| 2018-07-30
      
      ยง | 
    
  | 15:27 | <jynus> | restart and upgrade mariadb at db1118 | [production] | 
            
  | 15:14 | <bblack> | reboot cp1075 | [production] | 
            
  | 15:08 | <godog> | upgrade hp raid firmware on ms-be1028 - T141756 | [production] | 
            
  | 15:05 | <Amir1> | running ladsgroup@mwmaint1001:~$ mwscript extensions/ORES/maintenance/PurgeScoreCache.php  on all wikis | [production] | 
            
  | 13:58 | <zfilipin@deploy1001> | rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.14 | [production] | 
            
  | 13:39 | <ema> | reboot cp2006 for kernel update | [production] | 
            
  | 12:59 | <zeljkof> | EU SWAT finished | [production] | 
            
  | 12:58 | <zfilipin@deploy1001> | Synchronized php-1.32.0-wmf.14/extensions/AbuseFilter/: SWAT: [[gerrit:448856|Fix jQuery selector when editing filters (T200604)]] (duration: 00m 55s) | [production] | 
            
  | 12:54 | <zfilipin@deploy1001> | Synchronized php-1.32.0-wmf.14/extensions/FileImporter/: SWAT: [[gerrit:448801|Fix flipped array indexes in template removal code (T200406)]] (duration: 00m 57s) | [production] | 
            
  | 12:45 | <godog> | fix group write on deploy1001 /srv/mediawiki-staging/php-1.32.0-wmf.14/.git/objects | [production] | 
            
  | 12:40 | <godog> | reboot ms-be2040 with page_poisoning=1 - T199198 | [production] | 
            
  | 12:33 | <tgr@deploy1001> | Finished scap: T190015 Create separate user group for editing sitewide CSS/JavaScript that does not include administrators by default (duration: 60m 10s) | [production] | 
            
  | 11:39 | <ema> | pool cp2006, upgraded to stretch T200445 | [production] | 
            
  | 11:39 | <moritzm> | upgrading intel-microcode  to 20180703  on the servers which have microcode updates enabled (reboots to pick up the new version will be coordinated separately) | [production] | 
            
  | 11:33 | <tgr@deploy1001> | Started scap: T190015 Create separate user group for editing sitewide CSS/JavaScript that does not include administrators by default | [production] | 
            
  | 11:12 | <ladsgroup@deploy1001> | Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:449137|Enable reading from change_tag_def everywhere (T199334)]] (duration: 00m 55s) | [production] | 
            
  | 11:10 | <volans> | upgrading cumin to 3.0.2 on the remaining cumin masters (neodymium, labpuppetmaster*) | [production] | 
            
  | 11:01 | <volans> | upgrading cumin to 3.0.2 on sarin | [production] | 
            
  | 10:50 | <volans> | uploaded cumin_3.0.2-1_amd64.deb to apt.wikimedia.org jessie-wikimedia | [production] | 
            
  | 10:25 | <godog> | run xfs_repair on sdc1 on ms-be2040 - T199198 | [production] | 
            
  | 10:04 | <akosiaris@deploy1001> | scap-helm help finished | [production] | 
            
  | 10:04 | <akosiaris@deploy1001> | scap-helm help cluster codfw completed | [production] | 
            
  | 10:04 | <akosiaris@deploy1001> | scap-helm help cluster eqiad completed | [production] | 
            
  | 10:04 | <akosiaris@deploy1001> | scap-helm help  [namespace: help, clusters: eqiad,codfw] | [production] | 
            
  | 10:03 | <akosiaris> | upgrade kubernetes staging cluster to 1.9.9 | [production] | 
            
  | 10:01 | <akosiaris> | reboot kubestage1001 for kubernetes upgrade | [production] | 
            
  | 09:59 | <akosiaris> | reboot kubestage1002 for kubernetes upgrade | [production] | 
            
  | 09:31 | <moritzm> | uploaded intel-microcode 20180703 for jessie-wikimedia/stretch-wikimedia to apt.wikimedia.org (tested successfully on a number of canary hosts for approx two weeks) | [production] | 
            
  | 09:30 | <jynus> | migrate dbtree to dbmonitor1001 | [production] | 
            
  | 09:25 | <akosiaris@deploy1001> | scap-helm mathoid finished | [production] | 
            
  | 09:14 | <akosiaris@deploy1001> | scap-helm mathoid cluster codfw completed | [production] | 
            
  | 09:13 | <akosiaris@deploy1001> | scap-helm mathoid cluster eqiad completed | [production] | 
            
  | 09:09 | <akosiaris@deploy1001> | scap-helm mathoid upgrade production stable/mathoid [namespace: mathoid, clusters: eqiad,codfw] | [production] | 
            
  | 09:08 | <akosiaris> | upgrade mathoid helm chart from 0.0.5 to 0.0.9 | [production] | 
            
  | 09:07 | <akosiaris@deploy1001> | scap-helm mathoid finished | [production] | 
            
  | 09:07 | <akosiaris@deploy1001> | scap-helm mathoid cluster codfw completed | [production] | 
            
  | 09:07 | <akosiaris@deploy1001> | scap-helm mathoid cluster eqiad completed | [production] | 
            
  | 09:07 | <akosiaris@deploy1001> | scap-helm mathoid upgrade -h [namespace: mathoid, clusters: eqiad,codfw] | [production] | 
            
  | 08:49 | <marostegui> | Deploy schema change on dbstore1002:s8 T144010 T51190 T199368 | [production] | 
            
  | 08:07 | <elukey@deploy1001> | Finished deploy [eventlogging/analytics@54d43e4]: Band aid for T200630 (duration: 00m 05s) | [production] | 
            
  | 08:07 | <elukey@deploy1001> | Started deploy [eventlogging/analytics@54d43e4]: Band aid for T200630 | [production] | 
            
  | 07:52 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Give some traffic to db1120 to on x1 (duration: 00m 53s) | [production] | 
            
  | 07:43 | <moritzm> | installing opencv security updates | [production] | 
            
  | 07:39 | <moritzm> | installing mercurial security updates | [production] | 
            
  | 07:32 | <marostegui> | Deploy schema change on db2045 (s8 codfw master) this will generate lag on s8 codfw T144010 T51190 T199368 | [production] | 
            
  | 07:29 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Add db1120 to x1 T196376 (duration: 00m 54s) | [production] | 
            
  | 07:28 | <marostegui@deploy1001> | Synchronized wmf-config/db-codfw.php: Add db1120 to x1 T196376 (duration: 00m 59s) | [production] | 
            
  | 05:28 | <marostegui> | Deploy schema change on db1068 (commonswiki master) - T51190 | [production] | 
            
  | 05:00 | <marostegui> | Deploy schema change on db1062 (s7 primary master) T144010 T51190 T199368 | [production] | 
            
  | 03:13 | <l10nupdate@deploy1001> | ResourceLoader cache refresh completed at Mon Jul 30 03:13:42 UTC 2018 (duration 10m 29s) | [production] |