| 2017-03-23
      
      § | 
    
  | 11:01 | <jynus@tin> | Synchronized wmf-config/db-codfw.php: Pool all es2XXX servers, depool es2014 for maintenance (duration: 00m 43s) | [production] | 
            
  | 10:59 | <hashar> | Actually restarting Jenkins for email plugins upgrades | [production] | 
            
  | 10:20 | <hashar> | Jenkins jobs got slightly blocked because I forgot to cancel the shutdown when jobs had to run. | [production] | 
            
  | 09:58 | <hashar> | Jenkins: upgrading plugins email-ext and mailer | [production] | 
            
  | 09:14 | <hashar> | Jenkins upgrading SSH Slaves plugin. Might cause disruption in CI | [production] | 
            
  | 08:47 | <moritzm> | repooled mw1261 now that T161095 is deployed | [production] | 
            
  | 08:29 | <marostegui> | Stop db1070 MySQL db1070 for maintenance - T137191 | [production] | 
            
  | 08:06 | <moritzm> | installing audiofile security updates | [production] | 
            
  | 07:37 | <marostegui> | Deploy schema change s4 on db1068 and labsdb1001 T160415 - T73563 | [production] | 
            
  | 07:36 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Repool db1068, depool db1059 T160415 - T73563 (duration: 00m 43s) | [production] | 
            
  | 07:08 | <marostegui> | Stop MySQL db1082 for maintenance - https://phabricator.wikimedia.org/T137191 | [production] | 
            
  | 06:52 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Depool db1082 - T137191 (duration: 00m 44s) | [production] | 
            
  | 03:14 | <l10nupdate@tin> | ResourceLoader cache refresh completed at Thu Mar 23 03:14:05 UTC 2017 (duration 5m 47s) | [production] | 
            
  | 03:08 | <l10nupdate@tin> | scap sync-l10n completed (1.29.0-wmf.17) (duration: 14m 39s) | [production] | 
            
  | 02:35 | <l10nupdate@tin> | scap sync-l10n completed (1.29.0-wmf.16) (duration: 13m 26s) | [production] | 
            
  | 01:46 | <mutante> | added ottomata and milimetric to "wmf-deployers" in Gerrit web ui, both have existing (deployment resp. root) shell already (T161157) | [production] | 
            
  
    | 2017-03-22
      
      § | 
    
  | 22:59 | <RainbowSprinkles> | gerrit: Quick service restart, picking up new config | [production] | 
            
  | 21:25 | <awight> | reenabling Jenkins orphan rectifier job | [production] | 
            
  | 21:18 | <andrewbogott> | rebooting labvirt1001 because it is being terrible.  https://phabricator.wikimedia.org/T159835 | [production] | 
            
  | 21:05 | <demon@tin> | Synchronized wmf-config/InitialiseSettings-labs.php: No-op, beta (duration: 00m 47s) | [production] | 
            
  | 21:00 | <demon@tin> | Synchronized wmf-config/CommonSettings-labs.php: No-op, beta (duration: 00m 43s) | [production] | 
            
  | 20:52 | <awight> | disabling Ingenico orphan rectifier | [production] | 
            
  | 20:43 | <thcipriani@tin> | rebuilt wikiversions.php and synchronized wikiversions files: Group0 to php-1.29.0-wmf.17 | [production] | 
            
  | 20:05 | <thcipriani@tin> | Synchronized php-1.29.0-wmf.17/extensions/ZeroPortal/includes/ApiZeroPortal.php: [[gerrit:344190|Failure to parse json config should result in a usable error]] T161036 (duration: 00m 42s) | [production] | 
            
  | 20:04 | <demon@tin> | Synchronized wmf-config/InitialiseSettings.php: Remove redundant whitelist read list for grantswiki (duration: 00m 44s) | [production] | 
            
  | 19:55 | <thcipriani@tin> | Synchronized php-1.29.0-wmf.17/extensions/Flow: [[gerrit:344188|Make sure topiclist queries always join against workflow table]] T121644 (duration: 00m 59s) | [production] | 
            
  | 19:45 | <thcipriani@tin> | Synchronized php-1.29.0-wmf.17/includes/Revision.php: [[gerrit:344077|Make Revision::getRevisionText() cache the converted text]] (duration: 00m 44s) | [production] | 
            
  | 19:44 | <mutante> | rsyncing /srv of netmon1001 to /srv/netmon1001 on gerrit2001 (T125020) | [production] | 
            
  | 19:37 | <jynus> | deploying m2 dns additions on codfw | [production] | 
            
  | 19:12 | <jynus@tin> | Synchronized wmf-config/db-eqiad.php: Pool db1094 with full weight (duration: 00m 43s) | [production] | 
            
  | 17:10 | <_joe_> | restarted ocg on ogc1001, not serving http queries | [production] | 
            
  | 16:55 | <jynus> | shutting down es2016's mariadb to clone to es2015 | [production] | 
            
  | 15:41 | <hashar@tin> | Synchronized php-1.29.0-wmf.16/languages/classes/LanguageKk.php: Check for string initialization in ucfirst() to make HHVM 3.18 happy - T161095 (duration: 00m 44s) | [production] | 
            
  | 15:40 | <hashar@tin> | Synchronized php-1.29.0-wmf.16/languages/classes/LanguageAz.php: Check for string initialization in ucfirst() to make HHVM 3.18 happy - T161095 (duration: 00m 48s) | [production] | 
            
  | 15:36 | <hashar> | Deploying LanguageAz.php and LanguageKk.php hotfix for HHVM 3.18 on mwdebug* and mw1261  - T161095 | [production] | 
            
  | 15:34 | <hashar@tin> | Synchronized php-1.29.0-wmf.17/languages/classes/LanguageKk.php: Check for string initialization in ucfirst() to make HHVM 3.18 happy - T161095 (duration: 00m 54s) | [production] | 
            
  | 15:33 | <hashar@tin> | Synchronized php-1.29.0-wmf.17/languages/classes/LanguageAz.php: Check for string initialization in ucfirst() to make HHVM 3.18 happy - T161095 (duration: 00m 59s) | [production] | 
            
  | 15:25 | <ema> | cp*: removed linux-image-amd64, linux-image-3.16.0-4-amd64 and linux-image-4.4.0-1-amd64 to reduce churn | [production] | 
            
  | 14:54 | <moritzm> | rebooting elastic2001 to Linux 4.9 | [production] | 
            
  | 14:35 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Restore db1087 original weight - T137191 (duration: 00m 44s) | [production] | 
            
  | 14:24 | <marostegui> | Deploy schema change s4 to db1068 - https://phabricator.wikimedia.org/T160415 https://phabricator.wikimedia.org/T73563 | [production] | 
            
  | 14:18 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Repool db1081, depool db1068 T160415 - T73563 (duration: 00m 43s) | [production] | 
            
  | 13:54 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Increase db1087 weight - T137191 (duration: 00m 47s) | [production] | 
            
  | 13:39 | <volans> | stopped ircecho to avoid the message spam | [production] | 
            
  | 13:15 | <dcausse> | eu swat done | [production] | 
            
  | 13:09 | <dcausse@tin> | Synchronized wmf-config/InitialiseSettings.php: [cirrus] Enable the completion suggester (duration: 00m 43s) | [production] | 
            
  | 13:07 | <bblack@puppetmaster1001> | conftool action : set/ttl=275; selector: dnsdisc=appservers-rw | [production] | 
            
  | 12:48 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Enable db1087 for API - T137191 (duration: 00m 42s) | [production] | 
            
  | 12:29 | <dcausse> | cirrus: reindexing lost writes (2017-03-21T13:30:00Z to 2017-03-21T17:50:00Z) during es5 upgrade in elastic@eqiad (T157479) | [production] | 
            
  | 12:26 | <marostegui> | Deploy schema change on s4 to db1081 and labsdb1011 - T160415 T73563 | [production] |