| 
      
        2018-04-23
      
      ยง
     | 
  
    
  | 17:24 | 
  <XioNoX> | 
  pushing firewall block on cr1/2-codfw - T175361 | 
  [production] | 
            
  | 17:18 | 
  <thcipriani@tin> | 
  Synchronized php: Group1 to 1.31.0-wmf.30 (duration: 01m 16s) | 
  [production] | 
            
  | 17:15 | 
  <thcipriani@tin> | 
  rebuilt and synchronized wikiversions files: Group1 to 1.31.0-wmf.30 | 
  [production] | 
            
  | 17:02 | 
  <vgutierrez> | 
  Depool and reimage lvs5002 as stretch - T191897 | 
  [production] | 
            
  | 16:15 | 
  <demon@tin> | 
  Pruned MediaWiki: 1.31.0-wmf.26 (duration: 03m 28s) | 
  [production] | 
            
  | 16:06 | 
  <marostegui> | 
  Deploy schema change on db1096:3316 - T191519 T188299 T190148 | 
  [production] | 
            
  | 16:06 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Depool db1096:3316 for alter table (duration: 01m 16s) | 
  [production] | 
            
  | 16:03 | 
  <gehel> | 
  restarting wdqs-updater on all nodes | 
  [production] | 
            
  | 15:55 | 
  <marostegui> | 
  Reload haproxy on dbproxy1010 to repool labsdb1010 | 
  [production] | 
            
  | 15:53 | 
  <bstorm_> | 
  Added slots, slot_roles, content and content_models to views on labsdb1010 | 
  [production] | 
            
  | 15:36 | 
  <dereckson@tin> | 
  Finished scap: Rebuild localisation cache to add Gorontalo (T189127) (duration: 08m 29s) | 
  [production] | 
            
  | 15:28 | 
  <dereckson@tin> | 
  Started scap: Rebuild localisation cache to add Gorontalo (T189127) | 
  [production] | 
            
  | 15:28 | 
  <dereckson@tin> | 
  scap aborted: Rebuild localisation cache to add Gorontalo (T189127)z (duration: 00m 01s) | 
  [production] | 
            
  | 15:28 | 
  <dereckson@tin> | 
  Started scap: Rebuild localisation cache to add Gorontalo (T189127)z | 
  [production] | 
            
  | 15:23 | 
  <dereckson@tin> | 
  scap sync-l10n completed (1.31.0-wmf.29) (duration: 00m 46s) | 
  [production] | 
            
  | 15:20 | 
  <dereckson@tin> | 
  Synchronized php-1.31.0-wmf.30/languages/messages/MessagesGor.php: Localisation for MediaWiki in Gorontalo (T189127) (duration: 01m 16s) | 
  [production] | 
            
  | 15:13 | 
  <dereckson@tin> | 
  Synchronized php-1.31.0-wmf.29/languages/messages/MessagesGor.php: Localisation for MediaWiki in Gorontalo (T189127) (duration: 01m 18s) | 
  [production] | 
            
  | 14:10 | 
  <ottomata> | 
  switching main -> analytics MirrorMaker to --new.consumer (temporarily stopping puppet on kafka101[234]) https://phabricator.wikimedia.org/T192387 | 
  [production] | 
            
  | 14:02 | 
  <zeljkof> | 
  EU SWAT finished | 
  [production] | 
            
  | 13:57 | 
  <zfilipin@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:428052|lfnwiki: add logo path and missing namespace names (T183561)]] (duration: 01m 15s) | 
  [production] | 
            
  | 13:55 | 
  <elukey> | 
  reimage analytics1067 to Debian Stretch - T192557 | 
  [production] | 
            
  | 13:53 | 
  <urandom> | 
  decommissioning Cassandra, restbase1010-c -- T189822 | 
  [production] | 
            
  | 13:47 | 
  <urandom> | 
  decommissioning Cassandra, restbase1010-c -- T189822 | 
  [production] | 
            
  | 13:43 | 
  <zfilipin@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:428055|euwikisource: add missing $wgMetaNamespace (T189465)]] (duration: 01m 16s) | 
  [production] | 
            
  | 13:35 | 
  <zfilipin@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:428053|gorwiki: add missing namespaces (T189109)]] (duration: 01m 17s) | 
  [production] | 
            
  | 13:28 | 
  <zfilipin@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:427940|Add logos for gorwiki (T192669)]] (duration: 01m 14s) | 
  [production] | 
            
  | 13:27 | 
  <zfilipin@tin> | 
  Synchronized static/images/project-logos/: SWAT: [[gerrit:427940|Add logos for gorwiki (T192669)]] (duration: 01m 16s) | 
  [production] | 
            
  | 13:17 | 
  <zfilipin@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:427956|Temp rate limit for arwiki due to mass vandalism (T192668)]] (duration: 01m 17s) | 
  [production] | 
            
  | 13:12 | 
  <jynus> | 
  restarting es2003 to test gerrit:427902 | 
  [production] | 
            
  | 12:59 | 
  <marostegui> | 
  Deploy schema change on dbstore1002 s6 - T191519 T188299 T190148 | 
  [production] | 
            
  | 12:58 | 
  <jynus> | 
  disabling puppet on several mysql hosts before deploying gerrit:427902 | 
  [production] | 
            
  | 12:40 | 
  <sbisson@tin> | 
  Finished deploy [kartotherian/deploy@2195dde]: Deploy kartotherian with new babel fallback rules (duration: 04m 52s) | 
  [production] | 
            
  | 12:35 | 
  <sbisson@tin> | 
  Started deploy [kartotherian/deploy@2195dde]: Deploy kartotherian with new babel fallback rules | 
  [production] | 
            
  | 11:50 | 
  <moritzm> | 
  reimaging mw1238,mw1239,mw1240 (app servers) to stretch | 
  [production] | 
            
  | 11:46 | 
  <moritzm> | 
  reimaging mw1285 (previous attempt had a hardware problem which failed to trigger the reboot via IPMI) ,mw1287,mw1288 (API servers) to stretch | 
  [production] | 
            
  | 11:41 | 
  <moritzm> | 
  installing poppler security updates | 
  [production] | 
            
  | 11:25 | 
  <mobrovac@tin> | 
  Finished deploy [citoid/deploy@b3c0818]: Add support for restful crossRef API and Wikidata QIDs - T108175 T176411 (duration: 03m 36s) | 
  [production] | 
            
  | 11:22 | 
  <mobrovac@tin> | 
  Started deploy [citoid/deploy@b3c0818]: Add support for restful crossRef API and Wikidata QIDs - T108175 T176411 | 
  [production] | 
            
  | 11:17 | 
  <mobrovac@tin> | 
  Finished deploy [restbase/deploy@3f3f989]: Add lfnwiki, inhwiki, gorwiki and euwikisource, take #2 - T192678 (duration: 07m 21s) | 
  [production] | 
            
  | 11:10 | 
  <mobrovac@tin> | 
  Started deploy [restbase/deploy@3f3f989]: Add lfnwiki, inhwiki, gorwiki and euwikisource, take #2 - T192678 | 
  [production] | 
            
  | 11:09 | 
  <mobrovac@tin> | 
  Finished deploy [restbase/deploy@3f3f989]: Add lfnwiki, inhwiki, gorwiki and euwikisource - T192678 (duration: 11m 47s) | 
  [production] | 
            
  | 11:00 | 
  <gehel> | 
  restarting wdqs updater on all wdqs notes | 
  [production] | 
            
  | 10:57 | 
  <mobrovac@tin> | 
  Started deploy [restbase/deploy@3f3f989]: Add lfnwiki, inhwiki, gorwiki and euwikisource - T192678 | 
  [production] | 
            
  | 10:26 | 
  <jdrewniak@tin> | 
  Synchronized portals: Wikimedia Portals Update: [[gerrit:428299|Bumping portals to master (T128546)]] (duration: 01m 16s) | 
  [production] | 
            
  | 10:25 | 
  <jdrewniak@tin> | 
  Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:428299|Bumping portals to master (T128546)]] (duration: 01m 17s) | 
  [production] | 
            
  | 09:56 | 
  <_joe_> | 
  restarting memcached on mc1020-1036 at 1 hour intervals - T184854 | 
  [production] | 
            
  | 09:13 | 
  <godog> | 
  Flashing Smart Array P840 in Slot 3 [ 4.52 -> 6.30 ] on ms-be2034 - T192721 T141756 | 
  [production] | 
            
  | 09:05 | 
  <_joe_> | 
  AMEND: restart memcached on mc1019 (T184854) | 
  [production] | 
            
  | 09:05 | 
  <_joe_> | 
  restart memcached on mw1019 (Ttail -f /var/log/etcdmirror-conftool-eqiad-wmnet/syslog.log | 
  [production] | 
            
  | 09:05 | 
  <vgutierrez> | 
  restarting pybal on lvs1006 | 
  [production] |