| 
      
        2018-01-16
      
      ยง
     | 
  
    
  | 22:40 | 
  <mobrovac@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: Use EventBus for htmlCacheUpdate jobs for all wikis but en, commons and wikidata - T182023 (duration: 01m 12s) | 
  [production] | 
            
  | 22:39 | 
  <ppchelko@tin> | 
  Finished deploy [cpjobqueue/deploy@19b9bdd]: Switch htmlCacheUpdates for all but en, commons, wikidata T182023 (duration: 00m 35s) | 
  [production] | 
            
  | 22:39 | 
  <ppchelko@tin> | 
  Started deploy [cpjobqueue/deploy@19b9bdd]: Switch htmlCacheUpdates for all but en, commons, wikidata T182023 | 
  [production] | 
            
  | 22:19 | 
  <thcipriani@tin> | 
  Synchronized php-1.31.0-wmf.17/extensions/WikimediaMessages/WikimediaMessages.hooks.php: [[gerrit:404583|Update access to ORES isModelEnabled()]] (duration: 01m 13s) | 
  [production] | 
            
  | 22:19 | 
  <ottomata> | 
  apt-get install librdkafka1=0.9.4-1~jessie1 librdkafka++1=0.9.4-1~jessie1 on scb* to put librdkafka back at node-rdkafka compat version (somehow this was upgraded yesterday...very dangerous!!) | 
  [production] | 
            
  | 22:16 | 
  <thcipriani@tin> | 
  Finished scap: testwiki to php-1.31.0-wmf.17 and rebuild l10n cache (duration: 25m 45s) | 
  [production] | 
            
  | 21:50 | 
  <thcipriani@tin> | 
  Started scap: testwiki to php-1.31.0-wmf.17 and rebuild l10n cache | 
  [production] | 
            
  | 21:15 | 
  <andrewbogott> | 
  rebooting labvirt1014 and 1015 | 
  [production] | 
            
  | 21:04 | 
  <thcipriani@tin> | 
  rebuilt and synchronized wikiversions files: all wikis to 1.31.0-wmf.16 | 
  [production] | 
            
  | 20:59 | 
  <andrewbogott> | 
  rebooting labvirt1013 | 
  [production] | 
            
  | 20:42 | 
  <demon@tin> | 
  Finished scap: wmf.17 files, no bootstrap of i18n tho (x2) (duration: 06m 33s) | 
  [production] | 
            
  | 20:35 | 
  <demon@tin> | 
  Started scap: wmf.17 files, no bootstrap of i18n tho (x2) | 
  [production] | 
            
  | 20:34 | 
  <demon@tin> | 
  scap aborted: wmf.17 files, no bootstrap of i18n tho (duration: 08m 59s) | 
  [production] | 
            
  | 20:34 | 
  <andrewbogott> | 
  rebooting labvirt1011 | 
  [production] | 
            
  | 20:32 | 
  <herron> | 
  re-enabling puppet agents | 
  [production] | 
            
  | 20:25 | 
  <demon@tin> | 
  Started scap: wmf.17 files, no bootstrap of i18n tho | 
  [production] | 
            
  | 20:24 | 
  <herron> | 
  temporarily disabling puppet agents while troubleshooting puppet crl | 
  [production] | 
            
  | 20:07 | 
  <thcipriani@tin> | 
  rebuilt and synchronized wikiversions files: group1 to 1.31.0-wmf.16 | 
  [production] | 
            
  | 20:03 | 
  <andrewbogott> | 
  rebooting labvirt1009 | 
  [production] | 
            
  | 19:46 | 
  <andrewbogott> | 
  rebooting labvirt1008 | 
  [production] | 
            
  | 19:46 | 
  <thcipriani@tin> | 
  Synchronized php-1.31.0-wmf.16/includes/Storage/RevisionStore.php: [[gerrit:403930|RevisionStore, fix loadSlotContent with no $blobFlags]] T184749 (duration: 01m 13s) | 
  [production] | 
            
  | 19:30 | 
  <twentyafterfour> | 
  restarted wikibugs (several attempts, eventually it worked) | 
  [production] | 
            
  | 18:50 | 
  <chasemp> | 
  reboot labvirt1020 | 
  [production] | 
            
  | 18:44 | 
  <chasemp> | 
  reboot labvirt1019 | 
  [production] | 
            
  | 18:35 | 
  <andrewbogott> | 
  rebooting labvirt1007 | 
  [production] | 
            
  | 18:30 | 
  <arlolra@tin> | 
  Finished deploy [parsoid/deploy@1026fd2]: Updating Parsoid to 231bfff (duration: 13m 13s) | 
  [production] | 
            
  | 18:25 | 
  <herron> | 
  removing ganeti VM puppetcompiler1001 | 
  [production] | 
            
  | 18:19 | 
  <moritzm> | 
  rebooting labmon1002 for kernel security update | 
  [production] | 
            
  | 18:17 | 
  <arlolra@tin> | 
  Started deploy [parsoid/deploy@1026fd2]: Updating Parsoid to 231bfff | 
  [production] | 
            
  | 17:59 | 
  <moritzm> | 
  rebooting labnet100[34] and labcontrol100[34] for kernel security update | 
  [production] | 
            
  | 17:52 | 
  <herron> | 
  re-enabled puppet agents | 
  [production] | 
            
  | 17:50 | 
  <andrewbogott> | 
  rebooting labvirt1005 | 
  [production] | 
            
  | 17:45 | 
  <herron> | 
  disabled puppet agents troubleshooting T184444 | 
  [production] | 
            
  | 17:31 | 
  <andrewbogott> | 
  rebooting labvirt1004 | 
  [production] | 
            
  | 17:11 | 
  <andrewbogott> | 
  upgrading and rebooting labvirt1002 | 
  [production] | 
            
  | 17:06 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Restore db1092 original weight (duration: 01m 12s) | 
  [production] | 
            
  | 16:53 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Increase weight for db1092 (duration: 01m 12s) | 
  [production] | 
            
  | 16:40 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Increase weight for db1092 (duration: 01m 12s) | 
  [production] | 
            
  | 16:29 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Slowly repool db1092 - T174569 (duration: 01m 08s) | 
  [production] | 
            
  | 16:16 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Fully repool db1101:3317 (duration: 01m 12s) | 
  [production] | 
            
  | 16:11 | 
  <oblivian@neodymium> | 
  conftool action : set/pooled=no; selector: cluster=api_appserver,name=mw120[1-8]\.eqiad\.wmnet | 
  [production] | 
            
  | 16:10 | 
  <_joe_> | 
  depooling mw1201-1208 from the API cluster, T185004 | 
  [production] | 
            
  | 16:09 | 
  <moritzm> | 
  rebooting praseodymium for kernel security update | 
  [production] | 
            
  | 16:08 | 
  <godog> | 
  bootstrap cassandra-c on restbase1018 | 
  [production] | 
            
  | 16:04 | 
  <chasemp> | 
  add arturo to acl*operations-team | 
  [production] | 
            
  | 16:03 | 
  <moritzm> | 
  rebooting labweb* hosts for kernel security update | 
  [production] | 
            
  | 15:57 | 
  <andrewbogott> | 
  rebooting labvirt1001 | 
  [production] | 
            
  | 15:56 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Increase weight for db1101:3317 after kernel upgrade (duration: 01m 12s) | 
  [production] | 
            
  | 15:44 | 
  <elukey> | 
  reboot druid1003 for kernel upgrades | 
  [production] | 
            
  | 15:41 | 
  <moritzm> | 
  rebooting achernar for kernel security update | 
  [production] |