| 
      
        2020-01-02
      
      §
     | 
  
    
  | 23:33 | 
  <ejegg> | 
  updated Fundraising CiviCRM from d534f4e966 to 217a1f8c63 | 
  [production] | 
            
  | 23:09 | 
  <ejegg> | 
  updated Fundraising CiviCRM from 6936aa0262 to d534f4e966 | 
  [production] | 
            
  | 22:44 | 
  <ejegg> | 
  updated fundraising CiviCRM from f4db7fdb31 to 6936aa0262 | 
  [production] | 
            
  | 20:48 | 
  <ejegg> | 
  updated Fundraising CiviCRM from abf0019c44 to f4db7fdb31 | 
  [production] | 
            
  | 20:30 | 
  <sbassett@deploy1001> | 
  Synchronized wmf-config/CommonSettings.php: Deploying revert of temporary patch for T241503 (permissions clean-up) (duration: 00m 53s) | 
  [production] | 
            
  | 19:57 | 
  <sbassett@deploy1001> | 
  Synchronized wmf-config/CommonSettings.php: Deploying temporary patch for T241503 (permissions clean-up) (duration: 00m 54s) | 
  [production] | 
            
  | 18:53 | 
  <ejegg> | 
  re-enabled fundraising cron jobs | 
  [production] | 
            
  | 18:29 | 
  <ejegg> | 
  disabled fundraising cron jobs | 
  [production] | 
            
  | 16:15 | 
  <moritzm> | 
  restarting Apache on graphite* hosts to pick up SASL security update | 
  [production] | 
            
  | 16:11 | 
  <moritzm> | 
  restarting Apache on webperf* hosts to pick up SASL security update | 
  [production] | 
            
  | 15:52 | 
  <moritzm> | 
  restarting Apache on puppetboard* hosts to pick up SASL security update | 
  [production] | 
            
  | 15:46 | 
  <moritzm> | 
  restarting FPM on parsoid canary to pick up SASL security update | 
  [production] | 
            
  | 14:27 | 
  <marostegui> | 
  Deploy schema change on s6 codfw master (db2129) with replication - T234052 | 
  [production] | 
            
  | 14:22 | 
  <marostegui> | 
  Deploy schema change on s5 eqiad hosts - T234052 | 
  [production] | 
            
  | 14:05 | 
  <moritzm> | 
  restarting PHP/Apache on mw canaries to pick up SASL security update | 
  [production] | 
            
  | 13:47 | 
  <moritzm> | 
  installing cyrus-sasl security updates on Stretch/Buster | 
  [production] | 
            
  | 13:23 | 
  <marostegui> | 
  Deploy schema change on s5 codfw master (db2123) with replication - T234052 | 
  [production] | 
            
  | 13:17 | 
  <moritzm> | 
  upgrading jessie servers to intel-microcode 3.20191115.2 | 
  [production] | 
            
  | 13:14 | 
  <foks> | 
  scramble password for Windy906 | 
  [production] | 
            
  | 13:00 | 
  <XioNoX> | 
  enable BFD traceoptions on cr1-eqiad and cr3-knams - T240659 | 
  [production] | 
            
  | 12:41 | 
  <moritzm> | 
  upgrade recently reimaged hosts to puppet 5 T239832 | 
  [production] | 
            
  | 12:32 | 
  <moritzm> | 
  upgrade recently reimaged hosts to facter 3 T239832 | 
  [production] | 
            
  | 12:09 | 
  <jmm@cumin2001> | 
  END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) | 
  [production] | 
            
  | 12:07 | 
  <jmm@cumin2001> | 
  START - Cookbook sre.hosts.downtime | 
  [production] | 
            
  | 11:53 | 
  <moritzm> | 
  restarting FPM on scandium to clear opcache health | 
  [production] | 
            
  | 11:42 | 
  <moritzm> | 
  reimaging mw2277 to validate fix for puppet5/facter3 installation on new installs T239832 | 
  [production] | 
            
  | 11:23 | 
  <arturo> | 
  import more openstack packages into stretch-wikimedia thirdparty/openstack-pike-stretch (T241347) | 
  [production] | 
            
  | 10:43 | 
  <filippo@cumin1001> | 
  END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) | 
  [production] | 
            
  | 10:40 | 
  <filippo@cumin1001> | 
  START - Cookbook sre.hosts.downtime | 
  [production] | 
            
  | 09:58 | 
  <filippo@cumin1001> | 
  END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) | 
  [production] | 
            
  | 09:58 | 
  <filippo@cumin1001> | 
  START - Cookbook sre.hosts.downtime | 
  [production] | 
            
  | 08:58 | 
  <marostegui@cumin1001> | 
  dbctl commit (dc=all): 'Depool db2076 T241647', diff saved to https://phabricator.wikimedia.org/P10021 and previous config saved to /var/cache/conftool/dbconfig/20200102-085806-marostegui.json | 
  [production] | 
            
  | 08:35 | 
  <marostegui> | 
  Upgrade db2090 | 
  [production] | 
            
  | 08:26 | 
  <marostegui> | 
  Upgrade db2075 | 
  [production] | 
            
  | 08:10 | 
  <marostegui> | 
  Deploy schema change on officewiki.flow_wiki_ref on s3 master (db1123) T241387 | 
  [production] | 
            
  | 07:49 | 
  <marostegui> | 
  Deploy schema change on techconductwiki.flow_wiki_ref (empty table) on s3 master (db1123) T241387 | 
  [production] | 
            
  | 07:26 | 
  <marostegui> | 
  Upgrade db2079 | 
  [production] | 
            
  | 07:18 | 
  <marostegui> | 
  Deploy schema change on labswiki.flow_wiki_ref (empty table) T241387 | 
  [production] | 
            
  | 06:46 | 
  <marostegui> | 
  Deploy schema change on db2131 - T241387 | 
  [production] | 
            
  | 06:44 | 
  <marostegui> | 
  Repool labsdb1009 | 
  [production] | 
            
  | 06:30 | 
  <marostegui> | 
  Upgrade labsdb1009 | 
  [production] | 
            
  | 06:29 | 
  <marostegui> | 
  Remove revision partitions from db2087:3316 T239453 | 
  [production] | 
            
  | 06:26 | 
  <marostegui@cumin1001> | 
  dbctl commit (dc=all): 'Depool db2087:3316 - T239453', diff saved to https://phabricator.wikimedia.org/P10020 and previous config saved to /var/cache/conftool/dbconfig/20200102-062650-marostegui.json | 
  [production] | 
            
  | 06:22 | 
  <marostegui> | 
  Depool labsdb1009 | 
  [production] | 
            
  | 00:22 | 
  <ejegg> | 
  re-enabled fundraising cron jobs | 
  [production] |