| 
      
        2019-08-21
      
      §
     | 
  
    
  | 15:07 | 
  <moritzm> | 
  installing python-cryptography update from Stretch point release | 
  [production] | 
            
  | 15:00 | 
  <jbond42> | 
  adding interface::add_ip6_mapped to media wiki servers | 
  [production] | 
            
  | 14:46 | 
  <elukey@deploy1001> | 
  Finished deploy [analytics/superset/deploy@868635a]: Upgrading superset to 0.34rc1 (duration: 00m 33s) | 
  [production] | 
            
  | 14:46 | 
  <elukey@deploy1001> | 
  Started deploy [analytics/superset/deploy@868635a]: Upgrading superset to 0.34rc1 | 
  [production] | 
            
  | 14:42 | 
  <moritzm> | 
  installing java-common update from Stretch point release | 
  [production] | 
            
  | 14:36 | 
  <moritzm> | 
  installing dns-root-data update from Stretch point release | 
  [production] | 
            
  | 14:29 | 
  <godog> | 
  silence average mw appserver latency alerts for 24h, too noisy | 
  [production] | 
            
  | 14:28 | 
  <elukey> | 
  swap turnilo backend in varnish from analytics-tool1002 to an-tool1007 | 
  [production] | 
            
  | 14:27 | 
  <moritzm> | 
  installing ca-certificates-java update from Stretch point release | 
  [production] | 
            
  | 14:10 | 
  <marostegui> | 
  Upgrade mysql on db2075 | 
  [production] | 
            
  | 13:12 | 
  <zfilipin@deploy1001> | 
  Synchronized php: group1 wikis to 1.34.0-wmf.19 (duration: 00m 55s) | 
  [production] | 
            
  | 13:11 | 
  <zfilipin@deploy1001> | 
  rebuilt and synchronized wikiversions files: group1 wikis to 1.34.0-wmf.19 | 
  [production] | 
            
  | 11:59 | 
  <jbond42> | 
  add ipv6 mapped address to mw codfw servers | 
  [production] | 
            
  | 11:41 | 
  <Amir1> | 
  EU SWAT is done | 
  [production] | 
            
  | 11:38 | 
  <jijiki> | 
  Restarting ores on ores1004 and ores1005 | 
  [production] | 
            
  | 11:37 | 
  <elukey> | 
  restart celery-ores-worker on ores1002 | 
  [production] | 
            
  | 10:57 | 
  <Urbanecm> | 
  Run scap pull on mwdebug1002 (T230601) | 
  [production] | 
            
  | 10:52 | 
  <Urbanecm> | 
  Move 0a87e3c's code to abusefilter.php on mwdebug1002 (T230601) | 
  [production] | 
            
  | 10:49 | 
  <Urbanecm> | 
  Previous log entry was for mwdebug1002 | 
  [production] | 
            
  | 10:49 | 
  <Urbanecm> | 
  Wrapped code added to CommonSettings.php in T230601 to wgExtensionFunctions | 
  [production] | 
            
  | 10:45 | 
  <Urbanecm> | 
  Run mwscript namespaceDupes.php --wiki=zhwikisource --add-prefix=FIXME --fix (T230548) | 
  [production] | 
            
  | 10:02 | 
  <moritzm> | 
  installing puppetdb1002 | 
  [production] | 
            
  | 09:46 | 
  <tarrow> | 
  finished enabling termbox on wikidatawiki | 
  [production] | 
            
  | 09:36 | 
  <tarrow@deploy1001> | 
  Synchronized wmf-config/InitialiseSettings.php: [[gerrit:531433|Enable Termbox on wikidatawiki (T230896)]] (duration: 00m 55s) | 
  [production] | 
            
  | 09:29 | 
  <moritzm> | 
  rebooting db2102 (reverting to a proper stretch 4.9 kernel, it used a bpo kernel due to some hardware debuging a while back) | 
  [production] | 
            
  | 09:20 | 
  <@> | 
  helmfile [EQIAD] Ran 'apply' command on namespace 'termbox' for release 'production' . | 
  [production] | 
            
  | 09:15 | 
  <@> | 
  helmfile [CODFW] Ran 'apply' command on namespace 'termbox' for release 'production' . | 
  [production] | 
            
  | 09:09 | 
  <jmm@cumin2001> | 
  END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) | 
  [production] | 
            
  | 09:09 | 
  <jmm@cumin2001> | 
  START - Cookbook sre.hosts.downtime | 
  [production] | 
            
  | 09:09 | 
  <@> | 
  helmfile [STAGING] Ran 'apply' command on namespace 'termbox' for release 'staging' . | 
  [production] | 
            
  | 09:07 | 
  <_joe_> | 
  uploaded python-poolcounter to stretch,buster | 
  [production] | 
            
  | 08:57 | 
  <@> | 
  helmfile [STAGING] Ran 'apply' command on namespace 'termbox' for release 'test' . | 
  [production] | 
            
  | 08:29 | 
  <moritzm> | 
  upgrading PHP on contint* | 
  [production] | 
            
  | 08:18 | 
  <moritzm> | 
  installing puppetdb2002 | 
  [production] | 
            
  | 08:11 | 
  <marostegui> | 
  Stop MySQL on db2052 T230883 | 
  [production] | 
            
  | 08:11 | 
  <marostegui> | 
  Remove db2052 from tendril and zarcillo T230883 | 
  [production] | 
            
  | 08:05 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Remove db2052 from config T230883 (duration: 00m 54s) | 
  [production] | 
            
  | 08:04 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-codfw.php: Remove db2052 from config T230883 (duration: 00m 54s) | 
  [production] | 
            
  | 07:58 | 
  <marostegui@cumin1001> | 
  dbctl commit (dc=all): 'Fully repool db1122', diff saved to https://phabricator.wikimedia.org/P8953 and previous config saved to /var/cache/conftool/dbconfig/20190821-075813-marostegui.json | 
  [production] | 
            
  | 07:56 | 
  <ema> | 
  upload@eqsin: rolling ats-backend-restart to enable compress plugin | 
  [production] | 
            
  | 05:45 | 
  <marostegui@cumin1001> | 
  dbctl commit (dc=all): 'More weight to db1122', diff saved to https://phabricator.wikimedia.org/P8952 and previous config saved to /var/cache/conftool/dbconfig/20190821-054542-marostegui.json | 
  [production] | 
            
  | 05:28 | 
  <eileen> | 
  civicrm revision changed from 0d1b7f107a to d7370a9d0b, config revision is 58cd6b7ae6 | 
  [production] | 
            
  | 05:26 | 
  <marostegui@cumin1001> | 
  dbctl commit (dc=all): 'More weight to db1122', diff saved to https://phabricator.wikimedia.org/P8951 and previous config saved to /var/cache/conftool/dbconfig/20190821-052613-marostegui.json | 
  [production] | 
            
  | 05:14 | 
  <marostegui@cumin1001> | 
  dbctl commit (dc=all): 'Slowly repool db1122 after restart', diff saved to https://phabricator.wikimedia.org/P8950 and previous config saved to /var/cache/conftool/dbconfig/20190821-051441-marostegui.json | 
  [production] | 
            
  | 05:05 | 
  <marostegui> | 
  Restart MySQL on db1122 for binlog format change - T230785 | 
  [production] | 
            
  | 05:05 | 
  <marostegui@cumin1001> | 
  dbctl commit (dc=all): 'Depool db1122 for binlog format change', diff saved to https://phabricator.wikimedia.org/P8949 and previous config saved to /var/cache/conftool/dbconfig/20190821-050501-marostegui.json | 
  [production] | 
            
  | 05:04 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Clarify db1122 status: candidate master for s2 - T230785 (duration: 00m 55s) | 
  [production] | 
            
  | 02:28 | 
  <ebernhardson@deploy1001> | 
  Finished deploy [search/mjolnir/deploy@adff5ad]: bulk_daemon: Handle non-integer status_code in json response (duration: 04m 09s) | 
  [production] | 
            
  | 02:24 | 
  <ebernhardson@deploy1001> | 
  Started deploy [search/mjolnir/deploy@adff5ad]: bulk_daemon: Handle non-integer status_code in json response | 
  [production] |