| 2019-09-26
      
      § | 
    
  | 06:29 | <marostegui@cumin1001> | dbctl commit (dc=all): ' Repool db2088:3312 db2084:3315 db2087:3316 db2086:3317 T233625', diff saved to https://phabricator.wikimedia.org/P9195 and previous config saved to /var/cache/conftool/dbconfig/20190926-062922-marostegui.json | [production] | 
            
  | 05:30 | <marostegui@cumin1001> | dbctl commit (dc=all): 'Fully pool db1081  - T230784', diff saved to https://phabricator.wikimedia.org/P9194 and previous config saved to /var/cache/conftool/dbconfig/20190926-053029-marostegui.json | [production] | 
            
  | 05:19 | <marostegui@cumin1001> | dbctl commit (dc=all): 'Increase weight for db1081 - T230784', diff saved to https://phabricator.wikimedia.org/P9193 and previous config saved to /var/cache/conftool/dbconfig/20190926-051916-marostegui.json | [production] | 
            
  | 05:09 | <marostegui@cumin1001> | dbctl commit (dc=all): 'Give some API weight to db1081 - T230784', diff saved to https://phabricator.wikimedia.org/P9192 and previous config saved to /var/cache/conftool/dbconfig/20190926-050937-marostegui.json | [production] | 
            
  | 05:07 | <marostegui@cumin1001> | dbctl commit (dc=all): 'Give some weight to db1081 - T230784', diff saved to https://phabricator.wikimedia.org/P9191 and previous config saved to /var/cache/conftool/dbconfig/20190926-050722-marostegui.json | [production] | 
            
  | 05:01 | <marostegui@cumin1001> | dbctl commit (dc=all): 'Promote db1138 to s4 master and remove read-only from s4 T230784', diff saved to https://phabricator.wikimedia.org/P9190 and previous config saved to /var/cache/conftool/dbconfig/20190926-050140-marostegui.json | [production] | 
            
  | 05:00 | <marostegui@cumin1001> | dbctl commit (dc=all): 'Set s4 as read-only for maintenance T230784', diff saved to https://phabricator.wikimedia.org/P9189 and previous config saved to /var/cache/conftool/dbconfig/20190926-050050-marostegui.json | [production] | 
            
  | 05:00 | <marostegui> | Starting s4 failover from db1081 to db1138 - T230784 | [production] | 
            
  | 04:15 | <marostegui@cumin1001> | dbctl commit (dc=all): 'Set db1138 with weight 0 T230784', diff saved to https://phabricator.wikimedia.org/P9188 and previous config saved to /var/cache/conftool/dbconfig/20190926-041508-marostegui.json | [production] | 
            
  | 04:10 | <marostegui> | Start pre-switchover s4 steps T230784 | [production] | 
            
  
    | 2019-09-25
      
      § | 
    
  | 21:59 | <bblack> | remove GRE MTU hacks on archiva1001 gerrit2001 cobalt install1002 - T232602 | [production] | 
            
  | 21:58 | <bblack> | remove GRE MTU hacks on eqiad caches (cp1xxx) - T232602 | [production] | 
            
  | 21:57 | <bblack> | remove GRE MTU hacks on esams caches (cp3xxx) - T232602 | [production] | 
            
  | 21:56 | <bblack> | remove GRE MTU hacks on eqsin caches (cp5xxx) - T232602 | [production] | 
            
  | 21:10 | <AndyRussG> | update fruec from 97128874bf to c591bd653b | [production] | 
            
  | 21:00 | <ejegg> | updated fundraising internal dashboard from 4473c65af0 to 69fdbec60d | [production] | 
            
  | 20:23 | <mholloway-shell@deploy1001> | Finished deploy [mobileapps/deploy@dbf4e7e]: Speed up querySelectors in domUtil (T229286) (duration: 05m 32s) | [production] | 
            
  | 20:20 | <hashar> | Upgrading CI Jenkins | [production] | 
            
  | 20:17 | <mholloway-shell@deploy1001> | Started deploy [mobileapps/deploy@dbf4e7e]: Speed up querySelectors in domUtil (T229286) | [production] | 
            
  | 19:28 | <twentyafterfour@deploy1001> | Synchronized php: group1 wikis to 1.34.0-wmf.24  refs T220749 (duration: 01m 03s) | [production] | 
            
  | 19:27 | <twentyafterfour@deploy1001> | rebuilt and synchronized wikiversions files: group1 wikis to 1.34.0-wmf.24  refs T220749 | [production] | 
            
  | 18:24 | <twentyafterfour@deploy1001> | Finished deploy [releng/phatality@42ba003]: trying again (duration: 03m 31s) | [production] | 
            
  | 18:21 | <twentyafterfour@deploy1001> | Started deploy [releng/phatality@42ba003]: trying again | [production] | 
            
  | 18:19 | <twentyafterfour@deploy1001> | Finished deploy [releng/phatality@42ba003]: deploy for version 5.6.15 (duration: 00m 50s) | [production] | 
            
  | 18:19 | <twentyafterfour@deploy1001> | Started deploy [releng/phatality@42ba003]: deploy for version 5.6.15 | [production] | 
            
  | 18:13 | <twentyafterfour@deploy1001> | Finished deploy [releng/phatality@8f05ba9]: Deploy phatality (duration: 00m 24s) | [production] | 
            
  | 18:13 | <twentyafterfour@deploy1001> | Started deploy [releng/phatality@8f05ba9]: Deploy phatality | [production] | 
            
  | 18:11 | <Amir1> | creating nqowiki is finished now | [production] | 
            
  | 18:10 | <ladsgroup@deploy1001> | Synchronized wmf-config/interwiki.php: Update interwiki cache (duration: 02m 39s) | [production] | 
            
  | 18:07 | <ladsgroup@deploy1001> | Synchronized dblists/rtl.dblist: Create nqowiki T230359 (duration: 01m 05s) | [production] | 
            
  | 18:01 | <Amir1> | creating nqowiki is going to take five more minutes | [production] | 
            
  | 17:57 | <ladsgroup@deploy1001> | Synchronized langlist: Create nqowiki T230359 (duration: 01m 02s) | [production] | 
            
  | 17:56 | <ladsgroup@deploy1001> | Synchronized static/images/project-logos/: Create nqowiki T230359 (duration: 01m 05s) | [production] | 
            
  | 17:54 | <ladsgroup@deploy1001> | Synchronized wmf-config/InitialiseSettings.php: Create nqowiki T230359 (duration: 01m 04s) | [production] | 
            
  | 17:51 | <ladsgroup@deploy1001> | rebuilt and synchronized wikiversions files: (no justification provided) | [production] | 
            
  | 17:47 | <ladsgroup@deploy1001> | Synchronized dblists: (no justification provided) (duration: 01m 04s) | [production] | 
            
  | 17:29 | <mutante> | DNS - adding nqo (N'Ko) to langlist for new nqo.wikipedia, approved by langcom https://meta.wikimedia.org/wiki/Requests_for_new_languages/Wikipedia_N'Ko (T230359) | [production] | 
            
  | 17:11 | <ladsgroup@deploy1001> | Synchronized php-1.34.0-wmf.23/extensions/WikimediaMaintenance/addWiki.php: Redefine RevisionStore service for the wiki being created (T212881) (duration: 01m 05s) | [production] | 
            
  | 17:08 | <ladsgroup@deploy1001> | Synchronized php-1.34.0-wmf.24/extensions/WikimediaMaintenance/addWiki.php: Redefine RevisionStore service for the wiki being created (T212881) (duration: 01m 04s) | [production] | 
            
  | 16:19 | <tgr@deploy1001> | Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:537801|GrowthExperiments: Enable WelcomeSurvey for euwiki (T233063)]] (duration: 01m 04s) | [production] | 
            
  | 16:06 | <kartik@deploy1001> | Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit|537628|Fix incorrect channel name for TranslationNotifications extension (T144780)]] (duration: 01m 06s) | [production] | 
            
  | 15:38 | <moritzm> | installing php5 security updates | [production] | 
            
  | 15:07 | <moritzm> | imported jenkins 2.176.4 for jessie/stretch T233214 | [production] | 
            
  | 14:57 | <filippo@cumin1001> | END (FAIL) - Cookbook sre.hosts.decommission (exit_code=True) | [production] | 
            
  | 14:57 | <filippo@cumin1001> | START - Cookbook sre.hosts.decommission | [production] | 
            
  | 14:55 | <ladsgroup@deploy1001> | Synchronized php-1.34.0-wmf.24/extensions/Wikibase/view/lib/resources.php: Revert "Merge valueview modules": T233800 (duration: 01m 04s) | [production] | 
            
  | 14:53 | <anomie@deploy1001> | Synchronized wmf-config/InitialiseSettings.php: Fix Draft namespace aliases (T233770) (duration: 01m 04s) | [production] | 
            
  | 14:52 | <onimisionipe> | pool wdqs1005 - lag issues have minimized. | [production] | 
            
  | 14:38 | <moritzm> | restarting apache on analytics-tool/an-tool to pick up Expat security update | [production] | 
            
  | 14:35 | <filippo@cumin1001> | END (FAIL) - Cookbook sre.hosts.decommission (exit_code=True) | [production] |