| 2020-01-07
      
      § | 
    
  | 08:11 | <ayounsi@deploy1001> | Started deploy [librenms/librenms@7a0f7aa]: Upgrade LibreNMS to 1.59 - T241962 | [production] | 
            
  | 07:42 | <marostegui@cumin1001> | dbctl commit (dc=all): 'Repool es1019', diff saved to https://phabricator.wikimedia.org/P10055 and previous config saved to /var/cache/conftool/dbconfig/20200107-074159-marostegui.json | [production] | 
            
  | 07:40 | <marostegui@cumin1001> | dbctl commit (dc=all): 'Depool es1019 for upgrade', diff saved to https://phabricator.wikimedia.org/P10054 and previous config saved to /var/cache/conftool/dbconfig/20200107-074035-marostegui.json | [production] | 
            
  | 07:39 | <marostegui@cumin1001> | dbctl commit (dc=all): 'Repool es1013', diff saved to https://phabricator.wikimedia.org/P10053 and previous config saved to /var/cache/conftool/dbconfig/20200107-073922-marostegui.json | [production] | 
            
  | 07:35 | <marostegui@cumin1001> | dbctl commit (dc=all): 'Depool es1013 for upgrade', diff saved to https://phabricator.wikimedia.org/P10052 and previous config saved to /var/cache/conftool/dbconfig/20200107-073543-marostegui.json | [production] | 
            
  | 07:35 | <marostegui@cumin1001> | dbctl commit (dc=all): 'Repool es1018', diff saved to https://phabricator.wikimedia.org/P10051 and previous config saved to /var/cache/conftool/dbconfig/20200107-073508-marostegui.json | [production] | 
            
  | 07:29 | <marostegui@cumin1001> | dbctl commit (dc=all): 'Depool es1018 for upgrade', diff saved to https://phabricator.wikimedia.org/P10050 and previous config saved to /var/cache/conftool/dbconfig/20200107-072930-marostegui.json | [production] | 
            
  | 07:15 | <marostegui> | Remove partitions from s5: db2084:3315 T239453 | [production] | 
            
  | 07:12 | <marostegui> | Remove partitions from revision table on s6: db1098 T239453 | [production] | 
            
  | 07:08 | <marostegui@cumin1001> | dbctl commit (dc=all): 'Depool db1098:3316 - T239453', diff saved to https://phabricator.wikimedia.org/P10049 and previous config saved to /var/cache/conftool/dbconfig/20200107-070850-marostegui.json | [production] | 
            
  | 07:05 | <marostegui> | Depool labsdb1011 | [production] | 
            
  | 07:03 | <marostegui> | Deploy schema change on s8 codfw (this will generate lag on s8 codfw) - T234052 | [production] | 
            
  | 06:48 | <marostegui@cumin1001> | dbctl commit (dc=all): 'Repool db2089:3316 - T239453', diff saved to https://phabricator.wikimedia.org/P10048 and previous config saved to /var/cache/conftool/dbconfig/20200107-064846-marostegui.json | [production] | 
            
  | 01:17 | <dzahn@cumin1001> | END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) | [production] | 
            
  | 01:15 | <ebernhardson@deploy1001> | Synchronized wmf-config/InitialiseSettings.php: InitialiseSettings - clean up groupOverrides layout / spacing (sync again) (duration: 00m 53s) | [production] | 
            
  | 01:14 | <ebernhardson@deploy1001> | Synchronized wmf-config/InitialiseSettings.php: InitialiseSettings - clean up groupOverrides layout / spacing (duration: 00m 54s) | [production] | 
            
  | 01:12 | <mutante> | ganeti - creating urldownloader2002.wikimedia.org in codfw_B with 1 CPU, 1 GB RAM, 10 GB disk, public IP (T241979) | [production] | 
            
  | 01:12 | <dzahn@cumin1001> | START - Cookbook sre.ganeti.makevm | [production] | 
            
  | 01:09 | <dzahn@cumin1001> | END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) | [production] | 
            
  | 01:04 | <mutante> | ganeti - creating urldownloader2001.wikimedia.org in codfw_A with 1 CPU, 1 GB RAM, 10 GB disk, public IP (T241979) | [production] | 
            
  | 01:04 | <dzahn@cumin1001> | START - Cookbook sre.ganeti.makevm | [production] | 
            
  | 01:03 | <ebernhardson@deploy1001> | Synchronized wmf-config/InitialiseSettings.php: Revert: "cirrus: Shift more_like to codfw cirrus cluster" (duration: 00m 54s) | [production] | 
            
  | 01:02 | <dzahn@cumin1001> | END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) | [production] | 
            
  | 00:59 | <mutante> | ganeti - creating urldownloader1002.wikimedia.org in eqiad_C with 1 CPU, 1 GB RAM, 10 GB disk, public IP (T241979) | [production] | 
            
  | 00:58 | <mutante> | ganeti - creating urldownloader1001.wikimedia.org in eqiad_A with 1 CPU, 1 GB RAM, 10 GB disk, public IP (T241979) | [production] | 
            
  | 00:57 | <dzahn@cumin1001> | START - Cookbook sre.ganeti.makevm | [production] | 
            
  | 00:57 | <ebernhardson@deploy1001> | Synchronized wmf-config/CirrusSearch-common.php: Revert "reduce query load on cirrus elastic clusters" (duration: 00m 54s) | [production] | 
            
  | 00:57 | <dzahn@cumin1001> | END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) | [production] | 
            
  | 00:56 | <ebernhardson@deploy1001> | sync-file aborted: Revery  (duration: 00m 00s) | [production] | 
            
  | 00:52 | <dzahn@cumin1001> | START - Cookbook sre.ganeti.makevm | [production] | 
            
  | 00:46 | <tgr@deploy1001> | Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:561927|GrowthExperiments: use local search in production (T235717)]] (duration: 00m 54s) | [production] | 
            
  | 00:45 | <tgr@deploy1001> | Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: [[gerrit:561927|GrowthExperiments: use local search in production (T235717)]] (duration: 00m 58s) | [production] | 
            
  | 00:27 | <tgr@deploy1001> | Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:562359|Enable Partial Blocks on every wiki excluding those that have opted-out (T218626)]] (duration: 00m 55s) | [production] | 
            
  
    | 2020-01-06
      
      § | 
    
  | 23:49 | <ejegg> | updated payments-wiki from 827e3235dc to c3ca3ad6a7 | [production] | 
            
  | 23:12 | <mutante> | mailman - running /usr/local/sbin/rename_list wikimediamy wikimedia-my (T241988) | [production] | 
            
  | 22:34 | <eileen> | civicrm revision changed from b7746c31aa to 51b6fca9b2, config revision is b8af24d7c8 | [production] | 
            
  | 21:28 | <Amir1> | starting rebuild of holes in new term store from Q1Mio to Q10Mio using screen in mwmaint1002 (T219123) | [production] | 
            
  | 20:06 | <ejegg> | updated fundraising civicrm from 5642a92223 to b7746c31aa | [production] | 
            
  | 20:02 | <mutante> | LDAP - added 'krli' (Kris Litson) to 'wmde' and 'nda' for superset access (T241722) | [production] | 
            
  | 19:39 | <ebernhardson@deploy1001> | Finished deploy [search/mjolnir/deploy@41a22b8]: Bump to latest master (duration: 06m 57s) | [production] | 
            
  | 19:32 | <ebernhardson@deploy1001> | Started deploy [search/mjolnir/deploy@41a22b8]: Bump to latest master | [production] | 
            
  | 19:26 | <Urbanecm> | Morning SWAT done | [production] | 
            
  | 19:25 | <urbanecm@deploy1001> | Synchronized dblists/commonsuploads.dblist: SWAT: 0f045c3: Enable local uploads on inh.wiki (T239925) (duration: 00m 54s) | [production] | 
            
  | 19:22 | <urbanecm@deploy1001> | Synchronized wmf-config/InitialiseSettings.php: SWAT: 7722ff3: 0bff587: Add www.digital.archives.go.jp/mediaphoto.mnhn.fr to the wgCopyUploadsDomains (T238476, T241637) (duration: 00m 54s) | [production] | 
            
  | 19:19 | <urbanecm@deploy1001> | Synchronized wmf-config/throttle.php: SWAT: 1324af9: Add throttle rule for ECLAC editathon in Santiago, Chile (T241414) (duration: 00m 54s) | [production] | 
            
  | 19:14 | <urbanecm@deploy1001> | Synchronized wmf-config/InitialiseSettings.php: SWAT: c3a3248: Add sandboxlink for eswikivoyage (T241163) (duration: 00m 58s) | [production] | 
            
  | 19:08 | <urbanecm@deploy1001> | Synchronized wmf-config/InitialiseSettings.php: SWAT: d7a19ca: Enable GeoData extension in ruwikinews (T239000) (duration: 00m 56s) | [production] | 
            
  | 18:49 | <ebernhardson@deploy1001> | Finished deploy [search/airflow@8db442c]: match cryptography package with debian buster (duration: 00m 53s) | [production] | 
            
  | 18:48 | <ebernhardson@deploy1001> | Started deploy [search/airflow@8db442c]: match cryptography package with debian buster | [production] | 
            
  | 18:17 | <ebernhardson@deploy1001> | Finished deploy [search/airflow@8ae8500]: Require apache-airflow[kerberos] python package (duration: 00m 27s) | [production] |