| 2019-12-12
      
      ยง | 
    
  | 16:09 | <moritzm> | installing libvorbis security updates | [production] | 
            
  | 16:02 | <cdanis> | T229686 upgrade python3-conftool and python3-conftool-dbctl on cumin hosts | [production] | 
            
  | 16:01 | <cdanis> | sudo -E reprepro -C main include buster-wikimedia conftool_1.3.0-1+deb10u1_amd64.changes | [production] | 
            
  | 15:59 | <cdanis> | T229686 โ๏ธ cdanis@install1002.wikimedia.org ~ ๐โ sudo -E reprepro -C main include stretch-wikimedia conftool_1.3.0-1_amd64.changes | [production] | 
            
  | 15:18 | <jmm@cumin1001> | END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) | [production] | 
            
  | 15:18 | <jmm@cumin1001> | START - Cookbook sre.hosts.decommission | [production] | 
            
  | 15:13 | <moritzm> | deleting puppetdb1001 in Ganeti  T228657 | [production] | 
            
  | 14:56 | <reedy@deploy1001> | Synchronized php-1.35.0-wmf.10/includes/specials/SpecialUserrights.php: T240574 (duration: 01m 02s) | [production] | 
            
  | 14:44 | <mholloway-shell@deploy1001> | Finished deploy [mobileapps/deploy@7dc11d4]: Update mobileapps to 65272a6 (duration: 06m 12s) | [production] | 
            
  | 14:40 | <onimisionipe> | depool maps1001 for postgres reinitialization - T239728 | [production] | 
            
  | 14:38 | <mholloway-shell@deploy1001> | Started deploy [mobileapps/deploy@7dc11d4]: Update mobileapps to 65272a6 | [production] | 
            
  | 14:30 | <onimisionipe> | pool maps1004 osm-import is complete - T239728 | [production] | 
            
  | 14:21 | <otto@deploy1001> | helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'logging-external' . | [production] | 
            
  | 14:18 | <otto@deploy1001> | helmfile [CODFW] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'logging-external' . | [production] | 
            
  | 14:18 | <elukey@cumin1001> | END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) | [production] | 
            
  | 14:16 | <otto@deploy1001> | helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'logging-external' . | [production] | 
            
  | 14:08 | <marostegui> | Upgrade db2085 and db2086 | [production] | 
            
  | 14:02 | <jbond42> | merge puppet-merge refactor | [production] | 
            
  | 13:38 | <hashar> | contint1001 / contint2001 : upgraded Zuul to 2.5.1-wmf11 # T203846 | [production] | 
            
  | 12:58 | <elukey@cumin1001> | START - Cookbook sre.hadoop.roll-restart-workers | [production] | 
            
  | 12:39 | <Urbanecm> | EU SWAT done | [production] | 
            
  | 12:38 | <urbanecm@deploy1001> | Synchronized wmf-config/InitialiseSettings.php: SWAT: 07652a6: Add 2020: Wikimania namespace (T240339) (duration: 01m 02s) | [production] | 
            
  | 12:37 | <moritzm> | installing NSS security updates on buster | [production] | 
            
  | 12:34 | <urbanecm@deploy1001> | Synchronized wmf-config/InitialiseSettings.php: SWAT: 1c58f09: Enable SandboxLink extension on hywwiki (T239387) (duration: 01m 03s) | [production] | 
            
  | 11:49 | <moritzm> | removing puppetdb2001 from Ganeti | [production] | 
            
  | 11:46 | <jmm@cumin2001> | END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) | [production] | 
            
  | 11:45 | <jmm@cumin2001> | START - Cookbook sre.hosts.decommission | [production] | 
            
  | 11:41 | <hashar> | Removing zuul package from Jessie CI instances # T240551 | [production] | 
            
  | 11:17 | <addshore@deploy1001> | Synchronized php-1.35.0-wmf.10/extensions/Wikibase: BACKPORTS: wikibase tainted refs https://gerrit.wikimedia.org/r/#/q/topic:backports-wd-tainted-1 (duration: 01m 08s) | [production] | 
            
  | 09:46 | <moritzm> | upgrading recently reimaged stretch hosts back to puppet 5 / facter 3 T239832 | [production] | 
            
  | 09:37 | <marostegui> | Retroactive: deploy schema change on db1102:3314 | [production] | 
            
  | 08:40 | <eileen> | process-control config revision is 4d25b656e2 | [production] | 
            
  | 08:34 | <godog> | cleanup puppetmaster1001:/run/confd-template | [production] | 
            
  | 07:28 | <eileen> | process-control config revision is ad8bec977d | [production] | 
            
  | 07:10 | <marostegui> | Upgrade db1137 | [production] | 
            
  | 07:09 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Depool db1137 for upgrade (duration: 01m 03s) | [production] | 
            
  | 06:47 | <marostegui> | Upgrade db1117 | [production] | 
            
  | 06:11 | <aborrero@cumin1001> | END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) | [production] | 
            
  | 06:11 | <aborrero@cumin1001> | START - Cookbook sre.hosts.downtime | [production] | 
            
  | 05:58 | <marostegui@cumin1001> | END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) | [production] | 
            
  | 05:58 | <marostegui@cumin1001> | START - Cookbook sre.hosts.decommission | [production] | 
            
  | 05:57 | <aborrero@cumin1001> | END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) | [production] | 
            
  | 05:57 | <aborrero@cumin1001> | START - Cookbook sre.hosts.downtime | [production] | 
            
  | 05:57 | <aborrero@cumin1001> | END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) | [production] | 
            
  | 05:57 | <aborrero@cumin1001> | START - Cookbook sre.hosts.downtime | [production] | 
            
  | 05:56 | <aborrero@cumin1001> | END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) | [production] | 
            
  | 05:56 | <aborrero@cumin1001> | START - Cookbook sre.hosts.downtime | [production] | 
            
  | 05:56 | <aborrero@cumin1001> | END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) | [production] | 
            
  | 05:56 | <aborrero@cumin1001> | START - Cookbook sre.hosts.downtime | [production] | 
            
  | 05:47 | <marostegui> | Deploy schema change on db1102:3314 | [production] |