51-100 of 10000 results (34ms)
2020-06-15 ยง
13:20 <hashar> Stopping zuul-merger on contint1001 to rebuild the virtualenv # T255424 [production]
13:19 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single [production]
13:04 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) [production]
12:58 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2091:3312, db2091:3314 - T253217', diff saved to https://phabricator.wikimedia.org/P11495 and previous config saved to /var/cache/conftool/dbconfig/20200615-125856-marostegui.json [production]
12:58 <vgutierrez> upgrade acme-chief to version 0.26 [production]
12:57 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single [production]
12:46 <vgutierrez> upload acme-chief 0.26 to apt.wm.o (buster) - T255249 [production]
12:43 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) [production]
12:38 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single [production]
12:34 <moritzm> rolling reboot on the ganeti cluster in eqsin (for security updates and to pick up the network changes to provides instances with a public IP) [production]
12:12 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
12:11 <marostegui> Upgrade db2134 [production]
12:09 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
12:01 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
11:59 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
11:57 <moritzm> reimaging sretest1002 to validate the reimage script on Buster [production]
11:43 <marostegui> Reimage dbproxy2003 which points to m3-master.codfw.wmnet (not in use) - T255408 [production]
11:40 <tgr@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:605543|GrowthExperiments: Switch on guidance feature (T239181)]] (duration: 00m 57s) [production]
11:10 <sukhe@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
11:10 <sukhe@cumin1001> START - Cookbook sre.hosts.downtime [production]
11:07 <hnowlan> regenerated certificates for restbase2009, restbase101[678], restbase201[012]. Did not roll-restart yet [production]
11:07 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) [production]
11:04 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single [production]
11:03 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
11:03 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
10:54 <moritzm> imported python-phabricator 0.7.0-2~wmf2 to apt.wikimedia.org/buster-wikimedia T245114 [production]
10:39 <jdrewniak@deploy1001> Synchronized portals: Wikimedia Portals Update: [[gerrit:605553| Bumping portals to master (605553)]] (duration: 00m 58s) [production]
10:38 <hnowlan> regenerated restbase2009's cassandra certificates [production]
10:38 <jdrewniak@deploy1001> Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:605553| Bumping portals to master (605553)]] (duration: 00m 58s) [production]
10:16 <jmm@cumin1001> END (ERROR) - Cookbook sre.hosts.reboot-single (exit_code=97) [production]
10:16 <jmm@cumin1001> START - Cookbook sre.hosts.reboot-single [production]
10:12 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T254820 [enwikivoyage] Undeploy the Listings extension (duration: 01m 00s) [production]
10:08 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:05 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:53 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:50 <filippo@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:46 <godog> run logstash benchmark on logstash1023 [production]
09:42 <volans> deploying esams mgmt DNS records automatically generated by Netbox ( operations/dns/+/604136/ ) - T233183 [production]
09:41 <volans@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
09:35 <volans@cumin1001> START - Cookbook sre.dns.netbox [production]
09:29 <elukey> update analytics-in4/6 filters on cr1-cr2 eqiad to update the Druid term (new nodes added) [production]
09:21 <jbond42> offlining puppetmaster1003 and 2003 for reboot [production]
09:17 <XioNoX> reduce ae device-count from 10 to 3 on asw2-a/b/c-eqiad [production]
09:14 <jmm@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:11 <jmm@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:55 <marostegui> Deploy schema change on db2123 (s5 codfw master) - T250066 [production]
08:50 <kart_> Updated cxserver to 2020-06-10-044445-production (T246319, T254959) [production]
08:46 <kartik@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'cxserver' for release 'production' . [production]
08:42 <kartik@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'cxserver' for release 'production' . [production]
08:39 <kartik@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'cxserver' for release 'staging' . [production]