2901-2950 of 10000 results (31ms)
2020-11-30 ยง
12:34 <hnowlan@deploy1001> Finished deploy [tilerator/deploy@97575e4]: Newer codfw maps hosts (duration: 00m 24s) [production]
12:34 <hnowlan@deploy1001> Started deploy [tilerator/deploy@97575e4]: Newer codfw maps hosts [production]
12:34 <hnowlan@deploy1001> Finished deploy [tilerator/deploy@97575e4]: Newer codfw maps hosts (duration: 00m 51s) [production]
12:33 <hnowlan@deploy1001> Started deploy [tilerator/deploy@97575e4]: Newer codfw maps hosts [production]
12:32 <Lucas_WMDE> Deployed patch for T260349 [production]
12:27 <hnowlan@deploy1001> Finished deploy [tilerator/deploy@97575e4]: New eqiad maps hosts (duration: 00m 03s) [production]
12:27 <hnowlan@deploy1001> Started deploy [tilerator/deploy@97575e4]: New eqiad maps hosts [production]
12:24 <hnowlan@deploy1001> Finished deploy [kartotherian/deploy@0a38bc5]: New eqiad maps hosts (duration: 00m 03s) [production]
12:24 <hnowlan@deploy1001> Started deploy [kartotherian/deploy@0a38bc5]: New eqiad maps hosts [production]
12:21 <hnowlan@puppetmaster1001> conftool action : set/pooled=no:weight=0; selector: dc=eqiad,cluster=maps,service=kartotherian,name=maps1005.eqiad.wmnet [production]
12:12 <urbanecm@deploy1001> Synchronized wmf-config/CommonSettings.php: 2922abe7b810f1b53b446af783dc9d51e6585225: Remove wgContentTranslationRESTBase config (T266213) (duration: 00m 57s) [production]
11:43 <marostegui> Sanitize clouddb1016:3318 - T267090 [production]
11:38 <ema> A:cp upgrade fifo-log-demux to 0.6.2 T268883 [production]
11:36 <jdrewniak@deploy1001> Synchronized portals: Wikimedia Portals Update: [[gerrit:644196| Bumping portals to master (T128546)]] (duration: 00m 57s) [production]
11:35 <jdrewniak@deploy1001> Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:644196| Bumping portals to master (T128546)]] (duration: 01m 01s) [production]
11:32 <ariel@deploy1001> Finished deploy [dumps/dumps@e8c6267]: allow page content fixup script to write output files to arbitrary dir (duration: 00m 04s) [production]
11:32 <ariel@deploy1001> Started deploy [dumps/dumps@e8c6267]: allow page content fixup script to write output files to arbitrary dir [production]
11:28 <ema> upload fifo-log-demux 0.6.2 to buster-wikimedia T268883 [production]
11:13 <marostegui@cumin1001> dbctl commit (dc=all): 'db1134 (re)pooling @ 100%: After schema change', diff saved to https://phabricator.wikimedia.org/P13473 and previous config saved to /var/cache/conftool/dbconfig/20201130-111321-root.json [production]
11:00 <hnowlan> bootstrapping maps1005 cassandra [production]
10:58 <marostegui@cumin1001> dbctl commit (dc=all): 'db1134 (re)pooling @ 75%: After schema change', diff saved to https://phabricator.wikimedia.org/P13472 and previous config saved to /var/cache/conftool/dbconfig/20201130-105818-root.json [production]
10:43 <marostegui@cumin1001> dbctl commit (dc=all): 'db1134 (re)pooling @ 50%: After schema change', diff saved to https://phabricator.wikimedia.org/P13471 and previous config saved to /var/cache/conftool/dbconfig/20201130-104314-root.json [production]
10:29 <ema@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) [production]
10:29 <marostegui> Compare data between clouddb1014:3312 clouddb1018:3312 labsdb1012 T267090 [production]
10:29 <marostegui> Compare data between clouddb1012:3312 clouddb1018:3312 labsdb1012 T267090 [production]
10:28 <marostegui@cumin1001> dbctl commit (dc=all): 'db1134 (re)pooling @ 25%: After schema change', diff saved to https://phabricator.wikimedia.org/P13470 and previous config saved to /var/cache/conftool/dbconfig/20201130-102811-root.json [production]
10:24 <akosiaris> applying https://gerrit.wikimedia.org/r/q/topic:%22k8s_config%22 series of patches [production]
10:18 <ema@cumin1001> START - Cookbook sre.hosts.reboot-single [production]
10:18 <ema> cp4031: reboot to test atsmtail/fifo-log-demux service dependencies -- https://gerrit.wikimedia.org/r/c/operations/puppet/+/643922 T256467 [production]
10:11 <ema> cp4032: upgrade varnish to 6.0.7-1wm1 T268736 [production]
10:06 <moritzm> installing NSS security updates [production]
09:57 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1134 for schema change', diff saved to https://phabricator.wikimedia.org/P13469 and previous config saved to /var/cache/conftool/dbconfig/20201130-095729-marostegui.json [production]
09:56 <marostegui@cumin1001> dbctl commit (dc=all): 'db1089 (re)pooling @ 100%: After schema change', diff saved to https://phabricator.wikimedia.org/P13468 and previous config saved to /var/cache/conftool/dbconfig/20201130-095621-root.json [production]
09:41 <marostegui@cumin1001> dbctl commit (dc=all): 'db1089 (re)pooling @ 75%: After schema change', diff saved to https://phabricator.wikimedia.org/P13467 and previous config saved to /var/cache/conftool/dbconfig/20201130-094117-root.json [production]
09:40 <marostegui> Stop MySQL on db1087 to clone clouddb1016:3318 T267090) [production]
09:39 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1087 from s8 and pool db1092 instead temporarily on vslow T267090', diff saved to https://phabricator.wikimedia.org/P13466 and previous config saved to /var/cache/conftool/dbconfig/20201130-093909-marostegui.json [production]
09:26 <marostegui@cumin1001> dbctl commit (dc=all): 'db1089 (re)pooling @ 50%: After schema change', diff saved to https://phabricator.wikimedia.org/P13465 and previous config saved to /var/cache/conftool/dbconfig/20201130-092614-root.json [production]
09:21 <marostegui@cumin1001> dbctl commit (dc=all): 'db1089+ (re)pooling @ 25%: After schema change', diff saved to https://phabricator.wikimedia.org/P13464 and previous config saved to /var/cache/conftool/dbconfig/20201130-092154-root.json [production]
08:51 <marostegui> Deploy schema change on db1089 [production]
08:51 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1089', diff saved to https://phabricator.wikimedia.org/P13463 and previous config saved to /var/cache/conftool/dbconfig/20201130-085101-marostegui.json [production]
08:41 <godog> swift eqiad-prod: add weight to ms-be106[0-3] - T268435 [production]
08:36 <marostegui> Compare data between clouddb1016:3315 labsdb1012 T267090 [production]
07:45 <marostegui@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) [production]
07:41 <marostegui@cumin1001> START - Cookbook sre.hosts.decommission [production]
07:25 <marostegui@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) [production]
07:18 <marostegui@cumin1001> START - Cookbook sre.hosts.decommission [production]
07:11 <marostegui> Deploy schema change on s1 codfw - T268004 [production]
07:05 <marostegui> Stop mysql on db1124:3318 to clone clouddb1016:3318, lag will show up on wikireplicas on s8 T267090 [production]
06:47 <marostegui@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) [production]
06:43 <marostegui@cumin1001> START - Cookbook sre.hosts.decommission [production]