3651-3700 of 10000 results (34ms)
2020-05-19 §
05:57 <volker-e@deploy1001> Finished deploy [design/style-guide@7bfbd2a]: Deploy design/style-guide: (duration: 00m 06s) [production]
05:57 <volker-e@deploy1001> Started deploy [design/style-guide@7bfbd2a]: Deploy design/style-guide: [production]
05:03 <marostegui@cumin1001> dbctl commit (dc=all): 'Set s2 and s8 as read-only=off for maintenance T251981', diff saved to https://phabricator.wikimedia.org/P11227 and previous config saved to /var/cache/conftool/dbconfig/20200519-050346-marostegui.json [production]
05:00 <marostegui@cumin1001> dbctl commit (dc=all): 'Set s2 and s8 as read-only for maintenance T251981', diff saved to https://phabricator.wikimedia.org/P11226 and previous config saved to /var/cache/conftool/dbconfig/20200519-050043-marostegui.json [production]
04:27 <marostegui> Repool labsdb1011 T249188 [production]
03:29 <volker-e@deploy1001> Finished deploy [design/style-guide@4b4bc51]: Deploy design/style-guide: (duration: 00m 07s) [production]
03:28 <volker-e@deploy1001> Started deploy [design/style-guide@4b4bc51]: Deploy design/style-guide: [production]
2020-05-18 §
23:50 <pt1979@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
23:47 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime [production]
23:25 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
23:23 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime [production]
23:12 <ryankemper> Restarted `wdqs-updater` across all wdqs nodes and restarted `wdqs-categories` across all nodes except 1010 (test wdqs server) and 1009 (automated deployment server) [production]
22:55 <Krinkle> Clear module_deps on dewiki (group2, old mw version, s5) to monitor regeneration [production]
22:48 <Krinkle> Clear module_deps on group0 (mostly s3) to monitor regeneration [production]
22:48 <Krinkle> Clear module_deps on commonswiki (group0, mostly s3) to monitor regeneration [production]
22:35 <Krinkle> Clear module_deps on commonswiki (group1, s4) to monitor regeneration [production]
22:33 <ryankemper@deploy1001> Finished deploy [wdqs/wdqs@4886dc3]: 0.3.32 (duration: 17m 12s) [production]
22:19 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
22:18 <Krinkle> Clear module_deps on s2 wikis to monitor regeneration [production]
22:16 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime [production]
22:15 <ryankemper@deploy1001> Started deploy [wdqs/wdqs@4886dc3]: 0.3.32 [production]
22:02 <Krinkle> Clear module_deps on hewiki (group1, s7) to monitor regeneration, ref T247028 [production]
21:40 <pt1979@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
21:37 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime [production]
21:23 <krinkle@deploy1001> Synchronized php-1.35.0-wmf.32/includes/resourceloader/dependencystore/: I015fa5885, I972a93806006 (duration: 01m 07s) [production]
21:14 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
21:11 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime [production]
20:27 <bsitzmann@deploy1001> Finished deploy [mobileapps/deploy@12efc14]: Update mobileapps to c960b349 (duration: 03m 31s) [production]
20:23 <bsitzmann@deploy1001> Started deploy [mobileapps/deploy@12efc14]: Update mobileapps to c960b349 [production]
19:07 <herron> performing rolling maintenance on kafka-main to pick up java security updates [production]
19:00 <krinkle@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Ic005093778d (duration: 01m 08s) [production]
18:58 <krinkle@deploy1001> Synchronized wmf-config/CommonSettings-labs.php: Ic005093778d (duration: 01m 06s) [production]
18:49 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
18:46 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime [production]
18:38 <volans> upgraded spicerack to 0.0.37-1 on cumin[12]001 [production]
18:24 <hnowlan@cumin1001> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) [production]
18:13 <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Fix English Wikipedia wordmark dimensions (T252143) (duration: 01m 06s) [production]
17:14 <XioNoX> update domain object for 56.15.185.in-addr.arpa - T247972 [production]
17:06 <bblack> dns1001 - removing downtimes, back in service - T241770 [production]
16:45 <bstorm_> updated views on labsdb1011 for the wb_terms changes T251598 [production]
16:32 <bblack@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
16:30 <bblack@cumin1001> START - Cookbook sre.hosts.downtime [production]
16:17 <bblack> dns1001 - reimaging for new NIC - T241770 [production]
16:10 <volans> uploaded spicerack_0.0.37-1_amd64.deb to apt.wikimedia.org stretch-wikimedia [production]
15:52 <hnowlan> rolling codfw cassandra for java security updates [production]
15:51 <hnowlan@cumin1001> START - Cookbook sre.cassandra.roll-restart [production]
15:30 <hnowlan@cumin1001> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) [production]
15:22 <elukey@cumin1001> END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) [production]
15:11 <Krinkle> krinkle@mc1021 Pruning the old `echo:seen:` Redis keys that didn't have a ttl yet [production]
14:57 <jayme@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'termbox' for release 'production' . [production]