1151-1200 of 10000 results (33ms)
2021-03-24 §
03:41 <ryankemper> T274204 Restarting `codfw` restart; the timestamp argument should prevent it from wasting time on nodes that have been rebooted already [production]
03:40 <ryankemper@cumin1001> START - Cookbook sre.elasticsearch.rolling-upgrade [production]
03:39 <ryankemper> T274204 Timed out waiting for write queues to empty: `[59/60, retrying in 60.00s] Attempt to run 'spicerack.elasticsearch_cluster.ElasticsearchClusters.wait_for_all_write_queues_empty' raised: Write queue not empty (had value of 241631) for partition 0 of topic codfw.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite.` [production]
03:38 <ryankemper@cumin1001> END (FAIL) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=99) [production]
02:38 <ryankemper> T274204 `sudo -i cookbook sre.elasticsearch.rolling-upgrade search_codfw "codfw cluster reboot" --task-id T274204 --nodes-per-run 3 --start-datetime 2021-03-24T02:29:39` on `ryankemper@cumin1001` tmux session `elasticsearch_rolling_upgrade_reboots` [production]
02:31 <ryankemper@cumin1001> START - Cookbook sre.elasticsearch.rolling-upgrade [production]
01:59 <ryankemper> T274204 For now I'll proceed to the reboots of `codfw` [production]
01:58 <ryankemper> T274204 `ctrl+c`'d out of run; relforge is relying on outdated config that is trying to talk to `relforge1002` which no longer exists. Need to refactor so that config no longer lives in spicerack [production]
01:58 <ryankemper@cumin1001> END (ERROR) - Cookbook sre.elasticsearch.rolling-upgrade-reboot (exit_code=97) [production]
01:49 <ryankemper> T274204 `sudo -i cookbook sre.elasticsearch.rolling-upgrade-reboot relforge "relforge cluster restarts" --task-id T274204 --nodes-per-run 3 --start-datetime 2021-03-24T01:45:59+00:00` on `ryankemper@cumin1001` tmux session `elasticsearch_rolling_upgrade_reboots` [production]
01:48 <ryankemper@cumin1001> START - Cookbook sre.elasticsearch.rolling-upgrade-reboot [production]
01:36 <eileen> civicrm revision changed from f36a0b08f0 to ad430721f6, config revision is 26b02db7ba [production]
00:22 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2378.codfw.wmnet with reason: REIMAGE [production]
00:18 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2378.codfw.wmnet with reason: REIMAGE [production]
00:18 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2377.codfw.wmnet with reason: REIMAGE [production]
00:16 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2377.codfw.wmnet with reason: REIMAGE [production]
2021-03-23 §
22:59 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pki-root1001.eqiad.wmnet with reason: REIMAGE [production]
22:57 <robh@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on pki-root1001.eqiad.wmnet with reason: REIMAGE [production]
22:33 <dwisehaupt> pushing 60f9baaf50b to fundraising hosts which will enable ssl by default for mysql client connections that use the host my.cnf file - T170321 [production]
22:19 <ebernhardson@deploy1002> Finished deploy [wikimedia/discovery/analytics@3fd7d7b]: partition ores dumps by namespace (duration: 02m 07s) [production]
22:17 <ebernhardson@deploy1002> Started deploy [wikimedia/discovery/analytics@3fd7d7b]: partition ores dumps by namespace [production]
22:09 <dzahn@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
22:05 <dzahn@cumin1001> START - Cookbook sre.dns.netbox [production]
21:27 <ppchelko@deploy1002> Finished deploy [restbase/deploy@531c474]: Add pageviews top-per-country endpoint (duration: 17m 58s) [production]
21:09 <ppchelko@deploy1002> Started deploy [restbase/deploy@531c474]: Add pageviews top-per-country endpoint [production]
21:04 <robh@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
21:00 <robh@cumin1001> START - Cookbook sre.dns.netbox [production]
20:59 <robh@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
20:40 <eileen> civicrm revision changed from 39d24e8b0a to f36a0b08f0, config revision is 26b02db7ba [production]
20:24 <robh@cumin1001> START - Cookbook sre.dns.netbox [production]
20:24 <robh@cumin1001> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
20:21 <robh@cumin1001> START - Cookbook sre.dns.netbox [production]
20:13 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts auth1002.eqiad.wmnet [production]
20:03 <robh@cumin1001> START - Cookbook sre.hosts.decommission for hosts auth1002.eqiad.wmnet [production]
20:02 <robh@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts auth1002.eqiad.wmnet [production]
20:01 <robh@cumin1001> START - Cookbook sre.hosts.decommission for hosts auth1002.eqiad.wmnet [production]
19:51 <jforrester@deploy1002> Finished deploy [integration/docroot@9de8c9d]: Add homer-public listing, added by volans (duration: 00m 08s) [production]
19:51 <jforrester@deploy1002> Started deploy [integration/docroot@9de8c9d]: Add homer-public listing, added by volans [production]
18:45 <otto@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Remove schema overrides for 6 finished EL migrations - T267347 T271164 T267351 T267348 T267343 T267353 (duration: 01m 07s) [production]
18:40 <legoktm@deploy1002> Synchronized php-1.36.0-wmf.36/vendor/: Bump wikimedia/parsoid to 0.13.0-a29 (duration: 01m 16s) [production]
18:20 <mbsantos@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'production' . [production]
18:18 <mbsantos@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'production' . [production]
18:16 <mbsantos@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'mobileapps' for release 'staging' . [production]
18:10 <legoktm@deploy1002> Synchronized wmf-config/ProductionServices.php: Add irc2001.wikimedia.org (running buster) as second irc server (T224579) (duration: 01m 08s) [production]
15:39 <akosiaris@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'cxserver' for release 'staging' . [production]
15:39 <akosiaris@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'cxserver' for release 'production' . [production]
15:38 <akosiaris@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'cxserver' for release 'production' . [production]
15:38 <akosiaris@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'cxserver' for release 'staging' . [production]
15:36 <akosiaris@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'cxserver' for release 'staging' . [production]
15:36 <akosiaris@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'cxserver' for release 'production' . [production]