7101-7150 of 10000 results (47ms)
2021-03-25 §
00:32 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2378.codfw.wmnet [production]
00:32 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2377.codfw.wmnet [production]
00:29 <legoktm> syncing facts for puppet-compiler [production]
00:23 <mutante> mw2377, mw2378 - reboot [production]
00:14 <twentyafterfour> phabricator update complete [production]
00:10 <twentyafterfour> deploying phabricator [production]
00:05 <ryankemper> T274204 `sudo -i cookbook sre.elasticsearch.rolling-upgrade search_eqiad "eqiad cluster reboot" --task-id T274204 --nodes-per-run 3 --start-datetime 2021-03-24T23:55:35` on `ryankemper@cumin1001` tmux session `elasticsearch_rolling_upgrade_reboots` [production]
2021-03-24 §
23:57 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2378.codfw.wmnet with reason: new_install [production]
23:57 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2378.codfw.wmnet with reason: new_install [production]
23:56 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2377.codfw.wmnet with reason: new_install [production]
23:56 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2377.codfw.wmnet with reason: new_install [production]
23:56 <ryankemper@cumin1001> START - Cookbook sre.elasticsearch.rolling-upgrade [production]
23:48 <mutante> generating new mcrouter certs for mw2377, mw2378 [production]
22:07 <ryankemper@cumin1001> END (PASS) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=0) [production]
22:07 <legoktm> disabled puppet on lists1002 while mailman3-web is broken [production]
21:49 <ryankemper@cumin2001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
21:19 <mutante> webperf2001 - restarted apache [production]
21:11 <hashar@deploy1002> Synchronized php: group1 wikis to 1.36.0-wmf.36 (duration: 01m 07s) [production]
21:10 <hashar@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.36.0-wmf.36 [production]
21:08 <kharlan@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . [production]
21:08 <kharlan@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . [production]
21:07 <hashar@deploy1002> Synchronized php-1.36.0-wmf.36/extensions/GrowthExperiments: LinkRecommendation: Modify path args for calls to API - T277865 (duration: 01m 07s) [production]
21:05 <hashar@deploy1002> Synchronized php-1.36.0-wmf.36/extensions/ProofreadPage: Revert "Add default TemplateStyles for an Index" - T278379 (duration: 01m 07s) [production]
21:04 <kharlan@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . [production]
21:04 <kharlan@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . [production]
21:02 <hashar@deploy1002> Synchronized php-1.36.0-wmf.36/extensions/GlobalUsage: Fix hook registration after class was namespaced - T278375 (duration: 01m 07s) [production]
20:59 <hashar@deploy1002> Synchronized wmf-config/env.php: multiversion: Move '@' operator in env.php closer to relevant statement (duration: 01m 07s) [production]
20:56 <kharlan@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . [production]
20:30 <kharlan@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . [production]
20:26 <kharlan@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . [production]
20:13 <ryankemper@cumin2001> START - Cookbook sre.wdqs.data-transfer [production]
20:13 <ryankemper@cumin2001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
20:10 <ryankemper@cumin2001> START - Cookbook sre.wdqs.data-transfer [production]
20:09 <ryankemper@cumin2001> START - Cookbook sre.wdqs.data-transfer [production]
20:07 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cumin2002.codfw.wmnet with reason: REIMAGE [production]
20:05 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime for 2:00:00 on cumin2002.codfw.wmnet with reason: REIMAGE [production]
19:59 <ryankemper@cumin2001> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) [production]
19:59 <ryankemper@cumin2001> START - Cookbook sre.wdqs.data-transfer [production]
19:56 <ryankemper> T267927 Host key is missing for `wdqs2008` leading to `data-transfer` cookbook failing, looking into resolving [production]
19:55 <ryankemper@cumin2001> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) [production]
19:55 <ryankemper@cumin2001> START - Cookbook sre.wdqs.data-transfer [production]
19:50 <ryankemper@cumin2001> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) [production]
19:50 <ryankemper@cumin2001> START - Cookbook sre.wdqs.data-transfer [production]
19:49 <ryankemper@cumin2001> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) [production]
19:49 <ryankemper@cumin2001> START - Cookbook sre.wdqs.data-transfer [production]
19:45 <ryankemper@cumin2001> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) [production]
19:45 <ryankemper@cumin2001> START - Cookbook sre.wdqs.data-transfer [production]
19:42 <ryankemper> T267927 Re-enabledpuppet on `wdqs2008` and ran puppet agent [production]
19:21 <ryankemper@cumin1001> START - Cookbook sre.elasticsearch.rolling-upgrade [production]
19:14 <hashar@deploy1002> rebuilt and synchronized wikiversions files: Revert group 1 to 1.36.0-wmf.35 [production]