1-50 of 10000 results (20ms)
2021-03-24 ยง
23:57 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2378.codfw.wmnet with reason: new_install [production]
23:57 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2378.codfw.wmnet with reason: new_install [production]
23:56 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2377.codfw.wmnet with reason: new_install [production]
23:56 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2377.codfw.wmnet with reason: new_install [production]
23:56 <ryankemper@cumin1001> START - Cookbook sre.elasticsearch.rolling-upgrade [production]
23:48 <mutante> generating new mcrouter certs for mw2377, mw2378 [production]
22:07 <ryankemper@cumin1001> END (PASS) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=0) [production]
22:07 <legoktm> disabled puppet on lists1002 while mailman3-web is broken [production]
21:49 <ryankemper@cumin2001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
21:19 <mutante> webperf2001 - restarted apache [production]
21:11 <hashar@deploy1002> Synchronized php: group1 wikis to 1.36.0-wmf.36 (duration: 01m 07s) [production]
21:10 <hashar@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.36.0-wmf.36 [production]
21:08 <kharlan@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . [production]
21:08 <kharlan@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . [production]
21:07 <hashar@deploy1002> Synchronized php-1.36.0-wmf.36/extensions/GrowthExperiments: LinkRecommendation: Modify path args for calls to API - T277865 (duration: 01m 07s) [production]
21:05 <hashar@deploy1002> Synchronized php-1.36.0-wmf.36/extensions/ProofreadPage: Revert "Add default TemplateStyles for an Index" - T278379 (duration: 01m 07s) [production]
21:04 <kharlan@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . [production]
21:04 <kharlan@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . [production]
21:02 <hashar@deploy1002> Synchronized php-1.36.0-wmf.36/extensions/GlobalUsage: Fix hook registration after class was namespaced - T278375 (duration: 01m 07s) [production]
20:59 <hashar@deploy1002> Synchronized wmf-config/env.php: multiversion: Move '@' operator in env.php closer to relevant statement (duration: 01m 07s) [production]
20:56 <kharlan@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . [production]
20:30 <kharlan@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . [production]
20:26 <kharlan@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . [production]
20:13 <ryankemper@cumin2001> START - Cookbook sre.wdqs.data-transfer [production]
20:13 <ryankemper@cumin2001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
20:10 <ryankemper@cumin2001> START - Cookbook sre.wdqs.data-transfer [production]
20:09 <ryankemper@cumin2001> START - Cookbook sre.wdqs.data-transfer [production]
20:07 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cumin2002.codfw.wmnet with reason: REIMAGE [production]
20:05 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime for 2:00:00 on cumin2002.codfw.wmnet with reason: REIMAGE [production]
19:59 <ryankemper@cumin2001> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) [production]
19:59 <ryankemper@cumin2001> START - Cookbook sre.wdqs.data-transfer [production]
19:56 <ryankemper> T267927 Host key is missing for `wdqs2008` leading to `data-transfer` cookbook failing, looking into resolving [production]
19:55 <ryankemper@cumin2001> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) [production]
19:55 <ryankemper@cumin2001> START - Cookbook sre.wdqs.data-transfer [production]
19:50 <ryankemper@cumin2001> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) [production]
19:50 <ryankemper@cumin2001> START - Cookbook sre.wdqs.data-transfer [production]
19:49 <ryankemper@cumin2001> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) [production]
19:49 <ryankemper@cumin2001> START - Cookbook sre.wdqs.data-transfer [production]
19:45 <ryankemper@cumin2001> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) [production]
19:45 <ryankemper@cumin2001> START - Cookbook sre.wdqs.data-transfer [production]
19:42 <ryankemper> T267927 Re-enabledpuppet on `wdqs2008` and ran puppet agent [production]
19:21 <ryankemper@cumin1001> START - Cookbook sre.elasticsearch.rolling-upgrade [production]
19:14 <hashar@deploy1002> rebuilt and synchronized wikiversions files: Revert group 1 to 1.36.0-wmf.35 [production]
19:07 <hashar@deploy1002> Synchronized php: group1 wikis to 1.36.0-wmf.36 (duration: 01m 21s) [production]
19:05 <hashar@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.36.0-wmf.36 [production]
19:03 <urbanecm@deploy1002> Synchronized wmf-config/config/shwiki.yaml: 0f3aa7278d17c88f27b7d58ceede82730fd4ddcd: shwiki: Enable Growth features in dark mode (T278240; 3/3) (duration: 01m 08s) [production]
19:01 <urbanecm@deploy1002> Synchronized dblists/growthexperiments.dblist: 0f3aa7278d17c88f27b7d58ceede82730fd4ddcd: shwiki: Enable Growth features in dark mode (T278240; 2/3) (duration: 01m 06s) [production]
19:00 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 0f3aa7278d17c88f27b7d58ceede82730fd4ddcd: shwiki: Enable Growth features in dark mode (T278240; 1/3) (duration: 01m 07s) [production]
18:54 <urbanecm@deploy1002> Synchronized wmf-config/config/eswiki.yaml: ced092071a9638d1e1c04602bd5bbed5cc3812e3: Enable Growth features on eswiki in dark mode (T278235; 3/3) (duration: 01m 06s) [production]
18:53 <urbanecm@deploy1002> Synchronized dblists/growthexperiments.dblist: ced092071a9638d1e1c04602bd5bbed5cc3812e3: Enable Growth features on eswiki in dark mode (T278235; 2/3) (duration: 01m 07s) [production]