2301-2350 of 10000 results (27ms)
2020-12-07 ยง
23:12 <ryankemper@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
23:10 <ryankemper@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
23:07 <ryankemper@cumin2001> START - Cookbook sre.wdqs.data-transfer [production]
23:07 <ryankemper@cumin2001> START - Cookbook sre.wdqs.data-transfer [production]
23:07 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
23:07 <ryankemper@cumin1001> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) [production]
23:07 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
23:07 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
22:48 <sbassett> Re-deployed security patch for T120883 (v2) [production]
22:39 <sbassett> Undeployed security patch for T120883 as it caused several errors [production]
22:35 <sbassett> Deployed security patch for T120883 [production]
22:30 <ryankemper@cumin2001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
22:30 <ryankemper@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
22:21 <ryankemper@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
22:17 <ryankemper@cumin2001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
21:01 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
20:59 <ryankemper@cumin2001> START - Cookbook sre.wdqs.data-transfer [production]
20:58 <ryankemper@cumin2001> START - Cookbook sre.wdqs.data-transfer [production]
20:58 <ryankemper@cumin1001> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) [production]
20:58 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
20:58 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
20:27 <mutante> mwmaint1002 - systemctl reset-failed to clear icinga alert [production]
20:16 <mutante> mwmaint1002 - mediawiki_job_wikidata-updateQueryServiceLag job failed to run [production]
20:00 <ryankemper@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
19:58 <ryankemper@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
19:57 <ryankemper@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
19:55 <ryankemper@cumin2001> START - Cookbook sre.hosts.downtime [production]
19:55 <ryankemper@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
19:55 <ryankemper@cumin2001> START - Cookbook sre.hosts.downtime [production]
19:53 <ryankemper@cumin1001> START - Cookbook sre.hosts.downtime [production]
19:53 <ryankemper@cumin1001> START - Cookbook sre.hosts.downtime [production]
19:39 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
19:38 <ryankemper> T269204 reimaging the following instances to debian buster => `eqiad public`:`wdqs1006`, `codfw public`:`wdqs2003`, `codfw internal`:`wdqs2006`, `test`:`wdqs1009` [production]
19:37 <jiji@cumin1001> START - Cookbook sre.hosts.downtime [production]
19:23 <ejegg> updated standalone SmashPig payments listener from 3029b07004 to e3103b96ca [production]
19:21 <ppchelko@deploy1001> Synchronized wmf-config/CommonSettings.php: gerrit:646679 Enable OldRevisionParserCache on labs and group0, CS.php (duration: 00m 59s) [production]
19:19 <ppchelko@deploy1001> Synchronized wmf-config/InitialiseSettings.php: gerrit:646679 Enable OldRevisionParserCache on labs and group0 (duration: 01m 00s) [production]
19:08 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 3ba3af905251badeb546c17f996f0860a69024a1: Remove Growth Study Screener Quick Survey Config (T269369) (duration: 01m 02s) [production]
18:57 <ryankemper@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
18:57 <ryankemper@cumin2001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
18:57 <ryankemper@cumin2001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
18:57 <ryankemper@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
18:52 <herron> systemctl restart icinga on alert1001 T269560 [production]
18:52 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
18:51 <ryankemper@cumin2001> START - Cookbook sre.wdqs.data-transfer [production]
18:51 <ryankemper@cumin2001> START - Cookbook sre.wdqs.data-transfer [production]
18:51 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
18:51 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
18:50 <ryankemper> T246345 Brought new `wdqs-internal` node `wdqs1011` into service: `sudo confctl select 'name=wdqs1011.eqiad.wmnet' set/pooled=yes:weight=10` [production]
18:49 <ryankemper@mwmaint1002> conftool action : set/pooled=yes:weight=10; selector: name=wdqs1011.eqiad.wmnet [production]