701-750 of 10000 results (61ms)
2022-08-11 §
00:58 <sukhe@puppetmaster1001> conftool action : set/pooled=no; selector: name=cp2042.codfw.wmnet,service=ats-be [production]
00:58 <sukhe@puppetmaster1001> conftool action : set/pooled=no; selector: name=cp2042.codfw.wmnet,service=ats-tls [production]
00:57 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on cp2042.codfw.wmnet with reason: host down; depooled and will debug tomorrow [production]
00:57 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on cp2042.codfw.wmnet with reason: host down; depooled and will debug tomorrow [production]
2022-08-10 §
21:25 <bking@cumin1001> conftool action : set/weight=10:pooled=yes; selector: name=wdqs1016.eqiad.wmnet [production]
21:23 <bking@cumin1001> conftool action : set/weight=10:pooled=yes; selector: name=wdqs1014.eqiad.wmnet [production]
21:10 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 16 hosts with reason: T309810 [production]
21:10 <bking@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on 16 hosts with reason: T309810 [production]
21:09 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on elastic[1101-1102].eqiad.wmnet with reason: T309810 [production]
21:09 <bking@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on elastic[1101-1102].eqiad.wmnet with reason: T309810 [production]
21:00 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
21:00 <cjming> end of UTC late backport window [production]
20:59 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
20:59 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
20:59 <cjming@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:820533|Remove unused $wgEnableMWSuggest]] (duration: 03m 04s) [production]
20:58 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
20:56 <cjming@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:820568|Enable new topic tool on dewiki (T313699)]] (duration: 03m 01s) [production]
20:34 <cjming@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:822093|testwiki: set $wgCdnMatchParameterOrder to false (T314868)]] (duration: 03m 20s) [production]
20:31 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
20:30 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
20:30 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
20:30 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
20:19 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
20:18 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
20:18 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
20:17 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
20:12 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
20:09 <bking@cumin1001> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) [production]
20:08 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
20:08 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
20:08 <cjming@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:820646|Start writing to cuc_actor everywhere except s4 and s8 (T233004)]] (duration: 03m 15s) [production]
20:07 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
19:51 <rzl@cumin1001> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for mc[2053-2054].codfw.wmnet [production]
19:51 <rzl@cumin1001> START - Cookbook sre.hosts.remove-downtime for mc[2053-2054].codfw.wmnet [production]
19:35 <rzl@cumin1001> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for parse[2019-2020].codfw.wmnet [production]
19:35 <rzl@cumin1001> START - Cookbook sre.hosts.remove-downtime for parse[2019-2020].codfw.wmnet [production]
19:35 <rzl@cumin1001> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for parse[2016-2018].codfw.wmnet [production]
19:35 <rzl@cumin1001> START - Cookbook sre.hosts.remove-downtime for parse[2016-2018].codfw.wmnet [production]
19:34 <rzl@cumin1001> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for mc2036.codfw.wmnet [production]
19:34 <rzl@cumin1001> START - Cookbook sre.hosts.remove-downtime for mc2036.codfw.wmnet [production]
19:28 <sukhe> testing ATS 9.1.3-1wm1 on cp4026: T309651 [production]
19:09 <ryankemper@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic1087.eqiad.wmnet with OS bullseye [production]
19:06 <ryankemper@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic1086.eqiad.wmnet with OS bullseye [production]
18:55 <ryankemper@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic1087.eqiad.wmnet with reason: host reimage [production]
18:51 <ryankemper@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic1086.eqiad.wmnet with reason: host reimage [production]
18:50 <ryankemper@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on elastic1087.eqiad.wmnet with reason: host reimage [production]
18:49 <ryankemper@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on elastic1086.eqiad.wmnet with reason: host reimage [production]
18:47 <bking@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
18:38 <ryankemper@cumin1001> START - Cookbook sre.hosts.reimage for host elastic1087.eqiad.wmnet with OS bullseye [production]
18:36 <ryankemper@cumin1001> START - Cookbook sre.hosts.reimage for host elastic1086.eqiad.wmnet with OS bullseye [production]