1-50 of 10000 results (71ms)
2023-10-10 ยง
22:45 <brett@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5001.eqsin.wmnet with OS bookworm [production]
22:41 <pt1979@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudvirt1064.eqiad.wmnet with OS bullseye [production]
22:40 <cstone> SmashPig upgraded from a78a91d9 to 211284b9 [production]
22:13 <pt1979@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1064.eqiad.wmnet with OS bullseye [production]
21:45 <cmooney@cumin1001> END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f6-eqiad [production]
21:43 <cmooney@cumin1001> START - Cookbook sre.network.tls for network device lsw1-f6-eqiad [production]
21:34 <brett@cumin2002> START - Cookbook sre.hosts.reimage for host ncredir5001.eqsin.wmnet with OS bookworm [production]
21:33 <brett@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5001.eqsin.wmnet with OS bookworm [production]
20:48 <taavi@deploy2002> Finished scap: Backport for [[gerrit:963388|Set READ_NEW for CA wikis on OATHAuth multiple devices (T242031)]] (duration: 08m 24s) [production]
20:43 <taavi@deploy2002> taavi: Continuing with sync [production]
20:41 <taavi@deploy2002> taavi: Backport for [[gerrit:963388|Set READ_NEW for CA wikis on OATHAuth multiple devices (T242031)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
20:40 <taavi@deploy2002> Started scap: Backport for [[gerrit:963388|Set READ_NEW for CA wikis on OATHAuth multiple devices (T242031)]] [production]
20:19 <hmonroy@deploy2002> Finished scap: Backport for [[gerrit:964599|diffs: add line number headings to inline diffs (T346460)]] (duration: 30m 26s) [production]
20:17 <eileen> civicrm upgraded from 4329014b to f2f1e23e [production]
20:14 <brett@cumin2002> START - Cookbook sre.hosts.reimage for host ncredir5001.eqsin.wmnet with OS bookworm [production]
20:13 <brett@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host ncredir5001.eqsin.wmnet with OS bookworm [production]
20:07 <hmonroy@deploy2002> musikanimal and hmonroy: Continuing with sync [production]
20:07 <hmonroy@deploy2002> musikanimal and hmonroy: Backport for [[gerrit:964599|diffs: add line number headings to inline diffs (T346460)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
19:49 <hmonroy@deploy2002> Started scap: Backport for [[gerrit:964599|diffs: add line number headings to inline diffs (T346460)]] [production]
19:43 <arnaudb@cumin1001> dbctl commit (dc=all): 'Depooling db2148 (T343198)', diff saved to https://phabricator.wikimedia.org/P52890 and previous config saved to /var/cache/conftool/dbconfig/20231010-194311-arnaudb.json [production]
19:43 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2148.codfw.wmnet with reason: Maintenance [production]
19:42 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2148.codfw.wmnet with reason: Maintenance [production]
19:42 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2138:3312 (T343198)', diff saved to https://phabricator.wikimedia.org/P52889 and previous config saved to /var/cache/conftool/dbconfig/20231010-194249-arnaudb.json [production]
19:33 <jforrester@deploy2002> helmfile [codfw] DONE helmfile.d/services/mathoid: apply [production]
19:33 <jforrester@deploy2002> helmfile [codfw] START helmfile.d/services/mathoid: apply [production]
19:33 <jforrester@deploy2002> helmfile [eqiad] DONE helmfile.d/services/mathoid: apply [production]
19:32 <jforrester@deploy2002> helmfile [eqiad] START helmfile.d/services/mathoid: apply [production]
19:32 <jforrester@deploy2002> helmfile [staging] DONE helmfile.d/services/mathoid: apply [production]
19:31 <jforrester@deploy2002> helmfile [staging] START helmfile.d/services/mathoid: apply [production]
19:29 <cmooney@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 18 hosts with reason: changing bgp rr config [production]
19:29 <cmooney@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on 18 hosts with reason: changing bgp rr config [production]
19:29 <cmooney@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 6 hosts with reason: changing bgp rr config [production]
19:29 <cmooney@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on 6 hosts with reason: changing bgp rr config [production]
19:27 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2138:3312', diff saved to https://phabricator.wikimedia.org/P52888 and previous config saved to /var/cache/conftool/dbconfig/20231010-192742-arnaudb.json [production]
19:26 <jforrester@deploy2002> helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply [production]
19:26 <jforrester@deploy2002> helmfile [eqiad] START helmfile.d/services/wikifunctions: apply [production]
19:26 <jforrester@deploy2002> helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply [production]
19:25 <jforrester@deploy2002> helmfile [codfw] START helmfile.d/services/wikifunctions: apply [production]
19:24 <jforrester@deploy2002> helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [production]
19:23 <jforrester@deploy2002> helmfile [staging] START helmfile.d/services/wikifunctions: apply [production]
19:22 <jforrester@deploy2002> helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [production]
19:22 <jforrester@deploy2002> helmfile [staging] START helmfile.d/services/wikifunctions: apply [production]
19:14 <brett@cumin2002> START - Cookbook sre.hosts.reimage for host ncredir5001.eqsin.wmnet with OS bookworm [production]
19:12 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2138:3312', diff saved to https://phabricator.wikimedia.org/P52887 and previous config saved to /var/cache/conftool/dbconfig/20231010-191236-arnaudb.json [production]
18:57 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2138:3312 (T343198)', diff saved to https://phabricator.wikimedia.org/P52886 and previous config saved to /var/cache/conftool/dbconfig/20231010-185730-arnaudb.json [production]
18:15 <bvibber> brion running TimedMediaHandler requeueTranscodes.php batch jobs on mwmaint2002. expect many deletions & new file stores on swift [production]
18:11 <ejegg> fundraising python tools upgraded from 2e19cd39 to 0c17296c [production]
18:10 <cmooney@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 6 hosts with reason: changing bgp rr config [production]
18:09 <cmooney@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on 6 hosts with reason: changing bgp rr config [production]
18:07 <cmooney@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 18 hosts with reason: changing bgp rr config [production]