1351-1400 of 10000 results (80ms)
2023-10-11 §
02:30 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P52893 and previous config saved to /var/cache/conftool/dbconfig/20231011-023019-arnaudb.json [production]
02:18 <vriley@cumin1001> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cp1104.mgmt.eqiad.wmnet with reboot policy FORCED [production]
02:15 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2148 (T343198)', diff saved to https://phabricator.wikimedia.org/P52892 and previous config saved to /var/cache/conftool/dbconfig/20231011-021513-arnaudb.json [production]
02:03 <vriley@cumin1001> START - Cookbook sre.hosts.provision for host cp1104.mgmt.eqiad.wmnet with reboot policy FORCED [production]
02:02 <vriley@cumin1001> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp1104 [production]
02:01 <vriley@cumin1001> START - Cookbook sre.network.configure-switch-interfaces for host cp1104 [production]
2023-10-10 §
22:45 <brett@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5001.eqsin.wmnet with OS bookworm [production]
22:41 <pt1979@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudvirt1064.eqiad.wmnet with OS bullseye [production]
22:40 <cstone> SmashPig upgraded from a78a91d9 to 211284b9 [production]
22:13 <pt1979@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1064.eqiad.wmnet with OS bullseye [production]
21:45 <cmooney@cumin1001> END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f6-eqiad [production]
21:43 <cmooney@cumin1001> START - Cookbook sre.network.tls for network device lsw1-f6-eqiad [production]
21:34 <brett@cumin2002> START - Cookbook sre.hosts.reimage for host ncredir5001.eqsin.wmnet with OS bookworm [production]
21:33 <brett@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5001.eqsin.wmnet with OS bookworm [production]
20:48 <taavi@deploy2002> Finished scap: Backport for [[gerrit:963388|Set READ_NEW for CA wikis on OATHAuth multiple devices (T242031)]] (duration: 08m 24s) [production]
20:43 <taavi@deploy2002> taavi: Continuing with sync [production]
20:41 <taavi@deploy2002> taavi: Backport for [[gerrit:963388|Set READ_NEW for CA wikis on OATHAuth multiple devices (T242031)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
20:40 <taavi@deploy2002> Started scap: Backport for [[gerrit:963388|Set READ_NEW for CA wikis on OATHAuth multiple devices (T242031)]] [production]
20:19 <hmonroy@deploy2002> Finished scap: Backport for [[gerrit:964599|diffs: add line number headings to inline diffs (T346460)]] (duration: 30m 26s) [production]
20:17 <eileen> civicrm upgraded from 4329014b to f2f1e23e [production]
20:14 <brett@cumin2002> START - Cookbook sre.hosts.reimage for host ncredir5001.eqsin.wmnet with OS bookworm [production]
20:13 <brett@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host ncredir5001.eqsin.wmnet with OS bookworm [production]
20:07 <hmonroy@deploy2002> musikanimal and hmonroy: Continuing with sync [production]
20:07 <hmonroy@deploy2002> musikanimal and hmonroy: Backport for [[gerrit:964599|diffs: add line number headings to inline diffs (T346460)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
19:49 <hmonroy@deploy2002> Started scap: Backport for [[gerrit:964599|diffs: add line number headings to inline diffs (T346460)]] [production]
19:43 <arnaudb@cumin1001> dbctl commit (dc=all): 'Depooling db2148 (T343198)', diff saved to https://phabricator.wikimedia.org/P52890 and previous config saved to /var/cache/conftool/dbconfig/20231010-194311-arnaudb.json [production]
19:43 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2148.codfw.wmnet with reason: Maintenance [production]
19:42 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2148.codfw.wmnet with reason: Maintenance [production]
19:42 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2138:3312 (T343198)', diff saved to https://phabricator.wikimedia.org/P52889 and previous config saved to /var/cache/conftool/dbconfig/20231010-194249-arnaudb.json [production]
19:33 <jforrester@deploy2002> helmfile [codfw] DONE helmfile.d/services/mathoid: apply [production]
19:33 <jforrester@deploy2002> helmfile [codfw] START helmfile.d/services/mathoid: apply [production]
19:33 <jforrester@deploy2002> helmfile [eqiad] DONE helmfile.d/services/mathoid: apply [production]
19:32 <jforrester@deploy2002> helmfile [eqiad] START helmfile.d/services/mathoid: apply [production]
19:32 <jforrester@deploy2002> helmfile [staging] DONE helmfile.d/services/mathoid: apply [production]
19:31 <jforrester@deploy2002> helmfile [staging] START helmfile.d/services/mathoid: apply [production]
19:29 <cmooney@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 18 hosts with reason: changing bgp rr config [production]
19:29 <cmooney@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on 18 hosts with reason: changing bgp rr config [production]
19:29 <cmooney@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 6 hosts with reason: changing bgp rr config [production]
19:29 <cmooney@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on 6 hosts with reason: changing bgp rr config [production]
19:27 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2138:3312', diff saved to https://phabricator.wikimedia.org/P52888 and previous config saved to /var/cache/conftool/dbconfig/20231010-192742-arnaudb.json [production]
19:26 <jforrester@deploy2002> helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply [production]
19:26 <jforrester@deploy2002> helmfile [eqiad] START helmfile.d/services/wikifunctions: apply [production]
19:26 <jforrester@deploy2002> helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply [production]
19:25 <jforrester@deploy2002> helmfile [codfw] START helmfile.d/services/wikifunctions: apply [production]
19:24 <jforrester@deploy2002> helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [production]
19:23 <jforrester@deploy2002> helmfile [staging] START helmfile.d/services/wikifunctions: apply [production]
19:22 <jforrester@deploy2002> helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [production]
19:22 <jforrester@deploy2002> helmfile [staging] START helmfile.d/services/wikifunctions: apply [production]
19:14 <brett@cumin2002> START - Cookbook sre.hosts.reimage for host ncredir5001.eqsin.wmnet with OS bookworm [production]
19:12 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2138:3312', diff saved to https://phabricator.wikimedia.org/P52887 and previous config saved to /var/cache/conftool/dbconfig/20231010-191236-arnaudb.json [production]