|
2023-11-21
ยง
|
| 09:12 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P{lvs[4008-4009].ulsfo.wmnet} and A:lvs (T351069) |
[production] |
| 09:10 |
<vgutierrez> |
updating pybal to 1.5.14 on ulsfo - T351069 |
[production] |
| 09:09 |
<vgutierrez@cumin1001> |
START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P{lvs[4008-4009].ulsfo.wmnet} and A:lvs (T351069) |
[production] |
| 09:05 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'db2180 (re)pooling @ 45%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53661 and previous config saved to /var/cache/conftool/dbconfig/20231121-090547-arnaudb.json |
[production] |
| 09:05 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'db2178 (re)pooling @ 45%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53660 and previous config saved to /var/cache/conftool/dbconfig/20231121-090516-arnaudb.json |
[production] |
| 08:50 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'db2180 (re)pooling @ 30%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53659 and previous config saved to /var/cache/conftool/dbconfig/20231121-085042-arnaudb.json |
[production] |
| 08:50 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'db2178 (re)pooling @ 30%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53658 and previous config saved to /var/cache/conftool/dbconfig/20231121-085011-arnaudb.json |
[production] |
| 08:41 |
<vgutierrez> |
updating pybal to 1.5.14 on lvs4010 - T351069 |
[production] |
| 08:38 |
<awight> |
scap window cancelled due to k8s error |
[production] |
| 08:37 |
<awight@deploy2002> |
Finished scap: Backport for [[gerrit:976151|Revert "Enable Reference Previews on all wikis"]] (duration: 07m 08s) |
[production] |
| 08:37 |
<vgutierrez> |
upload pybal 1.15.14 to apt.wm.o (bullseye-wikimedia) - T348837 |
[production] |
| 08:35 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'db2180 (re)pooling @ 15%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53657 and previous config saved to /var/cache/conftool/dbconfig/20231121-083537-arnaudb.json |
[production] |
| 08:35 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'db2178 (re)pooling @ 15%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53656 and previous config saved to /var/cache/conftool/dbconfig/20231121-083504-arnaudb.json |
[production] |
| 08:32 |
<awight@deploy2002> |
awight and trainbranchbot: Continuing with sync |
[production] |
| 08:31 |
<awight@deploy2002> |
awight and trainbranchbot: Backport for [[gerrit:976151|Revert "Enable Reference Previews on all wikis"]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 08:31 |
<jmm@cumin2002> |
END (FAIL) - Cookbook sre.puppet.migrate-role (exit_code=99) for role: ncredir |
[production] |
| 08:30 |
<awight@deploy2002> |
Started scap: Backport for [[gerrit:976151|Revert "Enable Reference Previews on all wikis"]] |
[production] |
| 08:28 |
<awight@deploy2002> |
Sync cancelled. |
[production] |
| 08:28 |
<awight@deploy2002> |
wmde-fisch and awight: Backport for [[gerrit:971882|Enable Reference Previews on all wikis (T282999)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 08:27 |
<awight@deploy2002> |
Started scap: Backport for [[gerrit:971882|Enable Reference Previews on all wikis (T282999)]] |
[production] |
| 08:24 |
<awight@deploy2002> |
Finished scap: Backport for [[gerrit:971882|Enable Reference Previews on all wikis (T282999)]] (duration: 15m 02s) |
[production] |
| 08:20 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'db2180 (re)pooling @ 10%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53655 and previous config saved to /var/cache/conftool/dbconfig/20231121-082032-arnaudb.json |
[production] |
| 08:20 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'db2178 (re)pooling @ 10%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53654 and previous config saved to /var/cache/conftool/dbconfig/20231121-082000-arnaudb.json |
[production] |
| 08:18 |
<awight@deploy2002> |
awight and wmde-fisch: Continuing with sync |
[production] |
| 08:16 |
<stevemunene@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host druid1011.eqiad.wmnet with OS bullseye |
[production] |
| 08:11 |
<awight@deploy2002> |
awight and wmde-fisch: Backport for [[gerrit:971882|Enable Reference Previews on all wikis (T282999)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 08:09 |
<awight@deploy2002> |
Started scap: Backport for [[gerrit:971882|Enable Reference Previews on all wikis (T282999)]] |
[production] |
| 08:05 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'db2180 (re)pooling @ 5%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53653 and previous config saved to /var/cache/conftool/dbconfig/20231121-080527-arnaudb.json |
[production] |
| 08:04 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'db2178 (re)pooling @ 5%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53652 and previous config saved to /var/cache/conftool/dbconfig/20231121-080455-arnaudb.json |
[production] |
| 07:52 |
<stevemunene@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on druid1011.eqiad.wmnet with reason: host reimage |
[production] |
| 07:50 |
<stevemunene@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on druid1011.eqiad.wmnet with reason: host reimage |
[production] |
| 07:46 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1210.eqiad.wmnet with OS bookworm |
[production] |
| 07:27 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1210.eqiad.wmnet with reason: host reimage |
[production] |
| 07:25 |
<stevemunene@cumin1001> |
START - Cookbook sre.hosts.reimage for host druid1011.eqiad.wmnet with OS bullseye |
[production] |
| 07:25 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db1210.eqiad.wmnet with reason: host reimage |
[production] |
| 07:24 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Depooling db2155 (T348183)', diff saved to https://phabricator.wikimedia.org/P53651 and previous config saved to /var/cache/conftool/dbconfig/20231121-072424-arnaudb.json |
[production] |
| 07:24 |
<arnaudb@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance |
[production] |
| 07:24 |
<arnaudb@cumin1001> |
START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance |
[production] |
| 07:24 |
<arnaudb@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2155.codfw.wmnet with reason: Maintenance |
[production] |
| 07:23 |
<arnaudb@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2155.codfw.wmnet with reason: Maintenance |
[production] |
| 07:23 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2147 (T348183)', diff saved to https://phabricator.wikimedia.org/P53650 and previous config saved to /var/cache/conftool/dbconfig/20231121-072346-arnaudb.json |
[production] |
| 07:12 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.reimage for host db1210.eqiad.wmnet with OS bookworm |
[production] |
| 07:08 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P53649 and previous config saved to /var/cache/conftool/dbconfig/20231121-070840-arnaudb.json |
[production] |
| 07:06 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 33452 |
[production] |
| 07:05 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'configure' for AS: 33452 |
[production] |
| 06:53 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P53648 and previous config saved to /var/cache/conftool/dbconfig/20231121-065333-arnaudb.json |
[production] |
| 06:38 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2147 (T348183)', diff saved to https://phabricator.wikimedia.org/P53647 and previous config saved to /var/cache/conftool/dbconfig/20231121-063827-arnaudb.json |
[production] |
| 02:32 |
<ejegg> |
fundraising civicrm upgraded from b3da5d3f to 3a8558e7 |
[production] |
| 01:35 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Depooling db2147 (T348183)', diff saved to https://phabricator.wikimedia.org/P53646 and previous config saved to /var/cache/conftool/dbconfig/20231121-013514-arnaudb.json |
[production] |
| 01:35 |
<arnaudb@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2147.codfw.wmnet with reason: Maintenance |
[production] |