|
2022-11-10
ยง
|
| 18:12 |
<ladsgroup@deploy1002> |
Started scap: Backport for [[gerrit:855666|Bump portals to HEAD (T273179)]] |
[production] |
| 18:09 |
<volans> |
upgrading spicerack to 5.0.0 on cumin hosts |
[production] |
| 18:05 |
<volans> |
uploaded spicerack_5.0.0 to apt.wikimedia.org bullseye-wikimedia |
[production] |
| 18:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2157 (T321130)', diff saved to https://phabricator.wikimedia.org/P39052 and previous config saved to /var/cache/conftool/dbconfig/20221110-180543-marostegui.json |
[production] |
| 18:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P39051 and previous config saved to /var/cache/conftool/dbconfig/20221110-180442-marostegui.json |
[production] |
| 18:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db2157 (T321130)', diff saved to https://phabricator.wikimedia.org/P39050 and previous config saved to /var/cache/conftool/dbconfig/20221110-180228-marostegui.json |
[production] |
| 18:02 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2157.codfw.wmnet with reason: Maintenance |
[production] |
| 18:02 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 5:00:00 on db2157.codfw.wmnet with reason: Maintenance |
[production] |
| 18:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2137:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P39049 and previous config saved to /var/cache/conftool/dbconfig/20221110-180206-marostegui.json |
[production] |
| 18:01 |
<volans> |
uploaded python3-gjson_0.3.0 to apt.wikimedia.org bullseye-wikimedia,unstable-wikimedia |
[production] |
| 17:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1135 (T321123)', diff saved to https://phabricator.wikimedia.org/P39048 and previous config saved to /var/cache/conftool/dbconfig/20221110-174935-marostegui.json |
[production] |
| 17:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1135 (T321123)', diff saved to https://phabricator.wikimedia.org/P39047 and previous config saved to /var/cache/conftool/dbconfig/20221110-174828-marostegui.json |
[production] |
| 17:48 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1135.eqiad.wmnet with reason: Maintenance |
[production] |
| 17:48 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db1135.eqiad.wmnet with reason: Maintenance |
[production] |
| 17:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1134 (T321123)', diff saved to https://phabricator.wikimedia.org/P39046 and previous config saved to /var/cache/conftool/dbconfig/20221110-174806-marostegui.json |
[production] |
| 17:47 |
<dcausse@deploy1002> |
Finished deploy [wikimedia/discovery/analytics@84dd7b5]: T320656: image_suggestions: schedule ad hoc dataset to fix improper suggestions (duration: 02m 18s) |
[production] |
| 17:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2137:3315', diff saved to https://phabricator.wikimedia.org/P39045 and previous config saved to /var/cache/conftool/dbconfig/20221110-174659-marostegui.json |
[production] |
| 17:44 |
<dcausse@deploy1002> |
Started deploy [wikimedia/discovery/analytics@84dd7b5]: T320656: image_suggestions: schedule ad hoc dataset to fix improper suggestions |
[production] |
| 17:37 |
<sukhe> |
[done] running sukhe@cumin2002:~$ homer "cr*-ulsfo*" commit "Gerrit 855583: sites.yaml: add lvs4008 (ulsfo hardware refresh)" |
[production] |
| 17:36 |
<sukhe> |
running sukhe@cumin2002:~$ homer "cr*-ulsfo*" commit "Gerrit 855583: sites.yaml: add lvs4008 (ulsfo hardware refresh)" |
[production] |
| 17:35 |
<sukhe@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs4008.ulsfo.wmnet with OS buster |
[production] |
| 17:34 |
<rzl> |
rzl@mwmaint1002:~$ sudo systemctl start mediawiki_job_growthexperiments-userImpactDelete.service # test run for T322706 T322541 |
[production] |
| 17:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1134', diff saved to https://phabricator.wikimedia.org/P39044 and previous config saved to /var/cache/conftool/dbconfig/20221110-173300-marostegui.json |
[production] |
| 17:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2137:3315', diff saved to https://phabricator.wikimedia.org/P39043 and previous config saved to /var/cache/conftool/dbconfig/20221110-173153-marostegui.json |
[production] |
| 17:28 |
<urandom> |
restarting bootstrap of aqs1016-a -- T307802 |
[production] |
| 17:26 |
<urandom> |
increasing stream throughput to 400mbit, aqs1011-{a,b} & aqs1013-{a,b} -- T307802 |
[production] |
| 17:26 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1194 (T322618)', diff saved to https://phabricator.wikimedia.org/P39042 and previous config saved to /var/cache/conftool/dbconfig/20221110-172611-ladsgroup.json |
[production] |
| 17:26 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1194.eqiad.wmnet with reason: Maintenance |
[production] |
| 17:25 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1194.eqiad.wmnet with reason: Maintenance |
[production] |
| 17:25 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1191 (T322618)', diff saved to https://phabricator.wikimedia.org/P39041 and previous config saved to /var/cache/conftool/dbconfig/20221110-172549-ladsgroup.json |
[production] |
| 17:23 |
<rzl> |
rzl@mwmaint1002:~$ sudo systemctl start mediawiki_job_growthexperiments-userImpactUpdateRecentlyEdited.service # test run for T322706 T322541 |
[production] |
| 17:18 |
<robh@cumin1001> |
END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ganeti1033'] |
[production] |
| 17:18 |
<rzl> |
rzl@mwmaint1002:~$ sudo systemctl start mediawiki_job_growthexperiments-userImpactUpdateRecentlyRegistered.service # test run for T322706 T322541 |
[production] |
| 17:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1134', diff saved to https://phabricator.wikimedia.org/P39040 and previous config saved to /var/cache/conftool/dbconfig/20221110-171753-marostegui.json |
[production] |
| 17:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2137:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P39039 and previous config saved to /var/cache/conftool/dbconfig/20221110-171646-marostegui.json |
[production] |
| 17:13 |
<sukhe@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs4008.ulsfo.wmnet with reason: host reimage |
[production] |
| 17:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db2137:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P39038 and previous config saved to /var/cache/conftool/dbconfig/20221110-171329-marostegui.json |
[production] |
| 17:13 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2137.codfw.wmnet with reason: Maintenance |
[production] |
| 17:13 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 5:00:00 on db2137.codfw.wmnet with reason: Maintenance |
[production] |
| 17:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2128 (T321130)', diff saved to https://phabricator.wikimedia.org/P39037 and previous config saved to /var/cache/conftool/dbconfig/20221110-171308-marostegui.json |
[production] |
| 17:10 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P39036 and previous config saved to /var/cache/conftool/dbconfig/20221110-171043-ladsgroup.json |
[production] |
| 17:10 |
<robh@cumin1001> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti1033'] |
[production] |
| 17:09 |
<robh@cumin1001> |
END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['ganeti1033'] |
[production] |
| 17:09 |
<sukhe@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on lvs4008.ulsfo.wmnet with reason: host reimage |
[production] |
| 17:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1134 (T321123)', diff saved to https://phabricator.wikimedia.org/P39035 and previous config saved to /var/cache/conftool/dbconfig/20221110-170247-marostegui.json |
[production] |
| 17:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1134 (T321123)', diff saved to https://phabricator.wikimedia.org/P39034 and previous config saved to /var/cache/conftool/dbconfig/20221110-170139-marostegui.json |
[production] |
| 17:01 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1134.eqiad.wmnet with reason: Maintenance |
[production] |
| 17:01 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db1134.eqiad.wmnet with reason: Maintenance |
[production] |
| 17:01 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1133.eqiad.wmnet with reason: Maintenance |
[production] |
| 17:01 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db1133.eqiad.wmnet with reason: Maintenance |
[production] |