|
2025-08-13
ยง
|
| 09:41 |
<btullis@cumin1003> |
START - Cookbook sre.dns.netbox |
[production] |
| 09:37 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2173 (T400854)', diff saved to https://phabricator.wikimedia.org/P81236 and previous config saved to /var/cache/conftool/dbconfig/20250813-093710-ladsgroup.json |
[production] |
| 09:34 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depooling db2173 (T400854)', diff saved to https://phabricator.wikimedia.org/P81235 and previous config saved to /var/cache/conftool/dbconfig/20250813-093423-ladsgroup.json |
[production] |
| 09:34 |
<ladsgroup@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2173.codfw.wmnet with reason: Maintenance |
[production] |
| 09:34 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2170 (T400854)', diff saved to https://phabricator.wikimedia.org/P81234 and previous config saved to /var/cache/conftool/dbconfig/20250813-093401-ladsgroup.json |
[production] |
| 09:33 |
<btullis@cumin1003> |
START - Cookbook sre.hosts.decommission for hosts an-worker1065.eqiad.wmnet |
[production] |
| 09:29 |
<vgutierrez> |
restarting varnish on cp5017 |
[production] |
| 09:18 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2170', diff saved to https://phabricator.wikimedia.org/P81233 and previous config saved to /var/cache/conftool/dbconfig/20250813-091853-ladsgroup.json |
[production] |
| 09:17 |
<btullis@cumin1003> |
END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts an-worker1065.eqiad.wmnet |
[production] |
| 09:17 |
<btullis@cumin1003> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts an-worker1065.eqiad.wmnet |
[production] |
| 09:11 |
<vgutierrez> |
restarting ATS on cp5017 |
[production] |
| 09:10 |
<urbanecm> |
Set newprojects mailman list to moderate posts from nonmembers (previous: discard) to debug an issue with new projects announcements (T393444) |
[production] |
| 09:06 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.addnode for new host ganeti1054.eqiad.wmnet to cluster eqiad and group A |
[production] |
| 09:06 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti1053.eqiad.wmnet to cluster eqiad and group A |
[production] |
| 09:03 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2170', diff saved to https://phabricator.wikimedia.org/P81231 and previous config saved to /var/cache/conftool/dbconfig/20250813-090346-ladsgroup.json |
[production] |
| 08:56 |
<cmooney@cumin1003> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 08:56 |
<cmooney@cumin1003> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add entries for nokia switches codfw - cmooney@cumin1003" |
[production] |
| 08:56 |
<cmooney@cumin1003> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add entries for nokia switches codfw - cmooney@cumin1003" |
[production] |
| 08:55 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.addnode for new host ganeti1053.eqiad.wmnet to cluster eqiad and group A |
[production] |
| 08:52 |
<cmooney@cumin1003> |
START - Cookbook sre.dns.netbox |
[production] |
| 08:37 |
<btullis@cumin1003> |
END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts an-worker1065.eqiad.wmnet |
[production] |
| 08:37 |
<btullis@cumin1003> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts an-worker1065.eqiad.wmnet |
[production] |
| 08:35 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti1053.eqiad.wmnet |
[production] |
| 08:34 |
<btullis@cumin1003> |
END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts an-worker1065.eqiad.wmnet |
[production] |
| 08:34 |
<btullis@cumin1003> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts an-worker1065.eqiad.wmnet |
[production] |
| 08:30 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P81227 and previous config saved to /var/cache/conftool/dbconfig/20250813-083023-ladsgroup.json |
[production] |
| 08:16 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P81226 and previous config saved to /var/cache/conftool/dbconfig/20250813-081516-ladsgroup.json |
[production] |
| 08:00 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2153 (T400854)', diff saved to https://phabricator.wikimedia.org/P81225 and previous config saved to /var/cache/conftool/dbconfig/20250813-080008-ladsgroup.json |
[production] |
| 07:57 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depooling db2153 (T400854)', diff saved to https://phabricator.wikimedia.org/P81224 and previous config saved to /var/cache/conftool/dbconfig/20250813-075721-ladsgroup.json |
[production] |
| 07:57 |
<ladsgroup@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2153.codfw.wmnet with reason: Maintenance |
[production] |
| 07:56 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2146 (T400854)', diff saved to https://phabricator.wikimedia.org/P81223 and previous config saved to /var/cache/conftool/dbconfig/20250813-075658-ladsgroup.json |
[production] |
| 07:52 |
<fabfur> |
manually upgrading haproxykafka on cp1111 to test new metrics (T400978) |
[production] |
| 07:49 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve1012.eqiad.wmnet |
[production] |
| 07:48 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1003.eqiad.wmnet |
[production] |
| 07:43 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host sretest1003.eqiad.wmnet |
[production] |
| 07:42 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ml-serve1012.eqiad.wmnet |
[production] |
| 07:41 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2146', diff saved to https://phabricator.wikimedia.org/P81222 and previous config saved to /var/cache/conftool/dbconfig/20250813-074150-ladsgroup.json |
[production] |
| 07:26 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2146', diff saved to https://phabricator.wikimedia.org/P81221 and previous config saved to /var/cache/conftool/dbconfig/20250813-072643-ladsgroup.json |
[production] |
| 07:15 |
<kartik@deploy1003> |
Finished scap sync-world: Backport for [[gerrit:1178036|Section Translation: Add Arakan Wikipedia (T392490)]] (duration: 11m 06s) |
[production] |
| 07:11 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2146 (T400854)', diff saved to https://phabricator.wikimedia.org/P81220 and previous config saved to /var/cache/conftool/dbconfig/20250813-071135-ladsgroup.json |
[production] |
| 07:10 |
<kartik@deploy1003> |
kartik: Continuing with sync |
[production] |
| 07:08 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depooling db2146 (T400854)', diff saved to https://phabricator.wikimedia.org/P81219 and previous config saved to /var/cache/conftool/dbconfig/20250813-070849-ladsgroup.json |
[production] |
| 07:08 |
<ladsgroup@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2146.codfw.wmnet with reason: Maintenance |
[production] |
| 07:08 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2145 (T400854)', diff saved to https://phabricator.wikimedia.org/P81218 and previous config saved to /var/cache/conftool/dbconfig/20250813-070826-ladsgroup.json |
[production] |
| 07:06 |
<kartik@deploy1003> |
kartik: Backport for [[gerrit:1178036|Section Translation: Add Arakan Wikipedia (T392490)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. |
[production] |
| 07:04 |
<kartik@deploy1003> |
Started scap sync-world: Backport for [[gerrit:1178036|Section Translation: Add Arakan Wikipedia (T392490)]] |
[production] |
| 06:53 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2145', diff saved to https://phabricator.wikimedia.org/P81216 and previous config saved to /var/cache/conftool/dbconfig/20250813-065318-ladsgroup.json |
[production] |
| 06:38 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2145', diff saved to https://phabricator.wikimedia.org/P81215 and previous config saved to /var/cache/conftool/dbconfig/20250813-063811-ladsgroup.json |
[production] |
| 06:23 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2145 (T400854)', diff saved to https://phabricator.wikimedia.org/P81214 and previous config saved to /var/cache/conftool/dbconfig/20250813-062303-ladsgroup.json |
[production] |
| 06:20 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depooling db2145 (T400854)', diff saved to https://phabricator.wikimedia.org/P81213 and previous config saved to /var/cache/conftool/dbconfig/20250813-062018-ladsgroup.json |
[production] |