|
2024-06-18
§
|
| 06:00 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1181 (T364069)', diff saved to https://phabricator.wikimedia.org/P65138 and previous config saved to /var/cache/conftool/dbconfig/20240618-060038-marostegui.json |
[production] |
| 05:55 |
<jynus@cumin2002> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2102.codfw.wmnet |
[production] |
| 05:55 |
<jynus@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 05:55 |
<jynus@cumin2002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2102.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin2002" |
[production] |
| 05:53 |
<jynus@cumin2002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2102.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin2002" |
[production] |
| 05:50 |
<jynus@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 05:45 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P65137 and previous config saved to /var/cache/conftool/dbconfig/20240618-054531-marostegui.json |
[production] |
| 05:44 |
<jynus@cumin2002> |
START - Cookbook sre.hosts.decommission for hosts db2102.codfw.wmnet |
[production] |
| 05:30 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P65136 and previous config saved to /var/cache/conftool/dbconfig/20240618-053024-marostegui.json |
[production] |
| 05:15 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1181 (T364069)', diff saved to https://phabricator.wikimedia.org/P65135 and previous config saved to /var/cache/conftool/dbconfig/20240618-051517-marostegui.json |
[production] |
| 05:00 |
<marostegui> |
dbmaint codfw s5 deploy schema change on db2213 T364299 |
[production] |
| 04:57 |
<marostegui> |
dbmaint eqiad s2 deploy schema change on db2207 T364299 |
[production] |
| 04:54 |
<marostegui> |
dbmaint eqiad s4 deploy schema change on db1160 T364299 |
[production] |
| 04:51 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1160.eqiad.wmnet with reason: Long schema change |
[production] |
| 04:51 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1160.eqiad.wmnet with reason: Long schema change |
[production] |
| 04:49 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db1160 T367378', diff saved to https://phabricator.wikimedia.org/P65134 and previous config saved to /var/cache/conftool/dbconfig/20240618-044908-root.json |
[production] |
| 04:48 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Promote db1238 to s4 primary and set section read-write T367378', diff saved to https://phabricator.wikimedia.org/P65133 and previous config saved to /var/cache/conftool/dbconfig/20240618-044806-marostegui.json |
[production] |
| 04:47 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Set s4 eqiad as read-only for maintenance - T367378', diff saved to https://phabricator.wikimedia.org/P65132 and previous config saved to /var/cache/conftool/dbconfig/20240618-044747-marostegui.json |
[production] |
| 04:47 |
<marostegui> |
Starting s4 eqiad failover from db1160 to db1238 - T367378 |
[production] |
| 04:21 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 33 hosts with reason: Primary switchover s4 T367378 |
[production] |
| 04:20 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Set db1238 with weight 0 T367378', diff saved to https://phabricator.wikimedia.org/P65131 and previous config saved to /var/cache/conftool/dbconfig/20240618-042054-marostegui.json |
[production] |
| 04:20 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 33 hosts with reason: Primary switchover s4 T367378 |
[production] |
| 04:02 |
<mwpresync@deploy1002> |
Pruned MediaWiki: 1.43.0-wmf.7 (duration: 02m 50s) |
[production] |
| 04:01 |
<mwpresync@deploy1002> |
Finished scap: testwikis wikis to 1.43.0-wmf.10 refs T361404 (duration: 58m 57s) |
[production] |
| 03:03 |
<mwpresync@deploy1002> |
Started scap: testwikis wikis to 1.43.0-wmf.10 refs T361404 |
[production] |
| 01:36 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1181 (T364069)', diff saved to https://phabricator.wikimedia.org/P65130 and previous config saved to /var/cache/conftool/dbconfig/20240618-013639-marostegui.json |
[production] |
| 01:36 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1181.eqiad.wmnet with reason: Maintenance |
[production] |
| 01:36 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1181.eqiad.wmnet with reason: Maintenance |
[production] |
| 01:36 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1174 (T364069)', diff saved to https://phabricator.wikimedia.org/P65129 and previous config saved to /var/cache/conftool/dbconfig/20240618-013616-marostegui.json |
[production] |
| 01:21 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P65128 and previous config saved to /var/cache/conftool/dbconfig/20240618-012109-marostegui.json |
[production] |
| 01:10 |
<brett@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=cp4044.ulsfo.wmnet |
[production] |
| 01:06 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P65127 and previous config saved to /var/cache/conftool/dbconfig/20240618-010601-marostegui.json |
[production] |
| 00:57 |
<brett@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4044.ulsfo.wmnet with OS bullseye |
[production] |
| 00:50 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1174 (T364069)', diff saved to https://phabricator.wikimedia.org/P65126 and previous config saved to /var/cache/conftool/dbconfig/20240618-005054-marostegui.json |
[production] |
| 00:34 |
<brett@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4044.ulsfo.wmnet with reason: host reimage |
[production] |
| 00:31 |
<brett@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cp4044.ulsfo.wmnet with reason: host reimage |
[production] |
| 00:28 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2204 (T352010)', diff saved to https://phabricator.wikimedia.org/P65125 and previous config saved to /var/cache/conftool/dbconfig/20240618-002823-ladsgroup.json |
[production] |
| 00:18 |
<zabe@deploy1002> |
Finished scap: Update interwiki cache (duration: 14m 03s) |
[production] |
| 00:13 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2204', diff saved to https://phabricator.wikimedia.org/P65124 and previous config saved to /var/cache/conftool/dbconfig/20240618-001316-ladsgroup.json |
[production] |
| 00:10 |
<brett@cumin2002> |
START - Cookbook sre.hosts.reimage for host cp4044.ulsfo.wmnet with OS bullseye |
[production] |
| 00:10 |
<brett@cumin2002> |
END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4044.ulsfo.wmnet with OS bullseye |
[production] |
| 00:05 |
<zabe> |
zabe@mwmaint1002:~$ mwscript extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --wiki=u4cwiki --cluster=all 2>&1 | tee /tmp/u4c.UpdateSearchIndexConfig.log # T366649 |
[production] |
| 00:04 |
<zabe@deploy1002> |
Started scap: Update interwiki cache |
[production] |
| 00:02 |
<zabe@deploy1002> |
Finished scap: T366649 (duration: 15m 16s) |
[production] |
| 00:00 |
<brett@cumin2002> |
START - Cookbook sre.hosts.reimage for host cp4044.ulsfo.wmnet with OS bullseye |
[production] |