|
2024-06-18
§
|
| 04:47 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Set s4 eqiad as read-only for maintenance - T367378', diff saved to https://phabricator.wikimedia.org/P65132 and previous config saved to /var/cache/conftool/dbconfig/20240618-044747-marostegui.json |
[production] |
| 04:47 |
<marostegui> |
Starting s4 eqiad failover from db1160 to db1238 - T367378 |
[production] |
| 04:21 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 33 hosts with reason: Primary switchover s4 T367378 |
[production] |
| 04:20 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Set db1238 with weight 0 T367378', diff saved to https://phabricator.wikimedia.org/P65131 and previous config saved to /var/cache/conftool/dbconfig/20240618-042054-marostegui.json |
[production] |
| 04:20 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 33 hosts with reason: Primary switchover s4 T367378 |
[production] |
| 04:02 |
<mwpresync@deploy1002> |
Pruned MediaWiki: 1.43.0-wmf.7 (duration: 02m 50s) |
[production] |
| 04:01 |
<mwpresync@deploy1002> |
Finished scap: testwikis wikis to 1.43.0-wmf.10 refs T361404 (duration: 58m 57s) |
[production] |
| 03:03 |
<mwpresync@deploy1002> |
Started scap: testwikis wikis to 1.43.0-wmf.10 refs T361404 |
[production] |
| 01:36 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1181 (T364069)', diff saved to https://phabricator.wikimedia.org/P65130 and previous config saved to /var/cache/conftool/dbconfig/20240618-013639-marostegui.json |
[production] |
| 01:36 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1181.eqiad.wmnet with reason: Maintenance |
[production] |
| 01:36 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1181.eqiad.wmnet with reason: Maintenance |
[production] |
| 01:36 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1174 (T364069)', diff saved to https://phabricator.wikimedia.org/P65129 and previous config saved to /var/cache/conftool/dbconfig/20240618-013616-marostegui.json |
[production] |
| 01:21 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P65128 and previous config saved to /var/cache/conftool/dbconfig/20240618-012109-marostegui.json |
[production] |
| 01:10 |
<brett@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=cp4044.ulsfo.wmnet |
[production] |
| 01:06 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P65127 and previous config saved to /var/cache/conftool/dbconfig/20240618-010601-marostegui.json |
[production] |
| 00:57 |
<brett@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4044.ulsfo.wmnet with OS bullseye |
[production] |
| 00:50 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1174 (T364069)', diff saved to https://phabricator.wikimedia.org/P65126 and previous config saved to /var/cache/conftool/dbconfig/20240618-005054-marostegui.json |
[production] |
| 00:34 |
<brett@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4044.ulsfo.wmnet with reason: host reimage |
[production] |
| 00:31 |
<brett@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cp4044.ulsfo.wmnet with reason: host reimage |
[production] |
| 00:28 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2204 (T352010)', diff saved to https://phabricator.wikimedia.org/P65125 and previous config saved to /var/cache/conftool/dbconfig/20240618-002823-ladsgroup.json |
[production] |
| 00:18 |
<zabe@deploy1002> |
Finished scap: Update interwiki cache (duration: 14m 03s) |
[production] |
| 00:13 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2204', diff saved to https://phabricator.wikimedia.org/P65124 and previous config saved to /var/cache/conftool/dbconfig/20240618-001316-ladsgroup.json |
[production] |
| 00:10 |
<brett@cumin2002> |
START - Cookbook sre.hosts.reimage for host cp4044.ulsfo.wmnet with OS bullseye |
[production] |
| 00:10 |
<brett@cumin2002> |
END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4044.ulsfo.wmnet with OS bullseye |
[production] |
| 00:05 |
<zabe> |
zabe@mwmaint1002:~$ mwscript extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --wiki=u4cwiki --cluster=all 2>&1 | tee /tmp/u4c.UpdateSearchIndexConfig.log # T366649 |
[production] |
| 00:04 |
<zabe@deploy1002> |
Started scap: Update interwiki cache |
[production] |
| 00:02 |
<zabe@deploy1002> |
Finished scap: T366649 (duration: 15m 16s) |
[production] |
| 00:00 |
<brett@cumin2002> |
START - Cookbook sre.hosts.reimage for host cp4044.ulsfo.wmnet with OS bullseye |
[production] |
|
2024-06-17
§
|
| 23:58 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2204', diff saved to https://phabricator.wikimedia.org/P65123 and previous config saved to /var/cache/conftool/dbconfig/20240617-235809-ladsgroup.json |
[production] |
| 23:52 |
<zabe@deploy1002> |
zabe: Continuing with sync |
[production] |
| 23:52 |
<brett@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp4044.ulsfo.wmnet |
[production] |
| 23:51 |
<zabe@deploy1002> |
zabe: T366649 synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 23:48 |
<zabe> |
zabe@mwmaint1002:~$ mwscript extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --wiki=arbcom_itwiki --cluster=all 2>&1 | tee /tmp/arbcom_it.UpdateSearchIndexConfig.log # T363825 |
[production] |
| 23:47 |
<zabe@deploy1002> |
Started scap: T366649 |
[production] |
| 23:46 |
<zabe> |
Create an 'Universal Code of Conduct Coordinating Committee (U4C)' private wiki # T366649 |
[production] |
| 23:44 |
<zabe@deploy1002> |
Finished scap: T363825 (duration: 15m 00s) |
[production] |
| 23:43 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2204 (T352010)', diff saved to https://phabricator.wikimedia.org/P65122 and previous config saved to /var/cache/conftool/dbconfig/20240617-234302-ladsgroup.json |
[production] |
| 23:34 |
<zabe@deploy1002> |
zabe: Continuing with sync |
[production] |
| 23:34 |
<zabe@deploy1002> |
zabe: T363825 synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 23:29 |
<zabe@deploy1002> |
Started scap: T363825 |
[production] |
| 23:29 |
<zabe> |
create private wiki for itwiki arbcom # T363825 |
[production] |
| 23:23 |
<cdobbins@cumin1002> |
conftool action : set/pooled=yes; selector: name=cp4043.ulsfo.wmnet |
[production] |
| 23:14 |
<cdobbins@cumin1002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4043.ulsfo.wmnet with OS bullseye |
[production] |
| 22:52 |
<cdobbins@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4043.ulsfo.wmnet with reason: host reimage |
[production] |
| 22:49 |
<cdobbins@cumin1002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cp4043.ulsfo.wmnet with reason: host reimage |
[production] |
| 22:42 |
<andrew@cumin1002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1041.eqiad.wmnet with OS bookworm |
[production] |
| 22:30 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1206 (T352010)', diff saved to https://phabricator.wikimedia.org/P65121 and previous config saved to /var/cache/conftool/dbconfig/20240617-223010-ladsgroup.json |
[production] |
| 22:28 |
<cdobbins@cumin1002> |
START - Cookbook sre.hosts.reimage for host cp4043.ulsfo.wmnet with OS bullseye |
[production] |
| 22:26 |
<cdobbins@cumin1002> |
END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4043.ulsfo.wmnet with OS bullseye |
[production] |
| 22:25 |
<eevans@cumin1002> |
END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching cassandra-dev200[2-3].codfw.wmnet: Apply Cassandra upgrade to 4.1.5 — T354970 - eevans@cumin1002 |
[production] |