|
2024-06-21
§
|
| 01:45 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2193.codfw.wmnet with reason: Maintenance |
[production] |
| 01:45 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2180 (T367856)', diff saved to https://phabricator.wikimedia.org/P65282 and previous config saved to /var/cache/conftool/dbconfig/20240621-014523-marostegui.json |
[production] |
| 01:30 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P65281 and previous config saved to /var/cache/conftool/dbconfig/20240621-013016-marostegui.json |
[production] |
| 01:15 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P65280 and previous config saved to /var/cache/conftool/dbconfig/20240621-011509-marostegui.json |
[production] |
| 01:00 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2180 (T367856)', diff saved to https://phabricator.wikimedia.org/P65279 and previous config saved to /var/cache/conftool/dbconfig/20240621-010002-marostegui.json |
[production] |
| 00:52 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1236 (T352010)', diff saved to https://phabricator.wikimedia.org/P65278 and previous config saved to /var/cache/conftool/dbconfig/20240621-005237-ladsgroup.json |
[production] |
| 00:37 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1236', diff saved to https://phabricator.wikimedia.org/P65277 and previous config saved to /var/cache/conftool/dbconfig/20240621-003730-ladsgroup.json |
[production] |
| 00:22 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1236', diff saved to https://phabricator.wikimedia.org/P65276 and previous config saved to /var/cache/conftool/dbconfig/20240621-002223-ladsgroup.json |
[production] |
| 00:08 |
<mutante> |
[cp3072:~] $ sudo systemctl start varnishkafka-webrequest.service |
[production] |
| 00:08 |
<mutante> |
[cp3067:~] $ sudo systemctl start logrotate |
[production] |
| 00:07 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1236 (T352010)', diff saved to https://phabricator.wikimedia.org/P65275 and previous config saved to /var/cache/conftool/dbconfig/20240621-000716-ladsgroup.json |
[production] |
| 00:00 |
<sukhe> |
restarting haproxy on cp3068 and cp3072 |
[production] |
|
2024-06-20
§
|
| 23:47 |
<zabe@deploy1002> |
Finished scap: Update interwiki cache (duration: 10m 12s) |
[production] |
| 23:36 |
<zabe@deploy1002> |
Started scap: Update interwiki cache |
[production] |
| 23:34 |
<zabe> |
zabe@mwmaint1002:~$ mwscript extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --wiki=btmwiki --cluster=all 2>&1 | tee /tmp/btmwiki.UpdateSearchIndexConfig.log # T368038 |
[production] |
| 23:33 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2161 (T364069)', diff saved to https://phabricator.wikimedia.org/P65274 and previous config saved to /var/cache/conftool/dbconfig/20240620-233346-marostegui.json |
[production] |
| 23:33 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2161.codfw.wmnet with reason: Maintenance |
[production] |
| 23:33 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2161.codfw.wmnet with reason: Maintenance |
[production] |
| 23:33 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2154 (T364069)', diff saved to https://phabricator.wikimedia.org/P65273 and previous config saved to /var/cache/conftool/dbconfig/20240620-233324-marostegui.json |
[production] |
| 23:33 |
<zabe@deploy1002> |
Finished scap: Creating btmwiki (T368038) (duration: 12m 20s) |
[production] |
| 23:20 |
<zabe@deploy1002> |
Started scap: Creating btmwiki (T368038) |
[production] |
| 23:20 |
<zabe> |
create Wikipedia Mandailing # T368038 |
[production] |
| 23:18 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P65272 and previous config saved to /var/cache/conftool/dbconfig/20240620-231817-marostegui.json |
[production] |
| 23:03 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P65271 and previous config saved to /var/cache/conftool/dbconfig/20240620-230310-marostegui.json |
[production] |
| 22:48 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2154 (T364069)', diff saved to https://phabricator.wikimedia.org/P65270 and previous config saved to /var/cache/conftool/dbconfig/20240620-224803-marostegui.json |
[production] |
| 22:39 |
<mutante> |
aphlict1002/aphlict2001 - systemctl stop aphlict_lograte.timer (and .service); systemctl disable aphlict_logrotate.timer (and .service); systemctl daemon-reload; systemctl reset-failed T367960 |
[production] |
| 22:33 |
<zabe@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: T361041 T363825 T366649 (duration: 09m 55s) |
[production] |
| 22:29 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2180 (T367856)', diff saved to https://phabricator.wikimedia.org/P65269 and previous config saved to /var/cache/conftool/dbconfig/20240620-222909-marostegui.json |
[production] |
| 22:29 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2180.codfw.wmnet with reason: Maintenance |
[production] |
| 22:28 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2180.codfw.wmnet with reason: Maintenance |
[production] |
| 22:28 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2169 (T367856)', diff saved to https://phabricator.wikimedia.org/P65268 and previous config saved to /var/cache/conftool/dbconfig/20240620-222847-marostegui.json |
[production] |
| 22:13 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P65267 and previous config saved to /var/cache/conftool/dbconfig/20240620-221340-marostegui.json |
[production] |
| 21:58 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P65266 and previous config saved to /var/cache/conftool/dbconfig/20240620-215833-marostegui.json |
[production] |
| 21:43 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2169 (T367856)', diff saved to https://phabricator.wikimedia.org/P65265 and previous config saved to /var/cache/conftool/dbconfig/20240620-214326-marostegui.json |
[production] |
| 21:12 |
<ebernhardson@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 21:12 |
<swfrench@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/page-analytics: apply |
[production] |
| 21:12 |
<swfrench@deploy1002> |
helmfile [eqiad] START helmfile.d/services/page-analytics: apply |
[production] |
| 21:11 |
<ebernhardson@deploy1002> |
helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 21:10 |
<swfrench@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/media-analytics: apply |
[production] |
| 21:09 |
<swfrench@deploy1002> |
helmfile [eqiad] START helmfile.d/services/media-analytics: apply |
[production] |
| 21:09 |
<brett> |
Include ncmonitor 1.0.0 in wikimedia-bookworm apt repo |
[production] |
| 21:09 |
<pfischer@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 21:08 |
<ebernhardson@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 21:08 |
<pfischer@deploy1002> |
helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 21:08 |
<swfrench@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/geo-analytics: apply |
[production] |
| 21:07 |
<swfrench@deploy1002> |
helmfile [eqiad] START helmfile.d/services/geo-analytics: apply |
[production] |
| 21:07 |
<ebernhardson@deploy1002> |
helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 21:06 |
<swfrench@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/editor-analytics: apply |
[production] |
| 21:06 |
<swfrench@deploy1002> |
helmfile [eqiad] START helmfile.d/services/editor-analytics: apply |
[production] |
| 21:05 |
<swfrench@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/edit-analytics: apply |
[production] |