|
2024-11-29
ยง
|
| 08:51 |
<brouberol@deploy2002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply |
[production] |
| 08:16 |
<moritzm> |
imported mapbox-geometry_2.0.3-1~wmf12u1 to component/maps T216826 |
[production] |
| 07:19 |
<marostegui@cumin2002> |
dbctl commit (dc=all): 'db1223 (re)pooling @ 100%: Repooling after corruption', diff saved to https://phabricator.wikimedia.org/P71427 and previous config saved to /var/cache/conftool/dbconfig/20241129-071905-root.json |
[production] |
| 07:10 |
<aqu@deploy2002> |
Finished deploy [airflow-dags/analytics@656d6df]: Generate canary events faster in Airflow (duration: 03m 15s) |
[production] |
| 07:06 |
<aqu@deploy2002> |
Started deploy [airflow-dags/analytics@656d6df]: Generate canary events faster in Airflow |
[production] |
| 07:03 |
<marostegui@cumin2002> |
dbctl commit (dc=all): 'db1223 (re)pooling @ 75%: Repooling after corruption', diff saved to https://phabricator.wikimedia.org/P71426 and previous config saved to /var/cache/conftool/dbconfig/20241129-070333-root.json |
[production] |
| 06:48 |
<marostegui@cumin2002> |
dbctl commit (dc=all): 'db1223 (re)pooling @ 50%: Repooling after corruption', diff saved to https://phabricator.wikimedia.org/P71425 and previous config saved to /var/cache/conftool/dbconfig/20241129-064801-root.json |
[production] |
| 06:28 |
<marostegui@cumin2002> |
dbctl commit (dc=all): 'Repool', diff saved to https://phabricator.wikimedia.org/P71424 and previous config saved to /var/cache/conftool/dbconfig/20241129-062833-marostegui.json |
[production] |
| 06:27 |
<marostegui@cumin2002> |
END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) db1223 quickly with 2 steps - Fixed corruption |
[production] |
| 06:26 |
<marostegui@cumin2002> |
START - Cookbook sre.mysql.pool db1223 quickly with 2 steps - Fixed corruption |
[production] |
| 05:52 |
<taavi@cumin1002> |
dbctl commit (dc=all): 'depool db1223, replication broken', diff saved to https://phabricator.wikimedia.org/P71423 and previous config saved to /var/cache/conftool/dbconfig/20241129-055245-taavi.json |
[production] |
| 04:54 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2229 (T328817)', diff saved to https://phabricator.wikimedia.org/P71422 and previous config saved to /var/cache/conftool/dbconfig/20241129-045409-ladsgroup.json |
[production] |
| 04:39 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2229', diff saved to https://phabricator.wikimedia.org/P71421 and previous config saved to /var/cache/conftool/dbconfig/20241129-043902-ladsgroup.json |
[production] |
| 04:23 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2229', diff saved to https://phabricator.wikimedia.org/P71420 and previous config saved to /var/cache/conftool/dbconfig/20241129-042355-ladsgroup.json |
[production] |
| 04:20 |
<tstarling@deploy2002> |
Finished scap sync-world: Backport for [[gerrit:1098915|addWiki.php tweaks]], [[gerrit:1098916|Run dumpInterwiki.php locally with no changes]], [[gerrit:1098917|Prepare id.wikivoyage.org for installation (T380726 T352113)]], [[gerrit:1099065|dumpInterwiki: read from preinstall.dblist (T352113)]], [[gerrit:1099066|addWiki: Move DB_ADMIN to core]], [[gerrit:1099064|addWiki: Add UpdateSearchIndexCon |
[production] |
| 04:12 |
<tstarling@deploy2002> |
tstarling: Continuing with sync |
[production] |
| 04:12 |
<tstarling@deploy2002> |
tstarling: Backport for [[gerrit:1098915|addWiki.php tweaks]], [[gerrit:1098916|Run dumpInterwiki.php locally with no changes]], [[gerrit:1098917|Prepare id.wikivoyage.org for installation (T380726 T352113)]], [[gerrit:1099065|dumpInterwiki: read from preinstall.dblist (T352113)]], [[gerrit:1099066|addWiki: Move DB_ADMIN to core]], [[gerrit:1099064|addWiki: Add UpdateSearchIndexConfig]], [[gerrit |
[production] |
| 04:08 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2229 (T328817)', diff saved to https://phabricator.wikimedia.org/P71419 and previous config saved to /var/cache/conftool/dbconfig/20241129-040846-ladsgroup.json |
[production] |
| 04:05 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depooling db2229 (T328817)', diff saved to https://phabricator.wikimedia.org/P71418 and previous config saved to /var/cache/conftool/dbconfig/20241129-040547-ladsgroup.json |
[production] |
| 04:05 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2229.codfw.wmnet with reason: Maintenance |
[production] |
| 04:05 |
<ladsgroup@cumin1002> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2229.codfw.wmnet with reason: Maintenance |
[production] |
| 04:05 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2224 (T328817)', diff saved to https://phabricator.wikimedia.org/P71417 and previous config saved to /var/cache/conftool/dbconfig/20241129-040523-ladsgroup.json |
[production] |
| 04:01 |
<tstarling@deploy2002> |
Started scap sync-world: Backport for [[gerrit:1098915|addWiki.php tweaks]], [[gerrit:1098916|Run dumpInterwiki.php locally with no changes]], [[gerrit:1098917|Prepare id.wikivoyage.org for installation (T380726 T352113)]], [[gerrit:1099065|dumpInterwiki: read from preinstall.dblist (T352113)]], [[gerrit:1099066|addWiki: Move DB_ADMIN to core]], [[gerrit:1099064|addWiki: Add UpdateSearchIndexConf |
[production] |
| 03:50 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2224', diff saved to https://phabricator.wikimedia.org/P71416 and previous config saved to /var/cache/conftool/dbconfig/20241129-035016-ladsgroup.json |
[production] |
| 03:35 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2224', diff saved to https://phabricator.wikimedia.org/P71415 and previous config saved to /var/cache/conftool/dbconfig/20241129-033509-ladsgroup.json |
[production] |
| 03:20 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2224 (T328817)', diff saved to https://phabricator.wikimedia.org/P71414 and previous config saved to /var/cache/conftool/dbconfig/20241129-032002-ladsgroup.json |
[production] |
| 03:17 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depooling db2224 (T328817)', diff saved to https://phabricator.wikimedia.org/P71413 and previous config saved to /var/cache/conftool/dbconfig/20241129-031705-ladsgroup.json |
[production] |
| 03:16 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2224.codfw.wmnet with reason: Maintenance |
[production] |
| 03:16 |
<ladsgroup@cumin1002> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2224.codfw.wmnet with reason: Maintenance |
[production] |
| 03:16 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2217 (T328817)', diff saved to https://phabricator.wikimedia.org/P71412 and previous config saved to /var/cache/conftool/dbconfig/20241129-031642-ladsgroup.json |
[production] |
| 03:04 |
<tstarling@deploy2002> |
scap failed: <KeyError> '1 dbs from /srv/mediawiki-staging/wikiversions.json are missing from /srv/mediawiki-staging/dblists/all.dblist: idwikivoyage' (scap version: 4.129.0) (duration: 00m 00s) |
[production] |
| 03:01 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P71411 and previous config saved to /var/cache/conftool/dbconfig/20241129-030133-ladsgroup.json |
[production] |
| 02:46 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P71410 and previous config saved to /var/cache/conftool/dbconfig/20241129-024625-ladsgroup.json |
[production] |
| 02:31 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2217 (T328817)', diff saved to https://phabricator.wikimedia.org/P71409 and previous config saved to /var/cache/conftool/dbconfig/20241129-023118-ladsgroup.json |
[production] |
| 02:28 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depooling db2217 (T328817)', diff saved to https://phabricator.wikimedia.org/P71408 and previous config saved to /var/cache/conftool/dbconfig/20241129-022822-ladsgroup.json |
[production] |
| 02:28 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2217.codfw.wmnet with reason: Maintenance |
[production] |
| 02:28 |
<ladsgroup@cumin1002> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2217.codfw.wmnet with reason: Maintenance |
[production] |
| 02:27 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2197.codfw.wmnet with reason: Maintenance |
[production] |
| 02:26 |
<ladsgroup@cumin1002> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2197.codfw.wmnet with reason: Maintenance |
[production] |
| 02:26 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2193 (T328817)', diff saved to https://phabricator.wikimedia.org/P71407 and previous config saved to /var/cache/conftool/dbconfig/20241129-022645-ladsgroup.json |
[production] |
| 02:11 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P71406 and previous config saved to /var/cache/conftool/dbconfig/20241129-021138-ladsgroup.json |
[production] |
| 01:56 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P71405 and previous config saved to /var/cache/conftool/dbconfig/20241129-015631-ladsgroup.json |
[production] |
| 01:41 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2193 (T328817)', diff saved to https://phabricator.wikimedia.org/P71404 and previous config saved to /var/cache/conftool/dbconfig/20241129-014124-ladsgroup.json |
[production] |
| 01:39 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depooling db2193 (T328817)', diff saved to https://phabricator.wikimedia.org/P71403 and previous config saved to /var/cache/conftool/dbconfig/20241129-013912-ladsgroup.json |
[production] |
| 01:39 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2193.codfw.wmnet with reason: Maintenance |
[production] |
| 01:38 |
<ladsgroup@cumin1002> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2193.codfw.wmnet with reason: Maintenance |
[production] |
| 01:38 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2180 (T328817)', diff saved to https://phabricator.wikimedia.org/P71402 and previous config saved to /var/cache/conftool/dbconfig/20241129-013850-ladsgroup.json |
[production] |
| 01:23 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P71401 and previous config saved to /var/cache/conftool/dbconfig/20241129-012343-ladsgroup.json |
[production] |
| 01:08 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P71400 and previous config saved to /var/cache/conftool/dbconfig/20241129-010835-ladsgroup.json |
[production] |
| 00:53 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2180 (T328817)', diff saved to https://phabricator.wikimedia.org/P71399 and previous config saved to /var/cache/conftool/dbconfig/20241129-005328-ladsgroup.json |
[production] |