|
2024-11-29
ยง
|
| 07:07 |
<raymond-ndibe@cloudcumin1001> |
START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor |
[toolsbeta] |
| 07:06 |
<aqu@deploy2002> |
Started deploy [airflow-dags/analytics@656d6df]: Generate canary events faster in Airflow |
[production] |
| 07:05 |
<raymond-ndibe@cloudcumin1001> |
END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor |
[toolsbeta] |
| 07:04 |
<raymond-ndibe@cloudcumin1001> |
START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor |
[toolsbeta] |
| 07:03 |
<marostegui@cumin2002> |
dbctl commit (dc=all): 'db1223 (re)pooling @ 75%: Repooling after corruption', diff saved to https://phabricator.wikimedia.org/P71426 and previous config saved to /var/cache/conftool/dbconfig/20241129-070333-root.json |
[production] |
| 06:53 |
<raymond-ndibe@cloudcumin1001> |
END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor |
[toolsbeta] |
| 06:53 |
<raymond-ndibe@cloudcumin1001> |
START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor |
[toolsbeta] |
| 06:48 |
<marostegui@cumin2002> |
dbctl commit (dc=all): 'db1223 (re)pooling @ 50%: Repooling after corruption', diff saved to https://phabricator.wikimedia.org/P71425 and previous config saved to /var/cache/conftool/dbconfig/20241129-064801-root.json |
[production] |
| 06:39 |
<raymond-ndibe@cloudcumin1001> |
END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor |
[toolsbeta] |
| 06:39 |
<raymond-ndibe@cloudcumin1001> |
START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor |
[toolsbeta] |
| 06:34 |
<raymond-ndibe@cloudcumin1001> |
END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor |
[toolsbeta] |
| 06:34 |
<raymond-ndibe@cloudcumin1001> |
START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor |
[toolsbeta] |
| 06:32 |
<raymond-ndibe@cloudcumin1001> |
END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor |
[toolsbeta] |
| 06:32 |
<raymond-ndibe@cloudcumin1001> |
START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor |
[toolsbeta] |
| 06:28 |
<marostegui@cumin2002> |
dbctl commit (dc=all): 'Repool', diff saved to https://phabricator.wikimedia.org/P71424 and previous config saved to /var/cache/conftool/dbconfig/20241129-062833-marostegui.json |
[production] |
| 06:27 |
<marostegui@cumin2002> |
END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) db1223 quickly with 2 steps - Fixed corruption |
[production] |
| 06:26 |
<marostegui@cumin2002> |
START - Cookbook sre.mysql.pool db1223 quickly with 2 steps - Fixed corruption |
[production] |
| 05:52 |
<taavi@cumin1002> |
dbctl commit (dc=all): 'depool db1223, replication broken', diff saved to https://phabricator.wikimedia.org/P71423 and previous config saved to /var/cache/conftool/dbconfig/20241129-055245-taavi.json |
[production] |
| 05:00 |
<raymond-ndibe@cloudcumin1001> |
END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor |
[toolsbeta] |
| 05:00 |
<raymond-ndibe@cloudcumin1001> |
START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor |
[toolsbeta] |
| 04:54 |
<raymond-ndibe@cloudcumin1001> |
END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor |
[toolsbeta] |
| 04:54 |
<raymond-ndibe@cloudcumin1001> |
START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor |
[toolsbeta] |
| 04:54 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2229 (T328817)', diff saved to https://phabricator.wikimedia.org/P71422 and previous config saved to /var/cache/conftool/dbconfig/20241129-045409-ladsgroup.json |
[production] |
| 04:51 |
<raymond-ndibe@cloudcumin1001> |
END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor |
[toolsbeta] |
| 04:51 |
<raymond-ndibe@cloudcumin1001> |
START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor |
[toolsbeta] |
| 04:40 |
<raymond-ndibe@cloudcumin1001> |
END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor |
[toolsbeta] |
| 04:40 |
<raymond-ndibe@cloudcumin1001> |
START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor |
[toolsbeta] |
| 04:39 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2229', diff saved to https://phabricator.wikimedia.org/P71421 and previous config saved to /var/cache/conftool/dbconfig/20241129-043902-ladsgroup.json |
[production] |
| 04:23 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2229', diff saved to https://phabricator.wikimedia.org/P71420 and previous config saved to /var/cache/conftool/dbconfig/20241129-042355-ladsgroup.json |
[production] |
| 04:20 |
<tstarling@deploy2002> |
Finished scap sync-world: Backport for [[gerrit:1098915|addWiki.php tweaks]], [[gerrit:1098916|Run dumpInterwiki.php locally with no changes]], [[gerrit:1098917|Prepare id.wikivoyage.org for installation (T380726 T352113)]], [[gerrit:1099065|dumpInterwiki: read from preinstall.dblist (T352113)]], [[gerrit:1099066|addWiki: Move DB_ADMIN to core]], [[gerrit:1099064|addWiki: Add UpdateSearchIndexCon |
[production] |
| 04:14 |
<raymond-ndibe@cloudcumin1001> |
END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor |
[toolsbeta] |
| 04:14 |
<raymond-ndibe@cloudcumin1001> |
START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor |
[toolsbeta] |
| 04:13 |
<raymond-ndibe@cloudcumin1001> |
END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor |
[toolsbeta] |
| 04:13 |
<raymond-ndibe@cloudcumin1001> |
START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor |
[toolsbeta] |
| 04:12 |
<tstarling@deploy2002> |
tstarling: Continuing with sync |
[production] |
| 04:12 |
<tstarling@deploy2002> |
tstarling: Backport for [[gerrit:1098915|addWiki.php tweaks]], [[gerrit:1098916|Run dumpInterwiki.php locally with no changes]], [[gerrit:1098917|Prepare id.wikivoyage.org for installation (T380726 T352113)]], [[gerrit:1099065|dumpInterwiki: read from preinstall.dblist (T352113)]], [[gerrit:1099066|addWiki: Move DB_ADMIN to core]], [[gerrit:1099064|addWiki: Add UpdateSearchIndexConfig]], [[gerrit |
[production] |
| 04:08 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2229 (T328817)', diff saved to https://phabricator.wikimedia.org/P71419 and previous config saved to /var/cache/conftool/dbconfig/20241129-040846-ladsgroup.json |
[production] |
| 04:05 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depooling db2229 (T328817)', diff saved to https://phabricator.wikimedia.org/P71418 and previous config saved to /var/cache/conftool/dbconfig/20241129-040547-ladsgroup.json |
[production] |
| 04:05 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2229.codfw.wmnet with reason: Maintenance |
[production] |
| 04:05 |
<ladsgroup@cumin1002> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2229.codfw.wmnet with reason: Maintenance |
[production] |
| 04:05 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2224 (T328817)', diff saved to https://phabricator.wikimedia.org/P71417 and previous config saved to /var/cache/conftool/dbconfig/20241129-040523-ladsgroup.json |
[production] |
| 04:01 |
<tstarling@deploy2002> |
Started scap sync-world: Backport for [[gerrit:1098915|addWiki.php tweaks]], [[gerrit:1098916|Run dumpInterwiki.php locally with no changes]], [[gerrit:1098917|Prepare id.wikivoyage.org for installation (T380726 T352113)]], [[gerrit:1099065|dumpInterwiki: read from preinstall.dblist (T352113)]], [[gerrit:1099066|addWiki: Move DB_ADMIN to core]], [[gerrit:1099064|addWiki: Add UpdateSearchIndexConf |
[production] |
| 03:50 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2224', diff saved to https://phabricator.wikimedia.org/P71416 and previous config saved to /var/cache/conftool/dbconfig/20241129-035016-ladsgroup.json |
[production] |
| 03:43 |
<raymond-ndibe@cloudcumin1001> |
END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api |
[tools] |
| 03:37 |
<raymond-ndibe@cloudcumin1001> |
START - Cookbook wmcs.toolforge.component.deploy for component builds-api |
[tools] |
| 03:37 |
<raymond-ndibe@cloudcumin1001> |
END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api |
[tools] |
| 03:35 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2224', diff saved to https://phabricator.wikimedia.org/P71415 and previous config saved to /var/cache/conftool/dbconfig/20241129-033509-ladsgroup.json |
[production] |
| 03:34 |
<raymond-ndibe@cloudcumin1001> |
START - Cookbook wmcs.toolforge.component.deploy for component builds-api |
[tools] |
| 03:34 |
<raymond-ndibe@cloudcumin1001> |
END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api |
[tools] |
| 03:34 |
<raymond-ndibe@cloudcumin1001> |
START - Cookbook wmcs.toolforge.component.deploy for component builds-api |
[tools] |