51-100 of 10000 results (143ms)
2025-09-09 ยง
22:25 <fceratto@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2181.codfw.wmnet with reason: Maintenance [production]
22:24 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2167 (T402763)', diff saved to https://phabricator.wikimedia.org/P83061 and previous config saved to /var/cache/conftool/dbconfig/20250909-222451-fceratto.json [production]
22:23 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P83060 and previous config saved to /var/cache/conftool/dbconfig/20250909-222305-fceratto.json [production]
22:20 <jhathaway@cumin1002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dse-k8s-worker1014.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
22:17 <jhathaway@cumin1002> START - Cookbook sre.hosts.provision for host dse-k8s-worker1014.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
22:09 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P83059 and previous config saved to /var/cache/conftool/dbconfig/20250909-220944-fceratto.json [production]
22:07 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P83058 and previous config saved to /var/cache/conftool/dbconfig/20250909-220757-fceratto.json [production]
22:04 <bking@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
22:04 <bking@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
21:55 <krinkle@deploy1003> Finished scap sync-world: Backport for [[gerrit:1184937|beta: Remove replica instance from wmgMainStashServers (T401227)]] (duration: 12m 03s) [production]
21:54 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P83057 and previous config saved to /var/cache/conftool/dbconfig/20250909-215436-fceratto.json [production]
21:52 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2173 (T402763)', diff saved to https://phabricator.wikimedia.org/P83056 and previous config saved to /var/cache/conftool/dbconfig/20250909-215249-fceratto.json [production]
21:49 <krinkle@deploy1003> krinkle, bd808: Continuing with sync [production]
21:48 <krinkle@deploy1003> krinkle, bd808: Backport for [[gerrit:1184937|beta: Remove replica instance from wmgMainStashServers (T401227)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
21:44 <fceratto@cumin1002> dbctl commit (dc=all): 'Depooling db2173 (T402763)', diff saved to https://phabricator.wikimedia.org/P83055 and previous config saved to /var/cache/conftool/dbconfig/20250909-214449-fceratto.json [production]
21:44 <fceratto@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2173.codfw.wmnet with reason: Maintenance [production]
21:44 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2170 (T402763)', diff saved to https://phabricator.wikimedia.org/P83054 and previous config saved to /var/cache/conftool/dbconfig/20250909-214425-fceratto.json [production]
21:43 <krinkle@deploy1003> Started scap sync-world: Backport for [[gerrit:1184937|beta: Remove replica instance from wmgMainStashServers (T401227)]] [production]
21:39 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2167 (T402763)', diff saved to https://phabricator.wikimedia.org/P83053 and previous config saved to /var/cache/conftool/dbconfig/20250909-213928-fceratto.json [production]
21:31 <fceratto@cumin1002> dbctl commit (dc=all): 'Depooling db2167 (T402763)', diff saved to https://phabricator.wikimedia.org/P83052 and previous config saved to /var/cache/conftool/dbconfig/20250909-213134-fceratto.json [production]
21:31 <fceratto@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2167.codfw.wmnet with reason: Maintenance [production]
21:31 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2166 (T402763)', diff saved to https://phabricator.wikimedia.org/P83051 and previous config saved to /var/cache/conftool/dbconfig/20250909-213112-fceratto.json [production]
21:29 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2170', diff saved to https://phabricator.wikimedia.org/P83050 and previous config saved to /var/cache/conftool/dbconfig/20250909-212918-fceratto.json [production]
21:24 <jhathaway@cumin1002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-worker1014.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
21:21 <jhathaway@cumin1002> START - Cookbook sre.hosts.provision for host dse-k8s-worker1014.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
21:16 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P83049 and previous config saved to /var/cache/conftool/dbconfig/20250909-211605-fceratto.json [production]
21:14 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2170', diff saved to https://phabricator.wikimedia.org/P83048 and previous config saved to /var/cache/conftool/dbconfig/20250909-211410-fceratto.json [production]
21:11 <jhathaway@cumin1002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-worker1014.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
21:09 <jhathaway@cumin1002> START - Cookbook sre.hosts.provision for host dse-k8s-worker1014.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
21:08 <cjming@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply [production]
21:07 <cjming@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply [production]
21:06 <tgr> UTC late deploys done [production]
21:04 <tgr@deploy1003> Finished scap sync-world: Backport for [[gerrit:1186592|Add $wgJwtPrivateKey / $wgJwtPublicKey in the fake privatre repo (T399631)]] (duration: 16m 16s) [production]
21:00 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P83047 and previous config saved to /var/cache/conftool/dbconfig/20250909-210058-fceratto.json [production]
20:59 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2170 (T402763)', diff saved to https://phabricator.wikimedia.org/P83046 and previous config saved to /var/cache/conftool/dbconfig/20250909-205903-fceratto.json [production]
20:58 <tgr@deploy1003> tgr: Continuing with sync [production]
20:57 <mutante> deploy1003/deploy2002 - find /srv/patches/ -name "*.patch" -not -perm 0664 -print0 | xargs -0 -r sudo chmod 0664 [production]
20:54 <tgr@deploy1003> tgr: Backport for [[gerrit:1186592|Add $wgJwtPrivateKey / $wgJwtPublicKey in the fake privatre repo (T399631)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
20:52 <jhathaway@cumin1002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-worker1014.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
20:52 <fceratto@cumin1002> dbctl commit (dc=all): 'Depooling db2170 (T402763)', diff saved to https://phabricator.wikimedia.org/P83045 and previous config saved to /var/cache/conftool/dbconfig/20250909-205226-fceratto.json [production]
20:52 <fceratto@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2170.codfw.wmnet with reason: Maintenance [production]
20:52 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2153 (T402763)', diff saved to https://phabricator.wikimedia.org/P83044 and previous config saved to /var/cache/conftool/dbconfig/20250909-205203-fceratto.json [production]
20:49 <jhathaway@cumin1002> START - Cookbook sre.hosts.provision for host dse-k8s-worker1014.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
20:48 <tgr@deploy1003> Started scap sync-world: Backport for [[gerrit:1186592|Add $wgJwtPrivateKey / $wgJwtPublicKey in the fake privatre repo (T399631)]] [production]
20:45 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2166 (T402763)', diff saved to https://phabricator.wikimedia.org/P83043 and previous config saved to /var/cache/conftool/dbconfig/20250909-204550-fceratto.json [production]
20:38 <cjming@deploy1003> Finished scap sync-world: Backport for [[gerrit:1180890|Set wgCampaignEventsCountrySchemaMigrationStage to MIGRATION_NEW (T397476)]] (duration: 15m 45s) [production]
20:37 <fceratto@cumin1002> dbctl commit (dc=all): 'Depooling db2166 (T402763)', diff saved to https://phabricator.wikimedia.org/P83042 and previous config saved to /var/cache/conftool/dbconfig/20250909-203754-fceratto.json [production]
20:37 <fceratto@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2166.codfw.wmnet with reason: Maintenance [production]
20:36 <vriley@cumin1003> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host es1056.eqiad.wmnet with OS bookworm [production]
20:36 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P83041 and previous config saved to /var/cache/conftool/dbconfig/20250909-203656-fceratto.json [production]