|
2025-08-05
ยง
|
| 14:01 |
<cgoubert@deploy1003> |
helmfile [staging-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 13:50 |
<elukey@cumin1003> |
END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bookworm |
[production] |
| 13:48 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1202 (T399728)', diff saved to https://phabricator.wikimedia.org/P80816 and previous config saved to /var/cache/conftool/dbconfig/20250805-134814-fceratto.json |
[production] |
| 13:45 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Depooling db1202 (T399728)', diff saved to https://phabricator.wikimedia.org/P80815 and previous config saved to /var/cache/conftool/dbconfig/20250805-134539-fceratto.json |
[production] |
| 13:45 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1202.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:45 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1194 (T399728)', diff saved to https://phabricator.wikimedia.org/P80814 and previous config saved to /var/cache/conftool/dbconfig/20250805-134515-fceratto.json |
[production] |
| 13:45 |
<elukey@cumin1003> |
START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bookworm |
[production] |
| 13:41 |
<mszabo@deploy1003> |
Started scap sync-world: Backport for [[gerrit:1175899|UserInfoCard: Cap maximum count for thanks given/received (T398354)]] |
[production] |
| 13:40 |
<elukey@cumin1003> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2331.codfw.wmnet with OS bookworm |
[production] |
| 13:40 |
<elukey@cumin1003> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1003" |
[production] |
| 13:39 |
<elukey@cumin1003> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1003" |
[production] |
| 13:30 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P80813 and previous config saved to /var/cache/conftool/dbconfig/20250805-133007-fceratto.json |
[production] |
| 13:23 |
<elukey@cumin1003> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2331.codfw.wmnet with reason: host reimage |
[production] |
| 13:20 |
<jdrewniak@deploy1003> |
Synchronized portals: Wikimedia Portals Update: [[gerrit:1175895| Bumping portals to master (T128546)]] (duration: 01m 45s) |
[production] |
| 13:18 |
<jdrewniak@deploy1003> |
Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:1175895| Bumping portals to master (T128546)]] (duration: 07m 07s) |
[production] |
| 13:17 |
<elukey@cumin1003> |
START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2331.codfw.wmnet with reason: host reimage |
[production] |
| 13:15 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P80812 and previous config saved to /var/cache/conftool/dbconfig/20250805-131500-fceratto.json |
[production] |
| 13:04 |
<elukey@cumin1003> |
START - Cookbook sre.hosts.reimage for host wikikube-worker2331.codfw.wmnet with OS bookworm |
[production] |
| 13:02 |
<brouberol@deploy1003> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 13:02 |
<brouberol@deploy1003> |
helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 13:02 |
<elukey@cumin1003> |
END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2331.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART |
[production] |
| 12:59 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1194 (T399728)', diff saved to https://phabricator.wikimedia.org/P80811 and previous config saved to /var/cache/conftool/dbconfig/20250805-125952-fceratto.json |
[production] |
| 12:57 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Depooling db1194 (T399728)', diff saved to https://phabricator.wikimedia.org/P80810 and previous config saved to /var/cache/conftool/dbconfig/20250805-125719-fceratto.json |
[production] |
| 12:57 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1194.eqiad.wmnet with reason: Maintenance |
[production] |
| 12:56 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1191 (T399728)', diff saved to https://phabricator.wikimedia.org/P80809 and previous config saved to /var/cache/conftool/dbconfig/20250805-125655-fceratto.json |
[production] |
| 12:52 |
<elukey@cumin1003> |
START - Cookbook sre.hosts.provision for host wikikube-worker2331.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART |
[production] |
| 12:41 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P80807 and previous config saved to /var/cache/conftool/dbconfig/20250805-124147-fceratto.json |
[production] |
| 12:35 |
<gkyziridis@deploy1003> |
helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . |
[production] |
| 12:35 |
<gkyziridis@deploy1003> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . |
[production] |
| 12:26 |
<gkyziridis@deploy1003> |
helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'llm' for release 'main' . |
[production] |
| 12:26 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P80806 and previous config saved to /var/cache/conftool/dbconfig/20250805-122640-fceratto.json |
[production] |
| 12:26 |
<gkyziridis@deploy1003> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . |
[production] |
| 12:11 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1191 (T399728)', diff saved to https://phabricator.wikimedia.org/P80805 and previous config saved to /var/cache/conftool/dbconfig/20250805-121132-fceratto.json |
[production] |
| 12:08 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Depooling db1191 (T399728)', diff saved to https://phabricator.wikimedia.org/P80803 and previous config saved to /var/cache/conftool/dbconfig/20250805-120857-fceratto.json |
[production] |
| 12:08 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1191.eqiad.wmnet with reason: Maintenance |
[production] |
| 12:08 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1181 (T399728)', diff saved to https://phabricator.wikimedia.org/P80802 and previous config saved to /var/cache/conftool/dbconfig/20250805-120835-fceratto.json |
[production] |
| 11:53 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P80801 and previous config saved to /var/cache/conftool/dbconfig/20250805-115327-fceratto.json |
[production] |
| 11:38 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P80800 and previous config saved to /var/cache/conftool/dbconfig/20250805-113820-fceratto.json |
[production] |
| 11:23 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1181 (T399728)', diff saved to https://phabricator.wikimedia.org/P80799 and previous config saved to /var/cache/conftool/dbconfig/20250805-112312-fceratto.json |
[production] |
| 11:20 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Depooling db1181 (T399728)', diff saved to https://phabricator.wikimedia.org/P80798 and previous config saved to /var/cache/conftool/dbconfig/20250805-112036-fceratto.json |
[production] |
| 11:20 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1181.eqiad.wmnet with reason: Maintenance |
[production] |
| 11:20 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1174 (T399728)', diff saved to https://phabricator.wikimedia.org/P80797 and previous config saved to /var/cache/conftool/dbconfig/20250805-112014-fceratto.json |
[production] |
| 11:15 |
<btullis@cumin1003> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts dumpsdata1003.eqiad.wmnet |
[production] |
| 11:15 |
<btullis@cumin1003> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 11:15 |
<btullis@cumin1003> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dumpsdata1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1003" |
[production] |
| 11:14 |
<btullis@cumin1003> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dumpsdata1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1003" |
[production] |
| 11:05 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P80796 and previous config saved to /var/cache/conftool/dbconfig/20250805-110506-fceratto.json |
[production] |
| 10:56 |
<jnuche@deploy1003> |
Finished deploy [releng/jenkins-deploy@62138e1] (releasing): T401180 (duration: 00m 32s) |
[production] |
| 10:56 |
<jnuche@deploy1003> |
Started deploy [releng/jenkins-deploy@62138e1] (releasing): T401180 |
[production] |
| 10:55 |
<btullis@cumin1003> |
START - Cookbook sre.dns.netbox |
[production] |