|
2025-06-24
ยง
|
| 16:26 |
<cmooney@cumin1003> |
END (FAIL) - Cookbook sre.dns.netbox-records (exit_code=99) Generate and push DNS records from Netbox data |
[production] |
| 16:25 |
<cmooney@cumin1003> |
START - Cookbook sre.dns.netbox-records Generate and push DNS records from Netbox data |
[production] |
| 16:23 |
<dcaro@cloudcumin1001> |
END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api |
[tools] |
| 16:23 |
<brett@cumin2002> |
START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P{cp11[13,15].eqiad.wmnet} and A:cp - Fix VSLbs() assert error and upgrade libvmod-wmfuniq to 0.2.0 (T396581) |
[production] |
| 16:21 |
<brett@cumin2002> |
START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P{cp11[12,14].eqiad.wmnet} and A:cp - Fix VSLbs() assert error and upgrade libvmod-wmfuniq to 0.2.0 (T396581) |
[production] |
| 16:20 |
<mvernon@cumin1002> |
START - Cookbook sre.swift.check-dbs Checking container DBs of wikipedia-commons-local-thumb.23 |
[production] |
| 16:20 |
<mvernon@cumin1002> |
END (PASS) - Cookbook sre.swift.check-dbs (exit_code=0) Checking container DBs of wikipedia-commons-local-thumb.22 |
[production] |
| 16:19 |
<dcaro@cloudcumin1001> |
START - Cookbook wmcs.toolforge.component.deploy for component components-api |
[tools] |
| 16:19 |
<dcaro@cloudcumin1001> |
END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api |
[toolsbeta] |
| 16:14 |
<dcaro@cloudcumin1001> |
START - Cookbook wmcs.toolforge.component.deploy for component components-api |
[toolsbeta] |
| 16:09 |
<vgutierrez@cumin1002> |
END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-text_eqiad - 7.1.1-2~bpo11+wmf2 upgrade (T396581) |
[production] |
| 16:08 |
<mvernon@cumin1002> |
START - Cookbook sre.swift.check-dbs Checking container DBs of wikipedia-commons-local-thumb.22 |
[production] |
| 16:08 |
<mvernon@cumin1002> |
END (PASS) - Cookbook sre.swift.check-dbs (exit_code=0) Checking container DBs of wikipedia-commons-local-thumb.21 |
[production] |
| 16:08 |
<vgutierrez@cumin1002> |
END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-upload_eqiad - 7.1.1-2~bpo11+wmf2 upgrade (T396581) |
[production] |
| 15:58 |
<mvernon@cumin1002> |
START - Cookbook sre.swift.check-dbs Checking container DBs of wikipedia-commons-local-thumb.21 |
[production] |
| 15:58 |
<mvernon@cumin1002> |
END (PASS) - Cookbook sre.swift.check-dbs (exit_code=0) Checking container DBs of wikipedia-commons-local-thumb.20 |
[production] |
| 15:49 |
<mvernon@cumin1002> |
START - Cookbook sre.swift.check-dbs Checking container DBs of wikipedia-commons-local-thumb.20 |
[production] |
| 15:48 |
<mvernon@cumin1002> |
END (PASS) - Cookbook sre.swift.check-dbs (exit_code=0) Checking container DBs of wikipedia-commons-local-thumb.1f |
[production] |
| 15:43 |
<mvernon@cumin1002> |
START - Cookbook sre.swift.check-dbs Checking container DBs of wikipedia-commons-local-thumb.1f |
[production] |
| 15:42 |
<mvernon@cumin1002> |
END (PASS) - Cookbook sre.swift.check-dbs (exit_code=0) Checking container DBs of wikipedia-commons-local-thumb.1e |
[production] |
| 15:40 |
<dancy@deploy1003> |
Finished scap sync-world: testing (duration: 03m 01s) |
[production] |
| 15:37 |
<dancy@deploy1003> |
Started scap sync-world: testing |
[production] |
| 15:36 |
<mvernon@cumin1002> |
START - Cookbook sre.swift.check-dbs Checking container DBs of wikipedia-commons-local-thumb.1e |
[production] |
| 15:36 |
<mvernon@cumin1002> |
END (PASS) - Cookbook sre.swift.check-dbs (exit_code=0) Checking container DBs of wikipedia-commons-local-thumb.1d |
[production] |
| 15:29 |
<mvernon@cumin1002> |
START - Cookbook sre.swift.check-dbs Checking container DBs of wikipedia-commons-local-thumb.1d |
[production] |
| 15:29 |
<mvernon@cumin1002> |
END (PASS) - Cookbook sre.swift.check-dbs (exit_code=0) Checking container DBs of wikipedia-commons-local-thumb.1c |
[production] |
| 15:29 |
<sukhe@cumin1002> |
END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:drmrs and A:cp - 9.2.11 upgrade (T397456) |
[production] |
| 15:22 |
<cdanis@dns1004> |
END - running authdns-update |
[production] |
| 15:21 |
<cdanis@dns1004> |
START - running authdns-update |
[production] |
| 15:20 |
<cdanis@dns1004> |
END - running authdns-update |
[production] |
| 15:19 |
<cdanis@dns1004> |
START - running authdns-update |
[production] |
| 15:17 |
<mvernon@cumin1002> |
START - Cookbook sre.swift.check-dbs Checking container DBs of wikipedia-commons-local-thumb.1c |
[production] |
| 15:16 |
<mvernon@cumin1002> |
END (PASS) - Cookbook sre.swift.check-dbs (exit_code=0) Checking container DBs of wikipedia-commons-local-thumb.1b |
[production] |
| 15:12 |
<andrew@cloudcumin1001> |
END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-33 |
[tools] |
| 15:09 |
<mvernon@cumin1002> |
START - Cookbook sre.swift.check-dbs Checking container DBs of wikipedia-commons-local-thumb.1b |
[production] |
| 15:09 |
<mvernon@cumin1002> |
END (PASS) - Cookbook sre.swift.check-dbs (exit_code=0) Checking container DBs of wikipedia-commons-local-thumb.1a |
[production] |
| 15:06 |
<andrewbogott> |
rebooting tools-k8s-worker-nfs-33, stuck processes |
[tools] |
| 15:06 |
<andrew@cloudcumin1001> |
START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-33 |
[tools] |
| 15:05 |
<brennen@deploy1003> |
Finished deploy [phabricator/deployment@adb2373]: deploy phab1004 for T397726 (duration: 00m 38s) |
[production] |
| 15:05 |
<cmooney@cumin1003> |
END (FAIL) - Cookbook sre.dns.netbox_records (exit_code=99) Generate and push DNS records from Netbox data |
[production] |
| 15:05 |
<brennen@deploy1003> |
Started deploy [phabricator/deployment@adb2373]: deploy phab1004 for T397726 |
[production] |
| 15:05 |
<dcaro@cloudcumin1001> |
END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api |
[tools] |
| 15:05 |
<aokoth@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab1004.eqiad.wmnet with reason: Phorge Update |
[production] |
| 15:04 |
<brennen@deploy1003> |
Finished deploy [phabricator/deployment@adb2373]: test deploy phab2002 for T397726 (duration: 00m 41s) |
[production] |
| 15:03 |
<brennen@deploy1003> |
Started deploy [phabricator/deployment@adb2373]: test deploy phab2002 for T397726 |
[production] |
| 15:03 |
<aokoth@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab2002.codfw.wmnet with reason: Phorge Update |
[production] |
| 15:03 |
<mvernon@cumin1002> |
START - Cookbook sre.swift.check-dbs Checking container DBs of wikipedia-commons-local-thumb.1a |
[production] |
| 15:03 |
<mvernon@cumin1002> |
END (PASS) - Cookbook sre.swift.check-dbs (exit_code=0) Checking container DBs of wikipedia-commons-local-thumb.19 |
[production] |
| 15:03 |
<cmooney@cumin1003> |
START - Cookbook sre.dns.netbox_records Generate and push DNS records from Netbox data |
[production] |
| 15:02 |
<komla@cloudcumin1001> |
END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T396840) |
[mwoffliner] |