|
2023-09-14
ยง
|
| 11:25 |
<hnowlan@deploy1002> |
Started deploy [restbase/deploy@e8a6ae4]: Disable wikifeeds announcements healthcheck |
[production] |
| 11:24 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host netflow3003.esams.wmnet |
[production] |
| 11:24 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow2003.codfw.wmnet |
[production] |
| 11:21 |
<brouberol@cumin2002> |
START - Cookbook sre.opensearch.roll-restart-reboot rolling reboot on A:datahubsearch |
[production] |
| 11:17 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host netflow2003.codfw.wmnet |
[production] |
| 11:17 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow1002.eqiad.wmnet |
[production] |
| 11:13 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host netflow1002.eqiad.wmnet |
[production] |
| 11:12 |
<brouberol@cumin2002> |
END (PASS) - Cookbook sre.opensearch.roll-restart-reboot (exit_code=0) rolling restart_daemons on A:datahubsearch |
[production] |
| 11:08 |
<stevemunene@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1137.eqiad.wmnet with OS bullseye |
[production] |
| 11:04 |
<brouberol> |
brouberol@cumin2002 START - Cookbook sre.opensearch.roll-restart-reboot rolling restart_daemons on A:datahubsearch - T344798 |
[production] |
| 11:02 |
<brouberol@cumin2002> |
START - Cookbook sre.opensearch.roll-restart-reboot rolling restart_daemons on A:datahubsearch |
[production] |
| 10:43 |
<stevemunene@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1137.eqiad.wmnet with reason: host reimage |
[production] |
| 10:41 |
<stevemunene@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1137.eqiad.wmnet with reason: host reimage |
[production] |
| 10:28 |
<jayme@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on conf2004.codfw.wmnet with reason: host reimage |
[production] |
| 10:27 |
<stevemunene@cumin1001> |
START - Cookbook sre.hosts.reimage for host an-worker1137.eqiad.wmnet with OS bullseye |
[production] |
| 10:25 |
<jayme@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on conf2004.codfw.wmnet with reason: host reimage |
[production] |
| 10:24 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dborch1001.wikimedia.org |
[production] |
| 10:20 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host dborch1001.wikimedia.org |
[production] |
| 10:18 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.aqs.roll-restart-reboot (exit_code=0) rolling reboot on A:aqs-eqiad |
[production] |
| 10:10 |
<jayme@cumin1001> |
START - Cookbook sre.hosts.reimage for host conf2004.codfw.wmnet with OS bullseye |
[production] |
| 10:07 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica1006.wikimedia.org |
[production] |
| 10:06 |
<elukey@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/eventstreams: sync |
[production] |
| 10:06 |
<elukey@deploy1002> |
helmfile [eqiad] START helmfile.d/services/eventstreams: sync |
[production] |
| 10:03 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ldap-replica1006.wikimedia.org |
[production] |
| 10:03 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica1005.wikimedia.org |
[production] |
| 09:59 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ldap-replica1005.wikimedia.org |
[production] |
| 09:55 |
<jmm@cumin2002> |
END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host cumin1001.eqiad.wmnet |
[production] |
| 09:52 |
<elukey> |
remove the 'mediawiki.revision-score' stream form eventstreams public API - T342116 |
[production] |
| 09:51 |
<elukey@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/eventstreams: sync |
[production] |
| 09:51 |
<elukey@deploy1002> |
helmfile [codfw] START helmfile.d/services/eventstreams: sync |
[production] |
| 09:50 |
<elukey@deploy1002> |
helmfile [staging] DONE helmfile.d/services/eventstreams: sync |
[production] |
| 09:49 |
<elukey@deploy1002> |
helmfile [staging] START helmfile.d/services/eventstreams: sync |
[production] |
| 09:49 |
<jayme> |
restarted navtiming on webperf2003 to pick up changed etcd service records |
[production] |
| 09:44 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host cumin1001.eqiad.wmnet |
[production] |
| 09:43 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-rw2001.wikimedia.org |
[production] |
| 09:39 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ldap-rw2001.wikimedia.org |
[production] |
| 09:36 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-rw1001.wikimedia.org |
[production] |
| 09:32 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ldap-rw1001.wikimedia.org |
[production] |
| 09:22 |
<mvernon@cumin2002> |
END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-eqiad |
[production] |
| 09:17 |
<mvernon@cumin2002> |
START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-eqiad |
[production] |
| 09:16 |
<mvernon@cumin2002> |
END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-codfw |
[production] |
| 09:10 |
<mvernon@cumin2002> |
START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-codfw |
[production] |
| 09:07 |
<moritzm> |
installing qemu security updates on ganeti-test |
[production] |
| 08:59 |
<btullis> |
running build-production-images on build2001 for T344910 |
[production] |
| 08:53 |
<godog> |
+50 to prometheus eqiad k8s-staging |
[production] |
| 08:45 |
<jayme> |
restarting confd fleet wide |
[production] |
| 08:45 |
<jmm@cumin2002> |
START - Cookbook sre.aqs.roll-restart-reboot rolling reboot on A:aqs-eqiad |
[production] |
| 08:43 |
<jayme> |
restarting primary lvs in codfw, eqsin, ulsfo |
[production] |
| 08:38 |
<jayme> |
restarted secondary lvs in codfw, eqsin, ulsfo |
[production] |
| 08:24 |
<jnuche@deploy1002> |
rebuilt and synchronized wikiversions files: group2 wikis to 1.41.0-wmf.26 refs T343728 |
[production] |