|
2023-09-14
ยง
|
| 09:51 |
<elukey@deploy1002> |
helmfile [codfw] START helmfile.d/services/eventstreams: sync |
[production] |
| 09:50 |
<elukey@deploy1002> |
helmfile [staging] DONE helmfile.d/services/eventstreams: sync |
[production] |
| 09:49 |
<elukey@deploy1002> |
helmfile [staging] START helmfile.d/services/eventstreams: sync |
[production] |
| 09:49 |
<jayme> |
restarted navtiming on webperf2003 to pick up changed etcd service records |
[production] |
| 09:44 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host cumin1001.eqiad.wmnet |
[production] |
| 09:43 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-rw2001.wikimedia.org |
[production] |
| 09:39 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ldap-rw2001.wikimedia.org |
[production] |
| 09:36 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-rw1001.wikimedia.org |
[production] |
| 09:32 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ldap-rw1001.wikimedia.org |
[production] |
| 09:22 |
<mvernon@cumin2002> |
END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-eqiad |
[production] |
| 09:17 |
<mvernon@cumin2002> |
START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-eqiad |
[production] |
| 09:16 |
<mvernon@cumin2002> |
END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-codfw |
[production] |
| 09:10 |
<mvernon@cumin2002> |
START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-codfw |
[production] |
| 09:07 |
<moritzm> |
installing qemu security updates on ganeti-test |
[production] |
| 08:59 |
<btullis> |
running build-production-images on build2001 for T344910 |
[production] |
| 08:53 |
<godog> |
+50 to prometheus eqiad k8s-staging |
[production] |
| 08:45 |
<jayme> |
restarting confd fleet wide |
[production] |
| 08:45 |
<jmm@cumin2002> |
START - Cookbook sre.aqs.roll-restart-reboot rolling reboot on A:aqs-eqiad |
[production] |
| 08:43 |
<jayme> |
restarting primary lvs in codfw, eqsin, ulsfo |
[production] |
| 08:38 |
<jayme> |
restarted secondary lvs in codfw, eqsin, ulsfo |
[production] |
| 08:24 |
<jnuche@deploy1002> |
rebuilt and synchronized wikiversions files: group2 wikis to 1.41.0-wmf.26 refs T343728 |
[production] |
| 07:57 |
<jayme@cumin1001> |
END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) _etcd._tcp.eqsin.wmnet on all recursors |
[production] |
| 07:56 |
<jayme@cumin1001> |
START - Cookbook sre.dns.wipe-cache _etcd._tcp.eqsin.wmnet on all recursors |
[production] |
| 07:56 |
<jayme@cumin1001> |
END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) _etcd._tcp.ulsfo.wmnet on all recursors |
[production] |
| 07:56 |
<jayme@cumin1001> |
START - Cookbook sre.dns.wipe-cache _etcd._tcp.ulsfo.wmnet on all recursors |
[production] |
| 07:56 |
<jayme@cumin1001> |
END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) _etcd._tcp.codfw.wmnet on all recursors |
[production] |
| 07:56 |
<jayme@cumin1001> |
START - Cookbook sre.dns.wipe-cache _etcd._tcp.codfw.wmnet on all recursors |
[production] |
| 07:44 |
<jmm@cumin2002> |
END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host debmonitor2003.codfw.wmnet |
[production] |
| 07:35 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host debmonitor2003.codfw.wmnet |
[production] |
| 07:32 |
<hashar> |
Backport & config deployment window completed. |
[production] |
| 07:21 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host debmonitor1003.eqiad.wmnet |
[production] |
| 07:17 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host debmonitor1003.eqiad.wmnet |
[production] |
| 07:13 |
<kartik@deploy1002> |
Finished scap: Backport for [[gerrit:956807|Enable MinT translation service on MediaWiki - rollout #4 (T341445)]] (duration: 10m 17s) |
[production] |
| 07:09 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt1001.wikimedia.org |
[production] |
| 07:06 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host apt1001.wikimedia.org |
[production] |
| 07:06 |
<kartik@deploy1002> |
abi and kartik: Continuing with sync |
[production] |
| 07:04 |
<kartik@deploy1002> |
abi and kartik: Backport for [[gerrit:956807|Enable MinT translation service on MediaWiki - rollout #4 (T341445)]] synced to the testservers mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) |
[production] |
| 07:02 |
<kartik@deploy1002> |
Started scap: Backport for [[gerrit:956807|Enable MinT translation service on MediaWiki - rollout #4 (T341445)]] |
[production] |
| 06:27 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Pre swichover tasks |
[production] |
| 06:26 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 26 hosts with reason: Pre swichover tasks |
[production] |
| 06:22 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Pre swichover tasks |
[production] |
| 06:22 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 26 hosts with reason: Pre swichover tasks |
[production] |
| 05:23 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Pre swichover tasks |
[production] |
| 05:23 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 6 hosts with reason: Pre swichover tasks |
[production] |
| 05:22 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Pre swichover tasks |
[production] |
| 05:21 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 6 hosts with reason: Pre swichover tasks |
[production] |
| 05:11 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 11 hosts with reason: Pre swichover tasks |
[production] |
| 05:10 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 11 hosts with reason: Pre swichover tasks |
[production] |
| 05:06 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on pc2013.codfw.wmnet,pc[1013-1014].eqiad.wmnet with reason: Pre swichover tasks |
[production] |
| 05:06 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on pc2013.codfw.wmnet,pc[1013-1014].eqiad.wmnet with reason: Pre swichover tasks |
[production] |