|
2023-09-14
§
|
| 09:07 |
<moritzm> |
installing qemu security updates on ganeti-test |
[production] |
| 08:59 |
<btullis> |
running build-production-images on build2001 for T344910 |
[production] |
| 08:53 |
<godog> |
+50 to prometheus eqiad k8s-staging |
[production] |
| 08:45 |
<jayme> |
restarting confd fleet wide |
[production] |
| 08:45 |
<jmm@cumin2002> |
START - Cookbook sre.aqs.roll-restart-reboot rolling reboot on A:aqs-eqiad |
[production] |
| 08:43 |
<jayme> |
restarting primary lvs in codfw, eqsin, ulsfo |
[production] |
| 08:38 |
<jayme> |
restarted secondary lvs in codfw, eqsin, ulsfo |
[production] |
| 08:24 |
<jnuche@deploy1002> |
rebuilt and synchronized wikiversions files: group2 wikis to 1.41.0-wmf.26 refs T343728 |
[production] |
| 07:57 |
<jayme@cumin1001> |
END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) _etcd._tcp.eqsin.wmnet on all recursors |
[production] |
| 07:56 |
<jayme@cumin1001> |
START - Cookbook sre.dns.wipe-cache _etcd._tcp.eqsin.wmnet on all recursors |
[production] |
| 07:56 |
<jayme@cumin1001> |
END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) _etcd._tcp.ulsfo.wmnet on all recursors |
[production] |
| 07:56 |
<jayme@cumin1001> |
START - Cookbook sre.dns.wipe-cache _etcd._tcp.ulsfo.wmnet on all recursors |
[production] |
| 07:56 |
<jayme@cumin1001> |
END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) _etcd._tcp.codfw.wmnet on all recursors |
[production] |
| 07:56 |
<jayme@cumin1001> |
START - Cookbook sre.dns.wipe-cache _etcd._tcp.codfw.wmnet on all recursors |
[production] |
| 07:44 |
<jmm@cumin2002> |
END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host debmonitor2003.codfw.wmnet |
[production] |
| 07:35 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host debmonitor2003.codfw.wmnet |
[production] |
| 07:32 |
<hashar> |
Backport & config deployment window completed. |
[production] |
| 07:21 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host debmonitor1003.eqiad.wmnet |
[production] |
| 07:17 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host debmonitor1003.eqiad.wmnet |
[production] |
| 07:13 |
<kartik@deploy1002> |
Finished scap: Backport for [[gerrit:956807|Enable MinT translation service on MediaWiki - rollout #4 (T341445)]] (duration: 10m 17s) |
[production] |
| 07:09 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt1001.wikimedia.org |
[production] |
| 07:06 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host apt1001.wikimedia.org |
[production] |
| 07:06 |
<kartik@deploy1002> |
abi and kartik: Continuing with sync |
[production] |
| 07:04 |
<kartik@deploy1002> |
abi and kartik: Backport for [[gerrit:956807|Enable MinT translation service on MediaWiki - rollout #4 (T341445)]] synced to the testservers mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) |
[production] |
| 07:02 |
<kartik@deploy1002> |
Started scap: Backport for [[gerrit:956807|Enable MinT translation service on MediaWiki - rollout #4 (T341445)]] |
[production] |
| 06:27 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Pre swichover tasks |
[production] |
| 06:26 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 26 hosts with reason: Pre swichover tasks |
[production] |
| 06:22 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Pre swichover tasks |
[production] |
| 06:22 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 26 hosts with reason: Pre swichover tasks |
[production] |
| 05:23 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Pre swichover tasks |
[production] |
| 05:23 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 6 hosts with reason: Pre swichover tasks |
[production] |
| 05:22 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Pre swichover tasks |
[production] |
| 05:21 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 6 hosts with reason: Pre swichover tasks |
[production] |
| 05:11 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 11 hosts with reason: Pre swichover tasks |
[production] |
| 05:10 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 11 hosts with reason: Pre swichover tasks |
[production] |
| 05:06 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on pc2013.codfw.wmnet,pc[1013-1014].eqiad.wmnet with reason: Pre swichover tasks |
[production] |
| 05:06 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on pc2013.codfw.wmnet,pc[1013-1014].eqiad.wmnet with reason: Pre swichover tasks |
[production] |
| 05:06 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on pc2012.codfw.wmnet,pc1012.eqiad.wmnet with reason: Pre swichover tasks |
[production] |
| 05:05 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on pc2012.codfw.wmnet,pc1012.eqiad.wmnet with reason: Pre swichover tasks |
[production] |
| 05:05 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on pc[2011,2014].codfw.wmnet,pc1011.eqiad.wmnet with reason: Pre swichover tasks |
[production] |
| 05:05 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on pc[2011,2014].codfw.wmnet,pc1011.eqiad.wmnet with reason: Pre swichover tasks |
[production] |
| 03:04 |
<rzl@deploy1002> |
helmfile [eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 03:03 |
<rzl@deploy1002> |
helmfile [eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 03:03 |
<rzl@deploy1002> |
helmfile [codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 02:58 |
<rzl@deploy1002> |
helmfile [codfw] START helmfile.d/admin 'apply'. |
[production] |
| 02:58 |
<rzl@deploy1002> |
helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 02:57 |
<rzl@deploy1002> |
helmfile [staging-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 02:56 |
<rzl@deploy1002> |
helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 02:54 |
<rzl@deploy1002> |
helmfile [staging-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 01:36 |
<urandom> |
starting RESTBase/Cassandra node rebuilds, cassandra-c/row D — T331713 |
[production] |