|
2023-07-04
§
|
| 09:42 |
<cgoubert@deploy1002> |
helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 09:42 |
<cgoubert@deploy1002> |
helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 09:41 |
<cgoubert@deploy1002> |
helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 09:38 |
<jayme> |
updated envoyproxy to 1.23.10 on all nodes - T300324 |
[production] |
| 09:37 |
<cgoubert@deploy1002> |
helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 09:37 |
<cgoubert@deploy1002> |
helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 09:36 |
<cgoubert@deploy1002> |
helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 09:36 |
<cgoubert@deploy1002> |
helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 09:07 |
<btullis@cumin1001> |
END (FAIL) - Cookbook sre.hadoop.roll-restart-workers (exit_code=99) restart workers for Hadoop analytics cluster: Roll restart of jvm daemons for openjdk upgrade. |
[production] |
| 09:02 |
<jnuche@deploy1002> |
Started scap: testwikis wikis to 1.41.0-wmf.16 refs T340244 |
[production] |
| 08:56 |
<btullis@cumin1001> |
START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop analytics cluster: Roll restart of jvm daemons for openjdk upgrade. |
[production] |
|
2023-07-03
§
|
| 22:00 |
<eileen> |
civicrm upgraded from 9e04c92d to 4ca2008d |
[production] |
| 20:18 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:eqiad and (A:swift-fe or A:swift-fe-canary or A:swift-fe-codfw or A:swift-fe-eqiad) |
[production] |
| 20:15 |
<effie> |
restarting swift proxies |
[production] |
| 20:14 |
<jiji@cumin1001> |
START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:eqiad and (A:swift-fe or A:swift-fe-canary or A:swift-fe-codfw or A:swift-fe-eqiad) |
[production] |
| 19:11 |
<cgoubert@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 19:11 |
<cgoubert@deploy1002> |
helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 19:10 |
<cgoubert@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 19:09 |
<cgoubert@deploy1002> |
helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 19:09 |
<cgoubert@deploy1002> |
helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 19:09 |
<cgoubert@deploy1002> |
helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 16:10 |
<effie> |
restarting pybal on lvs2013 |
[production] |
| 16:04 |
<effie> |
restarting pybal on lvs2014 |
[production] |
| 15:57 |
<effie> |
restarting pybal on lvs2014 |
[production] |
| 15:52 |
<effie> |
restarting pybal on lvs1019 |
[production] |
| 15:49 |
<effie> |
restarting pybal on lvs1020 |
[production] |
| 15:34 |
<jiji@cumin1001> |
conftool action : set/weight=10; selector: name=kubestagemaster1002.eqiad.wmnet |
[production] |
| 15:34 |
<jiji@cumin1001> |
conftool action : set/weight=10; selector: name=kubestagemaster2002.codfw.wmnet |
[production] |
| 15:34 |
<jiji@cumin1001> |
conftool action : set/pooled=yes; selector: name=kubestagemaster1002.eqiad.wmnet |
[production] |
| 15:34 |
<jiji@cumin1001> |
conftool action : set/pooled=yes; selector: name=kubestagemaster2002.codfw.wmnet |
[production] |
| 15:14 |
<jiji@cumin1001> |
conftool action : set/weight=10; selector: dc=codfw,cluster=kubernetes-staging,service=kubemaster |
[production] |
| 15:12 |
<jiji@cumin1001> |
conftool action : set/weight=10; selector: name=kubestagemaster1002.eqiad.wmnet |
[production] |
| 15:12 |
<jiji@cumin1001> |
conftool action : set/weight=10; selector: name=kubestagemaster1001.eqiad.wmnet |
[production] |
| 15:09 |
<moritzm> |
installing Java 8 security updates on Hadoop systems |
[production] |
| 14:25 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kubestagemaster2002.codfw.wmnet with OS bullseye |
[production] |
| 14:04 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubestagemaster2002.codfw.wmnet with reason: host reimage |
[production] |
| 14:01 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on kubestagemaster2002.codfw.wmnet with reason: host reimage |
[production] |
| 13:50 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.reimage for host kubestagemaster2002.codfw.wmnet with OS bullseye |
[production] |
| 13:37 |
<moritzm> |
installing openjdk-8 security updates |
[production] |
| 13:32 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1011.eqiad.wmnet |
[production] |
| 13:32 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1011.eqiad.wmnet |
[production] |
| 13:28 |
<urbanecm> |
UTC afternoon B&C window done |
[production] |
| 13:27 |
<urbanecm@deploy1002> |
Finished scap: Backport for [[gerrit:934624|SpecialLog: Fix issues related to IP users (T338042 T340929)]] (duration: 08m 32s) |
[production] |
| 13:25 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti1011.eqiad.wmnet |
[production] |
| 13:22 |
<volans@cumin1001> |
END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary |
[production] |
| 13:22 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1011.eqiad.wmnet |
[production] |
| 13:22 |
<volans@cumin1001> |
START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary |
[production] |
| 13:22 |
<jmm@cumin2002> |
END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1011.eqiad.wmnet |
[production] |
| 13:21 |
<urbanecm> |
Run `wikiadmin2023@10.64.16.184(idwiki)> DELETE FROM `category` WHERE cat_title = ''; ` (T336780) |
[production] |
| 13:20 |
<urbanecm@deploy1002> |
func and urbanecm: Backport for [[gerrit:934624|SpecialLog: Fix issues related to IP users (T338042 T340929)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet |
[production] |