|
2023-07-03
ยง
|
| 22:00 |
<eileen> |
civicrm upgraded from 9e04c92d to 4ca2008d |
[production] |
| 20:18 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:eqiad and (A:swift-fe or A:swift-fe-canary or A:swift-fe-codfw or A:swift-fe-eqiad) |
[production] |
| 20:15 |
<effie> |
restarting swift proxies |
[production] |
| 20:14 |
<jiji@cumin1001> |
START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:eqiad and (A:swift-fe or A:swift-fe-canary or A:swift-fe-codfw or A:swift-fe-eqiad) |
[production] |
| 19:11 |
<cgoubert@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 19:11 |
<cgoubert@deploy1002> |
helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 19:10 |
<cgoubert@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 19:09 |
<cgoubert@deploy1002> |
helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 19:09 |
<cgoubert@deploy1002> |
helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 19:09 |
<cgoubert@deploy1002> |
helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 16:10 |
<effie> |
restarting pybal on lvs2013 |
[production] |
| 16:04 |
<effie> |
restarting pybal on lvs2014 |
[production] |
| 15:57 |
<effie> |
restarting pybal on lvs2014 |
[production] |
| 15:52 |
<effie> |
restarting pybal on lvs1019 |
[production] |
| 15:49 |
<effie> |
restarting pybal on lvs1020 |
[production] |
| 15:34 |
<jiji@cumin1001> |
conftool action : set/weight=10; selector: name=kubestagemaster1002.eqiad.wmnet |
[production] |
| 15:34 |
<jiji@cumin1001> |
conftool action : set/weight=10; selector: name=kubestagemaster2002.codfw.wmnet |
[production] |
| 15:34 |
<jiji@cumin1001> |
conftool action : set/pooled=yes; selector: name=kubestagemaster1002.eqiad.wmnet |
[production] |
| 15:34 |
<jiji@cumin1001> |
conftool action : set/pooled=yes; selector: name=kubestagemaster2002.codfw.wmnet |
[production] |
| 15:14 |
<jiji@cumin1001> |
conftool action : set/weight=10; selector: dc=codfw,cluster=kubernetes-staging,service=kubemaster |
[production] |
| 15:12 |
<jiji@cumin1001> |
conftool action : set/weight=10; selector: name=kubestagemaster1002.eqiad.wmnet |
[production] |
| 15:12 |
<jiji@cumin1001> |
conftool action : set/weight=10; selector: name=kubestagemaster1001.eqiad.wmnet |
[production] |
| 15:09 |
<moritzm> |
installing Java 8 security updates on Hadoop systems |
[production] |
| 14:25 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kubestagemaster2002.codfw.wmnet with OS bullseye |
[production] |
| 14:04 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubestagemaster2002.codfw.wmnet with reason: host reimage |
[production] |
| 14:01 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on kubestagemaster2002.codfw.wmnet with reason: host reimage |
[production] |
| 13:50 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.reimage for host kubestagemaster2002.codfw.wmnet with OS bullseye |
[production] |
| 13:37 |
<moritzm> |
installing openjdk-8 security updates |
[production] |
| 13:32 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1011.eqiad.wmnet |
[production] |
| 13:32 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1011.eqiad.wmnet |
[production] |
| 13:28 |
<urbanecm> |
UTC afternoon B&C window done |
[production] |
| 13:27 |
<urbanecm@deploy1002> |
Finished scap: Backport for [[gerrit:934624|SpecialLog: Fix issues related to IP users (T338042 T340929)]] (duration: 08m 32s) |
[production] |
| 13:25 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti1011.eqiad.wmnet |
[production] |
| 13:22 |
<volans@cumin1001> |
END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary |
[production] |
| 13:22 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1011.eqiad.wmnet |
[production] |
| 13:22 |
<volans@cumin1001> |
START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary |
[production] |
| 13:22 |
<jmm@cumin2002> |
END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1011.eqiad.wmnet |
[production] |
| 13:21 |
<urbanecm> |
Run `wikiadmin2023@10.64.16.184(idwiki)> DELETE FROM `category` WHERE cat_title = ''; ` (T336780) |
[production] |
| 13:20 |
<urbanecm@deploy1002> |
func and urbanecm: Backport for [[gerrit:934624|SpecialLog: Fix issues related to IP users (T338042 T340929)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet |
[production] |
| 13:19 |
<urbanecm@deploy1002> |
Started scap: Backport for [[gerrit:934624|SpecialLog: Fix issues related to IP users (T338042 T340929)]] |
[production] |
| 13:18 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1011.eqiad.wmnet |
[production] |
| 13:18 |
<jmm@cumin2002> |
END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1011.eqiad.wmnet |
[production] |
| 13:10 |
<urbanecm@deploy1002> |
Finished scap: Backport for [[gerrit:935048|Set wgCollectionDisableSidebarLink for nowiki (T340981)]] (duration: 08m 09s) |
[production] |
| 13:09 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1011.eqiad.wmnet |
[production] |
| 13:07 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1010.eqiad.wmnet |
[production] |
| 13:07 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1010.eqiad.wmnet |
[production] |
| 13:04 |
<urbanecm@deploy1002> |
jhsoby and urbanecm: Backport for [[gerrit:935048|Set wgCollectionDisableSidebarLink for nowiki (T340981)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet |
[production] |
| 13:02 |
<urbanecm@deploy1002> |
Started scap: Backport for [[gerrit:935048|Set wgCollectionDisableSidebarLink for nowiki (T340981)]] |
[production] |
| 13:00 |
<btullis@deploy1002> |
helmfile [staging] DONE helmfile.d/services/datahub: sync on main |
[production] |
| 13:00 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti1010.eqiad.wmnet |
[production] |