|
2023-07-03
ยง
|
| 15:09 |
<moritzm> |
installing Java 8 security updates on Hadoop systems |
[production] |
| 14:25 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kubestagemaster2002.codfw.wmnet with OS bullseye |
[production] |
| 14:04 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubestagemaster2002.codfw.wmnet with reason: host reimage |
[production] |
| 14:01 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on kubestagemaster2002.codfw.wmnet with reason: host reimage |
[production] |
| 13:50 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.reimage for host kubestagemaster2002.codfw.wmnet with OS bullseye |
[production] |
| 13:37 |
<moritzm> |
installing openjdk-8 security updates |
[production] |
| 13:32 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1011.eqiad.wmnet |
[production] |
| 13:32 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1011.eqiad.wmnet |
[production] |
| 13:28 |
<urbanecm> |
UTC afternoon B&C window done |
[production] |
| 13:27 |
<urbanecm@deploy1002> |
Finished scap: Backport for [[gerrit:934624|SpecialLog: Fix issues related to IP users (T338042 T340929)]] (duration: 08m 32s) |
[production] |
| 13:25 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti1011.eqiad.wmnet |
[production] |
| 13:22 |
<volans@cumin1001> |
END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary |
[production] |
| 13:22 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1011.eqiad.wmnet |
[production] |
| 13:22 |
<volans@cumin1001> |
START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary |
[production] |
| 13:22 |
<jmm@cumin2002> |
END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1011.eqiad.wmnet |
[production] |
| 13:21 |
<urbanecm> |
Run `wikiadmin2023@10.64.16.184(idwiki)> DELETE FROM `category` WHERE cat_title = ''; ` (T336780) |
[production] |
| 13:20 |
<urbanecm@deploy1002> |
func and urbanecm: Backport for [[gerrit:934624|SpecialLog: Fix issues related to IP users (T338042 T340929)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet |
[production] |
| 13:19 |
<urbanecm@deploy1002> |
Started scap: Backport for [[gerrit:934624|SpecialLog: Fix issues related to IP users (T338042 T340929)]] |
[production] |
| 13:18 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1011.eqiad.wmnet |
[production] |
| 13:18 |
<jmm@cumin2002> |
END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1011.eqiad.wmnet |
[production] |
| 13:10 |
<urbanecm@deploy1002> |
Finished scap: Backport for [[gerrit:935048|Set wgCollectionDisableSidebarLink for nowiki (T340981)]] (duration: 08m 09s) |
[production] |
| 13:09 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1011.eqiad.wmnet |
[production] |
| 13:07 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1010.eqiad.wmnet |
[production] |
| 13:07 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1010.eqiad.wmnet |
[production] |
| 13:04 |
<urbanecm@deploy1002> |
jhsoby and urbanecm: Backport for [[gerrit:935048|Set wgCollectionDisableSidebarLink for nowiki (T340981)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet |
[production] |
| 13:02 |
<urbanecm@deploy1002> |
Started scap: Backport for [[gerrit:935048|Set wgCollectionDisableSidebarLink for nowiki (T340981)]] |
[production] |
| 13:00 |
<btullis@deploy1002> |
helmfile [staging] DONE helmfile.d/services/datahub: sync on main |
[production] |
| 13:00 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti1010.eqiad.wmnet |
[production] |
| 12:59 |
<kart_> |
Updated MinT to 2023-06-29-061037-production (T340709 + Fixed repeatation with Santali) |
[production] |
| 12:57 |
<kartik@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/machinetranslation: apply |
[production] |
| 12:51 |
<kartik@deploy1002> |
helmfile [eqiad] START helmfile.d/services/machinetranslation: apply |
[production] |
| 12:50 |
<btullis@deploy1002> |
helmfile [staging] START helmfile.d/services/datahub: apply on main |
[production] |
| 12:46 |
<kartik@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/machinetranslation: apply |
[production] |
| 12:42 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1010.eqiad.wmnet |
[production] |
| 12:41 |
<kartik@deploy1002> |
helmfile [codfw] START helmfile.d/services/machinetranslation: apply |
[production] |
| 12:35 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1009.eqiad.wmnet |
[production] |
| 12:35 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1009.eqiad.wmnet |
[production] |
| 12:33 |
<kartik@deploy1002> |
helmfile [staging] DONE helmfile.d/services/machinetranslation: apply |
[production] |
| 12:31 |
<kartik@deploy1002> |
helmfile [staging] START helmfile.d/services/machinetranslation: apply |
[production] |
| 12:29 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti1009.eqiad.wmnet |
[production] |
| 12:23 |
<kart_> |
Updated cxserver to 2023-07-03-045311-production (T285217) |
[production] |
| 12:22 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1009.eqiad.wmnet |
[production] |
| 12:18 |
<kartik@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/cxserver: apply |
[production] |
| 12:17 |
<kartik@deploy1002> |
helmfile [eqiad] START helmfile.d/services/cxserver: apply |
[production] |
| 12:12 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Awjrichards out of all services on: 1271 hosts |
[production] |
| 12:12 |
<kartik@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/cxserver: apply |
[production] |
| 12:12 |
<jmm@cumin2002> |
START - Cookbook sre.idm.logout Logging Awjrichards out of all services on: 1271 hosts |
[production] |
| 12:12 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Awjrichards out of all services on: 20 hosts |
[production] |
| 12:12 |
<jmm@cumin2002> |
START - Cookbook sre.idm.logout Logging Awjrichards out of all services on: 20 hosts |
[production] |
| 12:12 |
<kartik@deploy1002> |
helmfile [codfw] START helmfile.d/services/cxserver: apply |
[production] |