|
2022-01-11
ยง
|
| 15:52 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM acmechief-test1001.eqiad.wmnet |
[production] |
| 15:51 |
<ebernhardson> |
restart elasticserach_6@production-search-psi-eqiad on elastic1049 to resolve issue with full heap |
[production] |
| 15:47 |
<vgutierrez@cumin1001> |
START - Cookbook sre.ganeti.reboot-vm for VM acmechief-test1001.eqiad.wmnet |
[production] |
| 15:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T297191)', diff saved to https://phabricator.wikimedia.org/P18582 and previous config saved to /var/cache/conftool/dbconfig/20220111-154722-marostegui.json |
[production] |
| 15:46 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1144:3314 (T297191)', diff saved to https://phabricator.wikimedia.org/P18580 and previous config saved to /var/cache/conftool/dbconfig/20220111-154615-marostegui.json |
[production] |
| 15:46 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance |
[production] |
| 15:46 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance |
[production] |
| 15:46 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1141 (T297191)', diff saved to https://phabricator.wikimedia.org/P18579 and previous config saved to /var/cache/conftool/dbconfig/20220111-154608-marostegui.json |
[production] |
| 15:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P18578 and previous config saved to /var/cache/conftool/dbconfig/20220111-153103-marostegui.json |
[production] |
| 15:30 |
<hnowlan> |
Decommissioning cassandra instance restbase2009-a via nodetool |
[production] |
| 15:22 |
<arnoldokoth> |
systemctl reset-failed ifup@ens5.service on otrs1001 T273026 |
[production] |
| 15:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P18577 and previous config saved to /var/cache/conftool/dbconfig/20220111-151558-marostegui.json |
[production] |
| 15:10 |
<aokoth@cumin1001> |
END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM otrs1001.eqiad.wmnet |
[production] |
| 15:08 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM rpki1001.eqiad.wmnet |
[production] |
| 15:04 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.reboot-vm for VM rpki1001.eqiad.wmnet |
[production] |
| 15:02 |
<aokoth@cumin1001> |
START - Cookbook sre.ganeti.reboot-vm for VM otrs1001.eqiad.wmnet |
[production] |
| 15:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1141 (T297191)', diff saved to https://phabricator.wikimedia.org/P18576 and previous config saved to /var/cache/conftool/dbconfig/20220111-150054-marostegui.json |
[production] |
| 15:00 |
<aokoth@cumin1001> |
END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM etherpad1002.eqiad.wmnet |
[production] |
| 14:59 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1141 (T297191)', diff saved to https://phabricator.wikimedia.org/P18575 and previous config saved to /var/cache/conftool/dbconfig/20220111-145947-marostegui.json |
[production] |
| 14:59 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1141.eqiad.wmnet with reason: Maintenance |
[production] |
| 14:59 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1141.eqiad.wmnet with reason: Maintenance |
[production] |
| 14:59 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1142 (T297191)', diff saved to https://phabricator.wikimedia.org/P18574 and previous config saved to /var/cache/conftool/dbconfig/20220111-145939-marostegui.json |
[production] |
| 14:58 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM zookeeper-test1002.eqiad.wmnet |
[production] |
| 14:56 |
<aokoth@cumin1001> |
START - Cookbook sre.ganeti.reboot-vm for VM etherpad1002.eqiad.wmnet |
[production] |
| 14:48 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.reboot-vm for VM zookeeper-test1002.eqiad.wmnet |
[production] |
| 14:48 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ping1002.eqiad.wmnet |
[production] |
| 14:44 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.reboot-vm for VM ping1002.eqiad.wmnet |
[production] |
| 14:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P18573 and previous config saved to /var/cache/conftool/dbconfig/20220111-144435-marostegui.json |
[production] |
| 14:38 |
<XioNoX> |
disable ping-offload in eqiad |
[production] |
| 14:36 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 14:35 |
<marostegui> |
Upgrade pc1014 mysql |
[production] |
| 14:33 |
<taavi@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:751949|Clean up nova-network remains]] (2/2) (duration: 02m 40s) |
[production] |
| 14:32 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 14:32 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 14:31 |
<taavi@deploy1002> |
Synchronized wmf-config/CommonSettings.php: Config: [[gerrit:751949|Clean up nova-network remains]] (1/2) (duration: 02m 49s) |
[production] |
| 14:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P18572 and previous config saved to /var/cache/conftool/dbconfig/20220111-142930-marostegui.json |
[production] |
| 14:28 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 14:25 |
<taavi@deploy1002> |
Synchronized wmf-config/reverse-proxy.php: Config: [[gerrit:751952|reverse-proxy: add drmrs ranges (T282787)]] (duration: 01m 36s) |
[production] |
| 14:19 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbproxy1021.eqiad.wmnet with OS bullseye |
[production] |
| 14:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1142 (T297191)', diff saved to https://phabricator.wikimedia.org/P18571 and previous config saved to /var/cache/conftool/dbconfig/20220111-141425-marostegui.json |
[production] |
| 14:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1142 (T297191)', diff saved to https://phabricator.wikimedia.org/P18570 and previous config saved to /var/cache/conftool/dbconfig/20220111-141318-marostegui.json |
[production] |
| 14:13 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1142.eqiad.wmnet with reason: Maintenance |
[production] |
| 14:13 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1142.eqiad.wmnet with reason: Maintenance |
[production] |
| 14:13 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 12 hosts with reason: Maintenance |
[production] |
| 14:12 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on 12 hosts with reason: Maintenance |
[production] |
| 14:12 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance |
[production] |
| 14:12 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance |
[production] |
| 14:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1147 (T297191)', diff saved to https://phabricator.wikimedia.org/P18569 and previous config saved to /var/cache/conftool/dbconfig/20220111-141249-marostegui.json |
[production] |
| 13:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P18568 and previous config saved to /var/cache/conftool/dbconfig/20220111-135744-marostegui.json |
[production] |
| 13:50 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.reimage for host dbproxy1021.eqiad.wmnet with OS bullseye |
[production] |