|
2024-02-01
ยง
|
| 09:49 |
<joal@deploy2002> |
Started deploy [airflow-dags/analytics@6b84b7a]: (no justification provided) |
[production] |
| 09:43 |
<klausman@cumin2002> |
START - Cookbook sre.cassandra.roll-restart for nodes matching A:ml-cache-codfw: JRE update for DSA 5604 - klausman@cumin2002 |
[production] |
| 09:43 |
<klausman@cumin2002> |
END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:ml-cache-eqiad: JRE update for DSA 5604 - klausman@cumin2002 |
[production] |
| 09:36 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2119', diff saved to https://phabricator.wikimedia.org/P56046 and previous config saved to /var/cache/conftool/dbconfig/20240201-093621-marostegui.json |
[production] |
| 09:30 |
<vgutierrez@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host acmechief1001.eqiad.wmnet |
[production] |
| 09:26 |
<vgutierrez@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host acmechief1001.eqiad.wmnet |
[production] |
| 09:25 |
<klausman@cumin2002> |
START - Cookbook sre.cassandra.roll-restart for nodes matching A:ml-cache-eqiad: JRE update for DSA 5604 - klausman@cumin2002 |
[production] |
| 09:24 |
<vgutierrez@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host acmechief1002.eqiad.wmnet |
[production] |
| 09:21 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2119', diff saved to https://phabricator.wikimedia.org/P56045 and previous config saved to /var/cache/conftool/dbconfig/20240201-092115-marostegui.json |
[production] |
| 09:20 |
<vgutierrez@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host acmechief1002.eqiad.wmnet |
[production] |
| 09:18 |
<vgutierrez@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host acmechief2001.codfw.wmnet |
[production] |
| 09:14 |
<vgutierrez@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host acmechief2001.codfw.wmnet |
[production] |
| 09:12 |
<vgutierrez@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host acmechief2002.codfw.wmnet |
[production] |
| 09:08 |
<vgutierrez@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host acmechief2002.codfw.wmnet |
[production] |
| 09:06 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2119 (T355609)', diff saved to https://phabricator.wikimedia.org/P56044 and previous config saved to /var/cache/conftool/dbconfig/20240201-090607-marostegui.json |
[production] |
| 08:57 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2104 (re)pooling @ 100%: After switchover', diff saved to https://phabricator.wikimedia.org/P56043 and previous config saved to /var/cache/conftool/dbconfig/20240201-085743-root.json |
[production] |
| 08:52 |
<hashar> |
Restarted primary Gerrit on gerrit1003 |
[production] |
| 08:44 |
<vgutierrez@cumin1002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host acmechief-test2001.codfw.wmnet |
[production] |
| 08:42 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2104 (re)pooling @ 75%: After switchover', diff saved to https://phabricator.wikimedia.org/P56042 and previous config saved to /var/cache/conftool/dbconfig/20240201-084238-root.json |
[production] |
| 08:42 |
<hashar> |
Restarting Gerrit replica on gerrit2002 |
[production] |
| 08:41 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2119 (T355609)', diff saved to https://phabricator.wikimedia.org/P56041 and previous config saved to /var/cache/conftool/dbconfig/20240201-084126-marostegui.json |
[production] |
| 08:41 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2119.codfw.wmnet with reason: Maintenance |
[production] |
| 08:41 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2119.codfw.wmnet with reason: Maintenance |
[production] |
| 08:41 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2110 (T355609)', diff saved to https://phabricator.wikimedia.org/P56040 and previous config saved to /var/cache/conftool/dbconfig/20240201-084104-marostegui.json |
[production] |
| 08:40 |
<vgutierrez@cumin1002> |
START - Cookbook sre.hosts.reboot-single for host acmechief-test2001.codfw.wmnet |
[production] |
| 08:33 |
<vgutierrez@cumin1002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host acmechief-test1001.eqiad.wmnet |
[production] |
| 08:27 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2104 (re)pooling @ 50%: After switchover', diff saved to https://phabricator.wikimedia.org/P56039 and previous config saved to /var/cache/conftool/dbconfig/20240201-082733-root.json |
[production] |
| 08:26 |
<vgutierrez@cumin1002> |
START - Cookbook sre.hosts.reboot-single for host acmechief-test1001.eqiad.wmnet |
[production] |
| 08:25 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2110', diff saved to https://phabricator.wikimedia.org/P56038 and previous config saved to /var/cache/conftool/dbconfig/20240201-082558-marostegui.json |
[production] |
| 08:22 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti-test2003.codfw.wmnet |
[production] |
| 08:21 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2003.codfw.wmnet |
[production] |
| 08:15 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti-test2003.codfw.wmnet |
[production] |
| 08:13 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti-test2003.codfw.wmnet |
[production] |
| 08:12 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2104 (re)pooling @ 25%: After switchover', diff saved to https://phabricator.wikimedia.org/P56036 and previous config saved to /var/cache/conftool/dbconfig/20240201-081228-root.json |
[production] |
| 08:10 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2110', diff saved to https://phabricator.wikimedia.org/P56035 and previous config saved to /var/cache/conftool/dbconfig/20240201-081051-marostegui.json |
[production] |
| 07:57 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2104 (re)pooling @ 10%: After switchover', diff saved to https://phabricator.wikimedia.org/P56034 and previous config saved to /var/cache/conftool/dbconfig/20240201-075723-root.json |
[production] |
| 07:55 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2110 (T355609)', diff saved to https://phabricator.wikimedia.org/P56033 and previous config saved to /var/cache/conftool/dbconfig/20240201-075545-marostegui.json |
[production] |
| 07:45 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es2024 (re)pooling @ 100%: After switchover ', diff saved to https://phabricator.wikimedia.org/P56032 and previous config saved to /var/cache/conftool/dbconfig/20240201-074520-root.json |
[production] |
| 07:42 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2104 (re)pooling @ 5%: After switchover', diff saved to https://phabricator.wikimedia.org/P56031 and previous config saved to /var/cache/conftool/dbconfig/20240201-074218-root.json |
[production] |
| 07:33 |
<slyngs> |
Failover debmonitor to new Bookworm host |
[production] |
| 07:30 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2110 (T355609)', diff saved to https://phabricator.wikimedia.org/P56030 and previous config saved to /var/cache/conftool/dbconfig/20240201-073053-marostegui.json |
[production] |
| 07:30 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance |
[production] |
| 07:30 |
<moritzm> |
installing openjdk-11 security updates |
[production] |
| 07:30 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance |
[production] |
| 07:30 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2106 (T355609)', diff saved to https://phabricator.wikimedia.org/P56029 and previous config saved to /var/cache/conftool/dbconfig/20240201-073031-marostegui.json |
[production] |
| 07:30 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es2024 (re)pooling @ 75%: After switchover ', diff saved to https://phabricator.wikimedia.org/P56028 and previous config saved to /var/cache/conftool/dbconfig/20240201-073015-root.json |
[production] |
| 07:27 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2104 (re)pooling @ 1%: After switchover', diff saved to https://phabricator.wikimedia.org/P56027 and previous config saved to /var/cache/conftool/dbconfig/20240201-072713-root.json |
[production] |
| 07:22 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti-test2002.codfw.wmnet |
[production] |
| 07:22 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2002.codfw.wmnet |
[production] |
| 07:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2104 T356374', diff saved to https://phabricator.wikimedia.org/P56026 and previous config saved to /var/cache/conftool/dbconfig/20240201-071934-root.json |
[production] |