|
2021-01-18
ยง
|
| 14:31 |
<kormat@cumin1001> |
END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) |
[production] |
| 14:30 |
<arturo> |
updating packages in buster-wikimedia/thirdparty/ceph-nautilus-buster (T272296) |
[production] |
| 14:26 |
<kormat@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 14:26 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 14:18 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 13:34 |
<moritzm> |
uploaded wmf-sre-laptop 0.3.2 to apt.wikimedia.org |
[production] |
| 13:26 |
<volans> |
installed spicerack 0.0.48-1+deb10u1 on cumin hosts |
[production] |
| 13:12 |
<marostegui> |
Upgrade db2071 to 10.4.17 - T268457 |
[production] |
| 13:08 |
<XioNoX> |
add NAT rule on pfw3-eqiad - T272066 |
[production] |
| 12:56 |
<XioNoX> |
add NAT rule on pfw3-codfw - T272066 |
[production] |
| 12:36 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-fe2008.codfw.wmnet |
[production] |
| 12:33 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1127.eqiad.wmnet with reason: REIMAGE |
[production] |
| 12:32 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ms-fe2008.codfw.wmnet |
[production] |
| 12:31 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1127.eqiad.wmnet with reason: REIMAGE |
[production] |
| 12:27 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1126.eqiad.wmnet with reason: REIMAGE |
[production] |
| 12:25 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1126.eqiad.wmnet with reason: REIMAGE |
[production] |
| 12:22 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1125.eqiad.wmnet with reason: REIMAGE |
[production] |
| 12:21 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-fe2007.codfw.wmnet |
[production] |
| 12:20 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1125.eqiad.wmnet with reason: REIMAGE |
[production] |
| 12:18 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ms-fe2007.codfw.wmnet |
[production] |
| 12:14 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-fe2006.codfw.wmnet |
[production] |
| 12:13 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1124.eqiad.wmnet with reason: REIMAGE |
[production] |
| 12:11 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1124.eqiad.wmnet with reason: REIMAGE |
[production] |
| 12:08 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ms-fe2006.codfw.wmnet |
[production] |
| 12:08 |
<volans> |
uploaded spicerack_0.0.48 to apt.wikimedia.org buster-wikimedia |
[production] |
| 12:04 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-fe2005.codfw.wmnet |
[production] |
| 12:03 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1123.eqiad.wmnet with reason: REIMAGE |
[production] |
| 12:01 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1123.eqiad.wmnet with reason: REIMAGE |
[production] |
| 11:54 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ms-fe2005.codfw.wmnet |
[production] |
| 11:52 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1122.eqiad.wmnet with reason: REIMAGE |
[production] |
| 11:52 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-fe1008.eqiad.wmnet |
[production] |
| 11:49 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1122.eqiad.wmnet with reason: REIMAGE |
[production] |
| 11:48 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ms-fe1008.eqiad.wmnet |
[production] |
| 11:44 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-fe1007.eqiad.wmnet |
[production] |
| 11:40 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ms-fe1007.eqiad.wmnet |
[production] |
| 11:37 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1121.eqiad.wmnet with reason: REIMAGE |
[production] |
| 11:35 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1121.eqiad.wmnet with reason: REIMAGE |
[production] |
| 11:33 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-fe1006.eqiad.wmnet |
[production] |
| 11:28 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ms-fe1006.eqiad.wmnet |
[production] |
| 11:22 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1120.eqiad.wmnet with reason: REIMAGE |
[production] |
| 11:18 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1120.eqiad.wmnet with reason: REIMAGE |
[production] |
| 11:17 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-fe1005.eqiad.wmnet |
[production] |
| 11:11 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ms-fe1005.eqiad.wmnet |
[production] |
| 11:10 |
<hashar> |
Restarting Gerrit main instance on gerrit1001.wikimedia.org |
[production] |
| 11:08 |
<hashar> |
Restarting Gerrit replica on gerrit2001.wikimedia.org |
[production] |
| 10:58 |
<moritzm> |
installing python2.7 security updates on Stretch |
[production] |
| 10:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1074 (re)pooling @ 100%: After moving wikireplicas to another host', diff saved to https://phabricator.wikimedia.org/P13799 and previous config saved to /var/cache/conftool/dbconfig/20210118-102959-root.json |
[production] |
| 10:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1074 (re)pooling @ 75%: After moving wikireplicas to another host', diff saved to https://phabricator.wikimedia.org/P13798 and previous config saved to /var/cache/conftool/dbconfig/20210118-101456-root.json |
[production] |
| 10:00 |
<_joe_> |
restarting pybal on lvs1016, not talking to its etcd server |
[production] |
| 09:59 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1074 (re)pooling @ 50%: After moving wikireplicas to another host', diff saved to https://phabricator.wikimedia.org/P13797 and previous config saved to /var/cache/conftool/dbconfig/20210118-095952-root.json |
[production] |