|
2026-03-17
ยง
|
| 14:49 |
<topranks> |
stop announcing routes from ssw1-d8-eqiad to external peers (cr2-eqiad, other spines) T420351 |
[production] |
| 14:49 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti2034.codfw.wmnet |
[production] |
| 14:48 |
<filippo@cumin1003> |
START - Cookbook sre.hosts.reboot-single for host cloudrabbit2002-dev.codfw.wmnet |
[production] |
| 14:48 |
<filippo@cumin1003> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudrabbit2001-dev.codfw.wmnet |
[production] |
| 14:46 |
<mvernon@cumin1003> |
START - Cookbook sre.hosts.reboot-single for host thanos-be2007.codfw.wmnet |
[production] |
| 14:46 |
<mvernon@cumin1003> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be2006.codfw.wmnet |
[production] |
| 14:45 |
<jelto@cumin1003> |
START - Cookbook sre.hosts.reboot-single for host gitlab2002.wikimedia.org |
[production] |
| 14:44 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2034.codfw.wmnet |
[production] |
| 14:44 |
<taavi> |
deploying cr firewall changes from https://gerrit.wikimedia.org/r/c/operations/homer/public/+/1254211 |
[production] |
| 14:44 |
<topranks> |
stop announcing "direct" routes to ssw1-d8-eqiad from cr2-eqiad T420351 |
[production] |
| 14:44 |
<jmm@cumin2002> |
END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti2034.codfw.wmnet |
[production] |
| 14:44 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2034.codfw.wmnet |
[production] |
| 14:43 |
<moritzm> |
failover Ganeti master in codfw to ganeti2047 |
[production] |
| 14:42 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2050.codfw.wmnet |
[production] |
| 14:42 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet |
[production] |
| 14:41 |
<filippo@cumin1003> |
START - Cookbook sre.hosts.reboot-single for host cloudrabbit2001-dev.codfw.wmnet |
[production] |
| 14:41 |
<filippo@cumin1003> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudnet2007-dev.codfw.wmnet |
[production] |
| 14:40 |
<topranks> |
disabling EVPN IBGP peering from ssw1-d8-eqiad to ssw1-d1-eqiad to stop them reflecting routes T420351 |
[production] |
| 14:40 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1006.eqiad.wmnet |
[production] |
| 14:39 |
<jelto@cumin1003> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab2002.wikimedia.org |
[production] |
| 14:38 |
<inflatador> |
bking@requestctl remove `wdqs_highest_error_rate_ever_seen` requestctl rule as it is no longer needed |
[production] |
| 14:38 |
<mvernon@cumin1003> |
START - Cookbook sre.hosts.reboot-single for host thanos-be2006.codfw.wmnet |
[production] |
| 14:37 |
<mvernon@cumin1003> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be2005.codfw.wmnet |
[production] |
| 14:36 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet |
[production] |
| 14:35 |
<filippo@cumin1003> |
START - Cookbook sre.hosts.reboot-single for host cloudnet2007-dev.codfw.wmnet |
[production] |
| 14:35 |
<filippo@cumin1003> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudnet2006-dev.codfw.wmnet |
[production] |
| 14:34 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host sretest1006.eqiad.wmnet |
[production] |
| 14:34 |
<Daimona> |
Creating ce_event_goals DB table for the CampaignEvents extension in x1.testwiki, x1.test2wiki, x1.officewiki, and x1.wikishared # T411433 |
[production] |
| 14:34 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2050.codfw.wmnet |
[production] |
| 14:33 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2049.codfw.wmnet |
[production] |
| 14:33 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2049.codfw.wmnet |
[production] |
| 14:31 |
<jelto@cumin1003> |
START - Cookbook sre.hosts.reboot-single for host gitlab2002.wikimedia.org |
[production] |
| 14:30 |
<mvernon@cumin1003> |
START - Cookbook sre.hosts.reboot-single for host thanos-be2005.codfw.wmnet |
[production] |
| 14:27 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti2049.codfw.wmnet |
[production] |
| 14:27 |
<topranks> |
de-pref internet circuits landing on cr2-eqiad to shift traffic to cr1 T420351 |
[production] |
| 14:27 |
<filippo@cumin1003> |
START - Cookbook sre.hosts.reboot-single for host cloudnet2006-dev.codfw.wmnet |
[production] |
| 14:27 |
<filippo@cumin1003> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudnet2005-dev.codfw.wmnet |
[production] |
| 14:25 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2049.codfw.wmnet |
[production] |
| 14:25 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2047.codfw.wmnet |
[production] |
| 14:25 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2047.codfw.wmnet |
[production] |
| 14:23 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-test-presto1001.eqiad.wmnet |
[production] |
| 14:19 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host an-test-presto1001.eqiad.wmnet |
[production] |
| 14:19 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti2047.codfw.wmnet |
[production] |
| 14:19 |
<filippo@cumin1003> |
START - Cookbook sre.hosts.reboot-single for host cloudnet2005-dev.codfw.wmnet |
[production] |
| 14:19 |
<filippo@cumin1003> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudlb2004-dev.codfw.wmnet |
[production] |
| 14:17 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp-test1005.wikimedia.org |
[production] |
| 14:16 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2047.codfw.wmnet |
[production] |
| 14:14 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2046.codfw.wmnet |
[production] |
| 14:14 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2046.codfw.wmnet |
[production] |
| 14:13 |
<topranks> |
disable VRRP on cr2-eqiad interfaces facing ssw1-d8-eqiad T420351 |
[production] |