|
2026-03-17
ยง
|
| 15:02 |
<dreamyjazz@deploy2002> |
Started scap sync-world: Backport for [[gerrit:1254217|Create dblists for wikis where CheckUser and AbuseFilter are disabled (T420063 T420062)]] |
[production] |
| 15:02 |
<topranks> |
reset BGP session to ssw1-d8-eiqad from lsw1-d4-eqiad T420180 |
[production] |
| 15:02 |
<kevinbazira@deploy2002> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . |
[production] |
| 15:02 |
<filippo@cumin1003> |
START - Cookbook sre.hosts.reboot-single for host cloudservices2004-dev.codfw.wmnet |
[production] |
| 15:02 |
<filippo@cumin1003> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudrabbit2003-dev.codfw.wmnet |
[production] |
| 15:00 |
<mvernon@cumin1003> |
START - Cookbook sre.hosts.reboot-single for host thanos-be2009.codfw.wmnet |
[production] |
| 15:00 |
<mvernon@cumin1003> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be2008.codfw.wmnet |
[production] |
| 14:57 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2034.codfw.wmnet |
[production] |
| 14:57 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2034.codfw.wmnet |
[production] |
| 14:57 |
<jelto@cumin1003> |
START - Cookbook sre.hosts.reboot-single for host phab1004.eqiad.wmnet |
[production] |
| 14:55 |
<filippo@cumin1003> |
START - Cookbook sre.hosts.reboot-single for host cloudrabbit2003-dev.codfw.wmnet |
[production] |
| 14:55 |
<filippo@cumin1003> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudrabbit2002-dev.codfw.wmnet |
[production] |
| 14:54 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host durum4004.ulsfo.wmnet |
[production] |
| 14:53 |
<jmm@deploy2002> |
helmfile [staging] DONE helmfile.d/services/thumbor: apply |
[production] |
| 14:53 |
<jmm@deploy2002> |
helmfile [staging] START helmfile.d/services/thumbor: apply |
[production] |
| 14:52 |
<mvernon@cumin1003> |
START - Cookbook sre.hosts.reboot-single for host thanos-be2008.codfw.wmnet |
[production] |
| 14:52 |
<mvernon@cumin1003> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be2007.codfw.wmnet |
[production] |
| 14:51 |
<topranks> |
stop accepting routes on ssw1-d8-eqiad from external peers (cr2-eqiad, other spines) T420351 |
[production] |
| 14:51 |
<jelto@cumin1003> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab2002.wikimedia.org |
[production] |
| 14:50 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host durum4004.ulsfo.wmnet |
[production] |
| 14:49 |
<topranks> |
stop announcing routes from ssw1-d8-eqiad to external peers (cr2-eqiad, other spines) T420351 |
[production] |
| 14:49 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti2034.codfw.wmnet |
[production] |
| 14:48 |
<filippo@cumin1003> |
START - Cookbook sre.hosts.reboot-single for host cloudrabbit2002-dev.codfw.wmnet |
[production] |
| 14:48 |
<filippo@cumin1003> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudrabbit2001-dev.codfw.wmnet |
[production] |
| 14:46 |
<mvernon@cumin1003> |
START - Cookbook sre.hosts.reboot-single for host thanos-be2007.codfw.wmnet |
[production] |
| 14:46 |
<mvernon@cumin1003> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be2006.codfw.wmnet |
[production] |
| 14:45 |
<jelto@cumin1003> |
START - Cookbook sre.hosts.reboot-single for host gitlab2002.wikimedia.org |
[production] |
| 14:44 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2034.codfw.wmnet |
[production] |
| 14:44 |
<taavi> |
deploying cr firewall changes from https://gerrit.wikimedia.org/r/c/operations/homer/public/+/1254211 |
[production] |
| 14:44 |
<topranks> |
stop announcing "direct" routes to ssw1-d8-eqiad from cr2-eqiad T420351 |
[production] |
| 14:44 |
<jmm@cumin2002> |
END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti2034.codfw.wmnet |
[production] |
| 14:44 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2034.codfw.wmnet |
[production] |
| 14:43 |
<moritzm> |
failover Ganeti master in codfw to ganeti2047 |
[production] |
| 14:42 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2050.codfw.wmnet |
[production] |
| 14:42 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet |
[production] |
| 14:41 |
<filippo@cumin1003> |
START - Cookbook sre.hosts.reboot-single for host cloudrabbit2001-dev.codfw.wmnet |
[production] |
| 14:41 |
<filippo@cumin1003> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudnet2007-dev.codfw.wmnet |
[production] |
| 14:40 |
<topranks> |
disabling EVPN IBGP peering from ssw1-d8-eqiad to ssw1-d1-eqiad to stop them reflecting routes T420351 |
[production] |
| 14:40 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1006.eqiad.wmnet |
[production] |
| 14:39 |
<jelto@cumin1003> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab2002.wikimedia.org |
[production] |
| 14:38 |
<inflatador> |
bking@requestctl remove `wdqs_highest_error_rate_ever_seen` requestctl rule as it is no longer needed |
[production] |
| 14:38 |
<mvernon@cumin1003> |
START - Cookbook sre.hosts.reboot-single for host thanos-be2006.codfw.wmnet |
[production] |
| 14:37 |
<mvernon@cumin1003> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be2005.codfw.wmnet |
[production] |
| 14:36 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet |
[production] |
| 14:35 |
<filippo@cumin1003> |
START - Cookbook sre.hosts.reboot-single for host cloudnet2007-dev.codfw.wmnet |
[production] |
| 14:35 |
<filippo@cumin1003> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudnet2006-dev.codfw.wmnet |
[production] |
| 14:34 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host sretest1006.eqiad.wmnet |
[production] |
| 14:34 |
<Daimona> |
Creating ce_event_goals DB table for the CampaignEvents extension in x1.testwiki, x1.test2wiki, x1.officewiki, and x1.wikishared # T411433 |
[production] |
| 14:34 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2050.codfw.wmnet |
[production] |
| 14:33 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2049.codfw.wmnet |
[production] |