|
2025-07-07
ยง
|
| 11:25 |
<ladsgroup@deploy1003> |
Started scap sync-world: Backport for [[gerrit:1165169|Revert^2 "Clean up EventBus and jobs config"]] |
[production] |
| 11:23 |
<taavi@cloudcumin1001> |
END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld |
[tools] |
| 11:23 |
<taavi@cloudcumin1001> |
START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld |
[tools] |
| 11:23 |
<taavi@cloudcumin1001> |
END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld |
[toolsbeta] |
| 11:23 |
<taavi@cloudcumin1001> |
START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld |
[toolsbeta] |
| 11:22 |
<taavi@cloudcumin1001> |
END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld |
[toolsbeta] |
| 11:11 |
<taavi@cloudcumin1001> |
START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld |
[toolsbeta] |
| 11:10 |
<root@cumin1002> |
END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe |
[production] |
| 11:06 |
<root@cumin1002> |
START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe |
[production] |
| 11:05 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1250.eqiad.wmnet with reason: Maintenance |
[production] |
| 11:02 |
<moritzm> |
installing modsecurity-apache security updates |
[production] |
| 10:53 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1217.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:45 |
<elukey@cumin2002> |
END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm |
[production] |
| 10:45 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2234].codfw.wmnet with reason: Maintenance |
[production] |
| 10:35 |
<elukey@cumin2002> |
START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm |
[production] |
| 10:30 |
<elukey@cumin2002> |
END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm |
[production] |
| 10:24 |
<Emperor> |
remove swift-account-stats_machinetranslation:prod time & service from thanos-fe1004 T335491 |
[production] |
| 10:17 |
<elukey@cumin2002> |
START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm |
[production] |
| 10:13 |
<root@cumin1002> |
END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe |
[production] |
| 10:09 |
<root@cumin1002> |
START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe |
[production] |
| 09:58 |
<elukey@cumin2002> |
END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm |
[production] |
| 09:43 |
<elukey@cumin2002> |
START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm |
[production] |
| 09:42 |
<elukey@cumin2002> |
END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm |
[production] |
| 09:25 |
<elukey@cumin2002> |
START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm |
[production] |
| 09:21 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1250.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:18 |
<elukey@cumin2002> |
END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm |
[production] |
| 09:13 |
<marostegui> |
Failover m2 from db1250 to db1228 - T397633 |
[production] |
| 09:09 |
<elukey@cumin2002> |
START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm |
[production] |
| 09:06 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2233].codfw.wmnet,db[1217,1228,1250].eqiad.wmnet with reason: maintenance |
[production] |
| 08:45 |
<hashar> |
gerrit: change operations/* submit strategy to "Rebase if Necessary" and "Allow content merge" | T390719 |
[releng] |
| 08:45 |
<hashar> |
gerrit: change mediawiki/* submit strategy to "Rebase if Necessary" and "Allow content merge" | T390719 |
[releng] |
| 08:26 |
<taavi@cloudcumin1001> |
END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging |
[tools] |
| 08:21 |
<taavi@cloudcumin1001> |
START - Cookbook wmcs.toolforge.component.deploy for component logging |
[tools] |
| 08:19 |
<taavi@cloudcumin1001> |
END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging |
[toolsbeta] |
| 08:19 |
<taavi@cloudcumin1001> |
START - Cookbook wmcs.toolforge.component.deploy for component logging |
[toolsbeta] |
| 08:15 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1237.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:01 |
<oblivian@cumin1003> |
END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "Feature: logging of deny actions; add rename functionality - oblivian@cumin1003" |
[production] |
| 08:01 |
<oblivian@cumin1003> |
END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003 |
[production] |
| 08:00 |
<oblivian@cumin1003> |
START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003 |
[production] |
| 08:00 |
<oblivian@cumin1003> |
START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "Feature: logging of deny actions; add rename functionality - oblivian@cumin1003" |
[production] |
| 08:00 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1237.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:53 |
<vgutierrez> |
repooling cp7006 with Ia82b9354a5b9e7bd5443b4af0888325919ddb19e applied - T397917 |
[production] |
| 07:53 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db1237 T397612', diff saved to https://phabricator.wikimedia.org/P78763 and previous config saved to /var/cache/conftool/dbconfig/20250707-075308-root.json |
[production] |
| 07:52 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write T397612', diff saved to https://phabricator.wikimedia.org/P78762 and previous config saved to /var/cache/conftool/dbconfig/20250707-075254-root.json |
[production] |
| 07:51 |
<marostegui@dns1006> |
END - running authdns-update |
[production] |
| 07:50 |
<marostegui@dns1006> |
START - running authdns-update |
[production] |
| 07:25 |
<vgutierrez> |
depooling cp7006 to test Ia82b9354a5b9e7bd5443b4af0888325919ddb19e - T397917 |
[production] |
| 07:25 |
<marostegui> |
Starting x1 eqiad failover from db1237 to db1220 - T397612 |
[production] |
| 07:21 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Set db1220 with weight 0 T397612', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json |
[production] |
| 07:13 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 T397612 |
[production] |