851-900 of 10000 results (22ms)
2025-07-07 ยง
11:25 <ladsgroup@deploy1003> Started scap sync-world: Backport for [[gerrit:1165169|Revert^2 "Clean up EventBus and jobs config"]] [production]
11:23 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld [tools]
11:23 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld [tools]
11:23 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld [toolsbeta]
11:23 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld [toolsbeta]
11:22 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld [toolsbeta]
11:11 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld [toolsbeta]
11:10 <root@cumin1002> END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe [production]
11:06 <root@cumin1002> START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe [production]
11:05 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1250.eqiad.wmnet with reason: Maintenance [production]
11:02 <moritzm> installing modsecurity-apache security updates [production]
10:53 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1217.eqiad.wmnet with reason: Maintenance [production]
10:45 <elukey@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm [production]
10:45 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2234].codfw.wmnet with reason: Maintenance [production]
10:35 <elukey@cumin2002> START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm [production]
10:30 <elukey@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm [production]
10:24 <Emperor> remove swift-account-stats_machinetranslation:prod time & service from thanos-fe1004 T335491 [production]
10:17 <elukey@cumin2002> START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm [production]
10:13 <root@cumin1002> END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe [production]
10:09 <root@cumin1002> START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe [production]
09:58 <elukey@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm [production]
09:43 <elukey@cumin2002> START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm [production]
09:42 <elukey@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm [production]
09:25 <elukey@cumin2002> START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm [production]
09:21 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1250.eqiad.wmnet with reason: Maintenance [production]
09:18 <elukey@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm [production]
09:13 <marostegui> Failover m2 from db1250 to db1228 - T397633 [production]
09:09 <elukey@cumin2002> START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm [production]
09:06 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2233].codfw.wmnet,db[1217,1228,1250].eqiad.wmnet with reason: maintenance [production]
08:45 <hashar> gerrit: change operations/* submit strategy to "Rebase if Necessary" and "Allow content merge" | T390719 [releng]
08:45 <hashar> gerrit: change mediawiki/* submit strategy to "Rebase if Necessary" and "Allow content merge" | T390719 [releng]
08:26 <taavi@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging [tools]
08:21 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component logging [tools]
08:19 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging [toolsbeta]
08:19 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component logging [toolsbeta]
08:15 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1237.eqiad.wmnet with reason: Maintenance [production]
08:01 <oblivian@cumin1003> END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "Feature: logging of deny actions; add rename functionality - oblivian@cumin1003" [production]
08:01 <oblivian@cumin1003> END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003 [production]
08:00 <oblivian@cumin1003> START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Feature: logging of deny actions; add rename functionality - oblivian@cumin1003 [production]
08:00 <oblivian@cumin1003> START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "Feature: logging of deny actions; add rename functionality - oblivian@cumin1003" [production]
08:00 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1237.eqiad.wmnet with reason: Maintenance [production]
07:53 <vgutierrez> repooling cp7006 with Ia82b9354a5b9e7bd5443b4af0888325919ddb19e applied - T397917 [production]
07:53 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1237 T397612', diff saved to https://phabricator.wikimedia.org/P78763 and previous config saved to /var/cache/conftool/dbconfig/20250707-075308-root.json [production]
07:52 <marostegui@cumin1002> dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write T397612', diff saved to https://phabricator.wikimedia.org/P78762 and previous config saved to /var/cache/conftool/dbconfig/20250707-075254-root.json [production]
07:51 <marostegui@dns1006> END - running authdns-update [production]
07:50 <marostegui@dns1006> START - running authdns-update [production]
07:25 <vgutierrez> depooling cp7006 to test Ia82b9354a5b9e7bd5443b4af0888325919ddb19e - T397917 [production]
07:25 <marostegui> Starting x1 eqiad failover from db1237 to db1220 - T397612 [production]
07:21 <marostegui@cumin1002> dbctl commit (dc=all): 'Set db1220 with weight 0 T397612', diff saved to https://phabricator.wikimedia.org/P78760 and previous config saved to /var/cache/conftool/dbconfig/20250707-072157-root.json [production]
07:13 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 T397612 [production]