3401-3450 of 10000 results (15ms)
2025-03-17 §
10:17 <marostegui@cumin1002> START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section s2 [production]
10:17 <marostegui@cumin1002> END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section s3 [production]
10:16 <marostegui@cumin1002> START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section s3 [production]
10:15 <marostegui@cumin1002> END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section s4 [production]
10:14 <marostegui@cumin1002> START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section s4 [production]
10:14 <marostegui@cumin1002> END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section s5 [production]
10:13 <marostegui@deploy2002> marostegui: Continuing with sync [production]
10:12 <marostegui@cumin1002> START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section s5 [production]
10:12 <marostegui@cumin1002> END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section s8 [production]
10:11 <marostegui@deploy2002> marostegui: Backport for [[gerrit:1128356|Revert "db-production.php: Disable writes on es6"]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
10:10 <marostegui@cumin1002> START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section s8 [production]
10:10 <marostegui@cumin1002> END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section s7 [production]
10:09 <marostegui@cumin1002> START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section s7 [production]
10:09 <marostegui@cumin1002> END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section s6 [production]
10:07 <marostegui@cumin1002> START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section s6 [production]
10:07 <marostegui@deploy2002> Started scap sync-world: Backport for [[gerrit:1128356|Revert "db-production.php: Disable writes on es6"]] [production]
10:07 <marostegui@cumin1002> END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section x1 [production]
10:06 <marostegui@cumin1002> START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section x1 [production]
10:06 <marostegui@cumin1002> END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section es7 [production]
10:05 <marostegui@cumin1002> START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section es7 [production]
10:02 <marostegui@cumin1002> END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section es6 [production]
10:01 <marostegui@cumin1002> START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section es6 [production]
10:00 <marostegui@deploy2002> Finished scap sync-world: Backport for [[gerrit:1128351|db-production.php: Disable writes on es6 (T388626)]] (duration: 23m 25s) [production]
09:56 <vgutierrez@cumin1002> END (FAIL) - Cookbook sre.loadbalancer.upgrade (exit_code=1) upgradeing P{lvs4010.ulsfo.wmnet} and A:liberica [production]
09:55 <vgutierrez@cumin1002> START - Cookbook sre.loadbalancer.upgrade upgradeing P{lvs4010.ulsfo.wmnet} and A:liberica [production]
09:50 <marostegui@deploy2002> marostegui: Continuing with sync [production]
09:50 <marostegui@deploy2002> marostegui: Backport for [[gerrit:1128351|db-production.php: Disable writes on es6 (T388626)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
09:45 <vgutierrez@cumin1002> START - Cookbook sre.hosts.reimage for host lvs3010.esams.wmnet with OS bookworm [production]
09:37 <marostegui@deploy2002> Started scap sync-world: Backport for [[gerrit:1128351|db-production.php: Disable writes on es6 (T388626)]] [production]
09:25 <moritzm> installing intel-microcode security updates [production]
09:19 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1002.eqiad.wmnet [production]
09:14 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host sretest1002.eqiad.wmnet [production]
08:46 <moritzm> freed 28G of disk space on maps1009 [production]
08:30 <moritzm> updated bookworm installer image to Bookworm 12.10 T389034 [production]
08:23 <elukey@deploy2002> helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: sync [production]
08:23 <elukey@deploy2002> helmfile [staging] START helmfile.d/services/changeprop-jobqueue: sync [production]
08:23 <elukey@deploy2002> helmfile [staging] DONE helmfile.d/services/changeprop: sync [production]
08:23 <elukey@deploy2002> helmfile [staging] START helmfile.d/services/changeprop: sync [production]
07:29 <marostegui@cumin2002> END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section x1 [production]
07:28 <marostegui@cumin2002> START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section x1 [production]
07:17 <marostegui@cumin1002> END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section x1 [production]
07:17 <marostegui@cumin1002> START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section x1 [production]
07:16 <marostegui@cumin1002> END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 [production]
07:15 <marostegui@cumin1002> START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 [production]
07:14 <marostegui@cumin1002> END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section x1 [production]
07:13 <marostegui@cumin1002> START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section x1 [production]
2025-03-16 §
14:08 <claime> sudo postqueue -j | jq -r 'select(.sender == "vrts-bounce@wikimedia.org") | .queue_id' | sudo postsuper -d - # mx-out1001 [production]
13:59 <Emperor> sudo postqueue -j | jq -r ' select(.recipients[0].address == "vrts-bounce@wikimedia.org") | select(.recipients[1].address == null) | .queue_id' | sudo postsuper -d - # mx-in2001 [production]
13:39 <Emperor> restart postfix on mx-in2001 [production]
11:42 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38 [tools]