5251-5300 of 10000 results (113ms)
2025-03-17 §
10:00 <marostegui@deploy2002> Finished scap sync-world: Backport for [[gerrit:1128351|db-production.php: Disable writes on es6 (T388626)]] (duration: 23m 25s) [production]
09:56 <vgutierrez@cumin1002> END (FAIL) - Cookbook sre.loadbalancer.upgrade (exit_code=1) upgradeing P{lvs4010.ulsfo.wmnet} and A:liberica [production]
09:55 <vgutierrez@cumin1002> START - Cookbook sre.loadbalancer.upgrade upgradeing P{lvs4010.ulsfo.wmnet} and A:liberica [production]
09:50 <marostegui@deploy2002> marostegui: Continuing with sync [production]
09:50 <marostegui@deploy2002> marostegui: Backport for [[gerrit:1128351|db-production.php: Disable writes on es6 (T388626)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
09:45 <vgutierrez@cumin1002> START - Cookbook sre.hosts.reimage for host lvs3010.esams.wmnet with OS bookworm [production]
09:37 <marostegui@deploy2002> Started scap sync-world: Backport for [[gerrit:1128351|db-production.php: Disable writes on es6 (T388626)]] [production]
09:25 <moritzm> installing intel-microcode security updates [production]
09:19 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1002.eqiad.wmnet [production]
09:14 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host sretest1002.eqiad.wmnet [production]
08:46 <moritzm> freed 28G of disk space on maps1009 [production]
08:30 <moritzm> updated bookworm installer image to Bookworm 12.10 T389034 [production]
08:23 <elukey@deploy2002> helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: sync [production]
08:23 <elukey@deploy2002> helmfile [staging] START helmfile.d/services/changeprop-jobqueue: sync [production]
08:23 <elukey@deploy2002> helmfile [staging] DONE helmfile.d/services/changeprop: sync [production]
08:23 <elukey@deploy2002> helmfile [staging] START helmfile.d/services/changeprop: sync [production]
07:29 <marostegui@cumin2002> END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section x1 [production]
07:28 <marostegui@cumin2002> START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section x1 [production]
07:17 <marostegui@cumin1002> END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section x1 [production]
07:17 <marostegui@cumin1002> START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section x1 [production]
07:16 <marostegui@cumin1002> END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 [production]
07:15 <marostegui@cumin1002> START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 [production]
07:14 <marostegui@cumin1002> END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section x1 [production]
07:13 <marostegui@cumin1002> START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section x1 [production]
2025-03-16 §
14:08 <claime> sudo postqueue -j | jq -r 'select(.sender == "vrts-bounce@wikimedia.org") | .queue_id' | sudo postsuper -d - # mx-out1001 [production]
13:59 <Emperor> sudo postqueue -j | jq -r ' select(.recipients[0].address == "vrts-bounce@wikimedia.org") | select(.recipients[1].address == null) | .queue_id' | sudo postsuper -d - # mx-in2001 [production]
13:39 <Emperor> restart postfix on mx-in2001 [production]
2025-03-15 §
18:11 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti2046.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
18:00 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host ganeti2046.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
00:51 <jhancock@cumin2002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti2046 [production]
00:51 <jhancock@cumin2002> START - Cookbook sre.network.configure-switch-interfaces for host ganeti2046 [production]
2025-03-14 §
21:25 <zabe> zabe@mwmaint2002:~$ mwscript extensions/WikimediaMaintenance/migrateESRefToContentTableStage2.php testwiki --delete /home/zabe/afl_text_table_deletedump/testwiki --sleep 0.3 # T381599 [production]
21:06 <zabe> zabe@mwmaint2002:~$ mwscript extensions/AbuseFilter/maintenance/MigrateESRefToAflTable.php testwiki --dump /home/zabe/afl_text_table_dump/testwiki --deletedump /home/zabe/afl_text_table_deletedump/testwiki --sleep 0.3 # T381599 [production]
16:51 <root@deploy2002> helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'sync'. [production]
16:51 <root@deploy2002> helmfile [aux-k8s-codfw] START helmfile.d/admin 'sync'. [production]
16:15 <slyngshede@cumin1002> DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Vivian Rook out of all services on: 2288 hosts [production]
16:14 <slyngshede@cumin1002> DONE (FAIL) - Cookbook sre.idm.logout (exit_code=99) Logging Vivian Rook out of all services on: 2288 hosts [production]
16:05 <sukhe@cumin1002> END (FAIL) - Cookbook sre.loadbalancer.admin (exit_code=1) pooling A:liberica-canary [production]
16:05 <sukhe@cumin1002> START - Cookbook sre.loadbalancer.admin pooling A:liberica-canary [production]
16:04 <sukhe@cumin1002> END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling A:liberica-canary [production]
16:04 <sukhe@cumin1002> START - Cookbook sre.loadbalancer.admin depooling A:liberica-canary [production]
16:04 <sukhe@cumin1002> END (FAIL) - Cookbook sre.loadbalancer.admin (exit_code=1) pooling A:liberica-canary [production]
16:03 <sukhe@cumin1002> START - Cookbook sre.loadbalancer.admin pooling A:liberica-canary [production]
16:01 <sukhe@cumin1002> END (FAIL) - Cookbook sre.loadbalancer.admin (exit_code=1) pooling A:liberica-canary [production]
16:00 <sukhe@cumin1002> START - Cookbook sre.loadbalancer.admin pooling A:liberica-canary [production]
16:00 <sukhe@cumin1002> END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling A:liberica-canary [production]
16:00 <sukhe@cumin1002> START - Cookbook sre.loadbalancer.admin depooling A:liberica-canary [production]
15:50 <slyngshede@cumin1002> DONE (FAIL) - Cookbook sre.idm.logout (exit_code=99) Logging Vivian Rook out of all services on: 2288 hosts [production]
15:41 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be2075.codfw.wmnet with OS bullseye [production]
15:41 <jhancock@cumin2002> START - Cookbook sre.hosts.reimage for host ms-be2075.codfw.wmnet with OS bullseye [production]