7401-7450 of 10000 results (104ms)
2024-03-20 ยง
15:51 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 6 hosts with reason: Remove circular replication in es4 T358200 [production]
15:50 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 0:30:00 on 6 hosts with reason: Remove circular replication in es4 T358200 [production]
15:49 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 6 hosts with reason: Remove circular replication in es5 T358200 [production]
15:49 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 0:30:00 on 6 hosts with reason: Remove circular replication in es5 T358200 [production]
15:48 <moritzm> installing usbutils bugfix updates from Bookworm point release [production]
15:45 <fabfur@cumin1002> conftool action : set/pooled=no; selector: name=cp4037.ulsfo.wmnet [production]
15:30 <fabfur@cumin1002> conftool action : set/pooled=yes; selector: name=cp4037.ulsfo.wmnet [production]
15:30 <fabfur> repooling cp4037 for a little longer than last time (T358109) [production]
15:28 <moritzm> installing squid security updates [production]
15:18 <dani@deploy2002> helmfile [codfw] DONE helmfile.d/services/miscweb: apply [production]
15:18 <dani@deploy2002> helmfile [codfw] START helmfile.d/services/miscweb: apply [production]
15:18 <dani@deploy2002> helmfile [eqiad] DONE helmfile.d/services/miscweb: apply [production]
15:17 <dani@deploy2002> helmfile [eqiad] START helmfile.d/services/miscweb: apply [production]
15:17 <dani@deploy2002> helmfile [staging] DONE helmfile.d/services/miscweb: apply [production]
15:17 <dani@deploy2002> helmfile [staging] START helmfile.d/services/miscweb: apply [production]
15:11 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 16 hosts with reason: Remove circular replication in x1 T358200 [production]
15:10 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 0:30:00 on 16 hosts with reason: Remove circular replication in x1 T358200 [production]
14:59 <Lucas_WMDE> START lucaswerkmeister-wmde@mwmaint1002:~$ time mwscript extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --wiki viwiki --current --all --touched-after=20230613000000 --start '["17099868"]' 2>&1 | tee ~/T315510-viwiki-4; date # in tmux; note the changed mwmaint host :) [production]
14:50 <jiji@deploy2002> Unlocked for deployment [ALL REPOSITORIES]: Datacenter Switchover - T357547 (duration: 61m 28s) [production]
14:45 <Dreamy_Jazz> Starting MediaModeration scanning script - https://wikitech.wikimedia.org/wiki/MediaModeration [production]
14:36 <jiji@cumin1002> END (PASS) - Cookbook sre.switchdc.mediawiki.09-run-puppet-on-db-masters (exit_code=0) [production]
14:29 <jiji@cumin1002> START - Cookbook sre.switchdc.mediawiki.09-run-puppet-on-db-masters [production]
14:27 <jiji@cumin1002> END (PASS) - Cookbook sre.switchdc.mediawiki.09-restore-ttl (exit_code=0) [production]
14:27 <jiji@cumin1002> START - Cookbook sre.switchdc.mediawiki.09-restore-ttl [production]
14:26 <jiji@cumin1002> END (PASS) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=0) [production]
14:24 <jiji@cumin1002> START - Cookbook sre.switchdc.mediawiki.08-start-maintenance [production]
14:23 <jiji@cumin1002> END (PASS) - Cookbook sre.switchdc.mediawiki.08-restart-mw-jobrunner (exit_code=0) [production]
14:23 <root@deploy2002> helmfile [codfw] [main] DONE helmfile.d/services/mw-jobrunner : sync [production]
14:21 <root@deploy2002> helmfile [codfw] [canary] DONE helmfile.d/services/mw-jobrunner : sync [production]
14:21 <root@deploy2002> helmfile [codfw] [main] START helmfile.d/services/mw-jobrunner : sync [production]
14:21 <root@deploy2002> helmfile [codfw] [canary] START helmfile.d/services/mw-jobrunner : sync [production]
14:21 <jiji@cumin1002> START - Cookbook sre.switchdc.mediawiki.08-restart-mw-jobrunner [production]
14:18 <marostegui> Test write T357547 [production]
14:18 <jiji@cumin1002> END (PASS) - Cookbook sre.switchdc.mediawiki.07-set-readwrite (exit_code=0) [production]
14:18 <jiji@cumin1002> MediaWiki read-only period ends at: 2024-03-20 14:18:32.727570 [production]
14:18 <jiji@cumin1002> START - Cookbook sre.switchdc.mediawiki.07-set-readwrite [production]
14:18 <jiji@cumin1002> END (PASS) - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (exit_code=0) [production]
14:18 <jiji@cumin1002> START - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite [production]
14:17 <jiji@cumin1002> END (PASS) - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (exit_code=0) [production]
14:16 <jiji@cumin1002> START - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki [production]
14:16 <jiji@cumin1002> END (PASS) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=0) [production]
14:16 <jiji@cumin1002> START - Cookbook sre.switchdc.mediawiki.03-set-db-readonly [production]
14:15 <jiji@cumin1002> END (PASS) - Cookbook sre.switchdc.mediawiki.02-set-readonly (exit_code=0) [production]
14:15 <jiji@cumin1002> MediaWiki read-only period starts at: 2024-03-20 14:15:24.401121 [production]
14:15 <jiji@cumin1002> START - Cookbook sre.switchdc.mediawiki.02-set-readonly [production]
14:13 <jiji@cumin1002> END (PASS) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=0) [production]
14:13 <jiji@cumin1002> START - Cookbook sre.switchdc.mediawiki.01-stop-maintenance [production]
14:03 <jiji@cumin1002> END (FAIL) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=99) [production]
14:03 <jiji@cumin1002> START - Cookbook sre.switchdc.mediawiki.01-stop-maintenance [production]
14:02 <jiji@cumin1002> END (PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0) [production]