101-150 of 1133 results (10ms)
2025-01-06 §
07:13 <marostegui> dbmaint Switchover m3 (phabricator) eqiad master dbproxy1020 -> dbproxy1028 T368874 [production]
2025-01-03 §
11:05 <marostegui> Switchover es4 codfw master to es2043 dbmaint T381848 [production]
11:04 <marostegui@cumin1002> dbctl commit (dc=all): 'Switchover es4 codfw master', diff saved to https://phabricator.wikimedia.org/P71779 and previous config saved to /var/cache/conftool/dbconfig/20250103-110440-marostegui.json [production]
2025-01-02 §
11:00 <root@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 32 hosts with reason: Primary switchover s8 T381993 [production]
11:00 <root@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on 32 hosts with reason: Primary switchover s8 T381993 [production]
2024-12-24 §
10:21 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 25 hosts with reason: Primary switchover s5 T382743 [production]
10:21 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on 25 hosts with reason: Primary switchover s5 T382743 [production]
2024-12-05 §
05:20 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2134,2160].codfw.wmnet,db[1159,1213,1217].eqiad.wmnet with reason: m3 master switchover T381365 [production]
05:20 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on db[2134,2160].codfw.wmnet,db[1159,1213,1217].eqiad.wmnet with reason: m3 master switchover T381365 [production]
2024-12-03 §
07:57 <marostegui> Switchover es4 codfw master to es2022 dbmaint (this happened an hour ago) T381259 [production]
2024-10-15 §
15:43 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s3 T377164 [production]
15:42 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on 24 hosts with reason: Primary switchover s3 T377164 [production]
2024-10-10 §
07:07 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Primary switchover s7 T376868 [production]
07:07 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on 27 hosts with reason: Primary switchover s7 T376868 [production]
06:42 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s5 T376867 [production]
06:41 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on 24 hosts with reason: Primary switchover s5 T376867 [production]
2024-10-02 §
15:27 <swfrench@cumin1002> END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) pool all active/active services in eqiad: Datacenter Switchover - T370962 [production]
15:07 <swfrench@cumin1002> START - Cookbook sre.discovery.datacenter pool all active/active services in eqiad: Datacenter Switchover - T370962 [production]
2024-09-26 §
16:27 <swfrench-wmf> done with switchover day 3 deployment server switch - T370962 [production]
16:04 <swfrench@deploy2002> Finished scap sync-world: No-op deployment to verify switchover - T370962 (duration: 29m 53s) [production]
15:34 <swfrench@deploy2002> Started scap sync-world: No-op deployment to verify switchover - T370962 [production]
15:01 <swfrench-wmf> starting switchover day 3 deployment server switch - T370962 [production]
2024-09-25 §
15:57 <swfrench-wmf> switchover day 2 manual steps are complete - T370962 [production]
15:24 <swfrench@deploy1003> Unlocked for deployment [ALL REPOSITORIES]: Datacenter Switchover - T370962 (duration: 58m 14s) [production]
15:21 <swfrench@cumin1002> END (PASS) - Cookbook sre.switchdc.mediawiki.09-run-puppet-on-db-masters (exit_code=0) for datacenter switchover from eqiad to codfw [production]
15:12 <Dreamy_Jazz> Started MediaModeration script after datacenter switchover - https://wikitech.wikimedia.org/wiki/MediaModeration [production]
15:12 <swfrench@cumin1002> START - Cookbook sre.switchdc.mediawiki.09-run-puppet-on-db-masters for datacenter switchover from eqiad to codfw [production]
15:09 <swfrench@cumin1002> END (PASS) - Cookbook sre.switchdc.mediawiki.09-restore-ttl (exit_code=0) for datacenter switchover from eqiad to codfw [production]
15:08 <swfrench@cumin1002> START - Cookbook sre.switchdc.mediawiki.09-restore-ttl for datacenter switchover from eqiad to codfw [production]
15:07 <swfrench@cumin1002> END (PASS) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=0) for datacenter switchover from eqiad to codfw [production]
15:05 <swfrench@cumin1002> START - Cookbook sre.switchdc.mediawiki.08-start-maintenance for datacenter switchover from eqiad to codfw [production]
15:04 <swfrench@cumin1002> END (PASS) - Cookbook sre.switchdc.mediawiki.08-restart-mw-jobrunner (exit_code=0) for datacenter switchover from eqiad to codfw [production]
15:03 <swfrench@cumin1002> START - Cookbook sre.switchdc.mediawiki.08-restart-mw-jobrunner for datacenter switchover from eqiad to codfw [production]
15:01 <swfrench@cumin1002> END (PASS) - Cookbook sre.switchdc.mediawiki.07-set-readwrite (exit_code=0) for datacenter switchover from eqiad to codfw [production]
15:01 <swfrench@cumin1002> START - Cookbook sre.switchdc.mediawiki.07-set-readwrite for datacenter switchover from eqiad to codfw [production]
15:01 <swfrench@cumin1002> END (PASS) - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (exit_code=0) for datacenter switchover from eqiad to codfw [production]
15:00 <swfrench@cumin1002> START - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite for datacenter switchover from eqiad to codfw [production]
15:00 <swfrench@cumin1002> END (PASS) - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (exit_code=0) for datacenter switchover from eqiad to codfw [production]
15:00 <swfrench@cumin1002> START - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki for datacenter switchover from eqiad to codfw [production]
14:59 <swfrench@cumin1002> END (PASS) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=0) for datacenter switchover from eqiad to codfw [production]
14:59 <swfrench@cumin1002> START - Cookbook sre.switchdc.mediawiki.03-set-db-readonly for datacenter switchover from eqiad to codfw [production]
14:59 <swfrench@cumin1002> END (PASS) - Cookbook sre.switchdc.mediawiki.02-set-readonly (exit_code=0) for datacenter switchover from eqiad to codfw [production]
14:58 <swfrench@cumin1002> START - Cookbook sre.switchdc.mediawiki.02-set-readonly for datacenter switchover from eqiad to codfw [production]
14:52 <swfrench@cumin1002> END (PASS) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=0) for datacenter switchover from eqiad to codfw [production]
14:52 <swfrench@cumin1002> START - Cookbook sre.switchdc.mediawiki.01-stop-maintenance for datacenter switchover from eqiad to codfw [production]
14:46 <swfrench@cumin1002> END (FAIL) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=99) for datacenter switchover from eqiad to codfw [production]
14:45 <swfrench@cumin1002> START - Cookbook sre.switchdc.mediawiki.01-stop-maintenance for datacenter switchover from eqiad to codfw [production]
14:41 <swfrench@cumin1002> END (PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0) for datacenter switchover from eqiad to codfw [production]
14:35 <swfrench@cumin1002> START - Cookbook sre.switchdc.mediawiki.00-reduce-ttl for datacenter switchover from eqiad to codfw [production]
14:34 <swfrench@cumin1002> END (PASS) - Cookbook sre.switchdc.mediawiki.00-downtime-db-readonly-checks (exit_code=0) for datacenter switchover from eqiad to codfw [production]