101-150 of 1149 results (5ms)
2025-02-25 §
13:35 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Primary switchover s7 T387216 [production]
12:52 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Primary switchover es6 T387211 [production]
2025-02-06 §
06:59 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s5 T385148 [production]
2025-02-03 §
13:17 <cgoubert@deploy2002> Unlocked for deployment [MediaWiki]: Emergency s3 switchover T385457 (duration: 07m 36s) [production]
13:09 <cgoubert@deploy2002> Locking from deployment [MediaWiki]: Emergency s3 switchover T385457 [production]
13:06 <marostegui> Emergency s3 switchover T385457 [production]
13:01 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s3 [production]
2025-01-31 §
11:26 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s3 T385051 [production]
2025-01-30 §
09:32 <marostegui@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s5 T385147 [production]
2025-01-23 §
16:26 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Primary switchover s2 T384597 [production]
2025-01-15 §
12:57 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 34 hosts with reason: Primary switchover s1 T383689 [production]
12:57 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on 34 hosts with reason: Primary switchover s1 T383689 [production]
2025-01-13 §
09:48 <marostegui@cumin1002> dbctl commit (dc=all): 'Switchover es5 eqiad master dbmaint T382569', diff saved to https://phabricator.wikimedia.org/P71987 and previous config saved to /var/cache/conftool/dbconfig/20250113-094833-marostegui.json [production]
2025-01-08 §
06:19 <marostegui@cumin1002> dbctl commit (dc=all): 'Switchover es4 eqiad master dbmaint T382569', diff saved to https://phabricator.wikimedia.org/P71829 and previous config saved to /var/cache/conftool/dbconfig/20250108-061914-marostegui.json [production]
2025-01-07 §
08:38 <marostegui@cumin1002> dbctl commit (dc=all): 'Switchover es5 eqiad master dbmaint T382569', diff saved to https://phabricator.wikimedia.org/P71821 and previous config saved to /var/cache/conftool/dbconfig/20250107-083811-marostegui.json [production]
05:40 <marostegui> Switchover m5 eqiad proxy dbmaint eqiad T368874 [production]
2025-01-06 §
07:13 <marostegui> dbmaint Switchover m3 (phabricator) eqiad master dbproxy1020 -> dbproxy1028 T368874 [production]
2025-01-03 §
11:05 <marostegui> Switchover es4 codfw master to es2043 dbmaint T381848 [production]
11:04 <marostegui@cumin1002> dbctl commit (dc=all): 'Switchover es4 codfw master', diff saved to https://phabricator.wikimedia.org/P71779 and previous config saved to /var/cache/conftool/dbconfig/20250103-110440-marostegui.json [production]
2025-01-02 §
11:00 <root@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 32 hosts with reason: Primary switchover s8 T381993 [production]
11:00 <root@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on 32 hosts with reason: Primary switchover s8 T381993 [production]
2024-12-24 §
10:21 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 25 hosts with reason: Primary switchover s5 T382743 [production]
10:21 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on 25 hosts with reason: Primary switchover s5 T382743 [production]
2024-12-05 §
05:20 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2134,2160].codfw.wmnet,db[1159,1213,1217].eqiad.wmnet with reason: m3 master switchover T381365 [production]
05:20 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on db[2134,2160].codfw.wmnet,db[1159,1213,1217].eqiad.wmnet with reason: m3 master switchover T381365 [production]
2024-12-03 §
07:57 <marostegui> Switchover es4 codfw master to es2022 dbmaint (this happened an hour ago) T381259 [production]
2024-10-15 §
15:43 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s3 T377164 [production]
15:42 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on 24 hosts with reason: Primary switchover s3 T377164 [production]
2024-10-10 §
07:07 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Primary switchover s7 T376868 [production]
07:07 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on 27 hosts with reason: Primary switchover s7 T376868 [production]
06:42 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s5 T376867 [production]
06:41 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on 24 hosts with reason: Primary switchover s5 T376867 [production]
2024-10-02 §
15:27 <swfrench@cumin1002> END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) pool all active/active services in eqiad: Datacenter Switchover - T370962 [production]
15:07 <swfrench@cumin1002> START - Cookbook sre.discovery.datacenter pool all active/active services in eqiad: Datacenter Switchover - T370962 [production]
2024-09-26 §
16:27 <swfrench-wmf> done with switchover day 3 deployment server switch - T370962 [production]
16:04 <swfrench@deploy2002> Finished scap sync-world: No-op deployment to verify switchover - T370962 (duration: 29m 53s) [production]
15:34 <swfrench@deploy2002> Started scap sync-world: No-op deployment to verify switchover - T370962 [production]
15:01 <swfrench-wmf> starting switchover day 3 deployment server switch - T370962 [production]
2024-09-25 §
15:57 <swfrench-wmf> switchover day 2 manual steps are complete - T370962 [production]
15:24 <swfrench@deploy1003> Unlocked for deployment [ALL REPOSITORIES]: Datacenter Switchover - T370962 (duration: 58m 14s) [production]
15:21 <swfrench@cumin1002> END (PASS) - Cookbook sre.switchdc.mediawiki.09-run-puppet-on-db-masters (exit_code=0) for datacenter switchover from eqiad to codfw [production]
15:12 <Dreamy_Jazz> Started MediaModeration script after datacenter switchover - https://wikitech.wikimedia.org/wiki/MediaModeration [production]
15:12 <swfrench@cumin1002> START - Cookbook sre.switchdc.mediawiki.09-run-puppet-on-db-masters for datacenter switchover from eqiad to codfw [production]
15:09 <swfrench@cumin1002> END (PASS) - Cookbook sre.switchdc.mediawiki.09-restore-ttl (exit_code=0) for datacenter switchover from eqiad to codfw [production]
15:08 <swfrench@cumin1002> START - Cookbook sre.switchdc.mediawiki.09-restore-ttl for datacenter switchover from eqiad to codfw [production]
15:07 <swfrench@cumin1002> END (PASS) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=0) for datacenter switchover from eqiad to codfw [production]
15:05 <swfrench@cumin1002> START - Cookbook sre.switchdc.mediawiki.08-start-maintenance for datacenter switchover from eqiad to codfw [production]
15:04 <swfrench@cumin1002> END (PASS) - Cookbook sre.switchdc.mediawiki.08-restart-mw-jobrunner (exit_code=0) for datacenter switchover from eqiad to codfw [production]
15:03 <swfrench@cumin1002> START - Cookbook sre.switchdc.mediawiki.08-restart-mw-jobrunner for datacenter switchover from eqiad to codfw [production]
15:01 <swfrench@cumin1002> END (PASS) - Cookbook sre.switchdc.mediawiki.07-set-readwrite (exit_code=0) for datacenter switchover from eqiad to codfw [production]