101-150 of 1274 results (14ms)
2025-07-03 §
09:21 <fceratto@cumin1002> DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 22 hosts with reason: Primary switchover s5 T398593 [production]
2025-06-26 §
10:23 <fceratto@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 32 hosts with reason: Primary switchover s4 T397163 [production]
2025-06-25 §
13:08 <fceratto@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Primary switchover s8 T397164 [production]
2025-06-23 §
06:18 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 7 hosts with reason: Primary switchover es7 T397599 [production]
05:41 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 T397419 [production]
2025-06-19 §
08:58 <fceratto@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Primary switchover s8 T397164 [production]
07:25 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2233].codfw.wmnet,db[1164,1217,1250].eqiad.wmnet with reason: Primary switchover m2 T397182 [production]
2025-06-18 §
04:57 <root@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 23 hosts with reason: Primary switchover s6 T397198 [production]
2025-06-17 §
09:53 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2232].codfw.wmnet,db[1207,1217,1250].eqiad.wmnet with reason: Primary switchover m1 T396706 [production]
2025-06-16 §
07:05 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 25 hosts with reason: Primary switchover s2 T396976 [production]
2025-06-11 §
06:03 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 7 hosts with reason: Primary switchover es7 T396550 [production]
05:11 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 23 hosts with reason: Primary switchover s6 T396509 [production]
2025-06-04 §
05:42 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 7 hosts with reason: Primary switchover es7 T395982 [production]
2025-06-03 §
11:46 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 7 hosts with reason: Primary switchover es7 T395785 [production]
11:29 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 7 hosts with reason: Primary switchover es7 T395785 [production]
11:21 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 7 hosts with reason: Primary switchover es7 T395785 [production]
05:15 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 7 hosts with reason: Primary switchover es6 T395867 [production]
05:09 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 7 hosts with reason: Primary switchover es6 T395867 [production]
04:54 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on es2037.codfw.wmnet with reason: Primary switchover es6 T395420 [production]
04:49 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 7 hosts with reason: Primary switchover es6 T395420 [production]
2025-05-28 §
14:07 <ladsgroup@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Primary switchover s1 T389373 [production]
14:07 <ladsgroup@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 33 hosts with reason: Primary switchover s1 T389373 [production]
2025-05-20 §
13:15 <topranks> re-enable graceful switchover on cr1-codfw (T364092) [production]
12:05 <topranks> disable routing-engine sync / graceful-switchover on cr1-codfw ahead of JunOS upgrade on RE1 T364092 [production]
2025-04-30 §
17:13 <thcipriani> gerrit incident following switchover https://phabricator.wikimedia.org/T393034 [production]
2025-03-26 §
14:30 <hnowlan@cumin1002> END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) pool all active/active services in codfw: Datacentre switchover repool - T385155 [production]
14:08 <hnowlan@cumin1002> START - Cookbook sre.discovery.datacenter pool all active/active services in codfw: Datacentre switchover repool - T385155 [production]
14:06 <hnowlan@cumin1002> END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool site codfw [reason: Datacentre switchover repool, T385155] [production]
14:06 <hnowlan@cumin1002> START - Cookbook sre.dns.admin DNS admin: pool site codfw [reason: Datacentre switchover repool, T385155] [production]
2025-03-25 §
10:44 <fceratto@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 28 hosts with reason: Primary switchover s7 T389383 [production]
09:57 <fceratto@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s5 T389381 [production]
2025-03-24 §
14:44 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 33 hosts with reason: Primary switchover s4 T389378 [production]
13:33 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 13 hosts with reason: Primary switchover x1 T389795 [production]
11:54 <fceratto@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s3 T389377 [production]
10:43 <fceratto@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 34 hosts with reason: Primary switchover s1 T389373 [production]
10:27 <fceratto@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 34 hosts with reason: Primary switchover s1 T389373 [production]
06:53 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s6 T389382 [production]
06:18 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Primary switchover s2 T389376 [production]
2025-03-20 §
23:30 <kamila@deploy1003> Finished scap sync-world: Test deployment to validate deployment server switchover - T385155 (duration: 19m 42s) [production]
23:10 <kamila@deploy1003> Started scap sync-world: Test deployment to validate deployment server switchover - T385155 [production]
06:34 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 32 hosts with reason: Primary switchover s8 T389367 [production]
2025-03-19 §
15:34 <hnowlan@deploy2002> Unlocked for deployment [ALL REPOSITORIES]: Switchover followup (duration: 02m 47s) [production]
15:31 <hnowlan@deploy2002> Locking from deployment [ALL REPOSITORIES]: Switchover followup [production]
15:23 <hnowlan@deploy2002> hnowlan: Backport for [[gerrit:1127072|debug: reorder debug backends for eqiad switchover (T385155)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
15:18 <hnowlan@deploy2002> Started scap sync-world: Backport for [[gerrit:1127072|debug: reorder debug backends for eqiad switchover (T385155)]] [production]
14:41 <hnowlan@deploy2002> Unlocked for deployment [ALL REPOSITORIES]: Datacenter Switchover - T385155 (duration: 52m 40s) [production]
14:40 <hnowlan@cumin2002> END (PASS) - Cookbook sre.switchdc.mediawiki.09-run-puppet-on-db-masters (exit_code=0) for datacenter switchover from codfw to eqiad [production]
14:29 <hnowlan@cumin2002> START - Cookbook sre.switchdc.mediawiki.09-run-puppet-on-db-masters for datacenter switchover from codfw to eqiad [production]
14:23 <hnowlan@cumin2002> END (PASS) - Cookbook sre.switchdc.mediawiki.09-restore-ttl (exit_code=0) for datacenter switchover from codfw to eqiad [production]
14:22 <hnowlan@cumin2002> START - Cookbook sre.switchdc.mediawiki.09-restore-ttl for datacenter switchover from codfw to eqiad [production]