1-50 of 1149 results (11ms)
2025-03-26 §
14:30 <hnowlan@cumin1002> END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) pool all active/active services in codfw: Datacentre switchover repool - T385155 [production]
14:08 <hnowlan@cumin1002> START - Cookbook sre.discovery.datacenter pool all active/active services in codfw: Datacentre switchover repool - T385155 [production]
14:06 <hnowlan@cumin1002> END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool site codfw [reason: Datacentre switchover repool, T385155] [production]
14:06 <hnowlan@cumin1002> START - Cookbook sre.dns.admin DNS admin: pool site codfw [reason: Datacentre switchover repool, T385155] [production]
2025-03-25 §
10:44 <fceratto@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 28 hosts with reason: Primary switchover s7 T389383 [production]
09:57 <fceratto@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s5 T389381 [production]
2025-03-24 §
14:44 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 33 hosts with reason: Primary switchover s4 T389378 [production]
13:33 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 13 hosts with reason: Primary switchover x1 T389795 [production]
11:54 <fceratto@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s3 T389377 [production]
10:43 <fceratto@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 34 hosts with reason: Primary switchover s1 T389373 [production]
10:27 <fceratto@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 34 hosts with reason: Primary switchover s1 T389373 [production]
06:53 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s6 T389382 [production]
06:18 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Primary switchover s2 T389376 [production]
2025-03-20 §
23:30 <kamila@deploy1003> Finished scap sync-world: Test deployment to validate deployment server switchover - T385155 (duration: 19m 42s) [production]
23:10 <kamila@deploy1003> Started scap sync-world: Test deployment to validate deployment server switchover - T385155 [production]
06:34 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 32 hosts with reason: Primary switchover s8 T389367 [production]
2025-03-19 §
15:34 <hnowlan@deploy2002> Unlocked for deployment [ALL REPOSITORIES]: Switchover followup (duration: 02m 47s) [production]
15:31 <hnowlan@deploy2002> Locking from deployment [ALL REPOSITORIES]: Switchover followup [production]
15:23 <hnowlan@deploy2002> hnowlan: Backport for [[gerrit:1127072|debug: reorder debug backends for eqiad switchover (T385155)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
15:18 <hnowlan@deploy2002> Started scap sync-world: Backport for [[gerrit:1127072|debug: reorder debug backends for eqiad switchover (T385155)]] [production]
14:41 <hnowlan@deploy2002> Unlocked for deployment [ALL REPOSITORIES]: Datacenter Switchover - T385155 (duration: 52m 40s) [production]
14:40 <hnowlan@cumin2002> END (PASS) - Cookbook sre.switchdc.mediawiki.09-run-puppet-on-db-masters (exit_code=0) for datacenter switchover from codfw to eqiad [production]
14:29 <hnowlan@cumin2002> START - Cookbook sre.switchdc.mediawiki.09-run-puppet-on-db-masters for datacenter switchover from codfw to eqiad [production]
14:23 <hnowlan@cumin2002> END (PASS) - Cookbook sre.switchdc.mediawiki.09-restore-ttl (exit_code=0) for datacenter switchover from codfw to eqiad [production]
14:22 <hnowlan@cumin2002> START - Cookbook sre.switchdc.mediawiki.09-restore-ttl for datacenter switchover from codfw to eqiad [production]
14:21 <hnowlan@cumin2002> END (PASS) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=0) for datacenter switchover from codfw to eqiad [production]
14:19 <hnowlan@cumin2002> START - Cookbook sre.switchdc.mediawiki.08-start-maintenance for datacenter switchover from codfw to eqiad [production]
14:18 <hnowlan@cumin2002> END (PASS) - Cookbook sre.switchdc.mediawiki.08-restart-mw-jobrunner (exit_code=0) for datacenter switchover from codfw to eqiad [production]
14:18 <hnowlan@cumin2002> START - Cookbook sre.switchdc.mediawiki.08-restart-mw-jobrunner for datacenter switchover from codfw to eqiad [production]
14:17 <hnowlan@cumin2002> END (PASS) - Cookbook sre.switchdc.mediawiki.07-set-readwrite (exit_code=0) for datacenter switchover from codfw to eqiad [production]
14:17 <hnowlan@cumin2002> START - Cookbook sre.switchdc.mediawiki.07-set-readwrite for datacenter switchover from codfw to eqiad [production]
14:17 <hnowlan@cumin2002> END (PASS) - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (exit_code=0) for datacenter switchover from codfw to eqiad [production]
14:17 <hnowlan@cumin2002> START - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite for datacenter switchover from codfw to eqiad [production]
14:17 <hnowlan@cumin2002> END (PASS) - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (exit_code=0) for datacenter switchover from codfw to eqiad [production]
14:16 <hnowlan@cumin2002> START - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki for datacenter switchover from codfw to eqiad [production]
14:16 <hnowlan@cumin2002> END (PASS) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=0) for datacenter switchover from codfw to eqiad [production]
14:15 <hnowlan@cumin2002> START - Cookbook sre.switchdc.mediawiki.03-set-db-readonly for datacenter switchover from codfw to eqiad [production]
14:15 <hnowlan@cumin2002> END (PASS) - Cookbook sre.switchdc.mediawiki.02-set-readonly (exit_code=0) for datacenter switchover from codfw to eqiad [production]
14:15 <hnowlan@cumin2002> START - Cookbook sre.switchdc.mediawiki.02-set-readonly for datacenter switchover from codfw to eqiad [production]
14:14 <hnowlan@cumin2002> END (PASS) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=0) for datacenter switchover from codfw to eqiad [production]
14:14 <hnowlan@cumin2002> START - Cookbook sre.switchdc.mediawiki.01-stop-maintenance for datacenter switchover from codfw to eqiad [production]
14:13 <hnowlan@cumin2002> END (FAIL) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=99) for datacenter switchover from codfw to eqiad [production]
14:12 <hnowlan@cumin2002> START - Cookbook sre.switchdc.mediawiki.01-stop-maintenance for datacenter switchover from codfw to eqiad [production]
14:11 <hnowlan@cumin2002> END (PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0) for datacenter switchover from codfw to eqiad [production]
14:06 <hnowlan@cumin2002> START - Cookbook sre.switchdc.mediawiki.00-reduce-ttl for datacenter switchover from codfw to eqiad [production]
14:03 <hnowlan@cumin2002> END (PASS) - Cookbook sre.switchdc.mediawiki.00-downtime-db-readonly-checks (exit_code=0) for datacenter switchover from codfw to eqiad [production]
14:03 <hnowlan@cumin2002> START - Cookbook sre.switchdc.mediawiki.00-downtime-db-readonly-checks for datacenter switchover from codfw to eqiad [production]
14:03 <hnowlan@cumin2002> END (PASS) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0) for datacenter switchover from codfw to eqiad [production]
14:03 <hnowlan@cumin2002> START - Cookbook sre.switchdc.mediawiki.00-disable-puppet for datacenter switchover from codfw to eqiad [production]
13:48 <hnowlan@deploy2002> Locking from deployment [ALL REPOSITORIES]: Datacenter Switchover - T385155 [production]