501-550 of 1133 results (16ms)
2023-11-06 §
14:55 <marostegui@cumin1001> END (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 32 hosts with reason: Primary switchover s8 T349053 [production]
14:54 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 32 hosts with reason: Primary switchover s8 T349053 [production]
2023-10-31 §
06:33 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 34 hosts with reason: Primary switchover s4 T349820 [production]
06:33 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on 34 hosts with reason: Primary switchover s4 T349820 [production]
2023-10-17 §
10:46 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 32 hosts with reason: Primary switchover s8 T349077 [production]
10:46 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 32 hosts with reason: Primary switchover s8 T349077 [production]
05:17 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 32 hosts with reason: Primary switchover s8 T349053 [production]
05:17 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 32 hosts with reason: Primary switchover s8 T349053 [production]
2023-09-27 §
16:09 <kamila_> Pooled back eqiad for traffic after the DC switchover (T345263) [production]
14:29 <kamila@cumin1001> END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) pool all services in eqiad: Datacenter Switchover: eqiad repool - T345263 [production]
14:08 <kamila@cumin1001> START - Cookbook sre.discovery.datacenter pool all services in eqiad: Datacenter Switchover: eqiad repool - T345263 [production]
2023-09-20 §
14:09 <kamila@deploy2002> Unlocked for deployment [ALL REPOSITORIES]: Datacenter Switchover: MediaWiki - T346474 (duration: 12m 54s) [production]
13:56 <kamila@deploy2002> Locking from deployment [ALL REPOSITORIES]: Datacenter Switchover: MediaWiki - T346474 [production]
2023-09-19 §
16:04 <kamila_> DC Switchover: traffic - T346330 [production]
15:05 <kamila@deploy1002> Unlocked for deployment [ALL REPOSITORIES]: Datacenter Switchover: Services & Traffic - T346330 (duration: 34m 46s) [production]
14:30 <kamila@deploy1002> Locking from deployment [ALL REPOSITORIES]: Datacenter Switchover: Services & Traffic - T346330 [production]
14:28 <kamila@cumin1001> END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) depool all services in eqiad: Datacenter Switchover: Services - T346330 [production]
14:20 <kamila@deploy1002> Unlocked for deployment [ALL REPOSITORIES]: Datacenter Switchover: Services & Traffic - T346330 (duration: 19m 27s) [production]
14:01 <kamila@cumin1001> START - Cookbook sre.discovery.datacenter depool all services in eqiad: Datacenter Switchover: Services - T346330 [production]
14:00 <kamila@deploy1002> Locking from deployment [ALL REPOSITORIES]: Datacenter Switchover: Services & Traffic - T346330 [production]
2023-09-05 §
15:36 <kamila_> Datacenter switchover live test completed (T345588) [production]
15:35 <kamila@deploy1002> Unlocked for deployment [ALL REPOSITORIES]: Datacenter Switchover Live Test - T345588 (duration: 30m 45s) [production]
15:04 <kamila@deploy1002> Locking from deployment [ALL REPOSITORIES]: Datacenter Switchover Live Test - T345588 [production]
14:50 <kamila@cumin1001> END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) pool all active/active services in codfw: Datacenter Switchover Live test - T345588 [production]
14:26 <kamila@cumin1001> START - Cookbook sre.discovery.datacenter pool all active/active services in codfw: Datacenter Switchover Live test - T345588 [production]
14:25 <kamila@cumin1001> END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) depool all services in codfw: Datacenter Switchover Live test - T345588 [production]
14:01 <kamila@cumin1001> START - Cookbook sre.discovery.datacenter depool all services in codfw: Datacenter Switchover Live test - T345588 [production]
2023-08-31 §
04:57 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Primary switchover s6 T345223 [production]
04:56 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 26 hosts with reason: Primary switchover s6 T345223 [production]
2023-08-24 §
07:43 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 34 hosts with reason: Primary switchover s4 T344883 [production]
07:42 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 34 hosts with reason: Primary switchover s4 T344883 [production]
06:31 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 34 hosts with reason: Primary switchover s4 T344883 [production]
06:31 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 34 hosts with reason: Primary switchover s4 T344883 [production]
05:19 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 34 hosts with reason: Primary switchover s4 T344881 [production]
05:19 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 34 hosts with reason: Primary switchover s4 T344881 [production]
2023-08-22 §
06:27 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 35 hosts with reason: Primary switchover s1 T344666 [production]
06:26 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 35 hosts with reason: Primary switchover s1 T344666 [production]
05:13 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 35 hosts with reason: Primary switchover s1 T344621 [production]
05:13 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 35 hosts with reason: Primary switchover s1 T344621 [production]
2023-08-03 §
05:44 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Primary switchover s6 T343296 [production]
05:43 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 26 hosts with reason: Primary switchover s6 T343296 [production]
2023-07-29 §
16:57 <marostegui@cumin1001> dbctl commit (dc=all): 'Emergency switchover T343077', diff saved to https://phabricator.wikimedia.org/P49799 and previous config saved to /var/cache/conftool/dbconfig/20230729-165748-root.json [production]
16:36 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 25 hosts with reason: Primary switchover s5 T343077 [production]
16:36 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 25 hosts with reason: Primary switchover s5 T343077 [production]
2023-07-11 §
09:43 <hashar> Updating Zuul configuration which was stall to a version from March 29th after the switchover from contint2001 to contint2002 | T324659 T341556 [production]
2023-07-10 §
18:45 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 37 hosts with reason: Primary switchover s1 T341511 [production]
18:44 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 37 hosts with reason: Primary switchover s1 T341511 [production]
2023-07-04 §
17:53 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 32 hosts with reason: Primary switchover s8 T339223 [production]
17:52 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 32 hosts with reason: Primary switchover s8 T339223 [production]
2023-05-23 §
07:18 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Primary switchover es5 T337285 [production]