501-550 of 1149 results (13ms)
2024-01-24 §
12:39 <marostegui@cumin1002> dbctl commit (dc=all): 'db1231 (re)pooling @ 5%: After switchover', diff saved to https://phabricator.wikimedia.org/P55515 and previous config saved to /var/cache/conftool/dbconfig/20240124-123859-root.json [production]
12:23 <marostegui@cumin1002> dbctl commit (dc=all): 'db1231 (re)pooling @ 1%: After switchover', diff saved to https://phabricator.wikimedia.org/P55514 and previous config saved to /var/cache/conftool/dbconfig/20240124-122354-root.json [production]
12:02 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 28 hosts with reason: Primary switchover s6 T355760 [production]
12:01 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on 28 hosts with reason: Primary switchover s6 T355760 [production]
2024-01-23 §
11:07 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 28 hosts with reason: Primary switchover s6 T355660 [production]
11:06 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on 28 hosts with reason: Primary switchover s6 T355660 [production]
2023-12-01 §
05:32 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2134,2160].codfw.wmnet,db[1119,1159,1217].eqiad.wmnet with reason: m3 master switchover T352149 [production]
05:31 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on db[2134,2160].codfw.wmnet,db[1119,1159,1217].eqiad.wmnet with reason: m3 master switchover T352149 [production]
2023-11-30 §
05:41 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2134,2160].codfw.wmnet,db[1119,1159,1217].eqiad.wmnet with reason: m3 master switchover T352149 [production]
05:41 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on db[2134,2160].codfw.wmnet,db[1119,1159,1217].eqiad.wmnet with reason: m3 master switchover T352149 [production]
2023-11-14 §
10:08 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 34 hosts with reason: Primary switchover s4 T351184 [production]
10:07 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 34 hosts with reason: Primary switchover s4 T351184 [production]
2023-11-09 §
06:26 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 35 hosts with reason: Primary switchover s1 T350142 [production]
06:26 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on 35 hosts with reason: Primary switchover s1 T350142 [production]
2023-11-07 §
06:16 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 35 hosts with reason: Primary switchover s1 T350142 [production]
06:16 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on 35 hosts with reason: Primary switchover s1 T350142 [production]
2023-11-06 §
14:55 <marostegui@cumin1001> END (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 32 hosts with reason: Primary switchover s8 T349053 [production]
14:54 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 32 hosts with reason: Primary switchover s8 T349053 [production]
2023-10-31 §
06:33 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 34 hosts with reason: Primary switchover s4 T349820 [production]
06:33 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on 34 hosts with reason: Primary switchover s4 T349820 [production]
2023-10-17 §
10:46 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 32 hosts with reason: Primary switchover s8 T349077 [production]
10:46 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 32 hosts with reason: Primary switchover s8 T349077 [production]
05:17 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 32 hosts with reason: Primary switchover s8 T349053 [production]
05:17 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 32 hosts with reason: Primary switchover s8 T349053 [production]
2023-09-27 §
16:09 <kamila_> Pooled back eqiad for traffic after the DC switchover (T345263) [production]
14:29 <kamila@cumin1001> END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) pool all services in eqiad: Datacenter Switchover: eqiad repool - T345263 [production]
14:08 <kamila@cumin1001> START - Cookbook sre.discovery.datacenter pool all services in eqiad: Datacenter Switchover: eqiad repool - T345263 [production]
2023-09-20 §
14:09 <kamila@deploy2002> Unlocked for deployment [ALL REPOSITORIES]: Datacenter Switchover: MediaWiki - T346474 (duration: 12m 54s) [production]
13:56 <kamila@deploy2002> Locking from deployment [ALL REPOSITORIES]: Datacenter Switchover: MediaWiki - T346474 [production]
2023-09-19 §
16:04 <kamila_> DC Switchover: traffic - T346330 [production]
15:05 <kamila@deploy1002> Unlocked for deployment [ALL REPOSITORIES]: Datacenter Switchover: Services & Traffic - T346330 (duration: 34m 46s) [production]
14:30 <kamila@deploy1002> Locking from deployment [ALL REPOSITORIES]: Datacenter Switchover: Services & Traffic - T346330 [production]
14:28 <kamila@cumin1001> END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) depool all services in eqiad: Datacenter Switchover: Services - T346330 [production]
14:20 <kamila@deploy1002> Unlocked for deployment [ALL REPOSITORIES]: Datacenter Switchover: Services & Traffic - T346330 (duration: 19m 27s) [production]
14:01 <kamila@cumin1001> START - Cookbook sre.discovery.datacenter depool all services in eqiad: Datacenter Switchover: Services - T346330 [production]
14:00 <kamila@deploy1002> Locking from deployment [ALL REPOSITORIES]: Datacenter Switchover: Services & Traffic - T346330 [production]
2023-09-05 §
15:36 <kamila_> Datacenter switchover live test completed (T345588) [production]
15:35 <kamila@deploy1002> Unlocked for deployment [ALL REPOSITORIES]: Datacenter Switchover Live Test - T345588 (duration: 30m 45s) [production]
15:04 <kamila@deploy1002> Locking from deployment [ALL REPOSITORIES]: Datacenter Switchover Live Test - T345588 [production]
14:50 <kamila@cumin1001> END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) pool all active/active services in codfw: Datacenter Switchover Live test - T345588 [production]
14:26 <kamila@cumin1001> START - Cookbook sre.discovery.datacenter pool all active/active services in codfw: Datacenter Switchover Live test - T345588 [production]
14:25 <kamila@cumin1001> END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) depool all services in codfw: Datacenter Switchover Live test - T345588 [production]
14:01 <kamila@cumin1001> START - Cookbook sre.discovery.datacenter depool all services in codfw: Datacenter Switchover Live test - T345588 [production]
2023-08-31 §
04:57 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Primary switchover s6 T345223 [production]
04:56 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 26 hosts with reason: Primary switchover s6 T345223 [production]
2023-08-24 §
07:43 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 34 hosts with reason: Primary switchover s4 T344883 [production]
07:42 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 34 hosts with reason: Primary switchover s4 T344883 [production]
06:31 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 34 hosts with reason: Primary switchover s4 T344883 [production]
06:31 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 34 hosts with reason: Primary switchover s4 T344883 [production]
05:19 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 34 hosts with reason: Primary switchover s4 T344881 [production]