|
2024-09-24
§
|
| 15:50 |
<swfrench-wmf> |
switchover day 1 actions are complete - T370962 |
[production] |
| 15:49 |
<swfrench@cumin1002> |
END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) depool all services in eqiad: Datacenter Switchover - T370962 |
[production] |
| 15:23 |
<swfrench@cumin1002> |
START - Cookbook sre.discovery.datacenter depool all services in eqiad: Datacenter Switchover - T370962 |
[production] |
| 15:02 |
<swfrench@cumin1002> |
END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool site eqiad [reason: Datacenter Switchover, T370962] |
[production] |
| 15:01 |
<swfrench@cumin1002> |
START - Cookbook sre.dns.admin DNS admin: depool site eqiad [reason: Datacenter Switchover, T370962] |
[production] |
| 15:01 |
<swfrench-wmf> |
starting switchover day 1 - T370962 |
[production] |
| 14:21 |
<swfrench-wmf> |
scaled up mw-web ahead of traffic+services switchover - T370962 |
[production] |
| 14:19 |
<swfrench-wmf> |
scaled up mw-api-ext ahead of traffic+services switchover - T370962 |
[production] |
|
2024-09-19
§
|
| 18:02 |
<swfrench-wmf> |
scaling down mw-api-ext in eqiad after pre-switchover testing - T371273 |
[production] |
| 18:02 |
<swfrench-wmf> |
scaling down mw-web in eqiad after pre-switchover testing - T371273 |
[production] |
| 17:53 |
<swfrench@cumin2002> |
conftool action : set/pooled=true; selector: dnsdisc=mw-web-ro,name=codfw [reason: Reverting pre-switchover capacity validation - T371273] |
[production] |
| 17:49 |
<swfrench@cumin2002> |
conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=codfw [reason: Reverting pre-switchover capacity validation - T371273] |
[production] |
| 17:46 |
<swfrench@cumin2002> |
conftool action : set/pooled=true; selector: dnsdisc=mw-api-int-ro,name=codfw [reason: Reverting pre-switchover capacity validation - T371273] |
[production] |
| 17:17 |
<swfrench@cumin2002> |
conftool action : set/pooled=false; selector: dnsdisc=mw-web-ro,name=codfw [reason: Pre-switchover capacity validation - T371273] |
[production] |
| 17:08 |
<swfrench@cumin2002> |
conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=codfw [reason: Pre-switchover capacity validation - T371273] |
[production] |
| 17:02 |
<swfrench@cumin2002> |
conftool action : set/pooled=false; selector: dnsdisc=mw-api-int-ro,name=codfw [reason: Pre-switchover capacity validation - T371273] |
[production] |
| 16:50 |
<swfrench-wmf> |
scaling up mw-web in eqiad for pre-switchover testing - T371273 |
[production] |
| 16:48 |
<swfrench-wmf> |
scaling up mw-api-ext in eqiad for pre-switchover testing - T371273 |
[production] |
|
2024-09-18
§
|
| 16:26 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Primary switchover s7 T375050 |
[production] |
| 16:26 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 27 hosts with reason: Primary switchover s7 T375050 |
[production] |
| 05:49 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Primary switchover s7 T374807 |
[production] |
| 05:48 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 27 hosts with reason: Primary switchover s7 T374807 |
[production] |
| 05:37 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 32 hosts with reason: Primary switchover s4 T374804 |
[production] |
| 05:36 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 32 hosts with reason: Primary switchover s4 T374804 |
[production] |
| 05:24 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 25 hosts with reason: Primary switchover s6 T375047 |
[production] |
| 05:23 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 25 hosts with reason: Primary switchover s6 T375047 |
[production] |
|
2024-09-16
§
|
| 15:13 |
<swfrench@cumin1002> |
END (PASS) - Cookbook sre.switchdc.mediawiki.09-run-puppet-on-db-masters (exit_code=0) for datacenter switchover from codfw to eqiad |
[production] |
| 15:03 |
<swfrench@cumin1002> |
START - Cookbook sre.switchdc.mediawiki.09-run-puppet-on-db-masters for datacenter switchover from codfw to eqiad |
[production] |
| 15:02 |
<swfrench@cumin1002> |
END (PASS) - Cookbook sre.switchdc.mediawiki.09-restore-ttl (exit_code=0) for datacenter switchover from codfw to eqiad |
[production] |
| 15:01 |
<swfrench@cumin1002> |
START - Cookbook sre.switchdc.mediawiki.09-restore-ttl for datacenter switchover from codfw to eqiad |
[production] |
| 15:01 |
<swfrench@cumin1002> |
END (PASS) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=0) for datacenter switchover from codfw to eqiad |
[production] |
| 14:59 |
<swfrench@cumin1002> |
START - Cookbook sre.switchdc.mediawiki.08-start-maintenance for datacenter switchover from codfw to eqiad |
[production] |
| 14:58 |
<swfrench@cumin1002> |
END (PASS) - Cookbook sre.switchdc.mediawiki.08-restart-mw-jobrunner (exit_code=0) for datacenter switchover from codfw to eqiad |
[production] |
| 14:57 |
<swfrench@cumin1002> |
START - Cookbook sre.switchdc.mediawiki.08-restart-mw-jobrunner for datacenter switchover from codfw to eqiad |
[production] |
| 14:57 |
<swfrench@cumin1002> |
END (PASS) - Cookbook sre.switchdc.mediawiki.07-set-readwrite (exit_code=0) for datacenter switchover from codfw to eqiad |
[production] |
| 14:57 |
<swfrench@cumin1002> |
START - Cookbook sre.switchdc.mediawiki.07-set-readwrite for datacenter switchover from codfw to eqiad |
[production] |
| 14:57 |
<swfrench@cumin1002> |
END (PASS) - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (exit_code=0) for datacenter switchover from codfw to eqiad |
[production] |
| 14:57 |
<swfrench@cumin1002> |
START - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite for datacenter switchover from codfw to eqiad |
[production] |
| 14:56 |
<swfrench@cumin1002> |
END (PASS) - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (exit_code=0) for datacenter switchover from codfw to eqiad |
[production] |
| 14:56 |
<swfrench@cumin1002> |
START - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki for datacenter switchover from codfw to eqiad |
[production] |
| 14:55 |
<swfrench@cumin1002> |
END (PASS) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=0) for datacenter switchover from codfw to eqiad |
[production] |
| 14:54 |
<swfrench@cumin1002> |
START - Cookbook sre.switchdc.mediawiki.03-set-db-readonly for datacenter switchover from codfw to eqiad |
[production] |
| 14:54 |
<swfrench@cumin1002> |
END (PASS) - Cookbook sre.switchdc.mediawiki.02-set-readonly (exit_code=0) for datacenter switchover from codfw to eqiad |
[production] |
| 14:54 |
<swfrench@cumin1002> |
START - Cookbook sre.switchdc.mediawiki.02-set-readonly for datacenter switchover from codfw to eqiad |
[production] |