1051-1100 of 1274 results (11ms)
|
2021-08-10
§
|
| 04:23 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Master switchover s2 T287454 |
[production] |
| 04:23 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 24 hosts with reason: Master switchover s2 T287454 |
[production] |
|
2021-07-22
§
|
| 18:10 |
<legoktm> |
testing dc switchover warmup script in eqiad |
[production] |
|
2021-06-30
§
|
| 09:35 |
<godog> |
start swiftrepl-mw on ms-fe2005 post-switchover (credentials were missing) - T162123 |
[production] |
|
2021-06-29
§
|
| 14:13 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 44 hosts with reason: DC switchover |
[production] |
| 14:13 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 44 hosts with reason: DC switchover |
[production] |
|
2021-06-28
§
|
| 22:31 |
<legoktm> |
starting DC switchover live test, which will "switch" us from codfw -> eqiad |
[production] |
| 18:40 |
<ebernhardson@deploy1002> |
Synchronized wmf-config/: T281515: Prepare Cirrus more_like for dc switchover (duration: 01m 02s) |
[production] |
|
2021-06-24
§
|
| 20:58 |
<legoktm> |
starting dry run and live test of DC switchover |
[production] |
|
2021-06-22
§
|
| 04:20 |
<marostegui> |
Start topology changes for s5 switchover T284529 |
[production] |
| 04:19 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Master switchover s5 T284529 |
[production] |
| 04:19 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 24 hosts with reason: Master switchover s5 T284529 |
[production] |
|
2021-06-21
§
|
| 04:48 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 21 hosts with reason: Master switchover s3 T284648 |
[production] |
| 04:48 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 21 hosts with reason: Master switchover s3 T284648 |
[production] |
|
2021-05-17
§
|
| 04:46 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Master switchover s6 T282124 |
[production] |
| 04:46 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 26 hosts with reason: Master switchover s6 T282124 |
[production] |
|
2021-04-28
§
|
| 04:07 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set db1163 with weight 0 before the switchover T278214', diff saved to https://phabricator.wikimedia.org/P15598 and previous config saved to /var/cache/conftool/dbconfig/20210428-040718-marostegui.json |
[production] |
|
2021-04-27
§
|
| 13:45 |
<akosiaris> |
switchover api-gateway, changeprop, cpjobqueue to use the new redis cluster servers (rdb2007-rdb2010) |
[production] |
|
2020-08-28
§
|
| 16:06 |
<rzl> |
starting one more live test of the data center switchover automation, no production impact is expected but there will be some SAL noise |
[production] |
|
2020-08-24
§
|
| 14:55 |
<rzl> |
switchover test complete, puppet re-enabled on cumin1001 |
[production] |
| 14:18 |
<rzl> |
disabling puppet on cumin1001 and starting a test of the DC switchover automation, expect some SAL noise but no production impact |
[production] |
|
2020-07-15
§
|
| 04:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set db1103 weight to 0 before the switchover T254871', diff saved to https://phabricator.wikimedia.org/P11908 and previous config saved to /var/cache/conftool/dbconfig/20200715-044432-marostegui.json |
[production] |
|
2020-07-08
§
|
| 07:48 |
<jynus> |
stop bacula-director on backup1001 in preparation for m1 switchover T256717 |
[production] |
|
2020-07-06
§
|
| 13:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Reduce es1024 weight in preparation for tomorrow's switchover T255755', diff saved to https://phabricator.wikimedia.org/P11750 and previous config saved to /var/cache/conftool/dbconfig/20200706-132634-marostegui.json |
[production] |
|
2020-06-25
§
|
| 07:08 |
<marostegui> |
Start pre switchover steps on m1 T254556 |
[production] |
|
2020-06-16
§
|
| 13:12 |
<XioNoX> |
add graceful-switchover to cr1-eqiad |
[production] |
|
2020-06-15
§
|
| 08:22 |
<marostegui> |
Switchover m3-master from dbproxy1008 to dbproxy1016 - T202367 |
[production] |
|
2020-06-08
§
|
| 11:28 |
<XioNoX> |
cr1-codfw add graceful-switchover - T243080 |
[production] |
|
2020-06-03
§
|
| 05:38 |
<XioNoX> |
deactivate graceful-switchover on cr3-esams - T254021 |
[production] |
|
2020-02-18
§
|
| 12:45 |
<XioNoX> |
remove graceful-switchover and nonstop-routing from cr2-esams - T243080 |
[production] |
|
2020-02-11
§
|
| 22:04 |
<XioNoX> |
switchover RE mastership back re0 on cr1-eqsin - T243080 |
[production] |
| 21:32 |
<XioNoX> |
switchover RE mastership on cr1-eqsin - T243080 |
[production] |
|
2020-02-06
§
|
| 10:21 |
<akosiaris> |
undo "switchover selectively eventgate-analytics.discovery.wmnet to codfw for mw1331 and mw1348". no effect observed |
[production] |
| 10:20 |
<akosiaris> |
undo "switchover selectively eventgate-analytics.discovery.wmnet to codfw for mw1331 and mw1348" |
[production] |
| 08:17 |
<akosiaris> |
switchover selectively eventgate-analytics.discovery.wmnet to codfw for mw1331 and mw1348 to |
[production] |
|
2020-01-23
§
|
| 18:35 |
<rlazarus> |
etcd main cluster switchover complete, eqiad is now read-write |
[production] |
| 16:00 |
<rlazarus> |
Starting etcd main cluster switchover from codfw to eqiad |
[production] |
|
2019-11-19
§
|
| 05:02 |
<marostegui> |
Start pre-switchover steps T235469 |
[production] |
|
2019-10-15
§
|
| 04:15 |
<marostegui> |
Start pre-switchover steps T234300 |
[production] |
|
2019-09-26
§
|
| 04:10 |
<marostegui> |
Start pre-switchover s4 steps T230784 |
[production] |
|
2019-09-24
§
|
| 04:13 |
<marostegui> |
Start pre switchover steps - T230783 |
[production] |
|
2019-09-17
§
|
| 04:11 |
<marostegui> |
Start s2 pre-switchover steps T230785 |
[production] |
|
2019-09-11
§
|
| 05:29 |
<marostegui> |
Switchover s1 codfw master db2048 -> db2112 T230106 |
[production] |
|
2019-09-10
§
|
| 15:37 |
<marostegui> |
Start pre-switchover for m1 steps T231403 |
[production] |
| 04:18 |
<marostegui> |
Start s8 (wikidata) pre switchover steps T230762 |
[production] |
|
2019-08-27
§
|
| 07:16 |
<marostegui> |
Switchover codfw s6 master from db2046 to db2129 T230106 |
[production] |
|
2019-08-20
§
|
| 08:05 |
<marostegui> |
Switchover s5 codfw master db2052 -> db2123 T230106 |
[production] |
| 05:18 |
<marostegui> |
Switchover m2 codfw master, db2044 -> db2067 T230705 |
[production] |
|
2019-08-12
§
|
| 07:34 |
<marostegui> |
Switchover s3 codfw master db2043 -> db2105 - T230106 |
[production] |
|
2019-08-08
§
|
| 07:39 |
<marostegui> |
Switchover x1 codfw master db2069 -> db2096 T220170 |
[production] |