901-950 of 1133 results (7ms)
2021-08-25 §
05:27 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:04:00 on 27 hosts with reason: Primary switchover s7 T289129 [production]
05:27 <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 1:04:00 on 27 hosts with reason: Primary switchover s7 T289129 [production]
2021-08-11 §
04:15 <root@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Master switchover s2 T287454 [production]
04:15 <root@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 24 hosts with reason: Master switchover s2 T287454 [production]
2021-08-10 §
12:23 <kormat> non-destructive (🤞) testing of db-switchover against s2/eqiad T288500 [production]
05:56 <marostegui@cumin1001> dbctl commit (dc=all): 'db2104 (re)pooling @ 100%: repool after failed switchover', diff saved to https://phabricator.wikimedia.org/P16987 and previous config saved to /var/cache/conftool/dbconfig/20210810-055642-root.json [production]
05:41 <marostegui@cumin1001> dbctl commit (dc=all): 'db2104 (re)pooling @ 75%: repool after failed switchover', diff saved to https://phabricator.wikimedia.org/P16986 and previous config saved to /var/cache/conftool/dbconfig/20210810-054139-root.json [production]
05:26 <marostegui@cumin1001> dbctl commit (dc=all): 'db2104 (re)pooling @ 50%: repool after failed switchover', diff saved to https://phabricator.wikimedia.org/P16985 and previous config saved to /var/cache/conftool/dbconfig/20210810-052635-root.json [production]
05:11 <marostegui@cumin1001> dbctl commit (dc=all): 'db2104 (re)pooling @ 25%: repool after failed switchover', diff saved to https://phabricator.wikimedia.org/P16984 and previous config saved to /var/cache/conftool/dbconfig/20210810-051131-root.json [production]
04:23 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Master switchover s2 T287454 [production]
04:23 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 24 hosts with reason: Master switchover s2 T287454 [production]
2021-07-22 §
18:10 <legoktm> testing dc switchover warmup script in eqiad [production]
2021-06-30 §
09:35 <godog> start swiftrepl-mw on ms-fe2005 post-switchover (credentials were missing) - T162123 [production]
2021-06-29 §
14:13 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 44 hosts with reason: DC switchover [production]
14:13 <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 44 hosts with reason: DC switchover [production]
2021-06-28 §
22:31 <legoktm> starting DC switchover live test, which will "switch" us from codfw -> eqiad [production]
18:40 <ebernhardson@deploy1002> Synchronized wmf-config/: T281515: Prepare Cirrus more_like for dc switchover (duration: 01m 02s) [production]
2021-06-24 §
20:58 <legoktm> starting dry run and live test of DC switchover [production]
2021-06-22 §
04:20 <marostegui> Start topology changes for s5 switchover T284529 [production]
04:19 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Master switchover s5 T284529 [production]
04:19 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 24 hosts with reason: Master switchover s5 T284529 [production]
2021-06-21 §
04:48 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 21 hosts with reason: Master switchover s3 T284648 [production]
04:48 <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 21 hosts with reason: Master switchover s3 T284648 [production]
2021-05-17 §
04:46 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Master switchover s6 T282124 [production]
04:46 <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 26 hosts with reason: Master switchover s6 T282124 [production]
2021-04-28 §
04:07 <marostegui@cumin1001> dbctl commit (dc=all): 'Set db1163 with weight 0 before the switchover T278214', diff saved to https://phabricator.wikimedia.org/P15598 and previous config saved to /var/cache/conftool/dbconfig/20210428-040718-marostegui.json [production]
2021-04-27 §
13:45 <akosiaris> switchover api-gateway, changeprop, cpjobqueue to use the new redis cluster servers (rdb2007-rdb2010) [production]
2020-08-28 §
16:06 <rzl> starting one more live test of the data center switchover automation, no production impact is expected but there will be some SAL noise [production]
2020-08-24 §
14:55 <rzl> switchover test complete, puppet re-enabled on cumin1001 [production]
14:18 <rzl> disabling puppet on cumin1001 and starting a test of the DC switchover automation, expect some SAL noise but no production impact [production]
2020-07-15 §
04:44 <marostegui@cumin1001> dbctl commit (dc=all): 'Set db1103 weight to 0 before the switchover T254871', diff saved to https://phabricator.wikimedia.org/P11908 and previous config saved to /var/cache/conftool/dbconfig/20200715-044432-marostegui.json [production]
2020-07-08 §
07:48 <jynus> stop bacula-director on backup1001 in preparation for m1 switchover T256717 [production]
2020-07-06 §
13:26 <marostegui@cumin1001> dbctl commit (dc=all): 'Reduce es1024 weight in preparation for tomorrow's switchover T255755', diff saved to https://phabricator.wikimedia.org/P11750 and previous config saved to /var/cache/conftool/dbconfig/20200706-132634-marostegui.json [production]
2020-06-25 §
07:08 <marostegui> Start pre switchover steps on m1 T254556 [production]
2020-06-16 §
13:12 <XioNoX> add graceful-switchover to cr1-eqiad [production]
2020-06-15 §
08:22 <marostegui> Switchover m3-master from dbproxy1008 to dbproxy1016 - T202367 [production]
2020-06-08 §
11:28 <XioNoX> cr1-codfw add graceful-switchover - T243080 [production]
2020-06-03 §
05:38 <XioNoX> deactivate graceful-switchover on cr3-esams - T254021 [production]
2020-02-18 §
12:45 <XioNoX> remove graceful-switchover and nonstop-routing from cr2-esams - T243080 [production]
2020-02-11 §
22:04 <XioNoX> switchover RE mastership back re0 on cr1-eqsin - T243080 [production]
21:32 <XioNoX> switchover RE mastership on cr1-eqsin - T243080 [production]
2020-02-06 §
10:21 <akosiaris> undo "switchover selectively eventgate-analytics.discovery.wmnet to codfw for mw1331 and mw1348". no effect observed [production]
10:20 <akosiaris> undo "switchover selectively eventgate-analytics.discovery.wmnet to codfw for mw1331 and mw1348" [production]
08:17 <akosiaris> switchover selectively eventgate-analytics.discovery.wmnet to codfw for mw1331 and mw1348 to [production]
2020-01-23 §
18:35 <rlazarus> etcd main cluster switchover complete, eqiad is now read-write [production]
16:00 <rlazarus> Starting etcd main cluster switchover from codfw to eqiad [production]
2019-11-19 §
05:02 <marostegui> Start pre-switchover steps T235469 [production]
2019-10-15 §
04:15 <marostegui> Start pre-switchover steps T234300 [production]
2019-09-26 §
04:10 <marostegui> Start pre-switchover s4 steps T230784 [production]
2019-09-24 §
04:13 <marostegui> Start pre switchover steps - T230783 [production]