901-950 of 1149 results (12ms)
2022-02-02 §
08:48 <root@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Switchover es4 T300127 [production]
08:48 <root@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 6 hosts with reason: Switchover es4 T300127 [production]
07:38 <root@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 6 hosts with reason: Switchover es4 T300127 [production]
07:38 <root@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on 6 hosts with reason: Switchover es4 T300127 [production]
2022-01-27 §
09:23 <root@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2078,2132].codfw.wmnet,db[1117,1128,1159].eqiad.wmnet with reason: Primary switchover m1 T299624 [production]
09:23 <root@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on db[2078,2132].codfw.wmnet,db[1117,1128,1159].eqiad.wmnet with reason: Primary switchover m1 T299624 [production]
2022-01-19 §
17:58 <herron> beginning logstash apifeatureusage switchover T297239 [production]
2021-11-11 §
05:14 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 31 hosts with reason: Primary switchover s8 T294321 [production]
05:13 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 31 hosts with reason: Primary switchover s8 T294321 [production]
2021-11-03 §
05:01 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 32 hosts with reason: Primary switchover s1 T293964 [production]
05:01 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 32 hosts with reason: Primary switchover s1 T293964 [production]
2021-09-20 §
17:02 <legoktm> repooled codfw (traffic/caches) 1 week after DC switchover [production]
2021-09-14 §
14:40 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 37 hosts with reason: DC switchover [production]
14:40 <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 37 hosts with reason: DC switchover [production]
2021-09-06 §
04:07 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 33 hosts with reason: Primary switchover s4 T289650 [production]
04:06 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on 33 hosts with reason: Primary switchover s4 T289650 [production]
2021-08-25 §
05:27 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:04:00 on 27 hosts with reason: Primary switchover s7 T289129 [production]
05:27 <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 1:04:00 on 27 hosts with reason: Primary switchover s7 T289129 [production]
2021-08-11 §
04:15 <root@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Master switchover s2 T287454 [production]
04:15 <root@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 24 hosts with reason: Master switchover s2 T287454 [production]
2021-08-10 §
12:23 <kormat> non-destructive (🤞) testing of db-switchover against s2/eqiad T288500 [production]
05:56 <marostegui@cumin1001> dbctl commit (dc=all): 'db2104 (re)pooling @ 100%: repool after failed switchover', diff saved to https://phabricator.wikimedia.org/P16987 and previous config saved to /var/cache/conftool/dbconfig/20210810-055642-root.json [production]
05:41 <marostegui@cumin1001> dbctl commit (dc=all): 'db2104 (re)pooling @ 75%: repool after failed switchover', diff saved to https://phabricator.wikimedia.org/P16986 and previous config saved to /var/cache/conftool/dbconfig/20210810-054139-root.json [production]
05:26 <marostegui@cumin1001> dbctl commit (dc=all): 'db2104 (re)pooling @ 50%: repool after failed switchover', diff saved to https://phabricator.wikimedia.org/P16985 and previous config saved to /var/cache/conftool/dbconfig/20210810-052635-root.json [production]
05:11 <marostegui@cumin1001> dbctl commit (dc=all): 'db2104 (re)pooling @ 25%: repool after failed switchover', diff saved to https://phabricator.wikimedia.org/P16984 and previous config saved to /var/cache/conftool/dbconfig/20210810-051131-root.json [production]
04:23 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Master switchover s2 T287454 [production]
04:23 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 24 hosts with reason: Master switchover s2 T287454 [production]
2021-07-22 §
18:10 <legoktm> testing dc switchover warmup script in eqiad [production]
2021-06-30 §
09:35 <godog> start swiftrepl-mw on ms-fe2005 post-switchover (credentials were missing) - T162123 [production]
2021-06-29 §
14:13 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 44 hosts with reason: DC switchover [production]
14:13 <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 44 hosts with reason: DC switchover [production]
2021-06-28 §
22:31 <legoktm> starting DC switchover live test, which will "switch" us from codfw -> eqiad [production]
18:40 <ebernhardson@deploy1002> Synchronized wmf-config/: T281515: Prepare Cirrus more_like for dc switchover (duration: 01m 02s) [production]
2021-06-24 §
20:58 <legoktm> starting dry run and live test of DC switchover [production]
2021-06-22 §
04:20 <marostegui> Start topology changes for s5 switchover T284529 [production]
04:19 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Master switchover s5 T284529 [production]
04:19 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 24 hosts with reason: Master switchover s5 T284529 [production]
2021-06-21 §
04:48 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 21 hosts with reason: Master switchover s3 T284648 [production]
04:48 <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 21 hosts with reason: Master switchover s3 T284648 [production]
2021-05-17 §
04:46 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Master switchover s6 T282124 [production]
04:46 <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 26 hosts with reason: Master switchover s6 T282124 [production]
2021-04-28 §
04:07 <marostegui@cumin1001> dbctl commit (dc=all): 'Set db1163 with weight 0 before the switchover T278214', diff saved to https://phabricator.wikimedia.org/P15598 and previous config saved to /var/cache/conftool/dbconfig/20210428-040718-marostegui.json [production]
2021-04-27 §
13:45 <akosiaris> switchover api-gateway, changeprop, cpjobqueue to use the new redis cluster servers (rdb2007-rdb2010) [production]
2020-08-28 §
16:06 <rzl> starting one more live test of the data center switchover automation, no production impact is expected but there will be some SAL noise [production]
2020-08-24 §
14:55 <rzl> switchover test complete, puppet re-enabled on cumin1001 [production]
14:18 <rzl> disabling puppet on cumin1001 and starting a test of the DC switchover automation, expect some SAL noise but no production impact [production]
2020-07-15 §
04:44 <marostegui@cumin1001> dbctl commit (dc=all): 'Set db1103 weight to 0 before the switchover T254871', diff saved to https://phabricator.wikimedia.org/P11908 and previous config saved to /var/cache/conftool/dbconfig/20200715-044432-marostegui.json [production]
2020-07-08 §
07:48 <jynus> stop bacula-director on backup1001 in preparation for m1 switchover T256717 [production]
2020-07-06 §
13:26 <marostegui@cumin1001> dbctl commit (dc=all): 'Reduce es1024 weight in preparation for tomorrow's switchover T255755', diff saved to https://phabricator.wikimedia.org/P11750 and previous config saved to /var/cache/conftool/dbconfig/20200706-132634-marostegui.json [production]
2020-06-25 §
07:08 <marostegui> Start pre switchover steps on m1 T254556 [production]