401-450 of 1149 results (10ms)
2024-04-04 §
08:02 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 29 hosts with reason: Primary switchover s2 T361682 [production]
08:01 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on 29 hosts with reason: Primary switchover s2 T361682 [production]
2024-03-21 §
13:14 <jiji@deploy1002> Finished scap: Check new deployment server (deploy1002) post switchover - March 2024 (duration: 35m 20s) [production]
12:39 <jiji@deploy1002> Started scap: Check new deployment server (deploy1002) post switchover - March 2024 [production]
2024-03-20 §
16:22 <jiji@deploy2002> Finished scap: Backport for [[gerrit:1013083|debug.json: List primary DC servers first (switchover #5) (T357547)]] (duration: 24m 27s) [production]
16:03 <jiji@deploy2002> jiji: Backport for [[gerrit:1013083|debug.json: List primary DC servers first (switchover #5) (T357547)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
15:58 <jiji@deploy2002> Started scap: Backport for [[gerrit:1013083|debug.json: List primary DC servers first (switchover #5) (T357547)]] [production]
14:50 <jiji@deploy2002> Unlocked for deployment [ALL REPOSITORIES]: Datacenter Switchover - T357547 (duration: 61m 28s) [production]
13:48 <jiji@deploy2002> Locking from deployment [ALL REPOSITORIES]: Datacenter Switchover - T357547 [production]
2024-03-19 §
14:42 <effie> Traffic+Services switchover complete, codfw is depooled - Τ357547 [production]
14:40 <jiji@cumin1002> END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) depool all services in codfw: Northward DC Switchover, March 2024 - T357547 [production]
14:16 <jiji@cumin1002> START - Cookbook sre.discovery.datacenter depool all services in codfw: Northward DC Switchover, March 2024 - T357547 [production]
2024-03-14 §
06:06 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 16 hosts with reason: Primary switchover x1 T359919 [production]
06:06 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on 16 hosts with reason: Primary switchover x1 T359919 [production]
2024-03-12 §
09:11 <eoghan@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on gitlab1004.wikimedia.org with reason: Silencing alerts for switchover prep [production]
09:10 <eoghan@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on gitlab1004.wikimedia.org with reason: Silencing alerts for switchover prep [production]
09:10 <eoghan@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on gitlab1004.wikimedia.org with reason: Silencing alerts for switchover prep [production]
09:10 <eoghan@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on gitlab1004.wikimedia.org with reason: Silencing alerts for switchover prep [production]
2024-03-11 §
13:22 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 16 hosts with reason: Primary switchover x1 T359790 [production]
13:22 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on 16 hosts with reason: Primary switchover x1 T359790 [production]
2024-02-26 §
08:58 <slyngs> IDP switchover to idp2002 [production]
2024-02-24 §
10:24 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Primary switchover s7 T358423 [production]
10:23 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on 27 hosts with reason: Primary switchover s7 T358423 [production]
2024-02-20 §
06:03 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Primary switchover es4 T356372 [production]
06:03 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on 6 hosts with reason: Primary switchover es4 T356372 [production]
2024-02-19 §
13:40 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: es4 switchover T357904 [production]
13:40 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on 6 hosts with reason: es4 switchover T357904 [production]
05:53 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on pc[2011,2014].codfw.wmnet,pc[1011,1014].eqiad.wmnet with reason: Primary switchover pc1 T356371 [production]
05:53 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on pc[2011,2014].codfw.wmnet,pc[1011,1014].eqiad.wmnet with reason: Primary switchover pc1 T356371 [production]
2024-02-08 §
06:47 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 38 hosts with reason: Primary switchover s4 T355658 [production]
06:46 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on 38 hosts with reason: Primary switchover s4 T355658 [production]
2024-02-07 §
08:12 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 38 hosts with reason: Primary switchover s4 T356649 [production]
08:11 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on 38 hosts with reason: Primary switchover s4 T356649 [production]
2024-02-01 §
08:57 <marostegui@cumin1002> dbctl commit (dc=all): 'db2104 (re)pooling @ 100%: After switchover', diff saved to https://phabricator.wikimedia.org/P56043 and previous config saved to /var/cache/conftool/dbconfig/20240201-085743-root.json [production]
08:42 <marostegui@cumin1002> dbctl commit (dc=all): 'db2104 (re)pooling @ 75%: After switchover', diff saved to https://phabricator.wikimedia.org/P56042 and previous config saved to /var/cache/conftool/dbconfig/20240201-084238-root.json [production]
08:27 <marostegui@cumin1002> dbctl commit (dc=all): 'db2104 (re)pooling @ 50%: After switchover', diff saved to https://phabricator.wikimedia.org/P56039 and previous config saved to /var/cache/conftool/dbconfig/20240201-082733-root.json [production]
08:12 <marostegui@cumin1002> dbctl commit (dc=all): 'db2104 (re)pooling @ 25%: After switchover', diff saved to https://phabricator.wikimedia.org/P56036 and previous config saved to /var/cache/conftool/dbconfig/20240201-081228-root.json [production]
07:57 <marostegui@cumin1002> dbctl commit (dc=all): 'db2104 (re)pooling @ 10%: After switchover', diff saved to https://phabricator.wikimedia.org/P56034 and previous config saved to /var/cache/conftool/dbconfig/20240201-075723-root.json [production]
07:45 <marostegui@cumin1002> dbctl commit (dc=all): 'es2024 (re)pooling @ 100%: After switchover ', diff saved to https://phabricator.wikimedia.org/P56032 and previous config saved to /var/cache/conftool/dbconfig/20240201-074520-root.json [production]
07:42 <marostegui@cumin1002> dbctl commit (dc=all): 'db2104 (re)pooling @ 5%: After switchover', diff saved to https://phabricator.wikimedia.org/P56031 and previous config saved to /var/cache/conftool/dbconfig/20240201-074218-root.json [production]
07:30 <marostegui@cumin1002> dbctl commit (dc=all): 'es2024 (re)pooling @ 75%: After switchover ', diff saved to https://phabricator.wikimedia.org/P56028 and previous config saved to /var/cache/conftool/dbconfig/20240201-073015-root.json [production]
07:27 <marostegui@cumin1002> dbctl commit (dc=all): 'db2104 (re)pooling @ 1%: After switchover', diff saved to https://phabricator.wikimedia.org/P56027 and previous config saved to /var/cache/conftool/dbconfig/20240201-072713-root.json [production]
07:15 <marostegui@cumin1002> dbctl commit (dc=all): 'es2024 (re)pooling @ 50%: After switchover ', diff saved to https://phabricator.wikimedia.org/P56022 and previous config saved to /var/cache/conftool/dbconfig/20240201-071510-root.json [production]
07:01 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 28 hosts with reason: Primary switchover s2 T356374 [production]
07:00 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on 28 hosts with reason: Primary switchover s2 T356374 [production]
07:00 <marostegui@cumin1002> dbctl commit (dc=all): 'es2024 (re)pooling @ 25%: After switchover ', diff saved to https://phabricator.wikimedia.org/P56019 and previous config saved to /var/cache/conftool/dbconfig/20240201-070005-root.json [production]
06:45 <marostegui@cumin1002> dbctl commit (dc=all): 'es2024 (re)pooling @ 10%: After switchover ', diff saved to https://phabricator.wikimedia.org/P56017 and previous config saved to /var/cache/conftool/dbconfig/20240201-064500-root.json [production]
06:29 <marostegui@cumin1002> dbctl commit (dc=all): 'es2024 (re)pooling @ 5%: After switchover ', diff saved to https://phabricator.wikimedia.org/P56016 and previous config saved to /var/cache/conftool/dbconfig/20240201-062955-root.json [production]
06:21 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on pc[2011,2014].codfw.wmnet,pc[1011,1014].eqiad.wmnet with reason: Primary switchover pc1 T356068 [production]
06:21 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on pc[2011,2014].codfw.wmnet,pc[1011,1014].eqiad.wmnet with reason: Primary switchover pc1 T356068 [production]