201-250 of 1274 results (10ms)
2025-02-27 §
17:36 <hnowlan@cumin2002> START - Cookbook sre.switchdc.mediawiki.07-set-readwrite for datacenter switchover from eqiad to codfw [production]
17:36 <hnowlan@cumin2002> END (PASS) - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (exit_code=0) for datacenter switchover from eqiad to codfw [production]
17:36 <hnowlan@cumin2002> START - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite for datacenter switchover from eqiad to codfw [production]
17:36 <hnowlan@cumin2002> END (PASS) - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (exit_code=0) for datacenter switchover from eqiad to codfw [production]
17:35 <hnowlan@cumin2002> START - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki for datacenter switchover from eqiad to codfw [production]
17:35 <hnowlan@cumin2002> END (PASS) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=0) for datacenter switchover from eqiad to codfw [production]
17:34 <hnowlan@cumin2002> START - Cookbook sre.switchdc.mediawiki.03-set-db-readonly for datacenter switchover from eqiad to codfw [production]
17:34 <hnowlan@cumin2002> END (PASS) - Cookbook sre.switchdc.mediawiki.02-set-readonly (exit_code=0) for datacenter switchover from eqiad to codfw [production]
17:34 <hnowlan@cumin2002> START - Cookbook sre.switchdc.mediawiki.02-set-readonly for datacenter switchover from eqiad to codfw [production]
17:33 <hnowlan@cumin2002> END (PASS) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=0) for datacenter switchover from eqiad to codfw [production]
17:32 <hnowlan@cumin2002> START - Cookbook sre.switchdc.mediawiki.01-stop-maintenance for datacenter switchover from eqiad to codfw [production]
17:32 <hnowlan@cumin2002> END (PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0) for datacenter switchover from eqiad to codfw [production]
17:26 <hnowlan@cumin2002> START - Cookbook sre.switchdc.mediawiki.00-reduce-ttl for datacenter switchover from eqiad to codfw [production]
17:26 <hnowlan@cumin2002> END (FAIL) - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches (exit_code=99) for datacenter switchover from eqiad to codfw [production]
17:26 <hnowlan@cumin2002> START - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches for datacenter switchover from eqiad to codfw [production]
17:26 <hnowlan@cumin2002> END (PASS) - Cookbook sre.switchdc.mediawiki.00-downtime-db-readonly-checks (exit_code=0) for datacenter switchover from eqiad to codfw [production]
17:25 <hnowlan@cumin2002> START - Cookbook sre.switchdc.mediawiki.00-downtime-db-readonly-checks for datacenter switchover from eqiad to codfw [production]
17:25 <hnowlan@cumin2002> END (PASS) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0) for datacenter switchover from eqiad to codfw [production]
17:25 <hnowlan@cumin2002> START - Cookbook sre.switchdc.mediawiki.00-disable-puppet for datacenter switchover from eqiad to codfw [production]
13:02 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s6 T387433 [production]
2025-02-26 §
12:06 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Primary switchover es6 T387273 [production]
07:26 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Primary switchover es7 T387271 [production]
06:25 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Primary switchover s7 T387270 [production]
2025-02-25 §
16:12 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Primary switchover es7 T387224 [production]
15:51 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Primary switchover es7 T387224 [production]
13:35 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Primary switchover s7 T387216 [production]
12:52 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Primary switchover es6 T387211 [production]
2025-02-06 §
06:59 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s5 T385148 [production]
2025-02-03 §
13:17 <cgoubert@deploy2002> Unlocked for deployment [MediaWiki]: Emergency s3 switchover T385457 (duration: 07m 36s) [production]
13:09 <cgoubert@deploy2002> Locking from deployment [MediaWiki]: Emergency s3 switchover T385457 [production]
13:06 <marostegui> Emergency s3 switchover T385457 [production]
13:01 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s3 [production]
2025-01-31 §
11:26 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s3 T385051 [production]
2025-01-30 §
09:32 <marostegui@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s5 T385147 [production]
2025-01-23 §
16:26 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Primary switchover s2 T384597 [production]
2025-01-15 §
12:57 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 34 hosts with reason: Primary switchover s1 T383689 [production]
12:57 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on 34 hosts with reason: Primary switchover s1 T383689 [production]
2025-01-13 §
09:48 <marostegui@cumin1002> dbctl commit (dc=all): 'Switchover es5 eqiad master dbmaint T382569', diff saved to https://phabricator.wikimedia.org/P71987 and previous config saved to /var/cache/conftool/dbconfig/20250113-094833-marostegui.json [production]
2025-01-08 §
06:19 <marostegui@cumin1002> dbctl commit (dc=all): 'Switchover es4 eqiad master dbmaint T382569', diff saved to https://phabricator.wikimedia.org/P71829 and previous config saved to /var/cache/conftool/dbconfig/20250108-061914-marostegui.json [production]
2025-01-07 §
08:38 <marostegui@cumin1002> dbctl commit (dc=all): 'Switchover es5 eqiad master dbmaint T382569', diff saved to https://phabricator.wikimedia.org/P71821 and previous config saved to /var/cache/conftool/dbconfig/20250107-083811-marostegui.json [production]
05:40 <marostegui> Switchover m5 eqiad proxy dbmaint eqiad T368874 [production]
2025-01-06 §
07:13 <marostegui> dbmaint Switchover m3 (phabricator) eqiad master dbproxy1020 -> dbproxy1028 T368874 [production]
2025-01-03 §
11:05 <marostegui> Switchover es4 codfw master to es2043 dbmaint T381848 [production]
11:04 <marostegui@cumin1002> dbctl commit (dc=all): 'Switchover es4 codfw master', diff saved to https://phabricator.wikimedia.org/P71779 and previous config saved to /var/cache/conftool/dbconfig/20250103-110440-marostegui.json [production]
2025-01-02 §
11:00 <root@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 32 hosts with reason: Primary switchover s8 T381993 [production]
11:00 <root@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on 32 hosts with reason: Primary switchover s8 T381993 [production]
2024-12-24 §
10:21 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 25 hosts with reason: Primary switchover s5 T382743 [production]
10:21 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on 25 hosts with reason: Primary switchover s5 T382743 [production]
2024-12-05 §
05:20 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2134,2160].codfw.wmnet,db[1159,1213,1217].eqiad.wmnet with reason: m3 master switchover T381365 [production]
05:20 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on db[2134,2160].codfw.wmnet,db[1159,1213,1217].eqiad.wmnet with reason: m3 master switchover T381365 [production]