51-100 of 1133 results (10ms)
2025-02-27 §
17:57 <hnowlan@cumin2002> END (PASS) - Cookbook sre.switchdc.mediawiki.09-run-puppet-on-db-masters (exit_code=0) for datacenter switchover from eqiad to codfw [production]
17:46 <hnowlan@cumin2002> START - Cookbook sre.switchdc.mediawiki.09-run-puppet-on-db-masters for datacenter switchover from eqiad to codfw [production]
17:46 <hnowlan@cumin2002> END (PASS) - Cookbook sre.switchdc.mediawiki.09-restore-ttl (exit_code=0) for datacenter switchover from eqiad to codfw [production]
17:45 <hnowlan@cumin2002> START - Cookbook sre.switchdc.mediawiki.09-restore-ttl for datacenter switchover from eqiad to codfw [production]
17:45 <hnowlan@cumin2002> END (PASS) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=0) for datacenter switchover from eqiad to codfw [production]
17:42 <hnowlan@cumin2002> START - Cookbook sre.switchdc.mediawiki.08-start-maintenance for datacenter switchover from eqiad to codfw [production]
17:42 <hnowlan@cumin2002> END (PASS) - Cookbook sre.switchdc.mediawiki.08-restart-mw-jobrunner (exit_code=0) for datacenter switchover from eqiad to codfw [production]
17:41 <hnowlan@cumin2002> START - Cookbook sre.switchdc.mediawiki.08-restart-mw-jobrunner for datacenter switchover from eqiad to codfw [production]
17:36 <hnowlan@cumin2002> END (PASS) - Cookbook sre.switchdc.mediawiki.07-set-readwrite (exit_code=0) for datacenter switchover from eqiad to codfw [production]
17:36 <hnowlan@cumin2002> START - Cookbook sre.switchdc.mediawiki.07-set-readwrite for datacenter switchover from eqiad to codfw [production]
17:36 <hnowlan@cumin2002> END (PASS) - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (exit_code=0) for datacenter switchover from eqiad to codfw [production]
17:36 <hnowlan@cumin2002> START - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite for datacenter switchover from eqiad to codfw [production]
17:36 <hnowlan@cumin2002> END (PASS) - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (exit_code=0) for datacenter switchover from eqiad to codfw [production]
17:35 <hnowlan@cumin2002> START - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki for datacenter switchover from eqiad to codfw [production]
17:35 <hnowlan@cumin2002> END (PASS) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=0) for datacenter switchover from eqiad to codfw [production]
17:34 <hnowlan@cumin2002> START - Cookbook sre.switchdc.mediawiki.03-set-db-readonly for datacenter switchover from eqiad to codfw [production]
17:34 <hnowlan@cumin2002> END (PASS) - Cookbook sre.switchdc.mediawiki.02-set-readonly (exit_code=0) for datacenter switchover from eqiad to codfw [production]
17:34 <hnowlan@cumin2002> START - Cookbook sre.switchdc.mediawiki.02-set-readonly for datacenter switchover from eqiad to codfw [production]
17:33 <hnowlan@cumin2002> END (PASS) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=0) for datacenter switchover from eqiad to codfw [production]
17:32 <hnowlan@cumin2002> START - Cookbook sre.switchdc.mediawiki.01-stop-maintenance for datacenter switchover from eqiad to codfw [production]
17:32 <hnowlan@cumin2002> END (PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0) for datacenter switchover from eqiad to codfw [production]
17:26 <hnowlan@cumin2002> START - Cookbook sre.switchdc.mediawiki.00-reduce-ttl for datacenter switchover from eqiad to codfw [production]
17:26 <hnowlan@cumin2002> END (FAIL) - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches (exit_code=99) for datacenter switchover from eqiad to codfw [production]
17:26 <hnowlan@cumin2002> START - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches for datacenter switchover from eqiad to codfw [production]
17:26 <hnowlan@cumin2002> END (PASS) - Cookbook sre.switchdc.mediawiki.00-downtime-db-readonly-checks (exit_code=0) for datacenter switchover from eqiad to codfw [production]
17:25 <hnowlan@cumin2002> START - Cookbook sre.switchdc.mediawiki.00-downtime-db-readonly-checks for datacenter switchover from eqiad to codfw [production]
17:25 <hnowlan@cumin2002> END (PASS) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0) for datacenter switchover from eqiad to codfw [production]
17:25 <hnowlan@cumin2002> START - Cookbook sre.switchdc.mediawiki.00-disable-puppet for datacenter switchover from eqiad to codfw [production]
13:02 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s6 T387433 [production]
2025-02-26 §
12:06 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Primary switchover es6 T387273 [production]
07:26 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Primary switchover es7 T387271 [production]
06:25 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Primary switchover s7 T387270 [production]
2025-02-25 §
16:12 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Primary switchover es7 T387224 [production]
15:51 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Primary switchover es7 T387224 [production]
13:35 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Primary switchover s7 T387216 [production]
12:52 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Primary switchover es6 T387211 [production]
2025-02-06 §
06:59 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s5 T385148 [production]
2025-02-03 §
13:17 <cgoubert@deploy2002> Unlocked for deployment [MediaWiki]: Emergency s3 switchover T385457 (duration: 07m 36s) [production]
13:09 <cgoubert@deploy2002> Locking from deployment [MediaWiki]: Emergency s3 switchover T385457 [production]
13:06 <marostegui> Emergency s3 switchover T385457 [production]
13:01 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s3 [production]
2025-01-31 §
11:26 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s3 T385051 [production]
2025-01-30 §
09:32 <marostegui@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s5 T385147 [production]
2025-01-23 §
16:26 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Primary switchover s2 T384597 [production]
2025-01-15 §
12:57 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 34 hosts with reason: Primary switchover s1 T383689 [production]
12:57 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on 34 hosts with reason: Primary switchover s1 T383689 [production]
2025-01-13 §
09:48 <marostegui@cumin1002> dbctl commit (dc=all): 'Switchover es5 eqiad master dbmaint T382569', diff saved to https://phabricator.wikimedia.org/P71987 and previous config saved to /var/cache/conftool/dbconfig/20250113-094833-marostegui.json [production]
2025-01-08 §
06:19 <marostegui@cumin1002> dbctl commit (dc=all): 'Switchover es4 eqiad master dbmaint T382569', diff saved to https://phabricator.wikimedia.org/P71829 and previous config saved to /var/cache/conftool/dbconfig/20250108-061914-marostegui.json [production]
2025-01-07 §
08:38 <marostegui@cumin1002> dbctl commit (dc=all): 'Switchover es5 eqiad master dbmaint T382569', diff saved to https://phabricator.wikimedia.org/P71821 and previous config saved to /var/cache/conftool/dbconfig/20250107-083811-marostegui.json [production]
05:40 <marostegui> Switchover m5 eqiad proxy dbmaint eqiad T368874 [production]