1601-1650 of 10000 results (56ms)
2018-09-12 ยง
16:39 <ebernhardson@deploy1001> Synchronized wmf-config/InitialiseSettings.php: cirrussearch: eqiad -> local (codfw) (duration: 00m 50s) [production]
16:32 <jynus@deploy1001> Synchronized wmf-config/db-codfw.php: Reduce db2065, db2061 weights (duration: 00m 48s) [production]
16:14 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Decrease weight for db2049 (duration: 00m 49s) [production]
16:06 <jynus@deploy1001> Synchronized wmf-config/db-codfw.php: Reduce db2056 weight (duration: 00m 49s) [production]
15:56 <akosiaris> switch s*-master DNS records to codfw [production]
15:51 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Decrease weight for db2049 (duration: 00m 50s) [production]
15:38 <krinkle@deploy1001> Synchronized wmf-config/CommonSettings.php: 43b98e414 (duration: 00m 50s) [production]
15:32 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Decrease weight for db2070 (duration: 00m 49s) [production]
15:24 <END> (PASS) - Cookbook sre.switchdc.mediawiki.08-restart-parsoid (exit_code=0) (volans@sarin) [production]
15:22 <oblivian@sarin> conftool action : set/weight=15; selector: dc=codfw,cluster=api_appserver,service=apache2,name=mw21.* [production]
15:19 <oblivian@sarin> conftool action : set/weight=20; selector: dc=codfw,cluster=api_appserver,service=apache2,name=mw22[2-9].* [production]
15:17 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Tweak weights in s1 and s2 (duration: 00m 50s) [production]
15:15 <_joe_> repool mw2141 [production]
15:14 <_joe_> depooling mw2141 for investigation [production]
15:08 <START> - Cookbook sre.switchdc.mediawiki.08-restart-parsoid (volans@sarin) [production]
15:05 <END> (PASS) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=0) (volans@sarin) [production]
15:05 <START> - Cookbook sre.switchdc.mediawiki.08-restore-ttl (volans@sarin) [production]
15:05 <END> (PASS) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=0) (volans@sarin) [production]
15:03 <START> - Cookbook sre.switchdc.mediawiki.08-start-maintenance (volans@sarin) [production]
14:59 <END> (PASS) - Cookbook sre.switchdc.mediawiki.08-update-tendril (exit_code=0) (volans@sarin) [production]
14:58 <START> - Cookbook sre.switchdc.mediawiki.08-update-tendril (volans@sarin) [production]
14:52 <END> (PASS) - Cookbook sre.switchdc.mediawiki.07-set-readwrite (exit_code=0) (volans@sarin) [production]
14:51 <MediaWiki> read-only period ends at: 2018-09-12 14:51:58.936291 (volans@sarin) [production]
14:51 <START> - Cookbook sre.switchdc.mediawiki.07-set-readwrite (volans@sarin) [production]
14:51 <END> (PASS) - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (exit_code=0) (volans@sarin) [production]
14:51 <START> - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (volans@sarin) [production]
14:50 <END> (PASS) - Cookbook sre.switchdc.mediawiki.05-invert-redis-sessions (exit_code=0) (volans@sarin) [production]
14:50 <START> - Cookbook sre.switchdc.mediawiki.05-invert-redis-sessions (volans@sarin) [production]
14:50 <END> (PASS) - Cookbook sre.switchdc.mediawiki.04-switch-traffic (exit_code=0) (volans@sarin) [production]
14:48 <START> - Cookbook sre.switchdc.mediawiki.04-switch-traffic (volans@sarin) [production]
14:47 <END> (PASS) - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (exit_code=0) (volans@sarin) [production]
14:47 <START> - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (volans@sarin) [production]
14:47 <END> (PASS) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=0) (volans@sarin) [production]
14:46 <START> - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (volans@sarin) [production]
14:44 <END> (FAIL) - Cookbook sre.switchdc.mediawiki.02-set-readonly (exit_code=99) (volans@sarin) [production]
14:44 <MediaWiki> read-only period starts at: 2018-09-12 14:44:24.536913 (volans@sarin) [production]
14:44 <START> - Cookbook sre.switchdc.mediawiki.02-set-readonly (volans@sarin) [production]
14:41 <END> (PASS) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=0) (volans@sarin) [production]
14:41 <START> - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (volans@sarin) [production]
14:40 <END> (PASS) - Cookbook sre.switchdc.mediawiki.00-wipe-and-warmup-caches (exit_code=0) (volans@sarin) [production]
14:34 <jynus> stopping mariadb at db2054 [production]
14:29 <START> - Cookbook sre.switchdc.mediawiki.00-wipe-and-warmup-caches (volans@sarin) [production]
14:13 <jynus> starting replication on db2068 [production]
14:07 <END> (PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0) (volans@sarin) [production]
14:07 <START> - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (volans@sarin) [production]
14:06 <END> (PASS) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0) (volans@sarin) [production]
14:06 <START> - Cookbook sre.switchdc.mediawiki.00-disable-puppet (volans@sarin) [production]
14:04 <volans> starting switchover meny "cookbook sre.switchdc.mediawiki eqiad codfw" on sarin - T203777 [production]
14:03 <jynus> stopping db2068 [production]
13:59 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Depool db2054 and db2068 looks like they are stuck (duration: 00m 50s) [production]