|
2018-09-12
ยง
|
| 15:22 |
<oblivian@sarin> |
conftool action : set/weight=15; selector: dc=codfw,cluster=api_appserver,service=apache2,name=mw21.* |
[production] |
| 15:19 |
<oblivian@sarin> |
conftool action : set/weight=20; selector: dc=codfw,cluster=api_appserver,service=apache2,name=mw22[2-9].* |
[production] |
| 15:17 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Tweak weights in s1 and s2 (duration: 00m 50s) |
[production] |
| 15:15 |
<_joe_> |
repool mw2141 |
[production] |
| 15:14 |
<_joe_> |
depooling mw2141 for investigation |
[production] |
| 15:08 |
<START> |
- Cookbook sre.switchdc.mediawiki.08-restart-parsoid (volans@sarin) |
[production] |
| 15:05 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=0) (volans@sarin) |
[production] |
| 15:05 |
<START> |
- Cookbook sre.switchdc.mediawiki.08-restore-ttl (volans@sarin) |
[production] |
| 15:05 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=0) (volans@sarin) |
[production] |
| 15:03 |
<START> |
- Cookbook sre.switchdc.mediawiki.08-start-maintenance (volans@sarin) |
[production] |
| 14:59 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.08-update-tendril (exit_code=0) (volans@sarin) |
[production] |
| 14:58 |
<START> |
- Cookbook sre.switchdc.mediawiki.08-update-tendril (volans@sarin) |
[production] |
| 14:52 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.07-set-readwrite (exit_code=0) (volans@sarin) |
[production] |
| 14:51 |
<MediaWiki> |
read-only period ends at: 2018-09-12 14:51:58.936291 (volans@sarin) |
[production] |
| 14:51 |
<START> |
- Cookbook sre.switchdc.mediawiki.07-set-readwrite (volans@sarin) |
[production] |
| 14:51 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (exit_code=0) (volans@sarin) |
[production] |
| 14:51 |
<START> |
- Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (volans@sarin) |
[production] |
| 14:50 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.05-invert-redis-sessions (exit_code=0) (volans@sarin) |
[production] |
| 14:50 |
<START> |
- Cookbook sre.switchdc.mediawiki.05-invert-redis-sessions (volans@sarin) |
[production] |
| 14:50 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.04-switch-traffic (exit_code=0) (volans@sarin) |
[production] |
| 14:48 |
<START> |
- Cookbook sre.switchdc.mediawiki.04-switch-traffic (volans@sarin) |
[production] |
| 14:47 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (exit_code=0) (volans@sarin) |
[production] |
| 14:47 |
<START> |
- Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (volans@sarin) |
[production] |
| 14:47 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=0) (volans@sarin) |
[production] |
| 14:46 |
<START> |
- Cookbook sre.switchdc.mediawiki.03-set-db-readonly (volans@sarin) |
[production] |
| 14:44 |
<END> |
(FAIL) - Cookbook sre.switchdc.mediawiki.02-set-readonly (exit_code=99) (volans@sarin) |
[production] |
| 14:44 |
<MediaWiki> |
read-only period starts at: 2018-09-12 14:44:24.536913 (volans@sarin) |
[production] |
| 14:44 |
<START> |
- Cookbook sre.switchdc.mediawiki.02-set-readonly (volans@sarin) |
[production] |
| 14:41 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=0) (volans@sarin) |
[production] |
| 14:41 |
<START> |
- Cookbook sre.switchdc.mediawiki.01-stop-maintenance (volans@sarin) |
[production] |
| 14:40 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.00-wipe-and-warmup-caches (exit_code=0) (volans@sarin) |
[production] |
| 14:34 |
<jynus> |
stopping mariadb at db2054 |
[production] |
| 14:29 |
<START> |
- Cookbook sre.switchdc.mediawiki.00-wipe-and-warmup-caches (volans@sarin) |
[production] |
| 14:13 |
<jynus> |
starting replication on db2068 |
[production] |
| 14:07 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0) (volans@sarin) |
[production] |
| 14:07 |
<START> |
- Cookbook sre.switchdc.mediawiki.00-reduce-ttl (volans@sarin) |
[production] |
| 14:06 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0) (volans@sarin) |
[production] |
| 14:06 |
<START> |
- Cookbook sre.switchdc.mediawiki.00-disable-puppet (volans@sarin) |
[production] |
| 14:04 |
<volans> |
starting switchover meny "cookbook sre.switchdc.mediawiki eqiad codfw" on sarin - T203777 |
[production] |
| 14:03 |
<jynus> |
stopping db2068 |
[production] |
| 13:59 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Depool db2054 and db2068 looks like they are stuck (duration: 00m 50s) |
[production] |
| 13:52 |
<ema> |
restart varnish-be on cp1081 due to mbox lag and 503s |
[production] |
| 13:24 |
<marostegui> |
Reload haproxy on dbproxy1010 to repool labsdb1010 - T174047 |
[production] |
| 13:11 |
<otto@deploy1001> |
Finished deploy [eventlogging/analytics@5c6fab6]: Support loading plugins in eventlogging-processor - T203596 (duration: 00m 05s) |
[production] |
| 13:11 |
<otto@deploy1001> |
Started deploy [eventlogging/analytics@5c6fab6]: Support loading plugins in eventlogging-processor - T203596 |
[production] |
| 13:08 |
<jynus@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Disabling CentralNotice translations (duration: 00m 50s) |
[production] |
| 12:37 |
<jynus@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1110 with original load (duration: 00m 50s) |
[production] |
| 12:23 |
<marostegui> |
Reload haproxy on dbproxy1010 to depool labsdb1010 - https://phabricator.wikimedia.org/T174047 |
[production] |
| 12:17 |
<godog> |
repair sdm / sdj on ms-be2042 - T199198 |
[production] |
| 12:07 |
<elukey> |
delete meitnerium.wikimedia.org's ganeti VM (decommissioned) - T203087 |
[production] |