|
2018-10-09
ยง
|
| 11:45 |
<akosiaris> |
dry-run services switchover from codfw to eqiad in preparation for Thursday |
[production] |
| 11:37 |
<END> |
(ERROR) - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (exit_code=2) (volans@neodymium) |
[production] |
| 11:37 |
<START> |
- Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (volans@neodymium) |
[production] |
| 11:14 |
<volans> |
live-test of the inverted switchdc (eqiad->codfw) completed, all good - T203777 |
[production] |
| 11:14 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.08-update-tendril (exit_code=0) (volans@neodymium) |
[production] |
| 11:13 |
<START> |
- Cookbook sre.switchdc.mediawiki.08-update-tendril (volans@neodymium) |
[production] |
| 11:12 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=0) (volans@neodymium) |
[production] |
| 11:11 |
<START> |
- Cookbook sre.switchdc.mediawiki.08-start-maintenance (volans@neodymium) |
[production] |
| 11:11 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=0) (volans@neodymium) |
[production] |
| 11:11 |
<START> |
- Cookbook sre.switchdc.mediawiki.08-restore-ttl (volans@neodymium) |
[production] |
| 11:11 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.07-set-readwrite (exit_code=0) (volans@neodymium) |
[production] |
| 11:11 |
<[DRY-RUN]> |
MediaWiki read-only period ends at: 2018-10-09 11:11:05.042622 (volans@neodymium) |
[production] |
| 11:11 |
<START> |
- Cookbook sre.switchdc.mediawiki.07-set-readwrite (volans@neodymium) |
[production] |
| 11:08 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (exit_code=0) (volans@neodymium) |
[production] |
| 11:08 |
<START> |
- Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (volans@neodymium) |
[production] |
| 11:07 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.05-invert-redis-sessions (exit_code=0) (volans@neodymium) |
[production] |
| 11:07 |
<START> |
- Cookbook sre.switchdc.mediawiki.05-invert-redis-sessions (volans@neodymium) |
[production] |
| 11:06 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.04-switch-traffic (exit_code=0) (volans@neodymium) |
[production] |
| 11:04 |
<START> |
- Cookbook sre.switchdc.mediawiki.04-switch-traffic (volans@neodymium) |
[production] |
| 11:03 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (exit_code=0) (volans@neodymium) |
[production] |
| 11:03 |
<START> |
- Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (volans@neodymium) |
[production] |
| 11:00 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=0) (volans@neodymium) |
[production] |
| 10:59 |
<START> |
- Cookbook sre.switchdc.mediawiki.03-set-db-readonly (volans@neodymium) |
[production] |
| 10:56 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.02-set-readonly (exit_code=0) (volans@neodymium) |
[production] |
| 10:56 |
<[DRY-RUN]> |
MediaWiki read-only period starts at: 2018-10-09 10:56:12.213026 (volans@neodymium) |
[production] |
| 10:56 |
<START> |
- Cookbook sre.switchdc.mediawiki.02-set-readonly (volans@neodymium) |
[production] |
| 10:53 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=0) (volans@neodymium) |
[production] |
| 10:53 |
<START> |
- Cookbook sre.switchdc.mediawiki.01-stop-maintenance (volans@neodymium) |
[production] |
| 10:51 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.00-warmup-caches (exit_code=0) (volans@neodymium) |
[production] |
| 10:49 |
<onimisionipe> |
repooling wdqs2001 catched up on lag - T206423 |
[production] |
| 10:48 |
<START> |
- Cookbook sre.switchdc.mediawiki.00-warmup-caches (volans@neodymium) |
[production] |
| 10:47 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.00-warmup-caches (exit_code=0) (volans@neodymium) |
[production] |
| 10:41 |
<START> |
- Cookbook sre.switchdc.mediawiki.00-warmup-caches (volans@neodymium) |
[production] |
| 10:40 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0) (volans@neodymium) |
[production] |
| 10:40 |
<START> |
- Cookbook sre.switchdc.mediawiki.00-reduce-ttl (volans@neodymium) |
[production] |
| 10:37 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0) (volans@neodymium) |
[production] |
| 10:36 |
<START> |
- Cookbook sre.switchdc.mediawiki.00-disable-puppet (volans@neodymium) |
[production] |
| 10:35 |
<onimisionipe> |
deploying prometheus-blazegraph-exporter 0.6 on all wdqs clusters - T206123 |
[production] |
| 10:34 |
<volans> |
about to perform live-test of the inverted switchdc (eqiad->codfw), actions will be real but basically noop due to codfw being already active - T203777 |
[production] |
| 09:25 |
<elukey> |
swapped Hadoop's hive/oozie from analytics1003 to an-coord1001 |
[production] |
| 09:16 |
<ema> |
restart pybal on lvs1005 to pick up config changes (conf2001 -> conf1004) |
[production] |
| 09:00 |
<ema> |
re-enable puppet/pybal on lvs1002, IPv6 connectivity with phab1001 working again T201039 |
[production] |
| 08:16 |
<elukey> |
update puppet compiler facts |
[production] |
| 08:06 |
<onimisionipe> |
depooling wdqs2001 to catch up on lag -T206423 |
[production] |
| 07:03 |
<akosiaris> |
restart zuul and zuul-merger on contint1001 for the upgrade of zuul to finish |
[production] |
| 06:37 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1122 (duration: 00m 57s) |
[production] |
| 05:19 |
<marostegui> |
Stop MySQL on db1122 for binlog format change, mysql and kernel upgrade |
[production] |
| 05:14 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1122 (duration: 00m 59s) |
[production] |
| 02:41 |
<krinkle@deploy1001> |
Synchronized wmf-config/profiler.php: T176916 / T206092 - Ie86e88777c48 (duration: 00m 56s) |
[production] |
| 02:21 |
<krinkle@deploy1001> |
Synchronized wmf-config/arclamp.php: T176916 - Id79baae90: ensure file exists before Ie86e88777c48 (duration: 00m 57s) |
[production] |