|
2018-10-09
ยง
|
| 15:00 |
<joal> |
restart wikidata-article-placeholder oozie job |
[analytics] |
| 14:57 |
<joal> |
restart mediawiki-history denormalize oozie job |
[analytics] |
| 14:56 |
<joal> |
Restart check_denormalize oozie job |
[analytics] |
| 14:53 |
<joal> |
Restart clickstream oozie job to pick new spark-lib |
[analytics] |
| 13:56 |
<ottomata> |
bouncing oozie server on an-coord1001 |
[analytics] |
| 13:54 |
<moritzm> |
rebooting prometheus1004 for kernel security update |
[production] |
| 13:46 |
<joal> |
Restarting oozie-api job |
[analytics] |
| 13:41 |
<moritzm> |
rebooting prometheus1003 for kernel security update |
[production] |
| 13:36 |
<joal> |
fully restart projectview_geo oozier job |
[analytics] |
| 13:28 |
<moritzm> |
rebooting prometheus2004 for kernel security update |
[production] |
| 13:26 |
<joal> |
Full restart of aqs oozie job |
[analytics] |
| 13:25 |
<joal> |
full restart of projectview_hourly |
[analytics] |
| 13:14 |
<joal> |
rerun failed aqs-hourl jobs |
[analytics] |
| 13:13 |
<moritzm> |
rebooting prometheus2003 for kernel security update |
[production] |
| 12:54 |
<gehel> |
silencing wdqs-public lag alerts (service still functional, and SLO unclear) - T199228 |
[production] |
| 12:48 |
<elukey> |
re-run all the failed projectview-hourly-coord and aqs-hourly-coord workflows (restarting them via hue) |
[analytics] |
| 12:47 |
<elukey> |
re-run apis-wf-2018-10-9-8 |
[analytics] |
| 12:45 |
<moritzm> |
installing imagemagick security updates |
[production] |
| 11:47 |
<END> |
(ERROR) - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (exit_code=2) (volans@neodymium) |
[production] |
| 11:46 |
<START> |
- Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (volans@neodymium) |
[production] |
| 11:45 |
<akosiaris> |
dry-run services switchover from codfw to eqiad in preparation for Thursday |
[production] |
| 11:37 |
<END> |
(ERROR) - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (exit_code=2) (volans@neodymium) |
[production] |
| 11:37 |
<START> |
- Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (volans@neodymium) |
[production] |
| 11:14 |
<volans> |
live-test of the inverted switchdc (eqiad->codfw) completed, all good - T203777 |
[production] |
| 11:14 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.08-update-tendril (exit_code=0) (volans@neodymium) |
[production] |
| 11:13 |
<START> |
- Cookbook sre.switchdc.mediawiki.08-update-tendril (volans@neodymium) |
[production] |
| 11:12 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=0) (volans@neodymium) |
[production] |
| 11:11 |
<START> |
- Cookbook sre.switchdc.mediawiki.08-start-maintenance (volans@neodymium) |
[production] |
| 11:11 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=0) (volans@neodymium) |
[production] |
| 11:11 |
<START> |
- Cookbook sre.switchdc.mediawiki.08-restore-ttl (volans@neodymium) |
[production] |
| 11:11 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.07-set-readwrite (exit_code=0) (volans@neodymium) |
[production] |
| 11:11 |
<[DRY-RUN]> |
MediaWiki read-only period ends at: 2018-10-09 11:11:05.042622 (volans@neodymium) |
[production] |
| 11:11 |
<START> |
- Cookbook sre.switchdc.mediawiki.07-set-readwrite (volans@neodymium) |
[production] |
| 11:08 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (exit_code=0) (volans@neodymium) |
[production] |
| 11:08 |
<START> |
- Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (volans@neodymium) |
[production] |
| 11:07 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.05-invert-redis-sessions (exit_code=0) (volans@neodymium) |
[production] |
| 11:07 |
<START> |
- Cookbook sre.switchdc.mediawiki.05-invert-redis-sessions (volans@neodymium) |
[production] |
| 11:06 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.04-switch-traffic (exit_code=0) (volans@neodymium) |
[production] |
| 11:04 |
<START> |
- Cookbook sre.switchdc.mediawiki.04-switch-traffic (volans@neodymium) |
[production] |
| 11:03 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (exit_code=0) (volans@neodymium) |
[production] |
| 11:03 |
<START> |
- Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (volans@neodymium) |
[production] |
| 11:00 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=0) (volans@neodymium) |
[production] |
| 10:59 |
<START> |
- Cookbook sre.switchdc.mediawiki.03-set-db-readonly (volans@neodymium) |
[production] |
| 10:56 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.02-set-readonly (exit_code=0) (volans@neodymium) |
[production] |
| 10:56 |
<[DRY-RUN]> |
MediaWiki read-only period starts at: 2018-10-09 10:56:12.213026 (volans@neodymium) |
[production] |
| 10:56 |
<START> |
- Cookbook sre.switchdc.mediawiki.02-set-readonly (volans@neodymium) |
[production] |
| 10:53 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=0) (volans@neodymium) |
[production] |
| 10:53 |
<START> |
- Cookbook sre.switchdc.mediawiki.01-stop-maintenance (volans@neodymium) |
[production] |
| 10:51 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.00-warmup-caches (exit_code=0) (volans@neodymium) |
[production] |
| 10:49 |
<onimisionipe> |
repooling wdqs2001 catched up on lag - T206423 |
[production] |