201-250 of 10000 results (45ms)
2018-10-09 ยง
13:13 <moritzm> rebooting prometheus2003 for kernel security update [production]
12:54 <gehel> silencing wdqs-public lag alerts (service still functional, and SLO unclear) - T199228 [production]
12:45 <moritzm> installing imagemagick security updates [production]
11:47 <END> (ERROR) - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (exit_code=2) (volans@neodymium) [production]
11:46 <START> - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (volans@neodymium) [production]
11:45 <akosiaris> dry-run services switchover from codfw to eqiad in preparation for Thursday [production]
11:37 <END> (ERROR) - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (exit_code=2) (volans@neodymium) [production]
11:37 <START> - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (volans@neodymium) [production]
11:14 <volans> live-test of the inverted switchdc (eqiad->codfw) completed, all good - T203777 [production]
11:14 <END> (PASS) - Cookbook sre.switchdc.mediawiki.08-update-tendril (exit_code=0) (volans@neodymium) [production]
11:13 <START> - Cookbook sre.switchdc.mediawiki.08-update-tendril (volans@neodymium) [production]
11:12 <END> (PASS) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=0) (volans@neodymium) [production]
11:11 <START> - Cookbook sre.switchdc.mediawiki.08-start-maintenance (volans@neodymium) [production]
11:11 <END> (PASS) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=0) (volans@neodymium) [production]
11:11 <START> - Cookbook sre.switchdc.mediawiki.08-restore-ttl (volans@neodymium) [production]
11:11 <END> (PASS) - Cookbook sre.switchdc.mediawiki.07-set-readwrite (exit_code=0) (volans@neodymium) [production]
11:11 <[DRY-RUN]> MediaWiki read-only period ends at: 2018-10-09 11:11:05.042622 (volans@neodymium) [production]
11:11 <START> - Cookbook sre.switchdc.mediawiki.07-set-readwrite (volans@neodymium) [production]
11:08 <END> (PASS) - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (exit_code=0) (volans@neodymium) [production]
11:08 <START> - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (volans@neodymium) [production]
11:07 <END> (PASS) - Cookbook sre.switchdc.mediawiki.05-invert-redis-sessions (exit_code=0) (volans@neodymium) [production]
11:07 <START> - Cookbook sre.switchdc.mediawiki.05-invert-redis-sessions (volans@neodymium) [production]
11:06 <END> (PASS) - Cookbook sre.switchdc.mediawiki.04-switch-traffic (exit_code=0) (volans@neodymium) [production]
11:04 <START> - Cookbook sre.switchdc.mediawiki.04-switch-traffic (volans@neodymium) [production]
11:03 <END> (PASS) - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (exit_code=0) (volans@neodymium) [production]
11:03 <START> - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (volans@neodymium) [production]
11:00 <END> (PASS) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=0) (volans@neodymium) [production]
10:59 <START> - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (volans@neodymium) [production]
10:56 <END> (PASS) - Cookbook sre.switchdc.mediawiki.02-set-readonly (exit_code=0) (volans@neodymium) [production]
10:56 <[DRY-RUN]> MediaWiki read-only period starts at: 2018-10-09 10:56:12.213026 (volans@neodymium) [production]
10:56 <START> - Cookbook sre.switchdc.mediawiki.02-set-readonly (volans@neodymium) [production]
10:53 <END> (PASS) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=0) (volans@neodymium) [production]
10:53 <START> - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (volans@neodymium) [production]
10:51 <END> (PASS) - Cookbook sre.switchdc.mediawiki.00-warmup-caches (exit_code=0) (volans@neodymium) [production]
10:49 <onimisionipe> repooling wdqs2001 catched up on lag - T206423 [production]
10:48 <START> - Cookbook sre.switchdc.mediawiki.00-warmup-caches (volans@neodymium) [production]
10:47 <END> (PASS) - Cookbook sre.switchdc.mediawiki.00-warmup-caches (exit_code=0) (volans@neodymium) [production]
10:41 <START> - Cookbook sre.switchdc.mediawiki.00-warmup-caches (volans@neodymium) [production]
10:40 <END> (PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0) (volans@neodymium) [production]
10:40 <START> - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (volans@neodymium) [production]
10:37 <END> (PASS) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0) (volans@neodymium) [production]
10:36 <START> - Cookbook sre.switchdc.mediawiki.00-disable-puppet (volans@neodymium) [production]
10:35 <onimisionipe> deploying prometheus-blazegraph-exporter 0.6 on all wdqs clusters - T206123 [production]
10:34 <volans> about to perform live-test of the inverted switchdc (eqiad->codfw), actions will be real but basically noop due to codfw being already active - T203777 [production]
09:25 <elukey> swapped Hadoop's hive/oozie from analytics1003 to an-coord1001 [production]
09:16 <ema> restart pybal on lvs1005 to pick up config changes (conf2001 -> conf1004) [production]
09:00 <ema> re-enable puppet/pybal on lvs1002, IPv6 connectivity with phab1001 working again T201039 [production]
08:16 <elukey> update puppet compiler facts [production]
08:06 <onimisionipe> depooling wdqs2001 to catch up on lag -T206423 [production]
07:03 <akosiaris> restart zuul and zuul-merger on contint1001 for the upgrade of zuul to finish [production]