|
2018-09-10
§
|
| 11:11 |
<START> |
- Cookbook sre.switchdc.mediawiki.08-restore-ttl (volans@sarin) |
[production] |
| 11:09 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.07-set-readwrite (exit_code=0) (volans@sarin) |
[production] |
| 11:09 |
<[DRY-RUN]> |
MediaWiki read-only period ends at: 2018-09-10 11:09:40.587760 (volans@sarin) |
[production] |
| 11:09 |
<START> |
- Cookbook sre.switchdc.mediawiki.07-set-readwrite (volans@sarin) |
[production] |
| 11:08 |
<moritzm> |
uploaded a co-installable 4.14 kernel to apt.wikimedia.org (to be used for installer tests) |
[production] |
| 11:06 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/CommonSettings.php: [[gerrit:458770|Add ['null'] (T201009)]] (duration: 00m 50s) |
[production] |
| 10:57 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (exit_code=0) (volans@sarin) |
[production] |
| 10:57 |
<START> |
- Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (volans@sarin) |
[production] |
| 10:52 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.05-invert-redis-sessions (exit_code=0) (volans@sarin) |
[production] |
| 10:52 |
<START> |
- Cookbook sre.switchdc.mediawiki.05-invert-redis-sessions (volans@sarin) |
[production] |
| 10:50 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.04-switch-traffic (exit_code=0) (volans@sarin) |
[production] |
| 10:48 |
<START> |
- Cookbook sre.switchdc.mediawiki.04-switch-traffic (volans@sarin) |
[production] |
| 10:41 |
<jdrewniak@deploy1001> |
Synchronized portals: Wikimedia Portals Update: [[gerrit:459523|Bumping portals to master (T128546)]] (duration: 00m 50s) |
[production] |
| 10:40 |
<jdrewniak@deploy1001> |
Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:459523|Bumping portals to master (T128546)]] (duration: 00m 50s) |
[production] |
| 10:38 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (exit_code=0) (volans@sarin) |
[production] |
| 10:38 |
<START> |
- Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (volans@sarin) |
[production] |
| 10:29 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=0) (volans@sarin) |
[production] |
| 10:28 |
<START> |
- Cookbook sre.switchdc.mediawiki.03-set-db-readonly (volans@sarin) |
[production] |
| 10:27 |
<moritzm> |
uploaded linux-meta 1.20+deb9u2 for apt.wikimedia.org/stretch-wikimedia |
[production] |
| 10:25 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.02-set-readonly (exit_code=0) (volans@sarin) |
[production] |
| 10:25 |
<[DRY-RUN]> |
MediaWiki read-only period starts at: 2018-09-10 10:25:20.558408 (volans@sarin) |
[production] |
| 10:25 |
<START> |
- Cookbook sre.switchdc.mediawiki.02-set-readonly (volans@sarin) |
[production] |
| 10:12 |
<END> |
(FAIL) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=99) (volans@sarin) |
[production] |
| 10:12 |
<START> |
- Cookbook sre.switchdc.mediawiki.01-stop-maintenance (volans@sarin) |
[production] |
| 10:02 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.00-wipe-and-warmup-caches (exit_code=0) (volans@sarin) |
[production] |
| 09:53 |
<START> |
- Cookbook sre.switchdc.mediawiki.00-wipe-and-warmup-caches (volans@sarin) |
[production] |
| 09:36 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0) (volans@sarin) |
[production] |
| 09:36 |
<START> |
- Cookbook sre.switchdc.mediawiki.00-reduce-ttl (volans@sarin) |
[production] |
| 09:32 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0) (volans@sarin) |
[production] |
| 09:31 |
<START> |
- Cookbook sre.switchdc.mediawiki.00-disable-puppet (volans@sarin) |
[production] |
| 09:30 |
<volans> |
starting execution of "cookbook sre.switchdc.mediawiki --live-test codfw eqiad" - T199073 |
[production] |
| 08:22 |
<marostegui> |
Drop users metric and wikilytics from core databases |
[production] |
| 08:04 |
<marostegui> |
Drop unused root grants from core servers |
[production] |
| 07:46 |
<moritzm> |
installing ghostscript security updates |
[production] |
| 07:18 |
<volans> |
restarted pdfrender on scb2004 - T174916 |
[production] |
| 07:04 |
<oblivian@deploy1001> |
Synchronized wmf-config/throttle.php: Deploy throttle rule for Czech School T203909 (duration: 00m 51s) |
[production] |
| 02:51 |
<l10nupdate@deploy1001> |
ResourceLoader cache refresh completed at Mon Sep 10 02:51:00 UTC 2018 (duration 10m 52s) |
[production] |
| 02:40 |
<l10nupdate@deploy1001> |
scap sync-l10n completed (1.32.0-wmf.20) (duration: 13m 48s) |
[production] |
| 00:46 |
<tstarling@deploy1001> |
Synchronized wmf-config/set-time-limit.php: (no justification provided) (duration: 00m 49s) |
[production] |
| 00:12 |
<tstarling@deploy1001> |
Synchronized w/infinite-loop.php: Testing for T97192 (duration: 00m 48s) |
[production] |
| 00:07 |
<tstarling@deploy1001> |
Synchronized wmf-config/PhpAutoPrepend.php: T97192 (duration: 00m 49s) |
[production] |
| 00:04 |
<tstarling@deploy1001> |
Synchronized wmf-config/set-time-limit.php: T97192 (duration: 00m 52s) |
[production] |
|
2018-09-08
§
|
| 09:45 |
<gtirloni> |
tools restarted cron and truncated /var/log/exim4/paniclog (T196137) |
[production] |
| 04:22 |
<krinkle@deploy1001> |
Synchronized multiversion/: Ia27a8f7ed612f (duration: 00m 49s) |
[production] |
| 04:16 |
<krinkle@deploy1001> |
Synchronized wmf-config/profiler.php: Ia27a8f7ed612f (duration: 00m 54s) |
[production] |
| 01:10 |
<mutante> |
also rsyncing /var/lib/tor-instances/ data for second instance and restarting service (T196701) |
[production] |
| 00:53 |
<mutante> |
radium - stopping rsync.service |
[production] |
| 00:27 |
<mutante> |
torrelay1001 - reset internal state (sighup) with "arm" and pressing x twice |
[production] |
| 00:18 |
<mutante> |
to watch what is happenin on torrelay1001 - sudo -u debian-tor arm - if asked for password it's in passwords::tor in private |
[production] |
| 00:16 |
<mutante> |
tor relay switched over from radium to torrelay1001, fixed /var/lib/tor permissions, restarted service, flipped DNS CNAME (5M TTL), traffic can be seen with "arm", monitoring all green (T196701) |
[production] |