2251-2300 of 10000 results (70ms)
2018-09-10 §
10:25 <[DRY-RUN]> MediaWiki read-only period starts at: 2018-09-10 10:25:20.558408 (volans@sarin) [production]
10:25 <START> - Cookbook sre.switchdc.mediawiki.02-set-readonly (volans@sarin) [production]
10:12 <END> (FAIL) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=99) (volans@sarin) [production]
10:12 <START> - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (volans@sarin) [production]
10:02 <END> (PASS) - Cookbook sre.switchdc.mediawiki.00-wipe-and-warmup-caches (exit_code=0) (volans@sarin) [production]
09:53 <START> - Cookbook sre.switchdc.mediawiki.00-wipe-and-warmup-caches (volans@sarin) [production]
09:36 <END> (PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0) (volans@sarin) [production]
09:36 <START> - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (volans@sarin) [production]
09:32 <END> (PASS) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0) (volans@sarin) [production]
09:31 <START> - Cookbook sre.switchdc.mediawiki.00-disable-puppet (volans@sarin) [production]
09:30 <volans> starting execution of "cookbook sre.switchdc.mediawiki --live-test codfw eqiad" - T199073 [production]
08:34 <Amir1> ores ae96071 is going to beta [releng]
08:22 <marostegui> Drop users metric and wikilytics from core databases [production]
08:04 <marostegui> Drop unused root grants from core servers [production]
07:54 <joal> Manually restarting mediawiki-reduced oozie with manual addition of missing parameter [analytics]
07:46 <moritzm> installing ghostscript security updates [production]
07:18 <volans> restarted pdfrender on scb2004 - T174916 [production]
07:04 <oblivian@deploy1001> Synchronized wmf-config/throttle.php: Deploy throttle rule for Czech School T203909 (duration: 00m 51s) [production]
02:51 <l10nupdate@deploy1001> ResourceLoader cache refresh completed at Mon Sep 10 02:51:00 UTC 2018 (duration 10m 52s) [production]
02:40 <l10nupdate@deploy1001> scap sync-l10n completed (1.32.0-wmf.20) (duration: 13m 48s) [production]
00:46 <tstarling@deploy1001> Synchronized wmf-config/set-time-limit.php: (no justification provided) (duration: 00m 49s) [production]
00:12 <tstarling@deploy1001> Synchronized w/infinite-loop.php: Testing for T97192 (duration: 00m 48s) [production]
00:07 <tstarling@deploy1001> Synchronized wmf-config/PhpAutoPrepend.php: T97192 (duration: 00m 49s) [production]
00:04 <tstarling@deploy1001> Synchronized wmf-config/set-time-limit.php: T97192 (duration: 00m 52s) [production]
2018-09-09 §
19:47 <legoktm> deploying https://gerrit.wikimedia.org/r/459401 [releng]
18:17 <Krinkle> access #cvn-simplewikis add CabalBot bot [cvn]
10:01 <JeanFred> Deploy latest from Git master: 15d2b04, b8e7751, 62d36e9, 27e4a91, 613a23d, 4f8faef, edcb7ef (T203428), 0e3fa2d (T203903) [tools.heritage]
02:32 <legoktm> deploying https://gerrit.wikimedia.org/r/458980 [releng]
00:11 <legoktm> updating docker-pkg for https://gerrit.wikimedia.org/r/459265 [releng]
2018-09-08 §
20:22 <legoktm> killed https://integration.wikimedia.org/ci/job/maintenance-disconnect-full-disks/1503/ because it was taking 2+ hours [releng]
20:20 <legoktm> deployed https://gerrit.wikimedia.org/r/457077 [releng]
20:19 <legoktm> deleted mwext-PoolCounter-* jobs since they're now unused [releng]
10:35 <gtirloni> restarted cron and truncated /var/log/exim4/paniclog (T196137) [tools]
09:45 <gtirloni> tools restarted cron and truncated /var/log/exim4/paniclog (T196137) [production]
04:22 <krinkle@deploy1001> Synchronized multiversion/: Ia27a8f7ed612f (duration: 00m 49s) [production]
04:16 <krinkle@deploy1001> Synchronized wmf-config/profiler.php: Ia27a8f7ed612f (duration: 00m 54s) [production]
03:06 <legoktm> deploying https://gerrit.wikimedia.org/r/458990 https://gerrit.wikimedia.org/r/458996 [releng]
01:51 <legoktm> deploying https://gerrit.wikimedia.org/r/458956 https://gerrit.wikimedia.org/r/458957 [releng]
01:10 <mutante> also rsyncing /var/lib/tor-instances/ data for second instance and restarting service (T196701) [production]
00:53 <mutante> radium - stopping rsync.service [production]
00:27 <mutante> torrelay1001 - reset internal state (sighup) with "arm" and pressing x twice [production]
00:18 <legoktm> deployed https://gerrit.wikimedia.org/r/458882 [releng]
00:18 <mutante> to watch what is happenin on torrelay1001 - sudo -u debian-tor arm - if asked for password it's in passwords::tor in private [production]
00:16 <mutante> tor relay switched over from radium to torrelay1001, fixed /var/lib/tor permissions, restarted service, flipped DNS CNAME (5M TTL), traffic can be seen with "arm", monitoring all green (T196701) [production]
2018-09-07 §
23:26 <mutante> ms-be2042 - repairing /dev/sdj1 (T199198) [production]
23:25 <mutante> ms-be2041 - repairing /dev/sdh1 (T199198) [production]
23:23 <mutante> ms-be1041 - repairing xfs per https://wikitech.wikimedia.org/wiki/Swift/How_To#Repair_xfs_free_blocks_counter_corruption (T199198) [production]
22:17 <mutante> gerrit - restarting for config change to move log files to /var/log/gerrit/ [production]
22:16 <mutante> - cobalt (gerrit) - applying change to move log file location, manually moved logs to /var/log/gerrit, remove old log dir, let puppet re-create it, like on gerrit2001 [production]
21:31 <mutante> gerrit2001, moving gerrit logfiles to /var/log/gerrit, removing old gerrit logdir, letting puppet re-create it as symlink [production]