6651-6700 of 10000 results (30ms)
2014-08-17 §
09:22 <springle> ongoing schema change wikidatawiki & testwiki wb_entity_per_page.epp_redirect_target. osc_host.sh processes on terbium ok to kill in emergency [production]
04:34 <ottomata> restarted udp2log on oxygen [production]
03:05 <LocalisationUpdate> ResourceLoader cache refresh completed at Sun Aug 17 03:04:22 UTC 2014 (duration 4m 21s) [production]
02:49 <springle> killed stuff on labsdb1003 using all disk for temp tables. investigating [production]
02:24 <LocalisationUpdate> completed (1.24wmf17) at 2014-08-17 02:23:08+00:00 [production]
02:14 <LocalisationUpdate> completed (1.24wmf16) at 2014-08-17 02:13:35+00:00 [production]
2014-08-16 §
18:12 <bblack> (amssq33: and yes, removing from fe/be cache pools) [production]
18:11 <bblack> powering off amssq33, it's clipping network traffic at peak times due to bad ethernet connection negotiated down to 100Mbps (see existing RT 7933 in esams queue) [production]
18:02 <bblack> ms-be1006: syslog indicates it started generating repeated "BUG: soft lockup" 10 minutes before dying, in XFS kernel code again... [production]
17:55 <bblack> rebooting ms-be1006, ping-dead in icinga for 23m, console was unresponsive [production]
17:37 <bblack> restarted apache2 on palladium... looks like something went horribly wrong with its puppet of itself that somehow killed off puppetmaster service? [production]
03:07 <LocalisationUpdate> ResourceLoader cache refresh completed at Sat Aug 16 03:06:29 UTC 2014 (duration 6m 28s) [production]
02:27 <LocalisationUpdate> completed (1.24wmf17) at 2014-08-16 02:26:02+00:00 [production]
02:17 <LocalisationUpdate> completed (1.24wmf16) at 2014-08-16 02:16:00+00:00 [production]
2014-08-15 §
20:59 <kaldari> Synchronized php-1.24wmf16/extensions/MobileFrontend/less: fixing iOS search bug (duration: 00m 05s) [production]
17:58 <aude> Synchronized wmf-config/Wikibase.php: Enable redirects on test.wikidata (duration: 00m 07s) [production]
15:53 <aude> Synchronized php-1.24wmf17/extensions/Wikidata: Update test.wikidata (duration: 00m 07s) [production]
15:50 <aude> Synchronized php-1.24wmf17/extensions/Wikidata: Fix database error and snak value display on test wikidata (duration: 00m 09s) [production]
15:00 <ori> re-enabled puppet on mw1017 [production]
13:33 <ori> disabling puppet on mw1017 to test rsyslog config [production]
03:51 <LocalisationUpdate> ResourceLoader cache refresh completed at Fri Aug 15 03:50:23 UTC 2014 (duration 50m 22s) [production]
03:04 <LocalisationUpdate> completed (1.24wmf17) at 2014-08-15 03:03:49+00:00 [production]
02:34 <LocalisationUpdate> completed (1.24wmf16) at 2014-08-15 02:33:21+00:00 [production]
00:24 <ori> Finished scap: SWAT: cherry picks for TMH and Echo (duration: 14m 38s) [production]
00:09 <ori> Started scap: SWAT: cherry picks for TMH and Echo [production]
2014-08-14 §
23:24 <aude> Synchronized wmf-config/Wikibase.php: Bump cache epoch and add badges setting on test.wikidata (duration: 00m 32s) [production]
23:14 <aude> Finished scap: Update branch for test.wikidata (duration: 16m 48s) [production]
22:57 <aude> Started scap: Update branch for test.wikidata [production]
22:26 <aaron> Synchronized php-1.24wmf16/includes/DefaultSettings.php: 67bf481ce1644ff194d7565107d9b8ffe11bf4b7 (duration: 00m 07s) [production]
22:23 <aaron> Synchronized wmf-config/CommonSettings.php: Increased wgParsoidCacheUpdateTitlesPerJob to 12 to lower the backlog (duration: 00m 07s) [production]
22:13 <aude> Started scap: Update branch for test.wikidata [production]
21:49 <reedy> Synchronized php-1.24wmf17/includes/context/RequestContext.php: (no message) (duration: 00m 15s) [production]
21:11 <godog> restarted hhvm on mw1053 [production]
20:48 <_joe|away> stopping puppet, jobrunner on mw1053; HHVM is eating memory like godzilla [production]
19:29 <bblack> puppeting labmon1001, etc [production]
18:57 <reedy> Synchronized wmf-config/: (no message) (duration: 00m 14s) [production]
18:55 <reedy> Synchronized database lists: (no message) (duration: 00m 14s) [production]
18:26 <mutante> stopped ircecho on neon temporarily [production]
18:10 <reedy> rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.24wmf17 [production]
18:05 <reedy> rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.24wmf16 [production]
17:46 <AaronSchulz> /srv/deployment/jobrunner updated to 795baf3ca4ce8308597dd74e5242aa5bfbbe961d [production]
17:39 <aaron> Synchronized rpc: 6c0ece687bb6ff3fec0ca7e80a587525ebf18a70 (duration: 00m 08s) [production]
16:52 <_joe_> uploaded new hhvm package 3.3-dev+20140728+wmf4 [production]
16:23 <reedy> Synchronized php-1.24wmf17/extensions/CentralAuth/: (no message) (duration: 00m 13s) [production]
16:23 <reedy> Synchronized php-1.24wmf16/extensions/CentralAuth/: (no message) (duration: 00m 14s) [production]
15:49 <Reedy> Running sync-common on mw1053 [production]
15:49 <reedy> Finished scap: testwiki to 1.24wmf17 (duration: 33m 13s) [production]
15:47 <Jeff_Green> adjust wiki-mail._domainkey DNS record to allow sending from 'wiki*@" addresses, instead of just wiki@ [production]
15:23 <_joe_> powercycling mw1053, which looks like the victim of hhvm-induced ooms [production]
15:15 <reedy> Started scap: testwiki to 1.24wmf17 [production]