4501-4550 of 10000 results (33ms)
2014-08-16 §
18:02 <bblack> ms-be1006: syslog indicates it started generating repeated "BUG: soft lockup" 10 minutes before dying, in XFS kernel code again... [production]
17:55 <bblack> rebooting ms-be1006, ping-dead in icinga for 23m, console was unresponsive [production]
17:37 <bblack> restarted apache2 on palladium... looks like something went horribly wrong with its puppet of itself that somehow killed off puppetmaster service? [production]
03:07 <LocalisationUpdate> ResourceLoader cache refresh completed at Sat Aug 16 03:06:29 UTC 2014 (duration 6m 28s) [production]
02:27 <LocalisationUpdate> completed (1.24wmf17) at 2014-08-16 02:26:02+00:00 [production]
02:17 <LocalisationUpdate> completed (1.24wmf16) at 2014-08-16 02:16:00+00:00 [production]
2014-08-15 §
20:59 <kaldari> Synchronized php-1.24wmf16/extensions/MobileFrontend/less: fixing iOS search bug (duration: 00m 05s) [production]
17:58 <aude> Synchronized wmf-config/Wikibase.php: Enable redirects on test.wikidata (duration: 00m 07s) [production]
15:53 <aude> Synchronized php-1.24wmf17/extensions/Wikidata: Update test.wikidata (duration: 00m 07s) [production]
15:50 <aude> Synchronized php-1.24wmf17/extensions/Wikidata: Fix database error and snak value display on test wikidata (duration: 00m 09s) [production]
15:00 <ori> re-enabled puppet on mw1017 [production]
13:33 <ori> disabling puppet on mw1017 to test rsyslog config [production]
03:51 <LocalisationUpdate> ResourceLoader cache refresh completed at Fri Aug 15 03:50:23 UTC 2014 (duration 50m 22s) [production]
03:04 <LocalisationUpdate> completed (1.24wmf17) at 2014-08-15 03:03:49+00:00 [production]
02:34 <LocalisationUpdate> completed (1.24wmf16) at 2014-08-15 02:33:21+00:00 [production]
00:24 <ori> Finished scap: SWAT: cherry picks for TMH and Echo (duration: 14m 38s) [production]
00:09 <ori> Started scap: SWAT: cherry picks for TMH and Echo [production]
2014-08-14 §
23:24 <aude> Synchronized wmf-config/Wikibase.php: Bump cache epoch and add badges setting on test.wikidata (duration: 00m 32s) [production]
23:14 <aude> Finished scap: Update branch for test.wikidata (duration: 16m 48s) [production]
22:57 <aude> Started scap: Update branch for test.wikidata [production]
22:26 <aaron> Synchronized php-1.24wmf16/includes/DefaultSettings.php: 67bf481ce1644ff194d7565107d9b8ffe11bf4b7 (duration: 00m 07s) [production]
22:23 <aaron> Synchronized wmf-config/CommonSettings.php: Increased wgParsoidCacheUpdateTitlesPerJob to 12 to lower the backlog (duration: 00m 07s) [production]
22:13 <aude> Started scap: Update branch for test.wikidata [production]
21:49 <reedy> Synchronized php-1.24wmf17/includes/context/RequestContext.php: (no message) (duration: 00m 15s) [production]
21:11 <godog> restarted hhvm on mw1053 [production]
20:48 <_joe|away> stopping puppet, jobrunner on mw1053; HHVM is eating memory like godzilla [production]
19:29 <bblack> puppeting labmon1001, etc [production]
18:57 <reedy> Synchronized wmf-config/: (no message) (duration: 00m 14s) [production]
18:55 <reedy> Synchronized database lists: (no message) (duration: 00m 14s) [production]
18:26 <mutante> stopped ircecho on neon temporarily [production]
18:10 <reedy> rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.24wmf17 [production]
18:05 <reedy> rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.24wmf16 [production]
17:46 <AaronSchulz> /srv/deployment/jobrunner updated to 795baf3ca4ce8308597dd74e5242aa5bfbbe961d [production]
17:39 <aaron> Synchronized rpc: 6c0ece687bb6ff3fec0ca7e80a587525ebf18a70 (duration: 00m 08s) [production]
16:52 <_joe_> uploaded new hhvm package 3.3-dev+20140728+wmf4 [production]
16:23 <reedy> Synchronized php-1.24wmf17/extensions/CentralAuth/: (no message) (duration: 00m 13s) [production]
16:23 <reedy> Synchronized php-1.24wmf16/extensions/CentralAuth/: (no message) (duration: 00m 14s) [production]
15:49 <Reedy> Running sync-common on mw1053 [production]
15:49 <reedy> Finished scap: testwiki to 1.24wmf17 (duration: 33m 13s) [production]
15:47 <Jeff_Green> adjust wiki-mail._domainkey DNS record to allow sending from 'wiki*@" addresses, instead of just wiki@ [production]
15:23 <_joe_> powercycling mw1053, which looks like the victim of hhvm-induced ooms [production]
15:15 <reedy> Started scap: testwiki to 1.24wmf17 [production]
14:01 <_joe_> puppet re-enabled on the appserver [production]
12:38 <_joe_> stopping puppet on appservers while deploying a delicate change. [production]
09:30 <_joe_> the hhvm jobrunner is back in production, seems healthy, see https://logstash.wikimedia.org/#/dashboard/elasticsearch/hhvm_jobrunner [production]
08:09 <_joe_> reactivated the jobrunner on mw1053, with promising results. Puppettization pending (in ~ 1 hour) [production]
03:12 <LocalisationUpdate> ResourceLoader cache refresh completed at Thu Aug 14 03:11:33 UTC 2014 (duration 11m 32s) [production]
02:31 <LocalisationUpdate> completed (1.24wmf16) at 2014-08-14 02:29:52+00:00 [production]
02:17 <LocalisationUpdate> completed (1.24wmf15) at 2014-08-14 02:16:34+00:00 [production]
2014-08-13 §
21:58 <manybubbles> cirrus index rebuild is proceeding without trouble - I'm going to let it continue over night. [production]