3851-3900 of 10000 results (30ms)
2014-08-18 §
09:24 <godog> stop ircecho again on neon, disable puppet on neon [production]
09:11 <godog> restarted apache2 on strontium [production]
08:58 <godog> stopped ircecho on neon while diagnosing puppet failure [production]
03:13 <LocalisationUpdate> ResourceLoader cache refresh completed at Mon Aug 18 03:12:27 UTC 2014 (duration 12m 26s) [production]
03:06 <hoo> Ran sync-common on mw1053 to stop "Unrecognized job type 'ChangeNotification'." exceptions [production]
02:31 <LocalisationUpdate> completed (1.24wmf17) at 2014-08-18 02:30:17+00:00 [production]
02:20 <LocalisationUpdate> completed (1.24wmf16) at 2014-08-18 02:18:52+00:00 [production]
2014-08-17 §
21:08 <legoktm> running migrateAccount.php without --safe or --auto on terbium for bug 69291 [production]
18:45 <hashar> Zuul upgraded [production]
18:41 <hashar> Upgrading Zuul to latest version (that is not a friday afterall) [production]
09:22 <springle> ongoing schema change wikidatawiki & testwiki wb_entity_per_page.epp_redirect_target. osc_host.sh processes on terbium ok to kill in emergency [production]
04:34 <ottomata> restarted udp2log on oxygen [production]
03:05 <LocalisationUpdate> ResourceLoader cache refresh completed at Sun Aug 17 03:04:22 UTC 2014 (duration 4m 21s) [production]
02:49 <springle> killed stuff on labsdb1003 using all disk for temp tables. investigating [production]
02:24 <LocalisationUpdate> completed (1.24wmf17) at 2014-08-17 02:23:08+00:00 [production]
02:14 <LocalisationUpdate> completed (1.24wmf16) at 2014-08-17 02:13:35+00:00 [production]
2014-08-16 §
18:12 <bblack> (amssq33: and yes, removing from fe/be cache pools) [production]
18:11 <bblack> powering off amssq33, it's clipping network traffic at peak times due to bad ethernet connection negotiated down to 100Mbps (see existing RT 7933 in esams queue) [production]
18:02 <bblack> ms-be1006: syslog indicates it started generating repeated "BUG: soft lockup" 10 minutes before dying, in XFS kernel code again... [production]
17:55 <bblack> rebooting ms-be1006, ping-dead in icinga for 23m, console was unresponsive [production]
17:37 <bblack> restarted apache2 on palladium... looks like something went horribly wrong with its puppet of itself that somehow killed off puppetmaster service? [production]
03:07 <LocalisationUpdate> ResourceLoader cache refresh completed at Sat Aug 16 03:06:29 UTC 2014 (duration 6m 28s) [production]
02:27 <LocalisationUpdate> completed (1.24wmf17) at 2014-08-16 02:26:02+00:00 [production]
02:17 <LocalisationUpdate> completed (1.24wmf16) at 2014-08-16 02:16:00+00:00 [production]
2014-08-15 §
20:59 <kaldari> Synchronized php-1.24wmf16/extensions/MobileFrontend/less: fixing iOS search bug (duration: 00m 05s) [production]
17:58 <aude> Synchronized wmf-config/Wikibase.php: Enable redirects on test.wikidata (duration: 00m 07s) [production]
15:53 <aude> Synchronized php-1.24wmf17/extensions/Wikidata: Update test.wikidata (duration: 00m 07s) [production]
15:50 <aude> Synchronized php-1.24wmf17/extensions/Wikidata: Fix database error and snak value display on test wikidata (duration: 00m 09s) [production]
15:00 <ori> re-enabled puppet on mw1017 [production]
13:33 <ori> disabling puppet on mw1017 to test rsyslog config [production]
03:51 <LocalisationUpdate> ResourceLoader cache refresh completed at Fri Aug 15 03:50:23 UTC 2014 (duration 50m 22s) [production]
03:04 <LocalisationUpdate> completed (1.24wmf17) at 2014-08-15 03:03:49+00:00 [production]
02:34 <LocalisationUpdate> completed (1.24wmf16) at 2014-08-15 02:33:21+00:00 [production]
00:24 <ori> Finished scap: SWAT: cherry picks for TMH and Echo (duration: 14m 38s) [production]
00:09 <ori> Started scap: SWAT: cherry picks for TMH and Echo [production]
2014-08-14 §
23:24 <aude> Synchronized wmf-config/Wikibase.php: Bump cache epoch and add badges setting on test.wikidata (duration: 00m 32s) [production]
23:14 <aude> Finished scap: Update branch for test.wikidata (duration: 16m 48s) [production]
22:57 <aude> Started scap: Update branch for test.wikidata [production]
22:26 <aaron> Synchronized php-1.24wmf16/includes/DefaultSettings.php: 67bf481ce1644ff194d7565107d9b8ffe11bf4b7 (duration: 00m 07s) [production]
22:23 <aaron> Synchronized wmf-config/CommonSettings.php: Increased wgParsoidCacheUpdateTitlesPerJob to 12 to lower the backlog (duration: 00m 07s) [production]
22:13 <aude> Started scap: Update branch for test.wikidata [production]
21:49 <reedy> Synchronized php-1.24wmf17/includes/context/RequestContext.php: (no message) (duration: 00m 15s) [production]
21:11 <godog> restarted hhvm on mw1053 [production]
20:48 <_joe|away> stopping puppet, jobrunner on mw1053; HHVM is eating memory like godzilla [production]
19:29 <bblack> puppeting labmon1001, etc [production]
18:57 <reedy> Synchronized wmf-config/: (no message) (duration: 00m 14s) [production]
18:55 <reedy> Synchronized database lists: (no message) (duration: 00m 14s) [production]
18:26 <mutante> stopped ircecho on neon temporarily [production]
18:10 <reedy> rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.24wmf17 [production]
18:05 <reedy> rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.24wmf16 [production]