601-650 of 10000 results (26ms)
2014-11-30 §
05:50 <ori> 3:50 UTC: switch asw-c-eqiad lost connectivity with cabinet C4. Impact: phabricator down; gap in web request logs and some perf monitoring. Job queue and Recent Changes stream OK b/c redundant servers are up. [production]
03:41 <LocalisationUpdate> ResourceLoader cache refresh completed at Sun Nov 30 03:41:03 UTC 2014 (duration 41m 2s) [production]
02:18 <LocalisationUpdate> completed (1.25wmf10) at 2014-11-30 02:18:22+00:00 [production]
02:18 <l10nupdate> Synchronized php-1.25wmf10/cache/l10n: (no message) (duration: 00m 02s) [production]
02:11 <LocalisationUpdate> completed (1.25wmf9) at 2014-11-30 02:11:01+00:00 [production]
02:11 <l10nupdate> Synchronized php-1.25wmf9/cache/l10n: (no message) (duration: 00m 01s) [production]
2014-11-29 §
04:17 <LocalisationUpdate> ResourceLoader cache refresh completed at Sat Nov 29 04:17:41 UTC 2014 (duration 17m 40s) [production]
02:25 <LocalisationUpdate> completed (1.25wmf10) at 2014-11-29 02:25:53+00:00 [production]
02:25 <l10nupdate> Synchronized php-1.25wmf10/cache/l10n: (no message) (duration: 00m 01s) [production]
02:13 <LocalisationUpdate> completed (1.25wmf9) at 2014-11-29 02:13:35+00:00 [production]
02:13 <l10nupdate> Synchronized php-1.25wmf9/cache/l10n: (no message) (duration: 00m 01s) [production]
2014-11-28 §
13:19 <YuviPanda> restarted apache on palladium, things are recovering [production]
11:20 <qchris> Updated gerrit plugin its-phabricator-from-bugzilla to 97c5f02d3ca6259488a763515251c5cc57a11a51 [production]
11:20 <qchris> Updated gerrit plugin its-phabricator to 9edf90a182e43bfeea7ebbcb20d4a52b6213600d [production]
03:39 <LocalisationUpdate> ResourceLoader cache refresh completed at Fri Nov 28 03:39:52 UTC 2014 (duration 39m 51s) [production]
02:22 <LocalisationUpdate> completed (1.25wmf10) at 2014-11-28 02:22:28+00:00 [production]
02:22 <l10nupdate> Synchronized php-1.25wmf10/cache/l10n: (no message) (duration: 00m 04s) [production]
02:10 <LocalisationUpdate> completed (1.25wmf9) at 2014-11-28 02:10:06+00:00 [production]
02:10 <l10nupdate> Synchronized php-1.25wmf9/cache/l10n: (no message) (duration: 00m 01s) [production]
2014-11-27 §
17:55 <_joe_> restarted hhvm on mw1224, the alarm may have been lost in the puppet failure shower earlier [production]
17:24 <godog> removed /var/lib/carbon/whisper/archived/jenkins from tungsten [production]
17:09 <godog> upload txstatsd 0.7.0~bzr30-0ubuntu0+14 to precise-wikimedia on carbon [production]
16:49 <godog> upload missing txstatsd 1.0.0-1 _source package_ to carbon [production]
16:48 <godog> upload missing txstatsd 1.0.0-1 to carbon [production]
15:48 <hoo> Synchronized php-1.25wmf10/extensions/Wikidata/: Fixing a data model bug + enable Statements on Properties for testwikidata (duration: 00m 12s) [production]
15:34 <hoo> Synchronized wmf-config/Wikibase.php: Set "displayStatementsOnProperties" for wikidata/testwikidata (duration: 00m 06s) [production]
14:48 <akosiaris> upgrading librsvg throughout the fleet [production]
04:35 <springle> restarted squid3 on carbon, but glitches seem to be upstream [production]
04:23 <LocalisationUpdate> ResourceLoader cache refresh completed at Thu Nov 27 04:23:45 UTC 2014 (duration 23m 44s) [production]
03:45 <springle> puppet failures everywhere; transient apt timeout [production]
02:33 <LocalisationUpdate> completed (1.25wmf10) at 2014-11-27 02:33:36+00:00 [production]
02:33 <l10nupdate> Synchronized php-1.25wmf10/cache/l10n: (no message) (duration: 00m 01s) [production]
02:20 <LocalisationUpdate> completed (1.25wmf9) at 2014-11-27 02:20:52+00:00 [production]
02:20 <l10nupdate> Synchronized php-1.25wmf9/cache/l10n: (no message) (duration: 00m 02s) [production]
00:30 <catrope> Synchronized php-1.25wmf10/extensions/VisualEditor: SWAT (duration: 00m 06s) [production]
00:27 <ejegg> set TY batch size=400 [production]
00:09 <awight> disabled thank-you activity records [production]
00:09 <catrope> Synchronized php-1.25wmf10/extensions/VisualEditor: SWAT (duration: 00m 07s) [production]
00:09 <catrope> Synchronized php-1.25wmf9/extensions/VisualEditor: SWAT (duration: 00m 05s) [production]
00:07 <catrope> Synchronized wmf-config/InitialiseSettings.php: Re-enable VisualEditor on frwiktionary and svwiktionary (duration: 00m 06s) [production]
2014-11-26 §
23:57 <ejegg> set TY batch size=700 [production]
23:48 <YuviPanda> manually ran puppet merge on strontium, puppet merge on palladium didn't sync [production]
23:46 <ejegg> enabling TY mail send [production]
23:45 <ejegg> set TY batch size=1 [production]
23:44 <ejegg> updated crm from 96f66e6b6c947c4e4c32c4a4a32dc940dc3b1d60 to 68703898b7ebfb2a038f307f17788739114806e4 [production]
23:38 <hashar> Jenkins all happy after a restart. Crashing to bed [production]
22:56 <hashar> Killing Jenkins, it is deadlocked beyond repair [production]
22:46 <hashar> Jenkins still in deadlock, will hard restart Jenkins and Zuul soonish. [production]
22:38 <ejegg> disabled TY email sending [production]
22:36 <ejegg> enabled TY email sending [production]