| 2013-03-26
      
      ยง | 
    
  | 22:49 | <mutante> | search21 - remove sun-j2sdk , replaced by oracle-jdsdk | [production] | 
            
  | 22:47 | <mutante> | search21 - broken DPKG: iF  sun-j2sdk1.6 | [production] | 
            
  | 22:45 | <mutante> | mw2 - broken DPKG: pi  librsvg2-2 , pi  librsvg2-bin | [production] | 
            
  | 22:39 | <LeslieCarr> | did a service apache2 start on mw108* | [production] | 
            
  | 22:28 | <reedy> | synchronized wmf-config/InitialiseSettings.php | [production] | 
            
  | 22:24 | <reedy> | synchronized wmf-config/InitialiseSettings.php | [production] | 
            
  | 22:10 | <notpeter> | icinga mostly trustworthy again :) | [production] | 
            
  | 22:08 | <andrewbogott> | restarted nova-api on virt0 because it appeared to have crashed. | [production] | 
            
  | 21:31 | <andrewbogott> | restarted memcached on virt0 in a futile attempt to address https://bugzilla.wikimedia.org/show_bug.cgi?id=46583 | [production] | 
            
  | 21:29 | <mutante> | killed stuck snmptt procs on neon to fix the others.. apparently :p | [production] | 
            
  | 21:28 | <mutante> | stopping gmetad on neon | [production] | 
            
  | 20:51 | <kaldari> | synchronized wmf-config/CommonSettings.php  'syncing CommonSettings.php for Echo event tracking' | [production] | 
            
  | 20:41 | <mutante> | more degraded RAIDs:  db1001->RT-4832, db1028->4834 | [production] | 
            
  | 20:35 | <mutante> | broken disks/degraded RAID  on db servers: db11->RT-4828, db29->RT-4829, db45->RT-4831 | [production] | 
            
  | 20:34 | <kaldari> | synchronized php-1.21wmf12/extensions/ContributionReporting  'syncing ContributionReporting extension' | [production] | 
            
  | 20:07 | <asher> | synchronized wmf-config/db-eqiad.php  'raising db1051 to full weight' | [production] | 
            
  | 19:49 | <khorn> | synchronized php-1.21wmf12/extensions/CentralNotice  'Fixing CentralNotice bug in which it was possible to initialize multiple times' | [production] | 
            
  | 19:45 | <khorn> | synchronized php-1.21wmf11/extensions/CentralNotice  'Fixing CentralNotice bug in which it was possible to initialize multiple times' | [production] | 
            
  | 19:40 | <kaldari> | synchronized php-1.21wmf12/resources/startup.js  'syncing php-1.21wmf12/resources/startup.js' | [production] | 
            
  | 19:32 | <reedy> | synchronized php-1.21wmf12/extensions/Wikibase/client | [production] | 
            
  | 19:27 | <cmjohnson1> | mw1209 re-imaging | [production] | 
            
  | 19:20 | <kaldari> | synchronized php-1.21wmf12/extensions/Echo/modules/icons  're-syncing Echo icons dir' | [production] | 
            
  | 19:19 | <mlitn> | synchronized wmf-config/InitialiseSettings.php  'Include AFTv5 on frwiki' | [production] | 
            
  | 19:12 | <mlitn> | synchronized wmf-config  'Enable AFTv5 on frwiki' | [production] | 
            
  | 19:05 | <reedy> | synchronized php-1.21wmf12/extensions/Collection/ | [production] | 
            
  | 19:04 | <mlitn> | synchronized php-1.21wmf12/extensions/ArticleFeedbackv5  'Update ArticleFeedbackv5 to master' | [production] | 
            
  | 19:03 | <mlitn> | synchronized php-1.21wmf11/extensions/ArticleFeedbackv5  'Update ArticleFeedbackv5 to master' | [production] | 
            
  | 19:02 | <kaldari> | synchronized php-1.21wmf12/extensions/Echo/modules/icons  'syncing Echo icons dir' | [production] | 
            
  | 18:58 | <mutante> | starting gmetad on neon - we failed to resolve data source name ms-fe1001.eqiad.wmnet | [production] | 
            
  | 18:56 | <mutante> | restarting ircecho on fenari | [production] | 
            
  | 18:48 | <mutante> | stopping gmetad on neon | [production] | 
            
  | 18:44 | <mutante> | killing all snmp related processes on neon and restarting | [production] | 
            
  | 18:43 | <reedy> | synchronized php-1.21wmf12/extensions/MobileFrontend/includes/specials/SpecialMobileWatchlist.php | [production] | 
            
  | 18:43 | <reedy> | synchronized php-1.21wmf12/extensions/MobileFrontend/includes/specials/SpecialMobileWatchlist.php | [production] | 
            
  | 18:41 | <kaldari> | synchronized php-1.21wmf11/extensions/Echo/modules/icons  'syncing Echo icons dir' | [production] | 
            
  | 18:41 | <kaldari> | synchronized php-1.21wmf11/extensions/Echo/modules/icons  'syncing Echo icons dir' | [production] | 
            
  | 18:38 | <mutante> | restarting snmtrapd on neon | [production] | 
            
  | 18:25 | <mlitn> | synchronized wmf-config/InitialiseSettings.php  'Turning Thanks extension on on MediaWiki wiki' | [production] | 
            
  | 18:24 | <mlitn> | synchronized wmf-config/InitialiseSettings.php  'Turning Thanks extension on on MediaWiki wiki' | [production] | 
            
  | 18:13 | <krinkle> | synchronized php-1.21wmf12/resources/mediawiki/mediawiki.js  'I80af730daa815 fixing bug 46575' | [production] | 
            
  | 18:13 | <krinkle> | synchronized php-1.21wmf12/resources/mediawiki/mediawiki.js  'I80af730daa815 fixing bug 46575' | [production] | 
            
  | 18:12 | <Krinkle> | Change I34a488247 (mw.loader debug) has been reverted in wmf/1.21wmf12 and pulled on fenari | [production] | 
            
  | 17:51 | <cmjohnson1> | removing disk from slot 0 ms-be1012 | [production] | 
            
  | 17:14 | <Krinkle> | Change I34a488247 is now merged in wmf/1.21wmf12 and pulled on fenari for test.wikipedia.org, do not sync. | [production] | 
            
  | 17:04 | <ottomata> | added (crappy, but better than nothing) .deb for webstatscollector to apt.wikimedia.org | [production] | 
            
  | 15:24 | <cmjohnson1> | swapping disk 6 ms-be1012 | [production] | 
            
  | 14:34 | <hashar> | synchronized multiversion  '{{gerrit|47564}} wikiversions.cdb now vary by realm' | [production] | 
            
  | 14:34 | <hashar> | synchronized multiversion  '{{gerrit|47564}} wikiversions.cdb now vary by realm' | [production] | 
            
  | 14:09 | <cmjohnson1> | running sync-common on mw1209 | [production] | 
            
  | 10:23 | <nikerabbit> | synchronized php-1.21wmf12/extensions/UniversalLanguageSelector/UniversalLanguageSelector.php  'Testing fix for wikidata' | [production] |