| 2011-12-28
      
      § | 
    
  | 19:59 | <catrope> | synchronized wmf-config/InitialiseSettings.php  'Add wmgUseMarkAsHelpful, default false' | [production] | 
            
  | 19:56 | <RoanKattouw> | Creating DB table for MarkAsHelpful on all wikis | [production] | 
            
  | 19:49 | <catrope> | synchronizing Wikimedia installation... : Pushing out MarkAsHelpful code | [production] | 
            
  | 17:31 | <Jeff_Green> | srv191 shutdown, looks like failed hdd, see RT #2193 | [production] | 
            
  | 17:09 | <reedy> | synchronized wmf-config/InitialiseSettings.php  '[[bugzilla:33319|bug 33319]] [[bugzilla:33320|bug 33320]]' | [production] | 
            
  | 16:53 | <Reedy> | srv191 complaining of / being readonly | [production] | 
            
  | 16:51 | <reedy> | synchronized wmf-config/InitialiseSettings.php  '[[bugzilla:33319|bug 33319]] [[bugzilla:33320|bug 33320]]' | [production] | 
            
  | 16:02 | <Jeff_Green> | hume:/usr/local/bin/copy_impression_logs_from_storage3.pl deprecated, t'was an emergency hack deployed when storage3 dropped a disk | [production] | 
            
  | 14:51 | <mutante> | fixed mobile traffic logger checks - what they report is for real now. 2 procs on first server, 4 procs on the other three | [production] | 
            
  | 07:03 | <apergos> | maybe the cron job on these image scalers should run more often... cleaned up /tmp on srv221 | [production] | 
            
  | 02:01 | <LocalisationUpdate> | completed (1.18) at Wed Dec 28 02:04:37 UTC 2011 | [production] | 
            
  
    | 2011-12-27
      
      § | 
    
  | 21:21 | <catrope> | synchronized php-1.18/extensions/ArticleFeedbackv5/modules/ext.articleFeedbackv5/ext.articleFeedbackv5.startup.js  '[[rev:107427|r107427]]' | [production] | 
            
  | 19:39 | <LeslieCarr> | removed puppetdlock on brewster | [production] | 
            
  | 19:04 | <LeslieCarr> | removing manual localhost accept all iptables rule on locke (replacing with puppetized rule) | [production] | 
            
  | 18:39 | <Ryan_Lane> | clearing some logs on brewster and restarting lighttpd and squid | [production] | 
            
  | 18:21 | <Jeff_Green> | powercycled kaulen yet again | [production] | 
            
  | 17:47 | <LeslieCarr> | removed file�s out of srv223's /tmp directory to free up its space | [production] | 
            
  | 17:15 | <mark> | Power cycled amslvs3 | [production] | 
            
  | 17:06 | <apergos> | powercycled kaulen, it was (presumably) swapping away. report.cgi was to blame | [production] | 
            
  | 13:45 | <nikerabbit> | synchronizing Wikimedia installation... : I18ndeploy [[rev:107378|r107378]] | [production] | 
            
  | 13:07 | <nikerabbit> | synchronized wmf-config/  'cs gender aliases [[bugzilla:33367|bug 33367]]' | [production] | 
            
  | 02:01 | <LocalisationUpdate> | completed (1.18) at Tue Dec 27 02:04:40 UTC 2011 | [production] | 
            
  
    | 2011-12-24
      
      § | 
    
  | 22:29 | <reedy> | synchronized wmf-config/InitialiseSettings.php  'Bug 33200 - Enable Autopatrol user group and RC patol for sa wiki' | [production] | 
            
  | 22:21 | <reedy> | synchronized wmf-config/InitialiseSettings.php  'Bug 33291 Add autopatrolled flag right on sa.wikipedia' | [production] | 
            
  | 22:17 | <reedy> | synchronized wmf-config/InitialiseSettings.php  '[[bugzilla:33137|bug 33137]] Add transwiki namespace' | [production] | 
            
  | 22:15 | <reedy> | synchronized wmf-config/InitialiseSettings.php  '[[bugzilla:33137|bug 33137]] adding import sources for ptwikinews' | [production] | 
            
  | 22:00 | <Reedy> | Created flaggedrevs tables on ukwiki | [production] | 
            
  | 21:59 | <reedy> | synchronized docroot/wikipedia.org/favicon-si.ico | [production] | 
            
  | 21:46 | <reedy> | synchronized wmf-config/InitialiseSettings.php  'kill tenwiki setting' | [production] | 
            
  | 17:04 | <reedy> | synchronized closed.dblist  'Add tenwiki to closed.dblist per [[bugzilla:33185|bug 33185]]' | [production] | 
            
  | 17:02 | <reedy> | synchronized wmf-config/InitialiseSettings.php  '[[bugzilla:32641|bug 32641]]' | [production] | 
            
  | 16:50 | <reedy> | synchronized wmf-config/InitialiseSettings.php  'Part of [[bugzilla:33137|bug 33137]] Enable GoogleNewsSitemap on ptwikinews' | [production] | 
            
  | 16:46 | <reedy> | synchronized closed.dblist  'Locking readerfeedback.l.w.o per bg 33229' | [production] | 
            
  | 16:43 | <reedy> | synchronized wmf-config/InitialiseSettings.php  'Bug 33042 - Set wgBabelMainCategory to false for nlwiki' | [production] | 
            
  | 02:03 | <LocalisationUpdate> | completed (1.18) at Sat Dec 24 02:06:03 UTC 2011 | [production] | 
            
  
    | 2011-12-23
      
      § | 
    
  | 16:34 | <mutante> | kswapd crashed, there is a Call Trace, and there was a load spike before, guess it is https://bugs.launchpad.net/ubuntu/+bug/721896 or similar | [production] | 
            
  | 16:27 | <mutante> | first interesting syslog line when it started: formey kernel: [39665413.570024] INFO: task kswapd0:36 blocked for more than 120 seconds. | [production] | 
            
  | 16:10 | <mutante> | gerrit and svn back up | [production] | 
            
  | 16:07 | <mutante> | gerrit stopped working and formey would still ping but no ssh connect and no mgmt output, powercycling formey | [production] | 
            
  | 02:07 | <binasher> | db9 maintenance completed | [production] | 
            
  | 02:02 | <LocalisationUpdate> | completed (1.18) at Fri Dec 23 02:05:19 UTC 2011 | [production] | 
            
  | 01:57 | <binasher> | starting db9 maintenance - services will be unavailable for approx 15 minutes | [production] | 
            
  | 00:08 | <hashar> | synchronized php-1.18/extensions/WikimediaIncubator/IncubatorTest.php  'deploy [[rev:107120|r107120]] - ([[bugzilla:32772|bug 32772]]) fix up MoodBar and WikiLove on WIkimedia Incubator' | [production] |