| 2011-11-25
      
      § | 
    
  | 12:57 | <mutante> | starting to dist-upgrade hume | [production] | 
            
  | 12:33 | <mutante> | stopping nagios and rebooting spence to finish kernel upgrade | [production] | 
            
  | 12:26 | <mutante> | reinstalled wikimedia-task-appserver on fenari - fixed dpkg conflict php5-apc vs php-apc by manually installing php-apc first | [production] | 
            
  | 12:23 | <mutante> | dist-upgrading spence (within same release) | [production] | 
            
  | 12:23 | <mutante> | removing php5-apc from spence | [production] | 
            
  | 12:09 | <mark> | Fixed puppet in labs | [production] | 
            
  | 11:36 | <mutante> | fenari back up as lucid :) feel free to use again | [production] | 
            
  | 11:32 | <mutante> | side-note: sh: /home/wikipedia/bin/start-ircbot: not found | [production] | 
            
  | 11:29 | <mutante> | rebooting fenari to finish upgrade | [production] | 
            
  | 10:59 | <mutante> | starting fenari dist-upgrade | [production] | 
            
  | 09:38 | <mark> | whipped aluminium back into submission by killing rfaulk's python processes | [production] | 
            
  | 09:27 | <apergos> | email notification was off momentarily for the second and last batch of bugzilla cleanup.  now back on. | [production] | 
            
  | 07:50 | <apergos> | disabled user account creation again on bugzilla for the moment | [production] | 
            
  | 01:57 | <LocalisationUpdate> | failed | [production] | 
            
  | 01:52 | <Tim> | created arbcom 2011 vote on enwiki | [production] | 
            
  | 01:47 | <Tim> | on hume: running makeArbcomList.php in tstarling screen | [production] | 
            
  | 00:06 | <rainman-sr> | set the "last indexed" dates back by 2 days for all wikis so that the wrongly indexed pages get reindexes properly | [production] | 
            
  
    | 2011-11-24
      
      § | 
    
  | 20:44 | <mark> | Puppet is broken in labs | [production] | 
            
  | 19:32 | <mark> | Letting puppet upgrade itself on virt1 | [production] | 
            
  | 19:04 | <robla> | new account creation on BZ reenabled | [production] | 
            
  | 18:59 | <apergos> | added manually the editbugs right for all users on bugzilla; automatic right via regexp removed by robla | [production] | 
            
  | 18:12 | <mark> | Temporarily disabled puppet on sockpuppet and virt1 | [production] | 
            
  | 18:02 | <mark> | Installed new Puppet 2.7.7rc2 packages into the Wikimedia APT repository (lucid-wikimedia); hardy packages will follow if needed | [production] | 
            
  | 13:50 | <catrope> | synchronized php-1.18/extensions/GlobalUsage/GlobalUsage_body.php  '[[rev:104163|r104163]]' | [production] | 
            
  | 11:50 | <rainman-sr> | restarting search indexing on searchidx2 as Bug 32376 is resolved | [production] | 
            
  | 11:47 | <mutante> | activated use_large_installation_tweaks on nagios and restarting | [production] | 
            
  | 10:46 | <nikerabbit> | synchronized php-1.18/resources/startup.js  'Touching startup.js for Narayam' | [production] | 
            
  | 10:42 | <nikerabbit> | synchronized wmf-config/InitialiseSettings.php  'Bug 32619 - Narayam on Commons' | [production] | 
            
  | 10:06 | <apergos> | restarted search indexer on searchidx2 (should have also restarted incremental indexer), now that [[bugzilla:32376|bug 32376]] is resolved | [production] | 
            
  | 09:34 | <catrope> | synchronized php-1.18/includes/Export.php  '[[rev:104126|r104126]]' | [production] | 
            
  | 09:23 | <hashar> | synchronized wmf-config/InitialiseSettings.php  '[[bugzilla:31745|bug 31745]] disable wgUseRCPatrol on ruwikisource' | [production] | 
            
  | 01:57 | <LocalisationUpdate> | failed | [production] | 
            
  | 00:16 | <rainman-sr> | stopped search indexing on searchidx2 until [[bugzilla:32376|bug 32376]] is resolved to prevent more pages being wrongly indexed | [production] | 
            
  | 00:14 | <reedy> | synchronized php-1.18/extensions/cldr/LanguageNamesCs.php  'Sync for Danny B' | [production] | 
            
  
    | 2011-11-23
      
      § | 
    
  | 23:43 | <Tim> | restarted apaches to get new wmerrors | [production] | 
            
  | 23:22 | <K4-713> | synchronized payments cluster to [[rev:104100|r104100]] | [production] | 
            
  | 21:34 | <catrope> | synchronizing Wikimedia installation... : Deploying MoodBar updates | [production] | 
            
  | 21:26 | <reedy> | synchronizing Wikimedia installation... : Pushing fr frontend symlink | [production] | 
            
  | 21:21 | <RoanKattouw> | Adding new moodbar_feedback_response table on all wikis | [production] | 
            
  | 21:08 | <hashar:> | Created symlink in php-1.18/extensions/FlaggedRevs for frontend pointing to presentation. Should fix localisation update from twn which use trunk. See [[rev:102741|r102741]] | [production] | 
            
  | 20:08 | <catrope> | synchronized php-1.18/resources/jquery/jquery.localize.js  '[[rev:104079|r104079]]' | [production] | 
            
  | 19:22 | <midom> | synchronized php/extensions/PrefSwitch/PrefSwitch.classes.php | [production] | 
            
  | 19:08 | <domas> | added missing columns for PrefSwitch survey on few wikis, was broken for a .. year | [production] | 
            
  | 18:03 | <catrope> | synchronized php-1.18/includes/parser/ParserCache.php  '[[rev:104059|r104059]]' | [production] | 
            
  | 17:54 | <Ryan_Lane> | downgrading mysql packages on virt* | [production] | 
            
  | 16:23 | <Ryan_Lane> | shuting down db19 | [production] | 
            
  | 16:20 | <catrope> | synchronized php-1.18/extensions/GlobalUsage/GlobalUsageHooks.php  'fix fatal' | [production] | 
            
  | 16:17 | <catrope> | synchronized php-1.18/extensions/GlobalUsage/GlobalUsageHooks.php  'Prospective fix for GlobalUsage OOM issue' | [production] | 
            
  | 16:10 | <laner> | synchronized wmf-config/db.php  'Removing db19, for maintenance' | [production] | 
            
  | 15:56 | <Ryan_Lane> | nova upgrade is finished. there are still a few issues | [production] |