| 
      
        2011-11-25
      
      §
     | 
  
    
  | 18:10 | 
  <mutante> | 
  secure.wm is up again, thanks Rob | 
  [production] | 
            
  | 16:59 | 
  <mutante> | 
  singer down :/ - http://wikitech.wikimedia.org/view/Singer#fail - secure.wm and others affected | 
  [production] | 
            
  | 16:07 | 
  <mutante> | 
  power cycling singer | 
  [production] | 
            
  | 15:48 | 
  <mutante> | 
  restarting singer to finish upgrade | 
  [production] | 
            
  | 15:21 | 
  <mutante> | 
  starting dist-upgrade on singer | 
  [production] | 
            
  | 15:12 | 
  <mutante> | 
  removing rpm and alien package from fenari | 
  [production] | 
            
  | 14:47 | 
  <mutante> | 
  hooper - done, Blog and Etherpad accessible again | 
  [production] | 
            
  | 14:44 | 
  <mutante> | 
  hooper - rebooting for upgrade | 
  [production] | 
            
  | 14:41 | 
  <mutante> | 
  hooper - installing new kernel,apache2 upgrades,dist-upgrade (within lucid) | 
  [production] | 
            
  | 13:41 | 
  <mutante> | 
  rebooting hume to finish upgrade | 
  [production] | 
            
  | 12:57 | 
  <mutante> | 
  starting to dist-upgrade hume | 
  [production] | 
            
  | 12:33 | 
  <mutante> | 
  stopping nagios and rebooting spence to finish kernel upgrade | 
  [production] | 
            
  | 12:26 | 
  <mutante> | 
  reinstalled wikimedia-task-appserver on fenari - fixed dpkg conflict php5-apc vs php-apc by manually installing php-apc first | 
  [production] | 
            
  | 12:23 | 
  <mutante> | 
  dist-upgrading spence (within same release) | 
  [production] | 
            
  | 12:23 | 
  <mutante> | 
  removing php5-apc from spence | 
  [production] | 
            
  | 12:09 | 
  <mark> | 
  Fixed puppet in labs | 
  [production] | 
            
  | 11:36 | 
  <mutante> | 
  fenari back up as lucid :) feel free to use again | 
  [production] | 
            
  | 11:32 | 
  <mutante> | 
  side-note: sh: /home/wikipedia/bin/start-ircbot: not found | 
  [production] | 
            
  | 11:29 | 
  <mutante> | 
  rebooting fenari to finish upgrade | 
  [production] | 
            
  | 10:59 | 
  <mutante> | 
  starting fenari dist-upgrade | 
  [production] | 
            
  | 09:38 | 
  <mark> | 
  whipped aluminium back into submission by killing rfaulk's python processes | 
  [production] | 
            
  | 09:27 | 
  <apergos> | 
  email notification was off momentarily for the second and last batch of bugzilla cleanup.  now back on. | 
  [production] | 
            
  | 07:50 | 
  <apergos> | 
  disabled user account creation again on bugzilla for the moment | 
  [production] | 
            
  | 01:57 | 
  <LocalisationUpdate> | 
  failed | 
  [production] | 
            
  | 01:52 | 
  <Tim> | 
  created arbcom 2011 vote on enwiki | 
  [production] | 
            
  | 01:47 | 
  <Tim> | 
  on hume: running makeArbcomList.php in tstarling screen | 
  [production] | 
            
  | 00:06 | 
  <rainman-sr> | 
  set the "last indexed" dates back by 2 days for all wikis so that the wrongly indexed pages get reindexes properly | 
  [production] | 
            
  
    | 
      
        2011-11-24
      
      §
     | 
  
    
  | 20:44 | 
  <mark> | 
  Puppet is broken in labs | 
  [production] | 
            
  | 19:32 | 
  <mark> | 
  Letting puppet upgrade itself on virt1 | 
  [production] | 
            
  | 19:04 | 
  <robla> | 
  new account creation on BZ reenabled | 
  [production] | 
            
  | 18:59 | 
  <apergos> | 
  added manually the editbugs right for all users on bugzilla; automatic right via regexp removed by robla  | 
  [production] | 
            
  | 18:12 | 
  <mark> | 
  Temporarily disabled puppet on sockpuppet and virt1 | 
  [production] | 
            
  | 18:02 | 
  <mark> | 
  Installed new Puppet 2.7.7rc2 packages into the Wikimedia APT repository (lucid-wikimedia); hardy packages will follow if needed | 
  [production] | 
            
  | 13:50 | 
  <catrope> | 
  synchronized php-1.18/extensions/GlobalUsage/GlobalUsage_body.php  '[[rev:104163|r104163]]' | 
  [production] | 
            
  | 11:50 | 
  <rainman-sr> | 
  restarting search indexing on searchidx2 as Bug 32376 is resolved | 
  [production] | 
            
  | 11:47 | 
  <mutante> | 
  activated use_large_installation_tweaks on nagios and restarting | 
  [production] | 
            
  | 10:46 | 
  <nikerabbit> | 
  synchronized php-1.18/resources/startup.js  'Touching startup.js for Narayam' | 
  [production] | 
            
  | 10:42 | 
  <nikerabbit> | 
  synchronized wmf-config/InitialiseSettings.php  'Bug 32619 - Narayam on Commons' | 
  [production] | 
            
  | 10:06 | 
  <apergos> | 
  restarted search indexer on searchidx2 (should have also restarted incremental indexer), now that [[bugzilla:32376|bug 32376]] is resolved | 
  [production] | 
            
  | 09:34 | 
  <catrope> | 
  synchronized php-1.18/includes/Export.php  '[[rev:104126|r104126]]' | 
  [production] |