| 
      
        2011-11-28
      
      §
     | 
  
    
  | 21:24 | 
  <mark> | 
  Letting Puppet upgrade itself to 2.7.7rc2 on all Lucid servers | 
  [production] | 
            
  | 20:55 | 
  <Jeff_Green> | 
  cycling through payments servers with dist-upgrade/reboot | 
  [production] | 
            
  | 20:20 | 
  <mutante> | 
  installing apt security updates on hooper,spence,fenari,hume | 
  [production] | 
            
  | 20:12 | 
  <mutante> | 
  fenari - removed php5-wikidiff, autoremove unused packages, clean apt cache.. | 
  [production] | 
            
  | 20:06 | 
  <reedy> | 
  synchronized php-1.18/includes/api/ApiFormatXml.php  '[[rev:104476|r104476]]' | 
  [production] | 
            
  | 19:53 | 
  <mutante> | 
  APT pinned php5-pgsql on gallium - fixed Depends: php5-common (= 5.3.2-2wm1) | 
  [production] | 
            
  | 18:43 | 
  <hashar:> | 
  jenkins backup. The host was missing the SSL cert and apache did not start up. Fixed by Daniel Zahn https://gerrit.wikimedia.org/r/1156 | 
  [production] | 
            
  | 17:43 | 
  <mutante> | 
  gallium: installing new kernel and rebooting | 
  [production] | 
            
  | 17:42 | 
  <hashar:> | 
  wrongly stopped jenkins while mutante was doing an upgrade. Jenkins process dead for some minutes :( | 
  [production] | 
            
  | 17:40 | 
  <hashar:> | 
  restarting jenkins | 
  [production] | 
            
  | 17:15 | 
  <mutante> | 
  gallium - installing some upgrades: apache2..,apt,bind9/dnsutils,,php5..,openjdk,puppet..RT 2039 | 
  [production] | 
            
  | 16:38 | 
  <reedy> | 
  synchronizing Wikimedia installation... : Pushing [[rev:104448|r104448]] | 
  [production] | 
            
  | 16:37 | 
  <mutante> | 
  fenari: re-installed parsekit via pecl, added extension=parsekit.so to php5/cli/php.ini | 
  [production] | 
            
  | 15:22 | 
  <RoanKattouw> | 
  Started logmsgbot on fenari, now running as me. Need to puppetize it with ircecho now that fenari has been upgraded to lucid | 
  [production] | 
            
  | 15:21 | 
  <reedy> | 
  synchronized php-1.18/includes/api/ApiHelp.php  '[[rev:104439|r104439]]' | 
  [production] | 
            
  | 15:21 | 
  <Reedy> | 
  sync-file php-1.18/includes/api/ApiMain.php [[rev:104439|r104439]] | 
  [production] | 
            
  
    | 
      
        2011-11-25
      
      §
     | 
  
    
  | 18:10 | 
  <mutante> | 
  secure.wm is up again, thanks Rob | 
  [production] | 
            
  | 16:59 | 
  <mutante> | 
  singer down :/ - http://wikitech.wikimedia.org/view/Singer#fail - secure.wm and others affected | 
  [production] | 
            
  | 16:07 | 
  <mutante> | 
  power cycling singer | 
  [production] | 
            
  | 15:48 | 
  <mutante> | 
  restarting singer to finish upgrade | 
  [production] | 
            
  | 15:21 | 
  <mutante> | 
  starting dist-upgrade on singer | 
  [production] | 
            
  | 15:12 | 
  <mutante> | 
  removing rpm and alien package from fenari | 
  [production] | 
            
  | 14:47 | 
  <mutante> | 
  hooper - done, Blog and Etherpad accessible again | 
  [production] | 
            
  | 14:44 | 
  <mutante> | 
  hooper - rebooting for upgrade | 
  [production] | 
            
  | 14:41 | 
  <mutante> | 
  hooper - installing new kernel,apache2 upgrades,dist-upgrade (within lucid) | 
  [production] | 
            
  | 13:41 | 
  <mutante> | 
  rebooting hume to finish upgrade | 
  [production] | 
            
  | 12:57 | 
  <mutante> | 
  starting to dist-upgrade hume | 
  [production] | 
            
  | 12:33 | 
  <mutante> | 
  stopping nagios and rebooting spence to finish kernel upgrade | 
  [production] | 
            
  | 12:26 | 
  <mutante> | 
  reinstalled wikimedia-task-appserver on fenari - fixed dpkg conflict php5-apc vs php-apc by manually installing php-apc first | 
  [production] | 
            
  | 12:23 | 
  <mutante> | 
  dist-upgrading spence (within same release) | 
  [production] | 
            
  | 12:23 | 
  <mutante> | 
  removing php5-apc from spence | 
  [production] | 
            
  | 12:09 | 
  <mark> | 
  Fixed puppet in labs | 
  [production] | 
            
  | 11:36 | 
  <mutante> | 
  fenari back up as lucid :) feel free to use again | 
  [production] | 
            
  | 11:32 | 
  <mutante> | 
  side-note: sh: /home/wikipedia/bin/start-ircbot: not found | 
  [production] | 
            
  | 11:29 | 
  <mutante> | 
  rebooting fenari to finish upgrade | 
  [production] | 
            
  | 10:59 | 
  <mutante> | 
  starting fenari dist-upgrade | 
  [production] | 
            
  | 09:38 | 
  <mark> | 
  whipped aluminium back into submission by killing rfaulk's python processes | 
  [production] | 
            
  | 09:27 | 
  <apergos> | 
  email notification was off momentarily for the second and last batch of bugzilla cleanup.  now back on. | 
  [production] | 
            
  | 07:50 | 
  <apergos> | 
  disabled user account creation again on bugzilla for the moment | 
  [production] | 
            
  | 01:57 | 
  <LocalisationUpdate> | 
  failed | 
  [production] | 
            
  | 01:52 | 
  <Tim> | 
  created arbcom 2011 vote on enwiki | 
  [production] | 
            
  | 01:47 | 
  <Tim> | 
  on hume: running makeArbcomList.php in tstarling screen | 
  [production] | 
            
  | 00:06 | 
  <rainman-sr> | 
  set the "last indexed" dates back by 2 days for all wikis so that the wrongly indexed pages get reindexes properly | 
  [production] |