| 2011-11-29
      
      § | 
    
  | 21:10 | <Reedy> | Creating flagged revs tables on siwiki | [production] | 
            
  | 21:10 | <Reedy> | Creating flagged revs tables on bawiki | [production] | 
            
  | 21:10 | <Reedy> | Creating flagged revs tables on elwikinews | [production] | 
            
  | 21:09 | <Reedy> | Creating flagged revs tables on vecwiki | [production] | 
            
  | 21:09 | <Reedy> | Creating flagged revs tables on fiwiki | [production] | 
            
  | 21:08 | <Reedy> | Creating flagged revs tables on ptwikibooks | [production] | 
            
  | 21:08 | <Reedy> | Creating flagged revs tables on lawikisource | [production] | 
            
  | 20:53 | <reedy> | synchronized wmf-config/InitialiseSettings.php  'Bug 32637 - Add Some user group to hindi wikipedia' | [production] | 
            
  | 20:42 | <awjrichards> | synchronized wmf-config/InitialiseSettings.php  'Actually re-enabling ContributionReporting for test, donate, foundation, and meta wikis' | [production] | 
            
  | 20:36 | <mark> | Puppet is currently failing on sanger because it wants to downgrade puppet; I'll fix that tomorrow | [production] | 
            
  | 20:36 | <awjrichards> | synchronizing Wikimedia installation... : Pushing ContributionReporting changes to display a friendlier 'this page temporarily disabled' message | [production] | 
            
  | 19:56 | <cmjohnson1> | msw-c2-sdtpa replacement complete | [production] | 
            
  | 19:50 | <mark> | Upgraded sanger (hardy) to experimental snapshot release of Puppet 2.7.7rc2 | [production] | 
            
  | 19:26 | <cmjohnson1> | replacing msw-c2-sdtpa | [production] | 
            
  | 18:55 | <reedy> | synchronized wmf-config/InitialiseSettings.php  'Bug 32636 - Change UploadNavigationUrl to new link' | [production] | 
            
  | 18:55 | <notpeter> | restarting apache on formey | [production] | 
            
  | 17:32 | <reedy> | synchronized wmf-config/InitialiseSettings.php  'Bug 32642 - Enable e-mail notifications for watchlist (EnotifWatchlist) on WMRU wiki' | [production] | 
            
  | 15:27 | <RoanKattouw> | Creating /var/lock/update-special-pages and /var/lock/update-special-pages-small on hume, as empty files owned by apache. Without these lock files, the update-special-pages cron jobs fail immediately | [production] | 
            
  | 13:03 | <mark> | Fixed broken package situation on ms1002 | [production] | 
            
  | 13:01 | <mark> | Ran dist-upgrade on streber | [production] | 
            
  | 11:45 | <hashar> | synchronized wmf-config/codereview.php  'CodeReview: auto defer /trunk/extensions/AddThis an extension to add G+, facebook, twitter links to "like" stuff' | [production] | 
            
  | 11:18 | <mark> | Running reboot on streber, stuck on disk | [production] | 
            
  | 11:07 | <mark> | Running apt-get dist-upgrade && reboot on sockpuppet | [production] | 
            
  | 09:47 | <mutante> | transcode1 - installing security updates, new kernel, rebooting | [production] | 
            
  | 09:03 | <mutante> | same on zinc, upgrade apt itself and some libs | [production] | 
            
  | 09:00 | <mutante> | started ntpd and installed some upgrades on magnesium | [production] | 
            
  | 01:57 | <LocalisationUpdate> | failed | [production] | 
            
  
    | 2011-11-28
      
      § | 
    
  | 21:24 | <reedy> | synchronizing Wikimedia installation... : Pushing [[rev:104486|r104486]], [[rev:104490|r104490]] | [production] | 
            
  | 21:24 | <mark> | Letting Puppet upgrade itself to 2.7.7rc2 on all Lucid servers | [production] | 
            
  | 20:55 | <Jeff_Green> | cycling through payments servers with dist-upgrade/reboot | [production] | 
            
  | 20:20 | <mutante> | installing apt security updates on hooper,spence,fenari,hume | [production] | 
            
  | 20:12 | <mutante> | fenari - removed php5-wikidiff, autoremove unused packages, clean apt cache.. | [production] | 
            
  | 20:06 | <reedy> | synchronized php-1.18/includes/api/ApiFormatXml.php  '[[rev:104476|r104476]]' | [production] | 
            
  | 19:53 | <mutante> | APT pinned php5-pgsql on gallium - fixed Depends: php5-common (= 5.3.2-2wm1) | [production] | 
            
  | 18:43 | <hashar:> | jenkins backup. The host was missing the SSL cert and apache did not start up. Fixed by Daniel Zahn https://gerrit.wikimedia.org/r/1156 | [production] | 
            
  | 17:43 | <mutante> | gallium: installing new kernel and rebooting | [production] | 
            
  | 17:42 | <hashar:> | wrongly stopped jenkins while mutante was doing an upgrade. Jenkins process dead for some minutes :( | [production] | 
            
  | 17:40 | <hashar:> | restarting jenkins | [production] | 
            
  | 17:15 | <mutante> | gallium - installing some upgrades: apache2..,apt,bind9/dnsutils,,php5..,openjdk,puppet..RT 2039 | [production] | 
            
  | 16:38 | <reedy> | synchronizing Wikimedia installation... : Pushing [[rev:104448|r104448]] | [production] | 
            
  | 16:37 | <mutante> | fenari: re-installed parsekit via pecl, added extension=parsekit.so to php5/cli/php.ini | [production] | 
            
  | 15:22 | <RoanKattouw> | Started logmsgbot on fenari, now running as me. Need to puppetize it with ircecho now that fenari has been upgraded to lucid | [production] | 
            
  | 15:21 | <reedy> | synchronized php-1.18/includes/api/ApiHelp.php  '[[rev:104439|r104439]]' | [production] | 
            
  | 15:21 | <Reedy> | sync-file php-1.18/includes/api/ApiMain.php [[rev:104439|r104439]] | [production] |