| 2011-02-17
      
      § | 
    
  | 23:57 | <tstarling> | synchronized php-1.17/includes/DefaultSettings.php | [production] | 
            
  | 23:54 | <tstarling> | synchronized php-1.5/includes/DefaultSettings.php  'commenting out the $wgStylePath reference assignment, to try to fix the canary mismatch errors' | [production] | 
            
  | 23:42 | <Tim> | fixed logrotate on nfs2 and ran it with -f | [production] | 
            
  | 23:16 | <catrope> | synchronized php-1.17/includes/api/ApiQueryPageProps.php  'r82363' | [production] | 
            
  | 23:16 | <catrope> | synchronized php-1.17/includes/specials/SpecialContributions.php  'r82363' | [production] | 
            
  | 23:16 | <catrope> | synchronized php-1.17/includes/specials/SpecialNewpages.php  'r82363' | [production] | 
            
  | 23:08 | <catrope> | synchronized php-1.17/includes/MessageCache.php  'r82362' | [production] | 
            
  | 22:45 | <catrope> | synchronized php-1.17/extensions/WikiEditor/WikiEditor.hooks.php  'r82360' | [production] | 
            
  | 22:38 | <catrope> | synchronized php-1.17/includes/resourceloader/ResourceLoader.php  'Increment RL minifier cache version again' | [production] | 
            
  | 22:37 | <catrope> | synchronized php-1.17/includes/libs/JavaScriptDistiller.php  'r82358' | [production] | 
            
  | 22:25 | <catrope> | synchronized php-1.17/includes/resourceloader/ResourceLoader.php  'r82355. Will expire the entire ResourceLoader minifier cache' | [production] | 
            
  | 22:08 | <catrope> | synchronized php-1.17/includes/libs/JavaScriptDistiller.php  'Deploy minfier fixes' | [production] | 
            
  | 21:37 | <catrope> | synchronized php-1.17/wmf-config/CommonSettings.php  'Bump style version appendix' | [production] | 
            
  | 21:36 | <catrope> | synchronized php-1.17/extensions/LiquidThreads_alpha/jquery/plugins.js  'r82349' | [production] | 
            
  | 21:36 | <catrope> | synchronized php-1.17/extensions/LiquidThreads/jquery/plugins.js  'r82349' | [production] | 
            
  | 16:06 | <mark> | synchronized php-1.5/../php-1.17/includes/media/SVGMetadataExtractor.php | [production] | 
            
  | 15:55 | <mark> | Disabled apparmor profile for apache2 on srv178, which was spamming the log | [production] | 
            
  | 15:09 | <mark> | Temporarily disabled wikidiff on srv151 (puppet will reenable soon) | [production] | 
            
  | 14:53 | <mark> | Cleaned up srv178 | [production] | 
            
  | 04:39 | <demon> | synchronized php-1.17/includes/parser/Preprocessor_DOM.php  'Sync wfErrorLog removals, was filling up /home' | [production] | 
            
  | 04:15 | <apergos1> | killed udp2log on nfs2 and restarted it via init script after removing preprocessorfatal.log again, got back 160 gb, we had gone down to 6.5 gb free on /home on fenari | [production] | 
            
  | 02:47 | <Tim> | re-enabled snaprotate, bug 27471 does not involve data loss | [production] | 
            
  | 02:22 | <LocalisationUpdate> | failed | [production] | 
            
  | 01:59 | <Tim> | redirected /home/wikipedia/common/php symlink to php-1.17 | [production] | 
            
  | 01:37 | <Tim> | disabled snaprotate.pl temporarily while I investigate bug 27471. This has to be reverted within a few days or else the snapshots will fill | [production] | 
            
  | 00:04 | <Tim> | rebooting spence, its load is too high to let me get a terminal | [production] | 
            
  
    | 2011-02-16
      
      § | 
    
  | 21:17 | <catrope> | synchronized php-1.17/wmf-config/db.php  'Depool srv178 from ES' | [production] | 
            
  | 21:09 | <Ryan_Lane|dc> | on nfs2 | [production] | 
            
  | 21:09 | <Ryan_Lane|dc> | killing udp2log and starting it via init script | [production] | 
            
  | 21:08 | <Ryan_Lane|dc> | restarting udp2log on nfs2 | [production] | 
            
  | 21:07 | <Ryan_Lane|dc> | restarting rsyslog on nfs2 | [production] | 
            
  | 21:05 | <Ryan_Lane|dc> | restarting rsyslog on fenari | [production] | 
            
  | 21:00 | <RoanKattouw> | Removing /h/w/logs/preprocessorfatal.log , was 146GB | [production] | 
            
  | 20:59 | <catrope> | synchronized php-1.5/includes/parser/Preprocessor_DOM.php  'Remove debug logging, 146 GB of the same error message over and over again is enough' | [production] | 
            
  | 20:39 | <RobH> | srv178 powered down, full disk, preventing syncs, needs to be powered backup and fixed later | [production] | 
            
  | 20:10 | <catrope> | synchronized php-1.17/includes/resourceloader/ResourceLoaderWikiModule.php  'r82287' | [production] | 
            
  | 20:07 | <RoanKattouw> | Changed /home/wikipedia/bin/l10nupdate (L10Update cron job) to use php-1.17 instead of wmf-deployment | [production] | 
            
  | 20:04 | <RoanKattouw> | Fixed redirect loop on mlwiki a few minutes ago by changing the namespace names for NS_CATEGORY and NS_CATEGORY_TALK in MessagesMl.php | [production] | 
            
  | 20:04 | <catrope> | synchronized php-1.17/includes/Wiki.php  'Remove debugging code' | [production] | 
            
  | 20:03 | <RoanKattouw> | Restarted morebots as me on wikitech | [production] | 
            
  | 17:35 | <catrope> | synchronized php-1.17/languages/messages/MessagesKaa.php  'r82262' | [production] | 
            
  | 17:35 | <catrope> | synchronized php-1.17/languages/messages/MessagesCa.php  'r82262' | [production] | 
            
  | 17:28 | <catrope> | synchronized php-1.17/includes/Skin.php  'Fix stupid fatal' | [production] | 
            
  | 17:21 | <catrope> | synchronized php-1.17/includes/resourceloader/ResourceLoaderStartUpModule.php  'r82260' | [production] | 
            
  | 17:21 | <catrope> | synchronized php-1.17/includes/Skin.php  'r82260' | [production] |