| 2013-04-29
      
      § | 
    
  | 22:27 | <RoanKattouw> | Running scap | [production] | 
            
  | 22:22 | <RoanKattouw> | Running scap again | [production] | 
            
  | 22:18 | <catrope> | Started syncing Wikimedia installation... : | [production] | 
            
  | 22:12 | <hashar> | gallium: restarted apache2 to kill off http connections established by mod_proxy to the jenkins backend. | [production] | 
            
  | 22:03 | <hashar> | jenkins web server is no more responding on port 8080.  Might have to restart it. | [production] | 
            
  | 22:00 | <hashar> | gallium / jenkins : blackholed a web drawer using   ip route add blackhole <IP>/32 | [production] | 
            
  | 21:53 | <hashar> | Jenkins is in trouble | [production] | 
            
  | 21:52 | <reedy> | synchronized php-1.22wmf3/includes/DefaultSettings.php  'Fix wgVersion' | [production] | 
            
  | 21:31 | <notpeter> | upgrading db45 to precise | [production] | 
            
  | 21:24 | <catrope> | synchronized wmf-config/InitialiseSettings.php  'Enabling VisualEditor on viwiki and test2wiki' | [production] | 
            
  | 20:31 | <AaronSchulz> | Doing another original/math/timeline sync/copy run | [production] | 
            
  | 20:27 | <aaron> | synchronized wmf-config/filebackend.php  'Enabled multiwrites for ceph/swift' | [production] | 
            
  | 19:19 | <kaldari> | synchronized php-1.22wmf2/extensions/Echo/maintenance/setEmailOptionTemp.php | [production] | 
            
  | 19:09 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.22wmf3 | [production] | 
            
  | 19:06 | <reedy> | Finished syncing Wikimedia installation... : Scap again to see if that fixes ROOTPAGENAME and ROOTPAGENAMEE. Otherwise I'm reverting it | [production] | 
            
  | 18:46 | <reedy> | Started syncing Wikimedia installation... : Scap again to see if that fixes ROOTPAGENAME and ROOTPAGENAMEE. Otherwise I'm reverting it | [production] | 
            
  | 18:37 | <reedy> | synchronized php-1.22wmf2/.git/modules/extensions/WikiLove | [production] | 
            
  | 18:21 | <reedy> | synchronized php-1.22wmf3/extensions/ | [production] | 
            
  | 18:06 | <reedy> | Finished syncing Wikimedia installation... : test2wiki to 1.22wmf3 and rebuild l10n cache | [production] | 
            
  | 18:03 | <notpeter> | deploying new frontend squid conf | [production] | 
            
  | 17:50 | <LeslieCarr> | restarting icinga-wm | [production] | 
            
  | 17:35 | <cmjohnson1> | removing asw-c1 for replacement | [production] | 
            
  | 17:29 | <reedy> | Started syncing Wikimedia installation... : test2wiki to 1.22wmf3 and rebuild l10n cache | [production] | 
            
  | 17:08 | <mutante> | fix NRPE checks on ms-be2, running as "weird user id" bug | [production] | 
            
  | 17:04 | <mutante> | removing reinstalled ms-be2 from ssh_known_hosts with ssh-keygen and then making it world-readable again manually so deployers don't get errors :p | [production] | 
            
  | 17:04 | <csteipp> | synchronized php-1.22wmf2/includes/ | [production] | 
            
  | 17:04 | <reedy> | synchronized live-1.5 | [production] | 
            
  | 17:01 | <mutante> | attempt to fix NRPE checks on ms-be9, restart as correct user | [production] | 
            
  | 16:49 | <reedy> | synchronized docroot | [production] | 
            
  | 16:44 | <reedy> | synchronized php-1.22wmf3/  'initial sync of 1.22wmf3 files' | [production] | 
            
  | 03:25 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Mon Apr 29 03:25:35 UTC 2013 | [production] | 
            
  | 02:12 | <LocalisationUpdate> | completed (1.22wmf2) at Mon Apr 29 02:12:33 UTC 2013 | [production] | 
            
  
    | 2013-04-26
      
      § | 
    
  | 23:57 | <mutante> | upgrading mysql packages on marmontel (blog) | [production] | 
            
  | 23:53 | <mutante> | make SSL on the Blog use SSLCACertificatePath and c_rehash certs on kaulen | [production] | 
            
  | 20:11 | <hashar> | killed a node process on gallium. Probably a daemon that has gone standalone when jenkins died | [production] | 
            
  | 19:18 | <hashar> | jenkins died again sorry :(  But this time I most probably found out the reason why it is slowed down: a web spider attempt to crawl all our builds! | [production] | 
            
  | 18:03 | <mwalker> | Upgraded fundraising civicrm install from a3db312836b6ac004694f831071b1c7d00560fbf to a65fbcec0dbb5bb79a97aca220a998e56d08b067 | [production] | 
            
  | 13:49 | <mark> | Rebooting hooft | [production] | 
            
  | 13:47 | <mark> | dist upgrade of hooft to precise | [production] | 
            
  | 13:47 | <mark> | Moved udpmcast from hooft to nescio | [production] | 
            
  | 13:15 | <^demon> | gerrit replication freaked out, restarting | [production] | 
            
  | 06:37 | <hashar> | Restarted Jenkins. It was stuck in some while loop doing tons of stat() on unexisting file :(   Would be up in roughly 40 minutes :( | [production] | 
            
  | 03:45 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Fri Apr 26 03:45:44 UTC 2013 | [production] | 
            
  | 02:12 | <LocalisationUpdate> | completed (1.22wmf2) at Fri Apr 26 02:12:38 UTC 2013 | [production] |