| 2012-10-03
      
      § | 
    
  | 22:47 | <mutante> | reinstalling analytics1007 | [production] | 
            
  | 21:38 | <mutante> | deleting education-ar mailman list per [[bugzilla:40749|bug 40749]] | [production] | 
            
  | 20:55 | <logmsgbot_> | dzahn synchronized wmf-config/throttle.php  'IP cap lift for Grace Hopper event, [[bugzilla:40669|bug 40669]]' | [production] | 
            
  | 20:18 | <logmsgbot_> | reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else non wikipedia to 1.21wmf1 | [production] | 
            
  | 20:15 | <mutante> | powercycling analytics1007 | [production] | 
            
  | 20:15 | <logmsgbot_> | reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikimedia and wiktionary to 1.21wmf1 | [production] | 
            
  | 20:13 | <logmsgbot_> | reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks and wikinews to 1.21wmf1 | [production] | 
            
  | 20:12 | <logmsgbot_> | reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity and wikiquote to 1.21wmf1 | [production] | 
            
  | 20:09 | <logmsgbot_> | reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special, private and closed to 1.21wmf1 | [production] | 
            
  | 19:33 | <logmsgbot_> | preilly synchronized php-1.21wmf1/extensions/MobileFrontend  'update for landing page' | [production] | 
            
  | 19:32 | <logmsgbot_> | preilly synchronized php-1.20wmf12/extensions/MobileFrontend  'update for landing page' | [production] | 
            
  | 19:30 | <logmsgbot_> | preilly synchronized php-1.21wmf1/extensions/ZeroRatedMobileAccess  'update for landing page' | [production] | 
            
  | 19:28 | <logmsgbot_> | preilly synchronized php-1.20wmf12/extensions/ZeroRatedMobileAccess  'update for landing page' | [production] | 
            
  | 15:43 | <Reedy> | Close strategywiki, zawiktionary, zh_min_nanwikibooks | [production] | 
            
  | 15:41 | <logmsgbot_> | reedy synchronized wmf-config/InitialiseSettings.php  'WikiLove on itwikiquote' | [production] | 
            
  | 15:36 | <Reedy> | Created wikilove tables to itwikiquote | [production] | 
            
  | 15:18 | <Jeff_Green> | test-deploying SPF DNS records | [production] | 
            
  | 09:49 | <mark> | Doubled number of inodes on nas1-a:originals from 31876689 to 63753378 | [production] | 
            
  | 08:38 | <apergos> | stopped ms7 to ms8 replication | [production] | 
            
  | 08:18 | <apergos> | rebooting ms8 to unstick deadlocked zfs recv and zfs list processes | [production] | 
            
  | 07:20 | <Tim> | upgrading luasandbox on all apaches to version 1.1-1 | [production] | 
            
  | 06:28 | <apergos> | restarted lucene search on search1016 | [production] | 
            
  | 05:22 | <logmsgbot_> | tstarling synchronized php-1.21wmf1/cache/trusted-xff.cdb | [production] | 
            
  | 05:19 | <logmsgbot_> | tstarling synchronized php-1.20wmf12/cache/trusted-xff.cdb | [production] | 
            
  | 03:11 | <logmsgbot_> | tstarling synchronized wmf-config/db.php  'es10 back up' | [production] | 
            
  | 03:03 | <Tim> | on es10: started mysqld | [production] | 
            
  | 02:51 | <logmsgbot_> | LocalisationUpdate completed (1.21wmf1) at Wed Oct  3 02:51:35 UTC 2012 | [production] | 
            
  | 02:45 | <Tim> | es10 not responding to ssh, rebooting | [production] | 
            
  | 02:43 | <logmsgbot_> | tstarling synchronized wmf-config/db.php | [production] | 
            
  | 02:32 | <logmsgbot_> | LocalisationUpdate completed (1.20wmf12) at Wed Oct  3 02:31:56 UTC 2012 | [production] | 
            
  | 02:09 | <logmsgbot_> | tstarling synchronized wmf-config/InitialiseSettings.php  'API log' | [production] | 
            
  | 01:35 | <logmsgbot_> | krinkle synchronized live-1.5 | [production] | 
            
  | 01:29 | <logmsgbot_> | tstarling synchronized live-1.5 | [production] | 
            
  | 01:25 | <Tim> | on srv247, srv270, srv271, srv286, mw69: killing more stuck apt-get processes, will run apt-get upgrade | [production] | 
            
  | 01:19 | <Tim> | on srv233: killed frozen apt-get, running since August 17, running apt-get upgrade | [production] | 
            
  | 01:17 | <logmsgbot_> | krinkle Finished syncing Wikimedia installation... : Icd721011b40bb8d2c20aefa8b359a3e45413a07f | [production] | 
            
  | 01:09 | <Tim> | upgrading wikimedia-task-appserver on all apaches | [production] | 
            
  | 01:06 | <Tim> | on brewster: updated wikimedia-task-appserver | [production] | 
            
  | 00:43 | <mutante> | moved logmsgbot to -operations channel | [production] | 
            
  | 00:34 | <krinkle> | Started syncing Wikimedia installation... : Icd721011b40bb8d2c20aefa8b359a3e45413a07f | [production] | 
            
  | 00:25 | <Reedy> | Restarted Jenkins | [production] | 
            
  
    | 2012-10-02
      
      § | 
    
  | 23:45 | <olivneh> | synchronized php-1.20wmf12/extensions/E3Experiments/experiments/openTask.js  'Updating event format for OpenTask funnel' | [production] | 
            
  | 23:44 | <reedy> | synchronized php-1.21wmf1/extensions/Collection/ | [production] | 
            
  | 23:19 | <Ryan_Lane> | deploying Damianz's OpenStackManager change for adding/removing bastion users on shell access on labsconsole | [production] | 
            
  | 16:17 | <asher> | synchronized wmf-config/db.php  'adding db64 to s3 in place of db25' | [production] | 
            
  | 16:02 | <mark> | Reclaimed space on brewster:/ - restarted squdi | [production] | 
            
  | 15:19 | <mark> | Sending ALL upload traffic to upload-lb.eqiad (Varnish) | [production] | 
            
  | 14:57 | <mark> | Sending traffic from India, Korea, China, Singapore, Thailand, Vietnam to upload-lb.eqiad (Varnish) | [production] |