| 
      
        2014-02-06
      
      §
     | 
  
    
  | 11:03 | 
  <springle> | 
  synchronized wmf-config/db-eqiad.php  's2 increase db1024 load after warm up' | 
  [production] | 
            
  | 07:29 | 
  <Ryan_Lane> | 
  redeploying parsoid/deploy on wtp* | 
  [production] | 
            
  | 07:01 | 
  <springle> | 
  xtrabackup clone db1018 to db1009 (take #2) | 
  [production] | 
            
  | 06:48 | 
  <springle> | 
  synchronized wmf-config/db-eqiad.php  's2 pool db1024 warm up' | 
  [production] | 
            
  | 05:48 | 
  <ori> | 
  varnish on cp1054: CPU wait spiked at 05:27. dmesg|tail: XFS: possible memory allocation deadlock in kmem_alloc. not investigating further. | 
  [production] | 
            
  | 05:45 | 
  <ori> | 
  finished scap: no new code. testing scap changes. (again.) (duration: 05m 00s) | 
  [production] | 
            
  | 05:40 | 
  <ori> | 
  started scap: no new code. testing scap changes. (again.) | 
  [production] | 
            
  | 05:15 | 
  <springle> | 
  restart labsdb1003 mariadb instances | 
  [production] | 
            
  | 05:09 | 
  <Ryan_Lane> | 
  scratch that, will redploy parsoid/deploy in about an hour | 
  [production] | 
            
  | 05:08 | 
  <Ryan_Lane> | 
  redeploying parsoid/deploy on wtp* | 
  [production] | 
            
  | 04:28 | 
  <ori> | 
  finished scap: no new code. testing scap changes. (duration: 04m 35s) | 
  [production] | 
            
  | 04:24 | 
  <ori> | 
  started scap: no new code. testing scap changes. | 
  [production] | 
            
  | 03:22 | 
  <LocalisationUpdate> | 
  ResourceLoader cache refresh completed at 2014-02-06 03:22:26+00:00 | 
  [production] | 
            
  | 03:10 | 
  <springle> | 
  xtrabackup clone db1018 to db1024 | 
  [production] | 
            
  | 02:45 | 
  <springle> | 
  xtrabackup clone db1034 to db1009 | 
  [production] | 
            
  | 02:44 | 
  <LocalisationUpdate> | 
  completed (1.23wmf11) at 2014-02-06 02:44:10+00:00 | 
  [production] | 
            
  | 02:23 | 
  <springle> | 
  synchronized wmf-config/db-eqiad.php  's2 depool db1009 schema changes' | 
  [production] | 
            
  | 02:21 | 
  <LocalisationUpdate> | 
  completed (1.23wmf12) at 2014-02-06 02:21:47+00:00 | 
  [production] | 
            
  | 02:01 | 
  <springle> | 
  synchronized wmf-config/db-eqiad.php  's2 repool db1034 warm up' | 
  [production] | 
            
  | 01:06 | 
  <mutante> | 
  apt-get remove libnet-stomp-perl on neon, i just removed that from puppet but didn't think it should stay in as an "absent" package forever | 
  [production] | 
            
  | 01:01 | 
  <ori> | 
  finished scap: no-diff scap to test script changes (duration: 02m 34s) | 
  [production] | 
            
  | 00:58 | 
  <ori> | 
  started scap: no-diff scap to test script changes | 
  [production] | 
            
  | 00:44 | 
  <ebernhardson> | 
  finished deploying Special:Notifications fix to Echo and Flow | 
  [production] | 
            
  | 00:44 | 
  <ebernhardson> | 
  synchronized php-1.23wmf11/extensions/Flow/  'Update flow for Special:Notifications fix' | 
  [production] | 
            
  | 00:43 | 
  <ebernhardson> | 
  synchronized php-1.23wmf11/extensions/Echo/  'Update echo for Special:Notifications fix' | 
  [production] | 
            
  | 00:38 | 
  <ebernhardson> | 
  synchronized php-1.23wmf12/extensions/Flow/  'Update flow for Special:Notifications fix' | 
  [production] | 
            
  | 00:37 | 
  <ebernhardson> | 
  synchronized php-1.23wmf12/extensions/Echo/  'Update echo for Special:Notifications fix' | 
  [production] | 
            
  | 00:04 | 
  <ori> | 
  finished scap: (no message) (duration: 29m 26s) | 
  [production] | 
            
  | 00:04 | 
  <ori> | 
  rebuilt wikiversions.cdb and synchronized wikiversions files: | 
  [production] | 
            
  
    | 
      
        2014-02-05
      
      §
     | 
  
    
  | 23:34 | 
  <ori> | 
  started scap: (no message) | 
  [production] | 
            
  | 22:11 | 
  <ottomata> | 
  stopping puppet on analytics1021.  Trying to get it to catch up on replica lag | 
  [production] | 
            
  | 20:13 | 
  <mwalker> | 
  updated fundraising civicrm from ec1e1fc75193ac8c0e6237fc0b80f58b3f8d55fc to 309a69545b4c89b323bffc500f71bdee82c35e42 for logging updates | 
  [production] | 
            
  | 18:57 | 
  <yurik> | 
  synchronized php-1.23wmf12/extensions/ZeroRatedMobileAccess/ | 
  [production] | 
            
  | 18:49 | 
  <yurik> | 
  synchronized php-1.23wmf11/extensions/ZeroRatedMobileAccess/ | 
  [production] | 
            
  | 18:38 | 
  <yurik> | 
  synchronized docroot/bits/WikipediaMobileFirefoxOS | 
  [production] | 
            
  | 14:11 | 
  <paravoid> | 
  ms-be1002/sdd: megacli -DiscardPreservedCache, -CfgEachDskRaid0, puppet run | 
  [production] | 
            
  | 13:57 | 
  <paravoid> | 
  remove XO-Level3 avoided as-path from cr1/2-eqiad despite no ticket reply; seems to work now | 
  [production] | 
            
  | 13:57 | 
  <hashar> | 
  compressing Jenkins console logs on gallium.wikimedia.org using gzip -9 | 
  [production] | 
            
  | 08:47 | 
  <ori> | 
  finished scap: (no message) (duration: 03m 31s) | 
  [production] | 
            
  | 08:47 | 
  <ori> | 
  rebuilt wikiversions.cdb and synchronized wikiversions files: | 
  [production] | 
            
  | 08:44 | 
  <ori> | 
  started scap: (no message) | 
  [production] | 
            
  | 08:17 | 
  <Ryan_Lane> | 
  restart parsoid on wtp1012 | 
  [production] | 
            
  | 08:14 | 
  <Ryan_Lane> | 
  stopping parsoid on wtp1012 shortly | 
  [production] | 
            
  | 07:53 | 
  <ori> | 
  rebuilt wikiversions.cdb and synchronized wikiversions files: | 
  [production] | 
            
  | 07:51 | 
  <ori> | 
  rebuilt wikiversions.cdb and synchronized wikiversions files: | 
  [production] | 
            
  | 07:15 | 
  <ori> | 
  rebuilt wikiversions.cdb and synchronized wikiversions files: | 
  [production] | 
            
  | 06:59 | 
  <ori> | 
  finished scap: (no message) (duration: 00m 35s) | 
  [production] | 
            
  | 06:59 | 
  <ori> | 
  rebuilt wikiversions.cdb and synchronized wikiversions files: | 
  [production] | 
            
  | 06:59 | 
  <ori> | 
  started scap: (no message) | 
  [production] | 
            
  | 05:52 | 
  <springle> | 
  synchronized wmf-config/db-pmtpa.php  'update pmtpa config for s1 master change' | 
  [production] |