| 
      
        2014-03-12
      
      §
     | 
  
    
  | 23:49 | 
  <mwalker> | 
  synchronized php-1.23wmf16/extensions/CentralNotice/ | 
  [production] | 
            
  | 23:21 | 
  <mwalker> | 
  synchronized php-1.23wmf17/extensions/CentralNotice | 
  [production] | 
            
  | 23:20 | 
  <mwalker> | 
  synchronized php-1.23wmf16/extensions/CentralNotice | 
  [production] | 
            
  | 21:57 | 
  <bd808> | 
  Finished scap: another no-diff scap to test script changes (duration: 03m 25s) | 
  [production] | 
            
  | 21:54 | 
  <bd808> | 
  Started scap: another no-diff scap to test script changes | 
  [production] | 
            
  | 21:54 | 
  <ottomata> | 
  initiated controlled shutdown of an21, promoting an22 to leader of all partitions | 
  [production] | 
            
  | 21:51 | 
  <bd808> | 
  Finished scap: no-diff scap to test script changes (duration: 21m 42s) | 
  [production] | 
            
  | 21:30 | 
  <bd808> | 
  Started scap: no-diff scap to test script changes | 
  [production] | 
            
  | 21:14 | 
  <mutante> | 
  welcome to root, Chase, added key to root-auth-keys | 
  [production] | 
            
  | 20:05 | 
  <Coren> | 
  restarting parsoids as requested by gwicke | 
  [production] | 
            
  | 20:02 | 
  <gwicke> | 
  deployed Parsoid 004c7acc with deploy f97820a2; restart todo | 
  [production] | 
            
  | 19:12 | 
  <yurik> | 
  synchronized php-1.23wmf16/extensions/ZeroRatedMobileAccess/ | 
  [production] | 
            
  | 19:11 | 
  <yurik> | 
  synchronized php-1.23wmf17/extensions/ZeroRatedMobileAccess/ | 
  [production] | 
            
  | 19:06 | 
  <mutante> | 
  graceful apache on gallium | 
  [production] | 
            
  | 17:36 | 
  <yurik> | 
  synchronized php-1.23wmf17/extensions/ZeroRatedMobileAccess/ | 
  [production] | 
            
  | 17:27 | 
  <yurik> | 
  synchronized php-1.23wmf16/extensions/ZeroRatedMobileAccess/ | 
  [production] | 
            
  | 17:17 | 
  <awight> | 
  updated crm from f0837090e918b7016c07cb366b33c8d8f0d1c661 to 445f1f0ce489e65bf3765a23403f5b52b77f23cf | 
  [production] | 
            
  | 17:02 | 
  <reedy> | 
  synchronized wmf-config/  'noop' | 
  [production] | 
            
  | 16:56 | 
  <reedy> | 
  synchronized docroot and w | 
  [production] | 
            
  | 16:56 | 
  <reedy> | 
  synchronized wmf-config/  'I6f1f3f8af5b97aa0e537fbae308ce27b28071894' | 
  [production] | 
            
  | 16:10 | 
  <ottomata> | 
  initiating controlled shutdown of analytics1021 kafka broker to do some load testing and also fix runtime java version | 
  [production] | 
            
  | 15:56 | 
  <cmjohnson1> | 
  ms-be1005 going down to fix mgmt  | 
  [production] | 
            
  | 15:06 | 
  <hoo> | 
  syncing to mw120[1-3] failed | 
  [production] | 
            
  | 14:58 | 
  <hoo> | 
  synchronized php-1.23wmf17/extensions/CentralAuth/  'Fix global account deletion' | 
  [production] | 
            
  | 14:38 | 
  <springle> | 
  synchronized wmf-config/db-eqiad.php  's2 db1063 full steam' | 
  [production] | 
            
  | 14:24 | 
  <paravoid> | 
  deploying new swift ring @ eqiad, setting weight from 100 to 2000 on all disks | 
  [production] | 
            
  | 13:01 | 
  <cmjohnson1> | 
  shutting down and relocating mw1201, mw1202, mw1203 to d5-eqiad | 
  [production] | 
            
  | 10:41 | 
  <maxsem> | 
  synchronized wmf-config/mobile.php  'https://gerrit.wikimedia.org/r/118227' | 
  [production] | 
            
  | 10:17 | 
  <springle> | 
  started s4 dump for toolserver on db72 /a | 
  [production] | 
            
  | 09:52 | 
  <springle> | 
  synchronized wmf-config/db-eqiad.php  's2 pool db1063, warm up' | 
  [production] | 
            
  | 09:45 | 
  <hashar> | 
  rerestarting Jenkins | 
  [production] | 
            
  | 09:16 | 
  <hashar> | 
  kill -9 of Jenkins since it is unresponsive | 
  [production] | 
            
  | 09:13 | 
  <hashar> | 
  restarting Jenkins | 
  [production] | 
            
  | 09:12 | 
  <hashar> | 
  Jenkins broken again! Good morning. | 
  [production] | 
            
  | 06:18 | 
  <springle> | 
  xtrabackup clone db1018 to db1063 | 
  [production] | 
            
  | 05:38 | 
  <springle> | 
  synchronized wmf-config/db-eqiad.php  's1 pool db1061, warm up' | 
  [production] | 
            
  | 04:52 | 
  <springle> | 
  synchronized wmf-config/db-eqiad.php  's1 db1051 warm up' | 
  [production] | 
            
  | 02:43 | 
  <LocalisationUpdate> | 
  ResourceLoader cache refresh completed at Wed Mar 12 02:42:59 UTC 2014 (duration 42m 58s) | 
  [production] | 
            
  | 02:32 | 
  <springle> | 
  synchronized wmf-config/db-eqiad.php  's1 drop load during xtrabackup clone db1051 to db1061' | 
  [production] | 
            
  | 02:18 | 
  <LocalisationUpdate> | 
  completed (1.23wmf17) at 2014-03-12 02:18:12+00:00 | 
  [production] | 
            
  | 02:10 | 
  <LocalisationUpdate> | 
  completed (1.23wmf16) at 2014-03-12 02:10:25+00:00 | 
  [production] | 
            
  
    | 
      
        2014-03-11
      
      §
     | 
  
    
  | 19:36 | 
  <mutante> | 
  re-deleting salt keys for pmtpa appservers | 
  [production] | 
            
  | 19:03 | 
  <mutante> | 
  shut down mw86-mw125 (sdtpa row A, A5) | 
  [production] | 
            
  | 18:52 | 
  <mutante> | 
  shut down mw28-mw57 | 
  [production] | 
            
  | 18:35 | 
  <bd808> | 
  purged l10n cache for 1.23wmf15 | 
  [production] | 
            
  | 18:33 | 
  <bd808> | 
  purged l10n cache for 1.23wmf14 | 
  [production] | 
            
  | 18:08 | 
  <bd808> | 
  rebuilt wikiversions.cdb and synchronized wikiversions files: group1 to 1.23wmf17 | 
  [production] | 
            
  | 18:03 | 
  <bd808> | 
  updated /a/common to {{Gerrit|I06d07cc3e}}: beta: disable memcached accross datacenters | 
  [production] | 
            
  | 17:31 | 
  <mutante> | 
  shut down srv284-srv301 (sdtpa row B, B5) | 
  [production] | 
            
  | 16:08 | 
  <cmjohnson1> | 
  attempting to fix  ps1-b5 and ps1-b6 | 
  [production] |