| 2016-03-16
      
      ยง | 
    
  | 19:42 | <mutante> | rcs1001 - starting redis-server | [production] | 
            
  | 19:32 | <twentyafterfour> | Deploying 1.27.0-wmf.17 to group1 wikis | [production] | 
            
  | 19:31 | <godog> | restart pybal on lvs1005 T130143 | [production] | 
            
  | 19:18 | <legoktm> | created urlshortcodes table on wikishared db for UrlShortener | [production] | 
            
  | 19:16 | <legoktm@tin> | Synchronized wmf-config/CommonSettings.php: Configure UrlShortener extension in read-only mode (2/2) (duration: 00m 26s) | [production] | 
            
  | 19:15 | <legoktm@tin> | Synchronized wmf-config/InitialiseSettings.php: Configure UrlShortener extension in read-only mode (1/2) (duration: 00m 28s) | [production] | 
            
  | 19:14 | <aaron@tin> | Synchronized php-1.27.0-wmf.17/includes/jobqueue/Job.php: 3da38ce (duration: 00m 37s) | [production] | 
            
  | 19:12 | <legoktm@tin> | Finished scap: Building l10n cache for UrlShortener - T108557 (try #2) (duration: 63m 05s) | [production] | 
            
  | 19:11 | <gehel> | regenerating puppet SSL certificates for elasticsearch codfw cluster | [production] | 
            
  | 19:08 | <gehel> | restarting elasticsearch server elastic1002.eqiad.wmnet | [production] | 
            
  | 19:02 | <volans> | restarted mysql on dbstore2002 and all replicas except x1, still investigating T130128 | [production] | 
            
  | 19:01 | <godog> | restart pybal on lvs1011 T130143 | [production] | 
            
  | 18:20 | <mutante> | repooling rcs1001 | [production] | 
            
  | 18:18 | <gehel> | restarting elasticsearch server elastic1001.eqiad.wmnet | [production] | 
            
  | 18:14 | <mutante> | rebooting rcs1001 | [production] | 
            
  | 18:10 | <mutante> | depool rcs1001 | [production] | 
            
  | 18:09 | <legoktm@tin> | Started scap: Building l10n cache for UrlShortener - T108557 (try #2) | [production] | 
            
  | 18:09 | <legoktm> | deleting /var/lib/l10nupdate/caches/cache-1.27.0-wmf.12 on tin to free up some space | [production] | 
            
  | 18:05 | <mutante> | repooling rcs1002 | [production] | 
            
  | 17:59 | <mutante> | rcs1002 - traffic graph flat in ganglia, reboot | [production] | 
            
  | 17:57 | <mutante> | rcs1002 - the last message was about 1002 | [production] | 
            
  | 17:56 | <mutante> | rcs1001 - depool from rcstream service | [production] | 
            
  | 17:55 | <legoktm@tin> | scap aborted: Building l10n cache for UrlShortener - T108557 (duration: 02m 51s) | [production] | 
            
  | 17:52 | <legoktm@tin> | Started scap: Building l10n cache for UrlShortener - T108557 | [production] | 
            
  | 17:43 | <ebernhardson@tin> | Synchronized wmf-config/InitialiseSettings.php: Enable completion suggester as default on ja, zh, pl, ar and nlwiki (duration: 00m 29s) | [production] | 
            
  | 17:28 | <ebernhardson@tin> | Synchronized wmf-config/InitialiseSettings.php: Enable completion suggester as default on ru, fr, pt and itwiki (duration: 00m 28s) | [production] | 
            
  | 17:10 | <ebernhardson@tin> | Synchronized wmf-config/InitialiseSettings.php: Enable completion suggester as default on eswiki (duration: 00m 25s) | [production] | 
            
  | 17:09 | <krinkle@tin> | Synchronized php-1.27.0-wmf.17/extensions/Kartographer/extension.json: (no message) (duration: 00m 25s) | [production] | 
            
  | 16:46 | <gehel> | restarting elasticsearch server elastic2024.codfw.wmnet | [production] | 
            
  | 16:36 | <ebernhardson@tin> | Synchronized wmf-config/InitialiseSettings.php: Enable completion suggester as default on enwiki (duration: 00m 30s) | [production] | 
            
  | 16:32 | <krinkle@tin> | Synchronized wmf-config/CommonSettings-labs.php: (no message) (duration: 00m 31s) | [production] | 
            
  | 16:11 | <ebernhardson@tin> | Synchronized wmf-config/InitialiseSettings.php: Enable completion suggester as default on dewiki (duration: 00m 44s) | [production] | 
            
  | 16:11 | <moritzm> | upgraded exim packages on magnesium to the version from USN 2933, without it add_environment can't be set exim failed to start | [production] | 
            
  | 16:06 | <moritzm> | upgraded remaining exim binary packages on krypton to 4.84-8+deb8u2 (only exim4-daemon-heavy was at the version, resulting in the new add_environment variable being unknown and exim failing to start) | [production] | 
            
  | 16:02 | <gehel> | restarting elasticsearch server elastic2023.codfw.wmnet | [production] | 
            
  | 15:25 | <moritzm> | rebooting graphite1001 for kernel upgrade | [production] | 
            
  | 15:16 | <moritzm> | rebooting graphite2001 for kernel upgrade | [production] | 
            
  | 15:09 | <volans> | restarting mysql on dbstore2002 Aria engine crashed | [production] | 
            
  | 15:08 | <moritzm> | rebooting lithium for kernel upgrade | [production] | 
            
  | 15:07 | <jzerebecki@tin> | Synchronized wmf-config/InitialiseSettings.php: Whitelist feeds on mediawiki.org 622c186b2712c923fbcc48c27f65ebf396176f3e T127176 (duration: 00m 31s) | [production] | 
            
  | 14:52 | <volans> | Reimporting table bnwikisource.shorturls into dbstore2002 as InnoDB | [production] | 
            
  | 14:51 | <gehel> | restarting elasticsearch server elastic2022.codfw.wmnet | [production] | 
            
  | 14:36 | <volans> | REPAIR NO_WRITE_TO_BINLOG TABLE bnwikisource.shorturls replica failed for corrupted table | [production] | 
            
  | 14:18 | <papaul> | labstore200[3-4}] - signing puppet certs, salt-key, initial run | [production] | 
            
  | 14:06 | <papaul> | wasat signing puppet certs, salt-key, initial run | [production] | 
            
  | 14:06 | <gehel> | restarting elasticsearch server elastic2021.codfw.wmnet | [production] | 
            
  | 14:03 | <chasemp> | cleanup snapshots on labstore1001 | [production] | 
            
  | 14:02 | <volans> | Reimaging db2008 to jessie T130098 | [production] | 
            
  | 13:23 | <gehel> | restarting elasticsearch server elastic2020.codfw.wmnet | [production] | 
            
  | 13:15 | <elukey> | puppet re-enabled on analytics1027 | [production] |