| 
      
        2020-02-27
      
      ยง
     | 
  
    
  | 10:54 | 
  <vgutierrez> | 
  replacing lvs2006 with lvs2010 - T196560 T245984 | 
  [production] | 
            
  | 09:35 | 
  <jynus> | 
  upgrade and restart db1084 T246323 | 
  [production] | 
            
  | 09:03 | 
  <jynus@cumin1001> | 
  dbctl commit (dc=all): 'Depool db1098 (s6 & s7)', diff saved to https://phabricator.wikimedia.org/P10536 and previous config saved to /var/cache/conftool/dbconfig/20200227-090344-jynus.json | 
  [production] | 
            
  | 08:26 | 
  <jynus> | 
  killed SpecialFewestRevisions::reallyDoQuery long running query on db1101:s8, causing lag | 
  [production] | 
            
  | 08:14 | 
  <jynus@cumin1001> | 
  dbctl commit (dc=all): 'Depool db1098 at 50%', diff saved to https://phabricator.wikimedia.org/P10535 and previous config saved to /var/cache/conftool/dbconfig/20200227-081449-jynus.json | 
  [production] | 
            
  | 03:52 | 
  <pt1979@cumin2001> | 
  END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) | 
  [production] | 
            
  | 03:50 | 
  <pt1979@cumin2001> | 
  START - Cookbook sre.hosts.downtime | 
  [production] | 
            
  | 03:31 | 
  <pt1979@cumin2001> | 
  END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) | 
  [production] | 
            
  | 03:28 | 
  <pt1979@cumin2001> | 
  END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) | 
  [production] | 
            
  | 03:27 | 
  <pt1979@cumin2001> | 
  START - Cookbook sre.hosts.downtime | 
  [production] | 
            
  | 03:26 | 
  <pt1979@cumin2001> | 
  START - Cookbook sre.hosts.downtime | 
  [production] | 
            
  | 02:53 | 
  <pt1979@cumin2001> | 
  END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) | 
  [production] | 
            
  | 02:50 | 
  <pt1979@cumin2001> | 
  START - Cookbook sre.hosts.downtime | 
  [production] | 
            
  | 02:49 | 
  <pt1979@cumin2001> | 
  END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) | 
  [production] | 
            
  | 02:47 | 
  <pt1979@cumin2001> | 
  START - Cookbook sre.hosts.downtime | 
  [production] | 
            
  | 02:27 | 
  <pt1979@cumin2001> | 
  END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) | 
  [production] | 
            
  | 02:24 | 
  <pt1979@cumin2001> | 
  END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) | 
  [production] | 
            
  | 02:23 | 
  <pt1979@cumin2001> | 
  START - Cookbook sre.hosts.downtime | 
  [production] | 
            
  | 02:22 | 
  <pt1979@cumin2001> | 
  START - Cookbook sre.hosts.downtime | 
  [production] | 
            
  | 01:45 | 
  <pt1979@cumin2001> | 
  END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) | 
  [production] | 
            
  | 01:43 | 
  <pt1979@cumin2001> | 
  START - Cookbook sre.hosts.downtime | 
  [production] | 
            
  | 01:37 | 
  <pt1979@cumin2001> | 
  END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) | 
  [production] | 
            
  | 01:34 | 
  <pt1979@cumin2001> | 
  START - Cookbook sre.hosts.downtime | 
  [production] | 
            
  | 01:27 | 
  <XioNoX> | 
  re-enable BGP to telia in esams | 
  [production] | 
            
  | 01:13 | 
  <pt1979@cumin2001> | 
  END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) | 
  [production] | 
            
  | 01:10 | 
  <pt1979@cumin2001> | 
  START - Cookbook sre.hosts.downtime | 
  [production] | 
            
  | 00:56 | 
  <cdanis> | 
  repool esams ๐ ๐ | 
  [production] | 
            
  | 00:52 | 
  <pt1979@cumin2001> | 
  END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) | 
  [production] | 
            
  | 00:49 | 
  <pt1979@cumin2001> | 
  START - Cookbook sre.hosts.downtime | 
  [production] | 
            
  | 00:42 | 
  <pt1979@cumin2001> | 
  END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) | 
  [production] | 
            
  | 00:39 | 
  <pt1979@cumin2001> | 
  START - Cookbook sre.hosts.downtime | 
  [production] | 
            
  | 00:21 | 
  <jforrester@deploy1001> | 
  Synchronized w/extract2.php: T239975: Use Article::getPage()->getTouched(), not Article::getTouched (duration: 01m 04s) | 
  [production] | 
            
  | 00:17 | 
  <jforrester@deploy1001> | 
  Synchronized wmf-config/InitialiseSettings.php: Bonus sync for cache clearance (duration: 01m 04s) | 
  [production] | 
            
  | 00:15 | 
  <jforrester@deploy1001> | 
  Synchronized wmf-config/InitialiseSettings.php: T232140: Merge definition of wgLogos and wgLogo (duration: 01m 04s) | 
  [production] | 
            
  | 00:13 | 
  <jforrester@deploy1001> | 
  Synchronized wmf-config/CommonSettings.php: T232140: Stop setting wgLogoHD from wgLogos (duration: 01m 05s) | 
  [production] | 
            
  | 00:02 | 
  <jforrester@deploy1001> | 
  Synchronized wmf-config/InitialiseSettings.php: Bonus sync for cache clearance (duration: 01m 03s) | 
  [production] | 
            
  | 00:01 | 
  <jforrester@deploy1001> | 
  Synchronized wmf-config/InitialiseSettings.php: T246212 Stop setting wgULSLanguageDetection in IS, set in CS (duration: 01m 05s) | 
  [production] | 
            
  
    | 
      
        2020-02-26
      
      ยง
     | 
  
    
  | 23:59 | 
  <jforrester@deploy1001> | 
  Synchronized wmf-config/CommonSettings.php: T246212 Set wgULSLanguageDetection false in CS (duration: 01m 04s) | 
  [production] | 
            
  | 23:55 | 
  <jforrester@deploy1001> | 
  Synchronized wmf-config/InitialiseSettings.php: Bonus sync for cache clearance (duration: 01m 04s) | 
  [production] | 
            
  | 23:54 | 
  <James_F> | 
  jforrester@deploy1001 Synchronized wmf-config/InitialiseSettings.php: T246193 Stop setting wgAllowTitlesInSVG, never read (and this was default anyway) (duration: 01m 05s) | 
  [production] | 
            
  | 23:19 | 
  <pt1979@cumin2001> | 
  END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) | 
  [production] | 
            
  | 23:16 | 
  <pt1979@cumin2001> | 
  END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) | 
  [production] | 
            
  | 23:16 | 
  <pt1979@cumin2001> | 
  START - Cookbook sre.hosts.downtime | 
  [production] | 
            
  | 23:15 | 
  <pt1979@cumin2001> | 
  START - Cookbook sre.hosts.downtime | 
  [production] | 
            
  | 22:58 | 
  <dzahn@cumin1001> | 
  END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) | 
  [production] | 
            
  | 22:58 | 
  <dzahn@cumin1001> | 
  START - Cookbook sre.hosts.downtime | 
  [production] | 
            
  | 22:58 | 
  <dzahn@cumin1001> | 
  END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) | 
  [production] | 
            
  | 22:58 | 
  <dzahn@cumin1001> | 
  START - Cookbook sre.hosts.downtime | 
  [production] | 
            
  | 22:51 | 
  <pt1979@cumin2001> | 
  END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) | 
  [production] | 
            
  | 22:49 | 
  <pt1979@cumin2001> | 
  END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) | 
  [production] |