| 
      
        2016-11-08
      
      ยง
     | 
  
    
  | 17:19 | 
  <ema> | 
  upgrade finished -> cache_text codfw to varnish 4.1.3-1wm3 T150247 | 
  [production] | 
            
  | 17:15 | 
  <reedy@tin> | 
  Synchronized wmf-config/CommonSettings-labs.php: Add PageViewInfo, Remove dupe OATHAuth config (duration: 00m 47s) | 
  [production] | 
            
  | 17:13 | 
  <reedy@tin> | 
  Synchronized wmf-config/InitialiseSettings-labs.php: Add PageViewInfo (duration: 00m 46s) | 
  [production] | 
            
  | 17:12 | 
  <reedy@tin> | 
  Synchronized wmf-config/extension-list-labs: Add PageViewInfo (duration: 00m 46s) | 
  [production] | 
            
  | 16:31 | 
  <apergos> | 
  rolling restart of mw1204-1208 for new kernel | 
  [production] | 
            
  | 16:24 | 
  <jynus> | 
  performing schema change templatelinks on db1080 T139090 | 
  [production] | 
            
  | 16:21 | 
  <Krinkle> | 
  mwscript --deleteEqualMessages.php --wiki kkwiki (T45917) | 
  [production] | 
            
  | 16:18 | 
  <ema> | 
  upgrading cache_text codfw to varnish 4.1.3-1wm3 T150247 | 
  [production] | 
            
  | 16:11 | 
  <jynus@tin> | 
  Synchronized wmf-config/db-eqiad.php: Pool db1052; depool db1080; reorganize trafic weight for s1 -second try (duration: 00m 46s) | 
  [production] | 
            
  | 16:04 | 
  <jynus@tin> | 
  Synchronized wmf-config/db-eqiad.php: Pool db1052; depool db1080; reorganize trafic weight for s1 (duration: 00m 46s) | 
  [production] | 
            
  | 15:23 | 
  <hashar@tin> | 
  Synchronized php-1.29.0-wmf.1/extensions/Kartographer/modules/maplink/maplink.js: Search .mw-body instead of #content to support all the skins - T150148 (duration: 00m 47s) | 
  [production] | 
            
  | 14:09 | 
  <moritzm> | 
  rebooting chromium for kernel update | 
  [production] | 
            
  | 13:58 | 
  <hashar> | 
  European SWAT on hold while some memcached/elasticsearch issues are being figured out | 
  [production] | 
            
  | 13:42 | 
  <apergos> | 
  deferring reboots of mw1204-1216 and mw1170-1188 for a while | 
  [production] | 
            
  | 13:06 | 
  <moritzm> | 
  rolling reboot of restbase-test for kernel update | 
  [production] | 
            
  | 12:52 | 
  <ema> | 
  upgrading pinkunicorn to varnish 4.1.3-1wm3 T150247 | 
  [production] | 
            
  | 12:41 | 
  <moritzm> | 
  restarted ntp on mw1194, stuck in XFAC state | 
  [production] | 
            
  | 12:39 | 
  <moritzm> | 
  rebooting iron for kernel update | 
  [production] | 
            
  | 12:28 | 
  <moritzm> | 
  restarted ntp on mw1166, stuck in XFAC state | 
  [production] | 
            
  | 12:24 | 
  <moritzm> | 
  depooling/rebooting/repooling scb1002 for kernel update | 
  [production] | 
            
  | 12:23 | 
  <moritzm> | 
  restarted ntp on hafnium, stuck in XFAC state | 
  [production] | 
            
  | 12:12 | 
  <moritzm> | 
  depooling/rebooting/repooling scb1001 for kernel update | 
  [production] | 
            
  | 12:11 | 
  <jynus@tin> | 
  Synchronized wmf-config/db-eqiad.php: Pool back db1051 and api servers to high load after hw issues (duration: 02m 45s) | 
  [production] | 
            
  | 11:59 | 
  <apergos> | 
  rolling restart of mw1170-1216 for new kernel | 
  [production] | 
            
  | 11:54 | 
  <apergos> | 
  restart of mw1240, 1253 for new kernel | 
  [production] | 
            
  | 11:44 | 
  <moritzm> | 
  rebooting logstash1001 for kernel update | 
  [production] | 
            
  | 11:42 | 
  <moritzm> | 
  rearmed keyholder on mira | 
  [production] | 
            
  | 11:38 | 
  <apergos> | 
  rolling reboot of mw1161, mw1163-1169 for new kernel | 
  [production] | 
            
  | 11:37 | 
  <jynus> | 
  running schema change on db1045 (pagelinks) T139090 | 
  [production] | 
            
  | 11:30 | 
  <moritzm> | 
  rebooting mira for kernel update | 
  [production] | 
            
  | 11:19 | 
  <apergos> | 
  rolling restart of mw2080-2085 for new kernel | 
  [production] | 
            
  | 11:15 | 
  <jynus> | 
  running schema change on db2070 (pagelinks) T139090 | 
  [production] | 
            
  | 11:02 | 
  <mark> | 
  Activated cr2-eqiad bgp group IX4 | 
  [production] | 
            
  | 10:53 | 
  <apergos> | 
  rolling reboot of mw2090 - mw2096 for new kernel | 
  [production] | 
            
  | 10:41 | 
  <moritzm> | 
  rearmed keyholder on tin | 
  [production] | 
            
  | 10:39 | 
  <apergos> | 
  rebooting mw2086 - mw2089 for new kernel | 
  [production] | 
            
  | 10:36 | 
  <jynus> | 
  rebooting and upgrading db2012 | 
  [production] | 
            
  | 10:33 | 
  <moritzm> | 
  rebooting tin for kernel update | 
  [production] | 
            
  | 10:23 | 
  <moritzm> | 
  restarted ntp on mw2075, stuck in XFAC state | 
  [production] | 
            
  | 10:22 | 
  <moritzm> | 
  rebooting hafnium for kernel update | 
  [production] | 
            
  | 10:08 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Depool db1059 - T149079 T147305 (duration: 00m 57s) | 
  [production] | 
            
  | 10:06 | 
  <moritzm> | 
  rebooting rhenium for kernel update | 
  [production] | 
            
  | 09:59 | 
  <moritzm> | 
  rebooting oxygen for kernel update | 
  [production] | 
            
  | 09:57 | 
  <apergos> | 
  rebooting mw2075 - mw2079 for new kernel | 
  [production] | 
            
  | 09:49 | 
  <moritzm> | 
  rebooting install2001 for kernel update | 
  [production] | 
            
  | 09:42 | 
  <moritzm> | 
  rebooting bast2001 for kernel update | 
  [production] | 
            
  | 09:24 | 
  <moritzm> | 
  rebooting graphite1002 for kernel update | 
  [production] | 
            
  | 08:44 | 
  <marostegui> | 
  Deploy schema change s4 commonswiki.revision table - T147305 | 
  [production] | 
            
  | 08:21 | 
  <moritzm> | 
  rolling reboot of swift backend servers in esams for kernel update | 
  [production] | 
            
  | 08:09 | 
  <moritzm> | 
  rolling reboot of parsoid in eqiad for kernel update | 
  [production] |