| 
      
        2016-06-14
      
      ยง
     | 
  
    
  | 20:46 | 
  <gehel> | 
  adding new maps servers to LVS | 
  [production] | 
            
  | 20:09 | 
  <demon@tin> | 
  Finished scap: wikidata submodule update for wmf.6 (duration: 25m 51s) | 
  [production] | 
            
  | 19:43 | 
  <demon@tin> | 
  Started scap: wikidata submodule update for wmf.6 | 
  [production] | 
            
  | 19:30 | 
  <demon@tin> | 
  Finished scap: group0 to 1.28.0-wmf.6 (duration: 26m 43s) | 
  [production] | 
            
  | 19:03 | 
  <demon@tin> | 
  Started scap: group0 to 1.28.0-wmf.6 | 
  [production] | 
            
  | 18:56 | 
  <demon@tin> | 
  Purged l10n cache for 1.27.0-wmf.23 | 
  [production] | 
            
  | 18:54 | 
  <demon@tin> | 
  Purged l10n cache for 1.28.0-wmf.4 | 
  [production] | 
            
  | 18:54 | 
  <demon@tin> | 
  Purged l10n cache for 1.28.0-wmf.3 | 
  [production] | 
            
  | 18:54 | 
  <demon@tin> | 
  Purged l10n cache for 1.28.0-wmf.2 | 
  [production] | 
            
  | 18:53 | 
  <demon@tin> | 
  Purged l10n cache for 1.28.0-wmf.1 | 
  [production] | 
            
  | 17:22 | 
  <Dereckson> | 
  Run initSiteStats.php for arcwiki and htwiki (T137827) | 
  [production] | 
            
  | 16:48 | 
  <jynus@tin> | 
  Synchronized wmf-config/db-eqiad.php: Repool db1040, pool for the first time db1081, db1084, db1091 (duration: 00m 34s) | 
  [production] | 
            
  | 16:41 | 
  <godog> | 
  reimage ms-fe3002 with jessie T117972 | 
  [production] | 
            
  | 15:57 | 
  <yurik> | 
  deployed & restarted kartotherian (fixing spec.config tests) | 
  [production] | 
            
  | 15:54 | 
  <urandom> | 
  Restarting cassandra-metrics-collector on restbase1007 : T137304 | 
  [production] | 
            
  | 15:53 | 
  <thcipriani@tin> | 
  Synchronized wmf-config: SWAT: [[gerrit:291908|Beta: Enable Compact Language Links for new users]] (duration: 00m 31s) | 
  [production] | 
            
  | 15:41 | 
  <hashar@tin> | 
  scap aborted: testwiki to php-1.28.0-wmf.6 and rebuild l10n cache (duration: 01m 31s) | 
  [production] | 
            
  | 15:40 | 
  <hashar@tin> | 
  Started scap: testwiki to php-1.28.0-wmf.6 and rebuild l10n cache | 
  [production] | 
            
  | 15:35 | 
  <thcipriani@tin> | 
  Synchronized php-1.28.0-wmf.5/extensions/CentralAuth/includes/CentralAuthHooks.php: SWAT: [[gerrit:294318|Account for changed login process]] (duration: 00m 26s) | 
  [production] | 
            
  | 15:27 | 
  <thcipriani@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:294269|Add nonecho.dblist and echo.dblist]] PART III (duration: 00m 27s) | 
  [production] | 
            
  | 15:26 | 
  <thcipriani@tin> | 
  Synchronized wmf-config/CommonSettings.php: SWAT: [[gerrit:294269|Add nonecho.dblist and echo.dblist]] PART II (duration: 00m 26s) | 
  [production] | 
            
  | 15:26 | 
  <godog> | 
  reimage ms-fe3001 with jessie T117972 | 
  [production] | 
            
  | 15:25 | 
  <thcipriani@tin> | 
  Synchronized dblists: SWAT: [[gerrit:294269|Add nonecho.dblist and echo.dblist]] PART I (duration: 00m 28s) | 
  [production] | 
            
  | 15:19 | 
  <thcipriani@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:294269|Add nonecho.dblist and echo.dblist]] PART III (duration: 00m 28s) | 
  [production] | 
            
  | 15:18 | 
  <thcipriani@tin> | 
  Synchronized wmf-config/CommonSettings.php: SWAT: [[gerrit:294269|Add nonecho.dblist and echo.dblist]] PART II (duration: 00m 30s) | 
  [production] | 
            
  | 15:18 | 
  <thcipriani@tin> | 
  Synchronized dblists: SWAT: [[gerrit:294269|Add nonecho.dblist and echo.dblist]] PART I (duration: 00m 30s) | 
  [production] | 
            
  | 15:09 | 
  <yurik> | 
  deployed & restarted kartotherian | 
  [production] | 
            
  | 15:07 | 
  <thcipriani@tin> | 
  Synchronized dblists/visualeditor-default.dblist: SWAT: [[gerrit:292746|Enable VisualEditor by default on eleven Wikivoyages]] (duration: 01m 49s) | 
  [production] | 
            
  | 13:47 | 
  <hashar> | 
  T136971 Cutting MediaWiki branches 1.28.0-wmf.6 | 
  [production] | 
            
  | 13:40 | 
  <moritzm> | 
  installing apache trusty updates on codfw app servers | 
  [production] | 
            
  | 13:28 | 
  <paravoid> | 
  rebooting install2001, T137647 | 
  [production] | 
            
  | 12:58 | 
  <moritzm> | 
  installing apache trusty updates on canary app servers | 
  [production] | 
            
  | 12:55 | 
  <mobrovac> | 
  change-prop deployed f34fb06c99 | 
  [production] | 
            
  | 12:27 | 
  <jynus@tin> | 
  Synchronized wmf-config/db-eqiad.php: Repool db1024, pool for the first time db1090 with low weight (duration: 00m 38s) | 
  [production] | 
            
  | 11:30 | 
  <mobrovac> | 
  scb disabling puppet for 10 mins or so to keep change-prop down | 
  [production] | 
            
  | 11:15 | 
  <akosiaris> | 
  T134242 rebooting alsafi.wikimedia.org  hassaleh.codfw.wmnet   kraz.wikimedia.org mx2001.wikimedia.org   planet2001.codfw.wmnet   pollux.wikimedia.org   pybal-test2001.codfw.wmnet   pybal-test2002.codfw.wmnet  pybal-test2003.codfw.wmnet for qemu-kvm upgrade | 
  [production] | 
            
  | 11:13 | 
  <akosiaris> | 
  T134242 install qemu-system-common, qemu-system-x86 1:2.5+dfsg-4~bpo8+1 from jessie-backports on ganeti200{1,2,3,4,5,6} | 
  [production] | 
            
  | 11:04 | 
  <_joe_> | 
  pooling all the new codfw appservers that have been installed - mw2215-mw2240 (T135466) | 
  [production] | 
            
  | 10:56 | 
  <_joe_> | 
  pooling the new jessie appservers, mw1263-71 | 
  [production] | 
            
  | 10:52 | 
  <oblivian@palladium> | 
  conftool action : set/weight=30; selector: cluster=appserver,dc=eqiad,name=mw12[67].* | 
  [production] | 
            
  | 09:27 | 
  <godog> | 
  roll-restart swift proxy in codfw and eqiad | 
  [production] | 
            
  | 09:04 | 
  <hashar> | 
  gallium: manually removing cron entry zuul_repack from user zuul. Causes cron spam due to zuul merger no more being on gallium  T137418 | 
  [production] | 
            
  | 08:59 | 
  <jynus> | 
  stopping db1040 for cloning to new s4 hosts | 
  [production] | 
            
  | 08:28 | 
  <jynus@tin> | 
  Synchronized wmf-config/db-eqiad.php: Depool db1040 for cloning (duration: 00m 32s) | 
  [production] | 
            
  | 08:23 | 
  <_joe_> | 
  powercycling mw1154, unresponsive | 
  [production] | 
            
  | 07:19 | 
  <jynus> | 
  powercycling mw1156, could not regain control after OOM | 
  [production] | 
            
  | 07:18 | 
  <jynus@tin> | 
  Synchronized wmf-config/db-eqiad.php: Depool db1024, increase weight of db1082, db1087 and db1092 (duration: 10m 50s) | 
  [production] | 
            
  | 07:05 | 
  <_joe_> | 
  rolling reboot of mw2233-40 | 
  [production] | 
            
  | 06:47 | 
  <_joe_> | 
  rebooting mw2228 | 
  [production] | 
            
  | 06:43 | 
  <_joe_> | 
  rebooting mw2228 | 
  [production] |