| 
      
        2016-06-14
      
      §
     | 
  
    
  | 13:28 | 
  <paravoid> | 
  rebooting install2001, T137647 | 
  [production] | 
            
  | 12:58 | 
  <moritzm> | 
  installing apache trusty updates on canary app servers | 
  [production] | 
            
  | 12:55 | 
  <mobrovac> | 
  change-prop deployed f34fb06c99 | 
  [production] | 
            
  | 12:27 | 
  <jynus@tin> | 
  Synchronized wmf-config/db-eqiad.php: Repool db1024, pool for the first time db1090 with low weight (duration: 00m 38s) | 
  [production] | 
            
  | 11:30 | 
  <mobrovac> | 
  scb disabling puppet for 10 mins or so to keep change-prop down | 
  [production] | 
            
  | 11:15 | 
  <akosiaris> | 
  T134242 rebooting alsafi.wikimedia.org  hassaleh.codfw.wmnet   kraz.wikimedia.org mx2001.wikimedia.org   planet2001.codfw.wmnet   pollux.wikimedia.org   pybal-test2001.codfw.wmnet   pybal-test2002.codfw.wmnet  pybal-test2003.codfw.wmnet for qemu-kvm upgrade | 
  [production] | 
            
  | 11:13 | 
  <akosiaris> | 
  T134242 install qemu-system-common, qemu-system-x86 1:2.5+dfsg-4~bpo8+1 from jessie-backports on ganeti200{1,2,3,4,5,6} | 
  [production] | 
            
  | 11:04 | 
  <_joe_> | 
  pooling all the new codfw appservers that have been installed - mw2215-mw2240 (T135466) | 
  [production] | 
            
  | 10:56 | 
  <_joe_> | 
  pooling the new jessie appservers, mw1263-71 | 
  [production] | 
            
  | 10:52 | 
  <oblivian@palladium> | 
  conftool action : set/weight=30; selector: cluster=appserver,dc=eqiad,name=mw12[67].* | 
  [production] | 
            
  | 09:27 | 
  <godog> | 
  roll-restart swift proxy in codfw and eqiad | 
  [production] | 
            
  | 09:04 | 
  <hashar> | 
  gallium: manually removing cron entry zuul_repack from user zuul. Causes cron spam due to zuul merger no more being on gallium  T137418 | 
  [production] | 
            
  | 08:59 | 
  <jynus> | 
  stopping db1040 for cloning to new s4 hosts | 
  [production] | 
            
  | 08:28 | 
  <jynus@tin> | 
  Synchronized wmf-config/db-eqiad.php: Depool db1040 for cloning (duration: 00m 32s) | 
  [production] | 
            
  | 08:23 | 
  <_joe_> | 
  powercycling mw1154, unresponsive | 
  [production] | 
            
  | 07:19 | 
  <jynus> | 
  powercycling mw1156, could not regain control after OOM | 
  [production] | 
            
  | 07:18 | 
  <jynus@tin> | 
  Synchronized wmf-config/db-eqiad.php: Depool db1024, increase weight of db1082, db1087 and db1092 (duration: 10m 50s) | 
  [production] | 
            
  | 07:05 | 
  <_joe_> | 
  rolling reboot of mw2233-40 | 
  [production] | 
            
  | 06:47 | 
  <_joe_> | 
  rebooting mw2228 | 
  [production] | 
            
  | 06:43 | 
  <_joe_> | 
  rebooting mw2228 | 
  [production] | 
            
  | 06:29 | 
  <jynus@tin> | 
  Synchronized wmf-config/db-eqiad.php: Pool db1052, db1080, db1083, db1089 (duration: 01m 31s) | 
  [production] | 
            
  | 02:39 | 
  <l10nupdate@tin> | 
  ResourceLoader cache refresh completed at Tue Jun 14 02:39:50 UTC 2016 (duration 5m 59s) | 
  [production] | 
            
  | 02:33 | 
  <mwdeploy@tin> | 
  scap sync-l10n completed (1.28.0-wmf.5) (duration: 12m 14s) | 
  [production] | 
            
  
    | 
      
        2016-06-13
      
      §
     | 
  
    
  | 23:50 | 
  <dereckson@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: Add ORES to whitelisted beta features (T130211) (duration: 00m 23s) | 
  [production] | 
            
  | 23:42 | 
  <dereckson@tin> | 
  Synchronized php-1.28.0-wmf.5/extensions/ORES/includes/Hooks.php: Update links to beta features (duration: 00m 25s) | 
  [production] | 
            
  | 23:33 | 
  <ejegg> | 
  updated payments from 44102c59ac897c9acab470bf83369d233f9b736f to 2fc573cbb94e833c4144aa9dad79de8ec374bb09 | 
  [production] | 
            
  | 23:29 | 
  <dereckson@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: Update cross-wiki upload configuration ([[Gerrit:293355]]) (duration: 00m 23s) | 
  [production] | 
            
  | 23:10 | 
  <dereckson@tin> | 
  Synchronized portals: (no message) (duration: 00m 24s) | 
  [production] | 
            
  | 23:10 | 
  <dereckson@tin> | 
  Synchronized portals/prod/wikipedia.org/assets: (no message) (duration: 00m 24s) | 
  [production] | 
            
  | 22:51 | 
  <demon@tin> | 
  Synchronized wmf-config/CommonSettings.php: Update extension distributor settings (duration: 00m 24s) | 
  [production] | 
            
  | 22:42 | 
  <yurik> | 
  switched to scap3 and deployed tilerator. Deployed kartotherian. Restarted. | 
  [production] | 
            
  | 22:41 | 
  <dapatrick> | 
  Deployed patches for T129738 to wmf5 | 
  [production] | 
            
  | 22:36 | 
  <awight> | 
  update fundraising CRM revert from e684b7823e751558772a4de4ac23819bc601eb74 to bb9bf136dc0fa82d5d07ebeb33d696e54672b2d6 | 
  [production] | 
            
  | 22:11 | 
  <awight> | 
  Updating fundraising CRM from b7b46740d701942507dca0a98a75f3f87b6b31b1 to e684b7823e751558772a4de4ac23819bc601eb74 | 
  [production] | 
            
  | 19:15 | 
  <aaron@tin> | 
  Synchronized php-1.28.0-wmf.5/resources: ee2da9c2ae6fac93bf65d17b5ea48e5c47c87d47 (duration: 00m 35s) | 
  [production] | 
            
  | 18:20 | 
  <bblack> | 
  upgrading nginx (etc) on deployment-prep caches | 
  [production] | 
            
  | 18:11 | 
  <gehel> | 
  deploying latest GUI on WDQS, | 
  [production] | 
            
  | 17:58 | 
  <urandom> | 
  Upgrade of restbase1007.eqiad.wmnet (https://people.wikimedia.org/~eevans/debian/cassandra_2.2.6-wmf1_all.deb) complete : T137474 | 
  [production] | 
            
  | 17:55 | 
  <urandom> | 
  Restarting restbase1007-c.eqiad.wmnet : T137474 | 
  [production] | 
            
  | 17:52 | 
  <urandom> | 
  Restarting restbase1007-b.eqiad.wmnet : T137474 | 
  [production] | 
            
  | 17:47 | 
  <awight> | 
  Whitelist Special:PaypalExpressGatewayResult | 
  [production] | 
            
  | 17:43 | 
  <godog> | 
  enable proxy_http apache module on graphite1003 / graphite2002 and restart apache | 
  [production] | 
            
  | 17:38 | 
  <urandom> | 
  Restarting restbase1007-a.eqiad.wmnet : T137474 | 
  [production] | 
            
  | 17:37 | 
  <urandom> | 
  Upgrading restbase1007.eqiad.wmnet w/ https://people.wikimedia.org/~eevans/debian/cassandra_2.2.6-wmf1_all.deb : T137474 | 
  [production] | 
            
  | 17:35 | 
  <awight> | 
  update paymentswiki from 63fbe39fbc4d671fd2705ce9e42762b7c49564c2 to 44102c59ac897c9acab470bf83369d233f9b736f | 
  [production] | 
            
  | 16:51 | 
  <_joe_> | 
  powercycling mw1115 | 
  [production] | 
            
  | 16:49 | 
  <thcipriani@tin> | 
  Finished scap: Update l10n cache for ores (duration: 32m 04s) | 
  [production] | 
            
  | 16:17 | 
  <thcipriani@tin> | 
  Started scap: Update l10n cache for ores | 
  [production] | 
            
  | 15:59 | 
  <thcipriani@tin> | 
  Synchronized wmf-config/CommonSettings.php: SWAT: [[gerrit:269478|Enable ORES on fawiki]] PART II (duration: 00m 24s) | 
  [production] | 
            
  | 15:58 | 
  <thcipriani@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:269478|Enable ORES on fawiki]] PART I (duration: 00m 25s) | 
  [production] |