| 
      
        2016-05-17
      
      ยง
     | 
  
    
  | 23:10 | 
  <dereckson@tin> | 
  Synchronized portals: Turning off survey banner on www.wikipedia.org (T135235) (duration: 00m 26s) | 
  [production] | 
            
  | 23:09 | 
  <dereckson@tin> | 
  Synchronized portals/prod/wikipedia.org/assets: Turning off survey banner on www.wikipedia.org (T135235) (duration: 00m 25s) | 
  [production] | 
            
  | 22:40 | 
  <ori@tin> | 
  Synchronized php-1.28.0-wmf.1/includes/api/ApiStashEdit.php: Id1e0808c: Improve edit stash hit rate for logged-out users (duration: 00m 32s) | 
  [production] | 
            
  | 21:45 | 
  <gehel> | 
  stopping rolling restart of elasticsearch cluster for the night (T135499) | 
  [production] | 
            
  | 21:21 | 
  <ori@tin> | 
  Synchronized php-1.28.0-wmf.1/extensions/NavigationTiming: I62e20087c1: Expand coverage of conformance test (duration: 00m 28s) | 
  [production] | 
            
  | 21:15 | 
  <twentyafterfour> | 
  finished deploying wmf/1.28.0-wmf.2 | 
  [production] | 
            
  | 21:15 | 
  <ori@tin> | 
  Synchronized php-1.28.0-wmf.2/includes/api/ApiStashEdit.php: Id1e0808c: Improve edit stash hit rate for logged-out users (duration: 00m 35s) | 
  [production] | 
            
  | 20:52 | 
  <twentyafterfour@tin> | 
  rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.28.0-wmf.2 | 
  [production] | 
            
  | 20:52 | 
  <twentyafterfour@tin> | 
  Purged l10n cache for 1.27.0-wmf.22 | 
  [production] | 
            
  | 20:51 | 
  <twentyafterfour@tin> | 
  Purged l10n cache for 1.27.0-wmf.21 | 
  [production] | 
            
  | 20:51 | 
  <twentyafterfour@tin> | 
  Purged l10n cache for 1.27.0-wmf.20 | 
  [production] | 
            
  | 20:41 | 
  <twentyafterfour@tin> | 
  Finished scap: (no message) (duration: 27m 42s) | 
  [production] | 
            
  | 20:26 | 
  <bblack> | 
  rebooting kafka1013 from racadm | 
  [production] | 
            
  | 20:14 | 
  <twentyafterfour> | 
  syncing testwiki to wmf/1.28.0-wmf.2 | 
  [production] | 
            
  | 20:13 | 
  <twentyafterfour@tin> | 
  Started scap: (no message) | 
  [production] | 
            
  | 20:09 | 
  <bblack> | 
  starting upgrade of varnish4 packages on cache_maps | 
  [production] | 
            
  | 20:09 | 
  <bblack> | 
  finished upgrade of varnish4 packages on cache_misc | 
  [production] | 
            
  | 20:03 | 
  <ejegg> | 
  enabled banner impression loader | 
  [production] | 
            
  | 20:02 | 
  <ejegg> | 
  updated DjangoBannerStats from 30396a6df5bb9eadd5d4485a08fc4f6cb4096bd9 to 8172c4176ea2b78df0dccbfa052064f2739c64bd | 
  [production] | 
            
  | 20:00 | 
  <ejegg> | 
  disabled banner impression loader | 
  [production] | 
            
  | 19:58 | 
  <bblack> | 
  starting upgrade of varnish4 packages on cache_misc | 
  [production] | 
            
  | 19:01 | 
  <twentyafterfour> | 
  preparing to deploy wmf/1.28.0-wmf.2 | 
  [production] | 
            
  | 18:37 | 
  <matt_flaschen> | 
  Ran manual SQL write in production to work around T122262; see task for query. | 
  [production] | 
            
  | 18:36 | 
  <urandom> | 
  Restarting (failed) bootstrap of restbase2008-b.codfw.wmnet : T95253 | 
  [production] | 
            
  | 18:34 | 
  <urandom> | 
  Restart restbase2008-a.codfw.wmnet; Hail Mary pass for failed 2008-b bootstraps : T95253 | 
  [production] | 
            
  | 17:22 | 
  <ejegg> | 
  updated CiviCRM from 6e3581692bb81c2c507612e7cd66dcdf78ef3cc0 to 7952ba43a012cb6a2e8d16af19bb13ed520bd56f | 
  [production] | 
            
  | 16:23 | 
  <godog> | 
  disable mod_deflate and restart apache2 on graphite1001 T135515 | 
  [production] | 
            
  | 16:01 | 
  <twentyafterfour> | 
  branching wmf/1.28.0-wmf.2 | 
  [production] | 
            
  | 15:12 | 
  <thcipriani@tin> | 
  Synchronized portals: (no message) (duration: 00m 25s) | 
  [production] | 
            
  | 15:12 | 
  <thcipriani@tin> | 
  Synchronized portals/prod/wikipedia.org/assets: (no message) (duration: 00m 25s) | 
  [production] | 
            
  | 15:11 | 
  <thcipriani> | 
  running /srv/mediawiki-staging/portals/sync-portals for [[gerrit:289218]] | 
  [production] | 
            
  | 15:06 | 
  <thcipriani@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor for IP users on the Japanese Wikipedia [[gerrit:287646]] (duration: 00m 35s) | 
  [production] | 
            
  | 14:59 | 
  <elukey> | 
  disabled puppet on aqs* nodes as prep step to bootstrap the new testing cluster (https://gerrit.wikimedia.org/r/#/c/288373/5)  | 
  [production] | 
            
  | 14:46 | 
  <gehel> | 
  taking elastic1001 down for investigation (T135509) | 
  [production] | 
            
  | 14:37 | 
  <mobrovac> | 
  change-prop deploying 5b5a07a3 | 
  [production] | 
            
  | 14:15 | 
  <gehel> | 
  restarting elastic1001 - high load (T135509) | 
  [production] | 
            
  | 13:32 | 
  <hashar> | 
  upgrading Jenkins T133737 | 
  [production] | 
            
  | 13:24 | 
  <godog> | 
  bounce carbon/frontend-relay on graphite1001 to increase queue size T135385 | 
  [production] | 
            
  | 13:22 | 
  <elukey> | 
  memcahced restarted on mc1009 with -o slab_reassign,slab_automove,lru_crawler,lru_maintainer as part of a perf experiment (T129963) | 
  [production] | 
            
  | 12:21 | 
  <gehel> | 
  starting rolling restart of Elasticsearch equiad fro Java update (T135499) | 
  [production] | 
            
  | 12:16 | 
  <mobrovac> | 
  mathoid deployed 10c7cb8 | 
  [production] | 
            
  | 11:04 | 
  <elukey> | 
  updated facts on puppet compiler following https://wikitech.wikimedia.org/wiki/Nova_Resource:Puppet3-diffs | 
  [production] | 
            
  | 10:59 | 
  <jynus> | 
  disabling puppet and starting decom process of db1027 | 
  [production] | 
            
  | 10:30 | 
  <jynus> | 
  running schema change on s1 T130692 | 
  [production] | 
            
  | 09:59 | 
  <filippo@palladium> | 
  conftool action : set/pooled=yes; selector: mw2050.codfw.wmnet | 
  [production] | 
            
  | 09:14 | 
  <ema> | 
  upgrading ulsfo cache_misc to varnish 4 (T131501, T134989) | 
  [production] | 
            
  | 09:02 | 
  <ema> | 
  upgrading esams cache_misc to varnish 4 (T126206, T134989) | 
  [production] | 
            
  | 08:54 | 
  <jynus> | 
  increased pool stall limit to 500 on db1049 | 
  [production] | 
            
  | 08:53 | 
  <filippo@palladium> | 
  conftool action : set/pooled=no; selector: mw2050.codfw.wmnet | 
  [production] | 
            
  | 08:31 | 
  <ema> | 
  upgrading codfw cache_misc to varnish 4 (T126206, T134989) | 
  [production] |