| 
      
        2015-07-01
      
      §
     | 
  
    
  | 19:28 | 
  <ori> | 
  Repooling mw1152 for further testing of HHVM scaler | 
  [production] | 
            
  | 19:03 | 
  <hoo> | 
  Synchronized php-1.26wmf12/extensions/Wikidata/: Update DataModel to fix SnakList (duration: 00m 20s) | 
  [production] | 
            
  | 18:42 | 
  <hoo> | 
  Synchronized wmf-config/mobile-labs.php: consistency (duration: 00m 12s) | 
  [production] | 
            
  | 18:41 | 
  <hoo> | 
  Synchronized wmf-config/InitialiseSettings-labs.php: consistency (duration: 00m 31s) | 
  [production] | 
            
  | 18:02 | 
  <andrewbogott> | 
  restarted keystone on labcontrol1001 | 
  [production] | 
            
  | 17:03 | 
  <jgage> | 
  beginning puppet CA replacement procedure | 
  [production] | 
            
  | 16:06 | 
  <ejegg> | 
  enabled queue consumers | 
  [production] | 
            
  | 16:05 | 
  <akosiaris> | 
  re-enabling ntp everywhere | 
  [production] | 
            
  | 15:59 | 
  <ejegg> | 
  disabled queue consumers | 
  [production] | 
            
  | 15:30 | 
  <hoo> | 
  Synchronized php-1.26wmf12/extensions/Wikidata/: Remove alias uniqueness constraints (duration: 00m 21s) | 
  [production] | 
            
  | 15:06 | 
  <urandom> | 
  restbase1002: PWD=/home/eevans/restbase-mod-table-cassandra/maintenance; node thin_out_key_rev_value_data.js `hostname -i` local_group_wikimedia_T_parsoid_html 2>&1 | pv --line-mode | gzip -c > wikimedia_T_parsoid_html.log.gz | 
  [production] | 
            
  | 15:05 | 
  <bblack> | 
  re-enabling puppet on caches | 
  [production] | 
            
  | 14:59 | 
  <bblack> | 
  disabling puppet on caches (because puppet always breaks when you move files/modules around...) | 
  [production] | 
            
  | 13:57 | 
  <bblack> | 
  rebooting cp2001 (test kernel update) | 
  [production] | 
            
  | 11:32 | 
  <YuviPanda> | 
  rsync on labstore1002 finished, restarting to see what was skipped + errors | 
  [production] | 
            
  | 10:47 | 
  <moritzm> | 
  installed patch security updates on 862 hosts | 
  [production] | 
            
  | 10:42 | 
  <hashar> | 
  restarting Jenkins: upgrading Jenkins gearman plugin from 0.1.1-8-gf2024bd to 0.1.1-9-g08e9c42-change_192429_2   https://phabricator.wikimedia.org/T72597#1416913 | 
  [production] | 
            
  | 07:48 | 
  <mobrovac> | 
  restbase restarting cassandra on rb1005 | 
  [production] | 
            
  | 05:28 | 
  <LocalisationUpdate> | 
  ResourceLoader cache refresh completed at Wed Jul  1 05:28:38 UTC 2015 (duration 28m 37s) | 
  [production] | 
            
  | 05:27 | 
  <csteipp> | 
  deployed patch for T103765 | 
  [production] | 
            
  | 04:41 | 
  <krinkle> | 
  Synchronized php-1.26wmf12/includes/resourceloader/ResourceLoader.php: Iee884208c5c4b minify cache key (duration: 00m 11s) | 
  [production] | 
            
  | 03:10 | 
  <mutante> | 
  git pull on strontium | 
  [production] | 
            
  | 03:00 | 
  <LocalisationUpdate> | 
  completed (1.26wmf12) at 2015-07-01 03:00:21+00:00 | 
  [production] | 
            
  | 02:53 | 
  <l10nupdate> | 
  Synchronized php-1.26wmf12/cache/l10n: (no message) (duration: 10m 12s) | 
  [production] | 
            
  | 02:26 | 
  <LocalisationUpdate> | 
  completed (1.26wmf11) at 2015-07-01 02:26:55+00:00 | 
  [production] | 
            
  | 02:23 | 
  <l10nupdate> | 
  Synchronized php-1.26wmf11/cache/l10n: (no message) (duration: 06m 50s) | 
  [production] | 
            
  | 02:12 | 
  <springle> | 
  upgrade db1034 trusty | 
  [production] | 
            
  | 01:37 | 
  <ori> | 
  Depooled mw1152. Req error dashboard shows elevated 5xx rates correlating with the server getting pooled, but the logs don't appear to corroborate it. Odd. | 
  [production] | 
            
  | 01:03 | 
  <ori> | 
  Disabling Puppet on mw1152 for 12h to hack apache config to log locally | 
  [production] | 
            
  | 00:42 | 
  <ori> | 
  Synchronized wmf-config/CommonSettings.php: I9a8018981: Double $wgMaxShellMemory on HHVM scalers (512 Mb => 1024 Mb) (duration: 00m 12s) | 
  [production] | 
            
  | 00:34 | 
  <ori> | 
  pooled mw1152 (HHVM rendering) at weight 10 for testing | 
  [production] | 
            
  | 00:33 | 
  <gwicke> | 
  rolling cassandra restart done | 
  [production] | 
            
  | 00:23 | 
  <gwicke> | 
  starting rolling restart of cassandra nodes to apply new config | 
  [production] | 
            
  | 00:01 | 
  <greg-g> | 
  we're still here | 
  [production] | 
            
  
    | 
      
        2015-06-30
      
      §
     | 
  
    
  | 23:30 | 
  <hoo> | 
  Synchronized php-1.26wmf12/extensions/Wikidata/: Fix EntityParserOutputGenerator (duration: 00m 21s) | 
  [production] | 
            
  | 22:55 | 
  <ori> | 
  depooled mw1152 | 
  [production] | 
            
  | 22:52 | 
  <ori> | 
  Pooled HHVM image scaler (mw1152) at weight 1 for testing. | 
  [production] | 
            
  | 22:52 | 
  <gwicke> | 
  updated restbase1004 to openjdk-8 | 
  [production] | 
            
  | 22:46 | 
  <bblack> | 
  restarting gitblit on antimony, because Java is so 1996 | 
  [production] | 
            
  | 22:43 | 
  <tgr> | 
  running eval.php (along the lines of https://gerrit.wikimedia.org/r/#/c/221783) on commonswiki to fix T104395 | 
  [production] | 
            
  | 22:13 | 
  <catrope> | 
  Synchronized wmf-config/InitialiseSettings.php: Flow-occupy Wikipedia talk namespace on cawiki (duration: 00m 11s) | 
  [production] | 
            
  | 22:09 | 
  <matt_flaschen> | 
  Done converting wikitext namespace to Flow on Catalan Wikipedia | 
  [production] | 
            
  | 22:03 | 
  <matt_flaschen> | 
  Started convertNamespaceFromWikitext.php for Project_talk on Catalan Wikipedia | 
  [production] | 
            
  | 21:46 | 
  <RoanKattouw> | 
  Also ran populateContentModel.php --table=archive for talk namespaces on officewiki | 
  [production] | 
            
  | 21:46 | 
  <RoanKattouw> | 
  Ran populateContentModel.php --table=archive --ns=5 on officewiki | 
  [production] | 
            
  | 21:29 | 
  <RoanKattouw> | 
  Ran populateContentModel.php --table=page --ns=5 on cawiki | 
  [production] | 
            
  | 21:19 | 
  <catrope> | 
  Synchronized php-1.26wmf12/extensions/Flow: (no message) (duration: 00m 14s) | 
  [production] | 
            
  | 21:19 | 
  <catrope> | 
  Synchronized php-1.26wmf11/extensions/Flow: (no message) (duration: 00m 14s) | 
  [production] | 
            
  | 21:14 | 
  <catrope> | 
  Synchronized php-1.26wmf12/extensions/Flow: (no message) (duration: 00m 14s) | 
  [production] | 
            
  | 21:14 | 
  <catrope> | 
  Synchronized php-1.26wmf11/extensions/Flow: (no message) (duration: 00m 13s) | 
  [production] |