| 
      
        2015-12-10
      
      §
     | 
  
    
  | 13:39 | 
  <jynus@tin> | 
  Synchronized wmf-config/db-eqiad.php: Switchover es2 master es1011 -> es1015 (duration: 00m 37s) | 
  [production] | 
            
  | 13:30 | 
  <paravoid> | 
  reenabled puppet on seaborgium & forced a puppet run | 
  [production] | 
            
  | 13:05 | 
  <moritzm> | 
  restarted slapd on serpens | 
  [production] | 
            
  | 13:03 | 
  <moritzm> | 
  stopping slapd on serpens to update LDAP indices | 
  [production] | 
            
  | 12:54 | 
  <moritzm> | 
  re-enabled puppet on serpens / restarted slapd | 
  [production] | 
            
  | 12:53 | 
  <jynus> | 
  performing switchover of es2 es1011-> es1015 for master maintenance, no production impact is expected | 
  [production] | 
            
  | 12:13 | 
  <godog> | 
  powercycle mw1119, login on console sluggish and no ssh | 
  [production] | 
            
  | 12:11 | 
  <jynus> | 
  codfw es2 server restart finished | 
  [production] | 
            
  | 09:04 | 
  <jynus> | 
  rebooting, restarting and upgrading mysql on es2 (codfw external storage servers) | 
  [production] | 
            
  | 03:55 | 
  <l10nupdate@tin> | 
  ResourceLoader cache refresh completed at Thu Dec 10 03:55:07 UTC 2015 (duration 54m 36s) | 
  [production] | 
            
  | 03:00 | 
  <mwdeploy@tin> | 
  sync-l10n completed (1.27.0-wmf.8) (duration: 15m 54s) | 
  [production] | 
            
  | 02:28 | 
  <gwicke> | 
  updated restbase1008 to cassandra 2.1.12 | 
  [production] | 
            
  | 02:27 | 
  <mwdeploy@tin> | 
  sync-l10n completed (1.27.0-wmf.7) (duration: 10m 53s) | 
  [production] | 
            
  | 01:25 | 
  <ebernhardson@tin> | 
  Synchronized php-1.27.0-wmf.7/extensions/CirrusSearch/: Add master timeout parameter to mapping updates so it can be increased when the master is slow (duration: 00m 30s) | 
  [production] | 
            
  | 01:23 | 
  <ebernhardson@tin> | 
  Synchronized php-1.27.0-wmf.8/extensions/CirrusSearch/: Add master timeout parameter to mapping updates so it can be increased when the master is slow (duration: 00m 29s) | 
  [production] | 
            
  | 01:20 | 
  <ebernhardson@tin> | 
  Synchronized wmf-config/CirrusSearch-production.php: Increase elasticsearch master timeout for maint actions to 2 minutes (duration: 00m 31s) | 
  [production] | 
            
  | 01:05 | 
  <catrope@tin> | 
  Synchronized php-1.27.0-wmf.8/extensions/WikimediaMaintenance/: Fix refreshMessageBlobs script (duration: 00m 29s) | 
  [production] | 
            
  | 01:01 | 
  <catrope@tin> | 
  Synchronized php-1.27.0-wmf.8/extensions/WikimediaMaintenance/: Fix refreshMessageBlobs script (duration: 00m 29s) | 
  [production] | 
            
  | 00:54 | 
  <catrope@tin> | 
  Synchronized php-1.27.0-wmf.8/: MessageBlobStore changes (duration: 02m 29s) | 
  [production] | 
            
  | 00:51 | 
  <catrope@tin> | 
  Synchronized php-1.27.0-wmf.8/skins/Vector/: Make placeholder in logged-out personal bar greyed out (duration: 00m 29s) | 
  [production] | 
            
  | 00:26 | 
  <catrope@tin> | 
  Synchronized php-1.27.0-wmf.8/extensions/Thanks/: Fix bug with topic titles in Flow thank notifications (duration: 00m 28s) | 
  [production] | 
            
  | 00:18 | 
  <catrope@tin> | 
  Synchronized wmf-config/mobile.php: Enable A/B test to measure impact of collapsing content (duration: 00m 29s) | 
  [production] | 
            
  | 00:06 | 
  <catrope@tin> | 
  Synchronized wmf-config/CommonSettings.php: Set $wgExtDistGraphiteRenderApi (duration: 00m 28s) | 
  [production] | 
            
  | 00:02 | 
  <catrope@tin> | 
  Synchronized wmf-config/InitialiseSettings-labs.php: Shut up unmerged change warning (duration: 00m 30s) | 
  [production] | 
            
  
    | 
      
        2015-12-09
      
      §
     | 
  
    
  | 23:43 | 
  <papaul> | 
  signing salt-key complete | 
  [production] | 
            
  | 23:28 | 
  <hoo@tin> | 
  Synchronized php-1.27.0-wmf.8/extensions/Wikidata: Item/ Property id values need separate validators each (duration: 00m 39s) | 
  [production] | 
            
  | 23:26 | 
  <papaul> | 
  auth2001 signing puppet certs | 
  [production] | 
            
  | 21:52 | 
  <papaul> | 
  OS install complete on auth2001 | 
  [production] | 
            
  | 21:34 | 
  <subbu> | 
  finished deploying parsoid sha a0c626e4 | 
  [production] | 
            
  | 21:30 | 
  <hashar> | 
  Jenkins upgraded to 1.625.3 ( https://phabricator.wikimedia.org/T56599 ) | 
  [production] | 
            
  | 21:22 | 
  <subbu> | 
  restarted parsoid on wtp1003 as a canary | 
  [production] | 
            
  | 21:20 | 
  <hashar> | 
  Jenkins going down right now. | 
  [production] | 
            
  | 21:18 | 
  <subbu> | 
  starting parsoid deploy | 
  [production] | 
            
  | 21:10 | 
  <hashar> | 
  Stopping Jenkins | 
  [production] | 
            
  | 19:56 | 
  <thcipriani@tin> | 
  rebuilt wikiversions.php and synchronized wikiversions files: hewiki and cawiki to php-1.27.0-wmf.8 | 
  [production] | 
            
  | 19:46 | 
  <thcipriani@tin> | 
  Synchronized dblists/group1.dblist: Train: Add hewiki and cawiki to group1. Part II [[gerrit:257973]] (duration: 00m 29s) | 
  [production] | 
            
  | 19:45 | 
  <thcipriani@tin> | 
  Synchronized dblists/group1-wikipedia.dblist: Train: Add hewiki and cawiki to group1. Part I [[gerrit:257973]] (duration: 00m 33s) | 
  [production] | 
            
  | 19:10 | 
  <thcipriani@tin> | 
  rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.8 | 
  [production] | 
            
  | 19:02 | 
  <ebernhardson@tin> | 
  Synchronized php-1.27.0-wmf.7/extensions/CirrusSearch/includes/Job/Job.php: Revert manual adjustment made to CirrusSearch Job.php  (duration: 00m 29s) | 
  [production] | 
            
  | 18:56 | 
  <mutante> | 
  mw1003 - restarted hhvm (and apache a bit earlier too) | 
  [production] | 
            
  | 18:15 | 
  <Jamesofur> | 
  inserted decryption key into database for enWP Arbcom case | 
  [production] | 
            
  | 17:31 | 
  <akosiaris> | 
  restarted hhvm on mw1125, memory starvation | 
  [production] | 
            
  | 17:31 | 
  <akosiaris> | 
  restarted hhvm on mw1125 | 
  [production] | 
            
  | 17:17 | 
  <thcipriani@tin> | 
  Synchronized php-1.27.0-wmf.8/extensions/Wikidata: SWAT: Update Wikibase [[gerrit:257901]] (duration: 00m 49s) | 
  [production] | 
            
  | 17:05 | 
  <krenair@tin> | 
  Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 28s) | 
  [production] | 
            
  | 16:44 | 
  <thcipriani@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: Add rights to CU+OS groups on en.wikipedia.org [[gerrit:255129]] (duration: 00m 29s) | 
  [production] | 
            
  | 16:32 | 
  <thcipriani@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: Turn off language detection user test [[gerrit:254071]] (duration: 00m 29s) | 
  [production] | 
            
  | 16:25 | 
  <thcipriani@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: Set initial titlesuggest shard sizes [[gerrit:256974]] (duration: 00m 29s) | 
  [production] | 
            
  | 16:18 | 
  <thcipriani@tin> | 
  Synchronized dblists/group1.dblist: SWAT: Move Catalan and Hebrew Wikipedias to group1 [[gerrit:253371]] (duration: 00m 29s) | 
  [production] | 
            
  | 16:12 | 
  <thcipriani@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable importupload and some import sources for officewiki [[gerrit:256881]] (duration: 00m 42s) | 
  [production] |