| 
      
        2015-12-17
      
      §
     | 
  
    
  | 15:56 | 
  <akosiaris> | 
  depool sca1001, playing with cxserver config | 
  [production] | 
            
  | 15:24 | 
  <jynus> | 
  reinstall, reboot and reconfigure mysql at db1031 | 
  [production] | 
            
  | 15:03 | 
  <moritzm> | 
  installing git security updates | 
  [production] | 
            
  | 14:23 | 
  <_joe_> | 
  restarting HHVM on the first jobrunners | 
  [production] | 
            
  | 14:11 | 
  <jynus> | 
  soft-rebooting mw1004, responsive to ping, but not to salt, ssh | 
  [production] | 
            
  | 14:06 | 
  <godog> | 
  nodetool stop -- CLEANUP on restbase1002 | 
  [production] | 
            
  | 13:59 | 
  <jynus@tin> | 
  Synchronized wmf-config/db-eqiad.php: Depool db1031 (x1-slave) for maintenance (duration: 07m 30s) | 
  [production] | 
            
  | 13:11 | 
  <moritzm> | 
  starting to restart hhvm on application servers (to effect security updates for libxml2, openssl and others) | 
  [production] | 
            
  | 13:01 | 
  <akosiaris> | 
  moved old cruft /srv/deployment/cxserver/deploy/src/config.js out of the way | 
  [production] | 
            
  | 12:51 | 
  <akosiaris> | 
  repooled sca1002 for cxserver | 
  [production] | 
            
  | 12:51 | 
  <akosiaris> | 
  restarted cxserver on sca1002 | 
  [production] | 
            
  | 12:33 | 
  <akosiaris> | 
  depooling sca1002 for cxserver | 
  [production] | 
            
  | 12:33 | 
  <akosiaris> | 
  repooling sca1001 for cxserver | 
  [production] | 
            
  | 12:24 | 
  <kart_> | 
  Updated cxserver on sca1002 | 
  [production] | 
            
  | 12:08 | 
  <akosiaris> | 
  disable puppet, stop salt-minion on sca1002 | 
  [production] | 
            
  | 12:08 | 
  <akosiaris> | 
  depool sca1001 from cxserver service. | 
  [production] | 
            
  | 11:56 | 
  <jynus@tin> | 
  Synchronized wmf-config/db-eqiad.php: Repool db1041 with low weight (duration: 00m 37s) | 
  [production] | 
            
  | 11:33 | 
  <jynus> | 
  performing schema change on officewiki | 
  [production] | 
            
  | 11:31 | 
  <jynus> | 
  performing schema change on x1-master flowdb | 
  [production] | 
            
  | 11:20 | 
  <jynus> | 
  performing schema change on x1-master wikishared.cx_translations | 
  [production] | 
            
  | 10:47 | 
  <jynus> | 
  performing schema change on s7-master metawiki.oauth_registered_consumer | 
  [production] | 
            
  | 10:30 | 
  <godog> | 
  nodetool stop -- CLEANUP restbase1004 | 
  [production] | 
            
  | 02:53 | 
  <l10nupdate@tin> | 
  ResourceLoader cache refresh completed at Thu Dec 17 02:53:11 UTC 2015 (duration 7m 12s) | 
  [production] | 
            
  | 02:45 | 
  <mwdeploy@tin> | 
  sync-l10n completed (1.27.0-wmf.9) (duration: 08m 22s) | 
  [production] | 
            
  | 02:27 | 
  <mwdeploy@tin> | 
  sync-l10n completed (1.27.0-wmf.8) (duration: 10m 32s) | 
  [production] | 
            
  | 00:28 | 
  <ori@tin> | 
  Synchronized php-1.27.0-wmf.8/includes/api/ApiStashEdit.php: I552cf6b0420: Upgrade some ApiStashEdit logging calls to info() (duration: 00m 30s) | 
  [production] | 
            
  | 00:05 | 
  <catrope@tin> | 
  Synchronized wmf-config/CommonSettings.php: Password policy for sysadmin group (duration: 00m 29s) | 
  [production] | 
            
  
    | 
      
        2015-12-16
      
      §
     | 
  
    
  | 23:37 | 
  <ori@tin> | 
  Synchronized php-1.27.0-wmf.8/includes/api/ApiStashEdit.php: local hack some extra debug logging into ApiStashEdit (take 2) (duration: 00m 30s) | 
  [production] | 
            
  | 23:32 | 
  <ori@tin> | 
  Synchronized php-1.27.0-wmf.8/includes/api/ApiStashEdit.php: local hack some extra debug logging into ApiStashEdit (duration: 00m 30s) | 
  [production] | 
            
  | 23:27 | 
  <yurik@tin> | 
  Synchronized php-1.27.0-wmf.9/extensions/Graph/: Deployed Graph ext to master - protocol issue (duration: 00m 32s) | 
  [production] | 
            
  | 23:10 | 
  <legoktm> | 
  running foreachwiki extensions/CentralAuth/maintenance/checkLocalUser.php --verbose=1 --delete=1 (T119736) | 
  [production] | 
            
  | 23:07 | 
  <robh> | 
  fermium returned to normal service with new/renewed ssl cert for lists.w.o | 
  [production] | 
            
  | 22:58 | 
  <robh> | 
  disabled puppet on fermium for the lists.w.o cert update | 
  [production] | 
            
  | 22:58 | 
  <robh> | 
  librenms returned to normal service (puppet renabled on netmon1001) | 
  [production] | 
            
  | 22:48 | 
  <robh> | 
  puppet disabled on netmon1001 for librenms certificate update | 
  [production] | 
            
  | 21:31 | 
  <subbu> | 
  finished deploying parsoid 64029e12 | 
  [production] | 
            
  | 21:29 | 
  <bearND> | 
  mobileapps deployed sha1 9f91ad5 | 
  [production] | 
            
  | 21:23 | 
  <subbu> | 
  restarted parsoid on wtp1005 as a canary | 
  [production] | 
            
  | 21:19 | 
  <subbu> | 
  starting parsoid deploy | 
  [production] | 
            
  | 21:14 | 
  <bearND> | 
  starting mobileapps deploy | 
  [production] | 
            
  | 19:46 | 
  <ostriches> | 
  iridium: repacking MW / OPUP repositories as phd user. This needs to be a cron. | 
  [production] | 
            
  | 19:07 | 
  <thcipriani@tin> | 
  rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.9 | 
  [production] | 
            
  | 19:00 | 
  <thcipriani> | 
  updating group1 wikis to 1.27.0-wmf.9 | 
  [production] | 
            
  | 18:39 | 
  <jynus> | 
  performing schema change on ruwiki.geo_tags on db2046 | 
  [production] | 
            
  | 17:46 | 
  <jynus> | 
  setting mysql db1041 as db2029's master | 
  [production] | 
            
  | 17:21 | 
  <jynus> | 
  restarting and configuring mysql on db2029 (there will be an increase of errors- from pings, not real traffic) | 
  [production] | 
            
  | 17:08 | 
  <jynus> | 
  setting mysql db1022 as db2028's master | 
  [production] | 
            
  | 16:50 | 
  <demon@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: graph namespace removal/cleanup (duration: 00m 31s) | 
  [production] | 
            
  | 16:41 | 
  <gwicke> | 
  cleared snapshots on cassandra cluster | 
  [production] | 
            
  | 16:32 | 
  <gwicke> | 
  restarted `nodetool decommission` on restbase1004 | 
  [production] |