| 
      
        2015-12-17
      
      §
     | 
  
    
  | 12:08 | 
  <akosiaris> | 
  depool sca1001 from cxserver service. | 
  [production] | 
            
  | 11:56 | 
  <jynus@tin> | 
  Synchronized wmf-config/db-eqiad.php: Repool db1041 with low weight (duration: 00m 37s) | 
  [production] | 
            
  | 11:33 | 
  <jynus> | 
  performing schema change on officewiki | 
  [production] | 
            
  | 11:31 | 
  <jynus> | 
  performing schema change on x1-master flowdb | 
  [production] | 
            
  | 11:20 | 
  <jynus> | 
  performing schema change on x1-master wikishared.cx_translations | 
  [production] | 
            
  | 10:47 | 
  <jynus> | 
  performing schema change on s7-master metawiki.oauth_registered_consumer | 
  [production] | 
            
  | 10:30 | 
  <godog> | 
  nodetool stop -- CLEANUP restbase1004 | 
  [production] | 
            
  | 02:53 | 
  <l10nupdate@tin> | 
  ResourceLoader cache refresh completed at Thu Dec 17 02:53:11 UTC 2015 (duration 7m 12s) | 
  [production] | 
            
  | 02:45 | 
  <mwdeploy@tin> | 
  sync-l10n completed (1.27.0-wmf.9) (duration: 08m 22s) | 
  [production] | 
            
  | 02:27 | 
  <mwdeploy@tin> | 
  sync-l10n completed (1.27.0-wmf.8) (duration: 10m 32s) | 
  [production] | 
            
  | 00:28 | 
  <ori@tin> | 
  Synchronized php-1.27.0-wmf.8/includes/api/ApiStashEdit.php: I552cf6b0420: Upgrade some ApiStashEdit logging calls to info() (duration: 00m 30s) | 
  [production] | 
            
  | 00:05 | 
  <catrope@tin> | 
  Synchronized wmf-config/CommonSettings.php: Password policy for sysadmin group (duration: 00m 29s) | 
  [production] | 
            
  
    | 
      
        2015-12-16
      
      §
     | 
  
    
  | 23:37 | 
  <ori@tin> | 
  Synchronized php-1.27.0-wmf.8/includes/api/ApiStashEdit.php: local hack some extra debug logging into ApiStashEdit (take 2) (duration: 00m 30s) | 
  [production] | 
            
  | 23:32 | 
  <ori@tin> | 
  Synchronized php-1.27.0-wmf.8/includes/api/ApiStashEdit.php: local hack some extra debug logging into ApiStashEdit (duration: 00m 30s) | 
  [production] | 
            
  | 23:27 | 
  <yurik@tin> | 
  Synchronized php-1.27.0-wmf.9/extensions/Graph/: Deployed Graph ext to master - protocol issue (duration: 00m 32s) | 
  [production] | 
            
  | 23:10 | 
  <legoktm> | 
  running foreachwiki extensions/CentralAuth/maintenance/checkLocalUser.php --verbose=1 --delete=1 (T119736) | 
  [production] | 
            
  | 23:07 | 
  <robh> | 
  fermium returned to normal service with new/renewed ssl cert for lists.w.o | 
  [production] | 
            
  | 22:58 | 
  <robh> | 
  disabled puppet on fermium for the lists.w.o cert update | 
  [production] | 
            
  | 22:58 | 
  <robh> | 
  librenms returned to normal service (puppet renabled on netmon1001) | 
  [production] | 
            
  | 22:48 | 
  <robh> | 
  puppet disabled on netmon1001 for librenms certificate update | 
  [production] | 
            
  | 21:31 | 
  <subbu> | 
  finished deploying parsoid 64029e12 | 
  [production] | 
            
  | 21:29 | 
  <bearND> | 
  mobileapps deployed sha1 9f91ad5 | 
  [production] | 
            
  | 21:23 | 
  <subbu> | 
  restarted parsoid on wtp1005 as a canary | 
  [production] | 
            
  | 21:19 | 
  <subbu> | 
  starting parsoid deploy | 
  [production] | 
            
  | 21:14 | 
  <bearND> | 
  starting mobileapps deploy | 
  [production] | 
            
  | 19:46 | 
  <ostriches> | 
  iridium: repacking MW / OPUP repositories as phd user. This needs to be a cron. | 
  [production] | 
            
  | 19:07 | 
  <thcipriani@tin> | 
  rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.9 | 
  [production] | 
            
  | 19:00 | 
  <thcipriani> | 
  updating group1 wikis to 1.27.0-wmf.9 | 
  [production] | 
            
  | 18:39 | 
  <jynus> | 
  performing schema change on ruwiki.geo_tags on db2046 | 
  [production] | 
            
  | 17:46 | 
  <jynus> | 
  setting mysql db1041 as db2029's master | 
  [production] | 
            
  | 17:21 | 
  <jynus> | 
  restarting and configuring mysql on db2029 (there will be an increase of errors- from pings, not real traffic) | 
  [production] | 
            
  | 17:08 | 
  <jynus> | 
  setting mysql db1022 as db2028's master | 
  [production] | 
            
  | 16:50 | 
  <demon@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: graph namespace removal/cleanup (duration: 00m 31s) | 
  [production] | 
            
  | 16:41 | 
  <gwicke> | 
  cleared snapshots on cassandra cluster | 
  [production] | 
            
  | 16:32 | 
  <gwicke> | 
  restarted `nodetool decommission` on restbase1004 | 
  [production] | 
            
  | 16:31 | 
  <jynus> | 
  restarting and reconfiguring mysql on db1041 | 
  [production] | 
            
  | 15:54 | 
  <jynus@tin> | 
  Synchronized wmf-config/db-eqiad.php: Repool db1022, depool db1041 (duration: 00m 30s) | 
  [production] | 
            
  | 15:53 | 
  <godog> | 
  cassandra on restbase1004 couldn't finish decomissioning, out of disk space, running nodetool clean | 
  [production] | 
            
  | 15:28 | 
  <jynus> | 
  restarting and reconfiguring mysql on db2028 | 
  [production] | 
            
  | 15:17 | 
  <yurik> | 
  updated graphoid service | 
  [production] | 
            
  | 15:16 | 
  <moritzm> | 
  removed ecryptfs-utils across the cluster | 
  [production] | 
            
  | 15:09 | 
  <godog> | 
  roll-restart swift daemons on ms-be1* | 
  [production] | 
            
  | 15:04 | 
  <jynus> | 
  setting db2023 master to be now db1049 instead of m5-master | 
  [production] | 
            
  | 14:56 | 
  <mobrovac> | 
  restbase roll-restarting restbase | 
  [production] | 
            
  | 14:52 | 
  <godog> | 
  reenable and run puppet on restbase* and aqs* | 
  [production] | 
            
  | 14:49 | 
  <mobrovac> | 
  restbase restart rb1001 | 
  [production] | 
            
  | 14:46 | 
  <urandom> | 
  starting `nodetool cleanup' on restbase1009-a.eqiad (https://phabricator.wikimedia.org/T121535) | 
  [production] | 
            
  | 14:41 | 
  <urandom> | 
  starting `nodetool cleanup' on restbase1006.eqiad (https://phabricator.wikimedia.org/T121535) | 
  [production] | 
            
  | 14:17 | 
  <mobrovac> | 
  restbase reenabling and running puppet on canary rb1001 | 
  [production] | 
            
  | 14:08 | 
  <godog> | 
  reenable puppet on restbase test cluster | 
  [production] |