| 
      
        2015-07-07
      
      §
     | 
  
    
  | 19:15 | 
  <moritzm> | 
  installed PHP security updates on all trusty hosts | 
  [production] | 
            
  | 18:58 | 
  <ejegg> | 
  updated payments from a17ee221db0dbde70c92e24fc188379b6dbad613 to ec34ebf61e5962f66b807abdcb519ff323d41e8e | 
  [production] | 
            
  | 18:09 | 
  <twentyafterfour> | 
  restarted apache2 on iridium (phab hotfix) | 
  [production] | 
            
  | 17:10 | 
  <robh> | 
  OTRS update appears to be functioning normally.  As such, ending maintenance window. | 
  [production] | 
            
  | 17:06 | 
  <robh> | 
  otrs is now using the new sha256 cert | 
  [production] | 
            
  | 17:00 | 
  <robh> | 
  starting otrs maint window | 
  [production] | 
            
  | 16:58 | 
  <_joe_> | 
  restarted HHVM on mw1026, near to OOM | 
  [production] | 
            
  | 16:47 | 
  <twentyafterfour> | 
  applied hotfix for phabricator bug: https://secure.phabricator.com/D13544 | 
  [production] | 
            
  | 16:36 | 
  <mutante> | 
  protactinium - manual iptables rules replaced by puppet/ferm rules | 
  [production] | 
            
  | 16:11 | 
  <thcipriani> | 
  Synchronized php-1.26wmf12/extensions/ContentTranslation/extension.json: Remove default value for ContentTranslationCampaigns (duration: 00m 12s) | 
  [production] | 
            
  | 15:33 | 
  <jynus> | 
  manually editing table mediawiki.ipblocks to fully solve a former software bug | 
  [production] | 
            
  | 15:12 | 
  <Jeff_Green> | 
  ptr records for frack/codfw and authdns-update | 
  [production] | 
            
  | 15:10 | 
  <thcipriani> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Enable ContentTranslation in enwiki [[gerrit:222991]] (duration: 00m 13s) | 
  [production] | 
            
  | 14:21 | 
  <jynus> | 
  dropping optin_survey_old table from enwiki | 
  [production] | 
            
  | 13:23 | 
  <akosiaris> | 
  restarting gitblit on antimony | 
  [production] | 
            
  | 11:31 | 
  <mobrovac> | 
  restbase restarted cassandra on rb1005 | 
  [production] | 
            
  | 11:26 | 
  <godog> | 
  restart cassandra on restbase1004, heap exhausted | 
  [production] | 
            
  | 10:49 | 
  <godog> | 
  restarted cassandra on restbase1005, mutations through the roof | 
  [production] | 
            
  | 08:27 | 
  <godog> | 
  set operations/puppet/cassandra git submodule repo as hidden | 
  [production] | 
            
  | 06:11 | 
  <LocalisationUpdate> | 
  ResourceLoader cache refresh completed at Tue Jul  7 06:11:46 UTC 2015 (duration 11m 45s) | 
  [production] | 
            
  | 05:51 | 
  <krinkle> | 
  Synchronized php-1.26wmf12/extensions/WikiEditor/modules/jquery.wikiEditor.toolbar.js: I3e965dda1c4 (duration: 00m 12s) | 
  [production] | 
            
  | 02:27 | 
  <LocalisationUpdate> | 
  completed (1.26wmf12) at 2015-07-07 02:27:55+00:00 | 
  [production] | 
            
  | 02:24 | 
  <l10nupdate> | 
  Synchronized php-1.26wmf12/cache/l10n: (no message) (duration: 06m 09s) | 
  [production] | 
            
  | 01:12 | 
  <ori> | 
  Re-pooled mw1152 at 20:46 UTC, did not log it then. | 
  [production] | 
            
  | 00:41 | 
  <springle> | 
  upgrade db1041 trusty | 
  [production] | 
            
  | 00:37 | 
  <krenair> | 
  Synchronized php-1.26wmf12/extensions/CentralAuth/includes/CreateLocalAccountJob.php: https://gerrit.wikimedia.org/r/#/c/223211/ (duration: 00m 13s) | 
  [production] | 
            
  
    | 
      
        2015-07-06
      
      §
     | 
  
    
  | 23:50 | 
  <krenair> | 
  Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/221989/ (duration: 00m 12s) | 
  [production] | 
            
  | 23:49 | 
  <krenair> | 
  Synchronized w/static/images/project-logos/mrwikisource.png: https://gerrit.wikimedia.org/r/#/c/221989/ (duration: 00m 13s) | 
  [production] | 
            
  | 23:36 | 
  <krenair> | 
  Synchronized wmf-config/abusefilter.php: https://gerrit.wikimedia.org/r/#/c/223179/ - should be labs-only (duration: 00m 12s) | 
  [production] | 
            
  | 23:32 | 
  <krenair> | 
  Synchronized README: https://gerrit.wikimedia.org/r/#/c/222941/ - ... (duration: 00m 13s) | 
  [production] | 
            
  | 23:27 | 
  <krenair> | 
  Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/221809/ - should be a noop, just doc changes (duration: 00m 13s) | 
  [production] | 
            
  | 23:25 | 
  <krenair> | 
  Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/221808/ (duration: 00m 13s) | 
  [production] | 
            
  | 23:17 | 
  <krenair> | 
  Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/223185/ (duration: 00m 12s) | 
  [production] | 
            
  | 23:06 | 
  <krenair> | 
  Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/220970/ (duration: 00m 14s) | 
  [production] | 
            
  | 21:46 | 
  <gwicke> | 
  restarted cassandra instance on restbase1003; was low on memory and constantly writing small chunks | 
  [production] | 
            
  | 21:30 | 
  <andrewbogott> | 
  rebooting labvirt1005, again.  Somehow virtualization is turned off again | 
  [production] | 
            
  | 21:12 | 
  <subbu> | 
  deployed parsoid version 87a746e6 | 
  [production] | 
            
  | 21:04 | 
  <ori> | 
  Synchronized php-1.26wmf12/thumb.php: cdc75debaf: Add Content-Length header to thumb.php error responses (duration: 00m 13s) | 
  [production] | 
            
  | 21:02 | 
  <mutante> | 
  purging static-bz URL on varnish ... | 
  [production] | 
            
  | 20:39 | 
  <akosiaris> | 
  upload php5_5.3.10-1ubuntu3.19-wmf1 on apt.wikimedia.org/precise-wikimedia | 
  [production] | 
            
  | 20:15 | 
  <gwicke> | 
  restart cassandra instance on 1005 | 
  [production] | 
            
  | 20:04 | 
  <mobrovac> | 
  restbase restart cassandra on rb1005 | 
  [production] | 
            
  | 19:28 | 
  <krenair> | 
  Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/223040/ (duration: 00m 12s) | 
  [production] | 
            
  | 19:11 | 
  <gwicke> | 
  reduced compaction throughput from 160 to 100 mb/s across the cassandra cluster via 'nodetool -h <host> setcompactionthroughput 100' | 
  [production] | 
            
  | 18:51 | 
  <gwicke> | 
  restarted cassandra on restbase1001 with jdk8, see T104888 | 
  [production] | 
            
  | 18:22 | 
  <gwicke> | 
  restarted cassandra on restbase1004 with jdk8 | 
  [production] | 
            
  | 17:54 | 
  <Jeff_Green> | 
  authdns-update for new rigel A record | 
  [production] | 
            
  | 17:42 | 
  <jynus> | 
  Synchronized wmf-config/db-codfw.php: increase db2029 traffic to normal levels (duration: 00m 12s) | 
  [production] | 
            
  | 17:37 | 
  <gwicke> | 
  upgraded restbase1005 to jdk8 | 
  [production] | 
            
  | 17:35 | 
  <gwicke> | 
  restarting cassandra instance on restbase1005: out of heap | 
  [production] |