| 2014-11-20
      
      § | 
    
  | 00:50 | <maxsem> | Synchronized php-1.25wmf9/extensions/MobileFrontend/: https://gerrit.wikimedia.org/r/#/c/174613/ (duration: 00m 04s) | [production] | 
            
  | 00:41 | <catrope> | Synchronized wmf-config/InitialiseSettings.php: Change mobile wordmark image to relative URL (duration: 00m 04s) | [production] | 
            
  | 00:34 | <catrope> | Synchronized php-1.25wmf9/extensions/VisualEditor: SWAT (duration: 00m 04s) | [production] | 
            
  | 00:34 | <catrope> | Synchronized php-1.25wmf8/extensions/VisualEditor: SWAT (duration: 00m 04s) | [production] | 
            
  | 00:26 | <catrope> | Synchronized php-1.25wmf8/includes/media/: SWAT: don't apply EXIF rotation to chained thumbnails (duration: 00m 04s) | [production] | 
            
  | 00:13 | <catrope> | Synchronized wmf-config/: SWAT: temp debugging for SecurePoll (duration: 00m 04s) | [production] | 
            
  | 00:10 | <catrope> | Synchronized wmf-config/: SWAT (duration: 00m 04s) | [production] | 
            
  | 00:09 | <catrope> | Synchronized images/mobile/: SWAT: new Wikipedia wordmark for mobile (duration: 00m 03s) | [production] | 
            
  
    | 2014-11-19
      
      § | 
    
  | 22:01 | <ottomata> | starting trusty upgrade of analytics1041 | [production] | 
            
  | 21:38 | <yuvipandajs> | restarted txstatsd & carbon on labmon1001, recovering from missing points now | [production] | 
            
  | 21:35 | <reedy> | Synchronized wmf-config/: ContactPage for legal (duration: 00m 17s) | [production] | 
            
  | 21:34 | <reedy> | Synchronized docroot and w: (no message) (duration: 00m 15s) | [production] | 
            
  | 21:25 | <ottomata> | starting trusty upgrade of analytics1040 | [production] | 
            
  | 21:22 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.25wmf9 | [production] | 
            
  | 21:21 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.25wmf8 | [production] | 
            
  | 21:18 | <reedy> | Finished scap: testwiki to 1.25wmf9 and build l10n cache (duration: 105m 06s) | [production] | 
            
  | 20:57 | <ottomata> | starting trusty upgrade of analytics1038 | [production] | 
            
  | 20:23 | <ottomata> | starting trusty upgrade of analytics1038 | [production] | 
            
  | 19:49 | <bblack> | starting the long slow process of draining out esams traffic ahead of power maint event | [production] | 
            
  | 19:34 | <ottomata> | starting trusty upgrade of analytics1037 | [production] | 
            
  | 19:33 | <reedy> | Started scap: testwiki to 1.25wmf9 and build l10n cache | [production] | 
            
  | 19:30 | <andrewbogott> | upgrading other compute hosts:  virt1001-1009 | [production] | 
            
  | 19:25 | <ori> | disarming keyholder agent on tin to test alerts | [production] | 
            
  | 18:53 | <reedy> | Synchronized php-1.25wmf8/extensions/Wikidata: Ie105a80aa776769eb0dae8a44cda0b7dbe018fb5 (duration: 00m 22s) | [production] | 
            
  | 18:50 | <andrewbogott> | upgrading virt1006 | [production] | 
            
  | 18:36 | <andrewbogott> | upgrading labnet1001 | [production] | 
            
  | 18:33 | <andrewbogott> | upgraded glance on virt1000 to version icehouse | [production] | 
            
  | 18:29 | <ottomata> | starting trusty upgrade of 1036 | [production] | 
            
  | 18:18 | <reedy> | Synchronized wmf-config/: BounceHandler (duration: 00m 15s) | [production] | 
            
  | 18:17 | <reedy> | Synchronized php-1.25wmf8/extensions/BounceHandler/: Bump (duration: 00m 14s) | [production] | 
            
  | 17:57 | <andrewbogott> | disabled keystone-redis because the current package doesn't work with icehouse | [production] | 
            
  | 16:36 | <andrewbogott> | upgrading virt1000 | [production] | 
            
  | 16:16 | <andrewbogott> | moved virt1000 db backup to /a/osback because it was /way/ too big to fit in my homedir | [production] | 
            
  | 16:12 | <hashar> | Jenkins: uninstalled Jenkins statsd plugin ( https://phabricator.wikimedia.org/T1278 ). It is overloading the statsd server with a bunch of metrics we don't care about ( https://phabricator.wikimedia.org/T1075 ) | [production] | 
            
  | 15:57 | <andrewbogott> | backed up all labs openstack databases to virt1000:~andrew/osback/havana-db-backup.sql | [production] | 
            
  | 15:49 | <andrewbogott> | backing up labs configs in ~andrew/osback/<servicename> | [production] | 
            
  | 15:48 | <andrewbogott> | beginning upgrade of labs OpenStack from Havana to Icehouse | [production] | 
            
  | 09:44 | <hoo> | Synchronized php-1.25wmf8/extensions/Wikidata/: Fix url and commonsMedia UI editing (duration: 00m 42s) | [production] | 
            
  | 05:44 | <legoktm> | batchCAAntiSpoof finished with "34721605 user(s) done." | [production] | 
            
  | 04:57 | <ejegg> | updated tools from 419fb7aa32c6d0776056968378e358ee01985565 to fe9b463379fac35ad5e71a57fbbb95ae39e2356e | [production] | 
            
  | 04:42 | <Tim> | on mw1114: testing xhprof hack for T758 | [production] | 
            
  | 04:15 | <ori> | mw1020: disabled puppet & restarted hhvm w/hhvm.eval.perf_pid_map = true to test | [production] | 
            
  | 04:15 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Wed Nov 19 04:15:00 UTC 2014 (duration 14m 59s) | [production] | 
            
  | 02:26 | <LocalisationUpdate> | completed (1.25wmf8) at 2014-11-19 02:26:12+00:00 | [production] | 
            
  | 02:14 | <LocalisationUpdate> | completed (1.25wmf7) at 2014-11-19 02:14:12+00:00 | [production] | 
            
  | 01:28 | <qchris> | Restarted EventLogging mysql-m2 consumer to pick up switch to dbproxy1002 | [production] | 
            
  | 01:23 | <ori> | Synchronized php-1.25wmf7/extensions/SyntaxHighlight_GeSHi: I788e1beb8: Update SyntaxHighlight_GeSHi for cherry-picks (duration: 00m 05s) | [production] | 
            
  | 01:20 | <ori> | Synchronized php-1.25wmf8/extensions/SyntaxHighlight_GeSHi: Ibb0f7c24: Update SyntaxHighlight_GeSHi for cherry-picks (duration: 00m 05s) | [production] | 
            
  | 00:48 | <springle> | m2-master CNAME switch to dbproxy1002, and db1046 to primary backend | [production] | 
            
  | 00:38 | <ori> | EventLogging: deployed 423f7dd5b2b5 & restarted. | [production] |