| 2014-11-19
      
      § | 
    
  | 18:33 | <andrewbogott> | upgraded glance on virt1000 to version icehouse | [production] | 
            
  | 18:29 | <ottomata> | starting trusty upgrade of 1036 | [production] | 
            
  | 18:18 | <reedy> | Synchronized wmf-config/: BounceHandler (duration: 00m 15s) | [production] | 
            
  | 18:17 | <reedy> | Synchronized php-1.25wmf8/extensions/BounceHandler/: Bump (duration: 00m 14s) | [production] | 
            
  | 17:57 | <andrewbogott> | disabled keystone-redis because the current package doesn't work with icehouse | [production] | 
            
  | 16:36 | <andrewbogott> | upgrading virt1000 | [production] | 
            
  | 16:16 | <andrewbogott> | moved virt1000 db backup to /a/osback because it was /way/ too big to fit in my homedir | [production] | 
            
  | 16:12 | <hashar> | Jenkins: uninstalled Jenkins statsd plugin ( https://phabricator.wikimedia.org/T1278 ). It is overloading the statsd server with a bunch of metrics we don't care about ( https://phabricator.wikimedia.org/T1075 ) | [production] | 
            
  | 15:57 | <andrewbogott> | backed up all labs openstack databases to virt1000:~andrew/osback/havana-db-backup.sql | [production] | 
            
  | 15:49 | <andrewbogott> | backing up labs configs in ~andrew/osback/<servicename> | [production] | 
            
  | 15:48 | <andrewbogott> | beginning upgrade of labs OpenStack from Havana to Icehouse | [production] | 
            
  | 09:44 | <hoo> | Synchronized php-1.25wmf8/extensions/Wikidata/: Fix url and commonsMedia UI editing (duration: 00m 42s) | [production] | 
            
  | 05:44 | <legoktm> | batchCAAntiSpoof finished with "34721605 user(s) done." | [production] | 
            
  | 04:57 | <ejegg> | updated tools from 419fb7aa32c6d0776056968378e358ee01985565 to fe9b463379fac35ad5e71a57fbbb95ae39e2356e | [production] | 
            
  | 04:42 | <Tim> | on mw1114: testing xhprof hack for T758 | [production] | 
            
  | 04:15 | <ori> | mw1020: disabled puppet & restarted hhvm w/hhvm.eval.perf_pid_map = true to test | [production] | 
            
  | 04:15 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Wed Nov 19 04:15:00 UTC 2014 (duration 14m 59s) | [production] | 
            
  | 02:26 | <LocalisationUpdate> | completed (1.25wmf8) at 2014-11-19 02:26:12+00:00 | [production] | 
            
  | 02:14 | <LocalisationUpdate> | completed (1.25wmf7) at 2014-11-19 02:14:12+00:00 | [production] | 
            
  | 01:28 | <qchris> | Restarted EventLogging mysql-m2 consumer to pick up switch to dbproxy1002 | [production] | 
            
  | 01:23 | <ori> | Synchronized php-1.25wmf7/extensions/SyntaxHighlight_GeSHi: I788e1beb8: Update SyntaxHighlight_GeSHi for cherry-picks (duration: 00m 05s) | [production] | 
            
  | 01:20 | <ori> | Synchronized php-1.25wmf8/extensions/SyntaxHighlight_GeSHi: Ibb0f7c24: Update SyntaxHighlight_GeSHi for cherry-picks (duration: 00m 05s) | [production] | 
            
  | 00:48 | <springle> | m2-master CNAME switch to dbproxy1002, and db1046 to primary backend | [production] | 
            
  | 00:38 | <ori> | EventLogging: deployed 423f7dd5b2b5 & restarted. | [production] | 
            
  | 00:21 | <maxsem> | Synchronized wmf-config/mobile.php: https://gerrit.wikimedia.org/r/174303 (duration: 00m 05s) | [production] | 
            
  | 00:09 | <MaxSem> | gracefulled apache on mw1205 (suspect an APC bug) | [production] | 
            
  | 00:06 | <ori> | repooled mw1205 | [production] | 
            
  
    | 2014-11-18
      
      § | 
    
  | 23:34 | <ori> | Synchronized wmf-config: I76f2023a1: 'Undeploy AntiBot' (duration: 00m 04s) | [production] | 
            
  | 23:19 | <demon> | Synchronized wmf-config: undeploy antibot (duration: 00m 04s) | [production] | 
            
  | 22:46 | <bd808> | Updated zuul config on gallium to include I511b14e (Make cdb-phpunit job non-voting) | [production] | 
            
  | 22:27 | <^d> | elasticsearch: set a template to apply auto_expand_replicas 0-2 on all newly created indexes. | [production] | 
            
  | 22:00 | <hoo> | Synchronized wmf-config/Wikibase.php: Bump cache epoch (duration: 00m 07s) | [production] | 
            
  | 21:50 | <hoo> | Synchronized php-1.25wmf8/extensions/Wikidata/: Fix EntityIdLabelFormatter et al. (duration: 00m 17s) | [production] | 
            
  | 21:33 | <^d> | elasticsearch: set auto_expand_replicas to 0-2 on ttmserver(-test) like other indexes for extra redundancy. | [production] | 
            
  | 20:25 | <ori> | re-enabling puppet on all varnishes following deployment of Iac35f2329 | [production] | 
            
  | 20:10 | <legoktm> | running batchCAAntiSpoof.php on terbium | [production] | 
            
  | 20:02 | <legoktm> | ran populateGlobalRenameLogSearch.php on metawiki | [production] | 
            
  | 19:47 | <ori> | disabling Puppet on varnishes to push out Iac35f2329 | [production] | 
            
  | 19:40 | <reedy> | Synchronized wmf-config/Wikibase.php: bump epoch (duration: 00m 13s) | [production] | 
            
  | 19:40 | <ottomata> | starting trusty upgrade of analytics1032 | [production] | 
            
  | 19:28 | <reedy> | Synchronized wmf-config/: (no message) (duration: 00m 14s) | [production] | 
            
  | 19:23 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.25wmf8 | [production] | 
            
  | 19:14 | <ottomata> | starting trusty upgrade of analytics1031 | [production] | 
            
  | 19:05 | <ori> | depooled mw1205; out of sync | [production] | 
            
  | 19:03 | <maxsem> | Synchronized php-1.25wmf7/extensions/WikiGrok/: Retry sync (duration: 00m 07s) | [production] | 
            
  | 18:57 | <maxsem> | Synchronized php-1.25wmf7/extensions/WikiGrok/: Revert (duration: 00m 04s) | [production] | 
            
  | 18:54 | <maxsem> | Synchronized php-1.25wmf7/extensions/WikiGrok/: SQL backed version (duration: 00m 04s) | [production] | 
            
  | 18:49 | <maxsem> | Synchronized php-1.25wmf8/extensions/WikiGrok/: SQL backed version (duration: 00m 05s) | [production] | 
            
  | 18:47 | <ejegg> | updated crm from 71ec68e8da1de289c4e7adca090c0fdbccbd8b8a to ff89895638a0dd0600b2e4c0b6adfd1b8e402df5 | [production] | 
            
  | 18:47 | <ejegg> | updated crm | [production] |