| 2014-07-10
      
      § | 
    
  | 23:38 | <mwalker> | Finished scap: Updating Core, VE, and GuidedTour for scap, {{gerrit|145400}}, {{gerrit|145401}}, {{gerrit|145431}}, and {{gerrit|145460}} (duration: 16m 26s) | [production] | 
            
  | 23:22 | <mwalker> | Started scap: Updating Core, VE, and GuidedTour for scap, {{gerrit|145400}}, {{gerrit|145401}}, {{gerrit|145431}}, and {{gerrit|145460}} | [production] | 
            
  | 20:00 | <reedy> | Synchronized wmf-config/: (no message) (duration: 00m 14s) | [production] | 
            
  | 19:46 | <reedy> | Synchronized private: (no message) (duration: 00m 14s) | [production] | 
            
  | 19:45 | <csteipp> | deployed patch for bug65778 | [production] | 
            
  | 19:43 | <hashar> | Jenkins upgrading Gearman plugin from 0.0.6 to 0.0.7 . That fix the way jobs labels are registered with Gearman | [production] | 
            
  | 19:16 | <hashar> | Killed jenkins :-( | [production] | 
            
  | 18:37 | <reedy> | Synchronized wmf-config/: (no message) (duration: 00m 13s) | [production] | 
            
  | 18:36 | <reedy> | Synchronized database lists: (no message) (duration: 00m 14s) | [production] | 
            
  | 18:10 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.24wmf13 | [production] | 
            
  | 18:02 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.24wmf12 | [production] | 
            
  | 17:26 | <hoo> | Synchronized php-1.24wmf13/extensions/Wikidata/: Fix a UI issue and two API related flaws (same version as for wmf12) (duration: 00m 09s) | [production] | 
            
  | 17:21 | <hoo> | Synchronized php-1.24wmf12/extensions/Wikidata/: Fix a UI issue and two API related flaws (duration: 00m 14s) | [production] | 
            
  | 16:04 | <godog> | restarted pdns in turn on virt1000 and virt0 after opendj ulimit change | [production] | 
            
  | 15:56 | <hashar> | gallium running a rather long du command in a screen. Need to have a good figure at how much disk space each jobs consume | [production] | 
            
  | 15:50 | <reedy> | Finished scap: testwiki to 1.24wmf13 and build l10n cache (duration: 32m 09s) | [production] | 
            
  | 15:18 | <reedy> | Started scap: testwiki to 1.24wmf13 and build l10n cache | [production] | 
            
  | 15:15 | <ottomata> | reinstalling analytics1026 and analytics1027 | [production] | 
            
  | 14:10 | <godog> | ran swift-dispersion-populate on eqiad and esams swift clusters | [production] | 
            
  | 14:04 | <godog> | cycle-restarting swift proxy-server on ms-fe to apply config updates | [production] | 
            
  | 13:09 | <godog> | restart pdns on virt1000 | [production] | 
            
  | 12:48 | <springle> | ongoing schema changes: pl_from_namespace gerrit 117373. on terbium, osc_host.sh processes ok to kill in emergency | [production] | 
            
  | 12:43 | <godog> | restart opendj on virt1000 with higher ulimit -n | [production] | 
            
  | 12:29 | <godog> | restarted opendj on virt1000, ran out of fd | [production] | 
            
  | 10:29 | <godog> | restart profiler-to-carbon on tungsten, seemingly cpu spinning | [production] | 
            
  | 09:48 | <reedy> | Synchronized database lists: (no message) (duration: 00m 13s) | [production] | 
            
  | 09:30 | <oblivian> | gracefulled all apaches | [production] | 
            
  | 09:23 | <_joe_> | doing a tagged run to sync apache config | [production] | 
            
  | 09:07 | <hashar> | gallium err was July 5th and file was from a minute ago ... ignore me | [production] | 
            
  | 09:06 | <hashar> | gallium deleted /var/lib/puppet/state/agent_catalog_run.lock  from July 5th. Was preventing me to run <tt>puppet agent -tv</tt> | [production] | 
            
  | 08:02 | <oblivian> | gracefulled all apaches | [production] | 
            
  | 07:52 | <_joe_> | doing a tagged run of puppet on all appservers to sync apache config | [production] | 
            
  | 06:40 | <bblack> | all normally-ulsfo traffic is back on ulsfo | [production] | 
            
  | 05:53 | <awight> | edit CRM Drupal permissions | [production] | 
            
  | 03:47 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Thu Jul 10 03:46:36 UTC 2014 (duration 46m 35s) | [production] | 
            
  | 03:12 | <LocalisationUpdate> | completed (1.24wmf12) at 2014-07-10 03:11:48+00:00 | [production] | 
            
  | 02:49 | <mutante> | argon,netmon1001, graceful'led apaches | [production] | 
            
  | 02:48 | <mutante> | netmon1001 - DocumentRoot [/etc/apache2/undef] does not exist | [production] | 
            
  | 02:42 | <LocalisationUpdate> | completed (1.24wmf11) at 2014-07-10 02:41:29+00:00 | [production] | 
            
  | 02:38 | <mutante> | argon,calcium,iron,rhenium,bast1001,oxygen,netmon1001 - upgraded SSL | [production] | 
            
  | 01:47 | <mutante> | argon - Ignoring file 'puppet_base_2.7' in directory '/etc/apt/preferences.d/ | [production] | 
            
  | 01:41 | <awight> | update crm schema to wmf_civicrm 7020 | [production] |