| 
      
        2014-02-28
      
      §
     | 
  
    
  | 23:19 | 
  <aaron> | 
  synchronized wmf-config  'Removed unused SwiftCloudFiles extension' | 
  [production] | 
            
  | 20:15 | 
  <RobH> | 
  and i fixed it too, huzzah | 
  [production] | 
            
  | 20:14 | 
  <RobH> | 
  i killed bz during puppet run on server, fixing now | 
  [production] | 
            
  | 20:12 | 
  <RobH> | 
  fixing cert errors on etherpad.w.o, sorry if folks have service interruption | 
  [production] | 
            
  | 18:34 | 
  <Krinkle> | 
  Reloading zuul to deploy Ic3aeb6a5f13086b108 | 
  [production] | 
            
  | 14:15 | 
  <anomie> | 
  synchronized php-1.23wmf15/includes/htmlform/HTMLFormField.php  'Backport fix for bug 61942 to 1.23wmf15 (reedy already did wmf16 last night)' | 
  [production] | 
            
  | 09:39 | 
  <hashar> | 
  Jenkins restarted. | 
  [production] | 
            
  | 09:32 | 
  <hashar> | 
  restarting Jenkins, some jobs registration is broken :( | 
  [production] | 
            
  | 08:58 | 
  <andrewbogott> | 
  disabled puppet on labstore1001 to allow unattended file copies | 
  [production] | 
            
  | 03:50 | 
  <LocalisationUpdate> | 
  ResourceLoader cache refresh completed at 2014-02-28 03:50:40+00:00 | 
  [production] | 
            
  | 03:07 | 
  <reedy> | 
  updated /a/common to {{Gerrit|Icb3159198}}: Fix arrays for $wgContactConfig | 
  [production] | 
            
  | 03:06 | 
  <reedy> | 
  synchronized wmf-config/interwiki.cdb  'Updating interwiki cache' | 
  [production] | 
            
  | 03:05 | 
  <LocalisationUpdate> | 
  completed (1.23wmf16) at 2014-02-28 03:05:28+00:00 | 
  [production] | 
            
  | 02:34 | 
  <LocalisationUpdate> | 
  completed (1.23wmf15) at 2014-02-28 02:34:35+00:00 | 
  [production] | 
            
  | 02:26 | 
  <andrewbogott> | 
  rebuilt ldap indexes on virt1000 | 
  [production] | 
            
  | 02:26 | 
  <springle_> | 
  restarted apache on virt0 | 
  [production] | 
            
  | 01:38 | 
  <mutante> | 
  allowing group mwupld write in ./releases/1.19 and 1.21, the other dirs were already like that | 
  [production] | 
            
  | 01:26 | 
  <mutante> | 
  fixing permissions for 1.22.3 files on caesium, let mwupld group own them like all the other files | 
  [production] | 
            
  | 01:13 | 
  <mutante> | 
  pdf1 - revoke puppet cert, kill from stored configs,... | 
  [production] | 
            
  | 01:04 | 
  <mutante> | 
  pdf1 - disable monitoring - downtime until ∞  | 
  [production] | 
            
  | 00:06 | 
  <rdwrer> | 
  By what can only be described as kicking-down-the-door-style deployment, mwalker and I managed to deploy four FundraisingChart Jenkins jobs after about 15 tries each. | 
  [production] | 
            
  | 00:02 | 
  <mutante> | 
  restarting gitblit on antimony | 
  [production] | 
            
  | 00:02 | 
  <reedy> | 
  synchronized wmf-config/ | 
  [production] | 
            
  
    | 
      
        2014-02-27
      
      §
     | 
  
    
  | 23:56 | 
  <reedy> | 
  synchronized wmf-config/ | 
  [production] | 
            
  | 23:56 | 
  <reedy> | 
  updated /a/common to {{Gerrit|Iade498a40}}: Get rid of remaining references to $wmfExtendedVersionNumber | 
  [production] | 
            
  | 23:50 | 
  <reedy> | 
  synchronized wmf-config/ | 
  [production] | 
            
  | 23:45 | 
  <reedy> | 
  updated /a/common to {{Gerrit|Ibe70ebb6d}}: Remove ContactPageFundraiser from testwiki and donatewiki | 
  [production] | 
            
  | 23:38 | 
  <reedy> | 
  synchronized wmf-config/InitialiseSettings.php  'Remove ContactPageFundraiser from testwiki and donatewiki' | 
  [production] | 
            
  | 23:38 | 
  <mutante> | 
  streber - revoke puppet cert and salt key | 
  [production] | 
            
  | 23:36 | 
  <mutante|away> | 
  DNS update - removing streber (RT 2186) | 
  [production] | 
            
  | 21:03 | 
  <aude> | 
  synchronized php-1.23wmf15/extensions/EducationProgram | 
  [production] | 
            
  | 20:55 | 
  <aude> | 
  synchronized php-1.23wmf15/extensions/Wikidata | 
  [production] | 
            
  | 20:51 | 
  <mutante> | 
  DNS update - removing locke and squidlog which was a locke cname, locke is dead | 
  [production] | 
            
  | 20:50 | 
  <aude> | 
  synchronized php-1.23wmf16/extensions/EducationProgram | 
  [production] | 
            
  | 20:23 | 
  <bd808> | 
  Finished scap: full scap; rebuild php-1.23wmf16 l10n cache (duration: 06m 21s) | 
  [production] | 
            
  | 20:17 | 
  <bd808> | 
  Started scap: full scap; rebuild php-1.23wmf16 l10n cache | 
  [production] | 
            
  | 20:10 | 
  <bd808|deploy> | 
  scap-rebuild-cdbs failed with "Could not open directory '/upstream'." | 
  [production] | 
            
  | 20:09 | 
  <cmjohnson1> | 
  messing with serial connections ps1-b5 and ps1-b6-eqiad | 
  [production] | 
            
  | 20:09 | 
  <bd808> | 
  Finished scap: rebuild php-1.23wmf16 l10n cache (duration: 10m 20s) | 
  [production] | 
            
  | 20:08 | 
  <bd808> | 
  scap-rebuild-cdbs failed on 422 hosts | 
  [production] | 
            
  | 19:58 | 
  <bd808> | 
  Started scap: rebuild php-1.23wmf16 l10n cache | 
  [production] | 
            
  | 19:51 | 
  <aaron> | 
  rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.23wmf16 | 
  [production] | 
            
  | 19:44 | 
  <bd808|deploy> | 
  The problem with me and sync-wikiversions is my ssh-agent. It croaks when all 400+ connections happen at once. | 
  [production] | 
            
  | 19:30 | 
  <aaron> | 
  rebuilt wikiversions.cdb and synchronized wikiversions files: | 
  [production] | 
            
  | 19:28 | 
  <bd808|deploy> | 
  sync-wikiversions + dsh still doesn't like my ssh key, but I can ssh to mw1010 from tin | 
  [production] | 
            
  | 19:27 | 
  <bd808> | 
  rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.23wmf15 (third try) | 
  [production] | 
            
  | 19:26 | 
  <bd808|deploy> | 
  sync-wikiversions + dsh doesn't like my ssh key | 
  [production] | 
            
  | 19:24 | 
  <bd808> | 
  rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.23wmf15 (second try) | 
  [production] | 
            
  | 19:24 | 
  <bd808> | 
  rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.23wmf15 | 
  [production] | 
            
  | 19:23 | 
  <AaronSchulz> | 
  Manually purged the job queue aggregator | 
  [production] |