| 
      
        2014-02-28
      
      §
     | 
  
    
  | 02:26 | 
  <springle_> | 
  restarted apache on virt0 | 
  [production] | 
            
  | 01:38 | 
  <mutante> | 
  allowing group mwupld write in ./releases/1.19 and 1.21, the other dirs were already like that | 
  [production] | 
            
  | 01:26 | 
  <mutante> | 
  fixing permissions for 1.22.3 files on caesium, let mwupld group own them like all the other files | 
  [production] | 
            
  | 01:13 | 
  <mutante> | 
  pdf1 - revoke puppet cert, kill from stored configs,... | 
  [production] | 
            
  | 01:04 | 
  <mutante> | 
  pdf1 - disable monitoring - downtime until ∞  | 
  [production] | 
            
  | 00:06 | 
  <rdwrer> | 
  By what can only be described as kicking-down-the-door-style deployment, mwalker and I managed to deploy four FundraisingChart Jenkins jobs after about 15 tries each. | 
  [production] | 
            
  | 00:02 | 
  <mutante> | 
  restarting gitblit on antimony | 
  [production] | 
            
  | 00:02 | 
  <reedy> | 
  synchronized wmf-config/ | 
  [production] | 
            
  
    | 
      
        2014-02-27
      
      §
     | 
  
    
  | 23:56 | 
  <reedy> | 
  synchronized wmf-config/ | 
  [production] | 
            
  | 23:56 | 
  <reedy> | 
  updated /a/common to {{Gerrit|Iade498a40}}: Get rid of remaining references to $wmfExtendedVersionNumber | 
  [production] | 
            
  | 23:50 | 
  <reedy> | 
  synchronized wmf-config/ | 
  [production] | 
            
  | 23:45 | 
  <reedy> | 
  updated /a/common to {{Gerrit|Ibe70ebb6d}}: Remove ContactPageFundraiser from testwiki and donatewiki | 
  [production] | 
            
  | 23:38 | 
  <reedy> | 
  synchronized wmf-config/InitialiseSettings.php  'Remove ContactPageFundraiser from testwiki and donatewiki' | 
  [production] | 
            
  | 23:38 | 
  <mutante> | 
  streber - revoke puppet cert and salt key | 
  [production] | 
            
  | 23:36 | 
  <mutante|away> | 
  DNS update - removing streber (RT 2186) | 
  [production] | 
            
  | 21:03 | 
  <aude> | 
  synchronized php-1.23wmf15/extensions/EducationProgram | 
  [production] | 
            
  | 20:55 | 
  <aude> | 
  synchronized php-1.23wmf15/extensions/Wikidata | 
  [production] | 
            
  | 20:51 | 
  <mutante> | 
  DNS update - removing locke and squidlog which was a locke cname, locke is dead | 
  [production] | 
            
  | 20:50 | 
  <aude> | 
  synchronized php-1.23wmf16/extensions/EducationProgram | 
  [production] | 
            
  | 20:23 | 
  <bd808> | 
  Finished scap: full scap; rebuild php-1.23wmf16 l10n cache (duration: 06m 21s) | 
  [production] | 
            
  | 20:17 | 
  <bd808> | 
  Started scap: full scap; rebuild php-1.23wmf16 l10n cache | 
  [production] | 
            
  | 20:10 | 
  <bd808|deploy> | 
  scap-rebuild-cdbs failed with "Could not open directory '/upstream'." | 
  [production] | 
            
  | 20:09 | 
  <cmjohnson1> | 
  messing with serial connections ps1-b5 and ps1-b6-eqiad | 
  [production] | 
            
  | 20:09 | 
  <bd808> | 
  Finished scap: rebuild php-1.23wmf16 l10n cache (duration: 10m 20s) | 
  [production] | 
            
  | 20:08 | 
  <bd808> | 
  scap-rebuild-cdbs failed on 422 hosts | 
  [production] | 
            
  | 19:58 | 
  <bd808> | 
  Started scap: rebuild php-1.23wmf16 l10n cache | 
  [production] | 
            
  | 19:51 | 
  <aaron> | 
  rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.23wmf16 | 
  [production] | 
            
  | 19:44 | 
  <bd808|deploy> | 
  The problem with me and sync-wikiversions is my ssh-agent. It croaks when all 400+ connections happen at once. | 
  [production] | 
            
  | 19:30 | 
  <aaron> | 
  rebuilt wikiversions.cdb and synchronized wikiversions files: | 
  [production] | 
            
  | 19:28 | 
  <bd808|deploy> | 
  sync-wikiversions + dsh still doesn't like my ssh key, but I can ssh to mw1010 from tin | 
  [production] | 
            
  | 19:27 | 
  <bd808> | 
  rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.23wmf15 (third try) | 
  [production] | 
            
  | 19:26 | 
  <bd808|deploy> | 
  sync-wikiversions + dsh doesn't like my ssh key | 
  [production] | 
            
  | 19:24 | 
  <bd808> | 
  rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.23wmf15 (second try) | 
  [production] | 
            
  | 19:24 | 
  <bd808> | 
  rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.23wmf15 | 
  [production] | 
            
  | 19:23 | 
  <AaronSchulz> | 
  Manually purged the job queue aggregator | 
  [production] | 
            
  | 18:29 | 
  <bd808> | 
  Finished scap: testwiki to php-1.23wmf16 and rebuild l10n cache (duration: 20m 22s) | 
  [production] | 
            
  | 18:09 | 
  <bd808> | 
  Started scap: testwiki to php-1.23wmf16 and rebuild l10n cache | 
  [production] | 
            
  | 15:55 | 
  <bd808> | 
  /srv/scap, not /src/scap | 
  [production] | 
            
  | 15:55 | 
  <bd808> | 
  Eleven mediawiki-installation dsh group hosts have stale /src/scap checkouts: fenari, mw1010, mw1066, mw1091, mw1107, mw1143, mw1150, mw1189, mw1204, mw1205, mw43 | 
  [production] | 
            
  | 15:40 | 
  <akosiaris> | 
  disabled puppet on carbon, testing some autoinstall stuff | 
  [production] | 
            
  | 13:35 | 
  <Krinkle> | 
  Deploy integration/slave-scripts Icd2b25fd882b7953ee | 
  [production] | 
            
  | 07:22 | 
  <springle> | 
  synchronized wmf-config/db-eqiad.php  's1 depool db1034' | 
  [production] | 
            
  | 06:11 | 
  <Ryan_Lane2> | 
  testing salt restarts of parsoid in batches of 10% | 
  [production] | 
            
  | 03:11 | 
  <LocalisationUpdate> | 
  ResourceLoader cache refresh completed at 2014-02-27 03:11:24+00:00 | 
  [production] | 
            
  | 02:38 | 
  <LocalisationUpdate> | 
  completed (1.23wmf15) at 2014-02-27 02:37:59+00:00 | 
  [production] | 
            
  | 02:28 | 
  <springle> | 
  synchronized wmf-config/db-eqiad.php  's1 repool db1055' | 
  [production] | 
            
  | 02:22 | 
  <LocalisationUpdate> | 
  completed (1.23wmf14) at 2014-02-27 02:18:24+00:00 | 
  [production] | 
            
  | 01:31 | 
  <bd808> | 
  Finished scap: no-diff scap to test script changes (duration: 16m 44s) | 
  [production] | 
            
  | 01:14 | 
  <bd808> | 
  Started scap: no-diff scap to test script changes | 
  [production] | 
            
  | 00:56 | 
  <mwalker> | 
  updated fundraising tools on lutetium from 7c2e15b006fec25fd1881c67f6bce9e406759535 to 87dbe60d2557a38553ae2bb602f777b4096915b8 | 
  [production] |