| 2014-02-27
      
      § | 
    
  | 20:51 | <mutante> | DNS update - removing locke and squidlog which was a locke cname, locke is dead | [production] | 
            
  | 20:50 | <aude> | synchronized php-1.23wmf16/extensions/EducationProgram | [production] | 
            
  | 20:23 | <bd808> | Finished scap: full scap; rebuild php-1.23wmf16 l10n cache (duration: 06m 21s) | [production] | 
            
  | 20:17 | <bd808> | Started scap: full scap; rebuild php-1.23wmf16 l10n cache | [production] | 
            
  | 20:10 | <bd808|deploy> | scap-rebuild-cdbs failed with "Could not open directory '/upstream'." | [production] | 
            
  | 20:09 | <cmjohnson1> | messing with serial connections ps1-b5 and ps1-b6-eqiad | [production] | 
            
  | 20:09 | <bd808> | Finished scap: rebuild php-1.23wmf16 l10n cache (duration: 10m 20s) | [production] | 
            
  | 20:08 | <bd808> | scap-rebuild-cdbs failed on 422 hosts | [production] | 
            
  | 19:58 | <bd808> | Started scap: rebuild php-1.23wmf16 l10n cache | [production] | 
            
  | 19:51 | <aaron> | rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.23wmf16 | [production] | 
            
  | 19:44 | <bd808|deploy> | The problem with me and sync-wikiversions is my ssh-agent. It croaks when all 400+ connections happen at once. | [production] | 
            
  | 19:30 | <aaron> | rebuilt wikiversions.cdb and synchronized wikiversions files: | [production] | 
            
  | 19:28 | <bd808|deploy> | sync-wikiversions + dsh still doesn't like my ssh key, but I can ssh to mw1010 from tin | [production] | 
            
  | 19:27 | <bd808> | rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.23wmf15 (third try) | [production] | 
            
  | 19:26 | <bd808|deploy> | sync-wikiversions + dsh doesn't like my ssh key | [production] | 
            
  | 19:24 | <bd808> | rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.23wmf15 (second try) | [production] | 
            
  | 19:24 | <bd808> | rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.23wmf15 | [production] | 
            
  | 19:23 | <AaronSchulz> | Manually purged the job queue aggregator | [production] | 
            
  | 18:29 | <bd808> | Finished scap: testwiki to php-1.23wmf16 and rebuild l10n cache (duration: 20m 22s) | [production] | 
            
  | 18:09 | <bd808> | Started scap: testwiki to php-1.23wmf16 and rebuild l10n cache | [production] | 
            
  | 15:55 | <bd808> | /srv/scap, not /src/scap | [production] | 
            
  | 15:55 | <bd808> | Eleven mediawiki-installation dsh group hosts have stale /src/scap checkouts: fenari, mw1010, mw1066, mw1091, mw1107, mw1143, mw1150, mw1189, mw1204, mw1205, mw43 | [production] | 
            
  | 15:40 | <akosiaris> | disabled puppet on carbon, testing some autoinstall stuff | [production] | 
            
  | 13:35 | <Krinkle> | Deploy integration/slave-scripts Icd2b25fd882b7953ee | [production] | 
            
  | 07:22 | <springle> | synchronized wmf-config/db-eqiad.php  's1 depool db1034' | [production] | 
            
  | 06:11 | <Ryan_Lane2> | testing salt restarts of parsoid in batches of 10% | [production] | 
            
  | 03:11 | <LocalisationUpdate> | ResourceLoader cache refresh completed at 2014-02-27 03:11:24+00:00 | [production] | 
            
  | 02:38 | <LocalisationUpdate> | completed (1.23wmf15) at 2014-02-27 02:37:59+00:00 | [production] | 
            
  | 02:28 | <springle> | synchronized wmf-config/db-eqiad.php  's1 repool db1055' | [production] | 
            
  | 02:22 | <LocalisationUpdate> | completed (1.23wmf14) at 2014-02-27 02:18:24+00:00 | [production] | 
            
  | 01:31 | <bd808> | Finished scap: no-diff scap to test script changes (duration: 16m 44s) | [production] | 
            
  | 01:14 | <bd808> | Started scap: no-diff scap to test script changes | [production] | 
            
  | 00:56 | <mwalker> | updated fundraising tools on lutetium from 7c2e15b006fec25fd1881c67f6bce9e406759535 to 87dbe60d2557a38553ae2bb602f777b4096915b8 | [production] | 
            
  | 00:45 | <aude> | synchronized php-1.23wmf15/extensions/Wikidata  'Fix references display, for real' | [production] | 
            
  | 00:34 | <aude> | synchronized php-1.23wmf15/extensions/Wikidata  'Fix display of references' | [production] | 
            
  | 00:18 | <mflaschen> | synchronized wmf-config/InitialiseSettings.php  'Deploy GettingStarted and GuidedTour to additional wikis.  Add category-based suggestions to svwiki' | [production] | 
            
  
    | 2014-02-26
      
      § | 
    
  | 22:57 | <greg-g> | RobH set permissions on /srv/scap/docs directory to 2775 (was 2755) | [production] | 
            
  | 22:15 | <hashar> | Jenkins restarted (yeah it is that fast tonight!!!!!) | [production] | 
            
  | 22:05 | <hashar> | Restating Jenkins, it exploded. | [production] | 
            
  | 21:25 | <gwicke> | deployed Parsoid 804ead03 with deploy e22f0a0f76 | [production] | 
            
  | 21:18 | <mutante> | restarting parsoid on wtp1016 | [production] | 
            
  | 19:47 | <mutante> | DNS update - remove db9, already shutdown, (left mgmt) | [production] | 
            
  | 18:47 | <cmjohnson1> | es1006 swapping failed disk | [production] | 
            
  | 17:33 | <ottomata> | upgraded to librdkafka1 0.8.3 on cp3019, restarting varnishkafka | [production] | 
            
  | 16:39 | <paravoid> | disabling puppet on sodium | [production] | 
            
  | 15:26 | <ottomata> | rebooting analytics1004 | [production] | 
            
  | 04:17 | <ori> | enabling geo_cookie on cp1066 caused general protection fault, so reverted and restarted. | [production] | 
            
  | 03:14 | <LocalisationUpdate> | ResourceLoader cache refresh completed at 2014-02-26 03:14:37+00:00 | [production] | 
            
  | 02:41 | <LocalisationUpdate> | completed (1.23wmf15) at 2014-02-26 02:41:01+00:00 | [production] | 
            
  | 02:12 | <LocalisationUpdate> | completed (1.23wmf14) at 2014-02-26 02:12:36+00:00 | [production] |