| 2014-08-28
      
      § | 
    
  | 23:38 | <maxsem> | Synchronized php-1.24wmf19/extensions/Flow/: https://gerrit.wikimedia.org/r/#/c/156994/ (duration: 00m 05s) | [production] | 
            
  | 23:37 | <maxsem> | Synchronized php-1.24wmf19/extensions/Echo: https://gerrit.wikimedia.org/r/#/c/157008/ (duration: 00m 04s) | [production] | 
            
  | 23:35 | <maxsem> | Synchronized php-1.24wmf19/extensions/Thanks/: https://gerrit.wikimedia.org/r/#/c/156898/ (duration: 00m 04s) | [production] | 
            
  | 23:34 | <maxsem> | Synchronized php-1.24wmf19/extensions/MobileFrontend/: https://gerrit.wikimedia.org/r/#/c/156968/ (duration: 00m 05s) | [production] | 
            
  | 23:30 | <maxsem> | Synchronized php-1.24wmf18/extensions/GlobalCssJs/: https://gerrit.wikimedia.org/r/#/c/157009/ (duration: 00m 04s) | [production] | 
            
  | 23:27 | <K4-713> | Updated fraud filters on payments | [production] | 
            
  | 22:52 | <aaron> | Synchronized php-1.24wmf18/maintenance/findMissingFiles.php: (no message) (duration: 00m 07s) | [production] | 
            
  | 22:15 | <mutante> | restarted tools.morebots production instance - can i log now? | [production] | 
            
  | 22:13 | <cmjohnson1> | ms-fe1003 down for relocation | [production] | 
            
  | 22:13 | <mutante> | test | [production] | 
            
  | 21:15 | <robh> | bast2001.wikimedia.org now online in codfw. | [production] | 
            
  | 21:15 | <robh> | i never admin logged when install2001.wikimedia.org went online the other day, opps. | [production] | 
            
  | 21:15 | <ori> | last sync was of Iac37a2369: resourceloader: Don't register raw modules client-side | [production] | 
            
  | 21:14 | <ori> | Synchronized php-1.24wmf18/includes/resourceloader/ResourceLoaderStartUpModule.php: (no message) (duration: 00m 03s) | [production] | 
            
  | 20:57 | <krinkle> | Synchronized php-1.24wmf19/includes/resourceloader/ResourceLoaderStartUpModule.php: fd5b963458c19 (duration: 00m 06s) | [production] | 
            
  | 20:33 | <ottomata> | shutting down elastic1016 | [production] | 
            
  | 20:16 | <ottomata> | temporarily disable puppet on gadolinium | [production] | 
            
  | 19:36 | <reedy> | Synchronized wmf-config/: (no message) (duration: 00m 14s) | [production] | 
            
  | 19:12 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.24wmf19 | [production] | 
            
  | 19:09 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.24wmf18 | [production] | 
            
  | 19:07 | <reedy> | Finished scap: testwiki to 1.24wmf19 (duration: 43m 00s) | [production] | 
            
  | 18:25 | <godog> | install build-essential and fakeroot on tin | [production] | 
            
  | 18:24 | <reedy> | Started scap: testwiki to 1.24wmf19 | [production] | 
            
  | 17:26 | <aaron> | Synchronized rpc: 9564e93ecd4953126d91b99d7728f63401a4dc86 (duration: 00m 07s) | [production] | 
            
  | 17:13 | <^d> | elastic: excluded the elastic1016 node from shard allocation, shards draining so we can take it down for disk testing | [production] | 
            
  | 16:02 | <ottomata> | restarted webstats-collector on gadolinium | [production] | 
            
  | 13:18 | <mark> | Reactivated cr2-eqiad AS3257 transit link | [production] | 
            
  | 10:44 | <springle> | xtrabackup clone db1051 to db1073 | [production] | 
            
  | 10:18 | <godog> | restarting mailman on sodium | [production] | 
            
  | 08:52 | <godog> | restarted apache on mw1134 | [production] | 
            
  | 08:03 | <godog> | killed stray mailman processes on sodium (no pid file) and restarted mailman | [production] | 
            
  | 06:12 | <springle> | xtrabackup clone db1051 to db1072 | [production] | 
            
  
    | 2014-08-26
      
      § | 
    
  | 21:04 | <hashar> | Updating our Jenkins Job Builder fork 0268581..e5c0c61 . Will let us define variables in 'default' section and override them when invoking a job template ( https://review.openstack.org/#/c/100020/ ) | [production] | 
            
  | 19:58 | <bd808> | Ran sync-common on mw1053.eqiad.wmnet to recover from failure during last scap | [production] | 
            
  | 19:48 | <aude> | Finished scap: Update new messages for Wikibase (duration: 07m 16s) | [production] | 
            
  | 19:41 | <aude> | Started scap: Update new messages for Wikibase | [production] | 
            
  | 19:39 | <aude> | Synchronized wmf-config/Wikibase.php: add Wikibase badges css setting (duration: 00m 10s) | [production] | 
            
  | 19:26 | <aude> | Synchronized wmf-config/Wikibase.php: enable new serialization format for wikidata (duration: 00m 08s) | [production] | 
            
  | 19:10 | <reedy> | Synchronized php-1.24wmf18/extensions/Echo/: (no message) (duration: 00m 14s) | [production] | 
            
  | 19:05 | <aude> | Synchronized wmf-config/Wikibase.php: enable otherprojects sidebar beta feature (duration: 00m 15s) | [production] | 
            
  | 18:55 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.24wmf18 | [production] | 
            
  | 18:53 | <reedy> | Synchronized php-1.24wmf18/extensions/MassMessage: (no message) (duration: 00m 14s) | [production] | 
            
  | 18:53 | <reedy> | Synchronized php-1.24wmf17/extensions/MassMessage: (no message) (duration: 00m 16s) | [production] | 
            
  | 18:19 | <jgage> | Failover from analytics1010-eqiad-wmnet to analytics1004-eqiad-wmnet successful | [production] | 
            
  | 17:47 | <bd808> | Synchronized private/PrivateSettings.php: Syncing file rather than symlink (duration: 00m 04s) | [production] | 
            
  | 17:36 | <bd808> | mw1010.eqiad.wmnet was out of sync too. I suspect there is something wrong with the fanout update step in scap | [production] | 
            
  | 17:26 | <bd808> | /usr/local/apache/common-local out of date on mw1161.eqiad.wmnet; updated via sync-common | [production] | 
            
  | 17:25 | <bd808> | sync-* not updating terbium properly; sync-common from terbium manually got several config changes; maybe a problem with mw1161.eqiad.wmnet rsync mirror | [production] | 
            
  | 17:14 | <demon> | Synchronized wmf-config/InitialiseSettings.php: touch (duration: 00m 04s) | [production] | 
            
  | 17:12 | <demon> | Synchronized wmf-config/PrivateSettings.php: adjust swift auth url for cirrus (duration: 00m 04s) | [production] |