| 2013-01-08
      
      § | 
    
  | 23:56 | <maxsem> | Finished syncing Wikimedia installation... : Weekly mobile deployment | [production] | 
            
  | 23:06 | <maxsem> | Started syncing Wikimedia installation... : Weekly mobile deployment | [production] | 
            
  | 22:49 | <maxsem> | Started syncing Wikimedia installation... : Weekly mobile deployment | [production] | 
            
  | 22:17 | <tstarling> | synchronized php-1.21wmf7/includes/objectcache/SqlBagOStuff.php | [production] | 
            
  | 22:17 | <tstarling> | synchronized php-1.21wmf7/includes/objectcache/RedisBagOStuff.php | [production] | 
            
  | 22:16 | <tstarling> | synchronized php-1.21wmf7/includes/ArrayUtils.php | [production] | 
            
  | 22:16 | <tstarling> | synchronized php-1.21wmf7/includes/AutoLoader.php | [production] | 
            
  | 21:53 | <dzahn> | synchronized ./wmf-config/InitialiseSettings.php | [production] | 
            
  | 21:32 | <reedy> | synchronized php-1.21wmf7/extensions/TimedMediaHandler | [production] | 
            
  | 19:22 | <reedy> | synchronized wmf-config/ | [production] | 
            
  | 18:51 | <reedy> | synchronized php-1.21wmf7/extensions/TimedMediaHandler | [production] | 
            
  | 17:38 | <olivneh> | synchronized docroot/wikivoyage.org/favicon.ico  'New favicon.ico for wikivoyage' | [production] | 
            
  | 17:38 | <olivneh> | synchronized wmf-config/InitialiseSettings.php  'New logo for wikivoyage & eswikivoyage' | [production] | 
            
  | 15:59 | <hashar> | upgraded Zuul | [production] | 
            
  | 15:12 | <hashar> | updating Zuul code base : 6c4d51a...23ec1ba (forced update) | [production] | 
            
  | 14:53 | <hashar> | gallium: restarted Zuul which could not recover its Gerrit connection :/ | [production] | 
            
  | 14:37 | <cmjohnson1> | rebooting solr2 to fix HT in bios | [production] | 
            
  | 14:26 | <hashar> | gallium: running git fsck  in /var/lib/git/mediawiki/extensions/Wikibase.git . That repository might have ended up being corrupted somehow :( | [production] | 
            
  | 14:22 | <cmjohnson1> | rebooting solr1 to fix HT in bios | [production] | 
            
  | 14:17 | <^demon> | manganese is back up, gerrit's running fine. | [production] | 
            
  | 14:13 | <^demon> | rebooting manganese | [production] | 
            
  | 13:38 | <apergos> | and restarted them | [production] | 
            
  | 13:38 | <apergos> | stopped and/or shot all swift processes on ms-be2, some were in an odd state, had load spike | [production] | 
            
  | 12:52 | <apergos> | cleared out /var/lg/messages.[1-x] on older ms-be hosts, several had full root partitions; disabled logging to that facility in rsyslogd conf | [production] | 
            
  | 10:37 | <hashar> | stopping Zuul | [production] | 
            
  | 10:37 | <hashar> | shutdowning jenkins for an unscheduled maintenance and upgrading the installation. | [production] | 
            
  | 10:30 | <paravoid> | reprepro update for jenkins (fetched 1.480.2) | [production] | 
            
  | 10:29 | <paravoid> | installing new backported reprepro version to {lucid,precise}-wikimedia and upgrading it in brewster | [production] | 
            
  | 09:58 | <hashar> | synchronized wmf-config/InitialiseSettings.php  '{{gerrit|42232}}  set autoconfirm count to 10 @ fawiki' | [production] | 
            
  | 09:54 | <mlitn> | synchronized wmf-config/InitialiseSettings.php  'Increase AbuseFilter disable threshold for AFT entries to 20%' | [production] | 
            
  | 09:29 | <nikerabbit> | synchronized php-1.21wmf7/extensions/Translate/resources/js/ext.translate.special.translate.js | [production] | 
            
  | 02:51 | <LocalisationUpdate> | completed (1.21wmf6) at Tue Jan  8 02:51:08 UTC 2013 | [production] | 
            
  | 02:26 | <LocalisationUpdate> | completed (1.21wmf7) at Tue Jan  8 02:26:01 UTC 2013 | [production] | 
            
  | 01:16 | <Tim> | on deployment-squid.pmtpa.wmflabs: started squid | [production] | 
            
  | 01:05 | <notpeter> | temp stopping puppet on brewster | [production] | 
            
  | 00:16 | <notpeter> | starting innobackupex on db1004 | [production] | 
            
  
    | 2013-01-07
      
      § | 
    
  | 23:54 | <paravoid> | starting nginx on ssl3002 | [production] | 
            
  | 23:26 | <aaron> | synchronized php-1.21wmf7/includes/api/ApiUpload.php  'deployed e2df801500f9472f812388006206e3e3261d58d3' | [production] | 
            
  | 23:22 | <RobH> | solr1003 coming offline for comparison work per rt 4282 | [production] | 
            
  | 22:57 | <paravoid> | powercycling cp1034, dead for 18h, evidence of sudden memory exhaustion | [production] | 
            
  | 22:38 | <aaron> | synchronized php-1.21wmf7/includes/api/ApiUpload.php  'deployed 542bfaef8f4ca61db04bb60cbd1e617c731db038' | [production] | 
            
  | 22:37 | <aaron> | synchronized wmf-config/CommonSettings.php  'set wfWikiShellCmd hook.' | [production] | 
            
  | 22:35 | <cmjohnson1> | shutting down solr3 to check bios setup | [production] | 
            
  | 22:32 | <Ryan_Lane> | repooling ssl1001 with proxy_http_version 1.1 enabled | [production] | 
            
  | 21:51 | <binasher> | restarted backend varnish on cp102[27]. cp103[0-6] | [production] | 
            
  | 21:43 | <binasher> | restarting backend varnish on cp1030 (previously restarted cp102[346]) | [production] | 
            
  | 21:42 | <maxsem> | synchronized wmf-config/CommonSettings.php | [production] | 
            
  | 21:34 | <maxsem> | synchronized wmf-config/CommonSettings.php  'Reduce the load on Solr servers in pmtpa' | [production] | 
            
  | 21:18 | <hashar> | jenkins: splitting whitespace check out of mediawiki linting job. Will make whitespace a non voting change to avoid rejecting legitimate changes. | [production] | 
            
  | 21:08 | <maxsem> | synchronized wmf-config/CommonSettings.php  'Multi-server solr' | [production] |