| 2013-08-26
      
      § | 
    
  | 19:17 | <reedy> | synchronized wmf-config/CommonSettings.php | [production] | 
            
  | 19:08 | <reedy> | synchronized database lists files: wikivoyage, wikiversity and wiktioanry to 1.22wmf14 | [production] | 
            
  | 19:08 | <RobH> | for operations/dns.git | [production] | 
            
  | 19:08 | <RobH> | i accidentally (yea right) pushed .gitreview to master rather than head, so no gerrit log for my change, opps! | [production] | 
            
  | 18:59 | <reedy> | synchronized wmf-config/CommonSettings.php | [production] | 
            
  | 18:49 | <reedy> | synchronized wmf-config/ | [production] | 
            
  | 18:40 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: special, closed and wikimedia to 1.22wmf14 | [production] | 
            
  | 18:28 | <reedy> | synchronized wmf-config/InitialiseSettings.php  'Fix wmgHTTPSBlacklistCountries' | [production] | 
            
  | 17:06 | <csteipp> | synchronized php-1.22wmf13/extensions/LiquidThreads  'Fix bug53320' | [production] | 
            
  | 17:05 | <csteipp> | synchronized php-1.22wmf14/extensions/LiquidThreads  'Fix bug53320' | [production] | 
            
  | 16:55 | <demon> | synchronized wmf-config/CommonSettings.php  'GeoIP configuration for secure login' | [production] | 
            
  | 16:54 | <demon> | synchronized wmf-config/InitialiseSettings.php  'GeoIP configuration for secure login' | [production] | 
            
  | 16:52 | <demon> | synchronized wmf-config/InitialiseSettings.php  'mediawikiwiki to secure login' | [production] | 
            
  | 16:37 | <ottomata> | taking down analytics1006, analytics1008 and analytics1009 for reinstall | [production] | 
            
  | 16:29 | <ottomata> | bringing analytics1005 down for reinstall | [production] | 
            
  | 02:24 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Mon Aug 26 02:24:25 UTC 2013 | [production] | 
            
  | 02:16 | <LocalisationUpdate> | completed (1.22wmf14) at Mon Aug 26 02:16:07 UTC 2013 | [production] | 
            
  | 02:09 | <LocalisationUpdate> | completed (1.22wmf13) at Mon Aug 26 02:09:13 UTC 2013 | [production] | 
            
  
    | 2013-08-25
      
      § | 
    
  | 16:49 | <RoanKattouw> | Restarted Parsoid on a couple of boxes to clear some disk space. It appears wtp1001-1004 have been set up with larger, LVMed root partitions but I can't find any logs on when this was done, by whom, or why | [production] | 
            
  | 16:44 | <RoanKattouw> | Restarted Parsoid on wtp1012, was unresponsive | [production] | 
            
  | 16:40 | <RoanKattouw> | Strike that, my account on fenari is just being weird apparently | [production] | 
            
  | 16:39 | <RoanKattouw> | Investigating low disk space warnings on wtp1001-1024, but it looks like something weird happened to them, I get host key verification warnings or failures for all of them | [production] | 
            
  | 02:37 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Sun Aug 25 02:37:03 UTC 2013 | [production] | 
            
  | 02:25 | <LocalisationUpdate> | completed (1.22wmf14) at Sun Aug 25 02:25:16 UTC 2013 | [production] | 
            
  | 02:13 | <LocalisationUpdate> | completed (1.22wmf13) at Sun Aug 25 02:13:19 UTC 2013 | [production] | 
            
  | 01:03 | <demon> | Finished syncing Wikimedia installation... : | [production] | 
            
  | 00:52 | <demon> | Started syncing Wikimedia installation... : | [production] | 
            
  
    | 2013-08-23
      
      § | 
    
  | 23:57 | <mutante> | removing mailing list Wikipedia-ensino, creating list educacao-br | [production] | 
            
  | 20:57 | <marc> | synchronized docroot | [production] | 
            
  | 20:52 | <ori-l> | Re-starting eventlogging/consumer NAME=mysql-db1047 on vanadium | [production] | 
            
  | 20:43 | <root> | synchronized docroot | [production] | 
            
  | 20:37 | <ori-l> | Stopped eventlogging/consumer NAME=mysql-db1047 on vanadium | [production] | 
            
  | 20:33 | <binasher> | taking mysqld down on db1047 (s1 analytics + eventlog data) for unplanned maintenance | [production] | 
            
  | 19:53 | <RobH> | ns2.wikimedia.org glue record updated from 91.198.174.4 to 91.198.174.239 | [production] | 
            
  | 18:19 | <awight> | civicrm updated from e4f1228d954ec239911143 to 5c28984d7e46c95484 | [production] | 
            
  | 17:18 | <ottomata> | disabling puppet on analytics 1006, 1008, 1009, 1021 and 1022 in preparation for reinstallation and repaving next week | [production] | 
            
  | 17:18 | <awight> | updated civicrm from 257686e81cf6 to e4f1228d954ec23991 | [production] | 
            
  | 16:27 | <reedy> | synchronized php-1.22wmf14/extensions/WikiEditor/ | [production] | 
            
  | 16:25 | <reedy> | synchronized php-1.22wmf13/extensions/WikiEditor/ | [production] | 
            
  | 16:19 | <reedy> | synchronized wmf-config/ | [production] | 
            
  | 15:44 | <ottomata> | disabling puppet on analytics1024 and analytics1025 in preperation for reinstall of zookeeper nodes | [production] | 
            
  | 15:06 | <ottomata> | taking analytics1023 offline for reinstall | [production] | 
            
  | 10:15 | <paravoid> | upgrading ceph to v0.67.2 | [production] | 
            
  | 07:27 | <springle> | enabled dns reverse-lookups on db1016 mysql while replication catches up, to avoid slave sql thread stopping on SET PASSWORD / GRANT statements using hostnames | [production] | 
            
  | 06:08 | <springle> | ran mysql_upgrade on db1001 (due to missing/incorrect system tables errors in log) | [production] |