| 2013-07-23
      
      § | 
    
  | 21:27 | <reedy> | synchronized wmf-config/CommonSettings.php | [production] | 
            
  | 20:55 | <mwalker> | updating civicrm on aluminium from e4062d012d0ccfb6220fc3703fd42ad894151ec5 to 180ecd219da68a0b950dafb036078b7c285e2437 | [production] | 
            
  | 20:49 | <mwalker> | kludging payments data -- adding 7 hours to contribution timestamps between IDs 4380557 and 4605567 | [production] | 
            
  | 20:35 | <mwalker> | paused all audit and QC jenkins jobs on the payments cluster in preparation for fixing timestamps | [production] | 
            
  | 19:54 | <reedy> | synchronized php-1.22wmf11/extensions/Wikibase | [production] | 
            
  | 19:46 | <reedy> | synchronized php-1.22wmf11/extensions/Wikibase | [production] | 
            
  | 19:29 | <Jeff_Green> | dist-upgrade & reboot for payments* | [production] | 
            
  | 18:39 | <reedy> | synchronized php-1.22wmf10/extensions/Wikibase | [production] | 
            
  | 18:08 | <reedy> | synchronized wmf-config/InitialiseSettings.php | [production] | 
            
  | 18:06 | <reedy> | synchronized wmf-config/ | [production] | 
            
  | 18:02 | <reedy> | synchronized database lists files: | [production] | 
            
  | 18:00 | <reedy> | synchronized wmf-config/ | [production] | 
            
  | 17:58 | <Reedy> | Populated sites table on all wikivoyage wikis | [production] | 
            
  | 17:40 | <reedy> | synchronized wmf-config/ | [production] | 
            
  | 17:31 | <reedy> | synchronized wmf-config/ | [production] | 
            
  | 17:27 | <reedy> | synchronized php-1.22wmf11/extensions/Wikibase | [production] | 
            
  | 17:25 | <reedy> | synchronized php-1.22wmf10/extensions/Wikibase | [production] | 
            
  | 17:13 | <aaron> | synchronized wmf-config/InitialiseSettings.php  '26162fbc97ebbe25fee84bb333b0b0b2efe4c447' | [production] | 
            
  | 16:40 | <Jeff_Green> | authdns-update to propagate fundraisingdb-* cname changes | [production] | 
            
  | 15:57 | <Jeff_Green> | stop icinga notifications for db1025 mysql | [production] | 
            
  | 15:57 | <Jeff_Green> | stop icinga notifications for db1008 | [production] | 
            
  | 15:21 | <Jeff_Green> | dist-upgrade &reboot db1025 | [production] | 
            
  | 15:14 | <reedy> | synchronized wmf-config/InitialiseSettings.php | [production] | 
            
  | 15:13 | <reedy> | synchronized database lists files: | [production] | 
            
  | 15:13 | <Jeff_Green> | dist-upgrade & reboot payments2-4 | [production] | 
            
  | 14:57 | <Jeff_Green> | dist-upgrade & reboot payments2 | [production] | 
            
  | 14:47 | <ottomata> | stopping puppet on gadolinium, looking into filters causing socat process packet loss | [production] | 
            
  | 14:22 | <hashar> | Jenkins: lanthanum has been added as a Jenkins slave node though NO job should be running there.e | [production] | 
            
  | 10:46 | <mark> | synchronized wmf-config/squid.php  'Add ssl1005/ssl1006' | [production] | 
            
  | 05:44 | <paravoid> | pooling ssl1005/1006 for https/ipv6 | [production] | 
            
  | 05:34 | <paravoid> | authdns-update for cr1-ulsfo <-> cr2-eqiad, dns-auth-lb -> nsN(-new), ssl100x fixed IPv6 | [production] | 
            
  | 02:21 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Tue Jul 23 02:21:20 UTC 2013 | [production] | 
            
  | 02:12 | <LocalisationUpdate> | completed (1.22wmf10) at Tue Jul 23 02:12:36 UTC 2013 | [production] | 
            
  | 02:08 | <LocalisationUpdate> | completed (1.22wmf11) at Tue Jul 23 02:08:03 UTC 2013 | [production] | 
            
  | 01:01 | <Ryan_Lane> | enabled hyperthreading on ssl1005 | [production] | 
            
  | 00:39 | <Ryan_Lane> | depooling ssl1005/6 due to ipv6 issues | [production] | 
            
  | 00:39 | <Ryan_Lane> | pooled new https node ssl1006.wikimedia.org (with hyperthreading enabled) | [production] | 
            
  | 00:38 | <Ryan_Lane> | pooled new https node ssl1005.wikimedia.org | [production] | 
            
  
    | 2013-07-22
      
      § | 
    
  | 22:31 | <reedy> | synchronized php-1.22wmf11/extensions/UploadWizard | [production] | 
            
  | 22:03 | <cmjohnson1> | restarting ns2 | [production] | 
            
  | 21:59 | <cmjohnson1> | dns update | [production] | 
            
  | 21:47 | <LeslieCarr> | restarting powerdns on ns1 | [production] | 
            
  | 21:04 | <asher> | synchronized wmf-config/StartProfiler.php  'only randomly profile web reqs in eqiad' | [production] | 
            
  | 20:39 | <sbernardin> | hot swapping disk 13 on db78 storage array | [production] | 
            
  | 20:19 | <paravoid> | deploying bmc-config fix across all pmtpa/eqiad mgmt | [production] | 
            
  | 19:42 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: Everyhting else no wikipedia to 1.22wmf11 | [production] | 
            
  | 19:36 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: wikiquote, wikinews and wikibooks to 1.22wmf11 | [production] | 
            
  | 19:29 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity, wikivoyage and wiktionary to 1.22wmf11 | [production] | 
            
  | 19:26 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: special, wikimedia, private, fishbowl and closed to 1.22wmf11 | [production] | 
            
  | 19:25 | <paravoid> | restarting & repooling cp1047/cp1059 | [production] |