| 2013-03-22
      
      § | 
    
  | 22:07 | <mutante> | installing package upgrades on oxygen | [production] | 
            
  | 21:40 | <mutante> | killing nagios-nrpe on gallium and testing if puppet restores it properly | [production] | 
            
  | 21:03 | <aaron> | rebuilt wikiversions.cdb and synchronized wikiversions files: Set all non-Wikipedias back to 1.21wmf12 again. | [production] | 
            
  | 20:51 | <aaron> | synchronized php-1.21wmf12/includes/Title.php  'deployed 0f4914bb4ff2d13cb124cad94dc5ded663123a18' | [production] | 
            
  | 20:46 | <aaron> | synchronized php-1.21wmf12/includes/WikiPage.php  'deployed e372b635ae6d6a589f6d87f14fe4c9452d7a1b4d' | [production] | 
            
  | 20:37 | <aaron> | synchronized php-1.21wmf12/includes/Title.php  'Reverted follow-up fixes too' | [production] | 
            
  | 19:32 | <ottomata> | removed unicast udp2logging to gadolinium; gadolinium will consume from multicast stream | [production] | 
            
  | 18:44 | <K4-713> | synchronized payments cluster to fa2e240b6e92440b | [production] | 
            
  | 18:28 | <K4-713> | synchronized payments cluster to df3c8dbc937c971 | [production] | 
            
  | 17:33 | <aaron> | synchronized php-1.21wmf12/includes/WikiPage.php  'deployed 1862a57aef08432e8b1194a54dc179649f5cae63' | [production] | 
            
  | 09:08 | <MaxSem> | Recreating Solr index | [production] | 
            
  | 02:52 | <Tim> | on searchidx1001: started incremental indexer, apparently it died on March 21 at 02:08 when it ran at the same time as a cron job import | [production] | 
            
  | 02:50 | <binasher> | power cycling db1050, unresponsive on serial console | [production] | 
            
  | 02:48 | <asher> | synchronized wmf-config/db-eqiad.php  'pulling db1050, crashed' | [production] | 
            
  | 02:46 | <asher> | synchronized wmf-config/db-eqiad.php  'returning db1043' | [production] | 
            
  | 02:42 | <LocalisationUpdate> | completed (1.21wmf12) at Fri Mar 22 02:42:32 UTC 2013 | [production] | 
            
  | 02:37 | <asher> | synchronized wmf-config/db-eqiad.php  'pulling db1043, oops' | [production] | 
            
  | 02:30 | <LocalisationUpdate> | completed (1.21wmf11) at Fri Mar 22 02:30:31 UTC 2013 | [production] | 
            
  | 01:28 | <ori-l> | Finished correcting E3Experiments extension's entry point from 'Experiments.php' (which does not match the submodule name) to 'E3Experiments.php'. Updated extension-list, CommonSettings.php, and the extension itself in both wmf11 and 12. | [production] | 
            
  | 01:24 | <olivneh> | synchronized wmf-config/extension-list  'Remove temporary workaround for E3 entry-point migration (2/2)' | [production] | 
            
  | 01:23 | <olivneh> | synchronized wmf-config/CommonSettings.php  'Remove temporary workaround for E3 entry-point migration (1/2)' | [production] | 
            
  | 01:18 | <olivneh> | synchronized php-1.21wmf12/extensions/E3Experiments  'Updating E3Experiment extensions's entry point' | [production] | 
            
  | 01:18 | <olivneh> | synchronized php-1.21wmf11/extensions/E3Experiments  'Updating E3Experiment extensions's entry point' | [production] | 
            
  | 01:11 | <olivneh> | synchronized wmf-config/CommonSettings.php  'Temporarily accepting either E3Experiments.php or Experiments.php as extension entry point to facilitate clean-up' | [production] | 
            
  | 00:54 | <olivneh> | synchronized php-1.21wmf11/extensions/NavigationTiming  'NavigationTiming: log mobile mode' | [production] | 
            
  | 00:16 | <catrope> | synchronized wmf-config/ExtensionMessages-1.21wmf12.php  'Remove PHP errors' | [production] | 
            
  | 00:15 | <aaron> | synchronized php-1.21wmf12/includes/Title.php  'deployed cef327e945e2593fe0291880a7d1976cc5a2f248' | [production] | 
            
  | 00:09 | <kaldari> | synchronized php-1.21wmf12/extensions/Thanks  'syncing Thanks ext' | [production] | 
            
  
    | 2013-03-21
      
      § | 
    
  | 23:23 | <robh> | Finished syncing Wikimedia installation... : | [production] | 
            
  | 23:17 | <robh> | Started syncing Wikimedia installation... : | [production] | 
            
  | 23:03 | <bsitu> | Finished syncing Wikimedia installation... : Update Echo, PageTriage to master, Add Thanks | [production] | 
            
  | 22:43 | <bsitu> | Started syncing Wikimedia installation... : Update Echo, PageTriage to master, Add Thanks | [production] | 
            
  | 21:50 | <kaldari> | synchronized wmf-config/CommonSettings.php  'Syncing change to CommonSettings for Thanks ext' | [production] | 
            
  | 21:49 | <kaldari> | synchronized wmf-config/InitialiseSettings.php  'Syncing change to InitialiseSettings for Thanks ext' | [production] | 
            
  | 21:46 | <mutante> | running puppet and reloading zuul on gallium. enables hashar's new shiny https://integration.wikimedia.org/zuul/ | [production] | 
            
  | 21:38 | <RobH> | authdns-update for terbium | [production] | 
            
  | 21:05 | <RoanKattouw> | Added fenari back to /etc/dsh/groups/mediawiki-installation per https://rt.wikimedia.org/Ticket/Display.html?id=4794 | [production] | 
            
  | 21:02 | <cmjohnson> | Finished syncing Wikimedia installation... : | [production] | 
            
  | 20:57 | <binasher> | pulling mw1070, may have network issues | [production] | 
            
  | 20:57 | <kaldari> | synchronized php-1.21wmf12/extensions/PageTriage/modules/ext.pageTriage.views.list  'syncing PageTriage js on wmf12' | [production] | 
            
  | 20:56 | <hashar> | fenari disappeared from dsh group mediawiki-installation :-( | [production] | 
            
  | 20:52 | <mlitn> | synchronized php-1.21wmf12/extensions/ArticleFeedbackv5  'Update ArticleFeedbackv5 to master' | [production] | 
            
  | 20:51 | <mlitn> | synchronized php-1.21wmf11/extensions/ArticleFeedbackv5  'Update ArticleFeedbackv5 to master' | [production] | 
            
  | 20:48 | <awight> | payments cluster updated from 18fd11a4 to df3c8dbc9 | [production] | 
            
  | 20:42 | <cmjohnson> | Started syncing Wikimedia installation... : | [production] | 
            
  | 20:37 | <LeslieCarr> | restarting gmond on mw1153 | [production] | 
            
  | 20:19 | <mlitn> | synchronized php-1.21wmf12/extensions/ArticleFeedbackv5  'Update ArticleFeedbackv5 to master' | [production] | 
            
  | 20:19 | <olivneh> | synchronized php-1.21wmf12/extensions/NavigationTiming  'Enable NavigationTiming on mobile alpha and beta' | [production] | 
            
  | 20:18 | <mlitn> | synchronized php-1.21wmf11/extensions/ArticleFeedbackv5  'Update ArticleFeedbackv5 to master' | [production] | 
            
  | 20:18 | <olivneh> | synchronized php-1.21wmf11/extensions/NavigationTiming  'Enable NavigationTiming on mobile alpha and beta' | [production] |