| 2014-03-10
      
      § | 
    
  | 14:58 | <reedy> | synchronized wmf-config/InitialiseSettings.php | [production] | 
            
  | 14:57 | <reedy> | updated /a/common to {{Gerrit|Ibcbd10044}}: Disable and remove ContactPageFundraiser | [production] | 
            
  | 14:44 | <reedy> | synchronized wmf-config/  'Remove ContactPageFundraiser, SkinPerPage, FormPreloadPostCache and skins used on foundationwiki' | [production] | 
            
  | 14:26 | <ottomata> | put elastic1007 and elastic1013-1016 back in main elasticsearch pool | [production] | 
            
  | 09:11 | <hashar> | Jenkins restarted and proceeding jobs again | [production] | 
            
  | 09:06 | <hashar> | stopping Jenkins | [production] | 
            
  | 09:01 | <hashar> | Jenkins unresponsive for some reason yeah!!! | [production] | 
            
  | 02:57 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Mon Mar 10 02:57:32 UTC 2014 (duration 57m 31s) | [production] | 
            
  | 02:25 | <LocalisationUpdate> | completed (1.23wmf17) at 2014-03-10 02:25:14+00:00 | [production] | 
            
  | 02:14 | <LocalisationUpdate> | completed (1.23wmf16) at 2014-03-10 02:14:08+00:00 | [production] | 
            
  
    | 2014-03-08
      
      § | 
    
  | 03:49 | <mutante> | shut down srv250-srv247 (this was stdpa row D-D2) | [production] | 
            
  | 03:42 | <mutante> | shutting down srv193,srv235-srv247,srv248,249 | [production] | 
            
  | 03:28 | <andrewbogott> | upgrading nova-compute on virt100[1-6] | [production] | 
            
  | 03:15 | <mutante> | shut down Tampa mw hosts 21-27,49,58,75-85 (that is sdpta row D:D1) | [production] | 
            
  | 03:04 | <mutante> | shut down Tampa mw hosts 1-20 | [production] | 
            
  | 02:56 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Sat Mar  8 02:55:55 UTC 2014 (duration 55m 54s) | [production] | 
            
  | 02:25 | <LocalisationUpdate> | completed (1.23wmf17) at 2014-03-08 02:25:00+00:00 | [production] | 
            
  | 02:14 | <LocalisationUpdate> | completed (1.23wmf16) at 2014-03-08 02:14:25+00:00 | [production] | 
            
  
    | 2014-03-07
      
      § | 
    
  | 23:36 | <ori> | updated /a/common to {{Gerrit|If2530281b}}: Order branch directories as version numbers | [production] | 
            
  | 22:14 | <mutante> | deleting salt keys for all Tampa app servers removed from puppet | [production] | 
            
  | 21:54 | <hashar> | Jenkins restarted | [production] | 
            
  | 21:49 | <hashar> | restarting Jenkins it is broken | [production] | 
            
  | 21:45 | <mutante> | killing all tampa appservers from puppetstoredconfigs | [production] | 
            
  | 21:41 | <mutante> | disabling puppet agent on all tampa appservers | [production] | 
            
  | 21:15 | <hoo> | synchronized wmf-config/InitialiseSettings-labs.php  'Syncing beta-only change for consistency' | [production] | 
            
  | 20:28 | <mutante> | killing mw1-16 from puppet stored configs, icinga,.. | [production] | 
            
  | 20:26 | <mutante> | revoking puppet certs for Tampa appservers | [production] | 
            
  | 20:14 | <catrope> | synchronized php-1.23wmf16/extensions/VisualEditor/modules/ve-mw/dm/nodes/ve.dm.MWBlockImageNode.js  'Fix image corruption bug' | [production] | 
            
  | 20:14 | <catrope> | updated /a/common/php-1.23wmf16 to {{Gerrit|I4a10768ec}}: Update VisualEditor to wmf16 branch for cherry-pick | [production] | 
            
  | 18:17 | <bd808> | Restored pre Ic56177a versions of wmf-config/*pmtpa* config files to mw31 again. Something wiped them out since 20:23Z yesterday even though "mw31" is not found in any dsh group files on tin. | [production] | 
            
  | 17:53 | <mutante> | restarted squid on brewster | [production] | 
            
  | 03:31 | <LocalisationUpdate> | ResourceLoader cache refresh completed at 2014-03-07 03:31:43+00:00 | [production] | 
            
  | 02:43 | <LocalisationUpdate> | completed (1.23wmf17) at 2014-03-07 02:43:40+00:00 | [production] | 
            
  | 02:21 | <LocalisationUpdate> | completed (1.23wmf16) at 2014-03-07 02:21:46+00:00 | [production] | 
            
  | 02:14 | <manybubbles> | [Elasticsearch upgrade] done.  we'll take a while to catch up on jobs that piled up during the upgrade, but we'll get them in time. | [production] | 
            
  | 02:10 | <demon> | synchronized wmf-config/InitialiseSettings.php  'Turn Cirrus back on for all wikis how it was before' | [production] | 
            
  | 02:09 | <demon> | synchronized wmf-config/jobqueue-eqiad.php  'Turn Cirrus jobs back on' | [production] | 
            
  | 02:04 | <manybubbles> | [Elasticsearch upgrade] restoring more sane recovery speed | [production] | 
            
  | 01:39 | <mholmquist> | synchronized php-1.23wmf17/extensions/UniversalLanguageSelector/UniversalLanguageSelector.hooks.php  'Actually gate the beta feature for ULS' | [production] | 
            
  | 01:30 | <manybubbles> | [Elasticsearch upgrade] temporarily raising recovery speed | [production] | 
            
  | 01:29 | <jgonera> | synchronized php-1.23wmf16/extensions/MobileFrontend/  'Touch MobileFrontend.i18n.php to update RL cache' | [production] | 
            
  | 01:19 | <mholmquist> | synchronized wmf-config/InitialiseSettings.php  'Fix James_F's commit, follow-up, should gate ULS beta feature' | [production] | 
            
  | 01:18 | <mholmquist> | updated /a/common to {{Gerrit|I48e98d28f}}: Follow-up: Icf0bef96306661 – missing file(!) from commit | [production] | 
            
  | 01:18 | <mholmquist> | Finished scap: (no message) (duration: 10m 55s) | [production] | 
            
  | 01:07 | <rdwrer> | That scap was for ULS, VE, and MobileFrontend fixes and updates. | [production] | 
            
  | 01:07 | <mholmquist> | Started scap: (no message) | [production] | 
            
  | 01:03 | <manybubbles> | [Elasticsearch upgrade] Reenabling puppet | [production] | 
            
  | 01:03 | <hashar> | Jenkins back up | [production] | 
            
  | 01:03 | <hashar> | Jenkins backup | [production] |