| 2014-04-18
      
      § | 
    
  | 14:51 | <ottomata> | powering down stat1 for decom | [production] | 
            
  | 14:43 | <mutante> | ms-fe[14] - shutting down | [production] | 
            
  | 14:41 | <ottomata> | disabling puppet on stat1 for decom | [production] | 
            
  | 14:37 | <mutante> | ms-be 1-12, Tampa Swift boxes, shutdown | [production] | 
            
  | 14:24 | <mutante> | ms-fe[14] - stop puppet,revoke certs,remove icinga | [production] | 
            
  | 13:54 | <mutante> | ms-be1-12 - removing from puppet,salt,icinga | [production] | 
            
  | 13:07 | <mutante> | Bugzilla Apache, changed SSL cipher suite in I7e9adc182dc ,might cost a a few % performance but zirconium had plenty | [production] | 
            
  | 11:49 | <hashar> | removing mw-jenkinsbot (the wikimedia jenkins installation) from #wikimedia-labs | [production] | 
            
  | 10:10 | <hashar> | Jenkins upgraded to 1.532.3. | [production] | 
            
  | 10:06 | <hashar> | Upgrading Jenkins to latest LTS version 1.532.3 | [production] | 
            
  | 07:57 | <mutante> | DNS update - remove api.svc, arptest.pmtpa .. | [production] | 
            
  | 06:31 | <demon> | synchronized wmf-config/InitialiseSettings.php  'Next round of wikis done building Cirrus indexes, throw into beta mode' | [production] | 
            
  | 04:04 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Fri Apr 18 04:04:21 UTC 2014 (duration 4m 20s) | [production] | 
            
  | 03:06 | <LocalisationUpdate> | completed (1.24wmf1) at 2014-04-18 03:06:06+00:00 | [production] | 
            
  | 02:40 | <LocalisationUpdate> | completed (1.23wmf22) at 2014-04-18 02:39:51+00:00 | [production] | 
            
  | 00:21 | <ori> | synchronized wmf-config/CommonSettings.php  'Ie9b265be9: Enable GlobalCssJs on testwiki & test2wiki (2/2)' | [production] | 
            
  | 00:21 | <ori> | synchronized wmf-config/InitialiseSettings.php  'Ie9b265be9: Enable GlobalCssJs on testwiki & test2wiki (1/2)' | [production] | 
            
  | 00:20 | <ori> | updated /a/common to {{Gerrit|Ie9b265be9}}: Enable GlobalCssJs on testwiki & test2wiki | [production] | 
            
  | 00:18 | <ori> | Finished scap: Cherry-pick Ibe8e67ebf for MobileFrontend on 1.23wmf22 and 1.24wmf1; add GlobalCssJs extension to 1.24wmf1 and 1.23wmf22 (duration: 32m 53s) | [production] | 
            
  
    | 2014-04-17
      
      § | 
    
  | 23:45 | <ori> | Started scap: Cherry-pick Ibe8e67ebf for MobileFrontend on 1.23wmf22 and 1.24wmf1; add GlobalCssJs extension to 1.24wmf1 and 1.23wmf22 | [production] | 
            
  | 23:39 | <ori> | scap failed: CalledProcessError Command '/usr/local/bin/mw-update-l10n' returned non-zero exit status 1 (duration: 00m 24s) | [production] | 
            
  | 23:38 | <ori> | Started scap: Cherry-pick Ibe8e67ebf for MobileFrontend on 1.23wmf22 and 1.24wmf1; add GlobalCssJs extension to 1.24wmf1 | [production] | 
            
  | 23:37 | <ori> | updated /a/common to {{Gerrit|I2a2abd7f3}}: Add GlobalCssJs to extension-list | [production] | 
            
  | 23:29 | <ori> | synchronized wmf-config/InitialiseSettings.php  'I52378a4b4: Add meta to legalteamwiki import sources' | [production] | 
            
  | 23:28 | <ori> | updated /a/common to {{Gerrit|I52378a4b4}}: Add meta to legalteamwiki import sources | [production] | 
            
  | 23:27 | <ori> | synchronized wmf-config/CommonSettings.php  'I373df6138: Normalize TextExtracts config handling (2/2)' | [production] | 
            
  | 23:27 | <ori> | synchronized wmf-config/InitialiseSettings.php  'I373df6138: Normalize TextExtracts config handling (1/2)' | [production] | 
            
  | 23:26 | <ori> | updated /a/common to {{Gerrit|I373df6138}}: Normalize TextExtracts config handling | [production] | 
            
  | 23:24 | <ori> | synchronized wmf-config/InitialiseSettings.php  'I7841f74b0: Kill all vestiges of $wgMFRemovableClasses (2/2)' | [production] | 
            
  | 23:24 | <ori> | synchronized wmf-config/mobile.php  'I7841f74b0: Kill all vestiges of $wgMFRemovableClasses (1/2)' | [production] | 
            
  | 23:23 | <ori> | updated /a/common to {{Gerrit|I7841f74b0}}: Kill all vestiges of $wgMFRemovableClasses | [production] | 
            
  | 23:18 | <ori> | synchronized wmf-config/InitialiseSettings.php  'I1795c70d1: Create a FeaturedFeed for the Tech News bulletin (2/2)' | [production] | 
            
  | 23:18 | <ori> | synchronized wmf-config/FeaturedFeedsWMF.php  'I1795c70d1: Create a FeaturedFeed for the Tech News bulletin (1/2)' | [production] | 
            
  | 23:17 | <ori> | updated /a/common to {{Gerrit|I1795c70d1}}: Create a FeaturedFeed for the Tech News bulletin | [production] | 
            
  | 23:14 | <ori> | synchronized php-1.23wmf22/extensions/ApiSandbox  'I9a56b2c5a: Update ApiSandbox' | [production] | 
            
  | 23:12 | <K4-713> | updates antifraud rules in payments | [production] | 
            
  | 22:03 | <andrewbogott> | updated default labs precise image (heartbleed fix) | [production] | 
            
  | 20:41 | <manybubbles> | elastic1016 restarted and not freaking out any more. | [production] | 
            
  | 20:37 | <_joe_> | restarting gitblit in order to prevent crippling due to the usual memory leak | [production] | 
            
  | 20:28 | <manybubbles> | restarting elastic1016 - it is freaking out.  If it happens again I'll dig deeper, but for now I consider it a fluke of the rolling restarts today.... | [production] | 
            
  | 20:20 | <RobH> | sorry for the misc-web-lb issues folks, they should be resolved at this time (for now) | [production] | 
            
  | 20:19 | <paravoid> | lvs1002/1005: commenting first resolv.conf entry until we have a more permanent fix, restarting pybal | [production] | 
            
  | 20:18 | <paravoid> | disabling puppet on lvs1002/lvs1005 | [production] | 
            
  | 19:57 | <RobH> | still working on issue | [production] | 
            
  | 19:57 | <RobH> | both cp1043 and cp1044 seem online and serving nginx service, but pybal says they are down still working | [production] | 
            
  | 19:46 | <ottomata> | power off emery | [production] | 
            
  | 19:40 | <RobH> | replacing ticket.wikimedia.org cert/key, apache may hiccup | [production] | 
            
  | 19:33 | <RobH> | blog.w.o cert replacement successful | [production] | 
            
  | 19:30 | <ottomata> | disabling puppet on emery for decommission | [production] | 
            
  | 19:29 | <RobH> | blog.w.o certificate swap (yes, again ;), apache may hiccup | [production] |