| 2014-09-10
      
      § | 
    
  | 18:26 | <yurik> | Synchronized php-1.24wmf20/extensions/ZeroBanner: (no message) (duration: 01m 09s) | [production] | 
            
  | 18:22 | <yurik> | Synchronized php-1.24wmf19/extensions/ZeroBanner: (no message) (duration: 01m 11s) | [production] | 
            
  | 18:00 | <manybubbles> | cirrus index rebuild for test2wiki went well - doing the rest of group0 | [production] | 
            
  | 17:35 | <manybubbles> | rebuilding cirrus index for test2wiki to test some performance enhancements don't break anything.  test2wiki is too small to see any gain from the enhancements though. | [production] | 
            
  | 17:25 | <Reedy> | mw1126, mw1116, mw1122, mw1146, mw1121, mw1136, mw1114, mw1068 have been gracefulled | [production] | 
            
  | 17:10 | <bd808> | Restarted logstash on logstash1001 | [production] | 
            
  | 16:03 | <demon> | Synchronized wmf-config/InitialiseSettings.php: nlwiki cirrus (duration: 00m 04s) | [production] | 
            
  | 15:44 | <reedy> | Synchronized docroot and w: (no message) (duration: 00m 15s) | [production] | 
            
  | 15:02 | <demon> | Synchronized wmf-config/wikitech.php: no-op (duration: 00m 06s) | [production] | 
            
  | 09:13 | <godog> | rolling restart swift-proxy on ms-fe1* | [production] | 
            
  | 04:17 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Wed Sep 10 04:17:36 UTC 2014 (duration 17m 35s) | [production] | 
            
  | 03:08 | <LocalisationUpdate> | completed (1.24wmf20) at 2014-09-10 03:07:59+00:00 | [production] | 
            
  | 02:36 | <LocalisationUpdate> | completed (1.24wmf19) at 2014-09-10 02:36:00+00:00 | [production] | 
            
  | 02:28 | <ori> | updated salt key for iridium and restarted salt-minion | [production] | 
            
  | 02:18 | <mutante> | started salt-minion on iridium | [production] | 
            
  
    | 2014-09-09
      
      § | 
    
  | 23:15 | <Krinkle> | Reloading Zuul to deploy I26bc21ed2938e97e7ed6f6b | [production] | 
            
  | 23:15 | <demon> | Synchronized php-1.24wmf20/extensions/CirrusSearch: Various fixes for things (duration: 00m 05s) | [production] | 
            
  | 23:00 | <mutante> | added wikimedia.org to search in resolv.conf on terbium | [production] | 
            
  | 22:42 | <ebernhardson> | Synchronized wmf-config/InitialiseSettings.php: Deploy config change I158e7c6852 (duration: 00m 04s) | [production] | 
            
  | 22:23 | <Krinkle> | Reloading Zuul to deploy I27024680c74ca0130 | [production] | 
            
  | 22:21 | <ebernhardson> | Finished scap: Bump Echo and Flow versions in 1.24wmf19 (duration: 31m 25s) | [production] | 
            
  | 21:49 | <ebernhardson> | Started scap: Bump Echo and Flow versions in 1.24wmf19 | [production] | 
            
  | 20:42 | <akosiaris> | service gmetad restart on nickel.wikimedia.org due to ganglia web not working | [production] | 
            
  | 20:15 | <cscott> | updated OCG to version c9a2b4cf2502479eeabed07ab2de728695d96e46 | [production] | 
            
  | 19:05 | <mutante> | killed jgonera's screen session on stat1002 - puppet failed to deactivate otherwise | [production] | 
            
  | 18:46 | <reedy> | Synchronized wmf-config/: (no message) (duration: 00m 14s) | [production] | 
            
  | 18:32 | <reedy> | Synchronized wmf-config/: (no message) (duration: 00m 15s) | [production] | 
            
  | 18:31 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: Add cawikimedia | [production] | 
            
  | 18:28 | <reedy> | Synchronized multiversion/: (no message) (duration: 00m 14s) | [production] | 
            
  | 18:14 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.24wmf20 | [production] | 
            
  | 16:03 | <demon> | Synchronized wmf-config/InitialiseSettings.php: eswiki getting cirrus (duration: 00m 04s) | [production] | 
            
  | 15:32 | <bblack> | deploying large DNS change https://gerrit.wikimedia.org/r/#/c/158382/ - be on the lookout for any related fallout from here... | [production] | 
            
  | 15:27 | <marktraceur> | [SCAP] Deployed fix for oojs class names at James_F's behest, sorry for lack of message. | [production] | 
            
  | 15:26 | <marktraceur> | Synchronized php-1.24wmf20/extensions/MobileFrontend/less/modules/editor/VisualEditorOverlay.less: (no message) (duration: 00m 07s) | [production] | 
            
  | 15:08 | <marktraceur> | Synchronized php-1.24wmf20/tests/phpunit/includes/changes/OldChangesListTest.php: [SWAT] Fix undefined argument (css classes) in OldChangesList. (duration: 00m 07s) | [production] | 
            
  | 15:06 | <marktraceur> | Synchronized php-1.24wmf20/includes/changes/OldChangesList.php: [SWAT] Fix undefined argument (css classes) in OldChangesList. (duration: 00m 07s) | [production] | 
            
  | 11:29 | <_joe_> | git.wikimedia.org works now, no action needed | [production] | 
            
  | 11:26 | <MatmaRex> | git.wikimedia.org is down: Error: 503, Service Unavailable | [production] | 
            
  | 10:04 | <_joe_> | also re-enabling puppet | [production] | 
            
  | 10:03 | <_joe_> | restarting manually apache on mw1178,mw1192,mw1163,mw1130,mw1018 as they started with the wrong pidfile before my fix | [production] | 
            
  | 09:24 | <_joe_> | disabling puppet on appservers | [production] | 
            
  | 08:55 | <godog> | launched "iptables" on tin to check current rules and it loaded iptables modules, logging for future reference | [production] | 
            
  | 08:10 | <_joe_> | re-enabling puppet on appservers and imagescalers, change is good | [production] | 
            
  | 08:08 | <_joe_> | restarted apache2 on mw1018 | [production] | 
            
  | 08:07 | <_joe_> | stopping apache on mw1018 for inspection | [production] | 
            
  | 07:37 | <_joe_> | that was on appservers | [production] | 
            
  | 07:36 | <_joe_> | disabling puppet, releasing a potentially harmful apache change | [production] | 
            
  | 04:56 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Tue Sep  9 04:56:25 UTC 2014 (duration 56m 24s) | [production] | 
            
  | 03:44 | <LocalisationUpdate> | completed (1.24wmf20) at 2014-09-09 03:44:07+00:00 | [production] | 
            
  | 03:11 | <LocalisationUpdate> | completed (1.24wmf19) at 2014-09-09 03:11:27+00:00 | [production] |