| 2014-10-05
      
      § | 
    
  | 22:28 | <Coren> | Q183 superprotected as a safeguard | [production] | 
            
  | 22:27 | <hoo> | Q183 is on revision 116786096 again, please don't alter this further! | [production] | 
            
  | 22:21 | <qchris> | Updated gerrit's hooks-bugzilla to 6e1e659 (with hooks-its at a421db4) | [production] | 
            
  | 22:11 | <hoo> | WD:Q183 was frozen on version 120566337, see bug 71519 (and others) | [production] | 
            
  | 21:23 | <hoo> | Bypassed Wikibase restrictions and set https://www.wikidata.org/wiki/Q183 back to old serialization format | [production] | 
            
  | 20:08 | <Nemo_bis> | 22.03 < Ainali> It was just noticed on svwp village pump that http://stats.wikimedia.org is down | [production] | 
            
  | 16:39 | <paravoid> | restore ns1 routing to codfw | [production] | 
            
  | 11:23 | <paravoid> | adding static route for ns1 to rubidium (ns0) on cr1-eqiad to temporarily redirect its traffic while the codfw is offline | [production] | 
            
  | 03:20 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Sun Oct  5 03:19:55 UTC 2014 (duration 19m 53s) | [production] | 
            
  | 03:02 | <ori> | Synchronized wmf-config/CommonSettings.php: I707b5754: Enable LuaSandbox profiling when  is true (duration: 00m 07s) | [production] | 
            
  | 02:22 | <LocalisationUpdate> | completed (1.25wmf2) at 2014-10-05 02:22:47+00:00 | [production] | 
            
  | 02:13 | <LocalisationUpdate> | completed (1.25wmf1) at 2014-10-05 02:13:26+00:00 | [production] | 
            
  
    | 2014-10-03
      
      § | 
    
  | 22:46 | <bd808> | Restarting zuul on gallium | [production] | 
            
  | 22:41 | <bd808> | Trying a soft restart of zuul on gallium | [production] | 
            
  | 22:37 | <bd808> | NoConnectedServersError("No connected Gearman servers") in zuul.log on gallium | [production] | 
            
  | 22:33 | <bd808|deploy> | Updated integration/phpunit to 6c1d11d (Regenerate autoloader) | [production] | 
            
  | 22:31 | <subbu> | restarted Parsoid servers after another gradual cpu load creep | [production] | 
            
  | 22:19 | <aaron> | Synchronized wmf-config/InitialiseSettings.php: Fixed the parser cache type for labswiki (duration: 00m 03s) | [production] | 
            
  | 21:55 | <andrewbogott> | updated the defaut labs trusty image:  updated packages, updated ldap setup, new /var/log partition | [production] | 
            
  | 20:37 | <ori> | disabling puppet on rbf1002 to test bloom filter config | [production] | 
            
  | 20:31 | <manybubbles> | Synchronized wmf-config/CirrusSearch-labs.php: noop update to sync beta configs (duration: 00m 04s) | [production] | 
            
  | 20:30 | <cscott> | (the above was on ocg100x.eqiad.wmnet) | [production] | 
            
  | 20:30 | <cscott> | ran 'sudo -u ocg -g ocg nodejs-ocg scripts/run-garbage-collect.js -c /home/cscott/config.js' from /home/cscott/ocg/mw-ocg-service in order to clear caches (working around https://gerrit.wikimedia.org/r/164644 ) | [production] | 
            
  | 20:23 | <andrewbogott> | restarted zuul | [production] | 
            
  | 20:23 | <ori> | Synchronized wmf-config/mc.php: Ie1ed821a7: Set bloom cache config (duration: 00m 03s) | [production] | 
            
  | 19:34 | <ori> | when running puppet merge: fatal: Unable to create '/var/lib/git/operations/puppet/.git/refs/remotes/origin/production.lock': File exists. | [production] | 
            
  | 19:28 | <hashar> | Restarting Zuul sorry :-/ | [production] | 
            
  | 19:26 | <hashar> | Zuul in some kind of death loop | [production] | 
            
  | 19:05 | <mutante> | purging old 'searchqa' scripts and logs from iron (gerrit 164429 removes from puppet) | [production] | 
            
  | 18:46 | <mutante> | restored dbtree from manual backup (should have been synced by scripts) | [production] | 
            
  | 18:25 | <aaron> | Synchronized php-1.25wmf2/extensions/CentralAuth: (no message) (duration: 00m 05s) | [production] | 
            
  | 18:14 | <maxsem> | Synchronized php-1.25wmf2/extensions/MobileFrontend/: https://gerrit.wikimedia.org/r/#/c/164543/ (duration: 00m 04s) | [production] | 
            
  | 18:14 | <maxsem> | Synchronized php-1.25wmf1/extensions/MobileFrontend/: https://gerrit.wikimedia.org/r/#/c/164543/ (duration: 00m 04s) | [production] | 
            
  | 17:53 | <bblack> | restarted parsoid varnishes | [production] | 
            
  | 17:30 | <andrewbogott> | enabling puppet on tungsten which is disabled for mysterious reasons | [production] | 
            
  | 17:04 | <gwicke> | restarting parsoids after CPU spike | [production] | 
            
  | 16:50 | <ejegg> | updated crm from 6f66294607d132230ef82fb4867c37a8700bfd4e to ef68d9fe98a64e819ebbdddbe5e13f83037607ce | [production] | 
            
  | 15:34 | <_joe_> | purging varnish cache for parsoid (RT 8528) | [production] | 
            
  | 15:19 | <mark> | shutdown pdf2 & pdf3 | [production] | 
            
  | 15:15 | <bblack> | adding 10.2.1.0/24 aggregate in cr-[12].codfw | [production] | 
            
  | 15:14 | <bblack> | dropping 10.2.1.0/24 aggregate + static routes in cr2-pmtpa | [production] | 
            
  | 14:37 | <bblack> | testing dns server upgrade on baham | [production] |