| 2014-10-06
      
      § | 
    
  | 22:00 | <hoo> | Reverted wd:Q17939676 to 157541810 and edit=sysop | [production] | 
            
  | 21:47 | <cscott> | updated OCG to version bbdf4c6400cfbbc6030114ad16e1a6f7025eab2c | [production] | 
            
  | 20:21 | <awight> | backfilled recurring GC glitch from FR #2018, 3342 records affected. | [production] | 
            
  | 20:16 | <subbu> | deployed parsoid sha 13a53ab3 (deploy repo sha 38d44ada7) | [production] | 
            
  | 19:04 | <akosiaris> | issued cf disable and halt on nas1-a.pmtpa.wmnet nas1-b.pmtpa.wmnet. They are officially down :) | [production] | 
            
  | 17:22 | <_joe_> | hhvm load test finished | [production] | 
            
  | 17:03 | <_joe_> | depooling and repooling progressively hhvm appservers to do see performance under load | [production] | 
            
  | 16:57 | <Krinkle> | git-deploy: Deploying integration/slave-scripts 0b85d48 | [production] | 
            
  | 16:19 | <ori> | Synchronized wmf-config/CommonSettings.php: Set $wgPercentHHVM to 1 (duration: 00m 27s) | [production] | 
            
  | 15:54 | <cscott> | updated OCG to version aee3712b352f51f96569de0bcccf3facf654e688 | [production] | 
            
  | 15:45 | <GroggyPanda> | deleted graphite data for deployment-rsync02 by hand on labmon1001, since instance has been dead. Need to move to shinken + dynamic host.cfg | [production] | 
            
  | 15:37 | <manybubbles> | Synchronized php-1.25wmf1/extensions/Wikidata/: SWAT update wikidata (duration: 00m 10s) | [production] | 
            
  | 15:23 | <manybubbles> | Synchronized php-1.25wmf2/extensions/Wikidata/: SWAT update wikidata (duration: 00m 10s) | [production] | 
            
  | 15:22 | <hashar> | Zuul jobs proceeding again | [production] | 
            
  | 15:22 | <godog> | swiftrepl replicating non-sharded originals containers eqiad -> codfw | [production] | 
            
  | 15:22 | <manybubbles> | Synchronized wmf-config/: SWAT Add tracking categories for files with attribution problems (duration: 00m 06s) | [production] | 
            
  | 15:19 | <cscott> | ran 'sudo -u ocg -g ocg nodejs-ocg scripts/run-garbage-collect.js -c /home/cscott/config.js' from /home/cscott/ocg/mw-ocg-service in order to clear caches (working around https://gerrit.wikimedia.org/r/164644 ) on ocg100x.eqiad.wmnet | [production] | 
            
  | 14:51 | <cmjohnson1> | disconnecting Tampa servers | [production] | 
            
  | 13:46 | <godog> | starting test swiftrepl run on wikibooks eqiad -> codfw | [production] | 
            
  | 11:49 | <_joe_> | done restarting ocg servers | [production] | 
            
  | 11:34 | <_joe_> | rolling restart and cleaning of ocg nodes, trying to unlock pdf generation | [production] | 
            
  | 11:11 | <mark> | Shutdown tarin | [production] | 
            
  | 11:11 | <mark> | Shutdown sanger | [production] | 
            
  | 09:27 | <_joe_> | cleaned ocg another time | [production] | 
            
  | 09:07 | <mark> | Stopped dovecot on sanger | [production] | 
            
  | 08:06 | <_joe_> | cleaned ocg1001, again | [production] | 
            
  | 05:57 | <Nemo_bis> | "book creator seems stuck": PDF servers at 97 % CPU, little traffic, enough disk free for about 1 day more | [production] | 
            
  | 03:26 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Mon Oct  6 03:26:31 UTC 2014 (duration 26m 30s) | [production] | 
            
  | 02:59 | <springle> | Synchronized wmf-config/db-eqiad.php: depool db1060 (duration: 00m 06s) | [production] | 
            
  | 02:28 | <LocalisationUpdate> | completed (1.25wmf2) at 2014-10-06 02:28:42+00:00 | [production] | 
            
  | 02:17 | <LocalisationUpdate> | completed (1.25wmf1) at 2014-10-06 02:17:40+00:00 | [production] | 
            
  
    | 2014-10-05
      
      § | 
    
  | 22:28 | <Coren> | Q183 superprotected as a safeguard | [production] | 
            
  | 22:27 | <hoo> | Q183 is on revision 116786096 again, please don't alter this further! | [production] | 
            
  | 22:21 | <qchris> | Updated gerrit's hooks-bugzilla to 6e1e659 (with hooks-its at a421db4) | [production] | 
            
  | 22:11 | <hoo> | WD:Q183 was frozen on version 120566337, see bug 71519 (and others) | [production] | 
            
  | 21:23 | <hoo> | Bypassed Wikibase restrictions and set https://www.wikidata.org/wiki/Q183 back to old serialization format | [production] | 
            
  | 20:08 | <Nemo_bis> | 22.03 < Ainali> It was just noticed on svwp village pump that http://stats.wikimedia.org is down | [production] | 
            
  | 16:39 | <paravoid> | restore ns1 routing to codfw | [production] | 
            
  | 11:23 | <paravoid> | adding static route for ns1 to rubidium (ns0) on cr1-eqiad to temporarily redirect its traffic while the codfw is offline | [production] | 
            
  | 03:20 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Sun Oct  5 03:19:55 UTC 2014 (duration 19m 53s) | [production] | 
            
  | 03:02 | <ori> | Synchronized wmf-config/CommonSettings.php: I707b5754: Enable LuaSandbox profiling when  is true (duration: 00m 07s) | [production] | 
            
  | 02:22 | <LocalisationUpdate> | completed (1.25wmf2) at 2014-10-05 02:22:47+00:00 | [production] | 
            
  | 02:13 | <LocalisationUpdate> | completed (1.25wmf1) at 2014-10-05 02:13:26+00:00 | [production] |