6901-6950 of 10000 results (36ms)
2014-10-05 §
20:08 <Nemo_bis> 22.03 < Ainali> It was just noticed on svwp village pump that http://stats.wikimedia.org is down [production]
16:39 <paravoid> restore ns1 routing to codfw [production]
11:23 <paravoid> adding static route for ns1 to rubidium (ns0) on cr1-eqiad to temporarily redirect its traffic while the codfw is offline [production]
03:20 <LocalisationUpdate> ResourceLoader cache refresh completed at Sun Oct 5 03:19:55 UTC 2014 (duration 19m 53s) [production]
03:02 <ori> Synchronized wmf-config/CommonSettings.php: I707b5754: Enable LuaSandbox profiling when is true (duration: 00m 07s) [production]
02:22 <LocalisationUpdate> completed (1.25wmf2) at 2014-10-05 02:22:47+00:00 [production]
02:13 <LocalisationUpdate> completed (1.25wmf1) at 2014-10-05 02:13:26+00:00 [production]
2014-10-04 §
21:08 <_joe_> cleaning ocg1001 tmpfs from a 32 gb pdf file [production]
19:59 <jgage> restarted pdns on virt1000 for ldap config update [production]
07:08 <springle> powercycle es1004 [production]
03:27 <LocalisationUpdate> ResourceLoader cache refresh completed at Sat Oct 4 03:27:20 UTC 2014 (duration 27m 19s) [production]
02:25 <LocalisationUpdate> completed (1.25wmf2) at 2014-10-04 02:25:04+00:00 [production]
02:15 <LocalisationUpdate> completed (1.25wmf1) at 2014-10-04 02:15:02+00:00 [production]
01:01 <bblack> depooling cp1045 for persistent cache wipe [production]
00:01 <andrewbogott> updated the defaut labs precise image: updated ldap setup, new /var/log partition [production]
2014-10-03 §
22:46 <bd808> Restarting zuul on gallium [production]
22:41 <bd808> Trying a soft restart of zuul on gallium [production]
22:37 <bd808> NoConnectedServersError("No connected Gearman servers") in zuul.log on gallium [production]
22:33 <bd808|deploy> Updated integration/phpunit to 6c1d11d (Regenerate autoloader) [production]
22:31 <subbu> restarted Parsoid servers after another gradual cpu load creep [production]
22:19 <aaron> Synchronized wmf-config/InitialiseSettings.php: Fixed the parser cache type for labswiki (duration: 00m 03s) [production]
21:55 <andrewbogott> updated the defaut labs trusty image: updated packages, updated ldap setup, new /var/log partition [production]
20:37 <ori> disabling puppet on rbf1002 to test bloom filter config [production]
20:31 <manybubbles> Synchronized wmf-config/CirrusSearch-labs.php: noop update to sync beta configs (duration: 00m 04s) [production]
20:30 <cscott> (the above was on ocg100x.eqiad.wmnet) [production]
20:30 <cscott> ran 'sudo -u ocg -g ocg nodejs-ocg scripts/run-garbage-collect.js -c /home/cscott/config.js' from /home/cscott/ocg/mw-ocg-service in order to clear caches (working around https://gerrit.wikimedia.org/r/164644 ) [production]
20:23 <andrewbogott> restarted zuul [production]
20:23 <ori> Synchronized wmf-config/mc.php: Ie1ed821a7: Set bloom cache config (duration: 00m 03s) [production]
19:34 <ori> when running puppet merge: fatal: Unable to create '/var/lib/git/operations/puppet/.git/refs/remotes/origin/production.lock': File exists. [production]
19:28 <hashar> Restarting Zuul sorry :-/ [production]
19:26 <hashar> Zuul in some kind of death loop [production]
19:05 <mutante> purging old 'searchqa' scripts and logs from iron (gerrit 164429 removes from puppet) [production]
18:46 <mutante> restored dbtree from manual backup (should have been synced by scripts) [production]
18:25 <aaron> Synchronized php-1.25wmf2/extensions/CentralAuth: (no message) (duration: 00m 05s) [production]
18:14 <maxsem> Synchronized php-1.25wmf2/extensions/MobileFrontend/: https://gerrit.wikimedia.org/r/#/c/164543/ (duration: 00m 04s) [production]
18:14 <maxsem> Synchronized php-1.25wmf1/extensions/MobileFrontend/: https://gerrit.wikimedia.org/r/#/c/164543/ (duration: 00m 04s) [production]
17:53 <bblack> restarted parsoid varnishes [production]
17:30 <andrewbogott> enabling puppet on tungsten which is disabled for mysterious reasons [production]
17:04 <gwicke> restarting parsoids after CPU spike [production]
16:50 <ejegg> updated crm from 6f66294607d132230ef82fb4867c37a8700bfd4e to ef68d9fe98a64e819ebbdddbe5e13f83037607ce [production]
15:34 <_joe_> purging varnish cache for parsoid (RT 8528) [production]
15:19 <mark> shutdown pdf2 & pdf3 [production]
15:15 <bblack> adding 10.2.1.0/24 aggregate in cr-[12].codfw [production]
15:14 <bblack> dropping 10.2.1.0/24 aggregate + static routes in cr2-pmtpa [production]
14:37 <bblack> testing dns server upgrade on baham [production]
14:35 <manybubbles> Synchronized wmf-config/CirrusSearch-labs.php: Noop - just keeps beta config in sync (duration: 00m 04s) [production]
14:18 <Jeff_Green> launched iodine:/opt/otrs/bin/otrs.RebuildFulltextIndex.pl per bugzilla #64473 [production]
13:07 <Reedy> Updated minor_mime to varbinary(100) on image|filearchive|oldimage on foundationwiki [production]
12:31 <springle> Synchronized wmf-config/db-eqiad.php: depool es1004 (duration: 00m 06s) [production]
11:51 <reedy> Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 15s) [production]