4951-5000 of 10000 results (30ms)
2011-07-06 §
15:14 <RobH> search18 appears to be completely offline, investigating lom logs before rebooting. [production]
15:12 <RobH> search18 offline, logging into mgmt to check it out [production]
15:01 <RobH> eqiad humidity levels ticket dispatched for fufillment [production]
14:37 <mark> Paused rsyncs on ms5 [production]
14:04 <mark> Powercycled sq36 [production]
13:18 <^demon|away> fixed permissions on svn c/o on ci.tesla, ran svn cleanup. cruise control still not pleased and yelling about locks [production]
13:16 <mark> Upgrading firmware of scs-a1-sdtpa [production]
12:51 <mark> csw5-pmtpa crashed and reloaded [production]
11:53 <mark> Upgrading firmware of scs-c1-pmtpa [production]
2011-07-05 §
23:53 <^demon> scratch that....ssh just seems to have been rather slow in getting its act together. ci.tesla is just fine now [production]
23:50 <^demon> well now I've locked myself out of ci.tesla. Seems it doesn't start ssh on reboot...what a silly thing to do [production]
23:47 <^demon> rebooting ci.tesla since it was horribly hung up on the latest build--was it really stuck for the past 24hrs? [production]
23:35 <reedy> synchronized php-1.17/resources/mediawiki/mediawiki.js 'r91505' [production]
21:28 <pdhanda> ran sync-common-all 'Synced to r91494 for WikiLove' [production]
21:10 <pdhanda> synchronized php-1.17/resources/jquery.ui/themes/vector/jquery.ui.button.css 'Synced to r 91493.' [production]
21:00 <pdhanda> synchronized php-1.17/resources/jquery.ui/themes/vector/jquery.ui.button.css 'Synced to r 91490.' [production]
19:53 <reedy> synchronized php-1.17/includes/api/ApiQuery.php 'r91479' [production]
19:07 <catrope> synchronized php-1.17/extensions/Vector/modules/ext.vector.collapsibleTabs.js 'r91476' [production]
19:07 <catrope> synchronized php-1.17/extensions/Vector/modules/ext.vector.simpleSearch.js 'r91476' [production]
16:49 <RoanKattouw> Short CPU spike on the Apaches, approx 16:45-16:50 UTC. Things seem to be recovering now [production]
16:04 <mark> Put OSPF/OSPFv3 on csw1-sdtpa:e14/1 in active mode again; appears stable so far [production]
15:45 <mark> Reenabled csw1-sdtpa:e14/1 with OSPF/OSPFv3 in passive mode and high metric [production]
15:31 <mark> Shutdown csw1-sdtpa:e14/1 (l3 wave) [production]
15:24 <mark> Disabled IP load-sharing on csw1-esams (out of next-hops), disabled e14/1 (10G wave L3), reenabled it [production]
13:38 <mark> Rebooting streber; dpkg hung [production]
08:34 <mark> Rerouted traffic AS14907 -> AS43821 via 2828 [production]
07:11 <Tim> creating 256 tables on db40 for DBOC [production]
07:08 <tstarling> synchronized php-1.17/wmf-config/CommonSettings.php 'config change for the next DBOC trial, unused at present' [production]
07:04 <tstarling> synchronized php-1.17/includes/objectcache/SqlBagOStuff.php 'r91442: MySQL parser cache fix, ready for a new trial' [production]
06:54 <root> synchronizing Wikimedia installation... Revision: 91441: [production]
06:53 <tstarling> synchronized php-1.17/includes/AutoLoader.php [production]
06:53 <tstarling> synchronized php-1.17/includes/libs/IEUrlExtension.php [production]
06:48 <Tim> doing svn up/scap to get r91440 [production]
2011-07-04 §
19:29 <mark> Restarted lsearchd on search7 [production]
14:02 <apergos> restarted morebots [production]
04:58 <Tim> setting up lighttpd mod_auth on lily [production]
2011-07-01 §
20:18 <hashar:> ci.tesla: installed exiv2 (required for some Api tests) [production]
20:13 <andrewS> mw1-mw22 racked and drac setup [production]
19:49 <RobH> typo in my number sequence update for the new 74 mw servers, updating dns to fix [production]
19:09 <Ryan_Lane> Changing text backend for https in esams to textsvc IP instead of text.esams.wikimedia.org IP [production]
18:49 <Ryan_Lane> restarting pybal on amslvs1 and amslvs3 [production]
18:34 <midom> synchronized php-1.17/wmf-config/InitialiseSettings.php 'removing mysql from pcache, need to repartition, current scheme too costly on memory' [production]
18:25 <Ryan_Lane> adding textsvc.esams IP to amssq and knsq, and amslvs servers [production]
18:23 <hashar:> ci.tesla install wbritish-small package to get a word list in /usr/share/dict [production]
18:08 <Ryan_Lane> restarting pybal on lvs2 to add textsvc service [production]
17:40 <andrewS> Rob Halsell is a Wizard. Period. [production]
17:39 <midom> synchronized php-1.17/wmf-config/InitialiseSettings.php [production]
17:29 <midom> synchronized php-1.17/wmf-config/InitialiseSettings.php 'adding enwiki to the mix' [production]
17:27 <midom> synchronized php-1.17/wmf-config/InitialiseSettings.php 'sending a bit lower cache load to db40, though I guess that was initial shock before' [production]
17:25 <RobH> Aware that #wikipedia is down and fixed, in less than a minute, woot [production]