|
2011-07-06
§
|
| 15:37 |
<mark> |
Removed ms5:/etc/cron.d/mdadm |
[production] |
| 15:37 |
<mark> |
Stopped MD raid resync on ms5 |
[production] |
| 15:28 |
<RobH> |
search18 booted back up successfully |
[production] |
| 15:25 |
<RobH> |
api lag issues known due to search server failure, being worked presently |
[production] |
| 15:24 |
<RobH> |
search18 sas configuration bios confirms both disks are still in a non-degraded (according to it) mirror |
[production] |
| 15:23 |
<RobH> |
search18 randomly rebooted after checking disks before the login prompt |
[production] |
| 15:19 |
<RobH> |
rebooting search18 |
[production] |
| 15:14 |
<RobH> |
search18 appears to be completely offline, investigating lom logs before rebooting. |
[production] |
| 15:12 |
<RobH> |
search18 offline, logging into mgmt to check it out |
[production] |
| 15:01 |
<RobH> |
eqiad humidity levels ticket dispatched for fufillment |
[production] |
| 14:37 |
<mark> |
Paused rsyncs on ms5 |
[production] |
| 14:04 |
<mark> |
Powercycled sq36 |
[production] |
| 13:18 |
<^demon|away> |
fixed permissions on svn c/o on ci.tesla, ran svn cleanup. cruise control still not pleased and yelling about locks |
[production] |
| 13:16 |
<mark> |
Upgrading firmware of scs-a1-sdtpa |
[production] |
| 12:51 |
<mark> |
csw5-pmtpa crashed and reloaded |
[production] |
| 11:53 |
<mark> |
Upgrading firmware of scs-c1-pmtpa |
[production] |
|
2011-07-05
§
|
| 23:53 |
<^demon> |
scratch that....ssh just seems to have been rather slow in getting its act together. ci.tesla is just fine now |
[production] |
| 23:50 |
<^demon> |
well now I've locked myself out of ci.tesla. Seems it doesn't start ssh on reboot...what a silly thing to do |
[production] |
| 23:47 |
<^demon> |
rebooting ci.tesla since it was horribly hung up on the latest build--was it really stuck for the past 24hrs? |
[production] |
| 23:35 |
<reedy> |
synchronized php-1.17/resources/mediawiki/mediawiki.js 'r91505' |
[production] |
| 21:28 |
<pdhanda> |
ran sync-common-all 'Synced to r91494 for WikiLove' |
[production] |
| 21:10 |
<pdhanda> |
synchronized php-1.17/resources/jquery.ui/themes/vector/jquery.ui.button.css 'Synced to r 91493.' |
[production] |
| 21:00 |
<pdhanda> |
synchronized php-1.17/resources/jquery.ui/themes/vector/jquery.ui.button.css 'Synced to r 91490.' |
[production] |
| 19:53 |
<reedy> |
synchronized php-1.17/includes/api/ApiQuery.php 'r91479' |
[production] |
| 19:07 |
<catrope> |
synchronized php-1.17/extensions/Vector/modules/ext.vector.collapsibleTabs.js 'r91476' |
[production] |
| 19:07 |
<catrope> |
synchronized php-1.17/extensions/Vector/modules/ext.vector.simpleSearch.js 'r91476' |
[production] |
| 16:49 |
<RoanKattouw> |
Short CPU spike on the Apaches, approx 16:45-16:50 UTC. Things seem to be recovering now |
[production] |
| 16:04 |
<mark> |
Put OSPF/OSPFv3 on csw1-sdtpa:e14/1 in active mode again; appears stable so far |
[production] |
| 15:45 |
<mark> |
Reenabled csw1-sdtpa:e14/1 with OSPF/OSPFv3 in passive mode and high metric |
[production] |
| 15:31 |
<mark> |
Shutdown csw1-sdtpa:e14/1 (l3 wave) |
[production] |
| 15:24 |
<mark> |
Disabled IP load-sharing on csw1-esams (out of next-hops), disabled e14/1 (10G wave L3), reenabled it |
[production] |
| 13:38 |
<mark> |
Rebooting streber; dpkg hung |
[production] |
| 08:34 |
<mark> |
Rerouted traffic AS14907 -> AS43821 via 2828 |
[production] |
| 07:11 |
<Tim> |
creating 256 tables on db40 for DBOC |
[production] |
| 07:08 |
<tstarling> |
synchronized php-1.17/wmf-config/CommonSettings.php 'config change for the next DBOC trial, unused at present' |
[production] |
| 07:04 |
<tstarling> |
synchronized php-1.17/includes/objectcache/SqlBagOStuff.php 'r91442: MySQL parser cache fix, ready for a new trial' |
[production] |
| 06:54 |
<root> |
synchronizing Wikimedia installation... Revision: 91441: |
[production] |
| 06:53 |
<tstarling> |
synchronized php-1.17/includes/AutoLoader.php |
[production] |
| 06:53 |
<tstarling> |
synchronized php-1.17/includes/libs/IEUrlExtension.php |
[production] |
| 06:48 |
<Tim> |
doing svn up/scap to get r91440 |
[production] |
|
2011-07-01
§
|
| 20:18 |
<hashar:> |
ci.tesla: installed exiv2 (required for some Api tests) |
[production] |
| 20:13 |
<andrewS> |
mw1-mw22 racked and drac setup |
[production] |
| 19:49 |
<RobH> |
typo in my number sequence update for the new 74 mw servers, updating dns to fix |
[production] |
| 19:09 |
<Ryan_Lane> |
Changing text backend for https in esams to textsvc IP instead of text.esams.wikimedia.org IP |
[production] |
| 18:49 |
<Ryan_Lane> |
restarting pybal on amslvs1 and amslvs3 |
[production] |
| 18:34 |
<midom> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'removing mysql from pcache, need to repartition, current scheme too costly on memory' |
[production] |
| 18:25 |
<Ryan_Lane> |
adding textsvc.esams IP to amssq and knsq, and amslvs servers |
[production] |