|
2011-01-11
§
|
| 23:39 |
<Ryan_Lane> |
adding cron on nova-controller.tesla to svn up /wiki hourly |
[production] |
| 22:27 |
<RobH> |
have to reinstall the squids, wrong version written into lease |
[production] |
| 22:24 |
<awjr> |
archived hudson build files for 'donations queue consume' and emptied builds directory on grosley to allow the jobs to continue running (donations queue consume job was failing due to hitting max # of files in a dir for ext3 fs) |
[production] |
| 22:05 |
<RobH> |
correction, sq65, sq66, & sq71 |
[production] |
| 22:04 |
<RobH> |
sq62-sq64 back online, sq65-sq67 coming down for reinstall |
[production] |
| 21:48 |
<Ryan_Lane> |
adding exim::simple-mail-sender to owa1-3 |
[production] |
| 21:33 |
<RobH> |
sq59-sq61 in service, sq62-sq64 reinstalling |
[production] |
| 21:27 |
<Ryan_Lane> |
powercycling amssq31 |
[production] |
| 20:52 |
<RobH> |
sq59-61 reinstalled and online, pooled, partially up |
[production] |
| 20:42 |
<rainman-sr> |
enwiki.spell index somehow got corrupt, investigating and rebuilding it now on searchidx1 |
[production] |
| 18:57 |
<RobH> |
sq59-sq61 coming back offline, bad partitioning in automatied install, need to update squid configuration for these hosts |
[production] |
| 18:47 |
<RobH> |
sq59 having reinstall issues, skipping it and moving on |
[production] |
| 18:36 |
<RobH> |
sq61 reinstalled and back online |
[production] |
| 18:30 |
<RobH> |
sq60 reinstalled, back in service |
[production] |
| 18:03 |
<RobH> |
sq59-sq61 depooled for upgrade |
[production] |
| 16:15 |
<RobH> |
sync-docroot run to push updated tenwiki favicon |
[production] |
| 14:27 |
<mark_> |
Depooled amssq31 and amssq32 for SSD install |
[production] |
|
2011-01-10
§
|
| 21:01 |
<Ryan_Lane> |
patching python-nova on nova-compute*.tesla (see bug #lp700015) |
[production] |
| 20:59 |
<Ryan_Lane> |
err make that #lp681164 for ldap driver |
[production] |
| 20:59 |
<Ryan_Lane> |
patching ldap driver on nova-* (see bug #lp681030) |
[production] |
| 20:57 |
<Ryan_Lane> |
patching ec2 api on nova-controller.tesla (see bug #lp701216) |
[production] |
| 19:56 |
<RobH> |
updated dns with virt!-virt4 mgmt info |
[production] |
| 16:10 |
<RobH> |
singer was crashed, investigating why it suddenly had issues. It was pulling down db9, but it halted before it could damage anything. |
[production] |
| 16:04 |
<RobH> |
secure server, as well as blogs, offline, investigating server issue on singer |
[production] |
| 16:02 |
<RobH> |
db9 having issues and singer is as well, taking singer down since its already crashed |
[production] |
|
2011-01-07
§
|
| 23:42 |
<davidstrauss> |
Cloning civicrm -> dev_civicrm on Fundraising infrastructure. |
[production] |
| 23:33 |
<Ryan_Lane> |
installed dnsmasq on nova-controller.tesla |
[production] |
| 22:47 |
<Ryan_Lane> |
rebooting nova-* |
[production] |
| 22:45 |
<Ryan_Lane> |
reconfiguring network on nova-* to remove bridges (FlatDHCPManager handles that) |
[production] |
| 22:10 |
<Ryan_Lane> |
rebooting nova-controller.tesla |
[production] |
| 22:01 |
<Ryan_Lane> |
changing to FlatDHCPManager for networking driver on nova-controller.tesla |
[production] |
| 22:00 |
<Ryan_Lane> |
installing bridge-utils on nova-controller.tesla |
[production] |
| 21:52 |
<RobH> |
srv217 offline, but shows no errors in drac log, rebooted |
[production] |
| 19:03 |
<Ryan_Lane> |
disabling apparmor on nova-compute1/2 to ensure it isn't causing instance creation to fail |
[production] |
| 16:40 |
<Ryan_Lane|sleep> |
adding CNAME svn.mediawiki.org -> svn.wikimedia.org per bug 26474 |
[production] |
| 14:44 |
<RoanKattouw> |
Installed xdebug on prototype so I can turn it on when debugging fatals |
[production] |