|
2009-04-24
ยง
|
| 21:19 |
<rainman_> |
srv58 dead, making all non-major wikis search broken, transfering the service to search11/12.... |
[production] |
| 19:50 |
<Rob> |
srv90-srv99 ganglia installed. |
[production] |
| 19:50 |
<Rob> |
srv97 online |
[production] |
| 19:47 |
<Rob> |
srv98 online |
[production] |
| 19:46 |
<Rob> |
srv96 online |
[production] |
| 19:45 |
<Rob> |
srv99 online |
[production] |
| 19:42 |
<Rob> |
srv95 online |
[production] |
| 19:40 |
<Rob> |
srv92, srv93, and srv94 back online |
[production] |
| 19:39 |
<Rob> |
srv91 back online |
[production] |
| 19:24 |
<Rob> |
srv90 online |
[production] |
| 19:16 |
<Rob> |
srv90-srv99 reinstalled, currently looping though package installation |
[production] |
| 18:34 |
<mark> |
Fixed ganglia by installing the appropriate config files on the (reinstalled) aggregation hosts |
[production] |
| 18:28 |
<Rob> |
installed ganglia on all servers reinstalled to ubuntu apache thus far today. |
[production] |
| 18:27 |
<Rob> |
srv89 back online |
[production] |
| 18:17 |
<Rob> |
srv90-srv99 will be down over the next 30 minutes for ubuntufication. |
[production] |
| 18:16 |
<robh> |
synchronized php-1.5/mc-pmtpa.php 'some spares were actually down' |
[production] |
| 18:14 |
<robh> |
synchronized php-1.5/mc-pmtpa.php 'removed the 9x servers for reinstallation' |
[production] |
| 18:02 |
<Rob> |
srv84 ubuntufied and online |
[production] |
| 17:58 |
<Rob> |
srv83 ubuntufied and online |
[production] |
| 17:54 |
<Rob> |
srv82 ubuntufied and online |
[production] |
| 17:50 |
<Rob> |
srv81 reinstalled and online |
[production] |
| 17:47 |
<Rob> |
srv89 coming down for reinstall |
[production] |
| 17:44 |
<Rob> |
srv58 online |
[production] |
| 17:38 |
<Rob> |
srv57 online |
[production] |
| 17:26 |
<Rob> |
reinstalling srv58 |
[production] |
| 17:16 |
<mark> |
Set up switchport for srv57 on asw-c4-pmtpa |
[production] |
| 17:10 |
<Rob> |
reinstalling srv57 |
[production] |
| 17:08 |
<Rob> |
srv75 back online |
[production] |
| 17:02 |
<Rob> |
srv74 back online |
[production] |
| 16:47 |
<Rob> |
srv73 back online as apache |
[production] |
| 16:45 |
<robh> |
synchronized php-1.5/mc-pmtpa.php 'removed srv75' |
[production] |
| 16:43 |
<Rob> |
srv71, srv72 back online as apache |
[production] |
| 16:43 |
<Rob> |
taking down srv75 for reinstall |
[production] |
| 16:37 |
<fvassard> |
synchronized php-1.5/mc-pmtpa.php 'swapping out srv72 for srv100 and srv73 for srv101 while srv[72,73] are being ubuntified' |
[production] |
| 16:26 |
<Rob> |
srv72, srv73, and srv74 down for reinstallation |
[production] |
| 16:23 |
<root> |
synchronized php-1.5/mc-pmtpa.php 'swapping out srv71 for srv70 and srv74 for srv92 while srv[71,74] are being ubuntified' |
[production] |
| 16:05 |
<Rob> |
srv34 back online reinstalled as ubuntu |
[production] |
| 16:04 |
<Rob> |
reinstalling srv71 |
[production] |
| 16:04 |
<Fred> |
restarted apache on srv99 |
[production] |
| 15:21 |
<Rob> |
srv34 coming down for reinstall |
[production] |
| 15:13 |
<Rob> |
amane reinstalled for tomasz |
[production] |
| 14:59 |
<Rob> |
amane reinstall started |
[production] |
| 14:36 |
<rainman-sr> |
search9,10 also up; everything should be normal again |
[production] |
| 14:33 |
<Rob> |
amane shutting down for rain controller work |
[production] |
| 14:27 |
<rainman-sr> |
search5-8 back in search pool |
[production] |
| 14:16 |
<Rob> |
shutting down search9 & search10 for memory upgrade |
[production] |
| 14:15 |
<Rob> |
search7 & search8 memory upgraded, systems rebooted |
[production] |
| 14:07 |
<Rob> |
search5 and search6 back online. |
[production] |
| 14:05 |
<Rob> |
memory upgrade complete on search5 & search6, rebooted. |
[production] |
| 14:02 |
<rainman-sr> |
done with initial index warmup on search3,4, back in rotation |
[production] |