2009-04-24
ยง
|
19:50 |
<Rob> |
srv90-srv99 ganglia installed. |
[production] |
19:50 |
<Rob> |
srv97 online |
[production] |
19:47 |
<Rob> |
srv98 online |
[production] |
19:46 |
<Rob> |
srv96 online |
[production] |
19:45 |
<Rob> |
srv99 online |
[production] |
19:42 |
<Rob> |
srv95 online |
[production] |
19:40 |
<Rob> |
srv92, srv93, and srv94 back online |
[production] |
19:39 |
<Rob> |
srv91 back online |
[production] |
19:24 |
<Rob> |
srv90 online |
[production] |
19:16 |
<Rob> |
srv90-srv99 reinstalled, currently looping though package installation |
[production] |
18:34 |
<mark> |
Fixed ganglia by installing the appropriate config files on the (reinstalled) aggregation hosts |
[production] |
18:28 |
<Rob> |
installed ganglia on all servers reinstalled to ubuntu apache thus far today. |
[production] |
18:27 |
<Rob> |
srv89 back online |
[production] |
18:17 |
<Rob> |
srv90-srv99 will be down over the next 30 minutes for ubuntufication. |
[production] |
18:16 |
<robh> |
synchronized php-1.5/mc-pmtpa.php 'some spares were actually down' |
[production] |
18:14 |
<robh> |
synchronized php-1.5/mc-pmtpa.php 'removed the 9x servers for reinstallation' |
[production] |
18:02 |
<Rob> |
srv84 ubuntufied and online |
[production] |
17:58 |
<Rob> |
srv83 ubuntufied and online |
[production] |
17:54 |
<Rob> |
srv82 ubuntufied and online |
[production] |
17:50 |
<Rob> |
srv81 reinstalled and online |
[production] |
17:47 |
<Rob> |
srv89 coming down for reinstall |
[production] |
17:44 |
<Rob> |
srv58 online |
[production] |
17:38 |
<Rob> |
srv57 online |
[production] |
17:26 |
<Rob> |
reinstalling srv58 |
[production] |
17:16 |
<mark> |
Set up switchport for srv57 on asw-c4-pmtpa |
[production] |
17:10 |
<Rob> |
reinstalling srv57 |
[production] |
17:08 |
<Rob> |
srv75 back online |
[production] |
17:02 |
<Rob> |
srv74 back online |
[production] |
16:47 |
<Rob> |
srv73 back online as apache |
[production] |
16:45 |
<robh> |
synchronized php-1.5/mc-pmtpa.php 'removed srv75' |
[production] |
16:43 |
<Rob> |
srv71, srv72 back online as apache |
[production] |
16:43 |
<Rob> |
taking down srv75 for reinstall |
[production] |
16:37 |
<fvassard> |
synchronized php-1.5/mc-pmtpa.php 'swapping out srv72 for srv100 and srv73 for srv101 while srv[72,73] are being ubuntified' |
[production] |
16:26 |
<Rob> |
srv72, srv73, and srv74 down for reinstallation |
[production] |
16:23 |
<root> |
synchronized php-1.5/mc-pmtpa.php 'swapping out srv71 for srv70 and srv74 for srv92 while srv[71,74] are being ubuntified' |
[production] |
16:05 |
<Rob> |
srv34 back online reinstalled as ubuntu |
[production] |
16:04 |
<Rob> |
reinstalling srv71 |
[production] |
16:04 |
<Fred> |
restarted apache on srv99 |
[production] |
15:21 |
<Rob> |
srv34 coming down for reinstall |
[production] |
15:13 |
<Rob> |
amane reinstalled for tomasz |
[production] |
14:59 |
<Rob> |
amane reinstall started |
[production] |
14:36 |
<rainman-sr> |
search9,10 also up; everything should be normal again |
[production] |
14:33 |
<Rob> |
amane shutting down for rain controller work |
[production] |
14:27 |
<rainman-sr> |
search5-8 back in search pool |
[production] |
14:16 |
<Rob> |
shutting down search9 & search10 for memory upgrade |
[production] |
14:15 |
<Rob> |
search7 & search8 memory upgraded, systems rebooted |
[production] |
14:07 |
<Rob> |
search5 and search6 back online. |
[production] |
14:05 |
<Rob> |
memory upgrade complete on search5 & search6, rebooted. |
[production] |
14:02 |
<rainman-sr> |
done with initial index warmup on search3,4, back in rotation |
[production] |
13:59 |
<Rob> |
search5, search6 shutdown for memory upgrade |
[production] |