|
2009-04-24
§
|
| 17:10 |
<Rob> |
reinstalling srv57 |
[production] |
| 17:08 |
<Rob> |
srv75 back online |
[production] |
| 17:02 |
<Rob> |
srv74 back online |
[production] |
| 16:47 |
<Rob> |
srv73 back online as apache |
[production] |
| 16:45 |
<robh> |
synchronized php-1.5/mc-pmtpa.php 'removed srv75' |
[production] |
| 16:43 |
<Rob> |
srv71, srv72 back online as apache |
[production] |
| 16:43 |
<Rob> |
taking down srv75 for reinstall |
[production] |
| 16:37 |
<fvassard> |
synchronized php-1.5/mc-pmtpa.php 'swapping out srv72 for srv100 and srv73 for srv101 while srv[72,73] are being ubuntified' |
[production] |
| 16:26 |
<Rob> |
srv72, srv73, and srv74 down for reinstallation |
[production] |
| 16:23 |
<root> |
synchronized php-1.5/mc-pmtpa.php 'swapping out srv71 for srv70 and srv74 for srv92 while srv[71,74] are being ubuntified' |
[production] |
| 16:05 |
<Rob> |
srv34 back online reinstalled as ubuntu |
[production] |
| 16:04 |
<Rob> |
reinstalling srv71 |
[production] |
| 16:04 |
<Fred> |
restarted apache on srv99 |
[production] |
| 15:21 |
<Rob> |
srv34 coming down for reinstall |
[production] |
| 15:13 |
<Rob> |
amane reinstalled for tomasz |
[production] |
| 14:59 |
<Rob> |
amane reinstall started |
[production] |
| 14:36 |
<rainman-sr> |
search9,10 also up; everything should be normal again |
[production] |
| 14:33 |
<Rob> |
amane shutting down for rain controller work |
[production] |
| 14:27 |
<rainman-sr> |
search5-8 back in search pool |
[production] |
| 14:16 |
<Rob> |
shutting down search9 & search10 for memory upgrade |
[production] |
| 14:15 |
<Rob> |
search7 & search8 memory upgraded, systems rebooted |
[production] |
| 14:07 |
<Rob> |
search5 and search6 back online. |
[production] |
| 14:05 |
<Rob> |
memory upgrade complete on search5 & search6, rebooted. |
[production] |
| 14:02 |
<rainman-sr> |
done with initial index warmup on search3,4, back in rotation |
[production] |
| 13:59 |
<Rob> |
search5, search6 shutdown for memory upgrade |
[production] |
| 13:58 |
<Rob> |
search4 memory upgraded and system back online |
[production] |
| 13:55 |
<Rob> |
search3 ram upgraded and system is back online |
[production] |
| 13:50 |
<Rob> |
search3 upgraded, rebooting. |
[production] |
| 13:44 |
<Rob> |
shutdown search3 & search4 for memory upgrades |
[production] |
| 07:18 |
<tstarling> |
synchronized php-1.5/db.php |
[production] |
| 03:35 |
<andrew> |
synchronized php-1.5/InitialiseSettings.php |
[production] |
| 03:35 |
<Andrew> |
Deployed AbuseFilter to fiwiki |
[production] |
| 02:51 |
<tstarling> |
synchronized php-1.5/mc-pmtpa.php |
[production] |
| 02:46 |
<Tim> |
srv127 has corrupted root partition, needs reinstall or repair. Shut down with echo o > /proc/sysrq-trigger. |
[production] |
| 02:36 |
<tstarling> |
synchronized php-1.5/mc-pmtpa.php |
[production] |
| 02:31 |
<Tim> |
killed srv124 with /proc/sysrq-trigger. Was very slow on ssh and was giving odd 403 errors via HTTP. |
[production] |
| 02:21 |
<tstarling> |
synchronized php-1.5/README |
[production] |
| 02:12 |
<andrew> |
synchronized php-1.5/CommonSettings.php |
[production] |
| 02:10 |
<Andrew> |
srv127: rsync: mkstemp "/apache/common/php-1.5/.CommonSettings.php.TRNqkG" failed: Read-only file system (30) |
[production] |
| 01:15 |
<tstarling> |
synchronized php-1.5/db.php |
[production] |
| 01:14 |
<Tim> |
depooled db3 so that it can finish doing the querycache update without making lots of people wait for a MASTER_POS_WAIT |
[production] |
| 01:03 |
<tstarling> |
synchronized php-1.5/InitialiseSettings.php |
[production] |
| 01:03 |
<Tim> |
blacklisted Wantedtemplates on enwiki, has been running for more than a day. |
[production] |
| 00:54 |
<Tim> |
restarting trackBlobs.php on hume for afwiki and enwiki |
[production] |