|
2009-04-15
§
|
| 22:34 |
<mark> |
PowerDNS got confused by a commented DNS entry and broke zone wikimedia.org, fixed |
[production] |
| 22:32 |
<brion-codereview> |
DNS broken. mark's poking it |
[production] |
| 22:24 |
<mark> |
Temporarily removed AAAA record from mayflower in DNS |
[production] |
| 22:14 |
<brion-codereview> |
db9 tmpfs full, breaking anything using that db |
[production] |
| 22:00 |
<brion-codereview> |
ipv6 connectivity broken between isidore & mayflower, breaking codereview SVN updates |
[production] |
| 20:59 |
<brion> |
civicrm queries bogging down db9 affecting otrs performance. tom's looking into it |
[production] |
| 18:24 |
<robh> |
synchronized php-1.5/InitialiseSettings.php 'for subpages on ukwikimedia' |
[production] |
| 17:32 |
<robh> |
synchronized php-1.5/InitialiseSettings.php 'Bug 17898 Wiktionary is a bad interwiki prefix on ukwiktionary and mlwiktionary' |
[production] |
| 17:25 |
<robh> |
synchronized php-1.5/InitialiseSettings.php 'per bug 17773 Install Labeled Section Transclusion for dewikiversity' |
[production] |
| 14:33 |
<robh> |
synchronized php-1.5/InitialiseSettings.php 'Bug 17718 Disable CentralNotice on private/fishbowl wikis' |
[production] |
| 14:29 |
<robh> |
synchronized php-1.5/InitialiseSettings.php '18434 Enable the rollback feature on Commons' |
[production] |
| 14:19 |
<robh> |
synchronized php-1.5/InitialiseSettings.php '18307 Add autopatrolled group to English Wikisource' |
[production] |
| 14:12 |
<robh> |
synchronized php-1.5/InitialiseSettings.php 'Bug 17717 Enable subpages on main namespace of UK chapter website' |
[production] |
| 13:55 |
<robh> |
synchronized php-1.5/InitialiseSettings.php 'Bug 18428 cswikisource settings updates' |
[production] |
| 12:38 |
<Tim> |
restarting copy to ms3 |
[production] |
| 12:25 |
<Tim> |
rebooting ms3 with 2.6.28 kernel |
[production] |
| 12:18 |
<Tim> |
running xfs_check on ms3 |
[production] |
| 12:14 |
<Tim> |
restarting ms2 with domas's 2.6.28 kernel |
[production] |
| 12:06 |
<midom> |
synchronized php-1.5/db.php 'removing db25 - apparently it was down for more than a day' |
[production] |
| 11:58 |
<domas> |
db25 went down, resetting |
[production] |
| 11:08 |
<Tim> |
ms3 went down, no response on serial console, rebooting |
[production] |
| 11:05 |
<tstarling> |
synchronized php-1.5/db.php |
[production] |
| 08:32 |
<Tim> |
copy in progress, rsync over ssh controlled via screen on tstarling@zwinger |
[production] |
| 08:23 |
<Tim> |
shutting down mysqld on srv98,srv122,srv124,srv142,srv102,srv106,srv107 for data directory copy to ms3 |
[production] |
|
2009-04-14
§
|
| 23:48 |
<tfinc> |
synchronized php-1.5/extensions/ContributionTracking/ContributionTracking_body.php |
[production] |
| 23:39 |
<tfinc> |
synchronized php-1.5/extensions/ContributionTracking/ContributionTracking_body.php |
[production] |
| 23:37 |
<tfinc> |
synchronized php-1.5/reporting-setup.php |
[production] |
| 19:01 |
<Rob> |
replaced dead drive in ms4 |
[production] |
| 18:41 |
<Rob> |
srv78 back online |
[production] |
| 18:37 |
<Rob> |
srv78 was wonky and such, reinstalled to fix. |
[production] |
| 18:21 |
<Rob> |
srv90 reinstalled and redeployed |
[production] |
| 18:21 |
<Rob> |
memcached had stopped on srv89, restarted. |
[production] |
| 18:16 |
<Rob> |
all fans are good on srv86, bringing back online. |
[production] |
| 18:13 |
<Rob> |
srv86 has temp warnings, shutting down to check fans and such |
[production] |
| 17:59 |
<Rob> |
reinstalling srv90 from FC to ubuntu |
[production] |
| 17:52 |
<Rob> |
replaced bad fan in srv90 |
[production] |
| 17:40 |
<Rob> |
pulling srv90, overheating warnings. |
[production] |
| 17:38 |
<Rob> |
srv85 overheating due to dead fans. server is old and out of warranty, decommissioned but kept on site for parting out. |
[production] |
| 17:10 |
<Rob> |
bringing back up sq1, no memory on hand for upgrading these (They are ddr pc3200, all the spare memory we have is ddr2 or ddr 2700) |
[production] |
| 17:02 |
<Rob> |
pulling sq1 for memory upgrade. |
[production] |
| 16:55 |
<Rob> |
replaced bad patch for search9, LOM functions properly. |
[production] |
| 16:38 |
<Rob> |
Upgraded memory in search1 and search2 to a total of 16GB each (previously 8). |
[production] |
| 16:14 |
<Rob> |
Had to restart wikitech due to OOM issues, again. Perhaps it is time to up the memory in the machine or tweak settings. |
[production] |
| 05:05 |
<Andrew> |
testwiki problem seems to be a squid problem, can get srv123, srv84 to serve the main page with no problems by sending a request through netcat. Trying to connect to rr just gets no response |
[production] |
| 05:01 |
<Andrew> |
testwiki seems fubar, timing out on all pageviews. |
[production] |
| 03:47 |
<andrew> |
synchronized php-1.5/InitialiseSettings.php |
[production] |
| 03:45 |
<Andrew> |
Installing AbuseFilter on alswiki |
[production] |