|
2009-04-16
§
|
| 15:57 |
<brion> |
network borkage between Florida and Amsterdam. Visitors through AMS proxies can't reach sites. |
[production] |
| 15:55 |
<azafred> |
bounced apache on srv[73,86,88,93,108,114,139,141,154,181,194,204,213,99] |
[production] |
| 15:52 |
<Tim-away> |
started mysqld on srv98,srv122,srv124,srv142,srv106,srv107: done with them for now. srv102 still going. |
[production] |
| 15:30 |
<mark> |
Set up ms6 with SP management at ms6.ipmi.esams.wikimedia.org |
[production] |
| 14:13 |
<mark> |
Restoring traffic to Amsterdam cluster |
[production] |
| 14:06 |
<mark> |
Reloading csw1-esams |
[production] |
| 13:55 |
<mark> |
Reloading csw1-esams |
[production] |
| 13:53 |
<JeLuF> |
ms1 NFS issues again. Might be load related |
[production] |
| 13:49 |
<Tim> |
copying fedora ES data from ms3 to ms2 |
[production] |
| 13:44 |
<JeLuF> |
ms1 is reachable, no errors logged, NFS daemons running fine. After some minutes, NFS clients were able to access the server again. Root cause unknown. |
[production] |
| 13:38 |
<JeLuF> |
ms1 issues. On NFS slaves: "ls: cannot access /mnt/upload5/: Input/output error" |
[production] |
| 13:24 |
<mark> |
DNS scenario knams-down for upcoming core switch reboot |
[production] |
| 08:23 |
<river> |
pdns on bayle crashed, bindbackend parser seems rather fragile |
[production] |
| 03:01 |
<andrew> |
synchronized php-1.5/InitialiseSettings.php 'Deployed AbuseFilter to ptwiki' |
[production] |
|
2009-04-15
§
|
| 22:42 |
<tomaszf> |
adding ramdisk to db9 to speed up create tmp tables |
[production] |
| 22:34 |
<mark> |
PowerDNS got confused by a commented DNS entry and broke zone wikimedia.org, fixed |
[production] |
| 22:32 |
<brion-codereview> |
DNS broken. mark's poking it |
[production] |
| 22:24 |
<mark> |
Temporarily removed AAAA record from mayflower in DNS |
[production] |
| 22:14 |
<brion-codereview> |
db9 tmpfs full, breaking anything using that db |
[production] |
| 22:00 |
<brion-codereview> |
ipv6 connectivity broken between isidore & mayflower, breaking codereview SVN updates |
[production] |
| 20:59 |
<brion> |
civicrm queries bogging down db9 affecting otrs performance. tom's looking into it |
[production] |
| 18:24 |
<robh> |
synchronized php-1.5/InitialiseSettings.php 'for subpages on ukwikimedia' |
[production] |
| 17:32 |
<robh> |
synchronized php-1.5/InitialiseSettings.php 'Bug 17898 Wiktionary is a bad interwiki prefix on ukwiktionary and mlwiktionary' |
[production] |
| 17:25 |
<robh> |
synchronized php-1.5/InitialiseSettings.php 'per bug 17773 Install Labeled Section Transclusion for dewikiversity' |
[production] |
| 14:33 |
<robh> |
synchronized php-1.5/InitialiseSettings.php 'Bug 17718 Disable CentralNotice on private/fishbowl wikis' |
[production] |
| 14:29 |
<robh> |
synchronized php-1.5/InitialiseSettings.php '18434 Enable the rollback feature on Commons' |
[production] |
| 14:19 |
<robh> |
synchronized php-1.5/InitialiseSettings.php '18307 Add autopatrolled group to English Wikisource' |
[production] |
| 14:12 |
<robh> |
synchronized php-1.5/InitialiseSettings.php 'Bug 17717 Enable subpages on main namespace of UK chapter website' |
[production] |
| 13:55 |
<robh> |
synchronized php-1.5/InitialiseSettings.php 'Bug 18428 cswikisource settings updates' |
[production] |
| 12:38 |
<Tim> |
restarting copy to ms3 |
[production] |
| 12:25 |
<Tim> |
rebooting ms3 with 2.6.28 kernel |
[production] |
| 12:18 |
<Tim> |
running xfs_check on ms3 |
[production] |
| 12:14 |
<Tim> |
restarting ms2 with domas's 2.6.28 kernel |
[production] |
| 12:06 |
<midom> |
synchronized php-1.5/db.php 'removing db25 - apparently it was down for more than a day' |
[production] |
| 11:58 |
<domas> |
db25 went down, resetting |
[production] |
| 11:08 |
<Tim> |
ms3 went down, no response on serial console, rebooting |
[production] |
| 11:05 |
<tstarling> |
synchronized php-1.5/db.php |
[production] |
| 08:32 |
<Tim> |
copy in progress, rsync over ssh controlled via screen on tstarling@zwinger |
[production] |
| 08:23 |
<Tim> |
shutting down mysqld on srv98,srv122,srv124,srv142,srv102,srv106,srv107 for data directory copy to ms3 |
[production] |
|
2009-04-14
§
|
| 23:48 |
<tfinc> |
synchronized php-1.5/extensions/ContributionTracking/ContributionTracking_body.php |
[production] |
| 23:39 |
<tfinc> |
synchronized php-1.5/extensions/ContributionTracking/ContributionTracking_body.php |
[production] |
| 23:37 |
<tfinc> |
synchronized php-1.5/reporting-setup.php |
[production] |
| 19:01 |
<Rob> |
replaced dead drive in ms4 |
[production] |
| 18:41 |
<Rob> |
srv78 back online |
[production] |
| 18:37 |
<Rob> |
srv78 was wonky and such, reinstalled to fix. |
[production] |
| 18:21 |
<Rob> |
srv90 reinstalled and redeployed |
[production] |
| 18:21 |
<Rob> |
memcached had stopped on srv89, restarted. |
[production] |
| 18:16 |
<Rob> |
all fans are good on srv86, bringing back online. |
[production] |
| 18:13 |
<Rob> |
srv86 has temp warnings, shutting down to check fans and such |
[production] |
| 17:59 |
<Rob> |
reinstalling srv90 from FC to ubuntu |
[production] |