|
2009-04-16
§
|
| 22:48 |
<azafred> |
bounced apache on srv217. All threads were DED - dead |
[production] |
| 22:16 |
<tfinc> |
synchronized php-1.5/extensions/ContributionReporting/ContributionHistory_body.php |
[production] |
| 22:08 |
<tfinc> |
synchronized php-1.5/extensions/ContributionReporting/ContributionHistory_body.php |
[production] |
| 17:41 |
<domas> |
fantastic. I start _looking_ at stuff and it fixes itself. |
[production] |
| 17:35 |
<midom> |
synchronized php-1.5/includes/Revision.php 'live profiling hook' |
[production] |
| 17:28 |
<domas> |
db20 has kswapd deadlock, needs reboot soonish |
[production] |
| 17:18 |
<midom> |
synchronized php-1.5/InitialiseSettings.php 'disabled stats' |
[production] |
| 17:15 |
<midom> |
synchronized php-1.5/InitialiseSettings.php 'enabling udp stats' |
[production] |
| 16:18 |
<azafred> |
bounced apache on srv217 (no pid file so previous restart did not include this one) |
[production] |
| 15:57 |
<brion> |
network borkage between Florida and Amsterdam. Visitors through AMS proxies can't reach sites. |
[production] |
| 15:55 |
<azafred> |
bounced apache on srv[73,86,88,93,108,114,139,141,154,181,194,204,213,99] |
[production] |
| 15:52 |
<Tim-away> |
started mysqld on srv98,srv122,srv124,srv142,srv106,srv107: done with them for now. srv102 still going. |
[production] |
| 15:30 |
<mark> |
Set up ms6 with SP management at ms6.ipmi.esams.wikimedia.org |
[production] |
| 14:13 |
<mark> |
Restoring traffic to Amsterdam cluster |
[production] |
| 14:06 |
<mark> |
Reloading csw1-esams |
[production] |
| 13:55 |
<mark> |
Reloading csw1-esams |
[production] |
| 13:53 |
<JeLuF> |
ms1 NFS issues again. Might be load related |
[production] |
| 13:49 |
<Tim> |
copying fedora ES data from ms3 to ms2 |
[production] |
| 13:44 |
<JeLuF> |
ms1 is reachable, no errors logged, NFS daemons running fine. After some minutes, NFS clients were able to access the server again. Root cause unknown. |
[production] |
| 13:38 |
<JeLuF> |
ms1 issues. On NFS slaves: "ls: cannot access /mnt/upload5/: Input/output error" |
[production] |
| 13:24 |
<mark> |
DNS scenario knams-down for upcoming core switch reboot |
[production] |
| 08:23 |
<river> |
pdns on bayle crashed, bindbackend parser seems rather fragile |
[production] |
| 03:01 |
<andrew> |
synchronized php-1.5/InitialiseSettings.php 'Deployed AbuseFilter to ptwiki' |
[production] |
|
2009-04-15
§
|
| 22:42 |
<tomaszf> |
adding ramdisk to db9 to speed up create tmp tables |
[production] |
| 22:34 |
<mark> |
PowerDNS got confused by a commented DNS entry and broke zone wikimedia.org, fixed |
[production] |
| 22:32 |
<brion-codereview> |
DNS broken. mark's poking it |
[production] |
| 22:24 |
<mark> |
Temporarily removed AAAA record from mayflower in DNS |
[production] |
| 22:14 |
<brion-codereview> |
db9 tmpfs full, breaking anything using that db |
[production] |
| 22:00 |
<brion-codereview> |
ipv6 connectivity broken between isidore & mayflower, breaking codereview SVN updates |
[production] |
| 20:59 |
<brion> |
civicrm queries bogging down db9 affecting otrs performance. tom's looking into it |
[production] |
| 18:24 |
<robh> |
synchronized php-1.5/InitialiseSettings.php 'for subpages on ukwikimedia' |
[production] |
| 17:32 |
<robh> |
synchronized php-1.5/InitialiseSettings.php 'Bug 17898 Wiktionary is a bad interwiki prefix on ukwiktionary and mlwiktionary' |
[production] |
| 17:25 |
<robh> |
synchronized php-1.5/InitialiseSettings.php 'per bug 17773 Install Labeled Section Transclusion for dewikiversity' |
[production] |
| 14:33 |
<robh> |
synchronized php-1.5/InitialiseSettings.php 'Bug 17718 Disable CentralNotice on private/fishbowl wikis' |
[production] |
| 14:29 |
<robh> |
synchronized php-1.5/InitialiseSettings.php '18434 Enable the rollback feature on Commons' |
[production] |
| 14:19 |
<robh> |
synchronized php-1.5/InitialiseSettings.php '18307 Add autopatrolled group to English Wikisource' |
[production] |
| 14:12 |
<robh> |
synchronized php-1.5/InitialiseSettings.php 'Bug 17717 Enable subpages on main namespace of UK chapter website' |
[production] |
| 13:55 |
<robh> |
synchronized php-1.5/InitialiseSettings.php 'Bug 18428 cswikisource settings updates' |
[production] |
| 12:38 |
<Tim> |
restarting copy to ms3 |
[production] |
| 12:25 |
<Tim> |
rebooting ms3 with 2.6.28 kernel |
[production] |
| 12:18 |
<Tim> |
running xfs_check on ms3 |
[production] |
| 12:14 |
<Tim> |
restarting ms2 with domas's 2.6.28 kernel |
[production] |
| 12:06 |
<midom> |
synchronized php-1.5/db.php 'removing db25 - apparently it was down for more than a day' |
[production] |
| 11:58 |
<domas> |
db25 went down, resetting |
[production] |
| 11:08 |
<Tim> |
ms3 went down, no response on serial console, rebooting |
[production] |
| 11:05 |
<tstarling> |
synchronized php-1.5/db.php |
[production] |
| 08:32 |
<Tim> |
copy in progress, rsync over ssh controlled via screen on tstarling@zwinger |
[production] |
| 08:23 |
<Tim> |
shutting down mysqld on srv98,srv122,srv124,srv142,srv102,srv106,srv107 for data directory copy to ms3 |
[production] |