|
2012-10-15
§
|
| 09:49 |
<paravoid> |
restarting all imagescaler apaches, did not recover after swift outage |
[production] |
| 09:24 |
<paravoid> |
live-hacking swift on ms-be10, ms-be12, ms-be1 to remove fadvise calls |
[production] |
| 07:53 |
<Tim> |
increasing worker count to 100 on all swift backends, via puppet |
[production] |
| 07:37 |
<Tim> |
ms-be11 showed an immediate improvement in bandwidth out, but wchan still indicates that 48 is not enough, increasing to 100 |
[production] |
| 07:33 |
<Tim> |
experimentally doubling the worker count on ms-be11 since wchan indicates that the worker pool is exhausted |
[production] |
| 05:49 |
<apergos> |
ms-be12 reporting errors on disk /dev/sdi, I've rt'ed it an unmounted it in the meantime |
[production] |
| 04:34 |
<Tim> |
on ms-fe2: restarted swift-proxy due to mem leak |
[production] |
| 04:28 |
<Tim> |
on ms-fe3: restarting swift-proxy due to swap |
[production] |
| 04:26 |
<LeslieCarr> |
restarted swift proxy on ms-fe4 - it was in swap doom |
[production] |
| 02:25 |
<logmsgbot_> |
LocalisationUpdate completed (1.21wmf1) at Mon Oct 15 02:25:12 UTC 2012 |
[production] |
|
2012-10-13
§
|
| 21:39 |
<logmsgbot_> |
preilly Finished syncing Wikimedia installation... : update zero rated mobile access |
[production] |
| 21:18 |
<logmsgbot_> |
preilly Started syncing Wikimedia installation... : update zero rated mobile access |
[production] |
| 21:06 |
<logmsgbot_> |
preilly synchronized php-1.21wmf1/extensions/MobileFrontend 'update for landing page' |
[production] |
| 18:51 |
<logmsgbot_> |
reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to master' |
[production] |
| 18:41 |
<logmsgbot_> |
reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to original again..' |
[production] |
| 18:38 |
<logmsgbot_> |
reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to f980f2380b830e712edfdbce065784d1bcba82fd' |
[production] |
| 18:36 |
<logmsgbot_> |
reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to 4f898444ce5d9c2c9a5491f2303db15a3143a5c5' |
[production] |
| 18:34 |
<logmsgbot_> |
reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to 72b7d4d645489dd092605a2183a944fac33cb40b' |
[production] |
| 18:26 |
<logmsgbot_> |
reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to original' |
[production] |
| 18:25 |
<logmsgbot_> |
reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to 8415a4564bf429b76e113368a3e5e1c19fb84a47' |
[production] |
| 18:23 |
<logmsgbot_> |
reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to 72b7d4d645489dd092605a2183a944fac33cb40b' |
[production] |
| 18:21 |
<logmsgbot_> |
reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to 9f37005e78a068821c45d9d313394c6728f55b4a' |
[production] |
| 16:34 |
<logmsgbot_> |
reedy synchronized php-1.21wmf1/extensions/Collection |
[production] |
| 16:27 |
<logmsgbot_> |
reedy synchronized php-1.21wmf1/extensions/Collection |
[production] |
| 13:17 |
<logmsgbot_> |
reedy synchronized wmf-config/throttle.php |
[production] |
| 02:00 |
<logmsgbot_> |
LocalisationUpdate failed: git pull of extensions failed |
[production] |
|
2012-10-12
§
|
| 23:51 |
<logmsgbot_> |
asher synchronized wmf-config/CommonSettings.php 'enabling redis debug log' |
[production] |
| 23:40 |
<LeslieCarr> |
moving traffic back to esams via authdns-scenario |
[production] |
| 23:40 |
<LeslieCarr> |
reenabled nagios alerts |
[production] |
| 23:20 |
<logmsgbot_> |
asher synchronized wmf-config/CommonSettings.php 'enabling session multiwrites to redis' |
[production] |
| 23:16 |
<LeslieCarr> |
rebooting esams core switch stack, expect reachability problems to esams |
[production] |
| 23:01 |
<logmsgbot_> |
asher synchronized wmf-config/CommonSettings.php 'disabling session multiwrites to redis' |
[production] |
| 22:36 |
<LeslieCarr> |
switching europe traffic to pmtpa via authdns-scenario |
[production] |
| 22:36 |
<LeslieCarr> |
muted nagios |
[production] |
| 20:20 |
<AaronSchulz> |
Confirmed all swift -> netapp sync scripts finished (s4 one kept dying to random http errors) |
[production] |
| 19:58 |
<logmsgbot_> |
reedy synchronized wmf-config/ |
[production] |
| 19:01 |
<cmjohnson1> |
powering down search32 to replace cpu1 |
[production] |
| 18:33 |
<logmsgbot_> |
reedy synchronized wmf-config/InitialiseSettings.php 'AFT (non v5) doesn't use clicktracking now' |
[production] |
| 18:28 |
<logmsgbot_> |
reedy synchronized php-1.21wmf1/extensions/ArticleFeedback/ |
[production] |
| 18:19 |
<cmjohnson1> |
troubleshooting db42 w/IBM tech |
[production] |
| 17:46 |
<cmjohnson1> |
pc3 shutting down to replace hdd's with ssd's rt3703 |
[production] |
| 17:29 |
<cmjohnson1> |
pc2 shutting down to replaced hdd w/ ssd rt3703 |
[production] |
| 16:27 |
<logmsgbot_> |
reedy synchronized wmf-config/CommonSettings.php 'Fix extdist mount related warnings' |
[production] |
| 16:14 |
<cmjohnson1> |
running installer on storage3 |
[production] |
| 15:54 |
<paravoid> |
moving bits from esams to pmtpa (authdns normal scenario) |
[production] |
| 15:52 |
<cmjohnson1> |
storage 3 replacing disk 12 and rebuilding raid |
[production] |