2012-10-15
§
|
07:53 |
<Tim> |
increasing worker count to 100 on all swift backends, via puppet |
[production] |
07:37 |
<Tim> |
ms-be11 showed an immediate improvement in bandwidth out, but wchan still indicates that 48 is not enough, increasing to 100 |
[production] |
07:33 |
<Tim> |
experimentally doubling the worker count on ms-be11 since wchan indicates that the worker pool is exhausted |
[production] |
05:49 |
<apergos> |
ms-be12 reporting errors on disk /dev/sdi, I've rt'ed it an unmounted it in the meantime |
[production] |
04:34 |
<Tim> |
on ms-fe2: restarted swift-proxy due to mem leak |
[production] |
04:28 |
<Tim> |
on ms-fe3: restarting swift-proxy due to swap |
[production] |
04:26 |
<LeslieCarr> |
restarted swift proxy on ms-fe4 - it was in swap doom |
[production] |
02:25 |
<logmsgbot_> |
LocalisationUpdate completed (1.21wmf1) at Mon Oct 15 02:25:12 UTC 2012 |
[production] |
2012-10-13
§
|
21:39 |
<logmsgbot_> |
preilly Finished syncing Wikimedia installation... : update zero rated mobile access |
[production] |
21:18 |
<logmsgbot_> |
preilly Started syncing Wikimedia installation... : update zero rated mobile access |
[production] |
21:06 |
<logmsgbot_> |
preilly synchronized php-1.21wmf1/extensions/MobileFrontend 'update for landing page' |
[production] |
18:51 |
<logmsgbot_> |
reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to master' |
[production] |
18:41 |
<logmsgbot_> |
reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to original again..' |
[production] |
18:38 |
<logmsgbot_> |
reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to f980f2380b830e712edfdbce065784d1bcba82fd' |
[production] |
18:36 |
<logmsgbot_> |
reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to 4f898444ce5d9c2c9a5491f2303db15a3143a5c5' |
[production] |
18:34 |
<logmsgbot_> |
reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to 72b7d4d645489dd092605a2183a944fac33cb40b' |
[production] |
18:26 |
<logmsgbot_> |
reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to original' |
[production] |
18:25 |
<logmsgbot_> |
reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to 8415a4564bf429b76e113368a3e5e1c19fb84a47' |
[production] |
18:23 |
<logmsgbot_> |
reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to 72b7d4d645489dd092605a2183a944fac33cb40b' |
[production] |
18:21 |
<logmsgbot_> |
reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to 9f37005e78a068821c45d9d313394c6728f55b4a' |
[production] |
16:34 |
<logmsgbot_> |
reedy synchronized php-1.21wmf1/extensions/Collection |
[production] |
16:27 |
<logmsgbot_> |
reedy synchronized php-1.21wmf1/extensions/Collection |
[production] |
13:17 |
<logmsgbot_> |
reedy synchronized wmf-config/throttle.php |
[production] |
02:00 |
<logmsgbot_> |
LocalisationUpdate failed: git pull of extensions failed |
[production] |
2012-10-12
§
|
23:51 |
<logmsgbot_> |
asher synchronized wmf-config/CommonSettings.php 'enabling redis debug log' |
[production] |
23:40 |
<LeslieCarr> |
moving traffic back to esams via authdns-scenario |
[production] |
23:40 |
<LeslieCarr> |
reenabled nagios alerts |
[production] |
23:20 |
<logmsgbot_> |
asher synchronized wmf-config/CommonSettings.php 'enabling session multiwrites to redis' |
[production] |
23:16 |
<LeslieCarr> |
rebooting esams core switch stack, expect reachability problems to esams |
[production] |
23:01 |
<logmsgbot_> |
asher synchronized wmf-config/CommonSettings.php 'disabling session multiwrites to redis' |
[production] |
22:36 |
<LeslieCarr> |
switching europe traffic to pmtpa via authdns-scenario |
[production] |
22:36 |
<LeslieCarr> |
muted nagios |
[production] |
20:20 |
<AaronSchulz> |
Confirmed all swift -> netapp sync scripts finished (s4 one kept dying to random http errors) |
[production] |
19:58 |
<logmsgbot_> |
reedy synchronized wmf-config/ |
[production] |
19:01 |
<cmjohnson1> |
powering down search32 to replace cpu1 |
[production] |
18:33 |
<logmsgbot_> |
reedy synchronized wmf-config/InitialiseSettings.php 'AFT (non v5) doesn't use clicktracking now' |
[production] |
18:28 |
<logmsgbot_> |
reedy synchronized php-1.21wmf1/extensions/ArticleFeedback/ |
[production] |
18:19 |
<cmjohnson1> |
troubleshooting db42 w/IBM tech |
[production] |
17:46 |
<cmjohnson1> |
pc3 shutting down to replace hdd's with ssd's rt3703 |
[production] |
17:29 |
<cmjohnson1> |
pc2 shutting down to replaced hdd w/ ssd rt3703 |
[production] |
16:27 |
<logmsgbot_> |
reedy synchronized wmf-config/CommonSettings.php 'Fix extdist mount related warnings' |
[production] |
16:14 |
<cmjohnson1> |
running installer on storage3 |
[production] |
15:54 |
<paravoid> |
moving bits from esams to pmtpa (authdns normal scenario) |
[production] |
15:52 |
<cmjohnson1> |
storage 3 replacing disk 12 and rebuilding raid |
[production] |
15:32 |
<mark> |
Disabled csw1-esams:e8/2 |
[production] |
14:37 |
<mark> |
Rebooting cr1-esams |
[production] |