2010-08-11
§
|
11:19 |
<tstarling> |
synchronized php-1.5/includes/media/Bitmap.php 'reduced magick memory limit from 100M to 50M to stop hanging with vsize limit 300M' |
[production] |
10:46 |
<mark> |
Removed pattern check from nagios check_http |
[production] |
09:42 |
<tstarling> |
synchronized php-1.5/wmf-config/CommonSettings.php |
[production] |
09:38 |
<tstarling> |
synchronized php-1.5/wmf-config/CommonSettings.php |
[production] |
09:35 |
<Tim> |
rebooting srv223, went OOM and mostly died |
[production] |
09:32 |
<tstarling> |
synchronized php-1.5/includes/media/Bitmap.php 'temporary patch to stop scalers going OOM' |
[production] |
09:19 |
<Tim> |
temporarily increased memory limit on the image scalers, since the new convert tends to hang when it runs out of memory instead of crashing nicely |
[production] |
09:17 |
<tstarling> |
synchronized php-1.5/wmf-config/CommonSettings.php 'more memory for image scalers' |
[production] |
08:56 |
<Tim> |
upgrading imagemagick on image scalers to 6.6.2.6-1wm1, package recently committed to svn |
[production] |
02:48 |
<Tim> |
on techblog, disabled WP_DEBUG since it was messing up the admin panels with E_NOTICE messages |
[production] |
02:42 |
<Tim> |
disabled WP-SpamFree on techblog due to bug 19540 |
[production] |
2010-08-10
§
|
23:12 |
<Fred> |
upgraded Tridge to Lucid. Now rebooting. |
[production] |
22:04 |
<RobH> |
knsq10 back online |
[production] |
20:59 |
<RobH> |
knsq10 reinstalling |
[production] |
20:44 |
<RobH> |
knsq9 online |
[production] |
19:37 |
<RobH> |
handed off knsq8 to mark, reinstalling knsq9 |
[production] |
19:02 |
<^demon> |
disabled svn post-commit hook for parser tests, long-since broken |
[production] |
18:57 |
<mark> |
Stopping backend squid on amssq60 for testing |
[production] |
15:24 |
<RobH> |
knsq8 reinstalled, not yet online, will push online shortly |
[production] |
14:56 |
<mark> |
Setup RT on rt.wikimedia.org (streber) |
[production] |
14:32 |
<RobH> |
knsq30 online and in cluster, knsq8 coming down for work |
[production] |
14:18 |
<RobH> |
updated wordpress versions on blog.wikimedia.org and techblog.wikimedia.org |
[production] |
13:35 |
<RobH> |
finishing install on knsq30 |
[production] |
12:50 |
<Tim> |
installed schroot on stafford, for hardy versions of uupdate etc. |
[production] |
11:19 |
<mark> |
Fixed broken hourly cron job mw-serve |
[production] |
11:18 |
<mark> |
Changed su www-data into su mwlib in cleanup cronjob on pdf1 |
[production] |
10:23 |
<mark> |
Removed broken daily system health report on srv178 |
[production] |
10:22 |
<mark> |
Removed broken daily system health report on db4 |
[production] |
07:13 |
<andrew> |
synchronized php-1.5/extensions/CommunityApplications/SpecialCommunityApplications.php 'Merge r70798' |
[production] |
07:13 |
<andrew> |
synchronized php-1.5/extensions/CommunityApplications/CommunityApplications.i18n.php 'Merge r70798' |
[production] |
04:02 |
<RobH> |
knsq30 set to false in pybal, install half done, will finish tomorrow morning. |
[production] |
02:44 |
<RobH> |
knsq29 online and in cluster |
[production] |
02:30 |
<RobH> |
knsq30 reinstalling |
[production] |
00:09 |
<RobH> |
knsq28 back online |
[production] |
00:03 |
<RobH> |
knsq27 back online |
[production] |
00:03 |
<RobH> |
knsq29 reinstalling |
[production] |
2010-08-09
§
|
23:34 |
<RobH> |
knsq28 reinstalling |
[production] |
23:32 |
<RobH> |
knsq26 online |
[production] |
23:32 |
<RobH> |
knsq25 online |
[production] |
23:12 |
<RobH> |
continuing reinstallation, ignore errors for knsq27, reinstalling |
[production] |
22:33 |
<RobH> |
knsq23, knsq24 back online, knsq25, knsq26 still being reinstalled, knsq27-30 still online not yet reinstalled |
[production] |
21:51 |
<mark> |
Added Nagios router interfaces check for br1-knams (using puppet) |
[production] |
21:11 |
<mark> |
Unmounted /dev/sda6 (/a) on srv171, replaced it by /dev/mapper/nonredundant-data (LV with the same data and more space) |
[production] |
21:02 |
<RobH> |
knsq24, knsq25, knsq26, knsq27 coming down for reinstall and puppetfication |
[production] |
20:49 |
<RobH> |
knsq23 reinstall done and pushed back into cluster |
[production] |
18:40 |
<mark> |
Running apt-get upgrade on db9 |
[production] |
18:29 |
<mark> |
Fixed ganglia mess on sq45 |
[production] |
18:24 |
<mark> |
Powercycled sq45 |
[production] |
18:18 |
<mark> |
Added a new MegaCli64 to wikimedia-raid-utils, made check-raid.py use it instead (we have all 64 bit servers anyway), and deployed the new package to the repository. Puppet will upgrade it everywhere. |
[production] |
16:26 |
<Fred> |
fixed DPKG issue on transcode... another one of those conflicting gmond install |
[production] |