|
2010-08-11
§
|
| 09:35 |
<Tim> |
rebooting srv223, went OOM and mostly died |
[production] |
| 09:32 |
<tstarling> |
synchronized php-1.5/includes/media/Bitmap.php 'temporary patch to stop scalers going OOM' |
[production] |
| 09:19 |
<Tim> |
temporarily increased memory limit on the image scalers, since the new convert tends to hang when it runs out of memory instead of crashing nicely |
[production] |
| 09:17 |
<tstarling> |
synchronized php-1.5/wmf-config/CommonSettings.php 'more memory for image scalers' |
[production] |
| 08:56 |
<Tim> |
upgrading imagemagick on image scalers to 6.6.2.6-1wm1, package recently committed to svn |
[production] |
| 02:48 |
<Tim> |
on techblog, disabled WP_DEBUG since it was messing up the admin panels with E_NOTICE messages |
[production] |
| 02:42 |
<Tim> |
disabled WP-SpamFree on techblog due to bug 19540 |
[production] |
|
2010-08-10
§
|
| 23:12 |
<Fred> |
upgraded Tridge to Lucid. Now rebooting. |
[production] |
| 22:04 |
<RobH> |
knsq10 back online |
[production] |
| 20:59 |
<RobH> |
knsq10 reinstalling |
[production] |
| 20:44 |
<RobH> |
knsq9 online |
[production] |
| 19:37 |
<RobH> |
handed off knsq8 to mark, reinstalling knsq9 |
[production] |
| 19:02 |
<^demon> |
disabled svn post-commit hook for parser tests, long-since broken |
[production] |
| 18:57 |
<mark> |
Stopping backend squid on amssq60 for testing |
[production] |
| 15:24 |
<RobH> |
knsq8 reinstalled, not yet online, will push online shortly |
[production] |
| 14:56 |
<mark> |
Setup RT on rt.wikimedia.org (streber) |
[production] |
| 14:32 |
<RobH> |
knsq30 online and in cluster, knsq8 coming down for work |
[production] |
| 14:18 |
<RobH> |
updated wordpress versions on blog.wikimedia.org and techblog.wikimedia.org |
[production] |
| 13:35 |
<RobH> |
finishing install on knsq30 |
[production] |
| 12:50 |
<Tim> |
installed schroot on stafford, for hardy versions of uupdate etc. |
[production] |
| 11:19 |
<mark> |
Fixed broken hourly cron job mw-serve |
[production] |
| 11:18 |
<mark> |
Changed su www-data into su mwlib in cleanup cronjob on pdf1 |
[production] |
| 10:23 |
<mark> |
Removed broken daily system health report on srv178 |
[production] |
| 10:22 |
<mark> |
Removed broken daily system health report on db4 |
[production] |
| 07:13 |
<andrew> |
synchronized php-1.5/extensions/CommunityApplications/SpecialCommunityApplications.php 'Merge r70798' |
[production] |
| 07:13 |
<andrew> |
synchronized php-1.5/extensions/CommunityApplications/CommunityApplications.i18n.php 'Merge r70798' |
[production] |
| 04:02 |
<RobH> |
knsq30 set to false in pybal, install half done, will finish tomorrow morning. |
[production] |
| 02:44 |
<RobH> |
knsq29 online and in cluster |
[production] |
| 02:30 |
<RobH> |
knsq30 reinstalling |
[production] |
| 00:09 |
<RobH> |
knsq28 back online |
[production] |
| 00:03 |
<RobH> |
knsq27 back online |
[production] |
| 00:03 |
<RobH> |
knsq29 reinstalling |
[production] |
|
2010-08-09
§
|
| 23:34 |
<RobH> |
knsq28 reinstalling |
[production] |
| 23:32 |
<RobH> |
knsq26 online |
[production] |
| 23:32 |
<RobH> |
knsq25 online |
[production] |
| 23:12 |
<RobH> |
continuing reinstallation, ignore errors for knsq27, reinstalling |
[production] |
| 22:33 |
<RobH> |
knsq23, knsq24 back online, knsq25, knsq26 still being reinstalled, knsq27-30 still online not yet reinstalled |
[production] |
| 21:51 |
<mark> |
Added Nagios router interfaces check for br1-knams (using puppet) |
[production] |
| 21:11 |
<mark> |
Unmounted /dev/sda6 (/a) on srv171, replaced it by /dev/mapper/nonredundant-data (LV with the same data and more space) |
[production] |
| 21:02 |
<RobH> |
knsq24, knsq25, knsq26, knsq27 coming down for reinstall and puppetfication |
[production] |
| 20:49 |
<RobH> |
knsq23 reinstall done and pushed back into cluster |
[production] |
| 18:40 |
<mark> |
Running apt-get upgrade on db9 |
[production] |
| 18:29 |
<mark> |
Fixed ganglia mess on sq45 |
[production] |
| 18:24 |
<mark> |
Powercycled sq45 |
[production] |
| 18:18 |
<mark> |
Added a new MegaCli64 to wikimedia-raid-utils, made check-raid.py use it instead (we have all 64 bit servers anyway), and deployed the new package to the repository. Puppet will upgrade it everywhere. |
[production] |
| 16:26 |
<Fred> |
fixed DPKG issue on transcode... another one of those conflicting gmond install |
[production] |
| 16:14 |
<catrope> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 24735: Sanitize private/fishbowl config' |
[production] |
| 16:03 |
<catrope> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 24732: Portal and Book namespaces for yowiki' |
[production] |
| 13:59 |
<mark> |
srv110 decommissioned itself |
[production] |
| 13:55 |
<RobH> |
knsq23 coming down for reinstallation |
[production] |