|
2011-09-05
§
|
| 13:31 |
<mutante> |
sq31 shutdown -> RT #1431 (degraded RAID, just on one disk and squid would not run anyways) |
[production] |
| 13:18 |
<mark> |
Extended /var LVM LV on lily by 5 GB |
[production] |
| 13:00 |
<mutante> |
sq33 - dist-upgrade/kernel, reboot | sq31 - hardware failure? ->RT #1431 |
[production] |
| 12:22 |
<mutante> |
sq31 - installed smartmontools to check disk health, squid backend does not start due to missing /dev/sdd | srv266 run puppet manually |
[production] |
| 11:42 |
<mutante> |
srv217,srv266 - fix NTP/ntpdate | sq31 - dist-upgrade/kernel, start squid |
[production] |
| 11:34 |
<mutante> |
srv266 - power cycle (too often since July) |
[production] |
| 11:25 |
<mutante> |
sq76 - power up,dist upgrade/kernel,squid clean,reboot |
[production] |
| 11:12 |
<mutante> |
also cleared paniclogs on mchenry (spamd socket fail on Aug31) and lily (temp. failure to write log due to disk space, see Tim's log entries Sep 2nd) |
[production] |
| 11:05 |
<mutante> |
reviewd paniclog on srv207 due to non-zero size. ran out of mem once on Aug 11, cleared logs to stop email alerts |
[production] |
| 10:43 |
<mutante> |
sq55 - power up,dist upgrade/kernel,squid clean,reboot |
[production] |
| 02:27 |
<LocalisationUpdate> |
completed (1.17) at Mon Sep 5 02:29:45 UTC 2011 |
[production] |
|
2011-09-02
§
|
| 22:08 |
<catrope> |
synchronized php/extensions/UploadWizard/resources/mw.UploadWizardDetails.js '[[rev:96154|r96154]]' |
[production] |
| 22:08 |
<catrope> |
synchronized php/extensions/UploadWizard/includes/UploadWizardCampaign.php '[[rev:96154|r96154]]' |
[production] |
| 22:07 |
<catrope> |
synchronized php/extensions/UploadWizard/UploadWizard.config.php '[[rev:96154|r96154]]' |
[production] |
| 22:07 |
<catrope> |
synchronized php/extensions/UploadWizard/UploadWizard.i18n.php '[[rev:96154|r96154]]' |
[production] |
| 19:15 |
<catrope> |
synchronizing Wikimedia installation... : |
[production] |
| 19:14 |
<RoanKattouw> |
Running scap to repair any damage that might have been caused by the above |
[production] |
| 19:13 |
<RoanKattouw> |
Removed empty line in /etc/dsh/group/mediawiki-installation that seemed to be breaking dsh and therefore break sync |
[production] |
| 18:59 |
<catrope> |
synchronized php/resources/startup.js 'touch' |
[production] |
| 18:52 |
<LocalisationUpdate> |
completed (1.17) at Fri Sep 2 18:54:34 UTC 2011 |
[production] |
| 18:17 |
<RoanKattouw> |
Removing the entire LocalisationUpdate cache and regenerating it from scratch |
[production] |
| 18:17 |
<catrope> |
synchronized php/extensions/UploadWizard/UploadWizard.i18n.php '[[rev:96136|r96136]]' |
[production] |
| 17:52 |
<RoanKattouw> |
Put l10nupdate cron job on fenari back at 2:00 UTC |
[production] |
| 17:31 |
<catrope> |
synchronized php/includes/parser/Preprocessor_DOM.php 'Pass XML_PARSE_HUGE flag' |
[production] |
| 16:21 |
<Tim> |
restarting mailman on lily |
[production] |
| 15:45 |
<notpeter> |
starting lighttpd on dataset2 |
[production] |
| 15:11 |
<LocalisationUpdate> |
completed (1.17) at Fri Sep 2 15:13:32 UTC 2011 |
[production] |
| 14:53 |
<mutante> |
srv217 - apache2 kept becoming a defunct zombie, there were apache2 package updates, installed them |
[production] |
| 14:50 |
<RoanKattouw> |
Testing new l10nupdate cron job by hacking the time to 14:53 (one minute from now) |
[production] |
| 14:45 |
<RoanKattouw> |
Moved l10nupdate cron job from hume to fenari |
[production] |
| 14:30 |
<mutante> |
srv217 - there were zombies again "[apache2] <defunct>", kill/restart, hints on debugging? |
[production] |
| 14:12 |
<Jeff_Green> |
depooling payments4 for fundraising testing/staging |
[production] |
| 14:04 |
<catrope> |
synchronized wmf-config/InitialiseSettings.php 'Make $wgLogo protocol-relative on foundationwiki' |
[production] |
| 13:24 |
<mutante> |
fenari - loaded apache php module by creating symlinks from mods-available to mods-enabled (to fix http://noc.wikimedia.org/conf/) |
[production] |
| 12:47 |
<mutante> |
changed puppet to make fenari include class mediawiki::packages which installs wikimedia-task-appserver (RT#1420) |
[production] |
| 11:42 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php |
[production] |
| 11:10 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Bug 30680 - Enable Collection for KM WP' |
[production] |
| 10:56 |
<mutante> |
sq59 - power up,dist-upgrade/kernel,squid clean,reboot |
[production] |
| 10:03 |
<Tim> |
on lily: reduced /var reserved space to 1% to get an extra 2GB free |
[production] |
| 09:56 |
<Tim> |
on lily: did "apt-get clean" to get an extra 1 GB of space and restarted exim4 |
[production] |
| 09:51 |
<Tim> |
mail on lily is broken due to full /var partition |
[production] |
| 02:20 |
<LocalisationUpdate> |
completed (1.17) at Fri Sep 2 02:22:27 UTC 2011 |
[production] |
| 00:33 |
<aaron> |
synchronizing Wikimedia installation... : |
[production] |
| 00:26 |
<aaron> |
synchronizing Wikimedia installation... : |
[production] |
| 00:21 |
<AaronSchulz> |
updated set-group-write2 for het deploy |
[production] |