|
2012-01-01
§
|
| 21:28 |
<Ryan_Lane> |
restarted pdns-recursor on dobson |
[production] |
| 21:26 |
<Ryan_Lane> |
restarted pdns on ns2 about an hour ago |
[production] |
| 09:46 |
<apergos> |
restarted lucene search on srch 10, 11, then later on 3,4,9,1 |
[production] |
| 09:35 |
<apergos> |
removed log.1 from /a/search/logs on search6, it was 35gb |
[production] |
| 03:55 |
<Tim> |
fixed broken package on search7 and search11 |
[production] |
| 02:01 |
<LocalisationUpdate> |
completed (1.18) at Sun Jan 1 02:04:30 UTC 2012 |
[production] |
| 01:36 |
<Tim> |
adjusted FD limit in /etc/init.d/lsearchd on all search servers with sed |
[production] |
| 01:34 |
<Tim> |
increased FD limit on search6 and restarted lsearchd |
[production] |
| 00:46 |
<Tim> |
removed some logs on search6 to fix /a disk space exhaustion |
[production] |
|
2011-12-29
§
|
| 18:28 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php '[[bugzilla:33407|bug 33407]] 33408 Babel stuff for iswiki' |
[production] |
| 15:10 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Bug 33410 - Allow IP edits on wikimania2012.wikimedia.org' |
[production] |
| 08:29 |
<apergos> |
live hack on db31: added nagios ALL=NOPASSWD: /usr/bin/arcconf getconfig 1 to sudoers to shut up security warnings, this is a bad approach though |
[production] |
| 08:27 |
<apergos> |
live hack on hume: added ". /etc/apache2/envvars.default" to /etc/apache2/envars on hume, w/o it apache wouldn't start, which killed static.wp.o |
[production] |
| 07:20 |
<apergos> |
from 1 hour and 20 mins ago: Ryan_Lane: virt4 is back up. br103 had a bad MAC of 00... |
[production] |
| 07:18 |
<apergos> |
test |
[production] |
| 05:28 |
<Ryan_Lane> |
rebooting virt4 again |
[production] |
| 05:28 |
<Ryan_Lane> |
rebooting all migrated instances |
[production] |
| 05:11 |
<Ryan_Lane> |
virt4 has some networking issue |
[production] |
| 05:11 |
<Ryan_Lane> |
migrating all instances off of virt4 onto virt3 and virt2 |
[production] |
| 04:33 |
<p858snake|l> |
[14:32] <Ryan_Lane> !log restarting virt4 because upstart is an unreliable POS (from #mediawiki) |
[production] |
| 04:29 |
<Ryan_Lane> |
restarting virt4 because upstart is an unreliable POS |
[production] |
| 02:53 |
<Ryan_Lane> |
enabled instance live migration on virt2-4 |
[production] |
| 02:01 |
<LocalisationUpdate> |
completed (1.18) at Thu Dec 29 02:05:01 UTC 2011 |
[production] |
|
2011-12-28
§
|
| 23:00 |
<binasher> |
db31 wasn't puppetized, fixed |
[production] |
| 22:49 |
<catrope> |
synchronized php-1.18/extensions/MoodBar/ 'updating to trunk state' |
[production] |
| 22:49 |
<catrope> |
synchronized php-1.18/extensions/MarkAsHelpful/ 'updating to trunk state' |
[production] |
| 22:26 |
<catrope> |
synchronized php-1.18/extensions/MoodBar 'Forgot to sync this earlier' |
[production] |
| 21:25 |
<catrope> |
synchronized wmf-config/InitialiseSettings.php 'Enable MarkAsHelpful on enwiki' |
[production] |
| 20:08 |
<catrope> |
synchronized wmf-config/CommonSettings.php 'Add wmgUseMarkAsHelpful trigger code' |
[production] |
| 19:59 |
<catrope> |
synchronized wmf-config/InitialiseSettings.php 'Add wmgUseMarkAsHelpful, default false' |
[production] |
| 19:56 |
<RoanKattouw> |
Creating DB table for MarkAsHelpful on all wikis |
[production] |
| 19:49 |
<catrope> |
synchronizing Wikimedia installation... : Pushing out MarkAsHelpful code |
[production] |
| 17:31 |
<Jeff_Green> |
srv191 shutdown, looks like failed hdd, see RT #2193 |
[production] |
| 17:09 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php '[[bugzilla:33319|bug 33319]] [[bugzilla:33320|bug 33320]]' |
[production] |
| 16:53 |
<Reedy> |
srv191 complaining of / being readonly |
[production] |
| 16:51 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php '[[bugzilla:33319|bug 33319]] [[bugzilla:33320|bug 33320]]' |
[production] |
| 16:02 |
<Jeff_Green> |
hume:/usr/local/bin/copy_impression_logs_from_storage3.pl deprecated, t'was an emergency hack deployed when storage3 dropped a disk |
[production] |
| 14:51 |
<mutante> |
fixed mobile traffic logger checks - what they report is for real now. 2 procs on first server, 4 procs on the other three |
[production] |
| 07:03 |
<apergos> |
maybe the cron job on these image scalers should run more often... cleaned up /tmp on srv221 |
[production] |
| 02:01 |
<LocalisationUpdate> |
completed (1.18) at Wed Dec 28 02:04:37 UTC 2011 |
[production] |