|
2011-12-29
§
|
| 08:29 |
<apergos> |
live hack on db31: added nagios ALL=NOPASSWD: /usr/bin/arcconf getconfig 1 to sudoers to shut up security warnings, this is a bad approach though |
[production] |
| 08:27 |
<apergos> |
live hack on hume: added ". /etc/apache2/envvars.default" to /etc/apache2/envars on hume, w/o it apache wouldn't start, which killed static.wp.o |
[production] |
| 07:20 |
<apergos> |
from 1 hour and 20 mins ago: Ryan_Lane: virt4 is back up. br103 had a bad MAC of 00... |
[production] |
| 07:18 |
<apergos> |
test |
[production] |
| 05:28 |
<Ryan_Lane> |
rebooting virt4 again |
[production] |
| 05:28 |
<Ryan_Lane> |
rebooting all migrated instances |
[production] |
| 05:11 |
<Ryan_Lane> |
virt4 has some networking issue |
[production] |
| 05:11 |
<Ryan_Lane> |
migrating all instances off of virt4 onto virt3 and virt2 |
[production] |
| 04:33 |
<p858snake|l> |
[14:32] <Ryan_Lane> !log restarting virt4 because upstart is an unreliable POS (from #mediawiki) |
[production] |
| 04:29 |
<Ryan_Lane> |
restarting virt4 because upstart is an unreliable POS |
[production] |
| 02:53 |
<Ryan_Lane> |
enabled instance live migration on virt2-4 |
[production] |
| 02:01 |
<LocalisationUpdate> |
completed (1.18) at Thu Dec 29 02:05:01 UTC 2011 |
[production] |
|
2011-12-28
§
|
| 23:00 |
<binasher> |
db31 wasn't puppetized, fixed |
[production] |
| 22:49 |
<catrope> |
synchronized php-1.18/extensions/MoodBar/ 'updating to trunk state' |
[production] |
| 22:49 |
<catrope> |
synchronized php-1.18/extensions/MarkAsHelpful/ 'updating to trunk state' |
[production] |
| 22:26 |
<catrope> |
synchronized php-1.18/extensions/MoodBar 'Forgot to sync this earlier' |
[production] |
| 21:25 |
<catrope> |
synchronized wmf-config/InitialiseSettings.php 'Enable MarkAsHelpful on enwiki' |
[production] |
| 20:08 |
<catrope> |
synchronized wmf-config/CommonSettings.php 'Add wmgUseMarkAsHelpful trigger code' |
[production] |
| 19:59 |
<catrope> |
synchronized wmf-config/InitialiseSettings.php 'Add wmgUseMarkAsHelpful, default false' |
[production] |
| 19:56 |
<RoanKattouw> |
Creating DB table for MarkAsHelpful on all wikis |
[production] |
| 19:49 |
<catrope> |
synchronizing Wikimedia installation... : Pushing out MarkAsHelpful code |
[production] |
| 17:31 |
<Jeff_Green> |
srv191 shutdown, looks like failed hdd, see RT #2193 |
[production] |
| 17:09 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php '[[bugzilla:33319|bug 33319]] [[bugzilla:33320|bug 33320]]' |
[production] |
| 16:53 |
<Reedy> |
srv191 complaining of / being readonly |
[production] |
| 16:51 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php '[[bugzilla:33319|bug 33319]] [[bugzilla:33320|bug 33320]]' |
[production] |
| 16:02 |
<Jeff_Green> |
hume:/usr/local/bin/copy_impression_logs_from_storage3.pl deprecated, t'was an emergency hack deployed when storage3 dropped a disk |
[production] |
| 14:51 |
<mutante> |
fixed mobile traffic logger checks - what they report is for real now. 2 procs on first server, 4 procs on the other three |
[production] |
| 07:03 |
<apergos> |
maybe the cron job on these image scalers should run more often... cleaned up /tmp on srv221 |
[production] |
| 02:01 |
<LocalisationUpdate> |
completed (1.18) at Wed Dec 28 02:04:37 UTC 2011 |
[production] |
|
2011-12-27
§
|
| 21:21 |
<catrope> |
synchronized php-1.18/extensions/ArticleFeedbackv5/modules/ext.articleFeedbackv5/ext.articleFeedbackv5.startup.js '[[rev:107427|r107427]]' |
[production] |
| 19:39 |
<LeslieCarr> |
removed puppetdlock on brewster |
[production] |
| 19:04 |
<LeslieCarr> |
removing manual localhost accept all iptables rule on locke (replacing with puppetized rule) |
[production] |
| 18:39 |
<Ryan_Lane> |
clearing some logs on brewster and restarting lighttpd and squid |
[production] |
| 18:21 |
<Jeff_Green> |
powercycled kaulen yet again |
[production] |
| 17:47 |
<LeslieCarr> |
removed file�s out of srv223's /tmp directory to free up its space |
[production] |
| 17:15 |
<mark> |
Power cycled amslvs3 |
[production] |
| 17:06 |
<apergos> |
powercycled kaulen, it was (presumably) swapping away. report.cgi was to blame |
[production] |
| 13:45 |
<nikerabbit> |
synchronizing Wikimedia installation... : I18ndeploy [[rev:107378|r107378]] |
[production] |
| 13:07 |
<nikerabbit> |
synchronized wmf-config/ 'cs gender aliases [[bugzilla:33367|bug 33367]]' |
[production] |
| 02:01 |
<LocalisationUpdate> |
completed (1.18) at Tue Dec 27 02:04:40 UTC 2011 |
[production] |