3551-3600 of 10000 results (20ms)
2012-01-01 §
09:35 <apergos> removed log.1 from /a/search/logs on search6, it was 35gb [production]
03:55 <Tim> fixed broken package on search7 and search11 [production]
02:01 <LocalisationUpdate> completed (1.18) at Sun Jan 1 02:04:30 UTC 2012 [production]
01:36 <Tim> adjusted FD limit in /etc/init.d/lsearchd on all search servers with sed [production]
01:34 <Tim> increased FD limit on search6 and restarted lsearchd [production]
00:46 <Tim> removed some logs on search6 to fix /a disk space exhaustion [production]
2011-12-31 §
02:01 <LocalisationUpdate> completed (1.18) at Sat Dec 31 02:05:00 UTC 2011 [production]
2011-12-30 §
19:36 <LeslieCarr> added 31133 to as path list "AVOID-PATHS" and soft reset AMS-IX route server peerings [production]
14:47 <Jeff_Green> unbroke replication on fundraisingdb's by skipping a bad query on the slaves [production]
02:01 <LocalisationUpdate> completed (1.18) at Fri Dec 30 02:05:02 UTC 2011 [production]
01:00 <LeslieCarr> restarted puppetd on ms1002 [production]
2011-12-29 §
18:28 <reedy> synchronized wmf-config/InitialiseSettings.php '[[bugzilla:33407|bug 33407]] 33408 Babel stuff for iswiki' [production]
15:10 <reedy> synchronized wmf-config/InitialiseSettings.php 'Bug 33410 - Allow IP edits on wikimania2012.wikimedia.org' [production]
08:29 <apergos> live hack on db31: added nagios ALL=NOPASSWD: /usr/bin/arcconf getconfig 1 to sudoers to shut up security warnings, this is a bad approach though [production]
08:27 <apergos> live hack on hume: added ". /etc/apache2/envvars.default" to /etc/apache2/envars on hume, w/o it apache wouldn't start, which killed static.wp.o [production]
07:20 <apergos> from 1 hour and 20 mins ago: Ryan_Lane: virt4 is back up. br103 had a bad MAC of 00... [production]
07:18 <apergos> test [production]
05:28 <Ryan_Lane> rebooting virt4 again [production]
05:28 <Ryan_Lane> rebooting all migrated instances [production]
05:11 <Ryan_Lane> virt4 has some networking issue [production]
05:11 <Ryan_Lane> migrating all instances off of virt4 onto virt3 and virt2 [production]
04:33 <p858snake|l> [14:32] <Ryan_Lane> !log restarting virt4 because upstart is an unreliable POS (from #mediawiki) [production]
04:29 <Ryan_Lane> restarting virt4 because upstart is an unreliable POS [production]
02:53 <Ryan_Lane> enabled instance live migration on virt2-4 [production]
02:01 <LocalisationUpdate> completed (1.18) at Thu Dec 29 02:05:01 UTC 2011 [production]
2011-12-28 §
23:00 <binasher> db31 wasn't puppetized, fixed [production]
22:49 <catrope> synchronized php-1.18/extensions/MoodBar/ 'updating to trunk state' [production]
22:49 <catrope> synchronized php-1.18/extensions/MarkAsHelpful/ 'updating to trunk state' [production]
22:26 <catrope> synchronized php-1.18/extensions/MoodBar 'Forgot to sync this earlier' [production]
21:25 <catrope> synchronized wmf-config/InitialiseSettings.php 'Enable MarkAsHelpful on enwiki' [production]
20:08 <catrope> synchronized wmf-config/CommonSettings.php 'Add wmgUseMarkAsHelpful trigger code' [production]
19:59 <catrope> synchronized wmf-config/InitialiseSettings.php 'Add wmgUseMarkAsHelpful, default false' [production]
19:56 <RoanKattouw> Creating DB table for MarkAsHelpful on all wikis [production]
19:49 <catrope> synchronizing Wikimedia installation... : Pushing out MarkAsHelpful code [production]
17:31 <Jeff_Green> srv191 shutdown, looks like failed hdd, see RT #2193 [production]
17:09 <reedy> synchronized wmf-config/InitialiseSettings.php '[[bugzilla:33319|bug 33319]] [[bugzilla:33320|bug 33320]]' [production]
16:53 <Reedy> srv191 complaining of / being readonly [production]
16:51 <reedy> synchronized wmf-config/InitialiseSettings.php '[[bugzilla:33319|bug 33319]] [[bugzilla:33320|bug 33320]]' [production]
16:02 <Jeff_Green> hume:/usr/local/bin/copy_impression_logs_from_storage3.pl deprecated, t'was an emergency hack deployed when storage3 dropped a disk [production]
14:51 <mutante> fixed mobile traffic logger checks - what they report is for real now. 2 procs on first server, 4 procs on the other three [production]
07:03 <apergos> maybe the cron job on these image scalers should run more often... cleaned up /tmp on srv221 [production]
02:01 <LocalisationUpdate> completed (1.18) at Wed Dec 28 02:04:37 UTC 2011 [production]
2011-12-27 §
21:21 <catrope> synchronized php-1.18/extensions/ArticleFeedbackv5/modules/ext.articleFeedbackv5/ext.articleFeedbackv5.startup.js '[[rev:107427|r107427]]' [production]
19:39 <LeslieCarr> removed puppetdlock on brewster [production]
19:04 <LeslieCarr> removing manual localhost accept all iptables rule on locke (replacing with puppetized rule) [production]
18:39 <Ryan_Lane> clearing some logs on brewster and restarting lighttpd and squid [production]
18:21 <Jeff_Green> powercycled kaulen yet again [production]
17:47 <LeslieCarr> removed file�s out of srv223's /tmp directory to free up its space [production]
17:15 <mark> Power cycled amslvs3 [production]
17:06 <apergos> powercycled kaulen, it was (presumably) swapping away. report.cgi was to blame [production]