651-700 of 1295 results (3ms)
2009-01-15 §
18:27 <brion> nfs upload5 is not happy :( [production]
18:27 <brion> some sort of issues w/ media fileserver, we think, perhaps pressure due to some upload squid cache clearing? [production]
18:23 <RobH> sq14-aq16 offline, rebooting and cleaning cache [production]
18:16 <RobH> sq2, sq4, and sq10 were unresponsive and down. Restarted, cleaned cache, and brought back online. [production]
04:32 <Tim> increased squid max post size from 75MB to 110MB so that people can actually upload 100MB files as advertised in the media [production]
2009-01-14 §
19:21 <mark> Lower preffed paths from 13030 that were learned at NYIIX [production]
18:44 <brion> updated wikitech to current SVN and rebuilt text search index for new server to fix short words [production]
18:30 <RobH> removed the sysop and bcrat add/remove from bcrat permissions for eswiki [production]
18:22 <RobH> added groups for eswiki again per https://bugzilla.wikimedia.org/show_bug.cgi?id=16975 [production]
16:28 <RobH> added rollbacker group per https://bugzilla.wikimedia.org/show_bug.cgi?id=16975 [production]
2009-01-13 §
23:32 <Tim> fixed NRPE on db29 [production]
22:56 <Tim> cleaned up binlogs on db1 and ixia [production]
22:54 <brion> poking WP alias on frwiki [[bugzilla:16887]] [production]
21:11 <RobH> setup ganglia on erzurumi [production]
20:42 <brion> setting all pdf generators to use the new server [production]
20:40 <brion> testing pdf gen on erzurumi on testwiki [production]
20:35 <RobH> setup erzurumi for dev testing [production]
20:35 <RobH> some random updates on [[server roles]] to clean it up [production]
19:37 <mark> Restored normal situation, with 14907 -> 43821 traffic downpreffed to HGTN to avoid peering network congestion [production]
18:40 <mark> Retracted outbound announcement to all AMS-IX peers, 16265 and 13030 to force inbound via 1299 [production]
18:25 <mark> Undid any routing changes as they were not having the desired effect [production]
18:14 <mark> Prepended 43821 twice on outgoing announcements to 16265 to make pmtpa-esams path via nycx less attractive [production]
11:38 <Tim> reducing innodb_buffer_pool_size on db19, db21, db22, db29 [production]
09:15 <Tim> restarting mysqld on db23 again [production]
09:09 <Tim> restarting mysqld on db18 again [production]
07:08 <Tim> removed db23 from rotation, since I'm bringing it up soon and it will be lagged [production]
07:02 <Tim> shutting down mysqld on db18 for further mem usage tweak [production]
06:53 <Tim> fixed broken /etc/fstab on db23 via serial console [production]
06:42 <Tim> restarting db23 [production]
00:08 <Tim> repooling db18, has caught up [production]
2009-01-12 §
21:50 <brion> testing a scap after touching MessagesWuu.php to see if that clears borked serialized btis [production]
21:22 <RobH> erzurumi installed [production]
17:55 <brion> temporarily stopped apache on srv78, srv118 [production]
17:54 <brion> srv78 doesn't have upload5 mounted [production]
17:54 <brion> srv118 doesn't have upload5 mounted [production]
17:46 <RobH> fixed some settings for flaggedrevs in https://bugzilla.wikimedia.org/show_bug.cgi?id=14648 [production]
17:31 <RobH> per brion commented out db18 in db.php cuz its making other crap lag too much [production]
17:26 <RobH> updated flaggedrevs.php for https://bugzilla.wikimedia.org/show_bug.cgi?id=16365 [production]
17:23 <RobH> updated apache config on yongle for wap => mobile forwarding oversight per https://bugzilla.wikimedia.org/show_bug.cgi?id=16692 [production]
17:05 <brion> db18 is backlogged 191k seconds. depooling it if it's still in; complaints of hella lag [production]
15:32 <Tim> restarted mysqld on db18 with reduced memory usage, repooled [production]
14:12 <Tim> rebooting db18 [production]
13:20 <Tim> depooled db18 (is down) [production]
2009-01-10 §
16:08 <domas> rotated 300g sampled-1000.log ;-) [production]
07:09 <river> applied current OS patches to ms2 and rebooted [production]
01:21 <Tim> restarted apache on srv95,srv114,srv37,srv49 [production]
01:19 <Tim> cleaned up disk space on db1. Still looks suspiciously like the master... [production]
00:33 <brion> redirecting old bylaws.pdf to wiki page bylaws on wikimediafoundation.org (foundation.conf update) [production]
00:13 <brion> reconfigured exim on wikitech to hopefully actually send mail out. whether it reaches anything, we'll see [production]
00:12 <tomaszf> turned off fundraising banners [production]