4451-4500 of 5010 results (9ms)
2009-01-07 §
21:15 <RobH> shutdown db13 and db14 to fix lom lockup issue. [production]
20:52 <RobH> depooled db13 and db14 in db.php to reboot them and fix the SP lockup issue. [production]
20:49 <RobH> updating firmware on db16. [production]
20:43 <RobH> started mysql back up on db15 [production]
20:42 <RobH> cold reset of db16 to resolve lom issue. will update firmware upon boot. [production]
20:39 <RobH> swappned hostnames on ms3 and ms4, updated racktables and dns to reflect change [production]
20:24 <brion> disabled wikidiff2 since it's not installed, and this apparaently is nicely broken [production]
20:21 <RobH> db15 now responsive to lom and ready to be re-integrated into the cluster [production]
20:12 <RobH> db15 cold reset fixes the LOM non-responsive issue. Upgrading its firmware to prevent future issues. [production]
20:06 <brion> removed stray whitespace from wikitech config file which was breaking rss feeds [production]
19:23 <mark> Possibility that esams LVS was overloaded, split over 2 boxes (fuchsia & mint) [production]
19:19 <RobH> ms3 and ms4 are accessible via LOM and ready for setup/deployment [production]
19:05 <RobH> updated dns for ms3-ms5, updated dns for mangement for all media servers. [production]
19:05 <RobH> nope ;_; [production]
19:04 <RobH> did domas update the bot for the new wikitech? [production]
19:03 <brion> touching MessagesZh.php and re-trying scap; may not have properly updated [production]
17:40 <brion-plague> scapping -- merged r45507 zh specialpage alias fix to live. also r45499 (revert of Cite error thingy) seems to already have been merged [production]
13:58 <Tim> ran updateAutoPromote.php on all flaggedRevs wikis [production]
13:41 <Tim> scap [production]
13:21 <Tim> repooled db3 and db4 [production]
04:36 <brion-codereview> svn up'ing testwiki to r45489 [production]
2009-01-06 §
16:01 <mark> Changed 'knams' into 'esams' in DNS, kept a lot of old names in place [production]
15:26 <Tim> cleaned up binlogs on db1 [production]
13:09 <mark> Did some Traffic Engineering on the Amsterdam network [production]
11:58 <Tim> installed NRPE on new ES servers [production]
11:47 <domas> added db29 to s3 duty [production]
11:32 <Tim> locked clusters 18 and 19, updated nagios [production]
11:27 <Tim> fixed lack of schema on srv161 [production]
11:21 <Tim> retired cluster18 from the write list, added cluster20 and cluster21 [production]
11:18 <domas> test!!! [production]
11:18 <domas> test [production]
2009-01-03 §
23:57 <mark> Restored AAAA record on upload.wikimedia.org [production]
12:04 <domas> db17, db18 had OS/firmware updates, rebooted [production]
10:50 <domas> db19 RAID complaining about temperature, check-raid/kswapd/mysqld deadlock. upgrading RAID firmware, rebooting, etc [production]
01:23 <Tim> removed db3 and db4 from rotation again, to allow recompressTracked to go faster [production]
00:36 <Tim> depooled db19, is down [production]
00:32 <Tim> restarting recompressTracked with an extra wfWaitForSlaves() [production]
00:08 <Tim> repooled db3 and db4 [production]
2009-01-02 §
22:35 <Tim> depooled db3 and db4 temporarily [production]
20:54 <mark> Depooled db4 [production]
00:42 <Tim> restarting recompressTracked.php on hume [production]
2009-01-01 §
20:34 <brion> live-merging file delete fatal error fix from r45278 [production]
19:47 <brion> bumped meter image to 7 [production]
01:59 <brion> scapping! [production]
01:40 <brion> svn up'ing test.wiki to r45274 [production]
00:55 <brion> svn up'ing on test.wikipedia [production]
2008-12-31 §
18:40 <brion> fixed old whygive.wikimedia.org blog by copying de-conflicted WordPress source files out of the active blog where we fixed it after the 2.7 upgrade [production]
2008-12-30 §
23:02 <RobH> is leaving on a jet plane, weeeeeeeee.. in 8 hours. [production]
23:01 <RobH> all knams squids are now online. [production]
22:49 <RobH> knsq23-26 back in rotation, 3 more to go. [production]