401-450 of 6259 results (6ms)
2010-02-23 §
15:37 <Rob> FINALLY got in the last RMA drive for storage1, arrays were toast (losing 1/2 the drives in a raid10 and having the unlucky chance of them being paired), reinstall in progress [production]
12:50 <RoanKattouw> Cleaned up my dblist voodoo, everything should be back to normal now [production]
12:41 <RoanKattouw> Fixed sync-dblist, was broken [production]
12:31 <RoanKattouw> Syncing pmtpa.dblist anyway, seems I have to [production]
12:24 <RoanKattouw> Temporarily adding wikis listed in bug 20325 to pmtpa.dblist so I can export their contents. Please don't sync this [production]
2010-02-22 §
23:40 <Tim> restarted replication on ms2 [production]
23:39 <Tim> ms1 is up with a full copy of ms3 data, now replicating cluster22/rc1 [production]
21:09 <mark> synchronized php-1.5/wmf-config/InitialiseSettings.php 'Revert' [production]
20:30 <root> synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22522 - Allow bureaucrats to remove sysop flag on Simple English Wikitionary' [production]
20:09 <Rob> finished the setup of srv217 (just needed puppet fixed and files updated) [production]
20:02 <root> ran sync-common-all [production]
20:01 <mark> synchronized php-1.5/wmf-config/InitialiseSettings.php 'Set vector as default skin on nlwikimedia' [production]
19:44 <root> ran sync-common-all [production]
18:10 <Fred> ganglia is going to hiccup a little for the next few minutes while I restore data... "Don't panic" [production]
07:53 <Tim> rebooted srv206 via drac [production]
07:48 <tstarling> synchronized php-1.5/wmf-config/mc.php 'removing srv206, is down' [production]
06:19 <Tim> increased multicast TTL from 1 to 3 in gmond_template.erb, to fix ganglia on ms* [production]
05:54 <Tim> started ms2 -> ms1 copy using nc+tarpipe [production]
05:30 <Tim> shutting down mysql on ms2 for copy to ms1 [production]
05:09 <tstarling> synchronized php-1.5/wmf-config/db.php 'depooling ms2 for copy to ms1' [production]
2010-02-21 §
02:43 <Fred> puppettized user gmetric [production]
2010-02-20 §
22:37 <mark> Installed Ubuntu 8.04 on ms1 - now ready for ES deployment [production]
2010-02-19 §
20:48 <Andrew> Ignore Brion's last, the error was on his personal site :D [production]
20:45 <brion> "out of captcha images" - user registration broken. Check captcha dir? [production]
18:48 <Fred> removed memcached node srv193 and replaced with srv233 [production]
16:52 <Rob> updated InitiliseSettings with bug 22585 [production]
01:31 <Fred> puppetized gmond.conf generation using templates [production]
2010-02-18 §
23:51 <RoanKattouw> Fixed search script on yongle, removed UA check exemption for yongle [production]
23:51 <RoanKattouw> logmsgbot is broken again [production]
04:07 <Tim> re-running storageTypeStats.php on enwiki to identify rows with old_flags='object,utf-8', which need to be handled properly in fixBug20757.php [production]
2010-02-17 §
17:25 <Rob> pushed updates to wordpress installations [production]
17:04 <RoanKattouw> Synced wmf-config/checkers.php 'Allow missing UA from yongle' [production]
17:04 <RoanKattouw> Restarted logmsgbot [production]
16:47 <Fred> test [production]
07:24 <Fred> there is a problem with ganglia. It is being worked on. [production]
00:15 <RoanKattouw> Resynced srv151, was returning empty responses [production]
2010-02-16 §
23:31 <Fred> upgrading gmond to 3.1.2 everywhere. However, due to the newish module structure, there is a potential that ganglia will hickup while puppet does its job... [production]
22:03 <logmsgbot_> mark synchronized php-1.5/wmf-config/checkers.php 'Exceptions' [production]
20:30 <Rob> srv127 is online, but not in LVS. Its cert was accepted on sockpuppet, but puppetd --test results in a cert failure on retrieval from sockpuppet. [production]
20:17 <Rob> rebooting srv127 [production]
20:15 <mark> Increased membufs to 40 per COSS dir on the pmtpa upload backend squids [production]
19:51 <mark> Increased membufs per COSS dir from 10 to 20 on the new pmtpa squids [production]
18:10 <apergos> but documentation can save people precious time when things are on fire [production]
18:02 <mark> Documentation is not a substitution for thinking [production]
16:44 <mark> Fixed puppet on most servers [production]
16:23 <domas> anyone knows why mysqldump on snapshot3 is locking tables? maybe --single-transaction could work better?!!!? [production]
16:22 <RoanKattouw> Strike my last, I hear it'll fix itself in a day [production]
16:21 <RoanKattouw> Oops, meant to type srv187-189 [production]
16:21 <RoanKattouw> Ganglia not picking up data for srv1987-189, 193, 214-218, 250-253, 257 even though the boxes are up and gmond is running; been like this for 3 days [production]
15:21 <mark> Removing all puppet certs/private keys on all machines [production]