701-750 of 6618 results (7ms)
2010-03-04 §
19:44 <Tim> installed NRPE and RAID utils on ms1 [production]
19:40 <Tim> removed db19 from the mysql node group and nagios since it is being used for varnish now [production]
19:29 <Tim> fixed prototype.wikimedia.org (nagios critical), was waiting 10s on every request due to memcached being down [production]
2010-03-03 §
16:16 <apergos> reenables gmetric on ms8 a little bit ago, scrub finished with no errors (yeah right( [production]
06:06 <apergos> replication off again tonight, zfs nicely borked on ms8, zfs destroy snapshot X gives message "cannot destroy X, dataset already exists" . time for sun contract and call them [production]
05:53 <apergos> gmetrics turned off again on ms8, running zpool scrub export [production]
05:40 <apergos> reset SYs on ms8 worked (slowly). gmetrics reenabled. (and I thought morebots was just being a bad bot) [production]
05:39 <apergos> ( :-P ) starting catchup for the last day's lost replication ms7 -> ms8 [production]
05:22 <apergos> reboot failed, bootadm hung probably waiting on zfs. stop SYS claims to work but subsequent show SYS shows power state on. [production]
05:14 <apergos> rebootiing ms8 to clear zfs deadlock [production]
2010-03-02 §
23:32 <catrope> synchronized php-1.5/extensions/UsabilityInitiative/OptIn/SpecialOptIn.php 'r63191' [production]
23:31 <catrope> synchronized php-1.5/extensions/UsabilityInitiative/OptIn/OptIn.php 'r63191' [production]
23:31 <catrope> synchronized php-1.5/wmf-config/CommonSettings.php 'Tweak OptIn settings' [production]
22:36 <RoanKattouw> Rebuilding localisation cache [production]
20:15 <Tim> on hume: started recompressTracked.php for all wikis, to fix any newly-tracked pointers from fixBug20757.php and to complete enwiki [production]
03:48 <atglenn> leaving replication and gmetrics turned off on ms8 for thenight, will go back to looking at scat and mdb output tomorrow. [production]
02:27 <atglenn> turning off gmetric on ms8, the dfs are piling up behind a deadlocked zfs list [production]
02:13 <atglenn> turn of ms7 replication to ms8 temporarily to work out a zfs deadlock issue on ms8 [production]
2010-03-01 §
12:53 <kate> synchronized php-1.5/wmf-config/db.php [production]
12:51 <river> removing db17 from rotation for ts dump [production]
12:51 <kate> synchronized php-1.5/wmf-config/db.php [production]
11:48 <domas> recovered locke, ran log collection with per-process limit of 200MB [production]
11:39 <domas> locke was killed by awks using gigabytes of memory, mwahaha. [production]
2010-02-27 §
02:09 <Tim> rewrote domas's api.php fix and deployed my version [production]
01:56 <midom> synchronized php-1.5/includes/LocalisationCache.php 'removing profiling hooks' [production]
01:42 <domas> fixed api.php on test/srv124 [production]
2010-02-26 §
23:40 <Tim> disabled the temporary squid purge script, I'm assuming it's done its job now [production]
19:43 <catrope> synchronized php-1.5/includes/api/ApiBase.php 'r63014, r63015' [production]
18:24 <Fred> bugzilla now residing on Kaulen... Updating server roles. [production]
18:07 <Fred> switching bugzilla cname to new server. (both servers are pointing to the same DB, so switch should be transparent to most) [production]
17:28 <midom> synchronized php-1.5/includes/LocalisationCache.php 'profiling' [production]
17:26 <midom> synchronized php-1.5/includes/diff/DifferenceInterface.php 'removing profiling hook' [production]
17:23 <midom> synchronized php-1.5/includes/StubObject.php 'tracehooks' [production]
17:21 <midom> synchronized php-1.5/includes/StubObject.php 'tracehooks' [production]
14:38 <RoanKattouw> logmsgbot is missing in action AGAIN, starting it as me on fenari [production]
14:37 <RoanKattouw> Synced r63008 [production]
12:17 <domas> *sigh*, recovered locke after OOM, added few entries to limits.conf [production]
2010-02-25 §
23:48 <Tim> running the new fixBug20757.php on all wikis [production]
21:48 <RoanKattouw> Removed empty line from the end of all.dblist, broke account creation ^^ [production]
20:18 <RoanKattouw> Removed those closed wikis again, db lists back to normal [production]
20:08 <RoanKattouw> Temp adding some closed wikis to all.dblist to be able to dump them (again) [production]
19:01 <Rob> removed old servers will, bart, etc from DNS [production]
18:32 <Rob> pushing more dns changes for LOM on new servers. [production]
00:59 <Tim> made the squid purge script on locke also purge redirects from /w/index.php?, for bug 22639 [production]
00:32 <Tim> installed ack-grep on fenari [production]
2010-02-24 §
20:48 <Rob> IP conflicts on alsted/ersch/formey/kaulen. I must have typo'd something in one of their DRAC configurations because they are all messed up. will go into DC and physically review settings via console. [production]
16:10 <Rob> had formey and alsted ip info mixed up in dns, updating dns [production]
13:13 <RoanKattouw> Adding optin_survey table to wikis that didn't have it [production]
01:38 <atglenn> ganglia 3.1 packages up on ms7 and ms8 (see atg for solaris packages for other boxen) [production]
01:20 <domas> that was fun, let's do it again! [production]