751-800 of 8469 results (7ms)
2010-07-26 §
17:17 <mark> Installed Lucid on sq69 and sq70 [production]
14:41 <mark> Moved db19 back to vlan 2 [production]
14:34 <mark> Pooled sq68 in the bits.pmtpa varnish LVS pool [production]
14:06 <mark> Installed lucid on sq68 and deployed varnish for bits.pmtpa [production]
12:22 <mark> Started puppet on spence [production]
05:21 <Tim> removed a log on search12 to give it a tiny bit more space [production]
2010-07-25 §
21:55 <mark> Pooled sq67 (Varnish) in the bits.pmtpa LVS pool along with the text squids [production]
21:54 <mark> Moved bits.pmtpa.wikimedia.org DNS from 208.80.152.2 (text squids) to 208.80.152.118 (dedicated LVS service) [production]
21:21 <mark> Started puppetd on spence [production]
20:50 <mark> Setup Nagios monitoring for bits.pmtpa [production]
20:33 <mark> Added LVS service for bits.pmtpa.wikimedia.org on lvs4 [production]
19:53 <mark> Added LVS service ip 208.80.152.118 (bits.pmtpa.wikimedia.org) to all text squids and sq67 [production]
19:18 <mark> Fixed ganglia varnish monitoring on sq67 [production]
13:01 <domas> thumbs get i/o errors across multiple clients [production]
10:42 <domas> xmltypecheck loop filled / with error logging on srv243 ( http://p.defau.lt/?KtZMFW9x7xaEa2eDwuqMEw ) - had to do some cleanup. didn't livehack anything yet. [production]
2010-07-24 §
21:31 <mark> Deployed varnish with configuration for bits.wikimedia.org on sq67 - not active yet [production]
03:59 <apergos> running xml dump file consistency checking script on snapshot2 and snapshot3 in screen as root, expect these to run all night, maybe through the next day as well [production]
2010-07-23 §
19:17 <Rob> synced for Bug 24470 - Enable NewUserMessage extension on lv.wikipedia [production]
19:17 <root> synchronized php-1.5/wmf-config/InitialiseSettings.php [production]
18:21 <tomaszf> moving otrs-783-529404994.sql to tridge to free up space [production]
18:21 <kaldari> Backing up dev_civicrm database in preparation for drupal upgrade [production]
11:33 <RoanKattouw> Started Apache on srv278, had died mysteriously [production]
09:30 <catrope> synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 24505: Editprotected group for hiwiki' [production]
02:02 <Rob> dig tested against all three nameservers after update, all nominal [production]
01:59 <Rob> pushing dns update to fix pointer for bugs.wikimedia.org [production]
01:30 <Tim> running some statistics queries on db38 [production]
2010-07-22 §
19:31 <Rob> mail server works, bypass expiration issue, rob will fix when all of the employees are not hitting the mail server. [production]
18:13 <kate> synchronized php-1.5/wmf-config/db.php 'put ixia back' [production]
17:50 <Rob> wmf mail server cert expired, we know, working on replacing it now [production]
17:29 <mark> Fixed ganglia data sources by qualifying all hostnames in gmetad_pmtpa.conf; this must have broken by the resolv.conf change last week [production]
17:16 <catrope> synchronized php-1.5/skins/MonoBook.php 'r69735' [production]
17:15 <catrope> synchronized php-1.5/skins/Vector.php 'r69735' [production]
17:11 <catrope> synchronized php-1.5/wmf-config/CommonSettings.php 'Hide Navigable TOC preference' [production]
17:10 <atglenn> xml dumps stopped in preparation for removal of bad dumps and xml code fix push [production]
17:09 <atglenn> snapshot3 added back to sync cluster in preparation for xml dumps fixes [production]
16:50 <kate> synchronized php-1.5/wmf-config/db.php 'put db27 back' [production]
16:37 <mark> Installed Lucid on sq67 [production]
16:26 <mark> Made Lucid the default distribution for new installs [production]
16:21 <mark> Fixed sq73 [production]
15:56 <mark> Fixed ganglia cron job on spence. [production]
15:46 <kate> synchronized php-1.5/wmf-config/db.php 'put db7 back' [production]
15:29 <RoanKattouw> Looks like that resolved the zombie issue on srv86 [production]
15:15 <mark> Fixed puppet for app servers [production]
15:05 <RoanKattouw> Defunct Apache process running on srv86, Apache won't start. Need root to kill zombie [production]
15:04 <RoanKattouw> Started Apache on srv270, srv163, had died for some reason [production]
15:04 <RoanKattouw> Deployed r69728, r69729 (UsabilityInitiative, Vector updates) about 30 minutes ago [production]
15:03 <RoanKattouw> <RoanKattouw>\t!log Deploying r69728, r69729 (UsabilityInitiative, Vector updates) to test.wikipedia.org [production]
15:03 <RoanKattouw> <mark>\t!log Upgraded Locke to Ubuntu 10.04 Lucid [production]
15:03 <RoanKattouw> <logmsgbot>\t!log kate synchronized php-1.5/wmf-config/db.php 'removed ixia to dump s4 for TS' [production]
15:03 <RoanKattouw> <logmsgbot>\t!log kate synchronized php-1.5/wmf-config/db.php 'removed db7 to dump s6 for TS' [production]