551-600 of 8281 results (6ms)
2010-07-27 §
18:05 <Rob> bugzilla admin note: disregard the email bounces for the scireview domain user, their mail server will be fixed shortly [production]
17:58 <Rob> srv284 fixed, drac online, needs setup [production]
17:54 <Rob> !log srv281 was shutdown when I came into the DC. popped case to use as base for rebuilding another system. need to investigate its initial shutdown [production]
17:17 <mark> Reinstalled sq68 [production]
16:41 <atglenn> started up several more workers on snapshot3 doing xml dumps [production]
14:43 <Tim> running svn cleanup on some ExtensionDistributor working copies [production]
14:29 <Tim> cleaning up old ext-dist tarballs, removing all that were older than a month [production]
11:48 <mark> Stracing puppet on spence [production]
08:01 <apergos> started one thread of xml dumps from screen session on snapshot3 as root, if these look good tomorrow we'll crank up more of 'em [production]
2010-07-26 §
18:29 <Rob> moved up to srv268, all working [production]
18:25 <Rob> moved srv258-srv261 to new asw-b5-sdtpa ports, all seems shiny [production]
17:59 <Rob> hooked up additional ports for sq67/sq68 [production]
17:17 <mark> Installed Lucid on sq69 and sq70 [production]
14:41 <mark> Moved db19 back to vlan 2 [production]
14:34 <mark> Pooled sq68 in the bits.pmtpa varnish LVS pool [production]
14:06 <mark> Installed lucid on sq68 and deployed varnish for bits.pmtpa [production]
12:22 <mark> Started puppet on spence [production]
05:21 <Tim> removed a log on search12 to give it a tiny bit more space [production]
2010-07-25 §
21:55 <mark> Pooled sq67 (Varnish) in the bits.pmtpa LVS pool along with the text squids [production]
21:54 <mark> Moved bits.pmtpa.wikimedia.org DNS from 208.80.152.2 (text squids) to 208.80.152.118 (dedicated LVS service) [production]
21:21 <mark> Started puppetd on spence [production]
20:50 <mark> Setup Nagios monitoring for bits.pmtpa [production]
20:33 <mark> Added LVS service for bits.pmtpa.wikimedia.org on lvs4 [production]
19:53 <mark> Added LVS service ip 208.80.152.118 (bits.pmtpa.wikimedia.org) to all text squids and sq67 [production]
19:18 <mark> Fixed ganglia varnish monitoring on sq67 [production]
13:01 <domas> thumbs get i/o errors across multiple clients [production]
10:42 <domas> xmltypecheck loop filled / with error logging on srv243 ( http://p.defau.lt/?KtZMFW9x7xaEa2eDwuqMEw ) - had to do some cleanup. didn't livehack anything yet. [production]
2010-07-24 §
21:31 <mark> Deployed varnish with configuration for bits.wikimedia.org on sq67 - not active yet [production]
03:59 <apergos> running xml dump file consistency checking script on snapshot2 and snapshot3 in screen as root, expect these to run all night, maybe through the next day as well [production]
2010-07-23 §
19:17 <Rob> synced for Bug 24470 - Enable NewUserMessage extension on lv.wikipedia [production]
19:17 <root> synchronized php-1.5/wmf-config/InitialiseSettings.php [production]
18:21 <tomaszf> moving otrs-783-529404994.sql to tridge to free up space [production]
18:21 <kaldari> Backing up dev_civicrm database in preparation for drupal upgrade [production]
11:33 <RoanKattouw> Started Apache on srv278, had died mysteriously [production]
09:30 <catrope> synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 24505: Editprotected group for hiwiki' [production]
02:02 <Rob> dig tested against all three nameservers after update, all nominal [production]
01:59 <Rob> pushing dns update to fix pointer for bugs.wikimedia.org [production]
01:30 <Tim> running some statistics queries on db38 [production]
2010-07-22 §
19:31 <Rob> mail server works, bypass expiration issue, rob will fix when all of the employees are not hitting the mail server. [production]
18:13 <kate> synchronized php-1.5/wmf-config/db.php 'put ixia back' [production]
17:50 <Rob> wmf mail server cert expired, we know, working on replacing it now [production]
17:29 <mark> Fixed ganglia data sources by qualifying all hostnames in gmetad_pmtpa.conf; this must have broken by the resolv.conf change last week [production]
17:16 <catrope> synchronized php-1.5/skins/MonoBook.php 'r69735' [production]
17:15 <catrope> synchronized php-1.5/skins/Vector.php 'r69735' [production]
17:11 <catrope> synchronized php-1.5/wmf-config/CommonSettings.php 'Hide Navigable TOC preference' [production]
17:10 <atglenn> xml dumps stopped in preparation for removal of bad dumps and xml code fix push [production]
17:09 <atglenn> snapshot3 added back to sync cluster in preparation for xml dumps fixes [production]
16:50 <kate> synchronized php-1.5/wmf-config/db.php 'put db27 back' [production]
16:37 <mark> Installed Lucid on sq67 [production]
16:26 <mark> Made Lucid the default distribution for new installs [production]