101-150 of 7808 results (3ms)
2010-07-25 §
19:53 <mark> Added LVS service ip 208.80.152.118 (bits.pmtpa.wikimedia.org) to all text squids and sq67 [production]
19:18 <mark> Fixed ganglia varnish monitoring on sq67 [production]
13:01 <domas> thumbs get i/o errors across multiple clients [production]
10:42 <domas> xmltypecheck loop filled / with error logging on srv243 ( http://p.defau.lt/?KtZMFW9x7xaEa2eDwuqMEw ) - had to do some cleanup. didn't livehack anything yet. [production]
2010-07-24 §
21:31 <mark> Deployed varnish with configuration for bits.wikimedia.org on sq67 - not active yet [production]
03:59 <apergos> running xml dump file consistency checking script on snapshot2 and snapshot3 in screen as root, expect these to run all night, maybe through the next day as well [production]
2010-07-23 §
19:17 <Rob> synced for Bug 24470 - Enable NewUserMessage extension on lv.wikipedia [production]
19:17 <root> synchronized php-1.5/wmf-config/InitialiseSettings.php [production]
18:21 <tomaszf> moving otrs-783-529404994.sql to tridge to free up space [production]
18:21 <kaldari> Backing up dev_civicrm database in preparation for drupal upgrade [production]
11:33 <RoanKattouw> Started Apache on srv278, had died mysteriously [production]
09:30 <catrope> synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 24505: Editprotected group for hiwiki' [production]
02:02 <Rob> dig tested against all three nameservers after update, all nominal [production]
01:59 <Rob> pushing dns update to fix pointer for bugs.wikimedia.org [production]
01:30 <Tim> running some statistics queries on db38 [production]
2010-07-22 §
19:31 <Rob> mail server works, bypass expiration issue, rob will fix when all of the employees are not hitting the mail server. [production]
18:13 <kate> synchronized php-1.5/wmf-config/db.php 'put ixia back' [production]
17:50 <Rob> wmf mail server cert expired, we know, working on replacing it now [production]
17:29 <mark> Fixed ganglia data sources by qualifying all hostnames in gmetad_pmtpa.conf; this must have broken by the resolv.conf change last week [production]
17:16 <catrope> synchronized php-1.5/skins/MonoBook.php 'r69735' [production]
17:15 <catrope> synchronized php-1.5/skins/Vector.php 'r69735' [production]
17:11 <catrope> synchronized php-1.5/wmf-config/CommonSettings.php 'Hide Navigable TOC preference' [production]
17:10 <atglenn> xml dumps stopped in preparation for removal of bad dumps and xml code fix push [production]
17:09 <atglenn> snapshot3 added back to sync cluster in preparation for xml dumps fixes [production]
16:50 <kate> synchronized php-1.5/wmf-config/db.php 'put db27 back' [production]
16:37 <mark> Installed Lucid on sq67 [production]
16:26 <mark> Made Lucid the default distribution for new installs [production]
16:21 <mark> Fixed sq73 [production]
15:56 <mark> Fixed ganglia cron job on spence. [production]
15:46 <kate> synchronized php-1.5/wmf-config/db.php 'put db7 back' [production]
15:29 <RoanKattouw> Looks like that resolved the zombie issue on srv86 [production]
15:15 <mark> Fixed puppet for app servers [production]
15:05 <RoanKattouw> Defunct Apache process running on srv86, Apache won't start. Need root to kill zombie [production]
15:04 <RoanKattouw> Started Apache on srv270, srv163, had died for some reason [production]
15:04 <RoanKattouw> Deployed r69728, r69729 (UsabilityInitiative, Vector updates) about 30 minutes ago [production]
15:03 <RoanKattouw> <RoanKattouw>\t!log Deploying r69728, r69729 (UsabilityInitiative, Vector updates) to test.wikipedia.org [production]
15:03 <RoanKattouw> <mark>\t!log Upgraded Locke to Ubuntu 10.04 Lucid [production]
15:03 <RoanKattouw> <logmsgbot>\t!log kate synchronized php-1.5/wmf-config/db.php 'removed ixia to dump s4 for TS' [production]
15:03 <RoanKattouw> <logmsgbot>\t!log kate synchronized php-1.5/wmf-config/db.php 'removed db7 to dump s6 for TS' [production]
15:03 <RoanKattouw> <logmsgbot>\t!log kate synchronized php-1.5/wmf-config/db.php 'removing db27 to dump s3 for TS' [production]
15:02 <RoanKattouw> Started morebots as catrope. werdnum has a defunt instance running that I can't kill [production]
2010-07-21 §
20:02 <catrope> synchronized php-1.5/extensions/CodeReview/ui/CodeRevisionListView.php 'r69705' [production]
19:44 <Rob> all 5 new misc servers have mgmt working, will allocate network port in a bit and get them loaded up [production]
19:43 <mark> Setup logging for [[Special:Book]] on locke [production]
19:42 <Rob> all dns servers back online and happy [production]
19:38 <Rob> grosley back online providing services [production]
19:30 <Rob> grosley was down due to my dns issue, rebooting it, sorry about that. [production]
19:29 <Rob> restarted pdns on linne [production]
19:25 <Rob> fixed letter reversal in dns, repushed [production]
19:13 <Rob> linne pdns restarted [production]