2010-07-25
§
|
21:55 |
<mark> |
Pooled sq67 (Varnish) in the bits.pmtpa LVS pool along with the text squids |
[production] |
21:54 |
<mark> |
Moved bits.pmtpa.wikimedia.org DNS from 208.80.152.2 (text squids) to 208.80.152.118 (dedicated LVS service) |
[production] |
21:21 |
<mark> |
Started puppetd on spence |
[production] |
20:50 |
<mark> |
Setup Nagios monitoring for bits.pmtpa |
[production] |
20:33 |
<mark> |
Added LVS service for bits.pmtpa.wikimedia.org on lvs4 |
[production] |
19:53 |
<mark> |
Added LVS service ip 208.80.152.118 (bits.pmtpa.wikimedia.org) to all text squids and sq67 |
[production] |
19:18 |
<mark> |
Fixed ganglia varnish monitoring on sq67 |
[production] |
13:01 |
<domas> |
thumbs get i/o errors across multiple clients |
[production] |
10:42 |
<domas> |
xmltypecheck loop filled / with error logging on srv243 ( http://p.defau.lt/?KtZMFW9x7xaEa2eDwuqMEw ) - had to do some cleanup. didn't livehack anything yet. |
[production] |
2010-07-23
§
|
19:17 |
<Rob> |
synced for Bug 24470 - Enable NewUserMessage extension on lv.wikipedia |
[production] |
19:17 |
<root> |
synchronized php-1.5/wmf-config/InitialiseSettings.php |
[production] |
18:21 |
<tomaszf> |
moving otrs-783-529404994.sql to tridge to free up space |
[production] |
18:21 |
<kaldari> |
Backing up dev_civicrm database in preparation for drupal upgrade |
[production] |
11:33 |
<RoanKattouw> |
Started Apache on srv278, had died mysteriously |
[production] |
09:30 |
<catrope> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 24505: Editprotected group for hiwiki' |
[production] |
02:02 |
<Rob> |
dig tested against all three nameservers after update, all nominal |
[production] |
01:59 |
<Rob> |
pushing dns update to fix pointer for bugs.wikimedia.org |
[production] |
01:30 |
<Tim> |
running some statistics queries on db38 |
[production] |
2010-07-22
§
|
19:31 |
<Rob> |
mail server works, bypass expiration issue, rob will fix when all of the employees are not hitting the mail server. |
[production] |
18:13 |
<kate> |
synchronized php-1.5/wmf-config/db.php 'put ixia back' |
[production] |
17:50 |
<Rob> |
wmf mail server cert expired, we know, working on replacing it now |
[production] |
17:29 |
<mark> |
Fixed ganglia data sources by qualifying all hostnames in gmetad_pmtpa.conf; this must have broken by the resolv.conf change last week |
[production] |
17:16 |
<catrope> |
synchronized php-1.5/skins/MonoBook.php 'r69735' |
[production] |
17:15 |
<catrope> |
synchronized php-1.5/skins/Vector.php 'r69735' |
[production] |
17:11 |
<catrope> |
synchronized php-1.5/wmf-config/CommonSettings.php 'Hide Navigable TOC preference' |
[production] |
17:10 |
<atglenn> |
xml dumps stopped in preparation for removal of bad dumps and xml code fix push |
[production] |
17:09 |
<atglenn> |
snapshot3 added back to sync cluster in preparation for xml dumps fixes |
[production] |
16:50 |
<kate> |
synchronized php-1.5/wmf-config/db.php 'put db27 back' |
[production] |
16:37 |
<mark> |
Installed Lucid on sq67 |
[production] |
16:26 |
<mark> |
Made Lucid the default distribution for new installs |
[production] |
16:21 |
<mark> |
Fixed sq73 |
[production] |
15:56 |
<mark> |
Fixed ganglia cron job on spence. |
[production] |
15:46 |
<kate> |
synchronized php-1.5/wmf-config/db.php 'put db7 back' |
[production] |
15:29 |
<RoanKattouw> |
Looks like that resolved the zombie issue on srv86 |
[production] |
15:15 |
<mark> |
Fixed puppet for app servers |
[production] |
15:05 |
<RoanKattouw> |
Defunct Apache process running on srv86, Apache won't start. Need root to kill zombie |
[production] |
15:04 |
<RoanKattouw> |
Started Apache on srv270, srv163, had died for some reason |
[production] |
15:04 |
<RoanKattouw> |
Deployed r69728, r69729 (UsabilityInitiative, Vector updates) about 30 minutes ago |
[production] |
15:03 |
<RoanKattouw> |
<RoanKattouw>\t!log Deploying r69728, r69729 (UsabilityInitiative, Vector updates) to test.wikipedia.org |
[production] |
15:03 |
<RoanKattouw> |
<mark>\t!log Upgraded Locke to Ubuntu 10.04 Lucid |
[production] |
15:03 |
<RoanKattouw> |
<logmsgbot>\t!log kate synchronized php-1.5/wmf-config/db.php 'removed ixia to dump s4 for TS' |
[production] |
15:03 |
<RoanKattouw> |
<logmsgbot>\t!log kate synchronized php-1.5/wmf-config/db.php 'removed db7 to dump s6 for TS' |
[production] |