|
2010-07-22
§
|
| 19:31 |
<Rob> |
mail server works, bypass expiration issue, rob will fix when all of the employees are not hitting the mail server. |
[production] |
| 18:13 |
<kate> |
synchronized php-1.5/wmf-config/db.php 'put ixia back' |
[production] |
| 17:50 |
<Rob> |
wmf mail server cert expired, we know, working on replacing it now |
[production] |
| 17:29 |
<mark> |
Fixed ganglia data sources by qualifying all hostnames in gmetad_pmtpa.conf; this must have broken by the resolv.conf change last week |
[production] |
| 17:16 |
<catrope> |
synchronized php-1.5/skins/MonoBook.php 'r69735' |
[production] |
| 17:15 |
<catrope> |
synchronized php-1.5/skins/Vector.php 'r69735' |
[production] |
| 17:11 |
<catrope> |
synchronized php-1.5/wmf-config/CommonSettings.php 'Hide Navigable TOC preference' |
[production] |
| 17:10 |
<atglenn> |
xml dumps stopped in preparation for removal of bad dumps and xml code fix push |
[production] |
| 17:09 |
<atglenn> |
snapshot3 added back to sync cluster in preparation for xml dumps fixes |
[production] |
| 16:50 |
<kate> |
synchronized php-1.5/wmf-config/db.php 'put db27 back' |
[production] |
| 16:37 |
<mark> |
Installed Lucid on sq67 |
[production] |
| 16:26 |
<mark> |
Made Lucid the default distribution for new installs |
[production] |
| 16:21 |
<mark> |
Fixed sq73 |
[production] |
| 15:56 |
<mark> |
Fixed ganglia cron job on spence. |
[production] |
| 15:46 |
<kate> |
synchronized php-1.5/wmf-config/db.php 'put db7 back' |
[production] |
| 15:29 |
<RoanKattouw> |
Looks like that resolved the zombie issue on srv86 |
[production] |
| 15:15 |
<mark> |
Fixed puppet for app servers |
[production] |
| 15:05 |
<RoanKattouw> |
Defunct Apache process running on srv86, Apache won't start. Need root to kill zombie |
[production] |
| 15:04 |
<RoanKattouw> |
Started Apache on srv270, srv163, had died for some reason |
[production] |
| 15:04 |
<RoanKattouw> |
Deployed r69728, r69729 (UsabilityInitiative, Vector updates) about 30 minutes ago |
[production] |
| 15:03 |
<RoanKattouw> |
<RoanKattouw>\t!log Deploying r69728, r69729 (UsabilityInitiative, Vector updates) to test.wikipedia.org |
[production] |
| 15:03 |
<RoanKattouw> |
<mark>\t!log Upgraded Locke to Ubuntu 10.04 Lucid |
[production] |
| 15:03 |
<RoanKattouw> |
<logmsgbot>\t!log kate synchronized php-1.5/wmf-config/db.php 'removed ixia to dump s4 for TS' |
[production] |
| 15:03 |
<RoanKattouw> |
<logmsgbot>\t!log kate synchronized php-1.5/wmf-config/db.php 'removed db7 to dump s6 for TS' |
[production] |
| 15:03 |
<RoanKattouw> |
<logmsgbot>\t!log kate synchronized php-1.5/wmf-config/db.php 'removing db27 to dump s3 for TS' |
[production] |
| 15:02 |
<RoanKattouw> |
Started morebots as catrope. werdnum has a defunt instance running that I can't kill |
[production] |
|
2010-07-21
§
|
| 20:02 |
<catrope> |
synchronized php-1.5/extensions/CodeReview/ui/CodeRevisionListView.php 'r69705' |
[production] |
| 19:44 |
<Rob> |
all 5 new misc servers have mgmt working, will allocate network port in a bit and get them loaded up |
[production] |
| 19:43 |
<mark> |
Setup logging for [[Special:Book]] on locke |
[production] |
| 19:42 |
<Rob> |
all dns servers back online and happy |
[production] |
| 19:38 |
<Rob> |
grosley back online providing services |
[production] |
| 19:30 |
<Rob> |
grosley was down due to my dns issue, rebooting it, sorry about that. |
[production] |
| 19:29 |
<Rob> |
restarted pdns on linne |
[production] |
| 19:25 |
<Rob> |
fixed letter reversal in dns, repushed |
[production] |
| 19:13 |
<Rob> |
linne pdns restarted |
[production] |
| 19:06 |
<Rob> |
updating dns for 5 new misc servers mgmt |
[production] |
| 16:35 |
<Rob> |
lots more time wasted diagnosing evident mainboard failures on srv284, hopefully replacement will soon be inroute |
[production] |
| 16:10 |
<mark> |
Done basic setup of asw-b-sdtpa, added to RANCID and torrus |
[production] |
| 14:27 |
<Rob> |
ms1 is back, passing it off to someone who doesnt hate it |
[production] |
| 14:06 |
<Rob> |
working on ms1, no touchy |
[production] |
| 09:56 |
<catrope> |
synchronized php-1.5/includes/api/ApiLogin.php 'r69661' |
[production] |
| 02:23 |
<Tim> |
increasing nagios retry count for NTP from 8 to 15 |
[production] |