|
2010-07-17
§
|
| 18:48 |
<mark> |
Having puppet install wikimedia-raid-utils on all servers |
[production] |
| 18:12 |
<mark> |
srv278 rebooted for no apparent reason |
[production] |
| 17:21 |
<mark> |
Fixed ganglia mess on search19 |
[production] |
| 16:19 |
<mark> |
Having puppet install NRPE on all internal servers by default |
[production] |
| 15:52 |
<root> |
synchronized php-1.5/wmf-config/lucene.php 'Use fixed LVS service search-pool3 instead of search7 directly' |
[production] |
| 15:31 |
<mark> |
Removed fscking nscd from spence |
[production] |
| 15:20 |
<mark> |
Added internal LVS services to pmtpa.wmnet DNS |
[production] |
| 15:11 |
<mark> |
Doing reboot test of nescio (esams DNS) |
[production] |
| 15:08 |
<mark> |
Fixed puppet exchange of ssh hostkeys which has been broken for a while |
[production] |
| 14:30 |
<mark> |
Truncated all tables in the puppet db on db9 |
[production] |
| 12:30 |
<mark> |
Implemented domain search list in resolv.conf for fenari (use sparingly!) |
[production] |
| 12:24 |
<mark> |
Put /etc/resolv.conf under Puppet management on all servers. Setting timeout option to 3s, to avoid PyBal depools due to 5s timeout when the primary resolver is down. |
[production] |
| 11:38 |
<mark> |
Removed rkhunter and chkrootkit on bayes. What is the point with 2 year old software? Just creating more cron spam? :) |
[production] |
| 10:06 |
<mark> |
Fixed degraded raid on nescio |
[production] |
| 09:53 |
<mark> |
Removed X.org and gdm from bayes. Why was it installed? |
[production] |
| 09:30 |
<domas> |
doing maintenance on db13 and db15 (BIOS, OS, MySQL upgrades, resync of data) |
[production] |
|
2010-07-16
§
|
| 21:05 |
<mark> |
Removed old-style DNS monitoring from Nagios conf.php, now fully Puppet managed |
[production] |
| 21:03 |
<mark> |
Deployed authoritative DNS on nescio.esams, and moved the service IP |
[production] |
| 19:32 |
<mark> |
Deployed PowerDNS recursor on nescio, and moved the 91.198.174.6 service ip to it |
[production] |
| 19:16 |
<mark> |
Installed lucid on new server nescio.esams |
[production] |
| 13:49 |
<mark> |
Fixed entry in upload.wikimedia.org georecord, sent to text squids for a few ip ranges |
[production] |
| 12:27 |
<mark> |
Pointed country code 'eu' (127.0.255.1 or 65281) to esams in geodns; since geobackend uses a signed short, I had to mask it to 0x7ff / 32513 in the director map |
[production] |
| 10:49 |
<mark> |
Shutdown fuchsia for decommissioning |
[production] |
| 07:57 |
<catrope> |
synchronizing Wikimedia installation... Revision: 69416 |
[production] |
| 07:56 |
<RoanKattouw> |
Running scap to deploy r69416 |
[production] |
| 01:48 |
<tstarling> |
synchronized php-1.5/includes/diff/DifferenceInterface.php 'r69414' |
[production] |
|
2010-07-15
§
|
| 19:23 |
<mark> |
Replaced all occurrences of 'rr.wikimedia.org' with 'text.wikimedia.org' in DNS |
[production] |
| 19:14 |
<mark> |
Updated IP of deprecated record rr.esams.wikimedia.org |
[production] |
| 19:10 |
<mark> |
Started PyBal on amslvs1 with a new config; it automatically picked up the traffic for both text.esams (91.198.174.232) and bits.esams (91.198.174.233) |
[production] |
| 19:07 |
<mark> |
Stopped PyBal on amslvs1, BGP and OSPF did an automatic failover of bits.esams (91.198.174.233) to amslvs3 |
[production] |
| 18:59 |
<mark> |
Removed IP 91.198.174.2 (old text squids service ip) from amslvs1. Anyone still using the old IP after weeks will now be unable to reach our sites. |
[production] |
| 18:56 |
<mark> |
Depooled knsq1-knsq7 in PyBal |
[production] |
| 17:38 |
<Fred> |
fixed nfs mounts on Bayes. |
[production] |
| 15:35 |
<apergos> |
chowned /mnt/upload6/private/ExtensionDistributor/mw-snapshot/trunk/extensions tree to extdist. ExtensionDistributor apparently working now |
[production] |
| 15:01 |
<apergos> |
running svn cleanup on /mnt/upload6/private/ExtensionDistributor/mw-snapshot/trunk/extensions as extdist user |
[production] |
| 12:34 |
<tstarling> |
synchronizing Wikimedia installation... Revision: 69381 |
[production] |
| 12:18 |
<Tim> |
svn up/scap to r69380 |
[production] |
| 05:13 |
<jeluf> |
synchronized php-1.5/wmf-config/InitialiseSettings.php '24321 - ml.wikiquote.org lost its project namespace' |
[production] |
|
2010-07-14
§
|
| 23:44 |
<Fred> |
re-added ccron job to periodically save rrds on our ganglia server. (cron job seems to have vanished for some reason) |
[production] |
| 17:59 |
<catrope> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'Favicon for wikimaniateamwiki per Guillaume' |
[production] |
| 16:06 |
<Fred> |
restarted apache on mobile1 (had begun to return 500) |
[production] |
| 14:07 |
<mark> |
Fixed memcached on srv110 |
[production] |
| 12:19 |
<mark> |
Fixed ganglia and puppet on stafford |
[production] |
| 11:54 |
<mark> |
Migrated DNS monitoring to puppet |
[production] |
| 10:31 |
<mark> |
Migrated ZFS RAID nagios check to puppet |
[production] |
| 10:14 |
<mark> |
Migrated monitoring of lucene to puppet |
[production] |
| 09:37 |
<mark> |
Migrated monitoring of image scalers to puppet |
[production] |
| 08:49 |
<Tim> |
using stafford for some pbuilder experimentation |
[production] |