|
2012-01-18
ยง
|
| 22:25 |
<binasher> |
swapping s1 master to db36 |
[production] |
| 22:24 |
<asher> |
synchronized wmf-config/db.php 'starting swap of s1 master to db36, s1 in read-only' |
[production] |
| 22:13 |
<asher> |
synchronized wmf-config/db.php 'returning db36 to normal weight' |
[production] |
| 22:07 |
<asher> |
synchronized wmf-config/db.php 'returning db36 at a low weight' |
[production] |
| 21:59 |
<awjrichards> |
synchronized php/extensions/CongressLookup/SpecialCongressLookup.php '[[rev:109440|r109440]]' |
[production] |
| 21:58 |
<binasher> |
rebooting db36, upgrading kernel + mysql |
[production] |
| 21:56 |
<asher> |
synchronized wmf-config/db.php 'pulling db36 from s1 for mysql/kernel upgrades' |
[production] |
| 21:54 |
<Ryan_Lane> |
installing python-wurfl on stat1 |
[production] |
| 21:35 |
<Ryan_Lane> |
installing geoip-bin geoip-database libgeoip1 python-geoip on stat1 |
[production] |
| 21:13 |
<asher> |
synchronized wmf-config/db.php 'returning db38 at prior weight' |
[production] |
| 21:05 |
<Reedy> |
Run patch-ug_group-length-increase.sql on all wikis |
[production] |
| 21:04 |
<Reedy> |
Run patch-uploadstash_chunk.sql on all wikis |
[production] |
| 21:03 |
<Reedy> |
Run patch-jobs-add-timestamp.sql on all wikis |
[production] |
| 20:55 |
<awjr> |
update cl_zip5 table for CongressLookup to data in r\t109408 |
[production] |
| 20:43 |
<Reedy> |
Manually running cleanupUploadStash.php against commonswiki |
[production] |
| 20:42 |
<Reedy> |
Manually ran cleanupUploadStash.php against enwiki |
[production] |
| 20:31 |
<binasher> |
db38 in service at a low weight with new lucid kernel and current mysql build |
[production] |
| 20:30 |
<RobH> |
shutting down db17, confirmed not in db rotation and has no mysql instance active |
[production] |
| 20:30 |
<asher> |
synchronized wmf-config/db.php 'returning db38 at a lower weight' |
[production] |
| 20:28 |
<asher> |
synchronized wmf-config/db.php 'pulling db38 again' |
[production] |
| 20:26 |
<asher> |
synchronized wmf-config/db.php 'returning db38 to service' |
[production] |
| 20:17 |
<LeslieCarr> |
rebooting spence as it's once again gone crazy |
[production] |
| 20:11 |
<binasher> |
pulled db38, rebooting for kernel and mysql upgrades |
[production] |
| 20:11 |
<asher> |
synchronized wmf-config/db.php 'pulling db38 from s1 for upgrade' |
[production] |
| 20:04 |
<RobH> |
mw1102 coming down for mainboard replacement |
[production] |
| 20:03 |
<LeslieCarr> |
killing puppet processes on spence |
[production] |
| 19:28 |
<Reedy> |
Run patch-jobs-add-timestamp.sql on enwiki (jobs table is empty!) |
[production] |
| 19:01 |
<mutante> |
mw1108 - OS installed, added to puppet, finished catalog run, free for use |
[production] |
| 18:37 |
<mutante> |
pxe booting mw1108, OS install |
[production] |
| 18:36 |
<mutante> |
fixed DHCP config for mw1108 on brewster, had the string "Failed to connect to 10.65.1.108." where the MAC address should have been. |
[production] |
| 18:27 |
<RobH> |
searchidx1001 memory replaced per rt 2208 |
[production] |
| 18:20 |
<mutante> |
tried to PXE boot mw1108 but no DHCP offers received |
[production] |
| 18:15 |
<RobH> |
searchidx1001 memory being replaced |
[production] |
| 18:14 |
<LeslieCarr> |
re-preffing tele2 routes |
[production] |
| 18:11 |
<RobH> |
db1004 hard disk replaced per rt#2140, rebuilding |
[production] |
| 17:40 |
<LeslieCarr> |
Draining HE to perform maintenance on the physical port |
[production] |
| 16:57 |
<reedy> |
synchronized php-1.18/extensions/CongressLookup/SpecialCongressLookup.php '[[rev:109395|r109395]]' |
[production] |
| 14:45 |
<mark> |
Changed service IP addresses of lists.wikimedia.org in DNS to US prefixes |
[production] |
| 14:40 |
<mark> |
Disabled hold_domains on sodium and lily |
[production] |
| 14:28 |
<mark> |
Setup lily to route lists.wikimedia.org mails to sodium |
[production] |
| 14:21 |
<mark> |
rsync complete. Running dpkg-reconfigure mailman on sodium |
[production] |
| 13:43 |
<demon> |
synchronized php-1.18/extensions/CongressLookup/SpecialCongressLookup.php '[[rev:109362|r109362]]' |
[production] |
| 13:38 |
<mark> |
Started rsync of selected mailman directories under /var/lib/mailman from lily to sodium |
[production] |
| 13:37 |
<mark> |
Removed all test messages on the exim4 queue on sodium |
[production] |
| 13:37 |
<mark> |
Created /var LVM snapshot on lily |
[production] |
| 13:35 |
<mark> |
Stopped lighttpd on lily |
[production] |
| 13:34 |
<mark> |
Stopped mailman on lily and sodium |
[production] |
| 13:30 |
<mark> |
Set hold_domains = lists.wikimedia.org on lily, to hold new lists mails on the queue |
[production] |
| 13:30 |
<mark> |
Starting mailman migration |
[production] |
| 13:03 |
<mutante> |
restarted pdns on ns0 |
[production] |