|
2012-06-05
§
|
| 18:45 |
<notpeter> |
starting innobackupex dump from blondel to bellin |
[production] |
| 18:44 |
<notpeter> |
starting indexing on new searchidx2 |
[production] |
| 18:39 |
<mark> |
Added static routes 2002::/16 and 2001::/32 for 6to4 and teredo on the Tampa routers; these are redistributed in OSPF to eqiad |
[production] |
| 18:29 |
<notpeter> |
restart indexing on searchidex1001 |
[production] |
| 18:20 |
<mark> |
Replaced static LVS IPv6 routes with correct next-hops on cr1-eqiad and cr2-eqiad |
[production] |
| 18:09 |
<mark> |
Redistributing static routes in OSPF on cr1-eqiad and cr2-eqiad |
[production] |
| 18:04 |
<paravoid> |
rebooting capella to make sure things work after a reboot |
[production] |
| 17:53 |
<mark> |
Redistributed statics in OSPF3 on csw1-esams |
[production] |
| 17:14 |
<logmsgbot_> |
aaron synchronized php-1.20wmf4/includes/logging/LogEventsList.php 'deployed d9f146ac42f2884e76390d6bc979eb10032adf7f' |
[production] |
| 17:09 |
<mark> |
Added uRPF exception for 6to4 traffic on all routers |
[production] |
| 16:52 |
<mark> |
Pooled ssl1001 |
[production] |
| 15:44 |
<logmsgbot_> |
asher synchronized wmf-config/db.php 'returning es2 to service' |
[production] |
| 15:17 |
<binasher> |
rebooting es2 for kernel + mysql upgrade |
[production] |
| 15:16 |
<logmsgbot_> |
asher synchronized wmf-config/db.php 'pulling es2 for kernel+mysql upgrades' |
[production] |
| 14:56 |
<paravoid> |
rebooting amslvs3 & amslvs4 to reinstall with precise |
[production] |
| 14:20 |
<paravoid> |
rebooting lvs1006 to reinstall with precise |
[production] |
| 13:51 |
<cmjohnson1> |
shutting down bellin to replace main board |
[production] |
| 13:49 |
<notpeter> |
reimaging db1042 |
[production] |
| 13:40 |
<paravoid> |
rebooting lvs1005 to reinstall with precise |
[production] |
| 12:59 |
<paravoid> |
rebooting lvs2 to reinstall with precise |
[production] |
| 12:47 |
<Ryan_Lane> |
changing capella's subnet in DNS |
[production] |
| 12:10 |
<Ryan_Lane> |
rebuilding capella as precise |
[production] |
| 10:01 |
<logmsgbot_> |
asher synchronized wmf-config/db.php 'putting es1 in production' |
[production] |
| 09:53 |
<notpeter> |
cancel that, it's mid-cron. will do later |
[production] |
| 09:52 |
<notpeter> |
stopping indexing on searchidx1001 to rsync to searchidx2 |
[production] |
| 09:35 |
<binasher> |
rebooting es1 for kernel+mysql upgrade. dont need to pull from db.php because it was never correctly added or queried? |
[production] |
| 09:14 |
<mark> |
Built PyBal 1.01 for precise, and included it in the precise-wikimedia APT repository |
[production] |
| 08:45 |
<binasher> |
restarted mysql on es1004 with default innodb file format as barracuda |
[production] |
| 08:31 |
<notpeter> |
reimaging searchidx2 |
[production] |
| 02:36 |
<logmsgbot_> |
LocalisationUpdate completed (1.20wmf3) at Tue Jun 5 02:36:44 UTC 2012 |
[production] |
| 02:14 |
<logmsgbot_> |
LocalisationUpdate completed (1.20wmf4) at Tue Jun 5 02:14:17 UTC 2012 |
[production] |
|
2012-06-04
§
|
| 23:23 |
<logmsgbot_> |
asher synchronized wmf-config/db.php 'returning es4' |
[production] |
| 22:49 |
<binasher> |
started an experiment on es1004 - altering all es tables from myisam to innodb one at a time with file_per_table enabled |
[production] |
| 22:39 |
<binasher> |
stopping mysql on es4. all tables marked as having repair fails are in cluster22, resyncing just those from es1002 |
[production] |
| 21:52 |
<logmsgbot_> |
asher synchronized wmf-config/db.php 'pulling es4 again' |
[production] |
| 21:43 |
<logmsgbot_> |
asher synchronized wmf-config/db.php 'returning es4 to service' |
[production] |
| 21:16 |
<Ryan_Lane> |
restarting nginx on all ssl boxes again |
[production] |
| 21:07 |
<Ryan_Lane> |
force running puppet on all ssl hosts again |
[production] |
| 21:04 |
<Ryan_Lane> |
repooling ssl1, ssl1001, ssl3001 |
[production] |
| 21:03 |
<Ryan_Lane> |
restarting nginx on all ssl hosts |
[production] |
| 20:23 |
<binasher> |
rebooted es4 |
[production] |
| 20:18 |
<logmsgbot_> |
asher synchronized wmf-config/db.php 'pulling es4 for post-crash upgrade' |
[production] |
| 19:55 |
<logmsgbot_> |
kaldari synchronized php-1.20wmf4/extensions/PageTriage/PageTriage.hooks.php 'syncing PageTriage.hooks.php' |
[production] |
| 19:32 |
<Ryan_Lane> |
force running puppet on ssl servers |
[production] |
| 18:22 |
<Reedy> |
Nuked php-1.20wmf4 on mw64 then ran sync-common. Seems to have dealt with the permission errors |
[production] |
| 18:11 |
<logmsgbot_> |
reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.20wmf4 |
[production] |
| 16:48 |
<binasher> |
upgraded kernel on db1047 / analytics |
[production] |
| 16:09 |
<Ryan_Lane> |
restarting ircecho on manganese |
[production] |
| 15:21 |
<paravoid> |
reinstalling lvs1 with precise |
[production] |
| 15:13 |
<mark> |
Added new IPv6 LVS prefixes to all routers for uRPF filters; BGP import filters still need adjusting for dual-family sessions |
[production] |