701-750 of 10000 results (9ms)
2012-06-05 §
18:45 <notpeter> starting innobackupex dump from blondel to bellin [production]
18:44 <notpeter> starting indexing on new searchidx2 [production]
18:39 <mark> Added static routes 2002::/16 and 2001::/32 for 6to4 and teredo on the Tampa routers; these are redistributed in OSPF to eqiad [production]
18:29 <notpeter> restart indexing on searchidex1001 [production]
18:20 <mark> Replaced static LVS IPv6 routes with correct next-hops on cr1-eqiad and cr2-eqiad [production]
18:09 <mark> Redistributing static routes in OSPF on cr1-eqiad and cr2-eqiad [production]
18:04 <paravoid> rebooting capella to make sure things work after a reboot [production]
17:53 <mark> Redistributed statics in OSPF3 on csw1-esams [production]
17:14 <logmsgbot_> aaron synchronized php-1.20wmf4/includes/logging/LogEventsList.php 'deployed d9f146ac42f2884e76390d6bc979eb10032adf7f' [production]
17:09 <mark> Added uRPF exception for 6to4 traffic on all routers [production]
16:52 <mark> Pooled ssl1001 [production]
15:44 <logmsgbot_> asher synchronized wmf-config/db.php 'returning es2 to service' [production]
15:17 <binasher> rebooting es2 for kernel + mysql upgrade [production]
15:16 <logmsgbot_> asher synchronized wmf-config/db.php 'pulling es2 for kernel+mysql upgrades' [production]
14:56 <paravoid> rebooting amslvs3 & amslvs4 to reinstall with precise [production]
14:20 <paravoid> rebooting lvs1006 to reinstall with precise [production]
13:51 <cmjohnson1> shutting down bellin to replace main board [production]
13:49 <notpeter> reimaging db1042 [production]
13:40 <paravoid> rebooting lvs1005 to reinstall with precise [production]
12:59 <paravoid> rebooting lvs2 to reinstall with precise [production]
12:47 <Ryan_Lane> changing capella's subnet in DNS [production]
12:10 <Ryan_Lane> rebuilding capella as precise [production]
10:01 <logmsgbot_> asher synchronized wmf-config/db.php 'putting es1 in production' [production]
09:53 <notpeter> cancel that, it's mid-cron. will do later [production]
09:52 <notpeter> stopping indexing on searchidx1001 to rsync to searchidx2 [production]
09:35 <binasher> rebooting es1 for kernel+mysql upgrade. dont need to pull from db.php because it was never correctly added or queried? [production]
09:14 <mark> Built PyBal 1.01 for precise, and included it in the precise-wikimedia APT repository [production]
08:45 <binasher> restarted mysql on es1004 with default innodb file format as barracuda [production]
08:31 <notpeter> reimaging searchidx2 [production]
02:36 <logmsgbot_> LocalisationUpdate completed (1.20wmf3) at Tue Jun 5 02:36:44 UTC 2012 [production]
02:14 <logmsgbot_> LocalisationUpdate completed (1.20wmf4) at Tue Jun 5 02:14:17 UTC 2012 [production]
2012-06-04 §
23:23 <logmsgbot_> asher synchronized wmf-config/db.php 'returning es4' [production]
22:49 <binasher> started an experiment on es1004 - altering all es tables from myisam to innodb one at a time with file_per_table enabled [production]
22:39 <binasher> stopping mysql on es4. all tables marked as having repair fails are in cluster22, resyncing just those from es1002 [production]
21:52 <logmsgbot_> asher synchronized wmf-config/db.php 'pulling es4 again' [production]
21:43 <logmsgbot_> asher synchronized wmf-config/db.php 'returning es4 to service' [production]
21:16 <Ryan_Lane> restarting nginx on all ssl boxes again [production]
21:07 <Ryan_Lane> force running puppet on all ssl hosts again [production]
21:04 <Ryan_Lane> repooling ssl1, ssl1001, ssl3001 [production]
21:03 <Ryan_Lane> restarting nginx on all ssl hosts [production]
20:23 <binasher> rebooted es4 [production]
20:18 <logmsgbot_> asher synchronized wmf-config/db.php 'pulling es4 for post-crash upgrade' [production]
19:55 <logmsgbot_> kaldari synchronized php-1.20wmf4/extensions/PageTriage/PageTriage.hooks.php 'syncing PageTriage.hooks.php' [production]
19:32 <Ryan_Lane> force running puppet on ssl servers [production]
18:22 <Reedy> Nuked php-1.20wmf4 on mw64 then ran sync-common. Seems to have dealt with the permission errors [production]
18:11 <logmsgbot_> reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.20wmf4 [production]
16:48 <binasher> upgraded kernel on db1047 / analytics [production]
16:09 <Ryan_Lane> restarting ircecho on manganese [production]
15:21 <paravoid> reinstalling lvs1 with precise [production]
15:13 <mark> Added new IPv6 LVS prefixes to all routers for uRPF filters; BGP import filters still need adjusting for dual-family sessions [production]