|
2012-06-05
§
|
| 12:10 |
<Ryan_Lane> |
rebuilding capella as precise |
[production] |
| 10:01 |
<logmsgbot_> |
asher synchronized wmf-config/db.php 'putting es1 in production' |
[production] |
| 09:53 |
<notpeter> |
cancel that, it's mid-cron. will do later |
[production] |
| 09:52 |
<notpeter> |
stopping indexing on searchidx1001 to rsync to searchidx2 |
[production] |
| 09:35 |
<binasher> |
rebooting es1 for kernel+mysql upgrade. dont need to pull from db.php because it was never correctly added or queried? |
[production] |
| 09:14 |
<mark> |
Built PyBal 1.01 for precise, and included it in the precise-wikimedia APT repository |
[production] |
| 08:45 |
<binasher> |
restarted mysql on es1004 with default innodb file format as barracuda |
[production] |
| 08:31 |
<notpeter> |
reimaging searchidx2 |
[production] |
| 02:36 |
<logmsgbot_> |
LocalisationUpdate completed (1.20wmf3) at Tue Jun 5 02:36:44 UTC 2012 |
[production] |
| 02:14 |
<logmsgbot_> |
LocalisationUpdate completed (1.20wmf4) at Tue Jun 5 02:14:17 UTC 2012 |
[production] |
|
2012-06-04
§
|
| 23:23 |
<logmsgbot_> |
asher synchronized wmf-config/db.php 'returning es4' |
[production] |
| 22:49 |
<binasher> |
started an experiment on es1004 - altering all es tables from myisam to innodb one at a time with file_per_table enabled |
[production] |
| 22:39 |
<binasher> |
stopping mysql on es4. all tables marked as having repair fails are in cluster22, resyncing just those from es1002 |
[production] |
| 21:52 |
<logmsgbot_> |
asher synchronized wmf-config/db.php 'pulling es4 again' |
[production] |
| 21:43 |
<logmsgbot_> |
asher synchronized wmf-config/db.php 'returning es4 to service' |
[production] |
| 21:16 |
<Ryan_Lane> |
restarting nginx on all ssl boxes again |
[production] |
| 21:07 |
<Ryan_Lane> |
force running puppet on all ssl hosts again |
[production] |
| 21:04 |
<Ryan_Lane> |
repooling ssl1, ssl1001, ssl3001 |
[production] |
| 21:03 |
<Ryan_Lane> |
restarting nginx on all ssl hosts |
[production] |
| 20:23 |
<binasher> |
rebooted es4 |
[production] |
| 20:18 |
<logmsgbot_> |
asher synchronized wmf-config/db.php 'pulling es4 for post-crash upgrade' |
[production] |
| 19:55 |
<logmsgbot_> |
kaldari synchronized php-1.20wmf4/extensions/PageTriage/PageTriage.hooks.php 'syncing PageTriage.hooks.php' |
[production] |
| 19:32 |
<Ryan_Lane> |
force running puppet on ssl servers |
[production] |
| 18:22 |
<Reedy> |
Nuked php-1.20wmf4 on mw64 then ran sync-common. Seems to have dealt with the permission errors |
[production] |
| 18:11 |
<logmsgbot_> |
reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.20wmf4 |
[production] |
| 16:48 |
<binasher> |
upgraded kernel on db1047 / analytics |
[production] |
| 16:09 |
<Ryan_Lane> |
restarting ircecho on manganese |
[production] |
| 15:21 |
<paravoid> |
reinstalling lvs1 with precise |
[production] |
| 15:13 |
<mark> |
Added new IPv6 LVS prefixes to all routers for uRPF filters; BGP import filters still need adjusting for dual-family sessions |
[production] |
| 15:08 |
<cmjohnson1> |
physically power cycling lvs1 |
[production] |
| 15:02 |
<Ryan_Lane> |
depooling ssl1001 and ssl3001 |
[production] |
| 14:55 |
<Ryan_Lane> |
disabling puppet on all ssl hosts |
[production] |
| 13:27 |
<mark> |
Changed upload.esams.wikimedia.org CNAME to upload-lb.esams, effectively disabling the IPv6 selective answer script |
[production] |
| 12:23 |
<mark> |
Upgrading wikimedia-lvs-realserver to version 0.08 across the cluster (by Puppet) |
[production] |
| 12:18 |
<Ryan_Lane> |
depooling ssl1 |
[production] |
| 11:32 |
<mark> |
Copied wikimedia-lvs-realserver 0.08 from APT distribution precise-wikimedia to lucid-wikimedia |
[production] |
| 02:38 |
<LocalisationUpdate> |
completed (1.20wmf3) at Mon Jun 4 02:38:15 UTC 2012 |
[production] |
| 02:14 |
<LocalisationUpdate> |
completed (1.20wmf4) at Mon Jun 4 02:14:45 UTC 2012 |
[production] |
|
2012-06-03
§
|
| 15:45 |
<paravoid> |
aborting lvs1 install, partition map is not ready; putting it back to production as-is |
[production] |
| 15:31 |
<paravoid> |
reinstalling lvs1 with precise |
[production] |
| 15:10 |
<RobH> |
torrus failed to refresh via puppet (failed refresh takes too long) so manually running the refresh/rebuild command as puppet copied the updates to the system |
[production] |
| 14:55 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Bug 37237 - Change Wikisource namespace for Tamil wikisource' |
[production] |
| 14:49 |
<reedy> |
synchronized wmf-config/CommonSettings.php 'Bug 37211 - Set $wgUseCombinedLoginLink = false' |
[production] |
| 14:44 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Bug 37294 - Add English Wikibooks as import source at Vietnamese Wikibooks' |
[production] |
| 14:07 |
<midom> |
synchronized wmf-config/db.php |
[production] |
| 13:10 |
<hashar> |
synchronized wmf-config/InitialiseSettings.php '59753a9 (allow bureaucrats on frwiki to add+remove accountcreator group' |
[production] |
| 13:09 |
<hashar> |
synchronized wmf-config/CommonSettings.php 'Commits: 6bef518 (wgHTCPMulticast only used on production cluster) and 882dd69 (wgLoadScript only used on production) -- was not correctly deployed earlier' |
[production] |
| 13:02 |
<midom> |
synchronized wmf-config/db.php |
[production] |
| 10:18 |
<hashar> |
mw64: rsync: write failed on "/apache/common-local/wmf-config/CommonSettings.php": No space left on device (28) |
[production] |
| 10:17 |
<hashar> |
synchronized wmf-config/CommonSettings.php 'Commits: 6bef518 (wgHTCPMulticast only used on production cluster) and 882dd69 (wgLoadScript only used on production)' |
[production] |