|
2009-07-17
§
|
| 20:38 |
<domas> |
ms2 has broken disks.. |
[production] |
| 20:31 |
<brion> |
We're going to see about setting up the previously-idle ms5 so we can get our thumbnailing on |
[production] |
| 20:01 |
<brion> |
rob's poking raid rebuild on storage2 (dumps server) |
[production] |
| 19:03 |
<RobH_A90> |
eiximenis and dobson pulled for solid state drive testing, do not use for other tasks |
[production] |
| 18:28 |
<brion> |
synchronized wmf-deployment/wmf-config/InitialiseSettings.php 'enabling vector for rtl' |
[production] |
| 18:25 |
<atglenn> |
started mass move out of the way of thumbnail dirs and replacing with symlinks to ms4 |
[production] |
| 18:25 |
<brion> |
synchronized wmf-deployment/wmf-config/CommonSettings.php 'bump style version' |
[production] |
| 18:25 |
<brion> |
ran sync-common-all |
[production] |
| 18:24 |
<brion> |
running sync-common-all for UI updates. need to poke the style ver too :) |
[production] |
| 18:08 |
<brion> |
svn up'ing wmf-deployment for test.wikipedia.org. Merged UI fixes from usability team |
[production] |
| 18:03 |
<Fred> |
spun a couple more apache server into image scalers: srv219..srv224. |
[production] |
| 17:28 |
<rainman-sr> |
putting new location of initialisesettings to lsearch-global-2.1.conf so the incremental updater works again |
[production] |
| 17:20 |
<Fred> |
srv224 is now an image_scaler. Adjusted on lvs3, ganglia and dsh's node_list. |
[production] |
| 17:14 |
<Fred> |
db20 back online |
[production] |
| 16:50 |
<Fred> |
rebooting db20 as it is in a "state" |
[production] |
| 16:45 |
<brion> |
looks like we've lost internal /home NFS, which makes some of our internal services very unhappy. investigating... |
[production] |
| 16:43 |
<brion> |
ganglia out. |
[production] |
| 13:44 |
<apergos1> |
doing next round of removals on ms1 (/export/upload/wikipedia/en/thumb/2) to keep ahead of the game |
[production] |
| 04:15 |
<apergos> |
starting removal of /export/upload/wikipedia/en/thumb/1 on ms1 (moved away and symlink to ms4 done already) for more space |
[production] |
| 03:54 |
<brion> |
synchronized wmf-deployment/wmf-config/CommonSettings.php 'Disabling sitenotice from maintenance' |
[production] |
| 03:29 |
<brion> |
reenabling uploads & image deletion/undeletion |
[production] |
| 03:29 |
<brion> |
synchronized wmf-deployment/wmf-config/CommonSettings.php |
[production] |
| 03:28 |
<brion> |
remounting ms1 on apaches |
[production] |
| 00:49 |
<atglenn> |
only about 1gb gain on each so doing all of /export/upload/wikipedia/en/thumb/0 |
[production] |
| 00:39 |
<atglenn> |
removing more directories in /export/upload/wikipedia/en/thumb/0 on ms1 and replacing with symlinks to ms4 |
[production] |
| 00:30 |
<brion> |
synchronized wmf-deployment/includes/specials/SpecialUpload.php |
[production] |
| 00:30 |
<brion> |
synchronized wmf-deployment/includes/ImagePage.php |
[production] |
| 00:27 |
<brion> |
synchronized wmf-deployment/wmf-config/CommonSettings.php |
[production] |
| 00:20 |
<brion> |
temporarily disabling image delete/rename during maintenance |
[production] |
|
2009-07-16
§
|
| 23:56 |
<fvassard> |
synchronized php-1.5/wmf-config/CommonSettings.php 'Disabling uploads and setting captcha to not-fancy.' |
[production] |
| 23:29 |
<atglenn> |
removing the images in /export/upload/wikipedia/en/thumb/0/00 on ms1 (real dir is a symlink to ms4) to get back some space |
[production] |
| 22:51 |
<atglenn> |
sym link back in place, let's see what happens |
[production] |
| 22:47 |
<atglenn> |
reverting temporarily while we resolve mount issues for the ms4 share |
[production] |
| 22:40 |
<atglenn> |
...whether the image scalers will fall over if we force them to do (some) regeneration. |
[production] |
| 22:37 |
<atglenn> |
on ms1, /export/upload/wikipedia/en/thumb/0/00 symlinked to (shared from ms4) /mnt/thumbs/wikipedia/en/thumb/0/00 to test |
[production] |
| 21:29 |
<brion> |
robots.php for robots.txt generation now also working. yay! |
[production] |
| 21:28 |
<brion> |
synchronized live-1.5/robots.php |
[production] |
| 21:28 |
<brion> |
extract2.php now fixed up for new deployment; portal pages ok (www.wikipedia.org) |
[production] |
| 21:27 |
<brion> |
synchronized live-1.5/robots.php |
[production] |
| 21:26 |
<brion> |
synchronized extract2.php |
[production] |
| 21:26 |
<brion> |
synchronized extract2.php |
[production] |
| 21:22 |
<brion> |
synchronized extract2.php |
[production] |
| 21:18 |
<brion> |
ran sync-common-all |
[production] |
| 21:18 |
<brion> |
rsync messed up the php-1.5 directory to symlink translation. retrying as root |
[production] |
| 21:14 |
<brion> |
synchronized extract2.php |
[production] |
| 21:13 |
<brion> |
synchronized extract2.php |
[production] |
| 21:13 |
<atglenn> |
started copy of thumbnails to ms4, symlinks going in on ms1 (but no data removal yet) |
[production] |
| 21:11 |
<brion> |
synchronized live-1.5/extract2.php |
[production] |
| 21:10 |
<brion> |
synchronized live-1.5/robots.php |
[production] |
| 21:09 |
<brion> |
ran sync-common-all |
[production] |