|
2009-07-17
§
|
| 18:03 |
<Fred> |
spun a couple more apache server into image scalers: srv219..srv224. |
[production] |
| 17:28 |
<rainman-sr> |
putting new location of initialisesettings to lsearch-global-2.1.conf so the incremental updater works again |
[production] |
| 17:20 |
<Fred> |
srv224 is now an image_scaler. Adjusted on lvs3, ganglia and dsh's node_list. |
[production] |
| 17:14 |
<Fred> |
db20 back online |
[production] |
| 16:50 |
<Fred> |
rebooting db20 as it is in a "state" |
[production] |
| 16:45 |
<brion> |
looks like we've lost internal /home NFS, which makes some of our internal services very unhappy. investigating... |
[production] |
| 16:43 |
<brion> |
ganglia out. |
[production] |
| 13:44 |
<apergos1> |
doing next round of removals on ms1 (/export/upload/wikipedia/en/thumb/2) to keep ahead of the game |
[production] |
| 04:15 |
<apergos> |
starting removal of /export/upload/wikipedia/en/thumb/1 on ms1 (moved away and symlink to ms4 done already) for more space |
[production] |
| 03:54 |
<brion> |
synchronized wmf-deployment/wmf-config/CommonSettings.php 'Disabling sitenotice from maintenance' |
[production] |
| 03:29 |
<brion> |
reenabling uploads & image deletion/undeletion |
[production] |
| 03:29 |
<brion> |
synchronized wmf-deployment/wmf-config/CommonSettings.php |
[production] |
| 03:28 |
<brion> |
remounting ms1 on apaches |
[production] |
| 00:49 |
<atglenn> |
only about 1gb gain on each so doing all of /export/upload/wikipedia/en/thumb/0 |
[production] |
| 00:39 |
<atglenn> |
removing more directories in /export/upload/wikipedia/en/thumb/0 on ms1 and replacing with symlinks to ms4 |
[production] |
| 00:30 |
<brion> |
synchronized wmf-deployment/includes/specials/SpecialUpload.php |
[production] |
| 00:30 |
<brion> |
synchronized wmf-deployment/includes/ImagePage.php |
[production] |
| 00:27 |
<brion> |
synchronized wmf-deployment/wmf-config/CommonSettings.php |
[production] |
| 00:20 |
<brion> |
temporarily disabling image delete/rename during maintenance |
[production] |
|
2009-07-16
§
|
| 23:56 |
<fvassard> |
synchronized php-1.5/wmf-config/CommonSettings.php 'Disabling uploads and setting captcha to not-fancy.' |
[production] |
| 23:29 |
<atglenn> |
removing the images in /export/upload/wikipedia/en/thumb/0/00 on ms1 (real dir is a symlink to ms4) to get back some space |
[production] |
| 22:51 |
<atglenn> |
sym link back in place, let's see what happens |
[production] |
| 22:47 |
<atglenn> |
reverting temporarily while we resolve mount issues for the ms4 share |
[production] |
| 22:40 |
<atglenn> |
...whether the image scalers will fall over if we force them to do (some) regeneration. |
[production] |
| 22:37 |
<atglenn> |
on ms1, /export/upload/wikipedia/en/thumb/0/00 symlinked to (shared from ms4) /mnt/thumbs/wikipedia/en/thumb/0/00 to test |
[production] |
| 21:29 |
<brion> |
robots.php for robots.txt generation now also working. yay! |
[production] |
| 21:28 |
<brion> |
synchronized live-1.5/robots.php |
[production] |
| 21:28 |
<brion> |
extract2.php now fixed up for new deployment; portal pages ok (www.wikipedia.org) |
[production] |
| 21:27 |
<brion> |
synchronized live-1.5/robots.php |
[production] |
| 21:26 |
<brion> |
synchronized extract2.php |
[production] |
| 21:26 |
<brion> |
synchronized extract2.php |
[production] |
| 21:22 |
<brion> |
synchronized extract2.php |
[production] |
| 21:18 |
<brion> |
ran sync-common-all |
[production] |
| 21:18 |
<brion> |
rsync messed up the php-1.5 directory to symlink translation. retrying as root |
[production] |
| 21:14 |
<brion> |
synchronized extract2.php |
[production] |
| 21:13 |
<brion> |
synchronized extract2.php |
[production] |
| 21:13 |
<atglenn> |
started copy of thumbnails to ms4, symlinks going in on ms1 (but no data removal yet) |
[production] |
| 21:11 |
<brion> |
synchronized live-1.5/extract2.php |
[production] |
| 21:10 |
<brion> |
synchronized live-1.5/robots.php |
[production] |
| 21:09 |
<brion> |
ran sync-common-all |
[production] |
| 21:08 |
<brion> |
attempting to replace the old php-1.5 dir with wmf-deployment symlink |
[production] |
| 21:02 |
<brion> |
synchronized wmf-deployment/wmf-config/InitialiseSettings.php 'I think touching the new master InitialiseSettings will fix it' |
[production] |
| 21:01 |
<brion> |
synchronized wmf-deployment/includes/GlobalFunctions.php 'mkdir error trackdown hack' |
[production] |
| 20:54 |
<brion> |
synchronized wmf-deployment/wmf-config/missing.php |
[production] |
| 20:52 |
<brion> |
synchronized wmf-deployment/wmf-config/CommonSettings.php |
[production] |
| 20:52 |
<brion> |
synchronized wmf-deployment/wmf-config/reporting-setup.php |
[production] |
| 20:48 |
<brion> |
switching all sites to wmf-deployment branch |
[production] |
| 20:48 |
<brion> |
synchronized live-1.5/MWVersion.php |
[production] |
| 19:06 |
<Tim> |
copying ExtensionDistributor stuff to ms4:/export/ext-dist, from root screen on ms1 |
[production] |
| 19:01 |
<brion> |
Now running test.wikipedia.org, www.mediawiki.org, and meta.wikimedia.org on new deployment checkout |
[production] |