|
2011-04-07
§
|
| 21:43 |
<notpeter> |
removed a silly check for hooper that I made and restarted nagios |
[production] |
| 19:06 |
<Ryan_Lane> |
switching openstack deb repo back to trunk, and upgrading packages on nova-controller, since we are likely to target cactus now |
[production] |
| 15:40 |
<mark> |
Restarted rsyncs |
[production] |
| 15:26 |
<mark> |
Created a test btrfs snapshot of /export on ms6 |
[production] |
| 15:12 |
<mark> |
Temporarily stopped the rsyncs on ms5 to test zfs send performance |
[production] |
| 13:56 |
<mark> |
Reenabled ms6 as backend on esams.upload squids |
[production] |
| 13:11 |
<apergos> |
replaced ms4 in fstab on fenari with ms5 so we have thumbs mounted there |
[production] |
| 12:08 |
<mark> |
Restarted rsyncs on ms5 |
[production] |
| 12:07 |
<apergos> |
nginx conf file change to "thumb" added to puppet |
[production] |
| 12:00 |
<mark> |
Removed the test snapshot on ms5 |
[production] |
| 11:47 |
<apergos> |
edited in place /etc/nginx/sites-available/thumbs and /export/thumbs/scripts/thumb-handler.php to make thumbs generated on the fly return 200. they were returning 404 |
[production] |
| 10:25 |
<catrope> |
synchronized php-1.17/thumb.php 'Attempted fix for wrong temp/thumb paths' |
[production] |
| 10:20 |
<apergos> |
(after reports from en vp that the search index has not been updated for over 4 days) |
[production] |
| 10:19 |
<apergos> |
restarting search indexer on searchidx1 |
[production] |
| 09:35 |
<catrope> |
synchronized php-1.17/includes/specials/SpecialUploadStash.php 'debugging' |
[production] |
| 09:08 |
<catrope> |
synchronized php-1.17/includes/specials/SpecialUploadStash.php 'r85612' |
[production] |
| 08:23 |
<catrope> |
synchronized php-1.17/extensions/UploadWizard/UploadWizard.php 'Fix fatal due to missing API module' |
[production] |
| 08:17 |
<catrope> |
ran sync-common-all |
[production] |
| 08:16 |
<RoanKattouw> |
I meant srv196, not srv193 |
[production] |
| 08:15 |
<RoanKattouw> |
Deploying UploadWizard for real this time, forgot to svn up first. sync-common-all then clearMessageBlobs.php |
[production] |
| 08:14 |
<RoanKattouw> |
Commenting out srv193 in mediawiki-installation node list because its timeouts take forever |
[production] |
| 08:10 |
<RoanKattouw> |
srv196 is not responding to SSH or syncs from fenari (they time out after a looong time) but Nagios says SSH is fine. Should be fixed or temporarily depooled |
[production] |
| 08:08 |
<RoanKattouw> |
Clearing message blobs |
[production] |
| 08:07 |
<catrope> |
ran sync-common-all |
[production] |
| 08:04 |
<RoanKattouw> |
Scap broke with sudo stuff AGAIN, running sync-common-all |
[production] |
| 08:01 |
<RoanKattouw> |
Running scap to deploy UploadWizard changes |
[production] |
| 07:11 |
<apergos> |
turned em off again, started seeing timeouts. bah |
[production] |
| 06:39 |
<apergos> |
and two more... |
[production] |
| 06:31 |
<apergos> |
restarted two of the 8 rsyncs on ms5, keeping an eye on them |
[production] |
| 01:31 |
<domas> |
added nobarrier to xfs mount options on db32 and db37 |
[production] |
|
2011-04-04
§
|
| 23:28 |
<Ryan_Lane> |
uploading ircecho package to lucid-wikimedia repo, for nagios irc bot |
[production] |
| 22:22 |
<Ryan_Lane> |
upgrading wikimedia-task-appserver package on srv281 |
[production] |
| 22:22 |
<Ryan_Lane> |
uploading new version of wikimedia-task-appserver to lucid-wikimedia repo; merges back in 1.17 changes that were missing |
[production] |
| 22:04 |
<RobH> |
updated noc robots entry in its apache config on fenari |
[production] |
| 21:58 |
<Ryan_Lane> |
srv281 is acting as a temporary scaling server for testing of lucid imagescalers, and to help with thumbs load. |
[production] |
| 21:27 |
<Ryan_Lane> |
depooling srv281 from appservers |
[production] |
| 21:21 |
<Ryan_Lane> |
syncing apaches to get configuration pushed to srv281 |
[production] |
| 21:17 |
<Ryan_Lane> |
rebooting srv281 |
[production] |
| 21:01 |
<Ryan_Lane> |
adding srv281 to rendering cluster in pybal via fenari |
[production] |