|
2010-12-27
§
|
| 23:39 |
<Ryan_Lane> |
powercycling srv217 - it's dead |
[production] |
| 20:38 |
<Ryan_Lane> |
adding puppet schema to opendj on nova-controller.tesla |
[production] |
| 19:51 |
<Ryan_Lane> |
specifically chgrp'd all files to svnadm and added write permissions for the group |
[production] |
| 19:51 |
<Ryan_Lane> |
opened up file permissions on /srv/org/wikimedia/svn on formey so that ^demon can edit files |
[production] |
| 19:48 |
<Ryan_Lane> |
disabled selenium service on windows7-1, and launched selenium services manually, while logged in as the selenium user |
[production] |
| 18:04 |
<Ryan_Lane> |
installing puppet, puppet-el, puppetmaster, puppetmaster-passenger, and vim-puppet on nova-controller.tesla |
[production] |
| 17:56 |
<Ryan_Lane> |
adding new 1TB "archive" logical volume on storage3, formatting as ext3, and mounting at /archive |
[production] |
| 17:49 |
<Ryan_Lane> |
purging some bin logs on db9 to free up space |
[production] |
| 16:27 |
<Ryan_Lane> |
installing graphviz on formey via puppet for bug 26404 |
[production] |
| 09:21 |
<apergos> |
truncated log-all and log-index in /a/search/log on searchidx1 to get some space back |
[production] |
|
2010-12-25
§
|
| 23:39 |
<apergos> |
ran Platonides' script "purgeStaleMemcachedText.php" on s3/s7 projects, see /home/wikipedia/common/wmf-deployment/maintenance/purgeStaleMemcachedText.php on fenari (not added to the local branch yet) |
[production] |
| 21:14 |
<ariel> |
synchronized wmf-deployment/wmf-config/db.php 'read-only for s3/s7 while we sort out edit sync and cache issues' |
[production] |
| 20:17 |
<mark> |
New master bin-log db17-bin.003, position 406 |
[production] |
| 20:16 |
<mark> |
Switched master on s3, s7 from db27 to db17, just one slave db25 |
[production] |
| 20:16 |
<mark> |
synchronized php-1.5/wmf-config/db.php 'Going read-write on s3, s7' |
[production] |
| 20:13 |
<mark> |
synchronized php-1.5/wmf-config/db.php 'Switch to new master db17, slave db25 for s3/s7' |
[production] |
| 20:03 |
<mark> |
synchronized php-1.5/wmf-config/db.php |
[production] |
| 20:02 |
<mark> |
synchronized php-1.5/wmf-config/db.php 'Remove db39 from rotation' |
[production] |
| 19:58 |
<mark> |
Switching master to db17 |
[production] |
| 19:58 |
<mark> |
synchronized php-1.5/wmf-config/db.php |
[production] |
| 19:43 |
<Ryan_Lane> |
purging old bin logs on db27 |
[production] |
| 14:32 |
<apergos> |
turned off Indexes option on wikitech in apache config files, just like on the cluster |
[production] |
| 12:18 |
<apergos> |
powercycling searchidx1, can't get in through management console, nonresponsive to ssh, google says the weird ilom messages are a firmware issue, so... reset. |
[production] |
| 11:11 |
<rainman-sr> |
searchidx1 is down for about 5 days now, can someone take a look at it? Cannot ssh to it. Also search group dissapeared from ganglia. |
[production] |
| 00:22 |
<mark> |
Running apt-get dist-upgrade on mobile3 |
[production] |
| 00:21 |
<mark> |
Rebalanced LVS weights of mobile servers on lvs4 from 20/50/30 to 30/50/50 |
[production] |
| 00:06 |
<mark> |
powercycled mobile3 |
[production] |
|
2010-12-23
§
|
| 18:23 |
<apergos> |
added export of /data for download host to class in puppet |
[production] |
| 17:57 |
<Ryan_Lane> |
ignore that last message |
[production] |
| 17:57 |
<Cobi> |
TODO: Give Cobi root on servers |
[production] |
| 17:54 |
<Ryan_Lane> |
added vivek and aditya to nova-controller.tesla, nova-compute1, nova-compute2, and added them to sudoers |
[production] |
| 17:46 |
<apergos> |
added misc::download class for dataset2 in puppet (handles lighttpd for download.wm) |
[production] |
| 14:02 |
<hcatlin> |
deploying code update to m. |
[production] |
| 10:42 |
<tfinc> |
synchronized php-1.5/wmf-config/reporting-setup.php 'Flipping back to db10 since weve cleared up our space issues' |
[production] |
| 01:39 |
<apergos> |
dataset2 racked and installed (thanks rich, robh, mark). now copying data back off tridge, rsync in screen session from tridge as root. |
[production] |
| 00:40 |
<tfinc> |
synchronized php-1.5/extensions/ContributionTracking/ContributionTracking_body.php |
[production] |
| 00:40 |
<tfinc> |
synchronized php-1.5/extensions/ContributionTracking/ContributionTracking.i18n.php |
[production] |
| 00:23 |
<tomaszf_> |
updating IPN listener on grosley to r83 |
[production] |