|
2010-12-28
§
|
| 21:54 |
<Ryan_Lane> |
destroying 1TB logical volume on storage3 |
[production] |
| 21:54 |
<Ryan_Lane> |
umounting 1TB logical volume from /archive on storage3 and mounting 4TB xfs volume (data was moved prior to this) |
[production] |
| 19:09 |
<Ryan_Lane> |
moving archive data from /archive to /archive1 on storage3, will replace mount afterwards |
[production] |
| 19:08 |
<Ryan_Lane> |
creating new 4TB logical volume "archive1" on storage3, and putting an xfs filesystem on it |
[production] |
| 16:34 |
<mark> |
Defined special disk space check (with SMS notification) for mysql core databases |
[production] |
| 15:52 |
<midom> |
synchronized php-1.5/wmf-config/db.php 'adding db11, db18 and db39' |
[production] |
| 14:30 |
<midom> |
synchronized php-1.5/wmf-config/db.php 'enabling db37' |
[production] |
| 13:16 |
<domas> |
resumed replication on db11, db18, db37 and db39 |
[production] |
| 00:19 |
<Ryan_Lane> |
restarting apache on srv217 - puppet updated its apache2.conf file, but didn't restart it |
[production] |
| 00:12 |
<Ryan_Lane> |
powercycling amssq53 - it's dead |
[production] |
| 00:09 |
<Ryan_Lane> |
powercycling sq76 - it's dead |
[production] |
| 00:08 |
<Ryan_Lane> |
knsq14 isn't coming back up - may be broken |
[production] |
| 00:08 |
<Ryan_Lane> |
powercycling knsq24 - it's dead |
[production] |
| 00:02 |
<Ryan_Lane> |
powercycling knsq14 - it's dead |
[production] |
|
2010-12-27
§
|
| 23:54 |
<Ryan_Lane> |
starting apache on srv217, running puppet to ensure code is synched |
[production] |
| 23:52 |
<Ryan_Lane> |
restarting apache on srv231 |
[production] |
| 23:50 |
<laner> |
synchronized php-1.5/wmf-config/mc.php 'Fixing IP address for srv226' |
[production] |
| 23:48 |
<laner> |
synchronized php-1.5/wmf-config/mc.php 'Fixing IP address for srv227' |
[production] |
| 23:39 |
<Ryan_Lane> |
powercycling srv217 - it's dead |
[production] |
| 20:38 |
<Ryan_Lane> |
adding puppet schema to opendj on nova-controller.tesla |
[production] |
| 19:51 |
<Ryan_Lane> |
specifically chgrp'd all files to svnadm and added write permissions for the group |
[production] |
| 19:51 |
<Ryan_Lane> |
opened up file permissions on /srv/org/wikimedia/svn on formey so that ^demon can edit files |
[production] |
| 19:48 |
<Ryan_Lane> |
disabled selenium service on windows7-1, and launched selenium services manually, while logged in as the selenium user |
[production] |
| 18:04 |
<Ryan_Lane> |
installing puppet, puppet-el, puppetmaster, puppetmaster-passenger, and vim-puppet on nova-controller.tesla |
[production] |
| 17:56 |
<Ryan_Lane> |
adding new 1TB "archive" logical volume on storage3, formatting as ext3, and mounting at /archive |
[production] |
| 17:49 |
<Ryan_Lane> |
purging some bin logs on db9 to free up space |
[production] |
| 16:27 |
<Ryan_Lane> |
installing graphviz on formey via puppet for bug 26404 |
[production] |
| 09:21 |
<apergos> |
truncated log-all and log-index in /a/search/log on searchidx1 to get some space back |
[production] |
|
2010-12-25
§
|
| 23:39 |
<apergos> |
ran Platonides' script "purgeStaleMemcachedText.php" on s3/s7 projects, see /home/wikipedia/common/wmf-deployment/maintenance/purgeStaleMemcachedText.php on fenari (not added to the local branch yet) |
[production] |
| 21:14 |
<ariel> |
synchronized wmf-deployment/wmf-config/db.php 'read-only for s3/s7 while we sort out edit sync and cache issues' |
[production] |
| 20:17 |
<mark> |
New master bin-log db17-bin.003, position 406 |
[production] |
| 20:16 |
<mark> |
Switched master on s3, s7 from db27 to db17, just one slave db25 |
[production] |
| 20:16 |
<mark> |
synchronized php-1.5/wmf-config/db.php 'Going read-write on s3, s7' |
[production] |
| 20:13 |
<mark> |
synchronized php-1.5/wmf-config/db.php 'Switch to new master db17, slave db25 for s3/s7' |
[production] |
| 20:03 |
<mark> |
synchronized php-1.5/wmf-config/db.php |
[production] |
| 20:02 |
<mark> |
synchronized php-1.5/wmf-config/db.php 'Remove db39 from rotation' |
[production] |
| 19:58 |
<mark> |
Switching master to db17 |
[production] |
| 19:58 |
<mark> |
synchronized php-1.5/wmf-config/db.php |
[production] |
| 19:43 |
<Ryan_Lane> |
purging old bin logs on db27 |
[production] |
| 14:32 |
<apergos> |
turned off Indexes option on wikitech in apache config files, just like on the cluster |
[production] |
| 12:18 |
<apergos> |
powercycling searchidx1, can't get in through management console, nonresponsive to ssh, google says the weird ilom messages are a firmware issue, so... reset. |
[production] |
| 11:11 |
<rainman-sr> |
searchidx1 is down for about 5 days now, can someone take a look at it? Cannot ssh to it. Also search group dissapeared from ganglia. |
[production] |
| 00:22 |
<mark> |
Running apt-get dist-upgrade on mobile3 |
[production] |
| 00:21 |
<mark> |
Rebalanced LVS weights of mobile servers on lvs4 from 20/50/30 to 30/50/50 |
[production] |
| 00:06 |
<mark> |
powercycled mobile3 |
[production] |