|
2010-09-07
§
|
| 21:55 |
<RobH> |
sq69, sq70 both back online and in lvs pool |
[production] |
| 21:51 |
<Ryan_Lane> |
repooled sq68 in pybal bits config |
[production] |
| 21:49 |
<mark> |
Powercycled sq68 |
[production] |
| 21:48 |
<RobH> |
rebooting sq69, sq70 |
[production] |
| 21:41 |
<Ryan_Lane> |
set sq68 to false in pybal's bits configuration |
[production] |
| 19:56 |
<catrope> |
synchronized php-1.5/languages/messages/MessagesFrr.php 'r72545' |
[production] |
| 19:46 |
<apergos> |
restarted en wiki full history (with prefetch this time). |
[production] |
| 19:29 |
<robh> |
synchronized php-1.5/wmf-config/mc.php 'adding in srv258-289 into memcached spares, all test online' |
[production] |
| 18:47 |
<Ryan_Lane> |
split pdf1-3 and mobile1-3 away from misc cluster in ganglia; made two new clusters (pdf and mobile) |
[production] |
| 18:33 |
<apergos> |
starting xml full history dumps stage (paralellized) for en wik on snapshot2 in screen session from /backups-testing |
[production] |
| 17:38 |
<Ryan_Lane> |
changed password for Utilisateur:Bot de paille on frwiki |
[production] |
| 14:37 |
<mark> |
Powercycled mobile1 |
[production] |
| 05:13 |
<apergos> |
started meta-current phase of xml dumps for en wiki on snapshot2 in screen session |
[production] |
| 03:39 |
<Tim> |
installed packet loss monitor on locke |
[production] |
|
2010-09-03
§
|
| 23:44 |
<Ryan_Lane> |
added philippe to the wmffundraising alias |
[production] |
| 18:56 |
<RobH> |
srv123 pulled, wiping. also all memcached in updated pool test fine. |
[production] |
| 18:56 |
<robh> |
synchronized php-1.5/wmf-config/mc.php 'removing srv123 from memcached due to decommissioning' |
[production] |
| 18:38 |
<RobH> |
rt#54 db16 seems to not be throwing errors... installs ok, passed all VTS testing and memtesting. reinstalled the OS |
[production] |
| 11:45 |
<tomaszf> |
leaving lighttpd in non ipv6 mode till currently open sockets clear |
[production] |
| 10:39 |
<tomaszf> |
disabling ipv6 for lighttpd on dataset1 as its not working post upgrade |
[production] |
| 10:20 |
<tomaszf> |
upgrading to ubuntu 10.04 LTS on dataset1 |
[production] |
| 01:44 |
<Ryan_Lane> |
added pre-commit and post-commit hooks to puppet's svn to check syntax on pre-commit, and svn up on post-commit |
[production] |
|
2010-09-02
§
|
| 23:09 |
<Ryan_Lane> |
restarted mysql on db10 |
[production] |
| 23:09 |
<Ryan_Lane> |
set expire_logs_days to 14 on db10 |
[production] |
| 23:04 |
<Ryan_Lane> |
fixed password issue for debian-sys-maint on db10 |
[production] |
| 22:56 |
<Ryan_Lane> |
commented out skip_slave_start in the my.cnf on db10 |
[production] |
| 22:54 |
<Ryan_Lane> |
restarted db10 (specifically the mysql database) |
[production] |
| 22:54 |
<Ryan_Lane> |
restarted db10 |
[production] |
| 19:38 |
<tfinc> |
synchronized php-1.5/wmf-config/CommonSettings.php 'adding new thursday test banner to whitelist' |
[production] |
| 16:53 |
<RobH> |
reinstalling storage2 |
[production] |
| 16:09 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative/PrefSwitch/SpecialPrefSwitch.php 'Attempt to fix DB error' |
[production] |
| 16:06 |
<RobH> |
db16 is running VTS, DO NOT TOUCH THIS SERVER |
[production] |
| 15:51 |
<RobH> |
db34 unracked, going back for repair with Oracle Field Engineer. |
[production] |
| 15:02 |
<RobH> |
ignore errrors from db16, db28, db34. Today is sun server troubleshooting day. |
[production] |
| 14:58 |
<RobH> |
db28 is not in rotation, taking it down to continue hardware testing/repair |
[production] |
| 14:25 |
<RobH> |
srv206 back online and in cluster. |
[production] |
| 14:25 |
<robh> |
synchronized php-1.5/wmf-config/mc.php 'moved srv206 from down to spare' |
[production] |