|
2010-08-26
§
|
| 19:43 |
<tfinc> |
synchronized php-1.5/extensions/CentralNotice/CentralNotice.db.php 'CN Phase1 Deployment' |
[production] |
| 19:18 |
<Ryan_Lane> |
reconfiguring tesla's backend networks. Selenium grid nodes may be down periodically |
[production] |
| 18:57 |
<Fred> |
rebooting pdf1. Hard down. |
[production] |
| 18:45 |
<catrope> |
synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix' |
[production] |
| 18:44 |
<catrope> |
synchronized php-1.5/skins/common/wikibits.js 'r71723' |
[production] |
| 18:43 |
<RoanKattouw> |
Commented out srv217 in mediawiki-installation node list cause it's down |
[production] |
| 16:28 |
<RoanKattouw> |
Starting six instances of convertUserOptions.php , one for each DB cluster, on hume |
[production] |
| 11:17 |
<mark> |
Discarding mail to bugzilla-daemon@wikimedia.org on mchenry |
[production] |
|
2010-08-24
§
|
| 19:18 |
<Fred> |
puppetized Ryan Lane's account and privileges. |
[production] |
| 18:53 |
<Fred> |
mobile3 has been deployed and is now in rotation. |
[production] |
| 18:01 |
<Fred> |
adding mobile3.w.o to DNS |
[production] |
| 17:43 |
<Fred> |
starting deployment of Mobile3 |
[production] |
| 17:39 |
<RobH> |
racked mobile3, handing off to fred to install |
[production] |
| 15:34 |
<mark> |
Reran Nagios sync script to get MySQL monitoring back |
[production] |
| 13:07 |
<mark> |
Removed unused IPs in the squid+lvs and public services subnets from DNS |
[production] |
| 08:51 |
<tstarling> |
synchronized php-1.5/includes/media/Bitmap.php 'r71547' |
[production] |
| 08:19 |
<tstarling> |
synchronized php-1.5/includes/media/Bitmap.php 'live patch to chain -resize with -thumbnail to fix bug 24824' |
[production] |
|
2010-08-23
§
|
| 13:19 |
<Tim> |
on ms4: fastcgi re-enabled |
[production] |
| 13:06 |
<mark> |
Finished removing all 2009 daily zfs snapshots on ms4 |
[production] |
| 12:08 |
<Tim> |
on ms4: disabling fastcgi to try to get basic cached service |
[production] |
| 12:03 |
<Tim> |
on ms4: increasing max thread count to 256 |
[production] |
| 12:00 |
<mark> |
Removing all the 2009-10 daily zfs snapshots on ms4 |
[production] |
| 11:46 |
<mark> |
Removing all the 2009-09 daily zfs snapshots on ms4 |
[production] |
| 11:36 |
<mark> |
Removing all the 2009-08 daily zfs snapshots on ms4 |
[production] |
| 11:33 |
<mark> |
Removed oldest daily thumbs zfs snapshot on ms4 |
[production] |
| 10:43 |
<Tim> |
on ms4: restarting webserver7 with fcgi re-enabled, reduced thread pool count to 4 |
[production] |
| 10:42 |
<mark> |
Added swap2 to /etc/vfstab on ms4 |
[production] |
| 10:36 |
<mark> |
zfs create -V 4gb rpool/swap2; swap -a /dev/zol/dsk/rpool/swap2 (on ms4) |
[production] |
| 10:11 |
<Tim> |
trying to start webserver7 on ms4, will see if it crashes it again |
[production] |
| 09:56 |
<mark> |
svcadm disable puppetd on ms4 |
[production] |
| 09:11 |
<Tim> |
ms4 back up, after some mucking around with /etc/vfstab |
[production] |
| 08:48 |
<Tim> |
ms4 timeout on http, squid serving "cannot forward", will reboot |
[production] |
| 08:44 |
<Tim> |
ms4 not responding to ssh, giving "stub start error" on http, trying serial console, very slow |
[production] |
| 03:05 |
<domas> |
'ps -ef | grep php-cgi | awk '$3==1 { print $2 }' | xargs kill; rm /tmp/https-ms4-5351d5c9/stub.pid' to recover from ms4 fastcgi death, not sure what are the causes yet |
[production] |
| 00:50 |
<midom> |
synchronized php-1.5/wmf-config/db.php 'oops, wrong cluster' |
[production] |
| 00:49 |
<midom> |
synchronized php-1.5/wmf-config/db.php |
[production] |