|
2011-04-20
§
|
| 16:05 |
<Ryan_Lane> |
downgrading squid on knsq27, knsq28, knsq29, knsq30, knsq26, knsq25, knsq24 |
[production] |
| 16:02 |
<Ryan_Lane> |
downgrading squid on amssq31 |
[production] |
| 16:01 |
<Ryan_Lane> |
downgrading squid on amssq38 |
[production] |
| 16:01 |
<Ryan_Lane> |
downgrading squid on amssq39 |
[production] |
| 16:01 |
<Ryan_Lane> |
downgrading squid on amssq40 |
[production] |
| 16:01 |
<Ryan_Lane> |
downgrading squid on amssq41 |
[production] |
| 16:00 |
<Ryan_Lane> |
downgrading squid on amssq37 |
[production] |
| 15:59 |
<Ryan_Lane> |
downgrading squid on amssq36 |
[production] |
| 15:59 |
<Ryan_Lane> |
downgrading squid on amssq32 |
[production] |
| 15:59 |
<Ryan_Lane> |
downgrading squid on amssq33 |
[production] |
| 15:58 |
<Ryan_Lane> |
downgrading squid on amssq34 |
[production] |
| 15:58 |
<Ryan_Lane> |
downgrading squid on amssq35 |
[production] |
| 15:57 |
<mark> |
Downgrading Squid on sq71-78 |
[production] |
| 15:57 |
<Ryan_Lane> |
downgrading squid on amssq42 |
[production] |
| 15:56 |
<Ryan_Lane> |
downgrading squid on amssq43 |
[production] |
| 15:56 |
<Ryan_Lane> |
downgrading squid on amssq45 |
[production] |
| 15:54 |
<Ryan_Lane> |
downgrading squid onamssq45 |
[production] |
| 15:53 |
<Ryan_Lane> |
downgrading squid on knsq23 |
[production] |
| 15:33 |
<RobH> |
lowered maxclients more on singer, its going to slow down secure, but hopefully keep it online longer. |
[production] |
| 15:04 |
<Ryan_Lane> |
downgraded sq71 to squid-2.7.7 |
[production] |
| 14:51 |
<mark> |
Pulled squid 2.7.9 packages from the lucid-wikimedia reprepro APT repository, reinstated 2.7.7 |
[production] |
| 14:34 |
<mark> |
Downgrading squid on sq66 |
[production] |
| 14:15 |
<mark> |
Stopping squid on sq72 |
[production] |
| 14:14 |
<RobH> |
reduced singer apache maxclients from 400 to 200, hopefully will reduce singer crashes |
[production] |
| 14:09 |
<RobH> |
singer died, rebooting. |
[production] |
| 14:07 |
<RobH> |
secure, blogs on singer, bouncing due to singer cpu max |
[production] |
| 13:53 |
<RobH> |
singer threw invalid cert warning, cpu spike was occuring, restarting apache and load is normalizing |
[production] |
| 11:34 |
<catrope> |
ran sync-common-all |
[production] |
| 11:33 |
<RoanKattouw> |
Running sync-common-all to deploy r86464 |
[production] |
| 11:11 |
<catrope> |
synchronized php-1.17/includes/Exif.php 'Fix for bug 28615 (fatal on image description page)' |
[production] |
| 00:41 |
<awjr> |
deployment of r86441 to payments servers complete |
[production] |
| 00:37 |
<awjr> |
deploying changes to send full donor address information to PayflowPro (r86441) to all payments servers |
[production] |
| 00:31 |
<awjr> |
re-deploying changes to send full donor address information to PayflowPro on payments2.wikimedia.org (suspected cause of previous issue: local browser cacheing) |
[production] |
| 00:14 |
<midom> |
synchronized php-1.17/wmf-config/db.php |
[production] |
|
2011-04-19
§
|
| 23:24 |
<richcole1> |
rebooting db11 to check raid |
[production] |
| 22:57 |
<awjr> |
reverting to r84367 of deployment branch on payments2.wikimedia.org due to bad address information appearing in civicrm |
[production] |
| 22:53 |
<awjr> |
deploying changes to send full donor address information to PayflowPro payments2.wikimedia.org |
[production] |
| 22:20 |
<Ryan_Lane> |
virt1-4 installed |
[production] |
| 20:54 |
<Ryan_Lane> |
powercycling singer |
[production] |
| 20:53 |
<Ryan_Lane> |
restarting apache on spence |
[production] |
| 20:52 |
<mark> |
powercycled singer |
[production] |
| 20:35 |
<Ryan_Lane> |
powercycling srv196 |
[production] |
| 17:31 |
<Ryan_Lane> |
finished ams and kn text squids |
[production] |
| 17:17 |
<RobH> |
change live, etherpad still online and functional |
[production] |
| 17:16 |
<RobH> |
pushing change to hooper so it doesnt kill etherpad when other apache vhosts exist |
[production] |
| 17:16 |
<RobH> |
updated puppet entry for etherpad vhost |
[production] |
| 16:54 |
<Ryan_Lane> |
upgrading squid on all squid systems, slowly, over the course of the day |
[production] |
| 15:09 |
<mark> |
Killed all etherpad-user processes and started etherpad |
[production] |
| 15:09 |
<RoanKattouw> |
Removing data for the section edit link experiment from the clicktracking table. Data is backed up in /home/catrope/sel . This will delete 8.4M of the 9.6M rows in the clicktracking table |
[production] |
| 15:05 |
<RoanKattouw> |
Etherpad is broken, serving 500s. Have e-mailed Peter |
[production] |