|
2011-04-20
§
|
| 23:10 |
<rainman-sr> |
search indexing is up and running again, might take a couple of days for things to catch up though |
[production] |
| 23:07 |
<rainman-sr> |
doing a slow restart of the whole search cluster (2 minute intervals between host restarts) |
[production] |
| 22:52 |
<rainman-sr> |
starting search indexer on searchidx2 |
[production] |
| 16:29 |
<mark> |
Squids finished rebuilding, removed iptables rules |
[production] |
| 16:21 |
<mark> |
Removed iptables rules on sq77/78 |
[production] |
| 16:09 |
<mark> |
Downgraded sq65 |
[production] |
| 16:06 |
<mark> |
removed iptables rule on sq66 |
[production] |
| 16:05 |
<Ryan_Lane> |
downgrading squid on knsq27, knsq28, knsq29, knsq30, knsq26, knsq25, knsq24 |
[production] |
| 16:02 |
<Ryan_Lane> |
downgrading squid on amssq31 |
[production] |
| 16:01 |
<Ryan_Lane> |
downgrading squid on amssq38 |
[production] |
| 16:01 |
<Ryan_Lane> |
downgrading squid on amssq39 |
[production] |
| 16:01 |
<Ryan_Lane> |
downgrading squid on amssq40 |
[production] |
| 16:01 |
<Ryan_Lane> |
downgrading squid on amssq41 |
[production] |
| 16:00 |
<Ryan_Lane> |
downgrading squid on amssq37 |
[production] |
| 15:59 |
<Ryan_Lane> |
downgrading squid on amssq36 |
[production] |
| 15:59 |
<Ryan_Lane> |
downgrading squid on amssq32 |
[production] |
| 15:59 |
<Ryan_Lane> |
downgrading squid on amssq33 |
[production] |
| 15:58 |
<Ryan_Lane> |
downgrading squid on amssq34 |
[production] |
| 15:58 |
<Ryan_Lane> |
downgrading squid on amssq35 |
[production] |
| 15:57 |
<mark> |
Downgrading Squid on sq71-78 |
[production] |
| 15:57 |
<Ryan_Lane> |
downgrading squid on amssq42 |
[production] |
| 15:56 |
<Ryan_Lane> |
downgrading squid on amssq43 |
[production] |
| 15:56 |
<Ryan_Lane> |
downgrading squid on amssq45 |
[production] |
| 15:54 |
<Ryan_Lane> |
downgrading squid onamssq45 |
[production] |
| 15:53 |
<Ryan_Lane> |
downgrading squid on knsq23 |
[production] |
| 15:33 |
<RobH> |
lowered maxclients more on singer, its going to slow down secure, but hopefully keep it online longer. |
[production] |
| 15:04 |
<Ryan_Lane> |
downgraded sq71 to squid-2.7.7 |
[production] |
| 14:51 |
<mark> |
Pulled squid 2.7.9 packages from the lucid-wikimedia reprepro APT repository, reinstated 2.7.7 |
[production] |
| 14:34 |
<mark> |
Downgrading squid on sq66 |
[production] |
| 14:15 |
<mark> |
Stopping squid on sq72 |
[production] |
| 14:14 |
<RobH> |
reduced singer apache maxclients from 400 to 200, hopefully will reduce singer crashes |
[production] |
| 14:09 |
<RobH> |
singer died, rebooting. |
[production] |
| 14:07 |
<RobH> |
secure, blogs on singer, bouncing due to singer cpu max |
[production] |
| 13:53 |
<RobH> |
singer threw invalid cert warning, cpu spike was occuring, restarting apache and load is normalizing |
[production] |
| 11:34 |
<catrope> |
ran sync-common-all |
[production] |
| 11:33 |
<RoanKattouw> |
Running sync-common-all to deploy r86464 |
[production] |
| 11:11 |
<catrope> |
synchronized php-1.17/includes/Exif.php 'Fix for bug 28615 (fatal on image description page)' |
[production] |
| 00:41 |
<awjr> |
deployment of r86441 to payments servers complete |
[production] |
| 00:37 |
<awjr> |
deploying changes to send full donor address information to PayflowPro (r86441) to all payments servers |
[production] |
| 00:31 |
<awjr> |
re-deploying changes to send full donor address information to PayflowPro on payments2.wikimedia.org (suspected cause of previous issue: local browser cacheing) |
[production] |
| 00:14 |
<midom> |
synchronized php-1.17/wmf-config/db.php |
[production] |