|
2009-12-10
§
|
| 16:05 |
<Rob> |
srv217 is powered on, but not in LVS pool and is just running dell diagnostics, please do not touch. |
[production] |
| 15:47 |
<Rob> |
ms8 required controller board assembly to be reseated, system LOM now can control it properly. |
[production] |
| 15:39 |
<domas> |
btw, side effect of my maintenance is that job queue was unclogged, yay. |
[production] |
| 15:19 |
<Rob> |
bringing down srv217 to run diagnostics |
[production] |
| 14:38 |
<domas> |
s1 master switch to db22-bin.000001:106 |
[production] |
| 14:37 |
<midom> |
synchronized php-1.5/wmf-config/db.php |
[production] |
| 14:36 |
<midom> |
synchronized php-1.5/wmf-config/db.php |
[production] |
| 14:36 |
<midom> |
synchronized php-1.5/wmf-config/db.php |
[production] |
| 14:29 |
<midom> |
synchronized php-1.5/wmf-config/db.php |
[production] |
| 14:15 |
<midom> |
synchronized php-1.5/wmf-config/db.php |
[production] |
| 14:12 |
<midom> |
synchronized php-1.5/wmf-config/db.php |
[production] |
| 12:41 |
<midom> |
synchronized php-1.5/wmf-config/db.php |
[production] |
| 12:33 |
<midom> |
synchronized php-1.5/wmf-config/db.php |
[production] |
| 11:38 |
<midom> |
synchronized php-1.5/wmf-config/db.php |
[production] |
| 11:32 |
<midom> |
synchronized php-1.5/wmf-config/db.php 'taking down one more server oh noes' |
[production] |
| 11:30 |
<midom> |
synchronized php-1.5/wmf-config/db.php |
[production] |
| 11:29 |
<midom> |
synchronized php-1.5/wmf-config/db.php |
[production] |
| 11:18 |
<midom> |
synchronized php-1.5/wmf-config/db.php |
[production] |
| 04:55 |
<atglenn> |
unpacking the last three months of incrementals on ms7 |
[production] |
| 02:19 |
<midom> |
synchronized php-1.5/wmf-config/db.php |
[production] |
| 00:34 |
<midom> |
synchronized php-1.5/wmf-config/db.php 'stress testing hehehehehe' |
[production] |
| 00:27 |
<midom> |
synchronized php-1.5/wmf-config/db.php |
[production] |
| 00:26 |
<tstarling> |
synchronized php-1.5/wmf-config/db.php 'increasing db12 load' |
[production] |
| 00:24 |
<Andrew> |
db12 recovered, db26 still barely coping with the load |
[production] |
| 00:17 |
<tstarling> |
synchronized php-1.5/wmf-config/db.php 'unloading general load from db12, excessive lag >800s' |
[production] |
| 00:09 |
<Andrew> |
db12 seems not to be replicating at all. |
[production] |
| 00:09 |
<Andrew> |
make that db12, not db22 |
[production] |
| 00:06 |
<Andrew> |
s1 lagged, db12 282s, db26 33s -- no slave servers up-to-date enough to serve pages |
[production] |
|
2009-12-09
§
|
| 23:53 |
<fred> |
synchronized php-1.5/wmf-config/CommonSettings.php 'Adding 2009_Notice_43-44-45 to tracking.' |
[production] |
| 23:06 |
<midom> |
synchronized php-1.5/wmf-config/db.php 'taking down one more server oh noes' |
[production] |
| 22:59 |
<Fred> |
fixed noc.wikimedia.org |
[production] |
| 21:08 |
<midom> |
synchronized php-1.5/wmf-config/db.php 'db14 offline' |
[production] |
| 20:54 |
<atglenn> |
open support call with sun for ms8, won't power up (no POST), even after resetting ILOM |
[production] |
| 17:28 |
<andrew> |
ran sync-common-all |
[production] |
| 17:18 |
<Andrew> |
Updating production LiquidThreads installations to new beta version, branch of current alpha. |
[production] |
| 17:08 |
<andrew> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 20253 Install Extension:NewUserMessage on strategy wiki' |
[production] |
| 13:39 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads_alpha/lqt.js 'Merge r59886' |
[production] |
| 13:38 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php 'Merge r59885' |
[production] |
| 13:08 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php 'Merge r59883' |
[production] |
| 13:05 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads_alpha/jquery/js2.combined.js 'merge r59882' |
[production] |
| 12:50 |
<andrew> |
ran sync-common-all |
[production] |
| 12:39 |
<Andrew> |
Updating LiquidThreads to trunk state, using sync-common-all to deploy to apaches (since scap crashes the cluster) |
[production] |
| 00:04 |
<RoanKattouw> |
Manually reallocating edits from [[User:Until It Sleeps]] to [[User:The Thing That Should Not Be]] on enwiki |
[production] |