|
2012-07-06
§
|
| 15:21 |
<cmjohnson1> |
ms-be5 removing disk0 to replace with good disk |
[production] |
| 14:38 |
<RobHalsell> |
authdns-update to correct typo in mgmt dns entry |
[production] |
| 13:22 |
<mark> |
Inserted new pybal 1.04 package in the precise-wikimedia APT repository, and upgraded all precise LVS servers |
[production] |
| 11:31 |
<tstarling> |
Finished syncing Wikimedia installation... : |
[production] |
| 11:25 |
<tstarling> |
Started syncing Wikimedia installation... : |
[production] |
| 11:20 |
<Tim> |
running scap to delete docroot files per I48502d90 and Ie3afd137 |
[production] |
| 08:58 |
<mutante> |
dist-upgrading (unused) db10xx servers |
[production] |
| 07:43 |
<Tim> |
also trusted-xff.phps |
[production] |
| 07:40 |
<Tim> |
removing some untracked junk from /home/wikipedia/common |
[production] |
| 07:29 |
<mutante> |
continue to restart and upgrade downed mw10xx servers |
[production] |
| 02:26 |
<LocalisationUpdate> |
completed (1.20wmf6) at Fri Jul 6 02:26:22 UTC 2012 |
[production] |
| 01:35 |
<binasher> |
updated squid redirector to cover wiki(quotes|books|versity) |
[production] |
| 00:13 |
<Ryan_Lane> |
restarting gerrit |
[production] |
| 00:09 |
<Ryan_Lane> |
force running puppet on manganese, it'll restart gerrit |
[production] |
|
2012-07-05
§
|
| 23:54 |
<Ryan_Lane> |
force running puppet on formey and manganese, since a config change is involved, it's going to restart |
[production] |
| 23:45 |
<Ryan_Lane> |
restarting gerrit on manganese |
[production] |
| 23:38 |
<Ryan_Lane> |
upgrading gerrit on formey |
[production] |
| 23:21 |
<Ryan_Lane> |
updating database for gerrit |
[production] |
| 23:09 |
<Ryan_Lane> |
upgrading gerrit on manganese |
[production] |
| 23:08 |
<Ryan_Lane> |
stopping gerrit on manganese and disabling puppet |
[production] |
| 23:07 |
<Ryan_Lane> |
stopping gerrit on formey and disabling puppet |
[production] |
| 22:47 |
<preilly> |
synchronized wmf-config/mobile.php 'add subdomain check' |
[production] |
| 22:34 |
<reedy> |
synchronized wmf-config/ |
[production] |
| 22:17 |
<reedy> |
synchronized wmf-config/ 'Tidying config for randomrootpage' |
[production] |
| 21:56 |
<reedy> |
synchronized wmf-config/ 'Strike 2' |
[production] |
| 21:50 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'revert' |
[production] |
| 21:48 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Random root page enabled everywhere but wikipedias' |
[production] |
| 21:47 |
<kaldari> |
Finished syncing Wikimedia installation... : |
[production] |
| 21:34 |
<kaldari> |
Started syncing Wikimedia installation... : |
[production] |
| 21:18 |
<kaldari> |
synchronized wmf-config/InitialiseSettings.php 'Bumping AFTv5 lottery percentage to 20% of en.wiki' |
[production] |
| 21:15 |
<LeslieCarr> |
powercycling unresponsive mw1116 |
[production] |
| 21:09 |
<maplebed> |
added new ms-be pmtpa hosts to DNS |
[production] |
| 20:03 |
<preilly> |
synchronized php-1.20wmf6/extensions/MobileFrontend 'fix cache issue with token' |
[production] |
| 19:14 |
<Jeff_Green> |
*somebody* purged binary logs on blondel |
[production] |
| 18:58 |
<preilly> |
synchronized php-1.20wmf6/extensions/MobileFrontend 'fix cache issue' |
[production] |
| 18:54 |
<mark> |
Upgraded pybal on all precise LVS servers |
[production] |
| 18:43 |
<mark> |
Built new pybal_1.03 package and inserted it into the precise-wikimedia APT repository |
[production] |
| 17:41 |
<mutante> |
powercycling db1048, mw1136, mw1046 |
[production] |
| 17:32 |
<midom> |
synchronized wmf-config/InitialiseSettings.php 'reenabling db40' |
[production] |
| 17:30 |
<mutante> |
more powercycling and upgrading: mw1036, mw1134, mw1043 .. |
[production] |
| 17:26 |
<mutante> |
powercycling db1009,db1010 |
[production] |
| 17:22 |
<mutante> |
powercycling db1027,db1028 |
[production] |
| 17:14 |
<mutante> |
powercycling db1013 |
[production] |
| 16:50 |
<mutante> |
powercycling downed db1015 |
[production] |
| 16:44 |
<mutante> |
powercycling and upgrading more mw10xx servers, 1017,1023,1025 ... |
[production] |
| 16:27 |
<mutante> |
mw1002, mw1007,mw1009,mw1011 - crashed,powercycling,dist-upgrading+kernel,reboot |
[production] |
| 16:18 |
<mutante> |
powercycling mw1002 |
[production] |
| 15:37 |
<mutante> |
argon back up with new kernel,mysql,grub,.. looks happy afaict |
[production] |
| 15:33 |
<mutante> |
argon (limesurvey) fscked, dist-upgrading |
[production] |
| 15:28 |
<mutante> |
powercycling argon |
[production] |