|
2011-02-08
ยง
|
| 17:17 |
<RoanKattouw> |
Servers going back to normal, second deployment attempt failed. Same symptoms as the first: pcache miss rate in the 40s |
[production] |
| 17:12 |
<catrope> |
synchronizing Wikimedia installation... Revision: 81334: |
[production] |
| 17:12 |
<RoanKattouw> |
Rolling back 1.17 deployment again |
[production] |
| 17:04 |
<catrope> |
synchronized php-1.5/languages/messages/MessagesEo.php |
[production] |
| 16:56 |
<catrope> |
synchronized php-1.5/includes/parser/Preprocessor_DOM.php 'Attempting to shut up fatals for good' |
[production] |
| 16:52 |
<catrope> |
synchronized php-1.5/includes/parser/Preprocessor_DOM.php 'Another fatal workaround' |
[production] |
| 16:49 |
<catrope> |
synchronized php-1.5/includes/parser/Preprocessor_DOM.php 'Workaround for preprocessor fatal' |
[production] |
| 16:45 |
<catrope> |
synchronized php-1.5/includes/parser/Preprocessor_DOM.php 'Revert debug change' |
[production] |
| 16:44 |
<catrope> |
synchronized php-1.5/includes/parser/Preprocessor_DOM.php 'Debugging' |
[production] |
| 16:41 |
<catrope> |
synchronized php-1.5/wmf-config/secure.php 'Do not send load.php through bits on secure' |
[production] |
| 16:37 |
<catrope> |
synchronized php-1.5/extensions/SiteMatrix/SiteMatrix.php 'Fix class SiteMatrix now found fatal' |
[production] |
| 16:32 |
<catrope> |
synchronizing Wikimedia installation... Revision: 81755: |
[production] |
| 16:32 |
<RoanKattouw> |
Running scap |
[production] |
| 16:31 |
<RoanKattouw> |
Switching cluster to 1.17 |
[production] |
| 16:30 |
<RoanKattouw> |
Switching /home to 1.17 |
[production] |
| 16:29 |
<catrope> |
ran sync-common-all |
[production] |
| 16:29 |
<RoanKattouw> |
Running sync-common-all to push changes to php-1.17 |
[production] |
| 16:28 |
<RoanKattouw> |
Switching test back to 1.16 in preparation for sync-common-all |
[production] |
| 15:13 |
<RoanKattouw> |
test is now running 1.17wmf1 again |
[production] |
| 14:42 |
<RoanKattouw> |
Forgot to log that I put testwiki back on 1.16 about 30 mins ago |
[production] |
| 14:29 |
<catrope> |
synchronized php-1.5/languages/messages/MessagesJa.php 'Testing to see if this will fix the message caching issue' |
[production] |
| 14:00 |
<RoanKattouw> |
Moving directories on /home again, putting 1.17 on test so we can diagnose the cause of the overload |
[production] |
| 13:41 |
<catrope> |
synchronizing Wikimedia installation... Revision: 81334: |
[production] |
| 13:41 |
<RoanKattouw> |
Running scap |
[production] |
| 13:40 |
<RoanKattouw> |
1.17-rollback done |
[production] |
| 13:39 |
<RoanKattouw> |
Running 1.17-revert to move directories back cluster-wise |
[production] |
| 13:38 |
<RoanKattouw> |
Moving directories back on /home |
[production] |
| 13:38 |
<RoanKattouw> |
Rolling back 1.17 deployment |
[production] |
| 13:31 |
<catrope> |
synchronized php-1.5/includes/parser/Parser.php 'r81738, fixes registerWatcher fatal' |
[production] |
| 13:30 |
<catrope> |
synchronized php-1.5/extensions/NewUserMessage/NewUserMessage.class.php 'r81737, fixes fatal' |
[production] |
| 13:08 |
<catrope> |
synchronizing Wikimedia installation... Revision: 81732: |
[production] |
| 13:07 |
<RoanKattouw> |
Running scap to make sure the move takes effect properly, caches are purged, etc. |
[production] |
| 13:07 |
<RoanKattouw> |
Moving wmf-deployment to w-d.old and php-1.17 to wmf-deployment on all Apaches. Run 1.17-rollback to undo this |
[production] |
| 13:06 |
<RoanKattouw> |
Moving wmf-deployment to w-d.old and php-1.17 to wmf-deployment on /home |
[production] |
| 13:05 |
<catrope> |
ran sync-common-all |
[production] |
| 13:04 |
<RoanKattouw> |
Rerunning sync-common-all |
[production] |
| 13:03 |
<mark> |
Upgraded rsync to a version with higher tcp backlog on nfs2 |
[production] |
| 13:01 |
<RoanKattouw> |
sync-common-all failed for a few servers due to NFS problems. Mark is fixing it |
[production] |
| 12:58 |
<catrope> |
ran sync-common-all |
[production] |
| 12:57 |
<RoanKattouw> |
Moved old wmf-deployment/ and php-1.17/ back on /home , now syncing that out to the cluster |
[production] |
| 10:38 |
<mark> |
Power cycled amssq37, 43, 49 |
[production] |
| 10:34 |
<mark> |
Power cycled amssq35 |
[production] |
| 10:15 |
<mark> |
Deploying special case for test.wikipedia.org (=> srv193) on bits.pmtpa (varnish) |
[production] |
| 10:12 |
<catrope> |
synchronized live-1.5/load.php 'Revert failed hack' |
[production] |
| 10:01 |
<catrope> |
synchronized live-1.5/load.php 'Temp hack to make load.php work on test' |
[production] |
| 09:28 |
<root> |
ran sync-common-all |
[production] |
| 08:58 |
<root> |
ran sync-common-all |
[production] |
| 08:41 |
<mark> |
Unstuck SMS modem |
[production] |
| 07:50 |
<Tim> |
restarted apache on srv174 |
[production] |
| 07:22 |
<mark> |
Started Nagios bot |
[production] |