|
2012-05-17
ยง
|
| 22:01 |
<binasher> |
migrating centralauth.spoofuser to innodb via osc (13.5mil rows) |
[production] |
| 22:00 |
<binasher> |
migrated centralauth.global_group to innodb |
[production] |
| 21:53 |
<maplebed> |
reverted mobile change from this morning - testing completed. |
[production] |
| 21:42 |
<binasher> |
es1004 is replicating again |
[production] |
| 21:39 |
<binasher> |
resumed replication to es1002 |
[production] |
| 21:21 |
<catrope> |
synchronized php-1.20wmf3/extensions/ArticleFeedbackv5/ArticleFeedbackv5.php 'Deploy 24ddcdf507e615b1942147654ccde1bdc4ea4bfa' |
[production] |
| 21:20 |
<catrope> |
synchronized php-1.20wmf2/extensions/ArticleFeedbackv5/ArticleFeedbackv5.php 'Deploy 24ddcdf507e615b1942147654ccde1bdc4ea4bfa' |
[production] |
| 21:09 |
<aaron> |
synchronized wmf-config/swift.php '+commonswiki' |
[production] |
| 20:58 |
<aaron> |
synchronized wmf-config/swift.php 'revert change to itwiki' |
[production] |
| 20:57 |
<Jeff_Green> |
several package updates on payments* and silicon |
[production] |
| 20:56 |
<aaron> |
synchronized wmf-config/swift.php '+itwiki' |
[production] |
| 20:40 |
<aaron> |
synchronized wmf-config/swift.php |
[production] |
| 20:34 |
<maplebed> |
deployed change to swift and mediawiki for MW to write thumbnails to swift instead of rewrite.py with aaron |
[production] |
| 20:32 |
<maplebed> |
deployed parallel thumbnail purging for test, test2, and mediawiki with aaron |
[production] |
| 20:25 |
<aaron> |
synchronized wmf-config/swift.php 'Enabled thumb copy hook for testwikis and mw.org' |
[production] |
| 20:14 |
<binasher> |
completed securepoll_votes.vote_ip and all ipv6 schema migration |
[production] |
| 20:10 |
<aaron> |
synchronized wmf-config/CommonSettings.php |
[production] |
| 20:09 |
<aaron> |
synchronized wmf-config/swift.php |
[production] |
| 20:03 |
<aaron> |
synchronized wmf-config/swift.php 'Enabling new purge hook on testwikis again.' |
[production] |
| 20:03 |
<binasher> |
running securepoll_votes.vote_ip schema migration on s1 |
[production] |
| 20:01 |
<binasher> |
running securepoll_votes.vote_ip schema migration on all s2 dbs |
[production] |
| 19:19 |
<binasher> |
running securepoll_votes.vote_ip schema migration on all s4 + s3 dbs |
[production] |
| 19:17 |
<binasher> |
running securepoll_votes.vote_ip schema migration on all s5 dbs |
[production] |
| 19:16 |
<binasher> |
running securepoll_votes.vote_ip schema migration on all s6 dbs |
[production] |
| 19:02 |
<binasher> |
running securepoll_votes.vote_ip schema migration on all s7 dbs |
[production] |
| 18:49 |
<binasher> |
syncing cluster23 tables from es1002 to es1004 |
[production] |
| 18:46 |
<binasher> |
stopped replication on es1002 |
[production] |
| 18:44 |
<notpeter> |
restarting puppet on brewster |
[production] |
| 18:39 |
<catrope> |
synchronizing Wikimedia installation... : ArticleFeedbackv5 updates |
[production] |
| 18:13 |
<aaron> |
synchronized wmf-config/swift.php |
[production] |
| 18:08 |
<aaron> |
synchronized php-1.20wmf3/includes/filerepo 'deployed 103efda39dd57bc22898bd0e69932982c1cfd588' |
[production] |
| 18:00 |
<Jeff_Green> |
shutting down grosley for disk and RAM upgrades |
[production] |
| 17:42 |
<notpeter> |
temporarily turning off puppet on brewster for preseed hackz |
[production] |
| 17:20 |
<maplebed> |
flushing the mobile cache post-deploy |
[production] |
| 17:17 |
<maplebed> |
deploying config change to mobile - more zero IP addresses. gerrit [[rev:7867|r7867]] |
[production] |
| 15:31 |
<dzahn> |
synchronized php/cache/interwiki.cdb 'Updating interwiki cache' |
[production] |
| 15:30 |
<mutante> |
sync-common-file interwiki.cdb |
[production] |
| 15:30 |
<mutante> |
creating fresh interwiki.cdb from dumpInterwiki.php |
[production] |
| 15:30 |
<Jeff_Green> |
adding DNS records to wikimedia.org for RT #2960 |
[production] |
| 14:22 |
<mutante> |
adding gerrit project analytics/udplog parent analytics |
[production] |
| 13:44 |
<cmjohnson1> |
shutting down bellin for troubleshooting |
[production] |
| 09:04 |
<hashar> |
Site outage was due to our custom wfLogXFF() which uses wfErrorLog(). $wmfUdp2logDest not being global there, caused exception to be shown. |
[production] |
| 08:59 |
<hashar> |
Broken the cluster by having an invalid global set |
[production] |
| 08:58 |
<hashar> |
synchronized wmf-config/CommonSettings.php |
[production] |
| 08:47 |
<hashar> |
synchronizing Wikimedia installation... : |
[production] |
| 08:44 |
<hashar> |
running scap to apply https://gerrit.wikimedia.org/r/7702 |
[production] |
| 08:41 |
<hashar> |
Deploying https://gerrit.wikimedia.org/r/7702 which abstract out the udp2log destination |
[production] |
| 08:15 |
<hashar> |
WMFLabs seems to have recovered now |
[production] |
| 06:50 |
<hashar> |
WMFLabs dieing out, I/O latency raised constantly over the last 2 hours and eventually lead to situation where system (via ssh) is not usable anymore |
[production] |
| 03:41 |
<asher> |
synchronized wmf-config/db.php 'returning db12 and db46' |
[production] |