751-800 of 10000 results (14ms)
2012-05-17 ยง
22:01 <binasher> migrating centralauth.spoofuser to innodb via osc (13.5mil rows) [production]
22:00 <binasher> migrated centralauth.global_group to innodb [production]
21:53 <maplebed> reverted mobile change from this morning - testing completed. [production]
21:42 <binasher> es1004 is replicating again [production]
21:39 <binasher> resumed replication to es1002 [production]
21:21 <catrope> synchronized php-1.20wmf3/extensions/ArticleFeedbackv5/ArticleFeedbackv5.php 'Deploy 24ddcdf507e615b1942147654ccde1bdc4ea4bfa' [production]
21:20 <catrope> synchronized php-1.20wmf2/extensions/ArticleFeedbackv5/ArticleFeedbackv5.php 'Deploy 24ddcdf507e615b1942147654ccde1bdc4ea4bfa' [production]
21:09 <aaron> synchronized wmf-config/swift.php '+commonswiki' [production]
20:58 <aaron> synchronized wmf-config/swift.php 'revert change to itwiki' [production]
20:57 <Jeff_Green> several package updates on payments* and silicon [production]
20:56 <aaron> synchronized wmf-config/swift.php '+itwiki' [production]
20:40 <aaron> synchronized wmf-config/swift.php [production]
20:34 <maplebed> deployed change to swift and mediawiki for MW to write thumbnails to swift instead of rewrite.py with aaron [production]
20:32 <maplebed> deployed parallel thumbnail purging for test, test2, and mediawiki with aaron [production]
20:25 <aaron> synchronized wmf-config/swift.php 'Enabled thumb copy hook for testwikis and mw.org' [production]
20:14 <binasher> completed securepoll_votes.vote_ip and all ipv6 schema migration [production]
20:10 <aaron> synchronized wmf-config/CommonSettings.php [production]
20:09 <aaron> synchronized wmf-config/swift.php [production]
20:03 <aaron> synchronized wmf-config/swift.php 'Enabling new purge hook on testwikis again.' [production]
20:03 <binasher> running securepoll_votes.vote_ip schema migration on s1 [production]
20:01 <binasher> running securepoll_votes.vote_ip schema migration on all s2 dbs [production]
19:19 <binasher> running securepoll_votes.vote_ip schema migration on all s4 + s3 dbs [production]
19:17 <binasher> running securepoll_votes.vote_ip schema migration on all s5 dbs [production]
19:16 <binasher> running securepoll_votes.vote_ip schema migration on all s6 dbs [production]
19:02 <binasher> running securepoll_votes.vote_ip schema migration on all s7 dbs [production]
18:49 <binasher> syncing cluster23 tables from es1002 to es1004 [production]
18:46 <binasher> stopped replication on es1002 [production]
18:44 <notpeter> restarting puppet on brewster [production]
18:39 <catrope> synchronizing Wikimedia installation... : ArticleFeedbackv5 updates [production]
18:13 <aaron> synchronized wmf-config/swift.php [production]
18:08 <aaron> synchronized php-1.20wmf3/includes/filerepo 'deployed 103efda39dd57bc22898bd0e69932982c1cfd588' [production]
18:00 <Jeff_Green> shutting down grosley for disk and RAM upgrades [production]
17:42 <notpeter> temporarily turning off puppet on brewster for preseed hackz [production]
17:20 <maplebed> flushing the mobile cache post-deploy [production]
17:17 <maplebed> deploying config change to mobile - more zero IP addresses. gerrit [[rev:7867|r7867]] [production]
15:31 <dzahn> synchronized php/cache/interwiki.cdb 'Updating interwiki cache' [production]
15:30 <mutante> sync-common-file interwiki.cdb [production]
15:30 <mutante> creating fresh interwiki.cdb from dumpInterwiki.php [production]
15:30 <Jeff_Green> adding DNS records to wikimedia.org for RT #2960 [production]
14:22 <mutante> adding gerrit project analytics/udplog parent analytics [production]
13:44 <cmjohnson1> shutting down bellin for troubleshooting [production]
09:04 <hashar> Site outage was due to our custom wfLogXFF() which uses wfErrorLog(). $wmfUdp2logDest not being global there, caused exception to be shown. [production]
08:59 <hashar> Broken the cluster by having an invalid global set [production]
08:58 <hashar> synchronized wmf-config/CommonSettings.php [production]
08:47 <hashar> synchronizing Wikimedia installation... : [production]
08:44 <hashar> running scap to apply https://gerrit.wikimedia.org/r/7702 [production]
08:41 <hashar> Deploying https://gerrit.wikimedia.org/r/7702 which abstract out the udp2log destination [production]
08:15 <hashar> WMFLabs seems to have recovered now [production]
06:50 <hashar> WMFLabs dieing out, I/O latency raised constantly over the last 2 hours and eventually lead to situation where system (via ssh) is not usable anymore [production]
03:41 <asher> synchronized wmf-config/db.php 'returning db12 and db46' [production]