51-100 of 10000 results (12ms)
2012-05-17 §
20:34 <maplebed> deployed change to swift and mediawiki for MW to write thumbnails to swift instead of rewrite.py with aaron [production]
20:32 <maplebed> deployed parallel thumbnail purging for test, test2, and mediawiki with aaron [production]
20:25 <aaron> synchronized wmf-config/swift.php 'Enabled thumb copy hook for testwikis and mw.org' [production]
20:14 <binasher> completed securepoll_votes.vote_ip and all ipv6 schema migration [production]
20:10 <aaron> synchronized wmf-config/CommonSettings.php [production]
20:09 <aaron> synchronized wmf-config/swift.php [production]
20:03 <aaron> synchronized wmf-config/swift.php 'Enabling new purge hook on testwikis again.' [production]
20:03 <binasher> running securepoll_votes.vote_ip schema migration on s1 [production]
20:01 <binasher> running securepoll_votes.vote_ip schema migration on all s2 dbs [production]
19:19 <binasher> running securepoll_votes.vote_ip schema migration on all s4 + s3 dbs [production]
19:17 <binasher> running securepoll_votes.vote_ip schema migration on all s5 dbs [production]
19:16 <binasher> running securepoll_votes.vote_ip schema migration on all s6 dbs [production]
19:02 <binasher> running securepoll_votes.vote_ip schema migration on all s7 dbs [production]
18:49 <binasher> syncing cluster23 tables from es1002 to es1004 [production]
18:46 <binasher> stopped replication on es1002 [production]
18:44 <notpeter> restarting puppet on brewster [production]
18:39 <catrope> synchronizing Wikimedia installation... : ArticleFeedbackv5 updates [production]
18:13 <aaron> synchronized wmf-config/swift.php [production]
18:08 <aaron> synchronized php-1.20wmf3/includes/filerepo 'deployed 103efda39dd57bc22898bd0e69932982c1cfd588' [production]
18:00 <Jeff_Green> shutting down grosley for disk and RAM upgrades [production]
17:42 <notpeter> temporarily turning off puppet on brewster for preseed hackz [production]
17:20 <maplebed> flushing the mobile cache post-deploy [production]
17:17 <maplebed> deploying config change to mobile - more zero IP addresses. gerrit [[rev:7867|r7867]] [production]
15:31 <dzahn> synchronized php/cache/interwiki.cdb 'Updating interwiki cache' [production]
15:30 <mutante> sync-common-file interwiki.cdb [production]
15:30 <mutante> creating fresh interwiki.cdb from dumpInterwiki.php [production]
15:30 <Jeff_Green> adding DNS records to wikimedia.org for RT #2960 [production]
14:22 <mutante> adding gerrit project analytics/udplog parent analytics [production]
13:44 <cmjohnson1> shutting down bellin for troubleshooting [production]
09:04 <hashar> Site outage was due to our custom wfLogXFF() which uses wfErrorLog(). $wmfUdp2logDest not being global there, caused exception to be shown. [production]
08:59 <hashar> Broken the cluster by having an invalid global set [production]
08:58 <hashar> synchronized wmf-config/CommonSettings.php [production]
08:47 <hashar> synchronizing Wikimedia installation... : [production]
08:44 <hashar> running scap to apply https://gerrit.wikimedia.org/r/7702 [production]
08:41 <hashar> Deploying https://gerrit.wikimedia.org/r/7702 which abstract out the udp2log destination [production]
08:15 <hashar> WMFLabs seems to have recovered now [production]
06:50 <hashar> WMFLabs dieing out, I/O latency raised constantly over the last 2 hours and eventually lead to situation where system (via ssh) is not usable anymore [production]
03:41 <asher> synchronized wmf-config/db.php 'returning db12 and db46' [production]
02:48 <LocalisationUpdate> completed (1.20wmf2) at Thu May 17 02:48:02 UTC 2012 [production]
02:22 <LocalisationUpdate> completed (1.20wmf3) at Thu May 17 02:22:02 UTC 2012 [production]
02:18 <reedy> synchronized wmf-config/CommonSettings.php 'Enable SpecialCite everywhere' [production]
01:40 <Tim> on cp1004: reverted after TIME_WAIT client connections reached 38k with no sign of a plateau [production]
01:37 <Tim> on cp1004: trying tcp_tw_reuse=1 instead of tcp_tw_recycle [production]
01:00 <Tim> reverted after client-side TIME_WAIT connections rose rapidly from 367 to 9000 [production]
00:59 <Tim> experimentally setting net.ipv4.tcp_tw_recycle=0 on cp1004 [production]
2012-05-16 §
23:50 <aaron> synchronized php-1.20wmf3/includes/upload/UploadBase.php 'deployed 4b0a61227fce37202da2b62b7dc2474bd227873f' [production]
22:47 <aaron> synchronized wmf-config/CommonSettings.php 'Made mediawikiwiki use $wgSiteStatsAsyncFactor=1.' [production]
22:32 <aaron> synchronized wmf-config/CommonSettings.php 'Set $wgSiteStatsAsyncFactor=1 on testwikis.' [production]
21:45 <maplebed> reverted this morning's mobile push - tests completed [production]
21:12 <reedy> synchronized wmf-config/InitialiseSettings.php 'Bug 36913 - Enable Collection on kkwiki' [production]