351-400 of 10000 results (10ms)
2012-05-17 §
15:30 <Jeff_Green> adding DNS records to wikimedia.org for RT #2960 [production]
14:22 <mutante> adding gerrit project analytics/udplog parent analytics [production]
13:44 <cmjohnson1> shutting down bellin for troubleshooting [production]
09:04 <hashar> Site outage was due to our custom wfLogXFF() which uses wfErrorLog(). $wmfUdp2logDest not being global there, caused exception to be shown. [production]
08:59 <hashar> Broken the cluster by having an invalid global set [production]
08:58 <hashar> synchronized wmf-config/CommonSettings.php [production]
08:47 <hashar> synchronizing Wikimedia installation... : [production]
08:44 <hashar> running scap to apply https://gerrit.wikimedia.org/r/7702 [production]
08:41 <hashar> Deploying https://gerrit.wikimedia.org/r/7702 which abstract out the udp2log destination [production]
08:15 <hashar> WMFLabs seems to have recovered now [production]
06:50 <hashar> WMFLabs dieing out, I/O latency raised constantly over the last 2 hours and eventually lead to situation where system (via ssh) is not usable anymore [production]
03:41 <asher> synchronized wmf-config/db.php 'returning db12 and db46' [production]
02:48 <LocalisationUpdate> completed (1.20wmf2) at Thu May 17 02:48:02 UTC 2012 [production]
02:22 <LocalisationUpdate> completed (1.20wmf3) at Thu May 17 02:22:02 UTC 2012 [production]
02:18 <reedy> synchronized wmf-config/CommonSettings.php 'Enable SpecialCite everywhere' [production]
01:40 <Tim> on cp1004: reverted after TIME_WAIT client connections reached 38k with no sign of a plateau [production]
01:37 <Tim> on cp1004: trying tcp_tw_reuse=1 instead of tcp_tw_recycle [production]
01:00 <Tim> reverted after client-side TIME_WAIT connections rose rapidly from 367 to 9000 [production]
00:59 <Tim> experimentally setting net.ipv4.tcp_tw_recycle=0 on cp1004 [production]
2012-05-16 §
23:50 <aaron> synchronized php-1.20wmf3/includes/upload/UploadBase.php 'deployed 4b0a61227fce37202da2b62b7dc2474bd227873f' [production]
22:47 <aaron> synchronized wmf-config/CommonSettings.php 'Made mediawikiwiki use $wgSiteStatsAsyncFactor=1.' [production]
22:32 <aaron> synchronized wmf-config/CommonSettings.php 'Set $wgSiteStatsAsyncFactor=1 on testwikis.' [production]
21:45 <maplebed> reverted this morning's mobile push - tests completed [production]
21:12 <reedy> synchronized wmf-config/InitialiseSettings.php 'Bug 36913 - Enable Collection on kkwiki' [production]
21:09 <reedy> synchronized php-1.20wmf3/cache/interwiki.cdb 'Updating interwiki cache' [production]
21:09 <reedy> synchronized php-1.20wmf2/cache/interwiki.cdb 'Updating interwiki cache' [production]
20:05 <binasher> ran ipv6 migrations on globalblocks [production]
20:05 <binasher> converted centralauth.globalblocks from myisam to innodb [production]
19:57 <binasher> rebooting db12 for kernel upgrade [production]
19:51 <binasher> stopping mysql on db12 [production]
19:50 <binasher> recentchanges.rc_ip migration completed [production]
19:47 <reedy> rebuilt wikiversions.cdb and synchronized wikiversions files: Last bits of tidying up [production]
19:44 <binasher> rebooted db46 [production]
19:38 <binasher> shutting down mysql on db46, preparing to reboot for kernel upgrade [production]
19:32 <reedy> synchronized wmf-config/InitialiseSettings.php 'touch' [production]
19:28 <asher> synchronized wmf-config/db.php 'pulling db12 from enwiki, moving watchlist / special queries to db59' [production]
19:25 <binasher> running recentchanges.rc_ip (ipv6) schema migration on enwiki master (5.2mil rows) via os������c - batten down the hatches! [production]
19:17 <binasher> running recentchanges.rc_ip (ipv6) schema migration on s2 dbs via os������c [production]
19:16 <reedy> synchronized wmf-config/InitialiseSettings.php 'touch' [production]
19:14 <Reedy> manually ran ddsh -cM -g mediawiki-installation -o -oSetupTimeout=30 -F30 "sudo -u mwdeploy rsync -a 10.0.5.8::common/*.dblist /usr/local/apache/common-local" because sync-dblist is woefully out of date.. [production]
19:13 <notpeter> restarting ganglia on nickel [production]
19:09 <reedy> rebuilt wikiversions.cdb and synchronized wikiversions files: 12 more misc/wikimedia wikis to 1.20wmf3 [production]
18:59 <reedy> rebuilt wikiversions.cdb and synchronized wikiversions files: All closed wikis to 1.20wmf3 [production]
18:55 <reedy> rebuilt wikiversions.cdb and synchronized wikiversions files: All special wikis to 1.20wmf3 [production]
18:54 <reedy> rebuilt wikiversions.cdb and synchronized wikiversions files: All wikimedia wikis to 1.20wmf3 [production]
18:52 <reedy> rebuilt wikiversions.cdb and synchronized wikiversions files: All wikisource to 1.20wmf3 [production]
18:50 <reedy> rebuilt wikiversions.cdb and synchronized wikiversions files: All wikiquote to 1.20wmf3 [production]
18:48 <reedy> rebuilt wikiversions.cdb and synchronized wikiversions files: All wikiversity to 1.20wmf3 [production]
18:45 <reedy> rebuilt wikiversions.cdb and synchronized wikiversions files: All wiktibooks to 1.20wmf3 [production]
18:42 <reedy> rebuilt wikiversions.cdb and synchronized wikiversions files: All wiktionaries to 1.20wmf3 [production]