|
2012-05-17
§
|
| 13:44 |
<cmjohnson1> |
shutting down bellin for troubleshooting |
[production] |
| 09:04 |
<hashar> |
Site outage was due to our custom wfLogXFF() which uses wfErrorLog(). $wmfUdp2logDest not being global there, caused exception to be shown. |
[production] |
| 08:59 |
<hashar> |
Broken the cluster by having an invalid global set |
[production] |
| 08:58 |
<hashar> |
synchronized wmf-config/CommonSettings.php |
[production] |
| 08:47 |
<hashar> |
synchronizing Wikimedia installation... : |
[production] |
| 08:44 |
<hashar> |
running scap to apply https://gerrit.wikimedia.org/r/7702 |
[production] |
| 08:41 |
<hashar> |
Deploying https://gerrit.wikimedia.org/r/7702 which abstract out the udp2log destination |
[production] |
| 08:15 |
<hashar> |
WMFLabs seems to have recovered now |
[production] |
| 06:50 |
<hashar> |
WMFLabs dieing out, I/O latency raised constantly over the last 2 hours and eventually lead to situation where system (via ssh) is not usable anymore |
[production] |
| 03:41 |
<asher> |
synchronized wmf-config/db.php 'returning db12 and db46' |
[production] |
| 02:48 |
<LocalisationUpdate> |
completed (1.20wmf2) at Thu May 17 02:48:02 UTC 2012 |
[production] |
| 02:22 |
<LocalisationUpdate> |
completed (1.20wmf3) at Thu May 17 02:22:02 UTC 2012 |
[production] |
| 02:18 |
<reedy> |
synchronized wmf-config/CommonSettings.php 'Enable SpecialCite everywhere' |
[production] |
| 01:40 |
<Tim> |
on cp1004: reverted after TIME_WAIT client connections reached 38k with no sign of a plateau |
[production] |
| 01:37 |
<Tim> |
on cp1004: trying tcp_tw_reuse=1 instead of tcp_tw_recycle |
[production] |
| 01:00 |
<Tim> |
reverted after client-side TIME_WAIT connections rose rapidly from 367 to 9000 |
[production] |
| 00:59 |
<Tim> |
experimentally setting net.ipv4.tcp_tw_recycle=0 on cp1004 |
[production] |
|
2012-05-16
§
|
| 23:50 |
<aaron> |
synchronized php-1.20wmf3/includes/upload/UploadBase.php 'deployed 4b0a61227fce37202da2b62b7dc2474bd227873f' |
[production] |
| 22:47 |
<aaron> |
synchronized wmf-config/CommonSettings.php 'Made mediawikiwiki use $wgSiteStatsAsyncFactor=1.' |
[production] |
| 22:32 |
<aaron> |
synchronized wmf-config/CommonSettings.php 'Set $wgSiteStatsAsyncFactor=1 on testwikis.' |
[production] |
| 21:45 |
<maplebed> |
reverted this morning's mobile push - tests completed |
[production] |
| 21:12 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Bug 36913 - Enable Collection on kkwiki' |
[production] |
| 21:09 |
<reedy> |
synchronized php-1.20wmf3/cache/interwiki.cdb 'Updating interwiki cache' |
[production] |
| 21:09 |
<reedy> |
synchronized php-1.20wmf2/cache/interwiki.cdb 'Updating interwiki cache' |
[production] |
| 20:05 |
<binasher> |
ran ipv6 migrations on globalblocks |
[production] |
| 20:05 |
<binasher> |
converted centralauth.globalblocks from myisam to innodb |
[production] |
| 19:57 |
<binasher> |
rebooting db12 for kernel upgrade |
[production] |
| 19:51 |
<binasher> |
stopping mysql on db12 |
[production] |
| 19:50 |
<binasher> |
recentchanges.rc_ip migration completed |
[production] |
| 19:47 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Last bits of tidying up |
[production] |
| 19:44 |
<binasher> |
rebooted db46 |
[production] |
| 19:38 |
<binasher> |
shutting down mysql on db46, preparing to reboot for kernel upgrade |
[production] |
| 19:32 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'touch' |
[production] |
| 19:28 |
<asher> |
synchronized wmf-config/db.php 'pulling db12 from enwiki, moving watchlist / special queries to db59' |
[production] |
| 19:25 |
<binasher> |
running recentchanges.rc_ip (ipv6) schema migration on enwiki master (5.2mil rows) via os������c - batten down the hatches! |
[production] |
| 19:17 |
<binasher> |
running recentchanges.rc_ip (ipv6) schema migration on s2 dbs via os������c |
[production] |
| 19:16 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'touch' |
[production] |
| 19:14 |
<Reedy> |
manually ran ddsh -cM -g mediawiki-installation -o -oSetupTimeout=30 -F30 "sudo -u mwdeploy rsync -a 10.0.5.8::common/*.dblist /usr/local/apache/common-local" because sync-dblist is woefully out of date.. |
[production] |
| 19:13 |
<notpeter> |
restarting ganglia on nickel |
[production] |
| 19:09 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: 12 more misc/wikimedia wikis to 1.20wmf3 |
[production] |
| 18:59 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: All closed wikis to 1.20wmf3 |
[production] |
| 18:55 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: All special wikis to 1.20wmf3 |
[production] |
| 18:54 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: All wikimedia wikis to 1.20wmf3 |
[production] |
| 18:52 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: All wikisource to 1.20wmf3 |
[production] |
| 18:50 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: All wikiquote to 1.20wmf3 |
[production] |
| 18:48 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: All wikiversity to 1.20wmf3 |
[production] |
| 18:45 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: All wiktibooks to 1.20wmf3 |
[production] |
| 18:42 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: All wiktionaries to 1.20wmf3 |
[production] |
| 18:40 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: All wikinews to 1.20wmf3 |
[production] |
| 18:31 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: All non wikipedia en projects to 1.20wmf3 |
[production] |