|
2012-05-17
§
|
| 18:49 |
<binasher> |
syncing cluster23 tables from es1002 to es1004 |
[production] |
| 18:46 |
<binasher> |
stopped replication on es1002 |
[production] |
| 18:44 |
<notpeter> |
restarting puppet on brewster |
[production] |
| 18:39 |
<catrope> |
synchronizing Wikimedia installation... : ArticleFeedbackv5 updates |
[production] |
| 18:13 |
<aaron> |
synchronized wmf-config/swift.php |
[production] |
| 18:08 |
<aaron> |
synchronized php-1.20wmf3/includes/filerepo 'deployed 103efda39dd57bc22898bd0e69932982c1cfd588' |
[production] |
| 18:00 |
<Jeff_Green> |
shutting down grosley for disk and RAM upgrades |
[production] |
| 17:42 |
<notpeter> |
temporarily turning off puppet on brewster for preseed hackz |
[production] |
| 17:20 |
<maplebed> |
flushing the mobile cache post-deploy |
[production] |
| 17:17 |
<maplebed> |
deploying config change to mobile - more zero IP addresses. gerrit [[rev:7867|r7867]] |
[production] |
| 15:31 |
<dzahn> |
synchronized php/cache/interwiki.cdb 'Updating interwiki cache' |
[production] |
| 15:30 |
<mutante> |
sync-common-file interwiki.cdb |
[production] |
| 15:30 |
<mutante> |
creating fresh interwiki.cdb from dumpInterwiki.php |
[production] |
| 15:30 |
<Jeff_Green> |
adding DNS records to wikimedia.org for RT #2960 |
[production] |
| 14:22 |
<mutante> |
adding gerrit project analytics/udplog parent analytics |
[production] |
| 13:44 |
<cmjohnson1> |
shutting down bellin for troubleshooting |
[production] |
| 09:04 |
<hashar> |
Site outage was due to our custom wfLogXFF() which uses wfErrorLog(). $wmfUdp2logDest not being global there, caused exception to be shown. |
[production] |
| 08:59 |
<hashar> |
Broken the cluster by having an invalid global set |
[production] |
| 08:58 |
<hashar> |
synchronized wmf-config/CommonSettings.php |
[production] |
| 08:47 |
<hashar> |
synchronizing Wikimedia installation... : |
[production] |
| 08:44 |
<hashar> |
running scap to apply https://gerrit.wikimedia.org/r/7702 |
[production] |
| 08:41 |
<hashar> |
Deploying https://gerrit.wikimedia.org/r/7702 which abstract out the udp2log destination |
[production] |
| 08:15 |
<hashar> |
WMFLabs seems to have recovered now |
[production] |
| 06:50 |
<hashar> |
WMFLabs dieing out, I/O latency raised constantly over the last 2 hours and eventually lead to situation where system (via ssh) is not usable anymore |
[production] |
| 03:41 |
<asher> |
synchronized wmf-config/db.php 'returning db12 and db46' |
[production] |
| 02:48 |
<LocalisationUpdate> |
completed (1.20wmf2) at Thu May 17 02:48:02 UTC 2012 |
[production] |
| 02:22 |
<LocalisationUpdate> |
completed (1.20wmf3) at Thu May 17 02:22:02 UTC 2012 |
[production] |
| 02:18 |
<reedy> |
synchronized wmf-config/CommonSettings.php 'Enable SpecialCite everywhere' |
[production] |
| 01:40 |
<Tim> |
on cp1004: reverted after TIME_WAIT client connections reached 38k with no sign of a plateau |
[production] |
| 01:37 |
<Tim> |
on cp1004: trying tcp_tw_reuse=1 instead of tcp_tw_recycle |
[production] |
| 01:00 |
<Tim> |
reverted after client-side TIME_WAIT connections rose rapidly from 367 to 9000 |
[production] |
| 00:59 |
<Tim> |
experimentally setting net.ipv4.tcp_tw_recycle=0 on cp1004 |
[production] |
|
2012-05-16
§
|
| 23:50 |
<aaron> |
synchronized php-1.20wmf3/includes/upload/UploadBase.php 'deployed 4b0a61227fce37202da2b62b7dc2474bd227873f' |
[production] |
| 22:47 |
<aaron> |
synchronized wmf-config/CommonSettings.php 'Made mediawikiwiki use $wgSiteStatsAsyncFactor=1.' |
[production] |
| 22:32 |
<aaron> |
synchronized wmf-config/CommonSettings.php 'Set $wgSiteStatsAsyncFactor=1 on testwikis.' |
[production] |
| 21:45 |
<maplebed> |
reverted this morning's mobile push - tests completed |
[production] |
| 21:12 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Bug 36913 - Enable Collection on kkwiki' |
[production] |
| 21:09 |
<reedy> |
synchronized php-1.20wmf3/cache/interwiki.cdb 'Updating interwiki cache' |
[production] |
| 21:09 |
<reedy> |
synchronized php-1.20wmf2/cache/interwiki.cdb 'Updating interwiki cache' |
[production] |
| 20:05 |
<binasher> |
ran ipv6 migrations on globalblocks |
[production] |
| 20:05 |
<binasher> |
converted centralauth.globalblocks from myisam to innodb |
[production] |
| 19:57 |
<binasher> |
rebooting db12 for kernel upgrade |
[production] |
| 19:51 |
<binasher> |
stopping mysql on db12 |
[production] |
| 19:50 |
<binasher> |
recentchanges.rc_ip migration completed |
[production] |
| 19:47 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Last bits of tidying up |
[production] |
| 19:44 |
<binasher> |
rebooted db46 |
[production] |
| 19:38 |
<binasher> |
shutting down mysql on db46, preparing to reboot for kernel upgrade |
[production] |
| 19:32 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'touch' |
[production] |
| 19:28 |
<asher> |
synchronized wmf-config/db.php 'pulling db12 from enwiki, moving watchlist / special queries to db59' |
[production] |
| 19:25 |
<binasher> |
running recentchanges.rc_ip (ipv6) schema migration on enwiki master (5.2mil rows) via os������c - batten down the hatches! |
[production] |