|
2014-02-14
§
|
| 23:59 |
<aaron> |
synchronized php-1.23wmf13/extensions/Math '9e75a1b' |
[production] |
| 23:32 |
<Coren> |
tools Rebooted labstore4 -- XFS done got broken agun |
[production] |
| 22:53 |
<kaldari> |
synchronized php-1.23wmf13/extensions/VectorBeta/ 'sync update for VectorBeta on wmf13' |
[production] |
| 22:30 |
<kaldari> |
synchronized php-1.23wmf14/extensions/VectorBeta/ 'sync update for VectorBeta on wmf14' |
[production] |
| 22:23 |
<bd808> |
manually applied Ie56d3a5 on logstash100[123] hosts and restarted gmond |
[production] |
| 22:22 |
<bd808> |
enabled puppetd on logstash1002 |
[production] |
| 21:46 |
<bd808> |
disabled puppetd on logstash1002 to test ganglia monitor fix |
[production] |
| 21:43 |
<bd808> |
lostash fatalmonitor dashboard working again after restarts to backend |
[production] |
| 21:30 |
<bd808> |
Upgraded and restarted elasticsearch on logstash1003 |
[production] |
| 21:29 |
<bd808> |
Upgraded and restarted elasticsearch on logstash1001 |
[production] |
| 21:28 |
<bd808> |
Upgraded and restarted elasticsearch on logstash1002 |
[production] |
| 21:25 |
<awight_> |
crm updated from bd973432c957b6253f82b7b0590251402e75a0de to 20684a7e5dd4270e615ca52e6a17b25e880e47c4 |
[production] |
| 20:46 |
<Reedy> |
ULSFO down, traffic to Asia etc affected. Being worked on |
[production] |
| 20:43 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Revert Add local interwiki for metawiki' |
[production] |
| 20:10 |
<mwalker> |
updated payments wiki from from bad46aad4b28435d10570aac8173a7f8dca8751d to 9b936320b797bd01c4e61b1cd7c2e15b0820a24b for version stamp and other 1.22.3 changes |
[production] |
| 20:01 |
<ori> |
synchronized php-1.23wmf13/extensions/NavigationTiming 'Update NavigationTiming for schema revision to 7494934' |
[production] |
| 19:59 |
<ori> |
synchronized php-1.23wmf14/extensions/NavigationTiming 'Update NavigationTiming for schema revision to 7494934' |
[production] |
| 19:55 |
<bd808> |
during scap test snapshot[1234] reported "sudo: no tty present and no askpass program specified" |
[production] |
| 19:50 |
<bd808> |
finished scap: no-diff scap to test script changes (duration: 25m 26s) |
[production] |
| 19:29 |
<bd808> |
no-diff scap updated 366 JSON l10n files |
[production] |
| 19:25 |
<bd808> |
started scap: no-diff scap to test script changes |
[production] |
| 19:11 |
<bd808> |
Updating scap on mediawiki-installation dsh hosts |
[production] |
| 19:09 |
<aaron> |
synchronized wmf-config/mc.php 'Set retry_timeout to -1 for memcached in eqiad only' |
[production] |
| 19:02 |
<bd808> |
Updated /src/scap on tin to b2d8042 |
[production] |
| 18:29 |
<aaron> |
synchronized wmf-config/CommonSettings.php 'Set wgMathDisableTexFilter to fix performance regression' |
[production] |
| 18:24 |
<bd808> |
Starting ganglia-monitor on logstash1001. Filed bug 61384 about problem in elasticsearch_monitoring.py effecting the logstash cluster. |
[production] |
| 17:54 |
<bd808> |
Running gmond in foreground on logstash1001 to debug elasticsearch reporting |
[production] |
| 17:11 |
<bd808> |
Restarted ganglia-monitor on logstash1001 to see if that makes the elasticsearch and redis metrics show up in ganglia |
[production] |
| 16:16 |
<cmjohnson1> |
db1034 swapping cables |
[production] |
| 15:26 |
<manybubbles> |
aborted reindex due to https://bugzilla.wikimedia.org/show_bug.cgi?id=61377 |
[production] |
| 14:25 |
<manybubbles> |
beginning cirrus reindex of all wikipedias running cirrus except enwiki |
[production] |
| 14:14 |
<springle> |
synchronized wmf-config/db-eqiad.php 's1 db1056 full steam' |
[production] |
| 13:55 |
<springle> |
synchronized wmf-config/db-eqiad.php 's1 repool db1056 warm up' |
[production] |
| 13:39 |
<springle> |
synchronized wmf-config/db-eqiad.php 's6 pool db1030 depool db1010' |
[production] |
| 13:30 |
<apergos> |
powercycling ms-be1005, unresponsive even on mgmt console |
[production] |
| 11:49 |
<akosiaris> |
restarted parsoid on wtp1001 |
[production] |
| 11:47 |
<akosiaris> |
depooled wtp1004 |
[production] |
| 11:44 |
<akosiaris> |
restart parsoid on wtp1022. |
[production] |
| 10:50 |
<mark> |
restarted varnish backend on amssq57 |
[production] |
| 06:08 |
<Krinkle> |
Reloading Zuul to deploy Ie02531143511f418a6 |
[production] |