|
2014-02-15
§
|
| 14:50 |
<springle> |
synchronized wmf-config/db-eqiad.php 's1 substitute db1034 for db1055 during schema changes' |
[production] |
| 05:14 |
<springle> |
xtrabackup clone db1055 to db1010 |
[production] |
| 03:32 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at 2014-02-15 03:31:58+00:00 |
[production] |
| 02:50 |
<LocalisationUpdate> |
completed (1.23wmf14) at 2014-02-15 02:50:08+00:00 |
[production] |
| 02:29 |
<LocalisationUpdate> |
completed (1.23wmf13) at 2014-02-15 02:29:05+00:00 |
[production] |
| 02:09 |
<awight_> |
updated payments from 9e4d8b29581e2465d1acde8d7c2377fa6a8522a6 to ce6233998f4bc0266c2e027c44620a8ba9984681 |
[production] |
| 01:41 |
<awight_> |
updated payments from 9b936320b797bd01c4e61b1cd7c2e15b0820a24b to 9e4d8b29581e2465d1acde8d7c2377fa6a8522a6 |
[production] |
| 00:56 |
<ori> |
updated /a/common to {{Gerrit|I7e014f29e}}: Handle empty lines gracefully |
[production] |
| 00:52 |
<awight_> |
crm rolled back from 9385167c077ccef4c52db2f479b39e5907378b61 to 20684a7e5dd4270e615ca52e6a17b25e880e47c4 |
[production] |
| 00:50 |
<awight_> |
update crm from 20684a7e5dd4270e615ca52e6a17b25e880e47c4 to 9385167c077ccef4c52db2f479b39e5907378b61 |
[production] |
| 00:45 |
<ori> |
updated /a/common to {{Gerrit|Id87a90474}}: Remove unused third/fourth parameters for wikiversions |
[production] |
|
2014-02-14
§
|
| 23:59 |
<aaron> |
synchronized php-1.23wmf13/extensions/Math '9e75a1b' |
[production] |
| 23:32 |
<Coren> |
tools Rebooted labstore4 -- XFS done got broken agun |
[production] |
| 22:53 |
<kaldari> |
synchronized php-1.23wmf13/extensions/VectorBeta/ 'sync update for VectorBeta on wmf13' |
[production] |
| 22:30 |
<kaldari> |
synchronized php-1.23wmf14/extensions/VectorBeta/ 'sync update for VectorBeta on wmf14' |
[production] |
| 22:23 |
<bd808> |
manually applied Ie56d3a5 on logstash100[123] hosts and restarted gmond |
[production] |
| 22:22 |
<bd808> |
enabled puppetd on logstash1002 |
[production] |
| 21:46 |
<bd808> |
disabled puppetd on logstash1002 to test ganglia monitor fix |
[production] |
| 21:43 |
<bd808> |
lostash fatalmonitor dashboard working again after restarts to backend |
[production] |
| 21:30 |
<bd808> |
Upgraded and restarted elasticsearch on logstash1003 |
[production] |
| 21:29 |
<bd808> |
Upgraded and restarted elasticsearch on logstash1001 |
[production] |
| 21:28 |
<bd808> |
Upgraded and restarted elasticsearch on logstash1002 |
[production] |
| 21:25 |
<awight_> |
crm updated from bd973432c957b6253f82b7b0590251402e75a0de to 20684a7e5dd4270e615ca52e6a17b25e880e47c4 |
[production] |
| 20:46 |
<Reedy> |
ULSFO down, traffic to Asia etc affected. Being worked on |
[production] |
| 20:43 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Revert Add local interwiki for metawiki' |
[production] |
| 20:10 |
<mwalker> |
updated payments wiki from from bad46aad4b28435d10570aac8173a7f8dca8751d to 9b936320b797bd01c4e61b1cd7c2e15b0820a24b for version stamp and other 1.22.3 changes |
[production] |
| 20:01 |
<ori> |
synchronized php-1.23wmf13/extensions/NavigationTiming 'Update NavigationTiming for schema revision to 7494934' |
[production] |
| 19:59 |
<ori> |
synchronized php-1.23wmf14/extensions/NavigationTiming 'Update NavigationTiming for schema revision to 7494934' |
[production] |
| 19:55 |
<bd808> |
during scap test snapshot[1234] reported "sudo: no tty present and no askpass program specified" |
[production] |
| 19:50 |
<bd808> |
finished scap: no-diff scap to test script changes (duration: 25m 26s) |
[production] |
| 19:29 |
<bd808> |
no-diff scap updated 366 JSON l10n files |
[production] |
| 19:25 |
<bd808> |
started scap: no-diff scap to test script changes |
[production] |
| 19:11 |
<bd808> |
Updating scap on mediawiki-installation dsh hosts |
[production] |
| 19:09 |
<aaron> |
synchronized wmf-config/mc.php 'Set retry_timeout to -1 for memcached in eqiad only' |
[production] |
| 19:02 |
<bd808> |
Updated /src/scap on tin to b2d8042 |
[production] |
| 18:29 |
<aaron> |
synchronized wmf-config/CommonSettings.php 'Set wgMathDisableTexFilter to fix performance regression' |
[production] |
| 18:24 |
<bd808> |
Starting ganglia-monitor on logstash1001. Filed bug 61384 about problem in elasticsearch_monitoring.py effecting the logstash cluster. |
[production] |
| 17:54 |
<bd808> |
Running gmond in foreground on logstash1001 to debug elasticsearch reporting |
[production] |
| 17:11 |
<bd808> |
Restarted ganglia-monitor on logstash1001 to see if that makes the elasticsearch and redis metrics show up in ganglia |
[production] |
| 16:16 |
<cmjohnson1> |
db1034 swapping cables |
[production] |
| 15:26 |
<manybubbles> |
aborted reindex due to https://bugzilla.wikimedia.org/show_bug.cgi?id=61377 |
[production] |
| 14:25 |
<manybubbles> |
beginning cirrus reindex of all wikipedias running cirrus except enwiki |
[production] |
| 14:14 |
<springle> |
synchronized wmf-config/db-eqiad.php 's1 db1056 full steam' |
[production] |
| 13:55 |
<springle> |
synchronized wmf-config/db-eqiad.php 's1 repool db1056 warm up' |
[production] |
| 13:39 |
<springle> |
synchronized wmf-config/db-eqiad.php 's6 pool db1030 depool db1010' |
[production] |
| 13:30 |
<apergos> |
powercycling ms-be1005, unresponsive even on mgmt console |
[production] |
| 11:49 |
<akosiaris> |
restarted parsoid on wtp1001 |
[production] |
| 11:47 |
<akosiaris> |
depooled wtp1004 |
[production] |
| 11:44 |
<akosiaris> |
restart parsoid on wtp1022. |
[production] |
| 10:50 |
<mark> |
restarted varnish backend on amssq57 |
[production] |