|
2014-02-16
§
|
| 15:58 |
<ottomata> |
restarted varnishkafka on cp3019 |
[production] |
| 15:48 |
<ottomata> |
starting kafka leader replica election to production load across both brokers evenly. Not yet sure why analytics1022 was the leader for all toppars… |
[production] |
| 10:48 |
<apergos> |
for the record, after the reboot I added back the 10.0.0.45 and ran start-nfs, still not happy |
[production] |
| 10:11 |
<ori> |
labstore4. dmesg: XFS (dm-0): xfs_log_force: error 5 returned. Rebooting. |
[production] |
| 08:07 |
<matanya> |
Labs NFS Issues: cannot open directory .: Stale NFS file handle XFS seems broken again |
[production] |
| 02:42 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at 2014-02-16 02:42:25+00:00 |
[production] |
| 02:21 |
<LocalisationUpdate> |
completed (1.23wmf14) at 2014-02-16 02:20:57+00:00 |
[production] |
| 02:11 |
<LocalisationUpdate> |
completed (1.23wmf13) at 2014-02-16 02:11:05+00:00 |
[production] |
|
2014-02-15
§
|
| 22:30 |
<reedy> |
updated /a/common to {{Gerrit|Id33b8287c}}: Remove 1.23wmf1 through 1.23wmf5 |
[production] |
| 22:27 |
<reedy> |
updated /a/common to {{Gerrit|I04d387adf}}: s1 substitute db1034 for db1055 during schema changes |
[production] |
| 14:50 |
<springle> |
synchronized wmf-config/db-eqiad.php 's1 substitute db1034 for db1055 during schema changes' |
[production] |
| 05:14 |
<springle> |
xtrabackup clone db1055 to db1010 |
[production] |
| 03:32 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at 2014-02-15 03:31:58+00:00 |
[production] |
| 02:50 |
<LocalisationUpdate> |
completed (1.23wmf14) at 2014-02-15 02:50:08+00:00 |
[production] |
| 02:29 |
<LocalisationUpdate> |
completed (1.23wmf13) at 2014-02-15 02:29:05+00:00 |
[production] |
| 02:09 |
<awight_> |
updated payments from 9e4d8b29581e2465d1acde8d7c2377fa6a8522a6 to ce6233998f4bc0266c2e027c44620a8ba9984681 |
[production] |
| 01:41 |
<awight_> |
updated payments from 9b936320b797bd01c4e61b1cd7c2e15b0820a24b to 9e4d8b29581e2465d1acde8d7c2377fa6a8522a6 |
[production] |
| 00:56 |
<ori> |
updated /a/common to {{Gerrit|I7e014f29e}}: Handle empty lines gracefully |
[production] |
| 00:52 |
<awight_> |
crm rolled back from 9385167c077ccef4c52db2f479b39e5907378b61 to 20684a7e5dd4270e615ca52e6a17b25e880e47c4 |
[production] |
| 00:50 |
<awight_> |
update crm from 20684a7e5dd4270e615ca52e6a17b25e880e47c4 to 9385167c077ccef4c52db2f479b39e5907378b61 |
[production] |
| 00:45 |
<ori> |
updated /a/common to {{Gerrit|Id87a90474}}: Remove unused third/fourth parameters for wikiversions |
[production] |
|
2014-02-14
§
|
| 23:59 |
<aaron> |
synchronized php-1.23wmf13/extensions/Math '9e75a1b' |
[production] |
| 23:32 |
<Coren> |
tools Rebooted labstore4 -- XFS done got broken agun |
[production] |
| 22:53 |
<kaldari> |
synchronized php-1.23wmf13/extensions/VectorBeta/ 'sync update for VectorBeta on wmf13' |
[production] |
| 22:30 |
<kaldari> |
synchronized php-1.23wmf14/extensions/VectorBeta/ 'sync update for VectorBeta on wmf14' |
[production] |
| 22:23 |
<bd808> |
manually applied Ie56d3a5 on logstash100[123] hosts and restarted gmond |
[production] |
| 22:22 |
<bd808> |
enabled puppetd on logstash1002 |
[production] |
| 21:46 |
<bd808> |
disabled puppetd on logstash1002 to test ganglia monitor fix |
[production] |
| 21:43 |
<bd808> |
lostash fatalmonitor dashboard working again after restarts to backend |
[production] |
| 21:30 |
<bd808> |
Upgraded and restarted elasticsearch on logstash1003 |
[production] |
| 21:29 |
<bd808> |
Upgraded and restarted elasticsearch on logstash1001 |
[production] |
| 21:28 |
<bd808> |
Upgraded and restarted elasticsearch on logstash1002 |
[production] |
| 21:25 |
<awight_> |
crm updated from bd973432c957b6253f82b7b0590251402e75a0de to 20684a7e5dd4270e615ca52e6a17b25e880e47c4 |
[production] |
| 20:46 |
<Reedy> |
ULSFO down, traffic to Asia etc affected. Being worked on |
[production] |
| 20:43 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Revert Add local interwiki for metawiki' |
[production] |
| 20:10 |
<mwalker> |
updated payments wiki from from bad46aad4b28435d10570aac8173a7f8dca8751d to 9b936320b797bd01c4e61b1cd7c2e15b0820a24b for version stamp and other 1.22.3 changes |
[production] |
| 20:01 |
<ori> |
synchronized php-1.23wmf13/extensions/NavigationTiming 'Update NavigationTiming for schema revision to 7494934' |
[production] |
| 19:59 |
<ori> |
synchronized php-1.23wmf14/extensions/NavigationTiming 'Update NavigationTiming for schema revision to 7494934' |
[production] |
| 19:55 |
<bd808> |
during scap test snapshot[1234] reported "sudo: no tty present and no askpass program specified" |
[production] |
| 19:50 |
<bd808> |
finished scap: no-diff scap to test script changes (duration: 25m 26s) |
[production] |
| 19:29 |
<bd808> |
no-diff scap updated 366 JSON l10n files |
[production] |
| 19:25 |
<bd808> |
started scap: no-diff scap to test script changes |
[production] |
| 19:11 |
<bd808> |
Updating scap on mediawiki-installation dsh hosts |
[production] |
| 19:09 |
<aaron> |
synchronized wmf-config/mc.php 'Set retry_timeout to -1 for memcached in eqiad only' |
[production] |
| 19:02 |
<bd808> |
Updated /src/scap on tin to b2d8042 |
[production] |
| 18:29 |
<aaron> |
synchronized wmf-config/CommonSettings.php 'Set wgMathDisableTexFilter to fix performance regression' |
[production] |
| 18:24 |
<bd808> |
Starting ganglia-monitor on logstash1001. Filed bug 61384 about problem in elasticsearch_monitoring.py effecting the logstash cluster. |
[production] |
| 17:54 |
<bd808> |
Running gmond in foreground on logstash1001 to debug elasticsearch reporting |
[production] |
| 17:11 |
<bd808> |
Restarted ganglia-monitor on logstash1001 to see if that makes the elasticsearch and redis metrics show up in ganglia |
[production] |
| 16:16 |
<cmjohnson1> |
db1034 swapping cables |
[production] |