|
2014-05-19
§
|
| 19:16 |
<bd808> |
Added display of exception-json events to fatalmonitor logstash dashboard |
[production] |
| 19:01 |
<bd808> |
Purged l10n cache for 1.24wmf3 |
[production] |
| 19:00 |
<bd808> |
Purged l10n cache for 1.23wmf22 |
[production] |
| 18:59 |
<bd808> |
Purged l10n cache for 1.23wmf21 |
[production] |
| 18:59 |
<bd808> |
Purged l10n cache for 1.23wmf21 |
[production] |
| 18:46 |
<reedy> |
Finished scap: nooop to test for errors (duration: 02m 45s) |
[production] |
| 18:44 |
<reedy> |
Started scap: nooop to test for errors |
[production] |
| 18:43 |
<Reedy> |
rm -rf /usr/local/apache/common-local/php-1.23wmf20 against all apaches |
[production] |
| 18:37 |
<Reedy> |
Ran sync-common locally on mw1015 |
[production] |
| 18:32 |
<greg-g> |
mw1010.eqiad.wmnet::common for sync-common, not sure for cdb. (sez superm401) |
[production] |
| 18:23 |
<superm401> |
1 server failed for sync-common. 2 servers failed for sync-rebuild-cdbs |
[production] |
| 18:18 |
<mattflaschen> |
Finished scap: Deploy GettingStarted and enable experiment for de, en, fr, and it (duration: 18m 53s) |
[production] |
| 17:59 |
<mattflaschen> |
Started scap: Deploy GettingStarted and enable experiment for de, en, fr, and it |
[production] |
| 15:26 |
<manybubbles> |
synchronized php-1.24wmf5/resources/lib/oojs-ui/ 'fix panellayout' |
[production] |
| 15:18 |
<manybubbles> |
synchronized php-1.24wmf4/extensions/CirrusSearch/ 'adding url parameter to suppress snippets and one to suggest suggestions to cirrus' |
[production] |
| 15:12 |
<manybubbles> |
SWAT deployed cirrus update for wmf5 and looks good. doing for wmf4 now. |
[production] |
| 15:10 |
<manybubbles> |
synchronized php-1.24wmf5/extensions/CirrusSearch/ |
[production] |
| 14:53 |
<manybubbles> |
synchronized wmf-config/InitialiseSettings.php 'touched and synced InitializeSettings.php to make update to cirrus.dblist take hold - resyncing to mw1171' |
[production] |
| 14:52 |
<_joe_> |
powercycled mw1171, dead and serial console stuck |
[production] |
| 14:51 |
<manybubbles> |
synchronized cirrus.dblist 'Switch cirrus to the primary backend for zh-yue wikipedia - resyncing to mw1171' |
[production] |
| 14:40 |
<manybubbles> |
synchronized wmf-config/InitialiseSettings.php 'touched and synced InitializeSettings.php to make update to cirrus.dblist take hold' |
[production] |
| 14:38 |
<manybubbles> |
synchronized cirrus.dblist 'Switch cirrus to the primary backend for zh-yue wikipedia' |
[production] |
| 13:34 |
<Krinkle> |
Running deleteEqualMessages.php on mtwiki (bug 43917) |
[production] |
| 13:17 |
<Krinkle> |
Running deleteEqualMessages.php on zh_min_nanwiki (bug 43917) |
[production] |
| 13:07 |
<Krinkle> |
Running deleteEqualMessages.php on zh_yuewiki (bug 43917) |
[production] |
| 12:01 |
<Krinkle> |
Running deleteEqualMessages.php on suwiki (bug 43917) |
[production] |
| 03:10 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Mon May 19 03:09:00 UTC 2014 (duration 8m 59s) |
[production] |
| 02:26 |
<LocalisationUpdate> |
completed (1.24wmf5) at 2014-05-19 02:25:51+00:00 |
[production] |
| 02:14 |
<LocalisationUpdate> |
completed (1.24wmf4) at 2014-05-19 02:13:37+00:00 |
[production] |
| 00:30 |
<Tim> |
on osmium: stopping job runners in order to fix cgroup permissions issue |
[production] |
|
2014-05-17
§
|
| 03:08 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Sat May 17 03:07:12 UTC 2014 (duration 7m 11s) |
[production] |
| 02:25 |
<LocalisationUpdate> |
completed (1.24wmf5) at 2014-05-17 02:24:46+00:00 |
[production] |
| 02:15 |
<LocalisationUpdate> |
completed (1.24wmf4) at 2014-05-17 02:14:13+00:00 |
[production] |
| 01:58 |
<mutante> |
powercycling labsdb1003 |
[production] |
| 00:23 |
<ori> |
varnishadm on cp1056 confirms that varnish recognizes mw1151 as "sick" |
[production] |
| 00:20 |
<ori> |
stopping apache and disabling puppet on mw1151 so that varnish stops forwarding reqs to it |
[production] |
| 00:16 |
<Krinkle> |
On mw1151, Gadget::loadStructuredList() returns false, memcached has no value for 'enwiki:gadgets-definition:7' and is unable to store it. |
[production] |
| 00:11 |
<ori> |
timo identified weird RL responses as all originating in mw1151; dmesg shows ata1 disk troubles: "failed command: READ DMA EXT", "sd 0:0:0:0: [sda] Add. Sense: Unrecovered read error - auto reallocate failed" |
[production] |
| 00:04 |
<krinkle> |
synchronized php-1.24wmf4/includes/resourceloader/ResourceLoader.php 'I718fcf23d' |
[production] |