|
2014-03-10
§
|
| 21:55 |
<bd808> |
Restarted logstash on logstash1001; new events flowing in again now |
[production] |
| 21:49 |
<K4-713> |
synchronized payments to 5d20972. |
[production] |
| 21:47 |
<bd808> |
ganglia monitoring for elasticsearch on logstash cluster seems broken. Caused by 1.0.x upgrade having not happened there yet? |
[production] |
| 21:46 |
<bd808> |
Restarted elastcisearch on logstash1003; it was JVM heap thrashing at 98% heap used. |
[production] |
| 21:44 |
<RobH> |
arsenic reclaim per rt6522, ignore alerts |
[production] |
| 21:40 |
<sbernardin> |
ms-be5 swapping failed disk |
[production] |
| 21:34 |
<K4-713> |
synchronized payments cluster to 01f7af8 |
[production] |
| 21:27 |
<bd808> |
No new data in logstash since 14:56Z. Bryan will investigate. |
[production] |
| 20:26 |
<gwicke> |
Coren fixed up the Parsoid deploy by running "salt-run deploy.restart 'parsoid/deploy' '10%'" from the salt master as a work-around for [[bugzilla:61882]] |
[production] |
| 20:18 |
<gwicke> |
deployed Parsoid 681f7b8d2 using deploy 77d17489; service restart incomplete due to [[bugzilla:61882]] |
[production] |
| 19:44 |
<reedy> |
synchronized wmf-config/ 'Use Vips for images over 20MP' |
[production] |
| 19:17 |
<^d> |
enwiki search indexes don't look happy. sporadic reports of "search request timed out" from users, cpu usage pretty high. possibly unrelated bug logs getting spammed with: "Retry, temporal error for query..." |
[production] |
| 18:50 |
<awight> |
updated crm from b85722a7c14af6ec1b50d2cddcd9f84d8c88da3e to a1ca31664f771a48963425893b2b927a5148119b |
[production] |
| 18:27 |
<awight> |
crm updated from 1aa34fd565e818466515b70c6a00af2e30e47ce2 to b85722a7c14af6ec1b50d2cddcd9f84d8c88da3e |
[production] |
| 18:11 |
<krinkle> |
synchronized php-1.23wmf17/resources/mediawiki.api/mediawiki.api.watch.js 'I2ac9e0da0f1c825' |
[production] |
| 18:07 |
<demon> |
synchronized php-1.23wmf17/extensions/Elastica/Elastica.php 'I forgot how to use git' |
[production] |
| 18:03 |
<demon> |
synchronized php-1.23wmf17/extensions/Elastica/Elastica.php |
[production] |
| 17:37 |
<andrewbogott> |
upgrading mediawiki on wikitech |
[production] |
| 16:33 |
<hashar> |
restarted Zuul on gallium : leaked file descriptors (fixed upstream) |
[production] |
| 15:43 |
<hoo> |
synchronized multiversion/MWScript.php 'Path typo fix: {{Gerrit|I6a05447}}' |
[production] |
| 15:06 |
<hoo> |
synchronized wmf-config/session-labs.php 'Syncing labs file for consistency {{Gerrit|I5e1a3242}}' |
[production] |
| 15:05 |
<hoo> |
synchronized wmf-config/InitialiseSettings.php 'Enable Extension:GuidedTour on testwikidatawiki' |
[production] |
| 14:58 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php |
[production] |
| 14:57 |
<reedy> |
updated /a/common to {{Gerrit|Ibcbd10044}}: Disable and remove ContactPageFundraiser |
[production] |
| 14:44 |
<reedy> |
synchronized wmf-config/ 'Remove ContactPageFundraiser, SkinPerPage, FormPreloadPostCache and skins used on foundationwiki' |
[production] |
| 14:26 |
<ottomata> |
put elastic1007 and elastic1013-1016 back in main elasticsearch pool |
[production] |
| 09:11 |
<hashar> |
Jenkins restarted and proceeding jobs again |
[production] |
| 09:06 |
<hashar> |
stopping Jenkins |
[production] |
| 09:01 |
<hashar> |
Jenkins unresponsive for some reason yeah!!! |
[production] |
| 02:57 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Mon Mar 10 02:57:32 UTC 2014 (duration 57m 31s) |
[production] |
| 02:25 |
<LocalisationUpdate> |
completed (1.23wmf17) at 2014-03-10 02:25:14+00:00 |
[production] |
| 02:14 |
<LocalisationUpdate> |
completed (1.23wmf16) at 2014-03-10 02:14:08+00:00 |
[production] |
|
2014-03-08
§
|
| 03:49 |
<mutante> |
shut down srv250-srv247 (this was stdpa row D-D2) |
[production] |
| 03:42 |
<mutante> |
shutting down srv193,srv235-srv247,srv248,249 |
[production] |
| 03:28 |
<andrewbogott> |
upgrading nova-compute on virt100[1-6] |
[production] |
| 03:15 |
<mutante> |
shut down Tampa mw hosts 21-27,49,58,75-85 (that is sdpta row D:D1) |
[production] |
| 03:04 |
<mutante> |
shut down Tampa mw hosts 1-20 |
[production] |
| 02:56 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Sat Mar 8 02:55:55 UTC 2014 (duration 55m 54s) |
[production] |
| 02:25 |
<LocalisationUpdate> |
completed (1.23wmf17) at 2014-03-08 02:25:00+00:00 |
[production] |
| 02:14 |
<LocalisationUpdate> |
completed (1.23wmf16) at 2014-03-08 02:14:25+00:00 |
[production] |
|
2014-03-07
§
|
| 23:36 |
<ori> |
updated /a/common to {{Gerrit|If2530281b}}: Order branch directories as version numbers |
[production] |
| 22:14 |
<mutante> |
deleting salt keys for all Tampa app servers removed from puppet |
[production] |
| 21:54 |
<hashar> |
Jenkins restarted |
[production] |
| 21:49 |
<hashar> |
restarting Jenkins it is broken |
[production] |
| 21:45 |
<mutante> |
killing all tampa appservers from puppetstoredconfigs |
[production] |
| 21:41 |
<mutante> |
disabling puppet agent on all tampa appservers |
[production] |
| 21:15 |
<hoo> |
synchronized wmf-config/InitialiseSettings-labs.php 'Syncing beta-only change for consistency' |
[production] |
| 20:28 |
<mutante> |
killing mw1-16 from puppet stored configs, icinga,.. |
[production] |
| 20:26 |
<mutante> |
revoking puppet certs for Tampa appservers |
[production] |