|
2015-06-15
§
|
| 13:04 |
<_joe_> |
re-scaling down the recovery index bandwidth in ES to 20 mb/s |
[production] |
| 12:52 |
<demon> |
Synchronized wmf-config/PoolCounterSettings-common.php: partially turn search back on (duration: 00m 13s) |
[production] |
| 11:54 |
<_joe_> |
raised the ES index replica bandwidth limit to 60mb |
[production] |
| 11:31 |
<akosiaris> |
migrating etherpad.wikimedia.org to etherpad1001.eqiad.wmnet |
[production] |
| 11:15 |
<_joe_> |
raised the max bytes for ES recovery to 40mbps |
[production] |
| 10:49 |
<manybubbles> |
and we're yellow right now. |
[production] |
| 10:49 |
<manybubbles> |
the initial primaries stage - the red stage of the rolling restart - recovers quick-ish |
[production] |
| 10:48 |
<manybubbles> |
soon we should see it go yellow and stay that way while the replicas recover |
[production] |
| 10:48 |
<manybubbles> |
manybubbles is confident his mighty bitch slap of the elasticsearch cluster has set it further to the road to recovery |
[production] |
| 10:46 |
<jynus> |
disabled puppet on all elasticsearch nodes to avoid restarting services and other magic |
[production] |
| 10:44 |
<_joe_> |
disabled hot threads logging, ganglia on es nodes |
[production] |
| 10:44 |
<manybubbles> |
started Elasticsearch on all elasticsearch nodes |
[production] |
| 10:38 |
<manybubbles> |
stopping all elasticsearch servers - going for a full cluster resstart. |
[production] |
| 10:11 |
<manybubbles> |
restarting elasticsearch on elasticsearch1021 - that one is in a gc death spiral |
[production] |
| 09:26 |
<oblivian> |
Synchronized wmf-config/PoolCounterSettings-common.php: temporarily throttle down cirrussearch (duration: 00m 13s) |
[production] |
| 09:12 |
<oblivian> |
Synchronized wmf-config/PoolCounterSettings-common.php: temporarily throttle down cirrussearch (duration: 00m 13s) |
[production] |
| 07:35 |
<_joe_> |
attempting a fast restart of elastic1020 |
[production] |
| 07:21 |
<ori> |
Synchronized php-1.26wmf9/extensions/CirrusSearch/includes/Util.php: I504dac0c3: Add missing 'use \\Status;' to includes/Util.php (duration: 00m 13s) |
[production] |
| 04:56 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Mon Jun 15 04:56:39 UTC 2015 (duration 56m 38s) |
[production] |
| 03:31 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool db1057 (duration: 00m 12s) |
[production] |
| 02:23 |
<LocalisationUpdate> |
completed (1.26wmf9) at 2015-06-15 02:22:56+00:00 |
[production] |
| 02:19 |
<l10nupdate> |
Synchronized php-1.26wmf9/cache/l10n: (no message) (duration: 05m 46s) |
[production] |
|
2015-06-13
§
|
| 19:30 |
<bblack> |
repooled cp1071, cp3040 |
[production] |
| 18:53 |
<bblack> |
rebooting cp1071, cp3040 to look at BIOS-level things (depooled, icinga-downed) |
[production] |
| 17:08 |
<krinkle> |
Synchronized php-1.26wmf9/extensions/WikimediaEvents: T101806 (duration: 00m 12s) |
[production] |
| 15:47 |
<paravoid> |
labstore1001: stopping manage-nfs-volumes daemon |
[production] |
| 04:42 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Sat Jun 13 04:41:57 UTC 2015 (duration 41m 56s) |
[production] |
| 03:51 |
<Krinkle> |
Running deleteEqualMessages.php for sawiki (T45917) |
[production] |
| 03:49 |
<Krinkle> |
Running deleteEqualMessages.php for cewiki (T45917) |
[production] |
| 02:21 |
<LocalisationUpdate> |
completed (1.26wmf9) at 2015-06-13 02:20:58+00:00 |
[production] |
| 02:18 |
<l10nupdate> |
Synchronized php-1.26wmf9/cache/l10n: (no message) (duration: 05m 19s) |
[production] |
| 00:17 |
<gwicke> |
restarted cassandra on restbase1001 |
[production] |
| 00:13 |
<gwicke> |
restarted cassandra on restbase1002 |
[production] |
|
2015-06-12
§
|
| 22:57 |
<ejegg> |
rolled back SmashPig on listener from 15acdafef9d9682c417632e5ac5a5f2e5380f92e to e1e925c9fc2a60c1e14ef01d8b653dc09512f51f |
[production] |
| 22:40 |
<ejegg> |
updated SmashPig on listener from e1e925c9fc2a60c1e14ef01d8b653dc09512f51f to 15acdafef9d9682c417632e5ac5a5f2e5380f92e |
[production] |
| 22:24 |
<godog> |
upgrade and bounce carbon daemons on graphite2001 to investigate T101572 |
[production] |
| 21:16 |
<ori> |
Synchronized wmf-config/InitialiseSettings.php: I3694489ba: wgCanonicalServer->https for new HTTPS domains (duration: 00m 14s) |
[production] |
| 20:33 |
<krenair> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/217878/1 (duration: 00m 13s) |
[production] |
| 20:32 |
<krenair> |
Synchronized w/static/images/project-logos/dawiki-200k.png: https://gerrit.wikimedia.org/r/#/c/217878/1 (duration: 00m 16s) |
[production] |
| 20:15 |
<krenair> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/217670/ (duration: 00m 12s) |
[production] |
| 19:28 |
<ejegg> |
updated SmashPig on payments-listener from f9c3eaa99fa0fe8ef098d0fc876091d3676aa039 to 5a463400bc74706ba7bf6256cd0101014e792acb |
[production] |
| 19:28 |
<ejegg> |
updated SmashPig on payments-listener ccepting New Patients: |
[production] |
| 18:47 |
<ejegg> |
updated SmashPig on payments-listener from 7fed22ad933a6d3e371d60dfc6f8fdd0f9131510 to f9c3eaa99fa0fe8ef098d0fc876091d3676aa039 |
[production] |
| 18:45 |
<faidon> |
Synchronized wmf-config/InitialiseSettings.php: remove wmgHTTPSBlacklistCountries (duration: 00m 12s) |
[production] |