I have two elasticsearch cluster - cluster1 (version 2.4.x) and cluster2 (version 5.1.1). I am feeding live data to these two cluster from logstash (version 5.x) to achieve a stable state between cluster 1 and 2. The data is now in sync between these two cluster but the cluster 2's index size is almost double when compared to cluster 1's index size. Below is an example:
ES Version health status index pri rep docs.count docs.deleted store.size pri.store.size
2.4.x Old_Cluster green open index1-08 5 1 6520824 0 5.3gb 2.6gb
5.1.1 New_Cluster green open index2-08 5 1 6520824 0 9.3gb 4.6gb
As you can see above the docs.count is same between these two indexes but the size of the index2-08 is double the size of index1-08. Both clusters have similar configuration with respect to their versions.
The logstash (version 5.x) creates these two indexes using default mappings. This is lowering kibana's search capability. I am new to ELK thus am not sure if this something expected.
Can anyone please take a look and provide any suggestions on what might be the reason for this behaviour?
Related
I couldn't find a way of determining what would be the max creatable enclave using the SGX SDK. Is there any way of fetching these capabilities? This is especially useful in cloud environments where you can create virtual machines with EPC sections and you don't know the actual usable size of the provisioned EPC.
The only option I found to get the value of the EPC section is by filtering dmesg for the output of the SGX driver.
[ 2.451815] intel_sgx: EPC section 0x240000000-0x2bfffffff
If we convert the start and end of the section in decimals and subtract the end from the start, we get a value in bytes which we can convert to gibibytes or mebibytes.
Here are the calculations for this example and the result in gibibytes:
python3 -c 'print((0x2bfffffff - 0x240000000) / 1024 ** 3)'
1.9999999990686774
So i am new to Openwhisk and i am trying some stuff for various experiments i want to perform. I am trying to create a web action with a concurrency limit > than the default which is 1. As far as i understood this can happen by typing wsk action create <action_name> function.js --concurrency 2 --web true, although this provides an error and the action is not created. Im currently using the standalone run for OW. Any insights on the matter?
*error: Unable to create action '0': The request content was malformed:
requirement failed: concurrency 2 exceeds allowed threshold of 1 (code 5uCNnpmwwDzMMgkWu33UW9o0zCfVH4FQ)
**the action name i used for this example is "0"
Thanks a lot in advance!
I believe you just need to set the system-level concurrency limits to be greater than 1. If you've deployed OpenWhisk via Kubernetes, then you can just modify the mycluster.yaml file.
This here is the relevant part of the configuration. Simply set those values to be greater than one. For example:
concurrency:
min: 1
max: 10
std: 1
I am using AWS Redis for a project and ran into an Out of Memory (OOM) issue. In investigating the issue, I discovered a couple parameters that affect the amount of usable memory, but the math doesn't seem to work out for my case. Am I missing any variables?
I'm using:
3 shards, 3 nodes per shard
cache.t2.micro instance type
default.redis4.0.cluster.on cache parameter group
The ElastiCache website says cache.t2.micro has 0.555 GiB = 0.555 * 2^30 B = 595,926,712 B memory.
default.redis4.0.cluster.on parameter group has maxmemory = 581,959,680 (just under the instance memory) and reserved-memory-percent = 25%. 581,959,680 B * 0.75 = 436,469,760 B available.
Now, looking at the BytesUsedForCache metric in CloudWatch when I ran out of memory, I see nodes around 457M, 437M, 397M, 393M bytes. It shouldn't be possible for a node to be above the 436M bytes calculated above!
What am I missing; Is there something else that determines how much memory is usable?
I remember reading it somewhere but I can not find it right now. I believe BytesUsedForCache is a sum of RAM and SWAP used by Redis to store data/buffers.
As Elasticache's docs suggest that SWAP should not go higher than 300 MB.
I would suggest checking the SWAP metric at that time.
I am running an Aerospike cluster in Google Cloud. Following the recommendation on this post, I updated to the last version (3.11.1.1) and re-created all servers. In fact, this change cause my 5 servers to operate in a much lower CPU load (it was around 75% load before, now it is on 20%, as show in the graph bellow:
Because of this low load, I decided to reduce the cluster size to 4 servers. When I did this, my application started to receive the following error:
All batch queues are full
I found this discussion about the topic, recommending to change the parameters batch-index-threads and batch-max-unused-buffers with the command
asadm -e "asinfo -v 'set-config:context=service;batch-index-threads=NEW_VALUE'"
I tried many combinations of values (batch-index-threads with 2,4,8,16) and none of them solved the problem, and also changing the batch-index-threads param. Nothing solves my problem. I keep receiving the All batch queues are full error.
Here is my aerospace.conf relevant information:
service {
user root
group root
paxos-single-replica-limit 1 # Number of nodes where the replica count is automatically reduced to 1.
paxos-recovery-policy auto-reset-master
pidfile /var/run/aerospike/asd.pid
service-threads 32
transaction-queues 32
transaction-threads-per-queue 4
batch-index-threads 40
proto-fd-max 15000
batch-max-requests 30000
replication-fire-and-forget true
}
I use 300GB SSD disks on these servers.
A quick note which may or may not pertain to you:
A common mistake we have seen in the past is that developers decide to use 'batch get' as a general purpose 'get' for single and multiple record requests. The single record get will perform better for single record requests.
It's possible that you are being constrained by the network between the clients and servers. Reducing from 5 to 4 nodes reduced the aggregate pipe. In addition, removing a node will start cluster migrations which adds additional network load.
I would look at the batch-max-buffer-per-queue config parameter.
Maximum number of 128KB response buffers allowed in each batch index
queue. If all batch index queues are full, new batch requests are
rejected.
In conjunction with raising this value from the default of 255 you will want to also raise the batch-max-unused-buffers to batch-index-threads x batch-max-buffer-per-queue + 1 (at least). If you do not do that new buffers will be created and destroyed constantly, as the amount of free (unused) buffers is smaller than the ones you're using. The moment the batch response is served the system will strive to trim the buffers down to the max unused number. You will see this reflected in the batch_index_created_buffers metric constantly rising.
Be aware that you need to have enough DRAM for this. For example if you raise the batch-max-buffer-per-queue to 320 you will consume
40 (`batch-index-threads`) x 320 (`batch-max-buffer-per-queue`) x 128K = 1600MB
For the sake of performance the batch-max-unused-buffers should be set to 13000 which will have a max memory consumption of 1625MB (1.59GB) per-node.
I am trying to determine how many nodes I need for my EMR cluster. As part of best practices the recommendations are:
(Total Mappers needed for your job + Time taken to process) / (per instance capacity + desired time) as outlined here: http://www.slideshare.net/AmazonWebServices/amazon-elastic-mapreduce-deep-dive-and-best-practices-bdt404-aws-reinvent-2013, page 89.
The question is how to determine how many parallel mappers the instance will support since AWS don't publish? https://aws.amazon.com/emr/pricing/
Sorry if i missed something obvious.
Wayne
To determine the number of parallel mappers , you will need to check this documentation from EMR called Task Configuration where EMR had a predefined mapping set of configurations for every instance type which would determine the number of mappers/reducers.
http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hadoop-task-config.html
For example : Lets say you have 5 m1.xlarge core nodes. According to the default mapred-site.xml configuration values for that instance type from EMR docs, we have
mapreduce.map.memory.mb = 768
yarn.nodemanager.resource.memory-mb = 12288
yarn.scheduler.maximum-allocation-mb = 12288 (same as above)
You can simply divide the later with former setting to get the maximum number of mappers supported by one m1.xlarge node = (12288/768) = 16
So, for the 5 node cluster , it would a max of 16*5 = 80 mappers that can run in parallel (considering a map only job). The same is the case with max parallel Reducers(30). You can do similar math for a combination of mappers and reducers.
So, If you want to run more mappers in parallel , you can either re-size the cluster or reduce the mapreduce.map.memory.mb(and its heap mapreduce.map.java.opts) on every node and restart NM to
To understand what the above mapred-site.xml properties mean and why you do need to do those calculations , you can refer it here :
https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
Note : The above calculations and statements are true if EMR stays in its default configuration using YARN capacity scheduler with DefaultResourceCalculator. If for example , you configure your capacity scheduler to use DominantResourceCalculator, it will consider VCPU's + Memory on every nodes (not just memory's) to decide on parallel number of mappers.