I have been trying to use JMeter to test my server. I have a cloudsearch endpoint on AWS. I have to test if it can scale upto 25000 requests per second without failing. I have tried JMeter with a constant throughput timer with throughput = 1500000 per second and running 1000 threads. I ran it for 10 mins. But When I review the aggregate report it shows an average of only 25 requests per second. How do i get an average of around 25000 requests per second?
Constant Throughput Timer can only pause the threads to reach specified "Target Throughput" value so make sure you provide enough virtual users (threads) to generate desired "requests per minute" value.
You don't have enough Threads to achieve such Requests per second!!!
To get an average (~25000) requests per second, you have to increase the Number of threads.
Remember, The number of threads will impact results if your server faces slowdowns. If so and you don't have enough threads then you will not be injecting the expected load and end up with fewer transactions performed.
You need to increase the number of concurrent users to be at least 25000 (it assumes 1 second response time, if you have 2 seconds response time - you will need 50000)
JMeter default configuration is not suitable for high loads, it is good for tests development and debugging, however when it comes to the real load you need to consider some constraints, i.e.:
Run JMeter test in non-GUI mode
Increase JVM Heap Size and tune other parameters
Disable all listeners during the test
If above tips don't help follow other recommendations from 9 Easy Solutions for a JMeter Load Test “Out of Memory” Failure article or consider Distributed Testing
Related
We have configured AWS for distributed load testing using - https://aws.amazon.com/solutions/implementations/distributed-load-testing-on-aws/
Our requirement includes achieving 5k RPS.
Please help me in understanding the inputs that needs to be provided here
When we consider the system supports 5k RPS then - What should be the Task Count, Concurrency, Ramp Up and Hold For values in order to achieve 5k RPS using AWS DLT.
We are also trying to achieve it using jmeter concurrent threads. Hoping someone could help with values and explaining the usage for the same.
We don't know.
Have you tried read the documentation the link to which you provided yourself? I.e. for the Concurrency there is a chapter called Determine the number of users which suggests start with 200 and increase/decrease depending on the resources consumption.
The same applies to the Task Count, you may either go with a single container with default resources, increase container resources or increase the number of containers.
The number of hits per second will mostly depend on your application response time, for example given 200 recommended users if response time will be 1 second - you will have 200 RPS, if response time will be 2 seconds - you will get 100 RPS, if response time will be 0.5 seconds - you will get 400 RPS, etc. See What is the Relationship Between Users and Hits Per Second? article for more comprehensive explanation if needed. The throughput can also be controlled on JMeter side using Concurrency Thread Group and Throughput Shaping Timer but again, the container(s) must have sufficient resources in order to produce the desired load.
With regards to ramp-up - again, we don't know. Personally I tend to increase the load gradually so I could correlate increasing load with other metrics. JMeter documentation recommends starting with ramp-up period in seconds equal to number of users.
The same for the time to hold the load, i.e. after ramping up to the number of users required to conduct the load of 5K RPS I would recommend holding the load for the duration of ramp-up period to see how the system behaves, whether it stabilizes when the load stops increasing, are the response times static or they go up, etc.
I have a question about concurrent user in Jmeter.
If i use setting like this, is that means in first sec 500 thread will hit concurrently?
Then after the first hit, is the threads will repeatedly hit every minute?
It means that JMeter will start 500 threads and will keep them running for 20 minutes.
It doesn't mean that the threads will "repeatedly hit every minute"
All JMeter threads are independent, once started each thread (virtual user) starts executing Samplers upside down. When there are no more loops to iterate or samplers to execute (or test duration is exceeded) the thread is being shut down.
So in your case 500 threads will be repeatedly executing Samplers without any delay (as long as you're not using Timers anywhere) for 20 minutes.
The actual concurrency (number of requests per second) mainly depends on the nature of your test plan and your application response time, it can be checked using i.e. Transactions per Second listener
The preview graph in Concurrency Thread Group is useful to understand how it works.
The preview graph reacts immediately to changes in the fields, showing you the planned concurrency schedule.
Target Concurrency: Number of threads available at the end of the ramp-up period.
**Hold Target Rate Time **: Duration of the test after the ramp-up period. Duration of the test execution with the Target Concurrency
You have not set the ramp-up settings. Hence, JMeter will create 500 tests as the test is started and will run the test for 20 minutes.
Note: It is advisable to set a ramp-up to avoid
Unbearable load to the JMeter client machine (load agent) as the test is started
Unrealistic, sudden load to the target server
I'm refactoring a job that uploads ~1.2mln small files to AWS; previously this upload was made file by file on a 64 CPUs machine with processes. I switched to an async + multiprocess approach following S3 rate limits and best practices and performance guidelines to make it faster. With sample data I can achieve execution times as low as 1/10th. With production loads S3 is returning "SlowDown" errors.
Actually the business logic makes the folder structure like this:
s3://bucket/this/will/not/change/<shard-key>/<items>
The objects will be equally splitted across ~30 shard-keys, making every prefix contain ~40k items.
We have every process writing on its own prefix and launching batches of 3k PUT requests in async until completion. There is a sleep after the batch write operation to ensure that we do not send another batch before 1.1sec has passed, so we will respect the 3500 PUT requests per second.
The problem is that we receive SlowDown errors for ~1 hour and then the job writes all the files in ~15 minutes. If we lower the limit to 1k/sec this gets even worse, running for hours and never finishing.
This is the distribution of the errors over time for the 3k/sec limit:
We are using Python 3.6 with aiobotocore to run async.
Doing some sort of trial and error to try to understand how to mitigate this takes forever on production data and testing with a lower quantity of data gives us different results (flawlessly works).
Did I miss any documentation regarding how to make the system scale up correctly?
This is how application setup goes -
2 c4.8xlarge instances
10 m4.4xlarge jmeter clients generating load. Each client used 70 threads
While conducting load test on a simple GET request (685 bytes size page). I came across issue of reduced throughput after some time of test run. Throughput of about 18000 requests/sec is reached with 700 threads, remains at this level for 40 minutes and then drops. Thread count remains 700 throughout the test. I have executed tests with different load patterns but results have been same.
The application response time considerably low throughout the test -
According to ELB monitor, there is reduction in number of requests (and I suppose hence the lower throughput ) -
There are no errors encountered during test run. I also set connect timeout with http request but yet no errors.
I discussed this issue with aws support at length and according to them I am not blocked by any network limit during test execution.
Given number of threads remain constant during test run, what are these threads doing? Is there a metrics I can check on to find out number of requests generated (not Hits/sec) by a JMeter client instance?
Testplan - http://justpaste.it/qyb0
Try adding the following Test Elements:
HTTP Cache Manager
and especially DNS Cache Manager as it might be the situation where all your threads are hitting only one c4.8xlarge instance while the remaining one is idle. See The DNS Cache Manager: The Right Way To Test Load Balanced Apps article for explanation and details.
I have a python program which query youtube to get the video details. I use the version-3 api. I have multiple processes m and a python pool of 10 processes in each python process.
songs_pool = Pool()
songs_pool =Pool(processes=10)
return_pool = songs_pool.map(getVideo,songs_list)
I get some client errors when the value of m is increased to more than 2 and the pool is increased to >5. I get forbidden errors. When I check the number of requests in the google analytics,it shows that the number of requests are 250 per sec. But according to the documentation the limit is 3000 requests per sec. I dont understand why am I getting the client errors. Can you tell me if there is a way to not get this errors and run the program quicker.
if m = 2 and process = 10 , i get no errors but it takes so much time to complete.
But if I increase them , then I get client errors which are ~ 5% of the total requests.
The per-user-limit is 3000 requests per second from a single IP address, and as soon as you go above that in a given second you'll start getting the forbidden errors. The analytics you see in the developers console will only report your average number of requests over a 5 minute period; therefore, if you had zero requests for 4 minutes, then started running your routine, the console may show only 250 requests per second (as an average) but your app likely is overrunning the limit in a given period of time or two.
It seems that you're handling it in the best way possible if speed is your concern; you'll want to run it fast enough to get a very small number of errors (so you know you're staying up there at your limit). Another option, though, might be to look into using etags; if you find yourself requesting info on the same videos a lot, you can let etags tell you whether or not any info has changed (and if the API responds that nothing has changed, it doesn't count against either your quota or your reqests/sec.)