I'm using Gitlab shared runner with Docker (current runner version: 10.0.2, docker storage driver: overlay2), running on AWS t2.small instance. I started experiencing issues with builds slowing down after some time (it's hard to say when exactly they become slow) - they take ~10x more time to finish than before. After killing the instance problem disappears for a while and after some time it slows down again.
Things I already checked:
CPU usage on machine is around 20% the whole time
RAM usage is around 1,5 G during the heaviest build
IOPS on EBS are not exhausting all Burst Balance (e.g. right now burst balance is around 80%)
Download speed
What else might be causing this ?
Just in case, jobs that are running on this runner are mostly yarn install and yarn build of a medium-sized front-end React application.
You mention you're using a t2.small, what's the CPU credit balance on your instance when you see this slow-down?
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/t2-instances.html#t2-instances-cpu-credits
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/t2-instances.html#t2-instances-monitoring-cpu-credits
Related
I have a use case where I need to start EC2 instances on-demmand, so starting fast is relevant to our users. Currently our startup time is 2 minutes on average, varing according time of the day and instance type.
We are lauching it using the NodeJS SDK and straight from our custom AMI, not using lauching templates, and we noted that smaller image sizes launch faster, but unfortunately we are unable to reduce it.
When the instance starts I have a #reboot cronjob with an application that notifies an api that the instance is ready, everything is installed in the AMI,built for this purpose based on ubuntu 18, and no hard work is done when it starts. We are measuring this startup time based as the difference of the time when it was started and the time it notified is ready.
The startup time is higher when no instances with that AMI were launched recently, suggesting that AWS has some kind of cold start in this case. We also noticed that increasing the disk size from 30gb to 45gb increased this startup from 1 min to the 2 min average that I mentioned.
What strategies may I try to reduce this startup time?
My team had a monolithic service for a small scale project but for a re-architecture and scaling, we are planning to move to cloud services of Amazon AWS and evaluating for orchestration whether to run Luigi as a container task or use AWS Step Functions instead? I don't have any experience with any of them especially Luigi.
Can anyone point out any issues that they have seen with Luigi or how it can prove to be better than AWS if at all? Any other suggestions for the same.
Thanks in advance.
I don't know about how AWS does orchestration, but if you are planning to at any time scale to at least thousands of jobs, I would not recommend investing in Luigi. Luigi is extremely useful for small to medium(ish) projects. It provides a fantastic interface for defining jobs and ensuring job completion through atomic filesystem actions. However, the problem when it comes to Luigi is the framework for running jobs. Luigi requires constant communication to workers for them to run, which in my own experience destroyed network bandwidth when I tried to scale.
For my research, I will generate a network of 10,000 tasks on a light to medium workflow, using my university's cluster computing grid which runs SLURM. All of my tasks don't take that long to complete, maybe 5 min max each. I have tried the following three methods to use Luigi efficiently.
SciLuigi's slurm task to submit jobs to SLURM from a central luigi worker (not using central scheduler). This method works well if your jobs will be accepted quickly and run. However, it uses an unreasonable amount of resources on the scheduling node, as each worker is a new process. Further, it destroys any priority you would have in the system. A better method would be to first allocate many workers and then have them continually work on jobs.
The second method I attempted was just that. I started the Luigi central scheduler on my home server (because otherwise I could not monitor the state of work, just like in the above workflow) and started up workers on the SLURM cluster that all had the same configuration, so each of them could run any part of the experiment. The problem was, even with 500Mbps internet, past ~50 workers Luigi would stop functioning and so would my internet connection to my server. So, I began running jobs with only 50 workers, which drastically slowed my workflow. In addition, each worker had to register each job with the central scheduler (another huge pain point), which could take hours with only 50 workers.
To mitigate this startup time I decided to partition the root-task subtrees by their parameters and submit each to SLURM. So now the startup time is reasonably low, but I lost the ability for any worker to run any job, which is still pretty important. Also, I can still only work with ~50 workers. When I completed the subtrees, I ran one last job to finish the experiment.
In conclusion, Luigi is great for small to medium-small workflows, but once you start hitting 1,000+ tasks and workers, the framework quickly fails to keep up. I hope that my experiences provide some insight into the framework.
I have a persistent server that unpredictably receives new data from users, needing about 10 GPU instances to crank at the problem for about 5 minutes, and I send the answer back to the users. The server itself is a cheap always-persistent single CPU Google Cloud instance. When a user request comes in, my code launches my 10 created but stopped Google Cloud GPU instances with
gcloud compute instances start (instance list)
In the rare case if the stopped instances don't exist (sometimes they get wiped) that's detected and they're recreated with
gcloud beta compute instances create (...)
This system all works fine. My only complaint is that even with created but stopped instances, the launch time before my GPU code finally starts to run is about 5 minutes. Most of this is just the time for the instance itself to launch its Ubuntu host and call my code.. the delay once Ubuntu is running to start the GPU is only about 10 seconds.
How can I reduce this 5 minute delay? I imagine most of it comes from Google having to copy over the 4GB of instance data to the target machine, but the startup time of (vanilla) Ubuntu adds probably 1 more minute. I'm not even sure if I could quantify these two numbers independently, I only can measure the combined 3-7 minutes delay from the launch until my code starts responding.
I don't think Ubuntu OS startup time is the major startup latency contributor since I timed an actual machine with the same Ubuntu and same GPU on my desk from poweron boot up and it began running my GPU code in 46 seconds.
My goal is to get results back to my users as soon as possible, and that 5 minute startup delay is a bottleneck.
Would making a smaller instance SIZE of say 2GB help? What else can I do to reduce the latency?
2GB is large. That's a heckuva big image. You should be able to cut that down to 100MB, perhaps using Alpine instead of Ubuntu.
Copying 4GB of data is also less than ideal. Given that, I suspect the solution will be more of an architecture change than a code change.
But if you want to take a whack at everything which is NOT about your 4GB of data, there is a capability to prepare a custom image for your VMs. If you can build a slim custom image that will help.
There's good resources for learning more, the two I would start with include:
- Improve GCE Boot Times with Custom Images
- Three steps to Compute Engine startup-time bliss: Google Cloud Performance Atlas
We are running a docker container on AWS elastic beanstalk, which was running fine for a few weeks but suddenly started to experience very sudden CPU spikes (from ~5% to ~60% in a matter of minutes), who sometimes drop back down quickly, and sometimes stay high for enough time to produce an autoscaling event and spin up a few instances for extra help (Which are terminated some time after that when the CPU spike dies down).
The funny thing is, I wanted to investigate the problem today, so I've sshed into every instance (4 in total) and ran top on all of them, trying to locate the CPU consuming process, and was surprised to discover all instances have ~15% CPU busy (system + user combined), while the EBS monitoring page still shows the servers are at 60% CPU.
I've measured these figures for the good part of the hour, making sure the CPU high load stays high, while the top command still shows low values.
I've also tried to measure CPU for a while using the advice found here -https://askubuntu.com/questions/22021/how-to-log-cpu-load, and got the same very low CPU stats when querying the server directly.
My question is - is it possible AWS monitoring system is not showing me accurate data? Is there anyway to verify the data displayed in the monitoring page?
Any help would be appreciated.
I'm trying to run a locust.io load test on an ec2 instance - a t2.micro. I fire up 50 concurrent users, and initially everything works fine, with the CPU load reaching ~15%. After an hour or so though, the network out shows a drop of about 80% -
Any idea why this is happening? It's certainly not due to CPU credits. Maybe I reached the network limits for a t2 micro instance?
Thanks
Are you sure it's not a CPU credit issue? Can you check your cpu credits over that same time period to see how they look?
Or better yet, run the same test on a non-t2 instance. One that isn't limited in it's CPU usage.
t2.micro's consume CPU credits at usage about 10% of CPU.