I have a query that runs against a SQL Server instance that takes anywhere from 5 minutes to 75 minutes to complete. The response size is anywhere from a few rows to 1GB of data. I have a parquet writer that only has to wait until the query completes and results sent back and it will write the results to Google Cloud Storage.
What would be the best product to accomplish this, and is their one that would have roughly zero startup time? The two that came to mind for me were Cloud Functions and Cloud Run, but I've never used either.
Neither service meets your requirement of 75 minutes.
Cloud Functions times out at 540 seconds.
Cloud Functions Time Limits
Cloud Run times out at 60 minutes.
Cloud Run Request Timeout
For that sort of runtime, I would launch a container in Compute Engine Container-Optimized OS.
Container-Optimized OS
There is the possibility of configuring Cloud Run cpu-throttling so that you can run tasks in the background.
Run more workloads on Cloud Run with new CPU allocation controls
Note that you will be paying for the service on a constant basis as it is no longer running services (containers) on demand.
Related
I am relatively new to programming and beginner at AWS cloud. I was wondering if there was a service\way that normally my code runs on CPU instance like t3.medium in AWS but when a certain condition is true (like a file gets available in bucket) then runs my function on a GPU instance like g4dn.xlarge. I want to do this so that I don't incur charges for continuously running my code on a GPU instance. The function that needs to be run is taxing as it takes up-to 7 minutes on google colab. I might be using it 20-30 times a day. Thanks.
I am running Job Array on the AWS Batch using Fargate Spot environment.
The main goal is to do some work as quickly as possible. So, when I run 100 jobs I expect that all of these jobs will be run simultaneously.
But only approx 25 of them start immediately, the rest of jobs are waiting with RUNNABLE status.
The jobs run on compute environment with max. 256 CPU. Each job uses 1 CPU or even less.
I haven't found any limits or quotas that can influence the process of running jobs.
What could be the cause?
I've talked with AWS Support and they advised me not to use Fargate when I need to process a lot of jobs as quick as possible.
For large-scale job processing, On-Demand solution is recommended.
So, after changed Provisioning model to On-Demand number of concurrent jobs grown up to CPU limits determined in settings, this was what I needed.
I have deployed a selenium script on ECS Fargate which communicates with my server through API. Normally almost 300 scripts run at parallel and bombard my server with api requests. I am facing Net::Read::Timeout error because server is unable to respond in a given time frame. How can I limit ecs tasks running at parallel.
For example if I have ran 300 scripts, 50 scripts should run at parallel and remaining 250 scripts should be in pending state.
I think for your use case, you should have a look at AWS Batch, which supports Docker jobs, and job queues.
This question was about limiting concurrency on AWS Batch: AWS batch - how to limit number of concurrent jobs
Edit: btw, the same strategy could maybe be applied to ECS, as in assigning your scripts to only a few instances, so that more can't be provisioned until the previous ones have finished.
I am unclear how your script works and there may be many ways to peal this onion but one way that would be easier to implement assuming your tasks/scripts are long running is to create an ECS service and modify the number of tasks in it. You can start with a service that has 50 tasks and then update the service to 20 or 300 or any number you want. The service will deploy/remove tasks depending on the task count parameter you configured.
This of course assumes the tasks (and the script) run infinitely. If your script is such that it starts and it ends at some point (in a batch sort of way) then probably launching them with either AWS Batch or Step Functions would be a better approach.
I have several GCP VMs running and each has a crontab to query a RESTful API every minute. For about two hours, they were all doing this just fine, when all of the sudden, all of their Packet/sec rates plummeted at the same time. They each were at about 1000 packets/sec when this happened. They are all still running, just much much slower. Does google cloud limit this? I could not find anything in their documentation to indicate otherwise.
I am new to AWS and I am completely lost. I am losing hope.
What I want to do seems quite easy, but I cannot find any clear streamlined documentation.
I would like to have my current web server call some API with an http request, and get some value returned from the API. The values to return are computed by some Python script.
Now lambda would do the job, but I need to run it on an EC2 because of computational demand (16 cores needed).
On top of that, this would need to scale: process each api request on a different machine with its own 16 cores. If I have 100 calls, I should get 100 EC2 running in parallel.
When there is no API call, I should have 0 EC2 running and $0.00 charged.
Normally, for a given session, several calls will come in a period of about 10 mins, and these calls should be answered quickly. What I am thinking is maybe having a special first call that starts an EC2 for 20 mins. So that first call can take 20 seconds, but the subsequent ones should be very quickly answered.
Summary of what I need AWS to do:
- Receive "start" API Call (Amazon API Gateway ?)
- Start an EC2 specifically for that session for 20 mins
- Return EC2 address (?)
- Call script on EC2 through http requests
- Stop EC2 after 20 mins or after the last call
What would be the services to set up and configure for that ?
I can create a docker image with the script if needed. (is there actually another way ?)
Is it possible ? How ?
Thanks a lot
Edit: modified a bit the question to reflect time constraints.
It seems to me that you should look into AWS batch. This allows you to run a batch of jobs and you can use machines with more vCPUs and benefit from per second billing.
https://aws.amazon.com/batch/