Say you want to run
a webserver (which can be nginx + uwsgi)
filebeat to collect logs (it basically watches logfiles of nginx and send the log contents to another host specified in the configuration)
I can run the two services with docker-compose.
Question is how can I run the two services on ec2-based ECS
ECS task seems to be associated with a docker image.
And separate task (web task / filebeat task) seems to be run on separate EC2.
They need to run on the same ec2 host (because filebeat needs to collect logs outputted by webserver)
How can I achieve that?
Related
I configured an AWS Fargate cluster with a docker image that runs nodejs express app and listens on port 80. Now I can browse to the public IP and successfully the request is handled by AWS Fargate.
Is it right that the docker container now is running and still waiting for requests?
Isn't it consuming CPU and so I have to pay as long as the docker container is running?
Do I have to build a docker image that just handles a single request and exits to be really serverless?
Thank you
Is it right that the docker container now is running and still waiting for requests? Isn't it consuming CPU and so I have to pay as long as the docker container is running?
Yes, that's how ECS Fargate works. It's really no different from running a docker container on your local computer. It has to be up and running all the time in order to handle requests that come in.
Do I have to build a docker image that just handles a single request and exits to be really serverless?
The term "serverless" is a vague marketing term and means different things depending on who you ask. Amazon calls ECS Fargate serverless because you don't have to manage, or even know the details of, the server that is running the container. In contrast to ECS EC2 deployments, where you have to have EC2 servers up and running ahead of time and ECS just starts the containers on those EC2 servers.
If you want something that only runs, and only charges you, when a request comes in, then you would need to reconfigure your application to run on AWS Lambda instead of ECS Fargate.
I'm running web service (nginx - uwsgi) on ECS.
I'm running the two applications using supervisor.
Now I want to add another service (filebeat) which will read logs of the web servers and send to logstash on another machine.
I've been told it is good idea to separate applications (all applications run on it's own docker container and get rid of supervisor)
So I'm trying to add a filebeat container to the already running webserver
If I go to define task tab of ECS menu, it seems I'm launching a new ec2 / fargate instance, that's not what I want.
Because filebeat has to run on the same host as the webserver
How do I run filebeat docker container along with webserver container?
I am trying to scale a flask micro service in AWS ECS to handle production workloads. My application is using the flask-apschedueler to handle long running tasks. I am using uwsgi web server for deployment in ECS so I am packing the application inside the container along with uwsgi server. The nginx container is running separately on ECS cluster.
My uwsgi config uses a single process, single thread right now.
I have successfully deployed it on AWS ECS but wondering what to scale for handling production workloads. I am debating between these options
1) I can spin up multiple containers and nginx would round robin to all of them distributing requests equally through route 53 dns service
2) I can increase the number of processes in uwsgi config but that messes with my flask-apscheduler as I only need one instance of it running. The workarounds I found are not that neat
It would be great if someone can share how to go about this
The docker mentality is more along the lines of 'one process per task'. Anytime you have more than one task running on a container, you should rethink.
I would advise the first approach. Create a service to wrap your task in ECS and simply vary the 'Desired' number of tasks for that service to scale the service up and down as desired.
If you only need the scheduler running on one of the tasks, you should setup a separate service using the same image, but with an environment variable to tell your container to start the scheduler. Make it true on the scheduler service/task and false on the worker service/tasks. Those ENV variables can be set on the container definition inside your ECS task definition.
This would be the "docker way".
I'm new to aws and am having some issues with getting my mobile app back running again. Forgive me if this question seems vague.
For a school project we created a mobile app on aws and deployed using docker containers (another student managed these tasks). When trying to get my own key pair to ssh into my ec2 instance i detached the volume associated with my instance and reattached it after getting my own key pair. Now i can ssh into my instance but my front end cant talk to my web server.
So my question is, do i create a new application on elastic beanstalk to deploy my app? Even though when i run lsblk is shows a have a docker loop device and when i run docker images i see several that match the name of my application? or do i somehow get the container running again, docker run doesn't seem to be working.
No need, just upload a new update into Elastic Beanstalk. AWS will handle the rest.
FYI, Elastic Beanstalk - Single Docker Container update process (simple under the hood):
You upload the update into AWS.
AWS will put it on your S3.
Inside your EC2, there is an Elastic Beanstalk agent. It will check for a new update.
If there is an update, the agent will download the update file and extract it.
The agent will build a new Docker image.
If the build is success, it will generate a new config to tell Nginx (web proxy) the new web server container.
Nginx will be reloaded.
Your old docker container will be destroyed.
Don't change anything inside EC2 of Elastic Beanstalk, except you know what you do. Elastic Beanstalk is design for automate deployment and scaling. So, if you change something in your EC2 manually, it might be lost. Of course, you can modified your EC2 instance, but you need to automate it using .ebextensions or take an image.
Was wondering if someone can help with below amazon AWS question, seems a basic item but can't find any answers, getting very frustrated.
1) I have an EC2 instance running that has a third party process running in the background, and when called from command line it spits out a number.
2) I have a java web app that runs this command line and uses the output for the web gui etc..
But for the life of me, i cannot figure out how to deploy my java web app on the SAME existing EC2 that's running the process, every time i try to create an elastic beanstalk it creates a new EC2 instance.
How do i make the elastic beanstalk to run off the same existing EC2 i already have? I understand there are other workaround to pass the data remotely but this seems a fundamental requirement that is missing from AWS - that you cannot run your web app with backend/batch processes on the same EC2 instance?
Thank you
Elastic Beanstalk is basically a higher abstraction layer on EC2 and it's tightly coupled with it. That means at a minimum every time you deploy your application it will spin up an EC2 server.
The advantage is that you don't need to manage your EC2 instances, for example it will autoscale automatically depending on your traffic demand.
The disadvantage is that it theoretically doesn't allow you to tweak little things in the EC2 instance because you may mess up how the Elastic Beanstalk interprets your app. Also, I believe you cannot force your Elastic Beanstalk deployment to use a specific AMI.
If you want more flexibility in your app (which sounds like your do) I recommend do your own deployment for your application (No Elastic Beanstalk). That way you can run the your app and your jobs on the same EC2 Instance.
You can use custom AMI with Elastic Beanstalk.
AWS documentation has a guide on how to create and use a customized AMI: Using Custom AMIs
But then again, nobody's stopping you from running your background processes on the standard Elastic Beanstalk instance. I run background cron jobs and Flask application on one Elastic Beanstalk instance.
files:
"/tmp/cronjob-for-foobar" :
mode: "000777"
owner: ec2-user
group: ec2-user
content: |
# skip
# clean up files created by above cronjob
30 23 * * * rm $HOME/cron*.log
container_commands:
70-foobar-cronjobs:
command: crontab /tmp/cronjob-for-foobar
Obviously, you can have anything scheduled in cron, as long as you stay within your instance limits.