I have a cluster "my-cluster"
If I try and add an ECS instance, there are non available. However, if I create a cluster "default", then I have an instance available.
I have deleted the file /var/lib/ecs/data/ecs_agent_data.json as suggested here:
Why can't my ECS service register available EC2 instances with my ELB?
Where can I change my instance/load balancer to allow me to use an EC2 instance in "my-cluster" rather than having to use the "default" cluster?
Per the ECS Agent Configuration docs:
If you are manually starting the Amazon ECS container agent (for non-Amazon ECS-optimized AMIs), you can use these environment variables in the docker run command that you use to start the agent with the syntax --env=VARIABLE_NAME=VARIABLE_VALUE. For sensitive information, such as authentication credentials for private repositories, you should store your agent environment variables in a file and pass them all at once with the --env-file path_to_env_file option.
One of the environment variables in the list is ECS_CLUSTER. So start the agent like this:
docker run -e ECS_CLUSTER=my-cluster ...
If you're using the ECS-optimized AMI you can use an alternative approach as well.
Related
I am creating a ECS cluster (EC2 launch type) using the ecs-cli. I want to run a script to modify vm.max_map_count setting in /etc/sysctl.conf once the EC2 instance is created. At the moment, I am doing it manually by ssh'ing into the instance and running the script as sudo.
Is it possible to run automation script on the EC2 instance created as part of cluster creation? Any reference/documentation will be really helpful.
Thanks
Since you've tagged your question with amazon-cloudformation I assume that you are defining your ECS container instances using CFN.
If so, you can use UserData in your AWS::EC2::Instance to execute commands when the instances are launched:
Running commands on your Linux instance at launch
You are probably already using it to specify cluster name for the ECS agents running on your instances. So probably you already have something similar in your UserData;
echo ECS_CLUSTER=${ClusterName} >> /etc/ecs/ecs.config
echo ECS_BACKEND_HOST= >> /etc/ecs/ecs.config
You can extend the UserData with extra commands that would modify /etc/sysctl.conf.
There are some other possibilities, such as using SSM State Manager to perform actions when your instances launch.
I launched an ECS Optimised instance in ap-south-1 region of AWS from ami id: ami-0a8bf4e187339e2c1 using the link https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-optimized_AMI.html but there is no ecs agent present. Even /var/log/ecs directory is not present so I cannot check logs. I have correct cluster name configured in /etc/ecs/ecs.config
If you look at the instances in the EC2 console in AWS, can you see the AMI ID? Is it the AMI ID you expect?
Just to have a point of comparison, I just SSH'd to an ECS-optimized EC2 instances and I can see ecs-agent in a docker ps listing, I can see /var/log/ecs, so my first instinct is that this EC2 instance didn't end up using the AMI you expected it to.
If you want to check logs go to tasks and click on the task in which you wan to see logs and then click on logs yo will see the logs of your container.
I'm using Jenkins Pipeline and Packer to create AMI inside an AWS Account.
The Jenkins uses Kubernetes cluster as slave (using a cloud plugin that allows me to parameter docker pods template),
I have a pipeline that pull git project with the packer template in it and run packer validate command which is a success. Than, it runs packer build and i get the following error:
[1;31mBuild 'Amazon Linux 2 Classic' errored: No valid credential sources found for AWS Builder. Please see https://www.packer.io/docs/builders/amazon.html#specifying-amazon-credentials for more information on providing credentials for the AWS Builder.[0m
I also use Kube2iam to provide roles on my slave containers.
In my packer template, i don't define any aws credentials since I don't want to use it but role. Do you know if I have something to do inside the packer template to indicate the role to use ?
Best Regards,
Tony.
From what I understand, you are running Jenkins inside a Kubernetes cluster running on AWS EC2 instances? If so, the Jenkins agents running the build should be able to read available roles from the metadata of the instance they're running on.
In this case, the process would be to assign the desire IAM role to instances and Kubernetes should be able to handle that.
I have an EC2 with a role that gives it full control over others EC2.
This role allows calling aws ec2 ... without doing the aws configure step.
However, if I install docker and run a docker container inside that EC2, this container is not able to do the aws ec2 ... without configuring the awscli.
Is there some kind of folder to share of feature to enable in order to run awscli commands inside my container without configuring it with an accesskey/password ?
The aws command is utilizing the IAM instance profile assigned to the EC2 instance, which it is obtaining via the EC2 metadata service. You would need to share that metadata with the Docker container somehow.
Are you using the AWS ECS service? Or are you manually installing and managing docker on an EC2 instance? ECS handles this for you.
Otherwise you might look into something like this Lyft project designed to proxy the EC2 IAM role to the Docker container.
Does anyone have an set of instructions for creating an AWS AMI from a chef 12, Linux based OpsWorks instance?
AWS publishes instructions on Creating a Custom Linux AMI from an AWS OpsWorks Instance. But it looks like they are out of date for Chef 12, Linux based OpsWorks stacks.
For example, they do not say you should remove the /opt/chef or /var/chef folders.
Here are the current instructions from AWS:
Create a Custom Linux AMI from an AWS OpsWorks Instance
If you want to use a customized AWS OpsWorks Linux instance to create an AMI, you should be aware that every Amazon EC2 instance created by OpsWorks includes a unique identity. If you create a custom AMI from such an instance, it will include that identity and all instances based on the AMI will have the same identity. To ensure that the instances based on your custom AMI have a unique identity, you must remove the identity from the customized instance before creating the AMI.
To create a custom AMI from an AWS OpsWorks instance
Create a Linux stack and add one or more layers to define the configuration of the customized instance. You can use built-in layers, customized as appropriate, as well as fully custom layers. For more information, see Customizing AWS OpsWorks.
Edit the layers and disable AutoHealing.
Add an instance with your preferred Linux distribution to the layer or layers and start it. We recommend using an Amazon EBS-backed instance. Open the instance's details page and record its Amazon EC2 ID for later.
When the instance is online, log in with SSH and run the following commands, in order:
sudo /etc/init.d/monit stop
sudo /etc/init.d/opsworks-agent stop
sudo rm -rf /etc/aws/opsworks/ /opt/aws/opsworks/ /var/log/aws/opsworks/ /var/lib/aws/opsworks/ /etc/monit.d/opsworks-agent.monitrc /etc/monit/conf.d/opsworks-agent.monitrc /var/lib/cloud/
If you are creating an AMI based on Amazon Linux 2014.09, run rpm -e opsworks-agent-ruby to ensure that the agent is running.
If you are creating an AMI based on Ubuntu, run dpkg -r opsworks-agent-ruby to ensure that the agent is running.
This step depends on the instance type:
For an Amazon EBS-backed instance, use the AWS OpsWorks console to stop the instance and create the AMI as described in Creating an Amazon EBS-Backed Linux AMI..
For an instance store-backed instance, create the AMI as described in Creating an Instance Store-Backed Linux AMI and then use the AWS OpsWorks console to stop the instance.
When you create the AMI, be sure to include the certificate files. For example, you can call the ec2-bundle-vol command with the -i argument set to -i $(find /etc /usr /opt -name '.pem' -o -name '.crt' -o -name '*.gpg' | tr '\n' ','). Do not remove the apt public keys when bundling. The default ec2-bundle-vol command handles this task.
Clean up your stack by returning to the AWS OpsWorks console and deleting the instance from the stack.