Using Elastic beanstalk autoscaling for an application, recently added a new SQS queue as an environment variable.
I can see the variable using eb printenv *my-env* | grep "SQS":
SQS_QUEUE = https://sqs.xxxxxx.amazonaws.com/xxxxxxx/xxxxxxx
When I SSH to a newly created instance I can't see that config sudo /opt/elasticbeanstalk/bin/get-config environment --output YAML | grep "SQS"
If I make a code change and deploy then the instance gets the config - this isn't a practical solution when the scaling goes up / down multiple times a day.
I'm using a custom AMI but I can't find any documentation on AWS around environment variables - adding new ones should just work..!?
What am I missing??
Please double check. I think -output YAML is incorrect. The queue should be accessible through:
/opt/elasticbeanstalk/bin/get-config environment -key SQS_QUEUE
Related
Can we set Environment variable against an EC2 instance from AWS console?
Or do we need load any other service to achieve this?
Also can we load this variable insider Docker?
That appears to be out of scope for the AWS Console. EC2 however is basically AWS hosted VMs, So anything you could do normally on that OS can be done.
Just connect to the machine and do what you would normally do. (See the AWS CLI Guide for further help)
Using the AWS Console:
Stop the ec2 instance
For that ec2 instance, from top dropdown - select Actions > Instance Settings > View/Change User Data
Create a script that would add required environment variables to init scripts.
!/bin/sh echo 'export MY_ENV_VARIABLE=value' >> ~/.bashrc
or a script to /etc/init.d or other similar locations..
Start the EC2 instance for the Environment variables to be available.
If you want to pass the EC2 instance environment variable to the Docker container environment variable:
When you start a docker instance through "docker run" you can pass the value through --env param in the command line.
eg: docker run -e DOCKER_ENV_VARIABLE=$MY_ENV_VARIABLE ...
https://docs.docker.com/engine/reference/commandline/run/#options
You can use user_data to load and afterwards pass to your docker container using docker run -e CONTAINER_ENV_VAR=$EC2_ENV_VARor put in your Dockerfile.
for keys or any sensitive data I would advise you to use the Parameter Store, you can put passwords, users, or whatever data you want and use a service call chamber
I am using AWS code deploy agent and deploying my project to the server through bitbucket plugin.
The code deployment agent first executes the script files which has the command to execute my spring-boot project.
Since I have two environments one development and another production. I want the script to do things differently based on the environment i.e two different instances.
My plan is to fetch the aws static ip-address which is mapped and from that determine the environment
(production or stage).
How to fetch the elastic ip address through sh commands.
edited
Static IP will work.
Here is a more nature CodeDeploy way to solve this is - set up 2 CodeDeploy deployment groups, one for your development env and the other for your production env. Then in your script, you can use environment variables that CodeDeploy will set during the deployment to understand which env you are deploying to.
Here is a blog post about how to use CodeDeploy environment variables: https://aws.amazon.com/blogs/devops/using-codedeploy-environment-variables/
You could do the following:
id=$( curl http://169.254.169.254/latest/meta-data/instance-id )
eip=$( aws ec2 describe-addresses --filter Name=instance-id,Values=${id} | aws ec2 describe-addresses | jq .Addresses[].PublicIp --raw-output )
The above gets the instance-id from metadata, then uses the aws cli to look for elastic IPs filtered by the id from metadata. Using jq this output can then be parsed down to the IP you are looking for.
Query the metadata server
eip=`curl -s 169.254.169.254/latest/meta-data/public-ipv4`
echo $eip
The solution is completely off tangent to what I originally asked but it was enough for my requirement.
I just needed to know the environment I am in to do certain actions. So what I did was to set an environment variable in an independent script file where the environment variable is set and the value is that of the environment.
ex: let's say in a file env-variables.sh
export profile= stage
In the script file where the commands have to be executed based on the environment I access it this way
source /test/env-variables.sh
echo current profile is $profile
if [ $profile = stage ]
then
echo stage
elif [ $profile = production ]
then
echo production
else
echo failure
fi
Hope some one finds it useful
I am currently migrating my config management on AWS to Terraform to make it more pluggable. What I like is the possibility to manage rolling updates to an Autoscaling Group where Terraform waits until the new instances are in service before it destroys the old infrastructure.
This works fine with the "bare" infrastructure. But I ran into a problem when update the actual app instances.
The code is deployed via AWS CodeDeploy and I can tell Terraform to use the generated name of the new Autoscaling Group as deployment target but it doesn't deploy the code to the new instances on startup. When I manually select "deploy changes to the deployment group" the deployment starts successfully.
Any ideas how to automate this step?
https://www.terraform.io/docs/provisioners/local-exec.html might be able to do this. Couple assumptions
You've got something like aws-cli installed where you're running terraform.
You've got your dependencies setup so that your CodeDeploy step would be one of the last things executed. If that's not the case you can play with depends_on https://www.terraform.io/intro/getting-started/dependencies.html#implicit-and-explicit-dependencies
Once your code has been posted, you would just add a
resource "something" "some_name" {
# Whatever config you've setup for the resource
provisioner "local-exec" {
command = "aws deploy create-deployment"
}
}
FYI the aws deploy create-deployment command is not complete, so you'll have to play with that in your environment till you've got the values needed to trigger the rollout but hopefully this is enough to get you started.
You can trigger the deployment directly in your user-data in the
resource "aws_launch_configuration" "my-application" {
name = "my-application"
...
user_data = "${data.template_file.node-init.rendered}"
}
data "template_file" "node-init" {
template = "${file("${path.module}/node-init.yaml")}"
}
Content of my node-init.yaml, following recommendations of this documentation: https://aws.amazon.com/premiumsupport/knowledge-center/codedeploy-agent-launch-configuration/
write_files:
- path: /root/configure.sh
content: |
#!/usr/bin/env bash
REGION=$(curl 169.254.169.254/latest/meta-data/placement/availability-zone/ | sed 's/[a-z]$//')
yum update -y
yum install ruby wget -y
cd /home/ec2-user
wget https://aws-codedeploy-$REGION.s3.amazonaws.com/latest/install
chmod +x ./install
./install auto
# Add the following line for your node to update itself
aws deploy create-deployment --application-name=<my-application> --region=ap-southeast-2 --deployment-group-name=<my-deployment-group> --update-outdated-instances-only
runcmd:
- bash /root/configure.sh
In this implementation the node is responsible for triggering the deployment itself. This is working perfectly so far for me but can result in deployment fails if the ASG is creating several instances at the same time (in that case the failed instances will be terminated quickly because not healthy).
Of course, you need to add the sufficient permissions to the role associated to your nodes to trigger the deployment.
This is still a workaround and if someone knows solution behaving the same way as cfn-init, I am interested.
So I have a docker container running jenkins and an EC2 registry on AWS. I would like to have jenkins push containers back to the EC2 registry.
To do this, I would like to be able to automate the aws configure and get login steps on container startup. I figured that I would be able to
export AWS_ACCESS_KEY_ID=*
export AWS_SECRET_ACCESS_KEY=*
export AWS_DEFAULT_REGION=us-east-1
export AWS_DEFAULT_OUTPUT=json
Which I expected to cause aws configure to complete automatically, but that did not work. I then tried creating configs as per the AWS docs and repeating the process, which also did not work. I then tried using aws configure set also with no luck.
I'm going bonkers here, what am I doing wrong?
No real need to issue aws configure instead as long as you populate env vars
export AWS_ACCESS_KEY_ID=aaaa
export AWS_SECRET_ACCESS_KEY=bbbb
... also export zone and region
then issue
aws ecr get-login --region ${AWS_REGION}
you will achieve the same desired aws login status ... as far as troubleshooting I suggest you remote login into your running container instance using
docker exec -ti CONTAINER_ID_HERE bash
then manually issue above aws related commands interactively to confirm they run OK before putting same into your Dockerfile
Is there a way to do this from Beanstalk?
or do I have to do it myself once the new instance is up via .ebextensions or something like that?
This feature was not available at the time this question was asked. There were many requests for this ever since the tagging feature was first introduced in 2014.
Support for updating EB tags was eventually added in Oct 2017.
You can now manage tags in the Beanstalk Management Console, with the EB CLI or with the AWS CLI.
If you want to set the tags from within your Elastic Beanstalk instance, you can certainly SSH in, install the AWS CLI and set the tags via the command line. Or, whatever application you're deploying on Beanstalk can do so via the SDK (e.g. Java).
Via CLI:
http://docs.aws.amazon.com/cli/latest/reference/ec2/create-tags.html
create-tags [--dry-run | --no-dry-run] --resources <value> --tags <value> [--cli-input-json <value>] [--generate-cli-skeleton]
Via SDK:
http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/ec2/model/CreateTagsRequest.html
(pseudo-code)
Authorize via credentials
Instantiate CreateTagsRequest with specific resource ID
call setTags(Collection<<**String**>> tags) on the CreateTagsRequest object
From outside the environment, you can set the tags via the console or via EB CLI.
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.tagging.html
eb create --tags mytag1=value1,mytag2=value2
Note, you can also set regular environment variables via eb create.
eb create --envvars key=value
Answer
1. In order to have tags in elastic beanstalk that would be applied on any ec2 instance created for that environment - they must be set during the creation of the environment.
2. For existing environments, it's very simple to save configurations, create an alternative environment, set the tags right, and swap the URLs until you get your original env configured right.
This guy explains it well: http://www.boringgeek.com/add-or-update-tags-on-existing-elastic-beanstalk-environments/