I installed:
- rundeck - 2.10.2
- rundeck-ec2-nodes-plugin - 1.5.5
I connected my jobs with existing aws EC2 instances on my account and it works great.
I search a lot but i cannot find answer for my question:
Can i use rundeck to provision EC2 instance as a node and then execute job on it and ofcourse after job is finish terminate this EC2 instance automatically??
For now you can just wrap the aws cli to create a new EC2 instances (for example using a custom script plugin).
https://docs.aws.amazon.com/cli/latest/userguide/cli-ec2-launch.html#launching-instances
If you want to create a new instance and runs some process on that instances on the same workflow, probably you need to call the "Refresh Project Nodes" steps to refresh the node resources.
Luis
Related
Problem:
I have an EC2 instance running and I have made some modifications to the instance: installed docker, setup directories for certs, etc. Now, I am wanting to create the same instance but use infrastructure as code principals. Instead of remembering all the additions that I have done and creating a template by hand, I am trying to find a way to export my current EC2 instance into a json or yaml format so that I can terminate this instance and create another one that is equivalent to the one running.
I have tried:
aws ec2 describe-instances
Reading through the AWS CLI EC2 docs
Reading through the CloudFormation docs
Searched Google
Searched SO
Since you have no knowledge of how the instance was setup, the only choice is to create an Amazon Machine Image (AMI). This will create an exact copy of the disk, so everything you have installed will be available to any new instances launched from the AMI. The CloudFormation template can then be configured to launch instances using this AMI.
If, on the other hand, you knew all the commands that needed to be run to configure the instance, then you could provide a User Data script that would run when new instances first boot. This would configure the instances automatically and is the recommended way to configure instances because it is easy to modify and allows instances to launch with the latest version of the Operating System.
Such a script can be provided as part of a CloudFormation template.
See: Running commands on your Linux instance at launch - Amazon EC2
One option would be to create AMI from live instance and spin up new CF stack using the AMI.
Other would be importing resource: https://aws.amazon.com/blogs/aws/new-import-existing-resources-into-a-cloudformation-stack/
There is a tool (still in beta) developed by AWS called CloudFormer:
CloudFormer is a template creation beta tool that creates an AWS CloudFormation template from existing AWS resources in your account. You select any supported AWS resources that are running in your account, and CloudFormer creates a template in an Amazon S3 bucket.
The CloudFormer is an AWS managed template. Once you launch it, the template will create an AWS::EC2::Instance for you along with a number of other related resources. You will access the instance using URL through browser, and an AWS wizard will guide you from there.
Its tutorial even shows how to create a CloudFormation template from an existing EC2 instance.
Import the EC2 instance into CloudFormation then copy it’s template.
Read more: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/resource-import.html
Background:
We have several legacy applications that are running in AWS EC2 instances while we develop a new suite of applications. Our company updates their approved AMI's on a monthly basis, and requires all running instances to run the new AMI's. This forces us to regularly tear down the instances and rebuild them with the new AMI's. In order to comply with these requirements all infrastructure and application deployment must be fully automated.
Approach:
To achieve automation, I'm using Terraform to build the infrastructure and Ansible to deploy the applications. Terraform will create EC2 Instances, Security Groups, SSH Keys, Load Balancers, Route 53 records, and an Inventory file to be used by Ansible which includes the IP addresses of the created Instances. Ansible will then deploy the legacy applications to the hosts supplied by the Inventory file. I have a shell script to execute the first the Terrafrom script and then the Ansible playbooks.
Question:
To achieve full automation I need to run this process whenever an AMI is updated. The current AMI release is stored in Parameter store and Terraform can detect when there is a change, but I still need to manually trigger the job. We also have an AWS SNS topic to which I can subscribe to receive notification of new AMI releases. My initial thought was to simply put the Terraform/Ansible scripts on an EC2 instance and have a Cron job run them monthly. This would likely work, but I wonder if it is the best approach. For starters, I would need to use an EC2 instance which itself would need to be updated with new AMI's, so unless I have another process to do this I would need to do it manually. Second, although our AMI's could potentially be updated monthly, sometimes they are not. Therefore, I would sometimes be running the jobs unnecessarily. Of course I could simply somehow detect if the the AMI ID has changed and run the job accordingly, but it seems like a better approach would be to react to the AWS SNS topic.
Is it possible to run the Terrafrom/Ansible scripts without having them on a running EC2 instance? And how can I trigger the scripts in response to the SNS topic?
options i was testing to trigger ansible playbook in response to webhooks from alertmanager to have some form of self healing ( might be useful for you)
run ansible in aws lambda and have it frontend with api gaetway as webhook .. alertmanager trigger -> https://medium.com/#jacoelho/ansible-in-aws-lambda-980bb8b5791b
SNS receiver in AWS -> susbscriber-> AWS system manager - which supports ansible :
https://aws.amazon.com/blogs/mt/keeping-ansible-effortless-with-aws-systems-manager/
Alertmanager target jenkins webhook -> jenkins pipline uses ansible plugin to execute playbooks :
https://medium.com/appgambit/ansible-playbook-with-jenkins-pipeline-2846d4442a31
frontend ansible server with a webhook server which execute ansible commands as post actions
this can be flask based webserver or this git webhook provided below :
https://rubyfaby.medium.com/auto-remediation-with-prometheus-alert-manager-and-ansible-e4d7bdbb6abf
https://github.com/adnanh/webhook
you can also use AWX ( ansible tower in opensource form) which expose ansible server as a api endpoint ( webhook) - currently only webhooks supported - github and gitlab.
I've faced with the problem while using AWS SDK. Currently I am using SDK for golang, but solutions from other languages are welcome too!
I have ECS cluster created via SDK
Now I need to add EC2 containers for this cluster. My problem is that I can't use Amazon ECS Agent to specify cluster name via config:
#!/bin/bash
echo ECS_CLUSTER=your_cluster_name >> /etc/ecs/ecs.config
or something like that. I can use only SDK.
I found method called RegisterContainerInstance.
But it has note:
This action is only used by the Amazon ECS agent, and it is not
intended for use outside of the agent.
It doesn't look like working solution.
I need to understand how (if it's possible) to create working ECS clusterusing SDK only.
UPDATE:
My main target is that I need to start specified count of servers from my Docker image.
While I am investigating this task i've found that I need:
create ECS cluster
assign to it needed count of ec2 instances.
create Task with my Docker image.
run it on cluster manually or as service.
So I:
Created new cluster via CreateCluster method with name "test-cluster".
Created new task via RegisterTaskDefinition
Created new EC2 instance with ecsInstanceRole role with ecs-optimized AMI type, that is correct for my region.
And there place where problems had started.
Actual result: All new ec2 instances had attached to "default" cluster (AWS created it and attach instance to it).
If I am using ECS agent I can specify cluster name by using ECS_CLUSTER config env. But I am developing tool that use only SDK (without any ability of using ECS agent).
With RegisterTaskDefinition I haven't any possibility to specify cluster, so my question, how I can assign new EC2 instance exactly to specified cluster?
When I had tried to just start my task via RunTask method (with hoping that AWS somehow create instances for me or something like that) I receive an error:
InvalidParameterException: No Container Instances were found in your cluster.
I actually can't sort out which question you are asking. Do you need to add containers to the cluster, or add instances to the cluster? Those are very different.
Add instances to the cluster
This is not done with the ECS API, it is done with the EC2 API by creating EC2 instances with the correct ecsInstanceRole. See the Launching an Amazon ECS Container Instance documentation for more information.
Add containers to the cluster
This is done be defining a task definition, then running those tasks manually or as services. See the Amazon ECS Task Definitions for more information.
I have amazon ec2 instance with mapped to 1 volume of data.
This instance running my http and have my server code.
now i have to scale my app with creating new instance and load balancing.
But if i create new instance with cloning existing instance how can the code and http vhost file will be in sync.
Using snapshot i close instance first time.
But i want when one instance i upload my code that should sync with other instance also.
How can i achieve this? should i need to configure rsync from 1 instance to another instance?
"Baking" custom AMIs is a very simple way to do this. Start a new instance from your AMI (start with a snapshot of your current instance), make changes on it like update application/configurations/system, test, create new AMI from it, start new instances from that new AMI, test them and then swap old instance in the ELB with the new ones.
There are also many tools you can use to automate your application deployment like Puppet, Chef or one of Amazons offerings: CodeDeploy, OpsWorks, Elastic Beanstalk and I recommend you use one such tool eventually.
From your description you cloned your first web server (www1) to make a second web server (www2).
Now when you make code edits you want the code to be in sync between the two webservers.
Rsync can help to make that easy.
From the 2nd web server (www2)
rsync -chavzP --stats username#IPorNAMEofwww1:/path/to/copy/on/www1 /path/to/putfiles/on/www2
Once you get that working from the command line. Add it to a cronjob so it syncs on a schedule (hourly). It should only sync the changes, not every file.
I use CodeShip to deploy my app to AWS EC2 instances when a new app version is out.
CodeShip first packages my app into .zip and puts it in S3.
Each deployment package is being generated with different file name.
When I deploy with CodeShip, they create a new revision in CodeDeploy and deploys it to all current instances.
On the one direction, every time I deploy new version it's being deployed to all my already up instances, but how can I deploy the latest revision of my app from CodeDeploy to the new born instance?
Is there any way using the aws-sdk (Ruby or cli) to achieve this?
I use OpsWorks to maintain my instances and I use custom Chef cookbook to build the environment.
Thanks
Since you have a group of EC2 instances which you wish to have the same application running, consider to use the AutoScaling Group (ASG). Then, create a deployment group based on the ASG (not EC2 instance tags). Next time when any new machine is launched in this ASG, Code Deploy will automatically run and deploys the last successful deployment package. I've been using this method for many months now and it works perfectly.
From CodeDeploy http://aws.amazon.com/codedeploy/faqs/ , if you go to the concepts section and look at question "What is a deployment group" , in the answer you will get more idea about how deployment group works.
CodeDeploy uses the tags in the deployment group to find EC2 instances when creating a new deployment. So for new born instances, basically you just need to tag them with the same tag(s) in that deployment group, and then kick off a new deployment with the revision you want. CodeDeploy will find all the EC2 instances that are tagged with the tags in the deployment group.
But you should notice this, if you want to manually start up a new EC2 instance, there are several things you need to do. You can follow the steps here: http://docs.aws.amazon.com/codedeploy/latest/userguide/how-to-prepare-instances.html to launch a instance that works with CodeDeploy.
After the instance has been tagged and set up correctly, you can just kick off a new deployment with the latest revision as the current way you do it. The revision should be deployed to the new instance by CodeDeploy automatically.