We are trying to automate running of multiple powershell scripts in AWS EC2 instance. Any suggestion how this can be done. These execution has to be sequential.
In order to configure state on a fresh EC2 instance, you can use 'User Data' in order to:
a. Fully provision and configure state using powershell, see:
http://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/ec2-instance-metadata.html#instancedata-add-user-data
b. Install an agent for a configuration management tool such as Puppet, Chef, Ansible (as mentioned in the comment above). An example of provisioning stock Window's AMI's can be followed here: http://blog.rolpdog.com/2015/09/manage-stock-windows-amis-with-ansible.html
If you wish to trigger scripts on an existing Amazon EC2 instance (or a whole fleet of instances), also consider Running PowerShell Commands or Scripts with Amazon EC2 Run Command.
Related
Background:
We have several legacy applications that are running in AWS EC2 instances while we develop a new suite of applications. Our company updates their approved AMI's on a monthly basis, and requires all running instances to run the new AMI's. This forces us to regularly tear down the instances and rebuild them with the new AMI's. In order to comply with these requirements all infrastructure and application deployment must be fully automated.
Approach:
To achieve automation, I'm using Terraform to build the infrastructure and Ansible to deploy the applications. Terraform will create EC2 Instances, Security Groups, SSH Keys, Load Balancers, Route 53 records, and an Inventory file to be used by Ansible which includes the IP addresses of the created Instances. Ansible will then deploy the legacy applications to the hosts supplied by the Inventory file. I have a shell script to execute the first the Terrafrom script and then the Ansible playbooks.
Question:
To achieve full automation I need to run this process whenever an AMI is updated. The current AMI release is stored in Parameter store and Terraform can detect when there is a change, but I still need to manually trigger the job. We also have an AWS SNS topic to which I can subscribe to receive notification of new AMI releases. My initial thought was to simply put the Terraform/Ansible scripts on an EC2 instance and have a Cron job run them monthly. This would likely work, but I wonder if it is the best approach. For starters, I would need to use an EC2 instance which itself would need to be updated with new AMI's, so unless I have another process to do this I would need to do it manually. Second, although our AMI's could potentially be updated monthly, sometimes they are not. Therefore, I would sometimes be running the jobs unnecessarily. Of course I could simply somehow detect if the the AMI ID has changed and run the job accordingly, but it seems like a better approach would be to react to the AWS SNS topic.
Is it possible to run the Terrafrom/Ansible scripts without having them on a running EC2 instance? And how can I trigger the scripts in response to the SNS topic?
options i was testing to trigger ansible playbook in response to webhooks from alertmanager to have some form of self healing ( might be useful for you)
run ansible in aws lambda and have it frontend with api gaetway as webhook .. alertmanager trigger -> https://medium.com/#jacoelho/ansible-in-aws-lambda-980bb8b5791b
SNS receiver in AWS -> susbscriber-> AWS system manager - which supports ansible :
https://aws.amazon.com/blogs/mt/keeping-ansible-effortless-with-aws-systems-manager/
Alertmanager target jenkins webhook -> jenkins pipline uses ansible plugin to execute playbooks :
https://medium.com/appgambit/ansible-playbook-with-jenkins-pipeline-2846d4442a31
frontend ansible server with a webhook server which execute ansible commands as post actions
this can be flask based webserver or this git webhook provided below :
https://rubyfaby.medium.com/auto-remediation-with-prometheus-alert-manager-and-ansible-e4d7bdbb6abf
https://github.com/adnanh/webhook
you can also use AWX ( ansible tower in opensource form) which expose ansible server as a api endpoint ( webhook) - currently only webhooks supported - github and gitlab.
I have a shell script. Per rule in my organization we should save all the code in GitHub and use it from there. How can I run this script in a specific set of EC2 instances from GitHub using Jenkins?
You may consider the use of AWS Systems Manger to run shell commands on your EC2 instance.
https://docs.aws.amazon.com/systems-manager/latest/userguide/execute-remote-commands.html
You can find a sample use case here: https://ujjwalbhardwaj.me/post/trigger-actions-inside-ec2-instance-on-cloudwatch-events-using-aws-systems
Is there any way to install exe/MSI agents in AWS EC2 instances in an automated way?? In specific, I am looking for a counterpart of Azure's Custom Script Extension. [Free of cost]
Scenario:
I want to install BigFix and Datadog agents on 1000 Ec2 instances, this is a one time job, so I am not looking for any solution that involves Chef / Puppet, etc.,
Yes, you can pass a script to the instance that will be executed on the first boot (but not thereafter). It is often referred to as a User Data script.
See:
Running Commands on Your Windows Instance at Launch - Amazon Elastic Compute Cloud
Running Commands on Your Linux Instance at Launch - Amazon Elastic Compute Cloud
If you wish to install after the instance has started, use the AWS Systems Manager Run Command.
I have a release of my project. I build a docker image and deploy it on an ec2 instance.
Later, when I have a new release, I would like update the docker on ec2 remotely (without accessing the machine, just executing some service).
Is there a way how to do it without ECS and ElasticBeanstalk?
If it's not possible can I somehow re-run the cfn-init script?
My Research
https://aws.amazon.com/blogs/aws/new-ec2-run-command-remote-instance-management-at-scale/
You can manage your instances remotely (i.e. make changes without manually SSHing into the instance and typing commands) by using any of the many system management services out there. AWS offers Simple Systems Manager (SSM) of which the Run Command you linked is part. AWS also offers the OpsWorks service which uses Chef. You also have other products like Ansible and SaltStack, and you can optionally integrate the use of those services with the AWS SSM service.
Is there a command in the AWS cli to get a list of services running on my ec2 instances?
Normally I log into each ec2 instance individually, and check using Linux commands such as "netstat" or "ps -eaf".
There are around 400 instances, so if I do this manually it takes quite a lot of time. If any AWS commands are there to find the services running without actually logging into the instance, it would be great.
Thanks in advance.
The AWS Command Line Interface is a tool to manage your AWS services. You cannot use AWS-CLI to manage OS-level service. With AWS-CLI, you can control multiple AWS services from the command line and automate them through scripts.
Hope it helps :-)
You might have some luck with EC2 Run Command. From their docs:
Run Command provides a simple way of automating common administrative tasks
like executing Shell scripts and commands on Linux [...]
Run Command allows you to execute these commands across multiple
instances and provides visibility into the results