I have been facing issues with integrating chef with AWS autoscale.
In most of my searches it tells about bootstrapping an instance and then using it's AMI to launch other instance in just the same way.
Basic issue is, Chef recognises each host with it's hostname, which in the above case is all going to be same. However, I was hoping for something like a Role which integrates in to AWS and does the thing better for me. Any help/Ideas will be appreciated. I just hope someone has done it already.
Regards,
There are a lot of options but the general flow looks like this:
Create an AMI with Chef pre-installed and with your org validator key and a client.rb with the server URL set. Packer is great for this. Technically optional, you could do this from the user-data script, but it saves a few seconds on each server launch.
Configure the UserData field on the ASG to be a script (or cloud-init config if you want to get fancy but we'll ignore that option for now) that launches chef-client -r 'role[myrole] where myrole is usually based on the type of ASG you are building. This will use the validator key to register with the Chef Server automatically and set the run list based on the command line you give. You can use similar arguments to set the environment or policy name if you are using those features.
Include the chef-client cookbook/recipe in that role to install Chef as a daemon on the machine and to remove the validator key.
Related
Bear with me, what I am requesting may be impossible. I am a AWS noob.
So I am going to describe to you the situation I am in...
I am doing a freelance gig and was essentially handed the keys to AWS. That is, I was handed the root user login credentials for the AWS account that powers this website.
Now there are 3 EC2 instances. One of the instances is a linux box that, from what I am being told, is running a Django Python backend.
My new "service" if you will must exist within this instance.
How do I introduce new source code into this instance? Is there a way to pull down the existing source code that lives within it?
I am not be helped by any existing/previous developers so I am kind of just handed the AWS credentials and have no idea where to start.
Is this even possible. That is, is it possible to pull the source code from an EC2 instance and/or modify the code? How do I do this?
EC2 instances are just virtual machines. So you can use SSH/SCP/SFTP files to and from. You can use the AWS CLI tools to copy stuff from S3. Dealers choice...
Now to get into this instance... If you look in the web console you can find its IP(s), what the security groups (firewall rules), and the key pair name. Hopefully they gave you the keys. You need these to SSH in.
You'll also want to check to make sure there's a security group applied that has SSH open. Hopefully only to your IP :)
If you don't have the keys you'll have to create an AMI image of the instance so you can create a new one with a key pair you do have.
Amazon has a set of tools for you in Amazon CodeSuite.
The tool used for "deploying" the code is Amazon CodeDeploy. By using this service you install an agent onto your host, then when triggered it will pull down an artifact of a code base and install it matching hosts. You can even specify additional commands through the hook system.
But you also want to trigger this to happen, maybe even automatically? CodeDeploy can be orchestrated using the CodePipeline tool.
Using CloudFormation template, I brought up a Windows 2012 EC2 instance. Instance came up fine. I read that metadata related to this instance is all recorded in the Ec2config logs which is in one of the sub-folders of C:\Programfiles\Amazon\ directory.
Following are the steps that I am doing after EC2 instance comes up:
Rename the Administrator password (which doesn't work yet).
Set the time zone
Rename the hostname
Adding that server to the domain controller.
There should be some kind of logs on that EC2 instance about all these steps right? However, I can't find any. Any suggestions where I should be looking for the logs please?
You need to run cloud-init scripts to achieve all the tasks. I recommend writing PowerShell scripts for this.
Just refer the below repo, you will find useful template and scripts which do same activities.
https://github.com/aws-quickstart/quickstart-microsoft-sql
I have a hosted chef account working with AWS EC2 instances.
The instances can connect to hosted chef by way of chef-client and run the inital
run_list that is presetly a role called servers. Everything runs and installs
correctly on initial provision.
I should also mention I have autoscaling on ec2 instances that increses the amount of instances
provisioned on an 'as needed' basis and this works fine.
I found one talk that sort of makes sense but seems a bit more involved.
https://www.youtube.com/watch?v=yHub6E4DNvg
My questions are around how to maintain the servers after initial provisioning. First, if I have an update to my role to include more cookbooks or configuration, and how would I push this out to say X amount servers in a simple way.
Second, how would chef server know if a node has failed and notify the admin?
Any guidance?
That's quite vast question ...
For the easy one: failing node => the solution is to use report handlers see the doc here
For the keep in desired state, the idea is to have chef run periodically on the nodes, so it keep enforcing the conf defined in cookbooks, any change will be updated accordingly. (that's the way chef has been concepted).
If you want more control on which server run when you can:
use the push-jobs addon (commercial addon, not free)
use some orchestration tool (I personnaly use rundeck for adhoc runs in addition to periodic runs).
There's a lot of way to do this and which one to use is a question of preference and environment.
I think you could also use knife to run a chef-client based on a certain criteria:
knife ssh 'role:somerole' 'chef-client' -x username -P password
Here is chef's documentation: https://docs.chef.io/knife_ssh.html
I was able to successfully run Data pipeline for predefined templates. I wanted to use a customized AMI for my EC2 Resource as I require some libraries and packages to be installed.
I also have to run a python script as a part of the process.
So, I have created a Base Image of EC2 Resource with all packages required and the code that has to be run.
As part of my activity, I trigger a shell command activity, where I execute the python script as the command that has to be run.
The EC2 resource comes up successfully based on the Customized AMI that I have specified. I am able to login to that machine using the key pair that I specified but just that the Activity gets stuck in "WAITING FOR RUNNER" state.
I am not sure as to how to solve this problem. Please let me know if there are better ways to fix the same. Am I missing some basic step while trying to use a EC2 resource from an AMI.
Use Amazon Linux when creating your custom AMI and it will resolve this issue. Their OS comes preinstalled with tools that are used by data pipelines to communicate with the instance.
Are you running in a VPC or EC2Classic? I had the same problem when running in a VPC. When I checked run.out on the EC2 instance, I saw an error message:
Error in custom provider, java.lang.RuntimeException: java.net.UnknownHostException: . . . "
The TaskRunner was not able to resolve its own hostname, and was failing to start.
I solved this by setting the "DNS hostnames" setting to yes on my VPC in the VPC console. By default on new accounts it is set to no. This resolved the issue.
I realize this is old but if you are using a custom AMI with runson specified, you should make sure your custom AMI has all dependencies installed: https://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-custom-ami.html
I am running a couple of ubuntu ec2 instances, I want to run an automation script which will pull the code from Github whenever a new instance is booted from the AMI. The thing is presently I am sshing to the server and run the command git pull origin master and it will ask for password key.
How do I automate this process? So after booting the new instance from a AMI this script should:
Run
Pull the code and also the submodule
Create couple of files and configure it
Please help me to achieve it.
Thanks
This will probably take some time and configuring, but this might set you on the right path.
First, setup your ssh keys, so that you can automatically pull from a repo, without a password. Outlined here: https://help.github.com/articles/generating-ssh-keys
Next, create a startup script to issue the 'pull' command from Github. Here: https://help.ubuntu.com/community/UbuntuBootupHowto
Then save your AMI, When you start a new EC2 AMI, the script should run, pulling in your Github changes.
Also to note, make sure gits remote path is using SSH, if it is HTTPS, it will ALWAYS ask for a password.
Your best best would be to utilize the fact the Ubuntu utilizes CloudInit within its canonical image.
Using CloudInit, you can pass scripts (i.e. shell scripts) to execute at various start up stages as EC2 user-data.
It would be very easy for your to make your GIT command line sequence execute from such a script. He is link to documentation, which includes examples.
https://help.ubuntu.com/community/CloudInit
Create a user-password access to your ubuntu instance. Replicate this particular instance if you need multiple. Now you are free of the key access. If you need to automate a process in that instance cron it or send the script via ssh to that instance and let the cron to find and run it.