I want to use cloudwatch log service for the programs running on older AMIs (2008-2010). Is there a way I can install it on such machines?.
A workaround which I could think of, is to copy log files from these AMIs to the latest AMI with log service installed and upload the logs from there. But the downside is that I will end up paying cost for data transfer. Is there any alternate better way?
When Henry Hahn gives Amazon Deep Dive CloudWatch presentation and says: "I am gonna to do a direct install", you find what you need.
$ wget https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py
$ sudo python awslogs-agent-setup.py --region eu-west-1
(the --region can differ in your case)
accepts defaults.
It shall install a service called awslogs, which can be started/stopped as any other service.
Configuration file can be found at /var/awslogs/etc/awslogs.conf
For me, this worked for my Debian Jessie notebook which is definitely not an EC2 instance, so it shall work for your older EC2 instance as well.
I expect, this will work for RPi instance too (planning to try soon).
Related
I haven't been able to determine if there is a time sync process (such as ntpd or chronyd) running on the docker swarm I've deployed to AWS using Docker Community Edition (CE) for AWS.
I've ssh'd to a swarm manager, but ps doesn't show much, and I don't see anything in /etc or /etc/conf.d that looks relevant.
I don't really have a good understanding of cloudformation, but I can see that the created instances running the docker nodes used AMI image Moby Linux 18.09.2-ce-aws1 stable (ami-0f4fb04ea796afb9a). I created a new instance w/ that AMI so I could ssh there. Still no time sync process indications w/ ps or in /etc
I suppose one of the swarm control containers that is running may deal with sync'ing time (maybe docker4x/l4controller-aws:18.09.2-ce-aws1)? Or maybe the cloudformation template installed one on the instances? But I don't know how to verify that.
So if anyone can tell me if there is a time sync process running (and where)?
And if not, I feel there should be so how might I start one up?
You can verify resources that are created by cloud formation Docker-no-vpc.tmpl from the link you provided.
Second thing, do you think ntpd have something do with docker-swarm? or it should be installed on the underlying EC2 instance?
Do ssh to your ec2 instance and very the status of the service, normally all AWS AMI has ntpd installed.
or you can just type to check
ntpd
If you did not find, you can install it for your self or you can run docker swarm with custom AMI.
UCP requires that the system clocks on all the machines in a UCP
cluster be in sync or else it can start having issues checking the
status of the different nodes in the cluster. To ensure that the
clocks in a cluster are synced, you can use NTP to set each machine's
clock.
First, on each machine in the cluster, install NTP. For example, to
install NTP on an Ubuntu distribution, run:
sudo apt-get update && apt-get install ntp
#On CentOS and RHEL, run:
sudo yum install ntp
what-does-clock-skew-detected-mean
Last thing, do you really need the stack that is created by cloudformation?
EC2 instances + Auto Scaling groups
IAM profiles
DynamoDB Tables
SQS Queue
VPC + subnets and security groups
ELB
CloudWatch Log Group
I know the cloudformation ease our life, but if you do not know the template (what resouces will be created) do not try to run the template, otherwise you will bear sweet cost at the of the month.
Also will suggest exploring AWS ECS and EKS, these are service that are sepcifly designed for docker container.
I'm about to install and use Amazon Inspector. We have many EC2 instances behind ELB. Plus some EC2 instances are opened via Auto-Scale.
My question: Is the Amazon Inspector doing its work locally or globally, meaning is the monitoring being made on the instance that it is installed on or it can be configured to include all the instances of the infrastructure?
If Inspector should be applied on every EC2 instance, can the Auto-Scale be configured to open the new instances with Inspector already installed on them and if yes, how can i do that?
I asked a similar question on the Amazon forum but got no response.
In the end I used the following feature to customise the EC2 instances that my application gets deployed to:
https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-ec2.html
Basically off the root of your .war file you need a folder named '.ebextensions' and in there a .config file containing some commands to install the Inspector client.
So my file 'inspector-agent.config' looks like this:
# Errors get logged to /var/log/cfn-init.log. See Also /var/log/eb-tools.log
commands:
# Download the agent installation script
"01-agent-repository":
command: sudo wget https://inspector-agent.amazonaws.com/linux/latest/install
# Run the installation script
"02-run-installation-script":
command: sudo bash install
I've found the answer and the solution, You have to install Amazon Inspector on each EC2 in order to inspect them all using Amazon Inspector.
About the Auto-Scale, I've applied Amazon Inspector on the main EC2 servers and took an image from them (after inspecting all the EC2s and fix all the issues). Then I've configured the Auto-Scale to lunch to lunch from the new AMIs (The Inspected AMIs).
So I've deployed a Mesos cluster in AWS using the CloudFormation script / instructions found here with the default cluster settings (5 private slaves, one public slave, single master, all m3.xlarge), and installed HDFS on the cluster with the dcos command: dcos package install hdfs.
The HDFS service is apparently up and healthy according to the DC/OS web UI and Marathon:
(the problem) At this point I should be able to SSH into my slave nodes and execute hadoop fs commands, but that returns the error -bash: hadoop: command not found (basically telling me there is no hadoop installed here).
There are no errors coming from the STDOUT and STDERR logging for the HDFS service, but for what its worth there is a recurring "offer decline" message appearing in the logs:
Processing DECLINE call for offers: [ 5358a8d8-74b4-4f33-9418-b76578d6c82b-O8390 ] for framework 5358a8d8-74b4-4f33-9418-b76578d6c82b-0001 (hdfs) at scheduler-60fe6c75-9288-49bc-9180-f7a271c …
I'm sure I'm missing something silly.
So I figured out a solution to at least verifying HDFS is running on your Mesos DC/OS cluster after install.
SSH into your master with the dcos CLI: dcos node ssh --master-proxy --leader
Create a docker container with hadoop installed to query your HDFS: docker run -ti cloudera/quickstart hadoop fs -ls hdfs://namenode-0.hdfs.mesos:9001/
Why this isn't a good solution & what to look out for:
Previous documentation all points to a default URL of hdfs://hdfs/, which instead will throw a java.net.UnknownHostException. I don't like pointing directly to a namenode.
Other documentation suggests you can run hdfs fs ... commands when you SSH into your cluster - this does not work as documented.
The image I used just to test that you can access HDFS is > 4GB (better options?)
None of this is documented (or at least not clearly/completely, hence why I'm keeping this post updated). I had to dig through DC/OS slack chat to find an answer.
The Mesosphere/HDFS repo is a completely different version than the HDFS that is installed via dcos package install hdfs. That repo is no longer maintained and the new version isn't open sourced yet (hence the lack of current documentation I guess).
I'm hoping there is an easier way to interface with HDFS that I'm still missing. Any better solutions would still be very helpful!
I see that Impala 2.3 is only supported on Cloudera CDH 5.5 & above. Impala 2.2 can be installed on Amazon EMR as there is Bootstrap script available on GitHub & you don't require Cloudera installation.
However, I don't see any way to install Cloudera CDH 5.5 or 5.6 on Amazon EMR. I want to install Impala 2.3 so is there any way through which Impala 2.3 can be installed on Amazon EMR?
Well, my previous answer has been deleted as long as "does not provide an answer to the question". I'm not going to argue if it's better to have a partially incorrect answer to this question or if making categorical claims without foundation is a good answer :/.
In any case, I'm not giving up :)
Yes, it's possible to install "anything" on the paper.
Once you launch the EMR cluster, all instances will appear on your EC2 console. The only thing is that you have to be careful assigning the right permissions to access thru SSH to your instances. My suggestion is to create a specific security group with the access and assign this extra security group to the instances using the Advanced configuration of the cluster.
By having the proper configuration, you could ssh into any instance and install anything (you should be able to scp any file or download from internet if you have the proper configuration of your VPC). Note that the user will be "hadoop" instead "ec2-root" but this is documented on the EMR user guide.
Keep in mind that the cluster is "Terminated" so, the EMR instances are volatile and the installation is not going to survive the cluster termination.
On the other hand, using the latest versions of EMR AMIs and the latest capabilities of AWS (I think that it was all the time the case, but, it doesn't matter now) you should be able to create some actions on the bootstrap and install anything you want.
Using the "Advanced configuration" of your cluster, you can access to the "Bootstrap" actions to be executed on your cluster. You could even have different actions depending on the node type (master, core, tasks). You should store your scripts (and/or jar files) on an S3 bucket and made this bucket available to your cluster. On the paper, you could install Impala on these EC2 instances comprising the EMR cluster but I'm not sure if this will work.
For more information, you can read http://docs.aws.amazon.com//emr/latest/ManagementGuide/emr-plan-bootstrap.html
And for a previous version of EMR AMI and not so recent version of Impala you can read https://github.com/awslabs/emr-bootstrap-actions/tree/master/impala
Thanks Mark, you forced me to elaborate better my comment.
No, it is not possible to "install" anything on EMR because it's a PaaS provided by AWS. But if your goal is to run a newer version of Impala on AWS, there is an AWS Quick Start path for installing CDH 5.x (including Impala) that makes the process relatively easy.
http://aws.amazon.com/quickstart/
I have been using Spark on an EMR cluster for a few weeks now without problems - the setup was with the AMI 3.8.0 and Spark 1.3.1, and I passed '-x' as an argument to Spark (without this it didn't seem to be installed).
I want to upgrade to a more recent version of Spark and today spun up a cluster with the emr-4.1.0 AMI, containing Spark 1.5.0. When the cluster is up it claims to have successfully installed Spark (at least on the cluster management page on AWS) but when I ssh into 'hadoop#[IP address]' I don't see anything in the 'hadoop' directory, where in the previous version Spark was installed (I've also tried with other applications and had the same result, and tried to ssh in as ec2-user but Spark is also not installed there). When I spin up the cluster with the emr-4.1.0 AMI I don't have the option to pass the '-x' argument to Spark, and I'm wondering if there is something I'm missing.
Does anyone know what I'm doing wrong here?
Many thanks.
This was actually solved, rather trivially.
In the previous AMI all of the paths to Spark and other applications were soft links available in the hadoop folder. In the newer AMI these have been removed but the applications are still installed and can be accessed by 'spark-shell' (for example) at the command line.