So I have deployed a standalone cluster to Amazon EC2 using Terraform. It is using passwordless ssh to communicate with workers.
I start the master with the start master script, giving the public ip of the cluster to be the public dns of the ec2 instance.
I then start the slaves using the start-slaves script, having copied over a config/slaves file with the public ip addresses of the 2 EC2 instances that are available to run the workers. (they each have the spark deployment on them in the same location as the master)
In the UI the workers are registered and running:
However, when i submit any job to the cluster, it is never able to allocate resources showing the message:
Does anyone know how to solve this?
The logs show the workers starting and registering correctly, and the task i'm submitting is within the available resources. (have tried as little as 1 cpu core and 500mb).
Does anyone know why the task may not be being accepted?
Cheers
Related
I am planning to create virtual server(Microsoft SQL Server 2019 Web on Windows Server 2019) in AWS but the costing is too high for me, to reduce the price I wanted to stop my virtual server automatically for some hours by using other AWS service.
How can I shutdown the virtual server?
Will I be able to shutdown it?
How AMI and EC2 are different in nature ?
After searching by hours I found that EC2 and RDS can be shutdown automatically but didn't find any article related to that.
Link for Virtual Server I want to created :
https://aws.amazon.com/marketplace/pp/prodview-urbgwl6rh6cuu?sr=0-1&ref_=beagle&applicationId=AWSMPContessa
Link for help:
https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/terminating-instances.html
To terminate an instance using the console
Before you terminate an instance, verify that you won't lose any data by checking that your Amazon EBS volumes won't be deleted on termination and that you've copied any data that you need from your instance store volumes to persistent storage, such as Amazon EBS or Amazon S3.
Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.
In the navigation pane, choose Instances.
Select the instance, and choose Instance state, Terminate instance.
Choose Terminate when prompted for confirmation.
Yes
The EC2 is used for creating the virtual server instance. The AMI is the EC2 virtual machines image. ECS provides container services such as docker and the AWS lambda is used to run the code without a server.
You can learn more about the differences in this link
I am a kafka newbie. I've to list the topics in the MSK(https://console.aws.amazon.com/msk/home) . I login to msk but there is no console there to run the relevant commands to list the topics. So I am trying to figure out on which EC2 is the MSK running so I can log in that machine. But I've not been able to figure out how to map the relevant EC2 instance to my current running MSK.
MSK is a managed service, which means Amazon manages the servers for you. You don't get access to the underlying servers of managed services on Amazon. If they even use EC2 instances behind the scenes (they probably do) those EC2 servers won't be visible to you.
To list the topics on your MSK cluster, you would need to do something like this from an EC2 server in the same VPC:
bin/kafka-topics.sh --list --zookeeper <zookeeper endpoint>
Where <zookeeper endpoint> is the ZooKeeper endpoint of your MSK cluster.
See the official documentation on this subject here.
Ok, I am lost with where to to even troubleshoot this. I am trying to spin up a stack that has a basic app running in ECS. I will show the cloudformation below. But I keep getting:
service sos-ecs-SosEcsService-1RVB1U5QXTY9S was unable to place a task
because no container instance met all of its requirements. Reason: No
Container Instances were found in your cluster. For more information,
see the Troubleshooting section.
I get 2 EC2 instances up and running but neither appear in the ECS cluster instances.
Here are a few of my theories:
is my user_data correct? do i need to sub the values?
what about the health check
my app is a sinatra app that uses port 4567. am i missing something with that?
Also, I basically started with this, http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/quickref-ecs.html and just streamlined it. So here is my current json, https://gist.github.com/kidbrax/388e2c2ae4d622b3ac4806526ec0e502
On a side note, how could I simplify this to take out all autoscaling? Just want to get it working in some form or fashion?
In order for the ECS instance to join the cluster, the following conditions must be met:
The agent must be configured correctly to connect to the correct
cluster via the /etc/ecs/ecs.config file.
The ECS instance must be assigned the correct IAM role to allow the ECS agent to access the ECS endpoints.
The ECS instance must have a connection to the Internet to contact the control plane, either via igw or NAT.
The ECS agent on the ECS Instance should be running.
UserData that should be used to configure /etc/ecs/ecs.config file.
#!/bin/bash
echo ECS_CLUSTER=ClusterName >> /etc/ecs/ecs.config
You can check reason for COntainer Instance not registering with Cluster in /var/log/ecs/ecs-agent.log*
After reading Why can't my ECS service register available EC2 instances with my ELB? I realized the issue was my userdata. The values were not being substituted correctly and so the instances were joining the defualt cluster.
Unable to place a task because no container instance met all of its requirements. Reason: No Container Instances were found in your cluster.
This usually means that your instances booted, but they're not healthy to register to cluster.
Navigate to Load Balancing Target Group of your cluster, then check the following
Health status of the instances in Targets tab.
Attributes in Description tab (the values could be off).
Health checks parameters.
If your instances are terminated, check system logs of terminated instances, and for any errors in your userdata script (check in Launch Configurations).
If the instances running, SSH to it, and verify the following:
The cluster is correctly configured in /etc/ecs/ecs.config.
ECS agent is up and running (docker ps). If it's not, start manually by: start ecs.
Check ECS logs for any errors by: tail -f /var/log/ecs/*.
Related:
terraform-ecs. Registered container instance is showing 0
How do I find the cause of an EC2 autoscaling group "health check" failure? (no load balancer involved)
I have Cassandra cluster on AWS. I have it only for testing purposes. Because of that I need to stop my instances and start them some other time. I deployed cluster with Datastax AMI and everything seems alright. I stoped instances in EC2 Management console and waited for 'stoped' state. After that I started them. I connected to them and Cassandra did not work. Command nodetool could not reach localhost:7199. Security groups on AWS allow all inbound and outbound traffic. I tried to sudo service cassandra start, but I got the same error.
I need start all nodes of cluster without manual action on them.
Ah. You stopped instances from the EC2 management console. This erased everything from the ephemeral storage. You should only restart instances if necessary, but never stop them. Or use sudo service dse restart to restart the dse service only without restarting the entire instance. Now I'd suggest that your best bet is to redeploy the cluster (or at least this node) from scratch....
Here is some explanation of what the instance store (ephemeral store) is: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html
The data in an instance store persists only during the lifetime of its
associated instance. If an instance reboots (intentionally or
unintentionally), data in the instance store persists. However, data
on instance store volumes is lost under the following circumstances:
Failure of an underlying drive
Stopping an Amazon EBS-backed instance
Terminating an instance
Datastax uses the ephemeral storage to store data and commit logs, because this storage is much faster than the ebs storage. More details here: http://datastax.com/documentation/datastax_enterprise/4.6/datastax_enterprise/install/installAMI.html
I created a elastic beanstalk environment and it created an EC2 instance. Then I thought I don't actually need this yet so I'll stop the EC2 instance, but then it seemed to start another one.
So my question is if I have an EB instance will I be charged by the hour for the underlying EC2 image all the time or only when the the service it provides is being access via the public elasticip. And if Im charged all the time is there a way to halt a elastic beanstalk application or only delete it or instantiate to a new environment.
The auto scaling feature of Elastic Beanstalk will automatically start another instance if a current instance continues to fail a health check. Stopping individual instances outside of the environment will cause failed health checks and trigger a new instance to be spun up.
You will be charged when the components within the environment are running as stated by Amazon here:
There is no additional charge for Elastic Beanstalk – you only pay for the underlying AWS resources (e.g. Amazon EC2, Amazon S3) that your application consumes.
You can completely stop an environment through the CLI. I gave this answer to a previous question about starting and stopping Elastic Beanstalk:
The EB command line interface has an eb stop command. Here is a little bit about what the command actually does:
The eb stop command deletes the AWS resources that are running your application (such as the ELB and the EC2 instances). It however leaves behind all of the application versions and configuration settings that you had deployed, so you can quickly get started again. Eb stop is ideal when you are developing and testing your application and don’t need the AWS resources running over night. You can get going again by simply running eb start.