I am trying to test auto scale behavior in amazon ECS. Here is how i tested
import java.io.IOException;
import java.net.HttpURLConnection;
import java.net.URL;
public class LoadHttp {
private static final String GET_URL = "http://tournament-ecs-weather-lb-1526102172.ap-south-1.elb.amazonaws.com/status";
public static void main(String[] args) throws IOException {
System.out.println("GET DONE");
while(true){
sendGET();
}
}
private static void sendGET() throws IOException {
Thread t = new Thread(() -> {
URL obj;
try {
obj = new URL(GET_URL);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
con.setRequestMethod("GET");
con.setConnectTimeout(5000);
int responseCode = con.getResponseCode();
System.out.println(responseCode);
} catch (Exception e) {
e.printStackTrace();
}
});
t.start();
}
}
and configuration of auto scale of service that i am running
here is my auto scale group configuration
I am using ECS Service to create cluster and my service is running in docker container. Still after executing my test code i found my service is not getting autoscaled. Number of container instance is still one. If you need additional information I will be happy.
If I understood your question, the service is autoscaled when there are not enough resources available in the current instance to handle the requests.
In your particular case, you have set the Min value of the instances as 0 which tells that the system should react fine even there is no instance running. You need to have at least 1 for the min value to make the whole cluster run ok. I believe that for this reason your autoscaling is not launching any instance.
Also, I would suggest you to add the Elastic Load balancer to your autoscaling group which launch asks autoscaling group to launch more instances when the minimum number of instances specified in your autoscaling are not healthy.
Lastly, to test if the autoscaling group is working or not, you check the availability zone of your instance and note that down. Now terminate your instance. Since the running instance is terminated the autoscaling group should automatically launch an instance in different availability zone. If it launches new instance in different availability zone than the terminated instance then it works ok.
Hope this will help.
Related
I am deploying a queue processing ECS service using EC2 deployment using CDK. Here is my stack
from aws_cdk import core, aws_ecs_patterns, aws_ec2, aws_ecs
class EcsTestStack(core.Stack):
def __init__(self, scope: core.Construct, construct_id: str, **kwargs) -> None:
super().__init__(scope, construct_id, **kwargs)
_container_image = aws_ecs.ContainerImage.from_asset(
directory=".",
file='Dockerfile.ECS_Test',
exclude=["cdk.out"]
)
_scaling_steps = [{"upper": 0, "change": -1}, {"lower": 50, "change": +1}, {"lower": 100, "change": +2}]
_vpc = aws_ec2.Vpc(self, "ecs-test-vpc-v4", max_azs=3)
_cluster = aws_ecs.Cluster(self, "ecs-test-cluster-v4", vpc=_vpc)
_cluster.add_capacity("ecs-autoscaling-capacity-v4",
instance_type=aws_ec2.InstanceType("t2.small"),
min_capacity=1,
max_capacity=3)
self.ecs_test = aws_ecs_patterns.QueueProcessingEc2Service(
self,
"ECS_Test_Pattern_v4",
cluster=_cluster,
cpu=512,
scaling_steps=_scaling_steps,
memory_limit_mib=256,
image=_container_image,
min_scaling_capacity=1,
max_scaling_capacity=5,
)
The stack starts out with 1 task in the service and 1 EC2 instance. Based on my _scaling_steps, a new task should be added to the service when the number of messages in the queue > 50 and 2 new tasks should be added to the service. The stack starts out with 1 task in the service and 1 EC2 instance.
But when I add 200 new messages to the queue, I can see 1 new task added to my service and then I get this error message in the even.
service
EcsTestStackV4-ECSTestPatternv4QueueProcessingService5C84D200-c00N6R56hB0p
was unable to place a task because no container instance met all of
its requirements. The closest matching container-instance
81e5380c718c4b558567dc6cc1fb1926 has insufficient CPU units available.
I also notice that no new EC2 instances were added.
Question: how do I get more EC2 instances added to my cluster when the service scales up?
I can see 1 new task added to my service and then I get this error message in the even.
This is because t2.small has 1000 CPU units. So your two tasks take all of them, and there is no other instances to place your extra task on.
I also notice that no new EC2 instances were added.
You set min_capacity=1 so you have only instance. The _scaling_steps are for the tasks only, not for your instances in autoscaling group. If you want more instance you have to set min_capacity=2 or whatever value you want.
I guess you thought that QueueProcessingEc2Service scales both instances and tasks. Sadly this is not the case.
/* My Configuration */
EC2 Console limit : All Standard (A, C, D, H, I, M, R, T, Z) Spot Instance Requests 128vCPU
Region : ap-northeast-2
Current Instance count : 0
Currently, I've no instances running on AWS (all spot instance terminated).
And I using ASG(Auto Scaling Group) with Instance Launch Template (No spot option).
When I try to launch a new instance with Instance Launch Template, it's OK. (instance type : c5.large)
But when I try to launch a new spot instance with Instance Launch Template (Check spot option),
it tells me that "Max spot instance count exceeded".
ASG Activity)
Status : Failed
Description : Launching a new EC2 instance. Status Reason: Max spot instance count exceeded. Launching EC2 instance failed.
Help me please...
I have Spring Boot app which connects to Redis cluster, using Jedis Connection Factory:
RedisClusterConfiguration redisClusterConfiguration = new RedisClusterConfiguration(redisProperties.getCluster().getNodes());
redisClusterConfiguration.setPassword(redisProperties.getPassword());
jedisConnectionFactory = new JedisConnectionFactory(redisClusterConfiguration);
and reading list of nodes from application.yml:
spring:
redis:
host: 127.0.0.1
port: 6379
timeout: 300s
cluster:
nodes: 127.0.0.1:6380,127.0.0.1:6381,127.0.0.1:6382
Now we want to switch to Elasticache since we are hosting our Redis cluster on AWS anyway.
It would be done quite easily. If AmazonElastiCache lib could be used.
Then we could just connect to Elasticache cluster with AWS credentials, pull available nodes put it in the list and pass it to Jedis instead hardcoding them in application.yml, like:
//get cache cluster nodes using AWS api
private List<String> getClusterNodes(){
AmazonElastiCache client = AmazonElastiCacheClientBuilder.standard().withRegion(Regions.DEFAULT_REGION).build();
DescribeCacheClustersRequest describeCacheClustersRequest = new DescribeCacheClustersRequest();
describeCacheClustersRequest.setShowCacheNodeInfo(true);
List<CacheCluster> cacheClusterList = client.describeCacheClusters(describeCacheClustersRequest).getCacheClusters();
List<String> nodeList = new ArrayList<>();
try {
for (CacheCluster cacheCluster : cacheClusterList) {
for(CacheNode cacheNode :cacheCluster.getCacheNodes()) {
String nodeAddr = cacheNode.getEndpoint().getAddress() + ":" +cacheNode.getEndpoint().getPort();
nodeList.add(nodeAddr);
}
}
}
catch(Exception e) {
e.printStackTrace();
}
return nodeList;
}
But DevOps team said that they can't configure AWS access on all labs and they have reasons for it. Also instead of connecting to AWS and pulling all available clusters we need to connect to specific one by URL.
So I tried to pass Elasticache cluster url directly to Jedis as standalone and as a cluster in application.yml configuration.
In both cases connection is established, but when App tries to write to Elasticache its gets MOVED exception:
org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.springframework.data.redis.ClusterRedirectException: Redirect: slot 1209 to 10.10.10.011:6379.; nested exception is redis.clients.jedis.exceptions.JedisMovedDataException: MOVED 1209 10.10.10.102:6379
Which as I understand means that App tried to write to one of the nodes in Elasticache, but wasn't able to connect.
So the question would be is there a way to connect to Elasticache Redis cluster from Spring Boot app using only Elasticache cluster URL?
I know that it's doable if Elasticache Memecache is used.
Also Jedis driver is not a hard requirement.
Thank you.
After some research we learned that if AWS Elasticache cluster end-point is set as a node in RedisClusterConfiguration then driver (Jedis or Lettuce) is able to connect and find all the nodes in a Elasticache cluster. Also if one of the nodes goes down driver is able to communicate with Elasticache cluster through some other node.
We migrated to Lettuce driver while working on this upgrade as well, since Lettuce is default driver provided in Spring Boot Redis Started and supports latest Redis versions. Lettuce connections are designed to be thread-safe too, Jedis not.
Code example:
List<String> nodes = Collections.singletonList("****.***.****.****.cache.amazonaws.com:6379");
RedisClusterConfiguration clusterConfiguration = new RedisClusterConfiguration(nodes);
return new LettuceConnectionFactory(clusterConfiguration);
Inspired from Above Answer:, complete more detailed code
List<String> nodes = Collections.singletonList("<cluster-host-name>:<port>");
RedisClusterConfiguration clusterConfiguration = new RedisClusterConfiguration(nodes);
ClusterTopologyRefreshOptions topologyRefreshOptions = ClusterTopologyRefreshOptions.builder().closeStaleConnections(true)
.enableAllAdaptiveRefreshTriggers().build();
ClusterClientOptions clusterClientOptions = ClusterClientOptions.builder().autoReconnect(true)
.topologyRefreshOptions(topologyRefreshOptions).validateClusterNodeMembership(false)
.build();
//If you want to add tuning options
LettuceClientConfiguration lettuceClientConfiguration = LettuceClientConfiguration.builder().readFrom(ReadFrom.REPLICA_PREFERRED).clientOptions(clusterClientOptions).build();
LettuceConnectionFactory lettuceConnectionFactory = new LettuceConnectionFactory(clusterConfiguration, lettuceClientConfiguration);
lettuceConnectionFactory.afterPropertiesSet();//**this is REQUIRED**
StringRedisTemplate redisTemplate = new StringRedisTemplate(lettuceConnectionFactory);
Given an instance id, I want to get an EC2 instance info (for example, its running status, private IP, public IP).
I have done some research (i.e. looking at the sample code posted here Managing Amazon EC2 Instances)
but there is only sample code of getting the Amazon EC2 instances for your account and region.
I tried to modify the sample and here is what I came up with:
private static AmazonEC2 getEc2StandardClient() {
// Using StaticCredentialsProvider
final String accessKey = "access_key";
final String secretKey = "secret_key";
BasicAWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey);
return AmazonEC2ClientBuilder.standard()
.withRegion(Regions.AP_NORTHEAST_1)
.withCredentials(new AWSStaticCredentialsProvider(credentials))
.build();
}
public static void getInstanceInfo(String instanceId) {
final AmazonEC2 ec2 = getEc2StandardClient();
DryRunSupportedRequest<DescribeInstancesRequest> dryRequest =
() -> {
DescribeInstancesRequest request = new DescribeInstancesRequest()
.withInstanceIds(instanceId);
return request.getDryRunRequest();
};
DryRunResult<DescribeInstancesRequest> dryResponse = ec2.dryRun(dryRequest);
if(!dryResponse.isSuccessful()) {
System.out.println("Failed to get information of instance " + instanceId);
}
DescribeInstancesRequest request = new DescribeInstancesRequest()
.withInstanceIds(instanceId);
DescribeInstancesResult response = ec2.describeInstances(request);
Reservation reservation = response.getReservations().get(0);
Instance instance = reservation.getInstances().get(0);
System.out.println("Instance id: " + instance.getInstanceId(), ", state: " + instance.getState().getName() +
", public ip: " + instance.getPublicIpAddress() + ", private ip: " + instance.getPrivateIpAddress());
}
It is working fine but I wonder if it's the best practice to get info from a single instance.
but there is only sample code of getting the Amazon EC2 instances for your account and region.
Yes, you may get only instance information you have permission to read.
It is working fine but I wonder if it's the best practice to get info from a single instance
You have multiple options.
For getting EC2 metadata from any client (e.g. from your on-premise network) your code seems ok.
If you are running the code in the AWS environment (on an EC2, lambda, docker, ..) you may specify a service role allowed calling the describeInstances operation from the service. Then you don't need to specify the AWS credentials explicitly (DefaultAWSCredentialsProviderChain will work).
If you are getting the EC2 metadata from the instance itself, you can use the EC2 metadata service.
I have a test Lambda function setup using Java 8
So far I have
given the Lambda function an execution role AWSLambdaVPCAccessExecutionRole
attached the Lambda function to the only VPC I have on my account and selected all subnets within the VPC so that it may access the
my RDS instance in this case is open to the public, and I am able to access it via my Laptop (i.e. the Lambda code actually runs on remote hosts not inside the VPC)
security group assigned to the Lambda is the most permissive possible (i.e all traffic on all CIDR block)
However, I am still unable to access my RDS instance when running the Lambda function on AWS (but the same code works on my laptop, when run from a main() function)
Sample Code
public class Application implements RequestHandler<Object, Boolean> {
private Logger logger = Logger.getLogger(Application.class);
public Boolean handleRequest(Object object, Context context) {
try {
Class.forName("com.mysql.jdbc.Driver");
logger.info("Calling DriverManager.getConnection()");
Connection con = DriverManager.getConnection(
"jdbc:mysql://...endpoint...defaultdb",
"awsops",
"..."
);
Statement stmt = con.createStatement();
logger.info("Test Started!");
ResultSet result = stmt.executeQuery("SELECT\n" +
" last_name, COUNT(*)\n" +
"FROM\n" +
" users\n" +
"GROUP BY\n" +
" last_name\n" +
"HAVING COUNT(*) > 1");
if (result.next()) {
logger.info(result.getString("last_name"));
}
return true;
} catch (Exception e) {
logger.info(e.getLocalizedMessage());
}
return false;
}
}
Can you please help me understand what I could be doing wrong?
The CloudWatch logs shows that the function hangs at DriverManager.getConnection()
It turns out my RDS was launched with an automatically created security group that literally just whitelisted my personal IP address ... so it feels like I was able to connect to it from 'anywhere'
I had to update the security group of my RDS instance to allow traffic from the subnets where the Lambda's virtual network interface could be coming from