I am trying to use ElastiCache with a Spring app that is supposed to be deployed as a worker application in a worker environment on AWS.
The app has a a cron job which should run every 5 minutes and update some data on ElastiCache. The cron.yaml is defined as:
version: 1
cron:
- name: "memcache-dataset-update-job"
url: "/runcron"
schedule: "0/5 * * * *"
"/runcron" calls the following method:
#RequestMapping(method = RequestMethod.GET)
#ResponseStatus(value = HttpStatus.OK)
public void updateDataSet(){
try {
dataSet = initializeNewDataSet();
memcached = new MemcachedClient(new BinaryConnectionFactory(ClientMode.Dynamic),
AddrUtil.getAddresses(memcacheConfigEndpoint));
// Store a value (async) for one hour
memcached.set(dataSetKey, 1800, dataSetObject);
}
My question:
1. Should the request mapping be for http POST?
2. Do I need to define permissions in the IAM worker role to allow my app access to ElastiCache. If yes, how? I could not find any help here on AWS docs.
I found answer to my own questions:
1. Request mapping should be for HTTP POST method.
2. No permissions need to be defined in the IAM worker role for ElastiCache access. Just that the app should be within the same VPC as your cache cluster.
Related
I am working with aws boto3 lib and trying to retrieve certain values.
I first retrieve all cluster list, then fetch specific services, then call describe-service for them.
But I am unable to retrieve two fields Minimum tasks and Maximum tasks for services which get displayed on AWS ECS console page under Auto Scaling tab.
Anybody has any idea how to get these values from?
The ECS console hides this fact, but those are actually in the Application AutoScaling configuration, not the ECS configuration. I believe you would need to call describe_scalable_targets in ApplicationAutoScaling to get those values.
Thanks Mark B for help.
You are right and I understand that aws ecs service has to register with autoscaling service which is a separate service. I am providing sample cli and python code to retrieve these values for other now.
aws ecs describe-services --cluster MAGIC-Bonus-Wrappers --services service-name
aws application-autoscaling describe-scalable-targets --service-namespace ecs --resource-ids service/cluster-name/service-name
Python Code:
client = session.client('application-autoscaling')
response = client.describe_scalable_targets(
ServiceNamespace='ecs',
ResourceIds=[serviceId])
def_val = -1, -1
if "ScalableTargets" in response and len(response['ScalableTargets']) > 0 :
target = response['ScalableTargets'][0]
if 'MinCapacity' in target and 'MaxCapacity' in target:
return target['MinCapacity'], target['MaxCapacity']
else:
return def_val
I am trying to invoke Google cloud function which is Http triggered by cloud scheduler.
But whenever I try to run cloud scheduler it always says permission denied error
httpRequest: {
status: 403
}
insertId: "14igacagbanzk3b"
jsonPayload: {
#type: "type.googleapis.com/google.cloud.scheduler.logging.AttemptFinished"
jobName: "projects/***********/locations/europe-west1/jobs/twilio-cloud-scheduler"
status: "PERMISSION_DENIED"
targetType: "HTTP"
url: "https://europe-west1-********.cloudfunctions.net/function-2"
}
logName: "projects/*******/logs/cloudscheduler.googleapis.com%2Fexecutions"
receiveTimestamp: "2020-09-20T15:11:13.240092790Z"
resource: {
labels: {
job_id: "***********"
location: "europe-west1"
project_id: "**********"
}
type: "cloud_scheduler_job"
}
severity: "ERROR"
timestamp: "2020-09-20T15:11:13.240092790Z"
}
Solutions I tried -
Tried putting Google cloud function in the same region as the App engine as suggested by some users.
Gave access to Google provided cloud scheduler sa service-****#gcp-sa-cloudscheduler.iamaccount.gserviceaccount.com owner role and Cloud Functions Admin role
My cloud function has ingress setting of Allow all traffic.
My cloud scheduler only works when I run below command
gcloud functions add-iam-policy-binding cloud-function --member="allUsers" --role="roles/cloudfunctions.invoker"
On Cloud Scheduler page, you have to add a service account to use to call the private Cloud Function. In the Cloud Scheduler set up, you have to
Click on SHOW MORE on the bottom
Select Add OIDC token in the Auth Header section
Add a service account email in the service account email for the Scheduler
Fill in the Audience with the same base URL as the Cloud Functions (the URL provided when you deployed it)
The service account email for the Scheduler must be granted with the role cloudfunctions.invoker
In my case the problem was related to restricted ingress setting for the cloud function. I set it to 'allow internal traffic only', but that allows only traffic from services using VPC, whereas Cloud Scheduler doesn't as per doc explanation:
Internal-only HTTP functions can only be invoked by HTTP requests that are created within a VPC network, such as those from Kubernetes Engine, Compute Engine, or the App Engine Flexible Environment. This means that events created by or routed through Pub/Sub, Eventarc, Cloud Scheduler, Cloud Tasks and Workflows cannot trigger these functions.
So the proper way to do it is:
set the ingress to 'all traffic'
remove the permission for allUsers with role Cloud Function Invoker
add the permission for created service account with role Cloud Function Invoker
or just set that permission globally for the service account in IAM console(you could do that when creating service account as well)
If you tried all of the above (which should be the first things to look at, such as Add OIDC token, giving your service account role Cloud Function Invoker and/or Cloud Run Invoker (for 2nd gen functions) etc.), please also check the following:
For me the only thing that fixed this, was adding the following google internal service account to IAM:
service-YOUR_PROJECT_NUMBER#gcp-sa-cloudscheduler.iam.gserviceaccount.com
And give this internal service account the following role:
Cloud Scheduler Service Agent
See also:
https://cloud.google.com/scheduler/docs/http-target-auth
And especially for this case:
https://cloud.google.com/scheduler/docs/http-target-auth#add
we have an AWS cloudformation template, through which we are creating the Amazon MSK(Kafka) cluster. which is working fine.
Now we have multiple applications in our product stack which consume the Brokers endpoints which is created by the Amazon MSK. now to automate the product deployment we decided to create a Route53 recordset for the MSK broker endpoints. we are having hard time finding how we can get a broker endpoints of MSK cluster as an Outputs in AWS Cloudformation templates.
looking forward for suggestion/guidance on this.
Following on #joinEffort answer, this how i did it using custom resources as the CFN resource for an MKS::Cluster does not expose the broker URL:
(option 2 is using boto3 and calling AWS API
The description of the classes and mothods to use from CDK custom resource code can be found here:
https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Kafka.html#getBootstrapBrokers-property
Option 1: Using custom resource:
def get_bootstrap_servers(self):
create_params = {
"ClusterArn": self._platform_msk_cluster_arn
}
get_bootstrap_brokers = custom_resources.AwsSdkCall(
service='Kafka',
action='getBootstrapBrokers',
region='ap-southeast-2',
physical_resource_id=custom_resources.PhysicalResourceId.of(f'connector-{self._environment_name}'),
parameters = create_params
)
create_update_custom_plugin = custom_resources.AwsCustomResource(self,
'getBootstrapBrokers',
on_create=get_bootstrap_brokers,
on_update=get_bootstrap_brokers,
policy=custom_resources.AwsCustomResourcePolicy.from_sdk_calls(resources=custom_resources.AwsCustomResourcePolicy.ANY_RESOURCE)
)
return create_update_custom_plugin.get_response_field('BootstrapBrokerString')
Option 2: Using boto3:
client = boto3.client('kafka', region_name='ap-southeast-2')
response = client.get_bootstrap_brokers(
ClusterArn='xxx')
#From here u can get the broker urls:
json_response = json.loads(json.dumps(response))
Reff: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/kafka.html#Kafka.Client.get_bootstrap_brokers
You should be able to get it from below command. More can be found here.
aws kafka get-bootstrap-brokers --region us-east-1 --cluster-arn ClusterArn
UPDATED
Following the AWS instance scheduler I've been able to setup a scheduler that starts and stops at the beginning and end of the day.
However, the instances keep being terminated and reinstalled.
I have an Amazon Elastic Kubernetes Service (EKS) that returns the following CloudWatch log:
discovered the following log in my CloudWatch
13:05:30
2019-11-21 - 13:05:30.251 - INFO : Handler SchedulerRequestHandler scheduling request for service(s) rds, account(s) 612681954602, region(s) eu-central-1 at 2019-11-21 13:05:30.251936
13:05:30
2019-11-21 - 13:05:30.433 - INFO : Running RDS scheduler for account 612681954602 in region(s) eu-central-1
13:05:31
2019-11-21 - 13:05:31.128 - INFO : Fetching rds Instances for account 612681954602 in region eu-central-1
13:05:31
2019-11-21 - 13:05:31.553 - INFO : Number of fetched rds Instances is 2, number of schedulable resources is 0
13:05:31
2019-11-21 - 13:05:31.553 - INFO : Scheduler result {'612681954602': {'started': {}, 'stopped': {}}}
I don't know if it is my EKS that keeps rebooting my instances, but I really would love to keep them stopped until the next day.
How can I prevent my EC2 instances from automatically rebooting every time one has stopped? Or, even better, how can I deactivate my EKS stack automatically?
Update:
I discovered that EKS has a Cluster Autoscaler. Maybe this could be where the problem lies?
https://docs.aws.amazon.com/eks/latest/userguide/cluster-autoscaler.html
EKS node group would create an auto scaling group to manage the worker nodes. You need specify the minimum, maximum and desired size of worker nodes. Once any instance is stopped, the auto scaling group would create new instance to match the desired instance size.
Check below doc for details,
https://docs.aws.amazon.com/eks/latest/userguide/launch-workers.html
I've created a worker environment for my eb application in order to take advantage of its "periodic tasks" capabilities using cron.yaml (located in the root of my application). It's a simple sinatra app (for now) that I would like to use to use to issue requests to my corresponding web server environment.
However, I'm having trouble deploying via the eb cli. Below is what happens I run eb deploy.
╰─➤ eb deploy
Creating application version archive "4882".
Uploading myapp/4882.zip to S3. This may take a while.
Upload Complete.
INFO: Environment update is starting.
ERROR: Service:AmazonCloudFormation, Message:Stack named 'awseb-e-1a2b3c4d5e-stack'
aborted operation. Current state: 'UPDATE_ROLLBACK_IN_PROGRESS'
Reason: The following resource(s) failed to create: [AWSEBWorkerCronLeaderRegistry].
I've looked around the CloudFormation dashboard to see to check for possible errors. After reading a bit of about what I could find regarding AWSEBWorkerCronLeaderRegistry, I found it that it's most likely a DynamoDB table that gets updated/created. However, when I look in the DynamoDB dashboard, there are no tables listed.
As always, any help, feedback, or guidance is appreciated.
If you are reluctant to add full DynamoDB access (like I was), Beanstalk now provides a Managed Policy for Worker environment permissions (AWSElasticBeanstalkWorkerTier). You can try adding one of these to your instance profile role instead.
See http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/iam-instanceprofile.html
We had the same issue and fixed it by attaching AmazonDynamoDBFullAccess to Elastic Beanstalk role (which was named aws-elasticbeanstalk-ec2-role in our case).
I was using Codepipeline to deploy my worker and was getting the same error. Eventually I tried giving AWS-CodePipeline-Service the AmazonDynamoDBFullAccess policy and that seemed to resolve the issue.
As Anthony suggested, when triggering the deploy from other services such as CodePileline, its service role needs the dynamodb:CreateTable permission to create the Leader Registry table (more info below) in DynamoDB.
Adding Full Access permission is a bad practice and should be avoided. Also, the managed policy AWSElasticBeanstalkWorkerTier does not have the appropriate permissions since it is for the worker to access DynamoDB and check if they are the current leader.
1. Find the Role that is trying to create the table:
Go to CloudTrail > Event History
Filter Event Name: CreateTable
Make sure the error code is AccessDenied
Locate the role name (i.e. AWSCodePipelineServiceRole-us-east-1-dev):
2. Add the permissions:
Go to IAM > Roles
Find the role in the list
Attach a policy with:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "CreateCronLeaderTable",
"Effect": "Allow",
"Action": "dynamodb:CreateTable",
"Resource": "arn:aws:dynamodb:*:*:table/*-stack-AWSEBWorkerCronLeaderRegistry*"
}
]
}
3. Check results:
Redeploy by triggering the pipeline
Check Elasticbeanstalk for errors
Optionally go to CloudTrail and make sure the request succeded this time.
You may use this technique any time you are not sure of what permission should be attached to what.
About the Cron Leader Table
From the Periodic Tasks Documentation:
Elastic Beanstalk uses leader election to determine which instance in your worker environment queues the periodic task. Each instance attempts to become leader by writing to an Amazon DynamoDB table. The first instance that succeeds is the leader, and must continue to write to the table to maintain leader status. If the leader goes out of service, another instance quickly takes its place.
For those wondering, this DynamoDB table uses 10 RCU and 5 WCU which covered by the always free tier.