I'm attempting to use an AWS Lambda SelfManagedKafka event source mapping, as described in Using Lambda with self-managed Apache Kafka. Confluent Cloud is being used to host the Kafka cluster the event source will use.
On creation of the event source mapping (via SAM/CloudFormation template), though, the trigger is showing the following as the "Last processing result":
PROBLEM: Cluster failed to authorize Lambda.
Based on the guide, this error indicates...
that the provided user doesn't have all of the following required Kafka access control list (ACL) permissions: [list truncated]
I've verified that the listed permissions are in place for the group, cluster, and topic which leaves me to suspect that the issue has something to do with the following note:
The group name must match the event source mapping's UUID.
My tentative take on that note is that it's suggesting the group name has to match the physical ID of the event source mapping which is generated on creation. I've been unable to track down any further documentation that would elucidate that note. I'm wondering if anyone has encountered this before and can confirm?
If that is true, it would seem that the expected workflow is to create the event source mapping, get the physical ID from the output, and then create the Kafka ACLs.
After digging around further, I found at least a partial answer to this question by manually creating the event source mapping via the AWS Console. The UI enables one to optionally specify the name of the consumer group identifier, and includes the following help text for the input field:
The ID of a Kafka consumer group to join. If the consumer group ID you specify doesn't exist, or if you leave this field blank, Lambda generates a unique value.
If none is provided, the UUID of the event source mapping seems to be the value used.
As an alternative to taking that generated value and creating a consumer group in Kafka with that name, one can also add the CREATE permission on the cluster itself to the consumer's ACL. This enables Lambda to create the group on Kafka as well.
Related
I have followed the Datadog's documentation (here) for manually configure AWS account with Datadog. Set-up includes a Lambda function provided by Datadog (here) which is triggered by Cloudwatch log group and lambda pushes logs to Datadog.
Problem is that when logs are pushed, Datadog's log forwarder lambda changes name of function, tags and rest of the to small case. Now, when I use '!Ref ' while creating query for Datadog monitor using Cloudformation, then query contains autogenerated name of lambda which is mixture of small and upper case alphabets. But query does not work as Datadog changes name of the function while pushing logs.
Can someone suggest the work around here?
You could use a Cloudformation macro to transform strings in your template, in this case using the Lower operation to make your !Refs lowercase. Keep in mind that you need to define and deploy macros as lambdas.
How to pull data from AWS Security hub automatically using a scheduler ?
I am new to AWS on doing some analysis I found below :
In Security Hub data is in Json format , we don't have option to do Export to csv/excel ?
All Security hub findings/insights are automatically sent to eventbridge ? Is it true ? If yes where i can check the same in eventbridge ?
Are there any other options in order to pull data from security hub , every 12 hours automatically. I want to take the data from security hub and pass it to the ETL Process in order to apply some logic on this data ?
Is Eventbridge the only and best approach for this ?
On:
It is a JSON based but it's their own format named AWS Security Finding Format (ASFF)
It is true (for all resources that SecurityHub supports and is able to see). It should be noted that Each Security Hub Findings - Imported event contains a single finding.
In order to see those events you'll need to create an EventBridge rule based on the format for each type of event.
how to create rule for automatically sent events (Security Hub Findings - Imported)
In addition you can create a custom action in SecurityHub and then have an EventBridge event filter for it too.
Once you have that set up, the event could trigger an automatic action like:
Invoking an AWS Lambda function
Invoking the Amazon EC2 run command
Relaying the event to Amazon Kinesis Data Streams
Activating an AWS Step Functions state machine
Notifying an Amazon SNS topic or an Amazon SQS queue
Sending a finding to a third-party ticketing, chat, SIEM, or incident response and management tool.
In general, EventBridge is the way forward, but rather than using a scheduled based approach you'll need to resort to an event-based one.
In order to intercept all findings, instead of rule being triggered by just specific one, you'll need to adjust the filter and essentially create a catch-all rule for SecurityHub which will then trigger your ETL job.
EDIT (as requested in comment):
The filter in the rule would look like this:
{
"source": [
"aws.securityhub"
]
}
with regard to the ETL, it really depends on your use case, having Kinesis Data Firehose dumping it to S3 and then using Athena as you suggest on your own would work. Another common approach is to send the data to ElasticSearch (or now OpenSearch).
This blog post described them both, you can adjust it based on your needs.
EDIT 2:
Based on the discussion in the comments section if you really want to use a cron based approach you'll need to use the SDK based on your preferred language and create something around the GetFindings API that will poll for data from SecurityHub.
You can use this function in Python, which extracts data from SecurityHub to Azure Sentinel as an example
My requirement is to track any newly launch aws service/resources which does have 2 specific tags. What is the best way to do this? Right now thinking to have a lambda which will keep pinging any new resources is being launched or not which doesn't have specific tags.
Any other recommendation!
We do have a similar requirement where we want to ascertain that all of the newly created resources confirm certain requirements ( having specific tags and does not breach any security best practices).
We have implemented a soln which was based on AWS Config service
"AWS Config is a service that enables you to assess, audit, and evaluate the configurations of your AWS resources. Config continuously monitors and records your AWS resource configurations and allows you to automate the evaluation of recorded configurations against desired configurations. With Config, you can review changes in configurations and relationships between AWS resources, dive into detailed resource configuration histories, and determine your overall compliance against the configurations specified in your internal guidelines. This enables you to simplify compliance auditing, security analysis, change management, and operational troubleshooting." Reference: https://aws.amazon.com/config/
Consolidated list of the resources supported by AWS Config can be found at: https://docs.aws.amazon.com/config/latest/developerguide/resource-config-reference.html
The High-level design to achieve this as below:
The following needs to be done to implement the architecture
Enable Conf service in the region which you want to observe.
Enable config change recording from config console.
Create a lambda that evaluates the json event received for the config changes on AWS resources monitored by Config service and check for tags (in your case).
Create a custom config rule which observes the event for resources of your interest delegate those events to the lambda which you have created in step 3.
Lambda can send events to SNS for noncompliant resources. From SNS you can trigger an email
If given the correct permissions, Lambda functions will automatically create CloudWatch log groups to hold log output from the Lambda. Same for LambdaEdge functions with the addition that the log groups are created in each region in which the LambdaEdge function has run and the name of the log group includes the name of the region. The problem is that the retention time is set to forever and there is no way to change that unless you wait for the log group to be created and then change the retention config after the fact.
To address this, I would like to create those log groups preemptively in Terraform. The problem is that the region would need to be set in the provider meta argument or passed in the providers argument to a module. I had originally thought that I could get the set of all AWS regions using the aws_regions data source and then dynamically create a provider for each region. However, there is currently no way to dynamically generate providers (see https://github.com/hashicorp/terraform/issues/24476).
Has anyone solved this or a similar problem in some other way? Yes, I could create a script using the AWS CLI to do this, but I'd really like to keep everything in Terraform. Using Terragrunt is also an option, but I wanted to see if there were any solutions using pure Terraform before I go that route.
I have a Logger Lambda function that listens on a specific LogGroup and process specific log details.
I would like to attach the newly created LogGroups of other lambdas to that specific LogGroup, so it will process them as well - but since these other lambdas are created automatically, I need to do this automatically. Can I do it? How?
So there's no way to directly replicate the logs stored in a CloudWatch log group to another log group. You could do this by creating a subscription filter with a Lambda function to push the logs from each log group to the common one, but this would increase the costs for CloudWatch.
What I would suggest is either of the following:
Create a subscription filter for each of the log groups used by your Lambda functions to the common Lambda function so that it is triggered when logs are pushed to any of the log groups. This event can be set up after creating each function. Note, you would have to update the function policy of the common Lambda to allow it to be invoked from each log group (or just set up a wildcard).
Push all the logs for all of the functions to a single log group. This would take the least effort, but you would have to figure out how to effectively separate the logs per function (if that is required for your use case).