lambda monitoring using aws quicksight - amazon-web-services

i have few lambdas that use different other services like SSM, athena, dynamodb, s3, SQS, SNS for my process. i am almost done with all my development and would love to monitor it visually. I use X-ray and cloud watch as my regular log monitoring and analysis. I feel cloud watch dashboards is not so efficient way to visualize my stuff with multiple services. So i did a lambda that pulls trace data from my X-ray traces and outputs a nested json file something like below.
[
{
"id": "4707a33e472",
"name": "test-lambda",
"start_time": 1524714634.098,
"end_time": 1524714672.046,
"parent_id": "1b9122bc",
"aws": {
"function_arn": "arn:aws:lambda:us-east-1:9684596:function:test-lambda",
"resource_names": [
"test-lambda"
],
"account_id": "9684596"
},
"trace_id": "1-5ae14c88-41dca52ccec8c7d",
"origin": "AWS::Lambda::Function",
"subsegments": [
{
"id": "ab6420197c",
"name": "S3",
"start_time": 1524714671.7148032,
"end_time": 1524714671.8333395,
"http": {
"response": {
"status": 200
}
},
"aws": {
"id_2": "No9Gemg5b9Y2XREorBG+6a1KLXX7S6O3HtPZ3f6vUuU5F1dQE0nIE1WmwmRRHIqCjI=",
"operation": "DeleteObjects",
"region": "us-east-1",
"request_id": "E2709BB91B8"
},
"namespace": "aws"
},
{
"id": "370e11d6d",
"name": "SSM",
"start_time": 1524714634.0991564,
"end_time": 1524714634.194922,
"http": {
"response": {
"status": 200
}
},
"aws": {
"operation": "GetParameter",
"region": "us-east-1",
"request_id": "f901ed67-4904-bde0-f9ad15cc558b"
},
"namespace": "aws"
},
{
"id": "8423bf21354",
"name": "DynamoDB",
"start_time": 1524714671.9744427,
"end_time": 1524714671.981935,
"http": {
"response": {
"status": 200
}
},
"aws": {
"operation": "UpdateItem",
"region": "us-east-1",
"request_id": "3AHBI44JRJ2UJ72V88CJPV5L4JVV4K6Q9ASUAAJG",
"table_name": "test-dynamodb",
"resource_names": [
"test-dynamodb"
]
},
I only posted the first few line of x-ray trace json output, but it's pretty large to post here. AWS quicksight doesn't support nested json, my question is, is there a way to visualize all my lambdas in a better way using quicksight. I am not allowed to use other third party monitoring systems. Need help with this

Related

Get errorType:OK when trying to deploy function

I'm struggling to deploy my cloud function. I'm unsure what information to provide. My set up:
# main.py
def callRequest():
print("bla")
return(1)
Entry point for the function is callRequest.
After failing to deploy I see this red highlighted message under details:
Deployment failure:
Build failed: {"metrics":{},"error":{"buildpackId":"","buildpackVersion":"","errorType":"OK","canonicalCode":"OK","errorId":"","errorMessage":""},"stats":[{"buildpackId":"google.utils.archive-source","buildpackVersion":"0.0.1","totalDurationMs":47,"userDurationMs":46},{"buildpackId":"google.python.runtime","buildpackVersion":"0.9.1","totalDurationMs":9487,"userDurationMs":6307},{"buildpackId":"google.python.functions-framework","buildpackVersion":"0.9.6","totalDurationMs":53,"userDurationMs":52},{"buildpackId":"google.python.pip","buildpackVersion":"0.9.2","totalDurationMs":5832,"userDurationMs":5822},{"buildpackId":"google.utils.label","buildpackVersion":"0.0.2","totalDurationMs":0,"userDurationMs":0}],"warnings":null,"customImage":false}
In the logs I see a notice related to the attempted deploy:
{
"protoPayload": {
"#type": "type.googleapis.com/google.cloud.audit.AuditLog",
"authenticationInfo": {
"principalEmail": "myname#bla.com"
},
"requestMetadata": {
"callerIp": "152.170.106.184",
"callerSuppliedUserAgent": "Mozilla/5.0 (X11; Linux x86_64; rv:108.0) Gecko/20100101 Firefox/108.0,gzip(gfe),gzip(gfe)",
"requestAttributes": {
"time": "2023-01-11T13:15:30.667011Z",
"auth": {}
},
"destinationAttributes": {}
},
"serviceName": "cloudfunctions.googleapis.com",
"methodName": "google.cloud.functions.v1.CloudFunctionsService.UpdateFunction",
"authorizationInfo": [
{
"resource": "projects/my-project/locations/us-central1/functions/ga4-to-s3-1",
"permission": "cloudfunctions.functions.update",
"granted": true,
"resourceAttributes": {}
}
],
"resourceName": "projects/my-project/locations/us-central1/functions/ga4-to-s3-1",
"request": {
"updateMask": "entryPoint,sourceUploadUrl",
"#type": "type.googleapis.com/google.cloud.functions.v1.UpdateFunctionRequest",
"function": {
"name": "projects/my-project/locations/us-central1/functions/ga4-to-s3-1",
"runtime": "python39",
"serviceAccountEmail": "my-project#appspot.gserviceaccount.com",
"availableMemoryMb": 256,
"maxInstances": 3000,
"timeout": "60s",
"eventTrigger": {
"eventType": "google.pubsub.topic.publish",
"resource": "projects/my-project/topics/ga4-daily-extract-complete"
},
"secretEnvironmentVariables": [
{
"version": "latest",
"key": "PAT",
"secret": "PAT-GA4-S3-Extract",
"projectId": "1234567"
}
],
"sourceUploadUrl": "https://storage.googleapis.com/uploads-1234567.us-central1.cloudfunctions.appspot.com/123-456-789-abc-def.zip?GoogleAccessId=service-123456789#gcf-admin-robot.iam.gserviceaccount.com&Expires=12345&Signature=kjhgfghjkjhg%iuytfrghj8765467uhgfdfghj",
"entryPoint": "callRequest",
"ingressSettings": "ALLOW_ALL"
}
},
"resourceLocation": {
"currentLocations": [
"us-central1"
]
}
},
"insertId": "nlbq4xd9dhq",
"resource": {
"type": "cloud_function",
"labels": {
"project_id": "my-project",
"function_name": "ga4-to-s3-1",
"region": "us-central1"
}
},
"timestamp": "2023-01-11T13:15:30.423213Z",
"severity": "NOTICE",
"logName": "projects/my-project/logs/cloudaudit.googleapis.com%2Factivity",
"operation": {
"id": "operations/Z2E0LWV4dHJhY3QvdXMtY2VudHJhbDEvZ2E0LXRvLXMzLTEvbHA2QlowNzBTekk",
"producer": "cloudfunctions.googleapis.com",
"first": true
},
"receiveTimestamp": "2023-01-11T13:15:31.626931279Z"
}
I'm unsure where else to look? Any pointers or advice most welcome.
Found the similar issue discussed here and the issue resolved.
cloud build service account was missing the Cloud Build Service Account role.
I tried removing the Cloud Build Service Account role and deployed the function, I also got the same deployment errors
Try adding the Cloud Build Service Account role for the Google Cloud Build Service Account (project-number#cloudbuild.gserviceaccount.com) in the Google Cloud IAM console . This fixed symptom of a cloud function deploy with the message:
message=Build failed: {
"metrics":{},
"error":{
"buildpackId":"",
"buildpackVersion":"",
"errorType":"OK",
"canonicalCode":"OK",
"errorId":"",
"errorMessage":""
}
}
Also have a look at this github link1 & link2 which might help

How to get metric log insights for lambdas using boto3 (python sdk)

I want to get the MemoryUsedInMB attribute from Lambda CloudWatch logs insights. I tried the get_metrics_data function with invocations but was unsure on what the dimensions should be. Here is the command I used
aws cloudwatch get-metric-data --cli-input-json file://test_file.json
Here is the test_file.json
{
"MetricDataQueries": [
{
"Id": "myRequest",
"MetricStat": {
"Metric": {
"Namespace": "AWS/Lambda",
"MetricName": "Invocations"
},
"Period": 3600,
"Stat": "Sum"
},
"Label": "myRequestLabel",
"ReturnData": true
}
],
"StartTime": "2022-05-31T10:40:0000",
"EndTime": "2022-05-31T14:12:0000"
}
You need to use the following dimensions for the lambda function
'Dimensions': [
{
'Name': 'FunctionName',
'Value': 'YourFunctionNameHere'
}
]

All my Cloud Functions say, function is active but last deploy failed

Facing this issue with my Google Cloud Functions where from the very first function that I deployed to the ones I'm to upgrade today, are all saying the same thing on their status.
"Function is active, but the last deploy failed"
What may this be?
Here's the log visible for updating the function on the log explorer.
{
"protoPayload": {
"#type": "type.googleapis.com/google.cloud.audit.AuditLog",
"status": {},
"authenticationInfo": {
"principalEmail": "start#pyme.team"
},
"serviceName": "cloudfunctions.googleapis.com",
"methodName": "google.cloud.functions.v1.CloudFunctionsService.UpdateFunction",
"resourceName": "projects/pyme-webapp/locations/us-central1/functions/applicationSubmitted"
},
"insertId": "d1k3hyd3jfe",
"resource": {
"type": "cloud_function",
"labels": {
"region": "us-central1",
"function_name": "applicationSubmitted",
"project_id": "pyme-webapp"
}
},
"timestamp": "2022-02-02T20:23:05.726462Z",
"severity": "NOTICE",
"logName": "projects/pyme-webapp/logs/cloudaudit.googleapis.com%2Factivity",
"operation": {
"id": "operations/cHltZS13ZWJhcHAvdXMtY2VudHJhbDEvYXBwbGljYXRpb25TdWJtaXR0ZWQvaWdGS2o4bXpjbDA",
"producer": "cloudfunctions.googleapis.com",
"last": true
},
"receiveTimestamp": "2022-02-02T20:23:06.263576440Z"
}
Similarly, all I see on the log in the function itself is:
Image of the Function Log itself available
The exact error that I am seeing and am concerned about and with is this: Function Error with ORANGE HAZARD on update
Attaching another, even more detailed update log as well.
{
"protoPayload": {
"#type": "type.googleapis.com/google.cloud.audit.AuditLog",
"authenticationInfo": {
"principalEmail": "start#pyme.team"
},
"requestMetadata": {
"callerIp": "80.83.136.68",
"callerSuppliedUserAgent": "FirebaseCLI/10.0.1,gzip(gfe),gzip(gfe)",
"requestAttributes": {
"time": "2022-02-02T20:21:00.491300Z",
"auth": {}
},
"destinationAttributes": {}
},
"serviceName": "cloudfunctions.googleapis.com",
"methodName": "google.cloud.functions.v1.CloudFunctionsService.UpdateFunction",
"authorizationInfo": [
{
"resource": "projects/pyme-webapp/locations/us-central1/functions/workContracts",
"permission": "cloudfunctions.functions.update",
"granted": true,
"resourceAttributes": {}
}
],
"resourceName": "projects/pyme-webapp/locations/us-central1/functions/workContracts",
"request": {
"updateMask": "name,sourceUploadUrl,entryPoint,runtime,labels,httpsTrigger,availableMemoryMb,environmentVariables,sourceToken",
"function": {
"runtime": "nodejs16",
"availableMemoryMb": 512,
"entryPoint": "workContracts",
"name": "projects/pyme-webapp/locations/us-central1/functions/workContracts",
"sourceUploadUrl": "https://storage.googleapis.com/gcf-upload-us-central1-d393f99f-6b88-4b68-8202-d75b734aa7a1/64b2646f-35b6-4919-8e89-c662fc29f01f.zip?GoogleAccessId=service-748321615979#gcf-admin-robot.iam.gserviceaccount.com&Expires=1643835053&Signature=McjqD9mmo%2F1wLbvO6SklkHi%2B34nQEwcpz7cLOLNAF4RwG8bpHh8RThxFJwnGZo1F92iQnquRQyGYbJFuihP%2FUGrgW7cG6GmhVq2gkugDywngZXT9d7UTBG0wgKF29XcbZkwV3IX7oKKiUwf6Q6mzCOOoCrjc5LBxqJo9WvWDZynv8R75nVZTZ5IhekMdqAw%2BRvIBvooXa%2BuA3Sezhh%2Bz2BR1XtIyS21CY%2FkoPDaKPwvftr3%2Fjcyuzb2V39%2BSajQg3t0U7Gt6oSch9qUhl6gnknr6wphFGmC7t7h9l0LUbjHUDuaMNNoB1LXxI30CRNkRupf9XBKTKpKMf%2F0nAAMltA%3D%3D",
"httpsTrigger": {},
"labels": {
"deployment-tool": "cli-firebase"
}
},
"#type": "type.googleapis.com/google.cloud.functions.v1.UpdateFunctionRequest"
},
"resourceLocation": {
"currentLocations": [
"us-central1"
]
}
},
"insertId": "1g6c2gwd46lm",
"resource": {
"type": "cloud_function",
"labels": {
"region": "us-central1",
"function_name": "workContracts",
"project_id": "pyme-webapp"
}
},
"timestamp": "2022-02-02T20:21:00.307699Z",
"severity": "NOTICE",
"logName": "projects/pyme-webapp/logs/cloudaudit.googleapis.com%2Factivity",
"operation": {
"id": "operations/cHltZS13ZWJhcHAvdXMtY2VudHJhbDEvd29ya0NvbnRyYWN0cy96bHlTLUtwbzI2VQ",
"producer": "cloudfunctions.googleapis.com",
"first": true
},
"receiveTimestamp": "2022-02-02T20:21:00.985842395Z"
}
If this isn't the log to look for, just let me know what to find but I'd appreciate the help.
So turns out today morning, I login and check and everything is fine. I still have no logs stating the exact cause of the error but the same functions, the same code and the exact same deployment methods have worked and the function seems to be working fine.
This is concerning as separate cloud functions should never ever be changing on deployments.
A cloud function which takes in a POST METHOD and send data to SendGrid for example has nothing to do with a cloud function triggered by updates to the Firestore Database and if they're both deployed since the 5th of January and never touched again (in terms of edits), they should not be showing the same deployment error message across the board.
my temporal solution is to delete the function then deploy. It seems like it cannot be deployed while in use, i'm sorry i couldn't provide a better solution i will edit it as soon as possible.

How to export an existing CodePipeline to CloudFormation template

This is more of a lack of understanding on my part but I cannot seem to debug this.
I have created an codepipeline which runs terraform apply ( which internally creates the aws infrastructure for me ). the codepipeline seems to be working.
I need to implement the same codepipeline for another account, how can I do so.
I tried to get the json script using the below command.
aws codepipeline get-pipeline --name
I convert json script to yaml script.
When I try to run the yaml script on another account I get below error
Template format error: At least one Resources member must be defined.
ISSUES:
1.) Best Way I can export codepipeline to cloudformation template
2.) The approach which I used didn't work, how to solve it?
{
"pipeline": {
"name": "my-code-pipeline",
"roleArn": "arn:aws:iam::aws-account-id:role/service-role/AWSCodePipelineServiceRole-aws-region-my-code-pipeline",
"artifactStore": {
"type": "S3",
"location": "codepipeline-aws-region-45856771421"
},
"stages": [
{
"name": "Source",
"actions": [
{
"name": "Source",
"actionTypeId": {
"category": "Source",
"owner": "ThirdParty",
"provider": "GitHub",
"version": "1"
},
"runOrder": 1,
"configuration": {
"Branch": "master",
"OAuthToken": "****",
"Owner": "github-account-name",
"PollForSourceChanges": "false",
"Repo": "repo-name"
},
"outputArtifacts": [
{
"name": "SourceArtifact"
}
],
"inputArtifacts": [],
"region": "aws-region",
"namespace": "SourceVariables"
}
]
},
{
"name": "codebuild-for-terraform-init-and-plan",
"actions": [
{
"name": "codebuild-for-terraform-init",
"actionTypeId": {
"category": "Build",
"owner": "AWS",
"provider": "CodeBuild",
"version": "1"
},
"runOrder": 1,
"configuration": {
"ProjectName": "my-code-pipeline-build-stage"
},
"outputArtifacts": [],
"inputArtifacts": [
{
"name": "SourceArtifact"
}
],
"region": "aws-region"
}
]
},
{
"name": "manual-approve",
"actions": [
{
"name": "approval",
"actionTypeId": {
"category": "Approval",
"owner": "AWS",
"provider": "Manual",
"version": "1"
},
"runOrder": 1,
"configuration": {
"NotificationArn": "arn:aws:sns:aws-region:aws-account-id:Email-Service"
},
"outputArtifacts": [],
"inputArtifacts": [],
"region": "aws-region"
}
]
},
{
"name": "codebuild-for-terraform-apply",
"actions": [
{
"name": "codebuild-for-terraform-apply",
"actionTypeId": {
"category": "Build",
"owner": "AWS",
"provider": "CodeBuild",
"version": "1"
},
"runOrder": 1,
"configuration": {
"ProjectName": "codebuild-project-for-apply"
},
"outputArtifacts": [],
"inputArtifacts": [
{
"name": "SourceArtifact"
}
],
"region": "aws-region"
}
]
}
],
"version": 11
},
"metadata": {
"pipelineArn": "arn:aws:codepipeline:aws-region:aws-account-id:my-code-pipeline",
"created": "2020-09-17T13:12:50.085000+05:30",
"updated": "2020-09-21T15:46:19.613000+05:30"
}
}
The given code is the yaml template that I used to create cloudformation template.
The aws codepipeline get-pipeline --name CLI command returns information about the pipeline structure and pipeline metadata, but it is not the same format as a CloudFormation template (or the resource part of it).
There is no built-in support for exporting existing AWS resources to create a CloudFormation template, though you do have a couple of options.
Use former2 (built and maintained by AWS Hero, Ian Mckay) to generate a CloudFormation template from the resources you select.
Take the JSON output from the aws codepipeline get-pipeline --name command you used and manually craft a CloudFormation template. The pipeline will be one resource in the list of resources in the full template. The info it contains is pretty close, but needs some adjustments to conform to the CloudFormation resource specification for a CodePipeline, which you can find here. You'll also need to do the same for other resources that you need to bring into the template, with aws <service name> describe.
If you go with option 2 (and even if you don't), I recommend using cfn-lint with your code editor to help adhere to the spec.

Why is GetMetricData returning an empty set of values?

Using the JS AWS SDK and passing the following parameters:
{
"StartTime": 1548111915,
"EndTime": 1549321515,
"MetricDataQueries": [
{
"Id": "m1",
"MetricStat": {
"Metric": {
"MetricName": "NetworkOut",
"Namespace": "AWS/EC2",
"Dimensions": [
{
"Name": "InstanceId",
"Value": "i-[redacted]"
}
]
},
"Period": 300,
"Stat": "Average",
"Unit": "Gigabytes"
}
}
]
}
This is the output:
[
{
"Id": "m1",
"Label": "NetworkOut",
"Timestamps": [],
"Values": [],
"StatusCode": "Complete",
"Messages": []
}
]
The query closely matches the sample request found at https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html#API_GetMetricData_Examples
I am sure that the instance is a valid instance that has definitely had NetworkOut traffic during that date range.
What reason could account for the lack of elements in Values array?
A better solution was to omit "Unit" altogether, which allowed AWS to choose the appropriate unit, not only in scale but in category.
I tried it and got the same (empty) result as you.
I then changed Gigabytes to Bytes and got a result. So, it could be that you need to reduce your Unit size.
Here's the command I used for the AWS CLI:
aws cloudwatch get-metric-data --start-time 1548111915 --end-time 1549321515 --metric-data-queries '[
{
"Id": "m1",
"MetricStat": {
"Metric": {
"MetricName": "NetworkOut",
"Namespace": "AWS/EC2",
"Dimensions": [
{
"Name": "InstanceId",
"Value": "i-xxx"
}
]
},
"Period": 300,
"Stat": "Average",
"Unit": "Bytes"
}
}
]'
For future inquisitors, there are multiple reasons for which the aws cli silently returns an empty dataset instead of an error, because the input requirements are stricter than the standard user's expectations, but the output requirements are much looser. Examples
wrong unit
incomplete list of dimensions
typos, case-sensitivity, etc
References:
https://aws.amazon.com/premiumsupport/knowledge-center/cloudwatch-getmetricstatistics-data/
https://github.com/grafana/grafana/issues/9852#issuecomment-395023506