I got this log message:
com.amazonaws.services.s3.model.AmazonS3Exception: The specified key
does not exist. (Service: Amazon S3; Status Code: 404; Error Code:
NoSuchKey; Request ID: request_id; S3 Extended Request ID:extended_request_id)
Is it possible to get parameters for the request (in this case S3 key and bucket) by request_id and extended_request_id ?
The Request ID is received on the wire as x-amz-request-id and is styled as the Request ID in the S3 access logs..
This will not provide exhaustive information about the request parameters, depending on what you are trying to find, but it will show the bucket and key -- though you'll have to know which bucket's logs to look in, of course.
More detailed information about the request can potentially found in CloudTrail logs. Object level requests are not captured by CloudTrail by default, so you'd need to enable this. The request ID should appear in these logs as well.
The Extended Request ID, also called x-amz-id-2 is -- as far as I am aware -- only of use to AWS support when tracing things internally for you. Neither value is known to contain sensitive information. The extended ID may be a large random number or may be encrypted, but if it is encrypted, there is no documented way to decrypt it. The documentation calls it a "special token." A little bit more detail in the context of support is here.
In summary, there is no short/simple "lookup" method but it is possible, as noted above.
If you go to CloudWatch Log Insights you can run a query:
fields #requestId, #message, #timestamp | filter #message like /\"requestID\":\"REQUEST_ID\"/
You may see suggestions of something like:
fields #timestamp, #message
| filter #message like /REQUEST_ID/
This works but with continued testing it will start to fill up your search results with the searches you have done, so the first approach is better.
Related
I have api.log logs being sent to CloudWatch and I want to create a metric filter to extract the userId of the user tried to access application.
A sample log entry looks like:
2022-12-06T19:13:59.329Z 2a-b0bc-7a79c791f19c INFO Validated that user fakeId has access to the following gated roles: create, update and delete
And the value I would like to extract is: fakeId
I read through this guide and it seems pretty straight forward because user [fakeId] seconds is unique to just this line. This guide on metric filter syntax seems to only show examples for extracting values from JSON logs and this official example list doesn't cover it.
Based on the documentation and a few other stackoverflow answers, I tried these things:
[validation="Validated", that="that", user="user", userId, ...]
[,,user="user",userId,...]
[,,user=user,userId,...]
but it didn't. Any help would be really appreciated!
I am using a graphql API with AppSync that receives post requests from a lambda function that is triggered by AWS IoT with sensor data in the following JSON format:
{
"scoredata": {
"id": "240",
"distance": 124,
"timestamp": "09:21:11",
"Date": "04/16/2022"
}
}
The lambda function uses this JSON object to perform a post request on the graphql API, and AppSync puts this data in DynamoDB to be stored. My issue is that whenever I parse the JSON object within my lambda function to retrieve the id value, the id value does not match with the id value stored in DynamoDB; appsync is seemingly automatically generating an id.
Here is a screenshot of the request made to the graphql api from cloudwatch:
Here is what DynamoDB is storing:
I would like to know why the id in DynamoDB is shown as 964a3cb2-1d3d-4f1e-a94a-9e4640372963" when the post request id value is "240" and if there is anything I can do to fix this.
I can’t tell for certain but i’m guessing that dynamo db schema is autogenerating the id field on insert and using a uuid as the id type. An alternative would be to introduce a new property like score_id to store this extraneous id.
If you are using amplify most likely the request mapping templates you are generating automatically identify the "id" field as a unique identifier to be generated at runtime.
I recommend you to take a look at your VTL request template, you will most likely find something like this:
$util.qr($context.args.input.put("id", $util.defaultIfNull($ctx.args.input.id, $util.autoId())))
Surely the self-generated id comes from $util.autoId()
Probably some older version of Amplify could omit the verification $util.defaultIfNull($ctx.args.input.id,... and always overwrite the id by self-generating it.
Is there a simple way to retrieve all items from a DynamoDB table using a mapping template in an API Gateway endpoint? I usually use a lambda to process the data before returning it but this is such a simple task that a Lambda seems like an overkill.
I have a table that contains data with the following format:
roleAttributeName roleHierarchyLevel roleIsActive roleName
"admin" 99 true "Admin"
"director" 90 true "Director"
"areaManager" 80 false "Area Manager"
I'm happy with getting the data, doesn't matter the representation as I can later transform it further down in my code.
I've been looking around but all tutorials explain how to get specific bits of data through queries and params like roles/{roleAttributeName} but I just want to hit roles/ and get all items.
All you need to do is
create a resource (without curly braces since we dont need a particular item)
create a get method
use Scan instead of Query in Action while configuring the integration request.
Configurations as follows :
enter image description here
now try test...you should get the response.
to try it out on postman deploy the api first and then use the provided link into postman followed by your resource name.
API Gateway allows you to Proxy DynamoDB as a service. Here you have an interesting tutorial on how to do it (you can ignore the part related to index to make it work).
To retrieve all the items from a table, you can use Scan as the action in API Gateway. Keep in mind that DynamoDB limits the query sizes to 1MB either for Scan and Query actions.
You can also limit your own query before it is automatically done by using the Limit parameter.
AWS DynamoDB Scan Reference
I am trying to write a Lua library for Amazon SES that will allow me to send API requests. I've poured over the documentation and various examples but I am continuing to get the following error:
The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.
One of my functions somewhere along the line is formatting data incorrectly or doing something to cause the end result of my signing process to not match what Amazon is generating on their side, so my request is being rejected. However, Amazon doesn't provide any useful information in their error response such as showing me the canonical request that they generate so I can compare it to mine to see if there are any discrepencies. My best attempts to debug this is to use the examples they provide in their documentation (see below) as "known good" comparisons and to try and generate the same hashes with my functions... Except that they don't provide all of the necessary information to do so.
In Task 3 of their documentation process, they do share an example secrey key and I've been able to use that to verify that at least part of my code is indeed working as intended, but that key does not seem to generate the same hashes on the other tasks. Am I missing something here, or is there a better way to figure this problem out?
Below are the example keys I was able to pull out of various Task pages in their documentation:
api_key = "AKIDEXAMPLE"
api_secret = "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY"
In Amazon's Documentation for Task 1, they provide the final canonical request and a paired hash:
GET
/
Action=ListUsers&Version=2010-05-08
content-type:application/x-www-form-urlencoded; charset=utf-8
host:iam.amazonaws.com
x-amz-date:20150830T123600Z
content-type;host;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
f536975d06c0309214f805bb90ccff089219ecd68b2577efef23edd43b7e1a59
However when I use the above secret to hash the above canonical request, I get a different hash:
d2da54b4842d8ca1acf1cf197827f4d75a742918af868d472e883781624a8bb5
So they must being using a different secret in some examples without actually documenting them.. unless I missed something?
Documentation: https://docs.aws.amazon.com/general/latest/gr/sigv4_signing.html
WIP Code: https://hastebin.com/mezugukefu.lua
I am looking to setup some CloudFormation stuff that is able to find any email addresses in CloudWatch logs and let us know that one slipped through the cracks. I thought this would be a simple process of using a RegEx pattern that catches all the possible variations and email address can have, and using that as a filter. Having discovered that CloudWatch filtering does not support RegEx I've become a bit stumped as to how to write a filter that can be relied upon to catch any email address.
Has anyone done something similar to this, or know where a good place to start would be?
Amazon has launched a service called CloudWatch insights and it allows to filter messages logs. In the previous link you have examples of queries.
You need to select the CloudWatch Log Group and the period of time in which search.
Example:
fields #message
| sort #timestamp desc
| filter #message like /.*47768.*/
If you're exporting the logs somewhere (Like Sumologic, Datadog etc) thats a better place to do that alerting.
If not and you're exporting them into S3 then a triggered lambda function that runs the check might do the trick. Could be expensive long term though.
The solution that we landed upon was to pass stings through a RegEx pattern that recognises email addresses before they logged into AWS. Replacing any matches with [REDACTED]. Which is simple enough to do in a lambda.