I'm using DynamoDB to make a Room Booking Website on Django. Every time I refresh the page, the console throws me a ResourceNotFoundException - Requested resource not found, which seems to be happening when I use table.scan(). On certain pages the table still loads, but on others' I'm shown a debug error.
Here's a part of my code:
dynamodb = boto3.resource(
'dynamodb',
aws_access_key_id="XXXXXX",
aws_secret_access_key="XXXXXX",
region_name="eu-west-2"
)
table = dynamodb.Table(table_name)
response = table.scan(TableName=table_name)
I'm entirely sure that the table_name value contains the correct string.
What could be the problem?
You are confusing the Client-level scan method with the Resource-level scan method. The former requires you to provide a TableName parameter, while the latter does not (because it's a method on an existing Table object, so the table name is implicitly known).
Also, see Difference in boto3 between resource, client, and session?
Related
I am using AWS proxy with AWS API Gateway to interact with a DynamoDB table. I have an API resource, under which I have a GET method with the below configuration:
The API uses the Scan action as seen above to fetch all the items from the DynamoDB table. I also have the following request integration mapping template;
{
"TableName": tableName
}
Its really simple. But my problem is that I would like to add another GET method to get each item by their id, which will be supplied in the URL as a param. However, since I have already setup one GET method, I am not able to setup another to fetch only a single item. I am aware I can use mapping templates and Scan as given in the docs to conditionally fetch items if a param is given, but that would mean scanning the entire table, which is a waste each time I want to fetch a single item.
Is there any other way to do this?
I have an Athena database set up pointing at an S3 bucket containing ALB logs, and it all works correctly. I partition the table by a column called datetime and the idea is that it has the format YYYY/MM/DD.
I can manually create partitions through the Athena console, using the following command:
ALTER TABLE alb_logs ADD IF NOT EXISTS PARTITION (datetime='2019-08-01') LOCATION 's3://mybucket/AWSLogs/myaccountid/elasticloadbalancing/eu-west-1/2019/08/01/'
I have created a lambda to run daily to create a new partition, however this doesn't seem to work. I use the boto3 python client and execute the following:
result = athena.start_query_execution(
QueryString = "ALTER TABLE alb_logs ADD IF NOT EXISTS PARTITION (datetime='2019-08-01') LOCATION 's3://mybucket/AWSLogs/myaccountid/elasticloadbalancing/eu-west-1/2019/08/01/'",
QueryExecutionContext = {
'Database': 'web'
},
ResultConfiguration = {
"OutputLocation" : "s3://aws-athena-query-results-093305704519-eu-west-1/Unsaved/"
}
)
This appears to run successfully without any errors and the query execution even returns a QueryExecutionId as it should. However if I run SHOW PARTITIONS web.alb_logs; via the Athena console it hasn't created the partition.
I have a feeling it could be down to permissions, however I have given the lambda execution role full permissions to all resources on S3 and full permissions to all resources on Athena and it still doesn't seem to work.
Since Athena query execution is asynchronous your Lambda function never sees the result of the query execution, it just gets the result of starting the query.
I would be very surprised if this wasn't a permissions issue, but because of the above the error will not appear in the Lambda logs. What you can do is to log the query execution ID and look it up with the GetQueryExecution API call to see that the query succeeded.
Even better would be to rewrite your code to use the Glue APIs directly to add the partitions. Adding a partition is a quick and synchronous operation in Glue, which means you can make the API call and get a status in the same Lambda execution. Have a look at the APIs for working with partitions: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-catalog-partitions.html
I have a setup on AWS RDS with MariaDB 10.3. I have several DBs on the RDS instance. I'm trying to replicate a table (routes) from one DB (att) to another DB (pro) using triggers. I have triggers for create, update and delete. The create and update triggers works fine while the delete trigger gives the error message below. I've tested all triggers locally and they work.
My trigger looks like this.
CREATE DEFINER=`root`#`%` TRIGGER routes_delete AFTER DELETE ON
`routes` FOR EACH ROW
BEGIN
DELETE FROM `pro`.`routes`
WHERE `route_id` = OLD.route_id;
END
Error message
Query execution failed
Reason:
SQL Error [1442] [HY000]: (conn:349208) Can't update table 'routes' in
stored function/trigger because it is already used by statement which
invoked this stored function/trigger
Query is : DELETE FROM `att`.routes WHERE route_code = 78 AND company_id = 3
I don't understand what other statement is using the routes table since there is nothing else linked to it. What adjustment is needed to get this work on AWS RDS?
what other statement is using the routes table
The "other" query is the query that invoked the trigger.
...is already used by [the] statement which invoked this stored function/trigger
A trigger is not allowed to modify its own table. BEFORE INSERT and BEFORE UPDATE triggers can modify the current row before it is written to the table using the NEW alias, but that is the extent to which a trigger can modify the table where it is defined.
Triggers are subject to all the same limitations as stored functions, and a stored function...
Cannot make changes to a table that is already in use (reading or writing) by the statement invoking the stored function.
https://mariadb.com/kb/en/library/stored-function-limitations/
I have a problem with connection to DynamoDB. I get this exception:
com.amazonaws.services.dynamodb.model.ResourceNotFoundException:
Requested resource not found (Service: AmazonDynamoDB; Status Code:
400; Error Code: ResourceNotFoundException; Request ID: ..
But I have a table and region is correct.
From the docs it's either you don't have a Table with that name or it is in CREATING status.
I would double check to verify that the table does in fact exist, in the correct region, and you're using an access key that can reach it
My problem was stupid but maybe someone has the same... I changed recently the default credentials of aws (~/.aws/credentials), I was testing in another account and forgot to rollback the values to the regular account.
I spent 1 day researching the problem in my project and now I should repay a debt to humanity and reduce the entropy of the universe a little.
Usually, this message says that your client can't reach a table in your DB.
You should check the next things:
1. Your database is running
2. Your accessKey and secretKey are valid for the database
3. Your DB endpoint is valid and contains correct protocol ("http://" or "https://"), and correct hostname, and correct port
4. Your table was created in the database.
5. Your table was created in the database in the same region that you set as a parameter in credentials. Optional, because some
database environments (e.g. Testcontainers Dynalite) don't have an incorrect value for the region. And any nonempty region value will be correct
In my case problem was that I couldn't save and load data from a table in tests with DynamoDB substituted by Testcontainers and Dynalite. I found out that in our project tables creates by Spring component marked with #Component annotation. And in tests, we are using a global setting for lazy loading components to test, so our component didn't load by default because no one call it in the test explicitly. ¯_(ツ)_/¯
If DynamoDB table is in a different region, make sure to set it before initialising the DynamoDB by
AWS.config.update({region: "your-dynamoDB-region" });
This works for me:)
Always ensure that you do one of the following:
The right default region is set up in the AWS CLI configuration files on all the servers, development machines that you are working on.
The best choice is to always specify these constants explicitly in a separate class/config in your project. Always import this in code and use it in the boto3 calls. This will provide flexibility if you were to add or change based on the enterprise requirements.
If your resources are like mine and all over the place, you can define the region_name when you're creating the resource.
I do this for all my instantiations as it forces me to think about what I'm putting/calling where.
boto3.resource("dynamodb", region_name='us-east-2')
I was getting this issue in my .NetCore Application.
Following fixed the issue for me in Startup class --> ConfigureServices method
services.AddDefaultAWSOptions(
new AWSOptions
{
Region = RegionEndpoint.GetBySystemName("eu-west-2")
});
I got Error warning Lambda : lifecycleIteration=0 lambda handler returned an error: ResourceNotFoundException: Requested resource not found
I spent 1 week to fix the issue.
And so its root cause and steps to find issue is mentioned in below Git Issue thread and fixed it.
https://github.com/soto-project/soto/issues/595
Boto's S3 Key object contains last_modified date (available via parse_ts) but the base_field "date" (i.e., ctime) doesn't seem to be accessible, even though it's listed in key.base_fields.
Based on the table at http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html, it does seem that it is always automatically created (and I can't imagine a reason why it wouldn't be). It's probably just a simple matter of finding it somewhere in the object attributes, but I haven't been able to find it so far, although I did find the base_fields attribute which contains 'date'. (They're just a set and don't seem to have an available methods and I haven't been able to find documentation regarding ways to inspect them.)
For example, Amazon S3 maintains object creation date and size metadata and uses this information as part of object management.
Interestingly, create_time (system metadata field "Date" in link above) does not show up in the AWS S3 console, either, although last_modified is visible.
TL;DR: Because overwriting an S3 object is essentially creating a new one, the "last modified" and "creation" timestamp will always be the same.
Answering the old question, just in case others run into the same issue.
Amazon S3 maintains only the last modified date for each object.
For example, the Amazon S3 console shows the Last Modified date in the object Properties pane. When you initially create a new object, this date reflects the date the object is created. If you replace the object, the date changes accordingly. So when we use the term creation date, it is synonymous with the term last modified date.
Reference: https://docs.aws.amazon.com/AmazonS3/latest/dev/intro-lifecycle-rules.html
i suggest use
key.last_modified since key.date seems to return the last time you viewed the file
so something like this :
key = bucket.get_key(key.name)
print(key.last_modified)
After additional research, it appears that S3 key objects returned from a list() may not include this metadata field!
The Key objects returned by the iterator are obtained by parsing the results of a GET on the bucket, also known as the List Objects request. The XML returned by this request contains only a subset of the information about each key. Certain metadata fields such as Content-Type and user metadata are not available in the XML. Therefore, if you want these additional metadata fields you will have to do a HEAD request on the Key in the bucket. (docs)
In other words, looping through keys:
for key in conn.get_bucket(bucket_name).list():
print (key.date)
... does not return the complete key with creation date and some other system metadata. (For example, it's also missing ACL data).
Instead, to retrieve the complete key metadata, use this method:
key = bucket.get_key(key.name)
print (key.date)
This necessitates an additional HTTP request as the docs clearly state above. (See also my original issue report.)
Additional code details:
import boto
# get connection
conn = boto.connect_s3()
# get first bucket
bucket = conn.get_all_buckets()[0]
# get first key in first bucket
key = list(bucket.list())[0]
# get create date if available
print (getattr(key, "date", False))
# (False)
# access key via bucket.get_key instead:
k = bucket.get_key(key.name)
# check again for create_date
getattr(k, "date", False)
# 'Sat, 03 Jan 2015 22:08:13 GMT'
# Wait, that's the current UTC time..?
# Also print last_modified...
print (k.last_modified)
# 'Fri, 26 Apr 2013 02:41:30 GMT'
If you have versioning enabled for your S3 bucket, you can use list_object_versions and find the smallest date for the object you're looking for which should be the date it was created