Today I created datashare but when I try to run query on it. I get below error:
Query: select * from postgress_home_db.staging_datashare.site limit 100
ERROR: Publicly accessible consumer cannot access object in the database.
I tried to search reason behind it but did not get anything.
Below are the queries I used to create datashare.
Producer cluster:
1. CREATE DATASHARE postgres_home_ds;
2. ALTER DATASHARE postgres_home_ds ADD SCHEMA postgres_home_pod;
3. GRANT USAGE ON DATASHARE postgres_home_ds to NAMESPACE 'xyz'
Consumer Cluster:
CREATE DATABASE postgress_home_db from DATASHARE postgres_home_ds of NAMESPACE 'abc'
CREATE EXTERNAL SCHEMA postgress_home_datashare FROM REDSHIFT DATABASE 'postgress_home_db' SCHEMA 'staging_datashare'
I was able to fix the issue by setting PUBLICACCESSIBLE=True
ALTER DATASHARE datashare_name SET PUBLICACCESSIBLE=True
Related
I use Amazon Redshift Serverless and Query editor(v2) and I'm having trouble with user permissions.
The following error occurred when importing data(.csv) in S3.
ERROR: Load into table 'x' failed. Check 'sys_load_error_detail' system table for details.
Therefore, I executed the command Select * From stl_load_errors to check the error, but it did not work well.
ERROR: permission denied for relation stl_load_errors
I checked my user permissions using select * from pg_user;and they are presented as follows.
However, I don't see any problem, what is the problem?
(I use hoge.)
usename
usesysid
usecreatedb
usesuper
usecatupd
passwd
valuntil
useconfig
rdsdb
1
true
true
true
********
infinity
NULL
hoge
101
true
true
false
********
NULL
NULL
I have tried to look at the query in the Amazon Redshift Serverless (Preview) dashboard under "Query and Database Monitoring", but could not find any details about the error.
What should I do?
AWS Redshift serverless is not exposing the stl_load_errors table
Perhaps you should try SYS_LOAD_HISTORY and SYS_LOAD_ERROR_DETAIL.
https://docs.aws.amazon.com/redshift/latest/mgmt/serverless-monitoring.html
You can't query STL, STV, SVCS, SVL, and some SVV system tables and views with Amazon Redshift Serverless, except the following:
...see link above....
try
select * from sys_load_error_detail
I prepared a schema and tables using AWS SCT tool so my DMS job will have a landing place for data.
Even thought access to the database and schema has been granted:
GRANT ALL ON DATABASE my_db TO "dms_user";
GRANT ALL ON SCHEMA my_schema TO "dms_user";
GRANT ALL ON ALL TABLES IN SCHEMA my_schema TO "dms_user";
ALTER DEFAULT PRIVILEGES IN SCHEMA my_schema GRANT ALL ON TABLES TO "dms_user";
I'm getting error:
2022-03-25T22:26:48 [TARGET_LOAD ]E: RetCode: SQL_ERROR SqlState: XX000 NativeError: 30 Message: [Amazon][Amazon Redshift] (30) Error occurred while trying to execute a query: [SQLState XX000] ERROR: Load into table 'table_test' failed. Check 'stl_load_errors' system table for details. [1022502] (ar_odbc_stmt.c:4815)
2022-03-25T22:26:48 [TARGET_LOAD ]E: Failed to load schema.table_testfrom S3, file name: LOAD00000001.csv [1022509] (cloud_imp.c:2386)
2022-03-25T22:26:48 [TARGET_LOAD ]E: Failed to load ims_suretyradm_publish.dimaccount from S3, file name: LOAD00000001.csv [1022509] (cloud_imp.c:2386)
stl_load_errors table is empty...
I'll greatly appreciate any help/guidance on this.
I hope it is not the difference between "my_schema" and "my_schema_name" as these are likely different due to obfuscation error.
There are a number of places things can go sideways. Have you checked the permissions after the grant?
select HAS_SCHEMA_PRIVILEGE('dms_user', 'my_schema', 'create');
select HAS_SCHEMA_PRIVILEGE('dms_user', 'my_schema', 'usage');
Another resource is awslabs' Redshift github repo - https://github.com/awslabs/amazon-redshift-utils - there are a number of admin views there that explore permissions. Knowing which step in the process is not doing what you expect will narrow things down.
Also, remember that you will want to change the default ACL for the schema so the new objects created will be useable by the correct people. For example:
ALTER DEFAULT PRIVILEGES IN SCHEMA my_schema GRANT ALL ON TABLES TO dms_user;
I am trying to create a datashare in Redshift by following this documentation. When I type this command:
CREATE DATASHARE datashare_name
I get this message:
ERROR: CREATE DATASHARE is not enabled.
I also tried to make it using console, but same issue.
So how to enable data sharing in Redshift ?
From the documents:here
Data sharing via datashare is only available for ra3 instance types
The document lists ra3.16xlarge, ra3.4xlarge, and ra3.xlplus instance types for producer and consumer clusters.
So, if I were in your place - I would first go back and check my instance type. If still not sure, drop a simple CS ticket and ask them if anything has changed recently & documentation is not updated
I'm trying to connect to AWS RDS database from my code:
const {
DATABASE_HOSTNAME,
DATABASE_BASENAME, // ches
DATABASE_USERNAME,
DATABASE_PASSWORD,
} = process.env;
this.sequelize = new Sequelize(DATABASE_BASENAME, DATABASE_USERNAME, DATABASE_PASSWORD, {
dialect: 'mysql',
host: DATABASE_HOSTNAME,
});
This is my AWS RDS database. However when I run my code I'm getting error:
code: 'ER_BAD_DB_ERROR',
errno: 1049,
sqlState: '42000',
sqlMessage: "Unknown database 'ches'"
What I'm doing wrong? Is empty "DB name" field in instance configuration source of problems?
From your screenshot it can be seen that there is no default DB created.
DB Name:
-
By default, when you setup your RDS, it does not create a database for you. Instead you have a DB instance, which is listed under RDS->Databases.
When you setup your RDS you can use Additional configuration section to provide a name of database to create, e.g. ches. Without this you have a DB instance without an actual database.
For existing rds instances, you can use mysql client to connect to the instance and create a database using standard CREATE DATABASE ches; as you would normally do in a mysql.
I have an Athena database set up pointing at an S3 bucket containing ALB logs, and it all works correctly. I partition the table by a column called datetime and the idea is that it has the format YYYY/MM/DD.
I can manually create partitions through the Athena console, using the following command:
ALTER TABLE alb_logs ADD IF NOT EXISTS PARTITION (datetime='2019-08-01') LOCATION 's3://mybucket/AWSLogs/myaccountid/elasticloadbalancing/eu-west-1/2019/08/01/'
I have created a lambda to run daily to create a new partition, however this doesn't seem to work. I use the boto3 python client and execute the following:
result = athena.start_query_execution(
QueryString = "ALTER TABLE alb_logs ADD IF NOT EXISTS PARTITION (datetime='2019-08-01') LOCATION 's3://mybucket/AWSLogs/myaccountid/elasticloadbalancing/eu-west-1/2019/08/01/'",
QueryExecutionContext = {
'Database': 'web'
},
ResultConfiguration = {
"OutputLocation" : "s3://aws-athena-query-results-093305704519-eu-west-1/Unsaved/"
}
)
This appears to run successfully without any errors and the query execution even returns a QueryExecutionId as it should. However if I run SHOW PARTITIONS web.alb_logs; via the Athena console it hasn't created the partition.
I have a feeling it could be down to permissions, however I have given the lambda execution role full permissions to all resources on S3 and full permissions to all resources on Athena and it still doesn't seem to work.
Since Athena query execution is asynchronous your Lambda function never sees the result of the query execution, it just gets the result of starting the query.
I would be very surprised if this wasn't a permissions issue, but because of the above the error will not appear in the Lambda logs. What you can do is to log the query execution ID and look it up with the GetQueryExecution API call to see that the query succeeded.
Even better would be to rewrite your code to use the Glue APIs directly to add the partitions. Adding a partition is a quick and synchronous operation in Glue, which means you can make the API call and get a status in the same Lambda execution. Have a look at the APIs for working with partitions: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-catalog-partitions.html