I am creating a dataset in AWS Quicksight using custom SQL which I prepare/test in Athena. However, unless I define each join/table "databasename".table, the QS custom SQL fails. I have tried the below but it has failed. Is it possible to instruct the query to fun against a specific DB at the beginning of the query?
USING AwsDataCatalog."databasename"
In the data preparation, in the custom SQL page, on the left pane, you should be able to choose the database name (Schema).
If you do not set that, then it will use Athena's default schema so you have to fully qualify all table names.
Related
We are Trying to Drop and recreate a table via Informatica Mapping using the Pre_Sql option. Informatica throws an Insufficient privilege error even though we have granted privileges to the Informatica user, is it possible to drop and create a table Via pre SQL or is there any other method to accomplish this issue.
I am performing Mysql to bigquery data migration using jdbc to bigquery template in dataflow.
But while performing "select * from teable1" command on mysql, i also want to insert the selected data to another table in same database for some reason.
How can i perform both select and insert queries in dataflow template? I got error when used semicolon between two queries.
The Jdbc to Bigquery template will write all data you read to the table specified under "Bigquery output table" (<my-project>:<my-dataset>.<my-table>), so there is no need to write the insert statement.
(The parameter is "outputTable" for gcloud/REST)
As #PeterKim mentioned the JDBC to BigQuery termplate could be not the best approach for your use case.
You could try to use that template as reference and modify it to write into MySQL, in this post you will find an implementation about how to make an insert into MYSQL database.
After modifying the pipeline source code you can create a custom template.
I am trying to use pre aggregations over CLOUD SQL on Google Cloud Platform but the database is denying access and giving error Statement violates GTID consistency.
Any help is appreciated.
Cube.js done pre-aggregation by CREATE TABLE ... SELECT, but you are using MySQL on top of Google SQL with --enforce-gtid-consistency (has limitations).
Since only transactionally safe statements can be logged, there is a limitation to use CREATE TABLE ... SELECT (and some another SQL), because this statement is actually logged as two separate events.
There are two ways how to solve this issue:
1. Use pre-aggregations to an external database. (recommended way).
https://cube.dev/docs/pre-aggregations/#read-only-data-source-pre-aggregations
2. Use not documented flag loadPreAggregationWithoutMetaLock
Attention: This flag is an experimental and can be removed or changed in the feature..
Take a look at the source code
You can pass it directly in the driver constructor. This will produce two SQL statements to pass the limitation:
CREATE TABLE
INSERT INTO
Thanks
Is it possible to change SQL statement of the dataset via API call?
My Scenario: I have data in multiple tables in SQL Server. I have created a SQL query with joins to fetch the required data. I created a SQL server dataset by providing that query in the SQL Statement section and published it on the Power BI workspace. Now, I want to modify that SQL Statment programmatically.
I want to import this same .pbix file to create different datasets. The Idea is to use import date set api to import this dummy dataset and then programmatically change the db source and the SQL Statment, to customizes it for my different report need.
Any pointer or help is much appreciated.
For server name and database name, you can simply use parameters. Click the button to the left of the field to do this. You can make some changes in the query using parameters too, but this isn't very flexible. This can be done by defining text parameter and using it in in the M statement associated with the dataset’s Source step. For more information you may see this article:
https://www.red-gate.com/simple-talk/sql/bi/power-bi-introduction-working-with-parameters-in-power-bi-desktop-part-4/
Then you can use the Rest API to modify parameter values and refresh your datasets. You will need Update Parameters In Group and Refresh Dataset In Group API calls.
At of this writing, this is not supported by the Power BI REST API.
Possible workaround: Given you're using SQL Server, I'd suggest you create a VIEW in SQL Server with the statement you defined in your Power BI report, and change your report to point to that view instead.
Then, to modify the SQL statement, you just have to ALTER the view in the database.
I want to provide runtime values to the query in Select & Create table statements. What are the ways to parameterize Athena SQL queries?
I tried with PREPARE & EXECUTE statements from Presto however it is not working in Athena console. Do we need any external scripts like Python to call it?
PREPARE my_select1
FROM SELECT * from NATION;
EXECUTE my_select1 USING 1;
The SQL and HiveQL Reference documentation does not list PREPARE nor EXECUTE as available commands.
You would need to fully construct your SELECT statement before sending it to Amazon Athena.
You have to upgrade to Athena engine version 2 and now this seems to be supported as of 2021-03-12 but I can't find an official report:
https://docs.aws.amazon.com/athena/latest/ug/querying-with-prepared-statements.html
Athena does not support Parameterized queries. How ever you can create user-defined functions that you can call in the body of a query. Refer to this to know more about UDFs.