I want to provide runtime values to the query in Select & Create table statements. What are the ways to parameterize Athena SQL queries?
I tried with PREPARE & EXECUTE statements from Presto however it is not working in Athena console. Do we need any external scripts like Python to call it?
PREPARE my_select1
FROM SELECT * from NATION;
EXECUTE my_select1 USING 1;
The SQL and HiveQL Reference documentation does not list PREPARE nor EXECUTE as available commands.
You would need to fully construct your SELECT statement before sending it to Amazon Athena.
You have to upgrade to Athena engine version 2 and now this seems to be supported as of 2021-03-12 but I can't find an official report:
https://docs.aws.amazon.com/athena/latest/ug/querying-with-prepared-statements.html
Athena does not support Parameterized queries. How ever you can create user-defined functions that you can call in the body of a query. Refer to this to know more about UDFs.
Related
I have been working with AWS Athena for a while and need to do create a backup and version control of the views. I'm trying to build an automation for the backup to run daily and get all the views.
I tried to find a way to copy all the views created in Athena using boto3, but I couldn't find a way to do that. With Dbeaver I can see and export the views SQL script but from what I've seen only one at a time which not serve the goal.
I'm open for any way.
I try to find answer to my question in boto3 documentation and Dbeaver documentation. read thread on stack over flow and some google search did not took me so far.
Views and Tables are stored in the AWS Glue Data Catalog.
You can Query the AWS Glue Data Catalog - Amazon Athena to obtain information about tables, partitions, columns, etc.
However, if you want to obtain the DDL that was used to create the views, you will probably need to use SHOW CREATE TABLE [db_name.]table_name:
Analyzes an existing table named table_name to generate the query that created it.
Have you tried using get_query_results in boto3? get_query_results
I am creating a dataset in AWS Quicksight using custom SQL which I prepare/test in Athena. However, unless I define each join/table "databasename".table, the QS custom SQL fails. I have tried the below but it has failed. Is it possible to instruct the query to fun against a specific DB at the beginning of the query?
USING AwsDataCatalog."databasename"
In the data preparation, in the custom SQL page, on the left pane, you should be able to choose the database name (Schema).
If you do not set that, then it will use Athena's default schema so you have to fully qualify all table names.
I am trying to create an external function in Athena using AWS Lambda function. I am able to do so and query successfully using Athena query editor. Code is below.
using external function s3signedurl(col1 VARCHAR) returns varchar lambda 'customudf'
select incident, pdfloc, s3signedurl(pdfloc) as s3_signed_url
from "lambdapoc"."lambdametadata"
However I want to create a View on top of this so I can query it easily like a table/view (Select * from MYVIEW) from a Reporting Tool like Tableau OR any other tool which connects to Athena.
I don't seem to find the documentation for it or any examples on the Internet.
Please note: Column 's3_signed_url' data is dynamically generated from Lambda function and it will change everytime the query is executed.
Can anyone help?
I just got to know that "Using connectors or UDFs in Views are not supported". It is mentioned in their GitHub Repo.
Closing this question
Might be worth exploring if a Create Table As Select (CTAS) table would meet your needs. I believe it is compatible with UDFs
I am performing Mysql to bigquery data migration using jdbc to bigquery template in dataflow.
But while performing "select * from teable1" command on mysql, i also want to insert the selected data to another table in same database for some reason.
How can i perform both select and insert queries in dataflow template? I got error when used semicolon between two queries.
The Jdbc to Bigquery template will write all data you read to the table specified under "Bigquery output table" (<my-project>:<my-dataset>.<my-table>), so there is no need to write the insert statement.
(The parameter is "outputTable" for gcloud/REST)
As #PeterKim mentioned the JDBC to BigQuery termplate could be not the best approach for your use case.
You could try to use that template as reference and modify it to write into MySQL, in this post you will find an implementation about how to make an insert into MYSQL database.
After modifying the pipeline source code you can create a custom template.
I have a .sql file filled with Athena queries.
Is there a way I can tell Athena to run the sql queries saved in s3://my-bucket/path/to/queries.sql?
In MySQL can do something like this (based in SO answer), but curious if possible in Athena
mysql> source \home\user\Desktop\test.sql;
Is there a way I can tell Athena to run the sql queries saved in s3://my-bucket/path/to/queries.sql?
I think there is no direct way to tell Athena to run query stored in S3.
In MySQL can do something like this (based in SO answer), but curious if possible in Athena.
If you want to do it at all, then yes, you should be able to run the query using AWS CLI.
Your steps should be look like this.
Get the query from S3 using CLI and store in temp variable
Pass the query stored in a temp variable to Athena Query CLI
Hope this will help.