Now i following the Superset official document for connect Athena and apache-Superset..
I already made ./docker/requirements-local.txt and add a driver.. (echo "PyAthenaJDBC>1.0.9" >> ./docker/requirements-local.txt & echo "PyAthena>1.2.0" >> ./docker/requirements-local.txt )
But it was not working...
'awsathena+jdbc://{aws_access_key_id}:{aws_secret_access_key}#athena.{region_name}.amazonaws.com/{schema_name}?s3_staging_dir={s3_staging_dir}&...'
If you try to register Athena in the above url format, you will get an error like the picture above.
please tell me how...
I already made ./docker/requirements-local.txt and add a driver.. (echo "PyAthenaJDBC>1.0.9" >> ./docker/requirements-local.txt & echo "PyAthena>1.2.0" >> ./docker/requirements-local.txt )
But it was not working...
'awsathena+jdbc://{aws_access_key_id}:{aws_secret_access_key}#athena.{region_name}.amazonaws.com/{schema_name}?s3_staging_dir={s3_staging_dir}&...'
If you try to register Athena in the above url format, you will get an error like the picture above.
please tell me how...
The JDBC driver would require Java which doesn't exist on your EC2 yet.
Have you tried with the rest interface?
awsathena+rest://{aws_access_key_id}:{aws_secret_access_key}#athena.{region_name}.amazonaws.com/{schema_name}?s3_staging_dir={s3_staging_dir}&...
Link: https://docs.preset.io/docs/aws-athena
If the keys have special characters such as "/", then use URL encode e.g. "%2F" to escape it.
Note that you can use the IAM role on EC2, and skip the access key & secret key part.
Related
I am struggling to figure out how you update a secret in AWS Secrets Manager via Powershell when the secret has multiple key/values. My code is below with the standard change, but I have two key/values that I need to update. I tried a specific syntax with JSON, which did not take very well, and I overwrote the secure key with a simple text. I have attached my failed code as well, but I am stumped; I can not find references to the syntax for multiple key/values.
SINGLE SECRET(WORKS GREAT):
$sm_value = "my_secret_name"
Update-SECSecret -SecretId $sm_value -SecretString $Password
MULTI VALUE KEYS (FAILED):
Update-SECSecret -SecretId $sm_value -SecretString '{"Current":$newpassword,"Previous":$mycurrent}'
EXAMPLE AWS SECRET
Team, after looking at the documentation in Cloudformation and AWS CLI I concluded that my JSON string was incorrect. So after playing with it for an hour or so, I came up with this format that worked.
$mycurrent = "abc1234"
$newpassword = "zxy1234"
$json = #"
{"Current":"$newpassword","Previous":"$mycurrent"}
"#
Update-SECSecret -SecretId $sm_value -SecretString $json
It seems that updating secrets for multiple key/values requires a JSON file in order to work.
I have script written in bash where I create a key with a certain name.
#!/bin/bash
project_id="y"
secret_id="x"
secret_value="test"
gcloud config set project "$project_id"
gcloud secrets create "$secret_id" --replication-policy="automatic"
I want to be able to also directly add the secret-value to my secret, so that I do not have to go into my GCP account and set it manually (which would defeat the purpose). I have seen that it is possible to attach files through the following command, however there does not seem to be a similar command for a secret value.
--data-file="/path/to/file.txt"
From https://cloud.google.com/sdk/gcloud/reference/secrets/create#--data-file:
--data-file=PATH
File path from which to read secret data. Set this to "-" to read the secret data from stdin.
So set --data-file to - and pass the value over stdin. Note, if you use echo use -n to avoid adding a newline.
echo -n $secret_value | gcloud secrets create ... --data-file=-
I am sending comma-separated data to my kinesis stream, and I want my kinesis analytics app to recognize that there are two columns (both bigints). But when I populate my stream with some records and click "Discover Schema", it always gives me a schema of one column! Here's a screenshot:
I have tried many different delimiters to indicate columns, including comma, space, and comma-space, but none of these cause aws to detect my schema properly. At one point I gave up and edited the schema manually, which caused this error:
While I know that I have the option to keep the schema as a single column and use string and date-time manipulation to structure my data, I prefer not to do it this way... Any suggestions?
While I wasn't able to get the schema discovery tool to work, I realized that I am able to manually edit my schema and it works fine. I was getting that error because I had just populated the stream initially, and I was not continuously sending data.
Schema Discovery required me to send data to my input kinesis stream during the schema discovery. To do this for my Proof of Concept application I used the AWS CLI:
# emittokinesis.sh
JSON='{
"messageId": "31c14ee7-9bde-484d-af05-03509c2c33aa",
"myTest": "myValue"
}'
echo "$JSON"
JSONBASE64=$(echo ${JSON} | base64)
echo 'aws kinesis put-record --stream-name logstash-input-test --partition-key 1 --data "'${JSONBASE64}'"'
aws kinesis put-record --stream-name logstash-input-test --partition-key 1 --data "${JSONBASE64}"
I clicked the "Run Schema Discovery" button in the AWS UI and then quickly ran my shell script in a CMD window.
Once my initial schema was discovered I could manually edit the schema but it mostly matched what I expected based on my input JSON.
I am trying to access variable's value from remote location(file, etc.)
I read about input variables but still couldn't achieve what I am looking for.
My case is to get the value of the variable from some remote file.
I tried something like below
terraform apply -var-file="https://s3-ap-northeast-1.amazonaws.com/..."
but this give error saying no such file or directory.
Is there any way to load values on runtime from remote location ?
EDIT: I am using mysql provider to create database & users. For setting the user password, I want to use some remote location where my passwords are kept (maybe a s3 file).
PS : I saw that there is keybase available for passwords but I wanted to check if there are other ways for achievening this ?
It's hard to download and pass the file name to terraform apply, but you can do this with bash, This is how we do in remote server.
#!/bin/bash
varfile_name=terraformvar.tfvars
varfile=$(aws s3 cp s3://bucket-********/web-app/terraform/terraform_vars.tfvars ./$varfile_name)
if [ $? -eq 0 ]; then
terraform apply -var-file=$varfile_name
else
echo "failed to download file from s3"
fi
I am trying to connect Athena with Apache Zeppelin.I need to handle secret_key, Access_key, and Session_token. I am feeling hard to establish my connection with the Zeppelin JDBC interpreter.
I am following the steps as mentioned in this block,
If any one can help me out in establishing the connection with AWS Session token approach that would be helpful.
Thank You
The main docs for this are here:
https://docs.aws.amazon.com/athena/latest/ug/connect-with-jdbc.html
I found there are 2 driver versions, -1.1.0 and -1.0.1 . I could only get Zeppelin working with 1.1.0, and the links on that page don't go to that file, the only way to get it was using the aws s3 cp command
e.g.
aws s3 cp s3://athena-downloads/drivers/AthenaJDBC41-1.1.0.jar .
although I've given feedback on that page so it should be fixed soon.
Regarding the parameters, you use default.user and enter the Access_Key, default.password and enter the Secret_key. The default.driver should be com.amazonaws.athena.jdbc.AthenaDriver
The default.s3_staging_dir is actually the bucket where csv results are written so needs to match your athena settings.
There is no mention of where you might put a session token, however, you could always try putting it on the jdbc connection string ( which goes in default.url parameter value)
e.g.
jdbc:awsathena://athena.{REGION}.amazonaws.com:443?SessionToken=blahblahsomethingrealsessiontokengoeshere
but of course, replace {REGION} with the actual aws region and use your real session token.