Creating Amazon Quicksight datasets in code and using them through the console - amazon-web-services

I'm creating datasources/ datasets in code (boto3) but these don't show up in the console.
Even though the datasets are listed with list_data_sets, they don't seem to be available in the console.
I need to be able to create all the necessary datasets in code and then be able to use these to create new analyses/ dashboards in the console.
I'm using the Standard Edition of QuickSight.
Can this be done? Or, can it only be done in the Enterprise Edition? Or, not at all?
Thanks

According to the QuickSight pricing page "APIs" are not available in Standard Edition. Exactly what that means, I have no idea.
But, assuming it's possible to call create-data-set, one important thing to remember is that data set permissions are necessary in order for users to view them.
According to the boto docs, these permissions should be included in the following schema
Permissions=[
{
'Principal': 'string',
'Actions': [
'string',
]
},
]
In my code, I use the following to share with the all-users group (note the group principal, replace AWS_REGION and ACCOUNT_ID with your values)
Permissions= [
{
'Principal': 'arn:aws:quicksight:AWS_REGION:ACCOUNT_ID:group/default/all-users',
'Actions': [
'quicksight:DescribeDataSet',
'quicksight:DescribeDataSetPermissions',
'quicksight:PassDataSet',
'quicksight:DescribeIngestion',
'quicksight:ListIngestions'
]
}
],
I believe the same can be done for individual users, with an ARN resource of user/default/user.name instead of group/default/all-users.
For data sources, the set of permissions that I use is
'Actions': [
'quicksight:DescribeDataSource',
'quicksight:DescribeDataSourcePermissions',
'quicksight:UpdateDataSource',
'quicksight:UpdateDataSourcePermissions',
'quicksight:DeleteDataSource',
'quicksight:PassDataSource'
]

Related

AWS S3 Bucket Policy for CORS

I am trying to figure out how to follow these instructions to set up an S3 bucket on AWS. I have been trying most of the year but still can't make sense of the AWS documentation. I think this github repo readme may have been prepared at a time when the AWS S3 interface appeared differently (there is no CORS setting in the form to make the S3 bucket permissions now).
I asked this question earlier this year, and I have tried using the upvoted answer to make a bucket policy, which is precisely as shown in that answer, as:
[
{
"AllowedHeaders": [
"*"
],
"AllowedMethods": [
"POST",
"GET",
"PUT",
"DELETE",
"HEAD"
],
"AllowedOrigins": [
"*"
],
"ExposeHeaders": []
}
]
When I try this, I get an error that says:
Ln 1, Col 0Data Type Mismatch: The text does not match the expected
JSON data type Object. Learn more
The Learn More link goes to a page that suggests I can resolve this error by updating text to use the supported data type. I do not know what that means in this context. I don't know how to find out which condition keys require which type of data.
Resolving the error
Update the text to use the supported data type.
For example, the Version global condition key requires a String data
type. If you provide a date or an integer, the data type won't match.
Can anyone help with current instructions for how to create the CORS permissions that are consistent with the spirit of the instructions in the github readme for the repo?

AWS quicksight can't ingest csv from s3 but the same data uploaded as file works

I am new to quicksight and was just test driving (on the quicksight web console. I'm not using the command line in this entire thing) with some data (can't share, confidential business info). I have a strange issue. when I create a dataset by uploading the file, which is only 50 mb, it works fine and I can see a preview of the table and I am able to proceed to the visualization. But when I upload the same file to the s3 and make a manifest and submit it using the 'use s3' option in the creat dataset window, I get the INCORRECT_FIELD_COUNT error.
here's the manifest file:
{
"fileLocations": [
{
"URIs": [
"s3://testbucket/analytics/mydata.csv"
]
},
{
"URIPrefixes": [
"s3://testbucket/analytics/"
]
}
],
"globalUploadSettings": {
"format": "CSV",
"delimiter": ",",
"containsHeader": "true"
}
}
I know the data is not fully structured with some rows where a few columns are missing but how is it possible for quicksight to automatically infer and put NULLs into shorter rows when uploaded from local machine but not as an s3 file with the manifest? are there some different setttings that i'm missing?
I'm getting the same thing - looks like this is fairly new code. It'd be useful to know what the expected field count is, especially as it doesn't say if it's too few or too many (both are wrong). One of those technologies that looks promising, but I'd say there's a little maturing required.

Adding row level permission tag configuration to a dataset

I am trying to embed an AWS QuickSight dashboard for anonymous access. For that dataset used in the dashboard must have tags that specify row level security. From what I see the only way to do this is via update-data-set cli command (or related API request). But this is insane - for this command to work I have to specify additional parameters like dataset name or even physical table map. But I have no intention to modify those, I just need to add RLS tags. Is there a straightforward way to add RLS tags to an existing dataset?
I ended up generating skeleton JSON for update-data-set via --generate-cli-skeleton parameter, then filling it with data from describe-data-set command, and adding block
"RowLevelPermissionTagConfiguration": {
"Status": "ENABLED",
"TagRules": [
{
"TagKey": "my_tag",
"ColumnName": "my_column"
}
]
}
and supplying this resulting JSON file via update-data-set --cli-input-json file://thatfile.json
Cumbersome, but it worked.

Is there a way to create Quicksight analysis purely through code (boto3)?

What I currently have in my Quicksight account is a Data Source (Redshift), some datasets (some Redshift views) and an analysis (graphs and charts that use the datasets). I can view all of these on the AWS Quicksight Console. But when I use boto3 to create a data source and datasets, nothing shows up on the console. They do however show up when I use the list_data_sources and list_data_sets calls.
After this, I need to create all the graphs by code that I created manually. I can't currently find an option to do this through code. There is a 'create_template' api call which is supposed to create a template through an existing Quicksight analysis. But it requires the ARN of the analysis which I can't find.
Any suggestions on what to do?
Note: this only answers why the data sets/sources do not appear in the console. As for the other question, I assume mjgpy3 was of some help.
Summary
Add the permissions at the bottom of this post to your data set and data source in order for them to appear in the console. Make sure to fill in the principal arn with your details.
Details
In order for data sets and data sources to appear in the console when created via the API, you must ensure that the correct permissions have been added to them. Without adding the correct permissions, it is true that the CLI lists them whereas the console does not.
If you have created data sets/sources via the console, you can use the CLI (aws quicksight describe-data-set-permissions and aws quicksight describe-data-source-permissions) to view what permissions AWS gives them so that your account can interact with them.
I've tested this and these are what AWS assigns them as of 25/03/2020.
Data Set permissions:
"permissions": [
{
"Principal": "arn:aws:quicksight:<region>:<aws_account_id>:user/default/{IAM user name}",
"Actions": [
"quicksight:UpdateDataSetPermissions",
"quicksight:DescribeDataSet",
"quicksight:DescribeDataSetPermissions",
"quicksight:PassDataSet",
"quicksight:DescribeIngestion",
"quicksight:ListIngestions",
"quicksight:UpdateDataSet",
"quicksight:DeleteDataSet",
"quicksight:CreateIngestion",
"quicksight:CancelIngestion"
]
}
]
Data Source permissions:
"permissions": [
{
"Principal": "arn:aws:quicksight:<region>:<aws_account_id>:user/default/{IAM user name}",
"Actions": [
"quicksight:UpdateDataSourcePermissions",
"quicksight:DescribeDataSource",
"quicksight:DescribeDataSourcePermissions",
"quicksight:PassDataSource",
"quicksight:UpdateDataSource",
"quicksight:DeleteDataSource"
]
}
]
It sounds like your smaller question is regarding the ARN of the analysis.
The format of analysis ARNs is
arn:aws:quicksight:$AWS_REGION:$AWS_ACCOUNT_ID:analysis/$ANALYSIS_ID
Where
$AWS_REGION is replaced with the region in which the analysis lives
$AWS_ACCOUNT_ID is replaced with your AWS account ID
$ANALYSIS_ID is replaced with the analysis ID
If you're looking for the $ANALYSIS_ID it's the GUID-looking thing on the end of the URL for the analysis in the QuickSight URL
So, if you were on an analysis at the URL
https://quicksight.aws.amazon.com/sn/analyses/018ef6393-2c71-4842-9798-1aa2f0902804
the analysis ID would be 018ef6393-2c71-4842-9798-1aa2f0902804 (this is a fake ID I injected for this example).
Your larger question seems to be whether you can use the create_template API to duplicate your analysis. The answer at this moment (12/16/19) is, unfortunately, no.
You can use the create_dashboard API to publish a Dashboard from a template made with create_template but you can't create an Analysis from a template.
I'm answering this bit just to clarify since you may actually be okay with creating a dashboard (basically the published version of an analysis) rather than another analysis.
There are multiple ways you can find analysis id associated. Use any of the following.
A dashboard url has dashboard id included, Use this ID to execute API call describe-dashboard and you would see analysis ARN in the source entity.
Click on "save as" option on the dashboard and it would take you to the associated analysis. [ One might not see this option if a dashboard is created from a template ]
A dashboard ID can also be found by using list_dashboards API call. Print all the dashboard ID and name. You can match the ID with the given dashboard name.Look at the whole list because a dashboard id is unique but the dashboard name is not. One can have multiple dashboards with the same name.
Yes you can create lambda and trigger using cron Job
import boto3
quicksight = boto3.client('quicksight')
response = quicksight.create_ingestion(AwsAccountId=XXXXXXX,
DataSetId=YYYY,IngestionId=ZZZZ)
https://aws.amazon.com/blogs/big-data/automate-dataset-monitoring-in-amazon-quicksight/
https://aws.amazon.com/blogs/big-data/event-driven-refresh-of-spice-datasets-in-amazon-quicksight/
I've been playing with this as well and ran into the same issue. Make sure that your permissions are set up properly for the data source and the data set by referencing the quicksight user as follows:
arn:aws:quicksight:{region}:xxxxxxxxxx:user/default/{user}
I would include all the quicksight permissions found in the docs to start with and shave down from there. If nothing else, create the data source/set from the console, and then use the describe-* CLI call to see what they use.
It's kind of wonky.

Create an AWS Resource Group with Terraform

I am currently getting into Terraform and I am trying to structure the different resources that I am deploying by using tags and resource groups.
https://docs.aws.amazon.com/cli/latest/reference/resource-groups/index.html
I can easily add tags with Terraform and I can create the resource-group via aws cli but I really want to be able to do both with Terraform if possible.
The official Terraform docs currently seem to not support an aws_resource_group resource(I was able to find aws_inspector_resource_group and aws_iam_resource_group, which are different types of grouping resources) but I was wondering if anyone was able to achieve it via some kind of a workaround.
I would really appreciate any feedback on the matter.
Thanks in advance!
This has been released in aws provider 1.55.0: https://www.terraform.io/docs/providers/aws/r/resourcegroups_group.html
For anyone looking for a code example, try this:
resource "aws_resourcegroups_group" "code-resource" {
name = "code-resource"
resource_query {
query = <<JSON
{
"ResourceTypeFilters": [
"AWS::EC2::Instance"
],
"TagFilters": [
{
"Key": "Stage",
"Values": ["dev"]
}
]
}
JSON
}
}
Please update it to your liking and needs. also be sure to checkout the source documentation:
https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/resourcegroups_group