I have several customer projects that write analytic events into a BigQuery dataset. The setup is organised like this:
1) Each GCP project has its own set of GCP resources and some of them report analytics using BigQuery insert API.
2) There's a single "Main Analytics" project that intakes all the data from the different projects in a standardised table (all projects write in the same data format).
I've created a custom IAM role in "Main Analytics" with the required permissions to execute a row insert operation:
bigquery.datasets.get
bigquery.tables.get
bigquery.tables.updateData
For every customer project I've created a unique service account with the above role. This allows each resource in any project to authenticate and insert rows (but not create/delete tables).
Problem: What I really want to do is limit the service accounts to write only to a specific dataset that intakes all the data. The above IAM role allows the service account to list all datasets/tables in the "Main Analytics" project and to insert into them.
If I use dataset permissions - add the service account email as a user to the dataset ACL - then it would have to be WRITER dataset role which would allow the service account to create & delete tables in the dataset which is too broad.
Combining the IAM role with the dataset permissions results in a union so the wider WRITER permission take effect over the narrower IAM role.
Anyway I can configure roles/permissions to allow each service account to insert and only-insert to a specific dataset?
You can drop the bigquery.datasets.get permission from the custom IAM role so that they can’t list all the datasets, and then in the dataset's permissions give the READER role instead of WRITER to the user for that specific dataset.
Related
I have 10 tables under my dataset. I need to create "BigQuery Metadata Viewer" permission but would like to neglect 2 tables under my dataset. So that BigQuery Metadata Viewer policy only will be able to access 8 tables.
I see that there is "condition" tab but could not figure out how to apply such a condition here.
IAM condition is a nice way to solve that issue, but it's not available for BigQuery resources.
The solution here is to have 2 datasets
One with the 8 tables and the permission to view the metadata
one with the 2 other tables without the permission to view the metadata.
You can use the GRANT statement using the role bigquery.metadataViewer or dataviewer.You can set this role to table level, the user will have permission to a specific table, and won’t see listed tables. In this case, you need to know the name tables.
Take a look to this example:
GRANT `roles/bigquery.metadataViewer`
ON TABLE `my_dataset._my_table`
TO "user:user#domain.com"
Additionally, you can set this role at dataset level, this will grant access to read and list all the tables from the dataset.
Here’s an example:
GRANT `roles/bigquery.metadataViewer`
ON schema `project_name.dataset_name`
TO "user:mail#mail.com"
Problem: I have a project in BigQuery where all my data is stored. Within this project I created multiple datasets containing different views. Now I want to use different service accounts to query the different datasets containing different views via grafana (if that matters). These users should only be able to query the views (and therefore a specific dataset) meant for them.
What I tried: I granted BigQuery User, Viewer or Editor permissions (I tried all of them) at a dataset level (and also BigQuery Meatadata Viewer at a project level). When I query a view, I receive the error:
User does not have bigquery.jobs.create permission in project xy.
Questions: It is not clear to me if granting bigquery.jobs.create permission on project level, will allow the user to query all datasets instead of only the one I want him to access to.
Is there any way to allow the user to create jobs only on a single dataset?
Update October 2021
I've just seen that this question did go unanswered for me back then but still gets a lot of views. I believe the possibilities changed a bit since I asked the question so here is how I'm handling it now:
I give the respective service account the role roles/bigquery.jobUser on project level. This allows it to create jobs in general, however since I don't give any other permissions yet it cannot query data yet.
Then I give the role roles/bigquery.dataViewer on the dataset level. That makes it possible for the service account to query only the dataset I granted the permission on.
It is also possible to grant roles/bigquery.dataViewer on table level, what will restrict access to only the specific table.
In case you want the service account not only to query (view) the data, but also to insert or change it for example, replace roles/bigquery.dataViewer with the role having the necessary permissions (or assign that role in addition).
How to grant the permissions:
On dataset level
On table or view level
We had a same problem, how we solved was, created a custom role and assigned the custom role to the particular dataset.
You can grant bigquery.user role to a specific dataset as indicated in this guide. The bigquery.user role contains the bigquery.jobs.create permission as well as other basic permissions related to querying datasets. You can check the full list of permissions for this role in this list.
As suggested above, you can also create custom roles having only the exact permissions you want by following this piece of documentation.
I am trying to restrict a bigquery so that users can only access a specific datasets, i did so without any issues, but why user is not able to create scheduled queries? it is saying to enable api and only project
owner can able to schedule queries , is there anyway to add permissions to create a custom role so that users can query,create and schedule queries ?
//dinesh
Ensure that the person creating the transfer has the following required permissions in BigQuery:
bigquery.transfers.update permissions to create the transfer
bigquery.datasets.update permissions on the target dataset
The bigquery.admin predefined Cloud IAM role includes bigquery.transfers.update and bigquery.datasets.update permissions. You should check the official documentation and the Cloud IAM roles in BigQuery to see the Predefined roles and permissions.
I'm trying to figure out if I can create multiple service accounts and for each service account create a different Policy (or even a generic policy).
In this policy I want to set the default retention for a dataset/table.
Only I (admin) can change the retention after table creation.
This is very important to control costs.
Did anyone managed to do this?
In Google Cloud Platform (GCP) it is possible to create different service accounts with distinct roles. These roles give access to specific resources across different services. In addition to the already existing roles in Bigquery, GCP allows to set service accounts with customized roles.
To control costs, the Project Admin or BigQuery Admin can establish a particular expiration date for a dataset and grant access to other service accounts with restricted permissions like BigQuery Job User or BigQuery Data Viewer, for example. This way, all the tables included in the dataset will have a default expiration date (set by the administrator) that all the other service accounts could not modify.
I have a IAM user with Role: BigQuery Data Editor
In my data set I did Share dataset added the user with Can Edit privileges.
However when I'm running my script which access BigQuery I get error 403
When I add to my IAM user the Role BigQuery User The script works.
The scripts runs only SELECT query from a table in this data set.
I don't understand why I must grant BigQuery User for this to work.
According to the documentation https://cloud.google.com/bigquery/docs/access-control
Rationale: The dataEditor role extends bigquery.dataViewer by issuing
create, update, delete privileges for the tables within the dataset
roles/bigquery.dataViewer has bigquery.tables.getData which get table data
What am I doing wrong here?
Having access to the data and being able to retrieve it with a query are different things and that's where the confusion is coming from.
Per the documentation, roles/bigquery.dataEditor has the following permissions:
Read the dataset's metadata and to list tables in the dataset.
Create, update, get, and delete the dataset's tables.
This means that the user with this role has access and manipulation rights to the dataset's information and the tables in it. An example would be that a user with this role can see all the table information by navigating to it through the GCP console (schema, details and preview tabs) but when trying to run a query there, the following message will appear:
Access Denied: Project <PROJECT-ID>: The user <USER> does not have bigquery.jobs.create permission in project <PROJECT-ID>.
Now let's check the roles/bigquery.user permissions:
Permissions to run jobs, including queries, within the project.
The key element here is that the BigQuery User role can run jobs and the BigQuery DataEditor can't. BigQuery Jobs are the objects that manage the BigQuery tasks, this includes running queries.
With this information, it's clearer in the roles comparison matrix that for what you are trying to accomplish you'll need the BigQuery DataEditor role (Get table data/metadata) and the BigQuery User role (Create jobs/queries).