BigQuery: Separate project for storing and project for querying - google-cloud-platform

Is it possible to create a project in BigQuery to store data and another to query the data ? If yes, what rights should be given to the project querying the data to access the data stored by the other project ?
The idea would be to have a better control of costs.

Yes you can do that!
You have to give the roles/bigquery.dataViewer role to the user that will be querying the data (at least). What that account will be depends on the use-case. If you are going to query from BigQuery UI you have to give such permissions to the mail account with which you will log in GCP UI, but you can also give such permissions to particular users or service-accounts for programatic access too.
Here you have the documentation referring to BQ permissions and how to grant them.

Related

In GCP IAM what's the difference between "Viewer" and "BigQuery Data Viewer"?

I am trying to grant access to a table and cannot tell the difference between the "Viewer" and "BigQuery Data Viewer" roles. I do not want to give permissions to view other tables or datasets within the GCP Project or full access to BigQuery. Can someone tell me the difference between these two if the access is given through the Share option at the table level?
Screenshot of where Role is granted
At the moment I've assigned the user both roles and am not sure if I'm granting too much access. I want to follow the Principle of Least Privilege.
Viewer or roles/viewer is a top level, project level permission. Permissions for read-only actions that do not affect state, such as viewing (but not modifying) existing resources or data.
BigQuery Data Viewer its a resource based permission. Can be applied to BigQuery itself, or even to individual tables or views. Read all about its usage.
it has the following BQ roles:
bigquery.datasets.get
bigquery.datasets.getIamPolicy
bigquery.models.export
bigquery.models.getData
bigquery.models.getMetadata
bigquery.models.list
bigquery.routines.get
bigquery.routines.list
bigquery.tables.createSnapshot
bigquery.tables.export
bigquery.tables.get
bigquery.tables.getData
bigquery.tables.getIamPolicy
bigquery.tables.list
resourcemanager.projects.get
resourcemanager.projects.list

Google BigQuery: grant service account permissions to create jobs in only some specific datasets

Problem: I have a project in BigQuery where all my data is stored. Within this project I created multiple datasets containing different views. Now I want to use different service accounts to query the different datasets containing different views via grafana (if that matters). These users should only be able to query the views (and therefore a specific dataset) meant for them.
What I tried: I granted BigQuery User, Viewer or Editor permissions (I tried all of them) at a dataset level (and also BigQuery Meatadata Viewer at a project level). When I query a view, I receive the error:
User does not have bigquery.jobs.create permission in project xy.
Questions: It is not clear to me if granting bigquery.jobs.create permission on project level, will allow the user to query all datasets instead of only the one I want him to access to.
Is there any way to allow the user to create jobs only on a single dataset?
Update October 2021
I've just seen that this question did go unanswered for me back then but still gets a lot of views. I believe the possibilities changed a bit since I asked the question so here is how I'm handling it now:
I give the respective service account the role roles/bigquery.jobUser on project level. This allows it to create jobs in general, however since I don't give any other permissions yet it cannot query data yet.
Then I give the role roles/bigquery.dataViewer on the dataset level. That makes it possible for the service account to query only the dataset I granted the permission on.
It is also possible to grant roles/bigquery.dataViewer on table level, what will restrict access to only the specific table.
In case you want the service account not only to query (view) the data, but also to insert or change it for example, replace roles/bigquery.dataViewer with the role having the necessary permissions (or assign that role in addition).
How to grant the permissions:
On dataset level
On table or view level
We had a same problem, how we solved was, created a custom role and assigned the custom role to the particular dataset.
You can grant bigquery.user role to a specific dataset as indicated in this guide. The bigquery.user role contains the bigquery.jobs.create permission as well as other basic permissions related to querying datasets. You can check the full list of permissions for this role in this list.
As suggested above, you can also create custom roles having only the exact permissions you want by following this piece of documentation.

unable to create schedule queries in bigquery

I am trying to restrict a bigquery so that users can only access a specific datasets, i did so without any issues, but why user is not able to create scheduled queries? it is saying to enable api and only project
owner can able to schedule queries , is there anyway to add permissions to create a custom role so that users can query,create and schedule queries ?
//dinesh
Ensure that the person creating the transfer has the following required permissions in BigQuery:
bigquery.transfers.update permissions to create the transfer
bigquery.datasets.update permissions on the target dataset
The bigquery.admin predefined Cloud IAM role includes bigquery.transfers.update and bigquery.datasets.update permissions. You should check the official documentation and the Cloud IAM roles in BigQuery to see the Predefined roles and permissions.

BigQuery Dataset/Table expiration by Policy

I'm trying to figure out if I can create multiple service accounts and for each service account create a different Policy (or even a generic policy).
In this policy I want to set the default retention for a dataset/table.
Only I (admin) can change the retention after table creation.
This is very important to control costs.
Did anyone managed to do this?
In Google Cloud Platform (GCP) it is possible to create different service accounts with distinct roles. These roles give access to specific resources across different services. In addition to the already existing roles in Bigquery, GCP allows to set service accounts with customized roles.
To control costs, the Project Admin or BigQuery Admin can establish a particular expiration date for a dataset and grant access to other service accounts with restricted permissions like BigQuery Job User or BigQuery Data Viewer, for example. This way, all the tables included in the dataset will have a default expiration date (set by the administrator) that all the other service accounts could not modify.

Restrict access to users from selecting a database

How do I use athena workgroups to restrict access of a user to a particular database?
For e.g. I have a user "readonly" who should not be able to run select query on default database. Is this possible?
The way to restrict users from querying tables is to use IAM permissions. The permissions model in Athena is unfortunately more complicated than in an isolated data warehouse or RDBMS, since Athena is a part of a larger ecosystem that also includes S3 and Glue.
There is no specific permission for running SELECT. You can restrict users to run queries by controlling whether or not they are allowed to perform the athena:StartQueryExecution action, but you can't control what kind of queries they run.
Instead you need to think in terms of access to data, and access to the catalog.
To restrict reading you restrict the user's access to the data on S3. Even if a user is allowed to run a SELECT query they will get an error if they don't have permission to run s3:ListObject and s3:GetObject on the objects in the table's prefix.
You can also restrict a user's access to the catalog objects, i.e. the databases and tables – but that does not restrict their access to the data itself, think of it more as a restriction on creating, updating, and dropping databases and tables. Even if there is a way to restrict which databases and tables a user can see in the catalog, if they have permission to read the data they can read the data directly from S3 and skipping Athena.
You can find the documentation on how to control access to catalog objects here: https://docs.aws.amazon.com/athena/latest/ug/fine-grained-access-to-glue-resources.html
Workgroups in Athena can't be used to control access to data, nor to the catalog.