I am new to Google Cloud. I am trying to access google buckets to upload files. I use Google Storage object for accessing the bucket programmatically in Python. I am able to authenticate the storage object with 'key.json'. But I am unsure when the application will run in cloud how will it access 'key.json' file securely ? Also is there a way to authenticate storage object using access token in python ?
Thanks in advance!
But I am unsure when the application will run in cloud how will it
access 'key.json' file securely ?
Review the details that I wrote below. Once you have selected your environment you might not need to use a service account JSON file at all because the metadata server is available to provide your code with credentials. This is the best case and secure. On my personal website, I have written many articles that show how to create, manage and store Google credentials and secrets.
Also is there a way to authenticate storage object using access token
in python ?
All access is via an OAuth Access Token. The following link shows details using the metadata server which I cover in more detail below.
Authenticating applications directly with access tokens
There are three items to consider:
My code is not running in Google Cloud
My code is running in Google Cloud on a "compute" type of service with access to the metadata server
My code is running in Google Cloud without access to the metadata server.
1) My code is not running in Google Cloud
This means your code is running on your desktop or even in another cloud such as AWS. You are responsible for providing the method of authorization. There are two primary methods: 1) Service Account JSON key file; 2) Google OAuth User Authorization.
Service Account JSON key file
This is what you are using now with key.json. The credentials are stored in the file and are used to generate an OAuth Access Token. You must protect that file as it contains your Google Cloud secrets. You can specify the key.json directly in your code or via the environment variable GOOGLE_APPLICATION_CREDENTIALS
Google OAuth User Authorization
This method requires the user to log in to Google Accounts requesting an OAuth scope for Cloud Storage. The end result is an OAuth Access Token (just like a Service Account) that authorizes access to Cloud Storage.
Getting Started with Authentication
2) My code is running in Google Cloud on a "compute" type of service with access to the metadata server
Notice the word "metadata" server. For Google Cloud compute services, Google provides a metadata server that provides applications running on that compute service (Compute Engine, Cloud Functions, Cloud Run, etc) with credentials. If you use Google SDK Client libraries for your code, the libraries will automatically select the credentials for you. The metadata server can be disabled (denied access through role/scope removal), so you need to evaluate what you are running on.
Storing and retrieving instance metadata
3) My code is running in Google Cloud without access to the metadata server.
This is a similar scenario to #1. However, now you are limited to only using a service account unless this is a web server type of service that can present the Google Accounts authorization service to the user.
Related
Does anyone have an idea, how I can access the email address of the service account, which is running my cloud run service, at runtime?
When deploying the service to gcloud, I use a specific service account for running the service.
During runtime I need the email/ID of this service account, in order to do blob signing using IAMCredentialsService.
Is there a possibility to get the service account ID somehow? The ComputeCredential object I have at hand doesn't provide this information. Right now I have to set an environment variable which contains the service account email address, which I can use at runtime within the service.
In your cloud run container, you need to reach this URL (a GET)
http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/email
With this header
Metadata-Flavor: Google
If you have difficulty getting the value, provide your language and I will see if I can provide a code sample for you.
See more in documentation https://cloud.google.com/run/docs/reference/container-contract#metadata-server
Container instance metadata server
Cloud Run container instances expose a metadata server that you can use to retrieve details about your container instance, such as the project ID, region, instance ID or service accounts. It can also be used to generate tokens for the runtime service account.
You can access this data from the metadata server using simple HTTP requests to the http://metadata.google.internal/ endpoint with the Metadata-Flavor: Google header: no client libraries are required. For more information, see Getting metadata.
I know this question is probably a bit vague. I was trying to run one of the examples of Google NLP Library in Google Shell.
I have 0 experience with using API, JSON, Nodejs... I don't understand what they are and how to use them.
Please help
Here is the snapshot of the error:
The error message means that you are using user credentials instead of service account credentials.
When you connect to Google Cloud Shell, you are using your Google Accounts User Credentials. Those credentials are the ones that you used to log in to the Google Cloud Console. When you run an application in Google Cloud Shell, your application is using those credentials unless you explicitly specify different credentials.
The solution is to create a service account in the Google Cloud Console. Then in your program use the service account for credentials for your application.
Google Cloud Service Accounts
When you do not specify the application credentials, the Google Client libraries use a method to locate credentials called ADC (Application Default Credentials). I wrote an article that might help you understand ADC:
Google Cloud Application Default Credentials
The simplest method for you is to create the environment variable GOOGLE_APPLICATION_CREDENTIALS to point to the service account full path location before running your application. Change the path below to point to where the service account is stored on Cloud Shell. You will need to first create the service acount, download it and then upload to Cloud Shell.
export GOOGLE_APPLICATION_CREDENTIALS="$HOME/service-account.json"
Managing files with Cloud Shell
This link will provide more information on how to write applications that use service accounts.
Setting Up Authentication for Server to Server Production Applications
I'm deploying a Cloud Function which interacts with Google Drive. As far as I can tell, there's no way to use the implicit Cloud Function credentials or related environment variables to authorize with Drive, so I've had to resort to either including the credentials.json service account key file in the Cloud Function bundle or by setting an environment variable using that same key file during deployment of the Cloud Function.
It'd seem to me that you would be able to use the Cloud Function's implicit service account and credentials in order to achieve this.
You can specify a identity service account on Cloud Function. By the way, instead to use the default compute service account, you can use this one that you want.
Then you can share a document, or a shared drive with the service account email. By the way, the service account of the function will have access to this drive element.
However, it's not enough. When you will perform the request to the Google Drive API, you have to provide an authentication token in the header of the request. You can find here how to request the Cloud Function metadata server to get this token and then reuse it to your Google Drive API call.
In order to limit the number of service accounts to manage as well as handling their keys, I'm exploring other ways of accessing GCP resources from a developer laptop or desktop so I can run ad-hoc scripts or interactive programs (e.g. Jupyter notebook) that access GCP services.
Using gcloud auth application-default login generates, after authenticating via a web browser, a refresh token that can be used to get and renew access tokens that can be used to interact with GCP services.
The workflow I'm following is this:
Run gcloud auth application-default login. This generates a JSON file on my disk that
contains the refresh token.
Export the JSON file location as GOOGLE_APPLICATION_CREDENTIALS env variable
GOOGLE_APPLICATION_CREDENTIALS=/Users/my.username/.config/gcloud/application_default_credentials.json
Use that file to authenticate via Google auth library and interact with different GCP services.
This is convenient, as it reduces the need to circulate, secure and, if needed, share service account key files around team members. However, I have noticed that the refresh token provided does not expire and is still valid.
Unless I'm missing something here, this makes application_default_credentials.json file as sensitive as a service account key. If it gets lost or compromised it can be used to get access tokens without the need to re-authenticate, which is fairly insecure, IMO.
We're aware of the GCP security best practices recommend using service account (and their keys) for service-to-service workloads. This scenario I'm describing is for ad-hoc, development/testing of code from
a developer's or engineer's laptop. We think that forcing users to interactively authenticate via the web to get new tokens every few hours would be more secure and convenient than using long-lived service account keys stored in the hard drive.
I have read through [1] but I could not find a definitive answer.
Does anyone know if there is an expiration for these refresh tokens?
Is there a way of controlling and limiting their lifetimes (ideally to hours or minutes)?
What is the best/common practice for this scenario? Using a single service account (and key) per individual user?
[1] https://developers.google.com/identity/protocols/OAuth2#expiration
Note: User Credentials have Refresh Tokens too.
Does anyone know if there is an expiration for these refresh tokens?
Google OAuth Refresh Tokens do not expire. They can be revoked.
Is there a way of controlling and limiting their lifetimes (ideally to
hours or minutes)?
You could periodically revoke the Refresh Token which will invalidate the Access and Client ID tokens. This means that you are handling the Refresh Tokens which adds another security issue to manage.
What is the best/common practice for this scenario? Using a single
service account (and key) per individual user?
If you use User Credentials (the method where you log in to Google) you will receive SDK warnings and if you make a lot of API calls, you will become blocked. Google does not want you to use User credentials in place of Service Account credentials. The verification process for User Credentials requires more effort on Google's backend systems. User Credentials are assumed to be created in an insecure environment (web browsers) whereas Service Account credentials are assumed to be in a secure environment.
Best practices are to issue service account JSON key files to an individual application with only the required permissions for that application to operate. For example, if you create a tool that only needs to read Cloud Storage objects, create a service account with only read permissions. Periodically the service account keys should be rotated and new keys downloaded and old keys deleted. Each application should have its own service account JSON key file. I wrote an article on how to securely store JSON key files on Cloud Storage. This helps with rotating keys as your application just downloads the latest key when needed. (link). My article discusses Google Cloud Run, but the same principles apply.
I wish to use the Google cloud IAM ( identity access management) system for a new Google App Engine project. (Although it's not necessary to know, the front-end will be an angular JS, and the backend in Java.) However, once the user logs into my app using his or her browser and is then authenticated via Google Cloud IAM, I need to know whether it's possible to pass this " authenticated credential" to a Google compute VM. If so, how? The reason why need to pass this "authenticated credential" is that I wish to use the gsutil ( or similar) functionality on a Google compute VM and I want to use the same username to ensure that the security profile carries through properly. (Specifically, I intend to use gsutil to communicate with Google cloud storage, but I intend to do this from a Windows Server compute engine VM.)
I've been reading on the Google computer VM and Google cloud IAM, and they all talk about being able to pass the "service account" token, but there is no reference to how to pass a "authenticated user" credential so that the gsutil command that can access Google cloud storage on the Windows VM could use this authenticated user. (I want to avoid making the user authenticate both for my application as well as for the gsutil program running within the compute engine Windows VM.)
Is this possible? If not, any suggestions/workarounds?
One idea I had, though ugly, is as follows: every time a Windows compute engine VM is requested, we would dynamically create a new Google service account which had the same permissions as the logged in IAM-authenticated user. Then, we would uses Google service account within the Windows compute VM to contact Google cloud storage. This solves the problem of ensuring that the same privileges are communicated, though it creates a slightly different problem in that all the logs that are generated for access to the file will be using this dummy service account instead of the real users name.