I have created a Google Cloud function that can be invoked through HTTP. The access to the function is limited to the Service account only.
If I had a Django View which should invoke this function and expect a response?
Here is what I have tried
1) Before starting Django I set the environment variable
export GOOGLE_APPLICATION_CREDENTIALS
2) I tried invoking the function using a standalone code, but soon realised this was going nowhere, because I could not figure out the next step after this.
from google.oauth2 import service_account
from apiclient.http import call
SCOPES = ['https://www.googleapis.com/auth/cloud-platform']
SERVICE_ACCOUNT_FILE = 'credentials/credentials.json'
credentials = service_account.Credentials.from_service_account_file(
SERVICE_ACCOUNT_FILE, scopes=SCOPES)
Google's documentation does give you documentation around the API, but there is no sample code on how to invoke the methods or what to import within your Python code and what are the ways to invoke those methods.
How do you send a POST request with JSON data to an Cloud Function, with authorization through a service account?
**Edit
A couple hours later I did some more digging and figured this out partially
from google.oauth2 import service_account
import googleapiclient.discovery
import json
SCOPES = ['https://www.googleapis.com/auth/cloud-platform']
SERVICE_ACCOUNT_FILE = 'credentials/credentials.json'
credentials = service_account.Credentials.from_service_account_file(
SERVICE_ACCOUNT_FILE, scopes=SCOPES)
cloudfunction = googleapiclient.discovery.build('cloudfunctions', 'v1', credentials=credentials)
#projects/{project_id}/locations/{location_id}/functions/{function_id}.
path='some project path'
data='some data in json that works when invoked through the console'
data=json.dumps(data)
a=cloudfunction.projects().locations().functions().call(name=path, body=data).execute()
I get another error now.
Details: "[{'#type': 'type.googleapis.com/google.rpc.BadRequest', 'fieldViolations': [{'description': 'Invalid JSON payload received. Unknown name "": Root element must be a message.'}]}]">
I cant find any documentation on this. How should the JSON be formed?
making the json like {"message":{my actual payload}} doesn't work.
The requested documentation can be found here.
The request body argument should be an object with the following form:
{ # Request for the `CallFunction` method.
"data": "A String", # Input to be passed to the function.
}
The following modification on your code should work correctly:
from google.oauth2 import service_account
import googleapiclient.discovery
SCOPES = ['https://www.googleapis.com/auth/cloud-platform']
SERVICE_ACCOUNT_FILE = 'credentials/credentials.json'
credentials = service_account.Credentials.from_service_account_file(
SERVICE_ACCOUNT_FILE, scopes=SCOPES)
cloudfunction = googleapiclient.discovery.build('cloudfunctions', 'v1', credentials=credentials)
path ="projects/your-project-name/locations/cloud-function-location/functions/name-of-cloud-function"
data = {"data": "A String"}
a=cloudfunction.projects().locations().functions().call(name=path, body=data).execute()
Notice that very limited traffic is allowed since there are limits to the API calls.
Related
first question ever on StackOverflow.
I am trying to write a Cloud Function on gcp to login to vault via hvac.
https://hvac.readthedocs.io/en/stable/usage/auth_methods/gcp.html#login
It says here that a path to a SA json but I am writing this on Cloud Function.
Does anyone have an example on how to do this properly? The default cloud identity SA associated with the function has permission already to the vault address.
Thanks
In Cloud Functions you don't need the path to the Service Account key because the Cloud Identity SA is already loaded as the Application Default Credentials (ADC).
The code from the link you share it's okay for environments where you don't have configured the ADC or simply you prefer to use another account.
For Functions, the code can be simpler:
import time
import json
import googleapiclient.discovery
import google.auth
import hvac
credentials, project = google.auth.default(scopes=["https://www.googleapis.com/auth/cloud-platform"])
now = int(time.time())
expires = now + 900
payload = {
'iat': now,
'exp': expires,
'sub': credentials.service_account_email,
'aud': 'vault/my-role'
}
body = {'payload': json.dumps(payload)}
name = f'projects/{project}/serviceAccounts/{credentials.service_account_email}'
iam = googleapiclient.discovery.build('iam', 'v1', credentials=credentials)
request = iam.projects().serviceAccounts().signJwt(name=name, body=body)
resp = request.execute()
jwt = resp['signedJwt']
client.auth.gcp.login(
role='my-role',
jwt=jwt,
)
I have read extensively on how to access GCP Gmail API using service account and have given it domain-wide authority, using the instruction here:
https://support.google.com/a/answer/162106
Here is my service account:
Here is the scopes added to the domain-wide authority. You can see that the ID matches the service account.
One thing I notice is that my GCP project is an internal project, I havent' published it or anything, yet when I added the scope, it is not showing the service account email name but the project name. Does it make any difference? Do I need to set anything here? In the OAuth Consent Screen, I see the name of the project is being defined there. I have added all same scope on this screen too, not sure if it make any difference.
Here is my code:
from google.oauth2 import service_account
from googleapiclient import discovery
credentials_file = get_credentials('gmail.json')
scopes = ['https://www.googleapis.com/auth/gmail.readonly', 'https://www.googleapis.com/auth/gmail.labels', 'https://www.googleapis.com/auth/gmail.modify']
credentials = service_account.Credentials.from_service_account_info(credentials_file, scopes=scopes)
delegated_credentials = credentials.with_subject("abc#mydomain.com")
GMAIL_SERVICE = discovery.build('gmail', 'v1', credentials=delegated_credentials)
labels = GMAIL_SERVICE.users().labels().list(userId='me').execute()
Error message:
Google.auth.exceptions.RefreshError: ('unauthorized_client: Client is
unauthorized to retrieve access tokens using this method, or client
not authorized for any of the scopes requested.', {'error':
'unauthorized_client', 'error_description': 'Client is unauthorized to
retrieve access tokens using this method, or client not authorized for
any of the scopes requested.'})
Not sure I can answer precisely on the original question (I think not), but here how things are done in cloud functions developed by me. The following particular code snippet is written/adopted for this answer, and it was not tested:
import os
import google.auth
import google.auth.iam
from google.oauth2 import service_account
from google.auth.exceptions import MutualTLSChannelError
from google.auth.transport import requests
import googleapiclient.discovery
from google.cloud import error_reporting
GMAIL_SERV_ACCOUNT = "A service account which makes the API CALL"
OAUTH_TOKEN_URI = "https://accounts.google.com/o/oauth2/token"
GMAIL_SCOPES_LIST = ["https://mail.google.com/"] # for example
GMAIL_USER = "User's email address, who's email we would like to access. abc#mydomain.com - from your question"
# inside the cloud function code:
local_credentials, project_id = google.auth.default()
local_credentials.refresh(requests.Request())
signer = google.auth.iam.Signer(requests.Request(), local_credentials, GMAIL_SERV_ACCOUNT)
delegate_credentials = service_account.Credentials(
signer, GMAIL_SERV_ACCOUNT, OAUTH_TOKEN_URI, scopes=GMAIL_SCOPES_LIST, subject=GMAIL_USER)
delegate_credentials.refresh(requests.Request())
try:
email_api_service = googleapiclient.discovery.build(
'gmail', 'v1', credentials=delegate_credentials, cache_discovery=False)
except MutualTLSChannelError as err:
# handle it somehow, for example (stupid, artificial)
ER = error_reporting.Client(service="my-app", version=os.getenv("K_REVISION", "0"))
ER.report_exception()
return 0
So, the idea is to use my (or 'local') cloud function's service account to create credentials of a dedicated service account (GMAIL_SERV_ACCOUNT - which is used in many different cloud functions running under many different 'local' service accounts); then use that 'delegate' service account to get API service access.
I don't remember if the GMAIL_SERV_ACCOUNT should have any specific IAM roles. But I think the 'local' cloud function's service account should get roles/iam.serviceAccountTokenCreator for it.
Updated:
Some clarification on the IAM role. In terraform (I use it for my CICD) for a given functional component, it looks:
# this service account is an 'external' for the given functional component,
# it is managed in another repository and terraform state file
# so we should get it at first
data "google_service_account" "gmail_srv_account" {
project = "some project id"
account_id = "actual GMAIL_SERV_ACCOUNT account"
}
# now we provide IAM role for that working with it
# where 'google_service_account.local_cf_sa' is the service account,
# under which the given cloud function is running
resource "google_service_account_iam_member" "iam_token_creator_gmail_sa" {
service_account_id = data.google_service_account.gmail_srv_account.name
role = "roles/iam.serviceAccountTokenCreator"
member = "serviceAccount:${google_service_account.local_cf_sa.email}"
depends_on = [
google_service_account.local_cf_sa,
]
}
Which python library should i use to control or launch or delete an instance on my Google cloud platform from my private PC ?
I think the simplest way is to use the Google Compute Engine Python API Client Library. You can see a sample with examples here.
You can see the complete list of functions regarding instances in REST Resource: instances
As you can see, you might do:
import googleapiclient.discovery
compute = googleapiclient.discovery.build('compute', 'v1')
listInstance = compute.instances().list(project=project, zone=zone).execute()
stopInstance = compute.instances().stop(project=project, zone=zone, instance=instance_id).execute()
startInstance = compute.instances().start(project=project, zone=zone, instance=instance_id).execute()
deleteInstance = compute.instances().delete(project=project, zone=zone, instance=instance_id).execute()
Don't confuse the param name "instance" with the chosen name for the path parameter "resourceId". You can see on the right side or at the bottom of the page examples with the real parameter's name.
You also could directly call the the REST API (see example) with in Python if you prefer to using POST/PUT methods.
You also might want to use OAuth. As you can see in the examples of the links provided, it would be something like:
from googleapiclient import discovery
from oauth2client.client import GoogleCredentials
credentials = GoogleCredentials.get_application_default()
service = discovery.build('compute', 'v1', credentials=credentials)
# Project ID for this request.
project = 'my-project' # TODO: Update placeholder value.
# The name of the zone for this request.
zone = 'my-zone' # TODO: Update placeholder value.
# Name of the instance resource to start.
instance = 'my-instance' # TODO: Update placeholder value.
request = service.instances().start(project=project, zone=zone, instance=instance)
response = request.execute()
You may also want to check out libcloud.
I am trying to short an URL using Google API but using only the requests module.
The code looks like this:
import requests
Key = "" # found in https://developers.google.com/url-shortener/v1/getting_started#APIKey
api = "https://www.googleapis.com/urlshortener/v1/url"
target = "http://www.google.com/"
def goo_shorten_url(url=target):
payload = {'longUrl': url, "key":Key}
r = requests.post(api, params=payload)
print(r.text)
When I run goo_shorten_url it returns:
"error": {
"errors": [
{
"domain": "global",
"reason": "required",
"message": "Required",
"locationType": "parameter",
"location": "resource.longUrl"
}
],
"code": 400,
"message": "Required"
}
But the longUrl parameter is there!
What am I doing wrong?
At first, please confirm that "urlshortener api v1" is enabled at Google API Console.
Content-Type is required as a header. And please use data as a request parameter. The modified sample is as follows.
Modified sample :
import json
import requests
Key = "" # found in https://developers.google.com/url-shortener/v1/getting_started#APIKey
api = "https://www.googleapis.com/urlshortener/v1/url"
target = "http://www.google.com/"
def goo_shorten_url(url=target):
headers = {"Content-Type": "application/json"}
payload = {'longUrl': url, "key":Key}
r = requests.post(api, headers=headers, data=json.dumps(payload))
print(r.text)
If above script doesn't work, please use an access token. The scope is https://www.googleapis.com/auth/urlshortener. In the case of use of access token, the sample script is as follows.
Sample script :
import json
import requests
headers = {
"Authorization": "Bearer " + "access token",
"Content-Type": "application/json"
}
payload = {"longUrl": "http://www.google.com/"}
r = requests.post(
"https://www.googleapis.com/urlshortener/v1/url",
headers=headers,
data=json.dumps(payload)
)
print(r.text)
Result :
{
"kind": "urlshortener#url",
"id": "https://goo.gl/#####",
"longUrl": "http://www.google.com/"
}
Added 1 :
In the case of use tinyurl.com
import requests
URL = "http://www.google.com/"
r = requests.get("http://tinyurl.com/" + "api-create.php?url=" + URL)
print(r.text)
Added 2 :
How to use Python Quickstart
You can use Python Quickstart. If you don't have "google-api-python-client", please install it. After installed it, please copy paste a sample script from "Step 3: Set up the sample", and create it as a python script. Modification points are following 2 parts.
1. Scope
Before :
SCOPES = 'https://www.googleapis.com/auth/drive.metadata.readonly'
After :
SCOPES = 'https://www.googleapis.com/auth/urlshortener'
2. Script
Before :
def main():
"""Shows basic usage of the Google Drive API.
Creates a Google Drive API service object and outputs the names and IDs
for up to 10 files.
"""
credentials = get_credentials()
http = credentials.authorize(httplib2.Http())
service = discovery.build('drive', 'v3', http=http)
results = service.files().list(
pageSize=10,fields="nextPageToken, files(id, name)").execute()
items = results.get('files', [])
if not items:
print('No files found.')
else:
print('Files:')
for item in items:
print('{0} ({1})'.format(item['name'], item['id']))
After :
def main():
credentials = get_credentials()
http = credentials.authorize(httplib2.Http())
service = discovery.build('urlshortener', 'v1', http=http)
resp = service.url().insert(body={'longUrl': 'http://www.google.com/'}).execute()
print(resp)
After done the above modifications, please execute the sample script. You can get the short URL.
I am convinced that one CANNOT use ONLY requests to use google api for shorten an url.
Below I wrote the solution I ended up with,
It works, but it uses google api, which is ok, but I cannot find much documentation or examples about it (Not as much as I wanted).
To run the code remember to install google api for python first with
pip install google-api-python-client, then:
import json
from oauth2client.service_account import ServiceAccountCredentials
from apiclient.discovery import build
scopes = ['https://www.googleapis.com/auth/urlshortener']
path_to_json = "PATH_TO_JSON"
#Get the JSON file from Google Api [Website]
(https://console.developers.google.com/apis/credentials), then:
# 1. Click on Create Credentials.
# 2. Select "SERVICE ACCOUNT KEY".
# 3. Create or select a Service Account and
# 4. save the JSON file.
credentials = ServiceAccountCredentials.from_json_keyfile_name(path_to_json, scopes)
short = build("urlshortener", "v1",credentials=credentials)
request = short.url().insert(body={"longUrl":"www.google.com"})
print(request.execute())
I adapted this from Google's Manual Page.
The reason it has to be so complicated (more than I expected at first at least) is to avoid the OAuth2 authentication that requires the user (Me in this case) to press a button (to confirm that I can use my information).
As the question is not very clear this answer is divided in 4 parts.
Shortening URL Using:
1. API Key.
2. Access Token
3. Service Account
4. Simpler solution with TinyUrl.
API Key
At first, please confirm that "urlshortener api v1" is enabled at Google API Console.
Content-Type is required as a header. And please use data as a request parameter. The modified sample is as follows.
(Seems not to be working despite what the API manual says).
Modified sample :
import json
import requests
Key = "" # found in https://developers.google.com/url-shortener/v1/getting_started#APIKey
api = "https://www.googleapis.com/urlshortener/v1/url"
target = "http://www.google.com/"
def goo_shorten_url(url=target):
headers = {"Content-Type": "application/json"}
payload = {'longUrl': url, "key":Key}
r = requests.post(api, headers=headers, data=json.dumps(payload))
print(r.text)
Access Token:
If above script doesn't work, please use an access token. The scope is https://www.googleapis.com/auth/urlshortener. In the case of use of access token, the sample script is as follows.
This answer in Stackoverflow shows how to get an Access Token: Link.
Sample script :
import json
import requests
headers = {
"Authorization": "Bearer " + "access token",
"Content-Type": "application/json"
}
payload = {"longUrl": "http://www.google.com/"}
r = requests.post(
"https://www.googleapis.com/urlshortener/v1/url",
headers=headers,
data=json.dumps(payload)
)
print(r.text)
Result :
{
"kind": "urlshortener#url",
"id": "https://goo.gl/#####",
"longUrl": "http://www.google.com/"
}
Using Service Account
To avoid the user need to accept the OAuth authentication (with a pop up screen and all that) there is a solution that uses authentication from machine to machine using a Service Account (As mentioned in another proposed answer).
To run this part of the code remember to install google api for python first with pip install google-api-python-client, then:
import json
from oauth2client.service_account import ServiceAccountCredentials
from apiclient.discovery import build
scopes = ['https://www.googleapis.com/auth/urlshortener']
path_to_json = "PATH_TO_JSON"
#Get the JSON file from Google Api [Website]
(https://console.developers.google.com/apis/credentials), then:
# 1. Click on Create Credentials.
# 2. Select "SERVICE ACCOUNT KEY".
# 3. Create or select a Service Account and
# 4. save the JSON file.
credentials = ServiceAccountCredentials.from_json_keyfile_name(path_to_json, scopes)
short = build("urlshortener", "v1",credentials=credentials)
request = short.url().insert(body={"longUrl":"www.google.com"})
print(request.execute())
Adapted from Google's Manual Page.
Even simpler:
In the case of use tinyurl.com
import requests
URL = "http://www.google.com/"
r = requests.get("http://tinyurl.com/" + "api-create.php?url=" + URL)
print(r.text)
I was using gdata module to access, upload, download files from google doc. I have the oauth key and secret with me. Now I want to switch to google drive api. Learning and studying a bit on google drive api , it looks like a bit different in the authentication. I also have downloaded pydrive module so as I can start things up. But I am not able to authorize my server side python code to authorize/authenticate the user using my oauth keys and access my drive. Do any one has any spare know how on how I can use pydrive to access my drive with my previous auth keys. I just need a simple way to authenticate.
For using the gdata module we use either of these credentials-
1> username & password or
2> consumer oauth key and secret key.
Since you are trying to use oauth credentials, I think you want a Domain Wide Delegated Access for Google Drive, which will help you to achieve uploading/downloading files into any user's google drive through out the domain.
For this you need to generate a new Client ID of a Service Account Type from
Developer's Console
*.p12 file will get downloaded. Note the path where you save it.
Also note the email address of your Service account. These will be use while coding.
Below is the python code where u have to carefully edit-
PATH TO SERIVE ACCOUNT PRIVATE KEY, something#developer.gserviceaccount.com, EMAIL_ID#YOURDOMAIN.COM in order to run it properly and test it.
Hope this will help!
Resource- Google Drive API
import httplib2
import pprint
import sys
from apiclient.discovery import build
from oauth2client.client import SignedJwtAssertionCredentials
"""Email of the Service Account"""
SERVICE_ACCOUNT_EMAIL = 'something#developer.gserviceaccount.com'
"""Path to the Service Account's Private Key file"""
SERVICE_ACCOUNT_PKCS12_FILE_PATH = 'PATH TO SERIVE ACCOUNT PRIVATE KEY'
def createDriveService(user_email):
"""Build and returns a Drive service object authorized with the service accounts
that act on behalf of the given user.
Args:
user_email: The email of the user.
Returns:
Drive service object.
"""
f = file(SERVICE_ACCOUNT_PKCS12_FILE_PATH, 'rb')
key = f.read()
f.close()
credentials = SignedJwtAssertionCredentials(SERVICE_ACCOUNT_EMAIL, key,
scope='https://www.googleapis.com/auth/drive', sub=user_email)
http = httplib2.Http()
http = credentials.authorize(http)
return build('drive', 'v2', http=http)
drive_service=createDriveService('EMAIL_ID#YOURDOMAIN.COM')
result = []
page_token = None
while True:
try:
param = {}
if page_token:
param['pageToken'] = page_token
files = drive_service.files().list().execute()
#print files
result.extend(files['items'])
page_token = files.get('nextPageToken')
if not page_token:
break
except errors.HttpError, error:
print 'An error occurred: %s' % error
break
for f in result:
print '\n\nFile: ',f.get('title')
print "\n"