I'm sorry but English is not my first language and my English is broken.
I'm trying to download my 3000+ pictures using Google Photos API with Python.
First I downloaded MediaItems list with this code.
from pathlib import Path
from requests_oauthlib import OAuth2Session
import json
api_url = "https://photoslibrary.googleapis.com/v1/mediaItems"
scope = ["https://www.googleapis.com/auth/photoslibrary.readonly"]
def save_token(token):
token = {
"access_token": token.get("access_token"),
"refresh_token": token.get("refresh_token"),
"token_type": token.get("token_type"),
"expires_in": token.get("expires_in")
}
Path("token.json").write_text(json.dumps(token))
def load_token():
token = {
"access_token": "",
"refresh_token": "",
"token_type": "",
"expires_in": "-30",
}
path = Path("token.json")
if path.exists():
token = json.loads(path.read_text())
return token
def login():
auth_info = json.loads(Path("credentials.json").read_text()).get("installed", None)
assert auth_info is not None
token = load_token()
extras = {
"client_id": auth_info.get("client_id"),
"client_secret": auth_info.get("client_secret"),
}
google = OAuth2Session(
auth_info.get("client_id"),
scope=scope,
token=token,
auto_refresh_kwargs=extras,
token_updater=save_token,
auto_refresh_url=auth_info.get("token_uri"),
redirect_uri=auth_info.get("redirect_uris")[0]
)
if not google.authorized:
authorization_url, state = google.authorization_url(
auth_info.get("auth_uri"),
access_type="offline",
prompt="select_account"
)
print("Access {} and paste code.".format(authorization_url))
access_code = input(">>> ")
google.fetch_token(
auth_info.get("token_uri"),
client_secret=auth_info.get("client_secret"),
code=access_code
)
assert google.authorized
save_token(google.token)
return google
def test():
google = login()
response = google.get(api_url)
print(response.text)
if __name__ == "__main__":
test()
This code worked without problems and I downloaded about 30 json files (contains 3000 pictures information) with nextPageToken.
After that, I tried to download these pictures by this code.
The photo_info_list variable contains all MediaItems.
photo_download_format = "{base}=w{width}-h{height}"
def download_photos(photo_info_list):
google = login()
for photo_info in photo_info_list:
photo_id = photo_info.get("id", "dummy_id")
base_url = photo_info.get("baseUrl")
metadata = photo_info.get("mediaMetadata")
filename = photo_info.get("filename")
download_url = photo_download_format.format(
base=base_url,
width=metadata["width"],
height=metadata["height"]
)
response = google.get(download_url)
# save_picture
This code worked well for first 162 pictures (about 270MB) but then I got 403 forbidden error.
I deleted token and tried login procedures again, created another credentials but got the same errors.
Does anyone know what is the problem?
Any suggestion or information are really appreciate. Thank you!
baseUrls expire in 60 minutes after acquisition. Per the documentation for Google Photos APIs
You also shouldn't store baseUrls, which expire after approximately 60 minutes.
The most likely explanation is that your baseUrl has expired in the middle of downloading.
Related
I am trying to get data about a song playing on one of my devices from the spotify API. I have created a view that fetches data from the API and part of it looks like this:
class Song(viewsets.ModelViewSets):
....
room_code = request.data['room_code']
room = Room.objects.filter(code=room_code)[0]
host = room.host
endpoint = 'player/currently-playing'
response = execute_spotify_api_request(host, endpoint)
item = response.get('item')
duration = item.get('duration_ms')
progress = response.get('progress_ms')
album_cover = item.get('album').get('images')[0].get('url')
return Response(response, status=status.HTTP_200_OK)
The execute_spotify_api_request(host, endpoint) is a utility function and it looks like this:
def execute_spotify_api_request(session_id, endpoint, post_=False, put_=False):
tokens = get_user_tokens(session_id)
headers = {'Content-Type': 'application/json',
'Authorization': "Bearer " + tokens.access_token}
if post_:
post(BASE_URL + endpoint, headers=headers)
if put_:
post(BASE_URL + endpoint, headers=headers)
response = get(BASE_URL, {}, headers=headers)
try:
return response.json()
except:
return {'error': 'Could not retrieve a response'}
The full url from which im fetching is ""https://api.spotify.com/v1/me/player/currently-playing"
The problem is with the response that im getting from the API, the response is not an error but data that im not expecting to get. Im getting a response that looks like this:
response = {
"display_name": "Tanatswamanyakara",
"external_urls": {
"spotify": "https://open.spotify.com/user/dlnsysel6bndktbvduz6cl79w"
},
"followers": {
"href": null,
"total": 0
},
"href": "https://api.spotify.com/v1/users/dlnsysel6bndktbvduz6cl79w",
"id": "dlnsysel6bndktbvduz6cl79w",
"images": [],
"type": "user",
"uri": "spotify:user:dlnsysel6bndktbvduz6cl79w"
}
I was hoping to get data (the progress, title, duration, album, artist etc) about the song I am playing on my spotify account but instead I get that response, how do I fix that?
N.B
My access tokens and refresh tokens are working as they should. (so I think)
If the data is not what you are expecting then there's going to be something wrong with your API endpoint. You can use the Spotify developer console to generate the endpoint link, it would be worth debugging your execute_spotify_api_request code and the url it generates against the value in the console.
Having just re-read your code half way through answering, I've noticed that you aren't appending your endpoint variable to your GET url:
response = get(BASE_URL, {}, headers=headers)
This means that the get doesn't have 'player/currently-playing' and just returns the base URL which is probably 'https://api.spotify.com/v1/me/' - hence the response you receive is just your profile data.
I am trying to make a JWT call to storage API using the example listed here with some changes as below -
def generate_jwt():
"""Generates a signed JSON Web Token using a Google API Service Account."""
now = int(time.time())
sa_email = os.environ["FUNCTION_IDENTITY"]
expiry_length = 3600
# build payload
payload = {
'iat': now,
# expires after 'expiry_length' seconds.
"exp": now + expiry_length,
# iss must match 'issuer' in the security configuration in your
# swagger spec (e.g. service account email). It can be any string.
'iss': sa_email,
# aud must be either your Endpoints service name, or match the value
# specified as the 'x-google-audience' in the OpenAPI document.
'aud': "https://storage.googleapis.com",
# sub and email should match the service account's email address
'sub': sa_email,
'email': sa_email
}
# sign with keyfile
sa_keyfile="cred.json"
signer = google.auth.crypt.RSASigner.from_service_account_file(sa_keyfile)
jwt = google.auth.jwt.encode(signer, payload)
return jwt
and calliing it here
def make_jwt_request(signed_jwt, url="https://storage.googleapis.com/storage/v1/b/BUCKET_NAME"):
"""Makes an authorized request to the endpoint"""
headers = {
'Authorization': 'Bearer {}'.format(signed_jwt.decode('utf-8')),
'content-type': 'application/json',
"Host": "www.googleapis.com",
}
response = requests.get(url, headers=headers)
print(response.status_code, response.content)
response.raise_for_status()
but getting error as Couldn't parse the specified URI. Illegal URI.
I dont understand why it is a illegal URI. I tried with https://googleapis.com/storage/b/BUCKETNMAE but still same error. could not find anything on SO or google docs about this. any idea what wrong am I doing here ?
Google Cloud Storage does not accept a Signed JWT for authorization. Once you create the Signed JWT you must exchange the JWT for an Access Token.
Refer to my answer here or my article for a complete example in Python.
def exchangeJwtForAccessToken(signed_jwt):
'''
This function takes a Signed JWT and exchanges it for a Google OAuth Access Token
'''
auth_url = "https://www.googleapis.com/oauth2/v4/token"
params = {
"grant_type": "urn:ietf:params:oauth:grant-type:jwt-bearer",
"assertion": signed_jwt
}
r = requests.post(auth_url, data=params)
if r.ok:
return(r.json()['access_token'], '')
return None, r.text
I'm trying to use the JSON API for Google Cloud Storage to retrieve a file from Google Cloud Storage. I am not allowed to use the SDKs. Is it possible to create a JWT from a ServiceAccount.json file and use the JWT to access files from Google Cloud Storage? I have a script in node.js that generates a JWT from the service account, but i'm not sure if the audience is right
const jwt = require('jsonwebtoken');
const serviceAccount = require('./serviceAccount.json');
const issuedAt = Math.floor(Date.now() / 1000);
const TOKEN_DURATION_IN_SECONDS = 3600;
let params = {
'iss': serviceAccount.client_email,
'sub': serviceAccount.client_email,
'aud': serviceAccount.project_id,
'iat': issuedAt,
'exp': issuedAt + TOKEN_DURATION_IN_SECONDS,
};
let options = {
algorithm: 'RS256',
header: {
'kid': serviceAccount.private_key_id,
'typ': 'JWT',
'alg': 'RS256',
},
};
let token = jwt.sign(params, serviceAccount.private_key, options);
console.log(token);
I then use that JWT to call the Google Cloud Storage JSON API:
https://www.googleapis.com/storage/v1/b/test
Using the header: Authorization Bearer {token}
That simply resulted in a Invalid Credentials response.
A few questions:
I'm not sure what the 'aud' should be when creating the JWT. I've seen examples where it's a url and also where it's the projectId. Neither work for me.
One of the JSON API examples said the Authorization token should be an oauth token. Can I use a JWT instead or do I need to make a call using the JWT to get an access token?
Is my bucket path correct? Is the base folder for the bucket path your projectId? Should my path be /{projectId}/test. I've tried both and neither work.
Recap
This is an IoT project and I need embedded devices to download files from Google Cloud Storage. I need to create a web portal to upload files to (using Firebase Functions) and pass to the device either a bucket path or a private/signed URL that. The bottom line being I need to access a Google Cloud Storage bucket using a service account key. If there is an embedded SDK - great, but I couldn't find one for C. My only thought was to use the JSON API. If there is a way I can sign a URL which can only be accessed using a service account - that works too.
Thanks!
Yes, you can create your own Signed JWT from a service account Json (or P12) file and exchange the JWT for an Access Token that you then use as Authorization: Bearer TOKEN
I have written a number of articles on how to use Json and P12 credentials.
Google Cloud – Creating OAuth Access Tokens for REST API Calls
For your questions:
I'm not sure what the 'aud' should be when creating the JWT. I've seen
examples where it's a url and also where it's the projectId. Neither
work for me.
Set aud to "https://www.googleapis.com/oauth2/v4/token"
One of the JSON API examples said the Authorization token should be an
oauth token. Can I use a JWT instead or do I need to make a call using
the JWT to get an access token?
Some APIs accept signed JWTs, others expect an OAuth Access Token. It is just easier to always obtain the OAuth Access Token. In my example code below, I show you how.
Is my bucket path correct? Is the base folder for the bucket path your
projectId? Should my path be /{projectId}/test. I've tried both and
neither work.
Your url shold look like this (Python string building example)
url = "https://www.googleapis.com/storage/v1/b?project=" + project
Below I show you how to call two services (GCE and GCS). Most Google APIs will follow similar styles for building the REST API urls.
From the code in your question, you are missing the last step in the OAuth process. You need to exchange your Signed JWT for an Access Token.
def exchangeJwtForAccessToken(signed_jwt):
'''
This function takes a Signed JWT and exchanges it for a Google OAuth Access Token
'''
auth_url = "https://www.googleapis.com/oauth2/v4/token"
params = {
"grant_type": "urn:ietf:params:oauth:grant-type:jwt-bearer",
"assertion": signed_jwt
}
r = requests.post(auth_url, data=params)
if r.ok:
return(r.json()['access_token'], '')
return None, r.text
Here is a complete Python 3.x example that will list GCE instances. Below this code are changes to display GCS Buckets.
'''
This program lists lists the Google Compute Engine Instances in one zone
'''
import time
import json
import jwt
import requests
import httplib2
# Project ID for this request.
project = 'development-123456'
# The name of the zone for this request.
zone = 'us-west1-a'
# Service Account Credentials, Json format
json_filename = 'service-account.json'
# Permissions to request for Access Token
scopes = "https://www.googleapis.com/auth/cloud-platform"
# Set how long this token will be valid in seconds
expires_in = 3600 # Expires in 1 hour
def load_json_credentials(filename):
''' Load the Google Service Account Credentials from Json file '''
with open(filename, 'r') as f:
data = f.read()
return json.loads(data)
def load_private_key(json_cred):
''' Return the private key from the json credentials '''
return json_cred['private_key']
def create_signed_jwt(pkey, pkey_id, email, scope):
'''
Create a Signed JWT from a service account Json credentials file
This Signed JWT will later be exchanged for an Access Token
'''
# Google Endpoint for creating OAuth 2.0 Access Tokens from Signed-JWT
auth_url = "https://www.googleapis.com/oauth2/v4/token"
issued = int(time.time())
expires = issued + expires_in # expires_in is in seconds
# Note: this token expires and cannot be refreshed. The token must be recreated
# JWT Headers
additional_headers = {
'kid': pkey_id,
"alg": "RS256",
"typ": "JWT" # Google uses SHA256withRSA
}
# JWT Payload
payload = {
"iss": email, # Issuer claim
"sub": email, # Issuer claim
"aud": auth_url, # Audience claim
"iat": issued, # Issued At claim
"exp": expires, # Expire time
"scope": scope # Permissions
}
# Encode the headers and payload and sign creating a Signed JWT (JWS)
sig = jwt.encode(payload, pkey, algorithm="RS256", headers=additional_headers)
return sig
def exchangeJwtForAccessToken(signed_jwt):
'''
This function takes a Signed JWT and exchanges it for a Google OAuth Access Token
'''
auth_url = "https://www.googleapis.com/oauth2/v4/token"
params = {
"grant_type": "urn:ietf:params:oauth:grant-type:jwt-bearer",
"assertion": signed_jwt
}
r = requests.post(auth_url, data=params)
if r.ok:
return(r.json()['access_token'], '')
return None, r.text
def gce_list_instances(accessToken):
'''
This functions lists the Google Compute Engine Instances in one zone
'''
# Endpoint that we will call
url = "https://www.googleapis.com/compute/v1/projects/" + project + "/zones/" + zone + "/instances"
# One of the headers is "Authorization: Bearer $TOKEN"
headers = {
"Host": "www.googleapis.com",
"Authorization": "Bearer " + accessToken,
"Content-Type": "application/json"
}
h = httplib2.Http()
resp, content = h.request(uri=url, method="GET", headers=headers)
status = int(resp.status)
if status < 200 or status >= 300:
print('Error: HTTP Request failed')
return
j = json.loads(content.decode('utf-8').replace('\n', ''))
print('Compute instances in zone', zone)
print('------------------------------------------------------------')
for item in j['items']:
print(item['name'])
if __name__ == '__main__':
cred = load_json_credentials(json_filename)
private_key = load_private_key(cred)
s_jwt = create_signed_jwt(
private_key,
cred['private_key_id'],
cred['client_email'],
scopes)
token, err = exchangeJwtForAccessToken(s_jwt)
if token is None:
print('Error:', err)
exit(1)
gce_list_instances(token)
To display GCS Buckets instead, modify the code:
# Create the HTTP url for the Google Storage REST API
url = "https://www.googleapis.com/storage/v1/b?project=" + project
resp, content = h.request(uri=url, method="GET", headers=headers)
s = content.decode('utf-8').replace('\n', '')
j = json.loads(s)
print('')
print('Buckets')
print('----------------------------------------')
for item in j['items']:
print(item['name'])
I found this [Service account authorization without OAuth].(https://developers.google.com/identity/protocols/oauth2/service-account#jwt-auth
You can avoid having to make a network request to Google's authorization server before making an API call.
Available APIs are listed in https://github.com/googleapis/googleapis.
It looks like Google Cloud Storage api is not yet published as per the comments in the repository.
Were you able to use the cloud storage API with JWT?
I'm trying to make a chatbot in Hangouts Chat.
I'm referring this documentation to implement account linking.
Its default version is working but when I try to generate access_token and refresh token using Token Endpoint. It gives
{
"error": "invalid_grant",
"error_description": "Bad Request"
}
Here is my callback function code.
def on_oauth2_callback():
"""Handles the OAuth callback."""
print("IN CALLBACK ", flask.request.args)
oauth2_callback_args = OAuth2CallbackCipher.decrypt(
flask.request.args['state'])
user_name, redirect_url = (
oauth2_callback_args['user_name'],
oauth2_callback_args['redirect_url'])
oauth2_flow = flow.Flow.from_client_secrets_file(
OAUTH2_CLIENT_SECRET_FILE,
scopes=PEOPLE_API_SCOPES,
redirect_uri=flask.url_for('auth.on_oauth2_callback', _external=True),
state=flask.request.args['state'])
oauth2_flow.fetch_token(authorization_response=flask.request.url)
print("REDIRECT URL ", redirect_url)
auth_code = request.args.get('code')
data = {'code': auth_code,
'client_id': "xxxxxxxxxxxxxxx.apps.googleusercontent.com",
'client_secret': "xxxxxxxxxxxx",
'redirect_uri': redirect_url,
'grant_type': 'authorization_code'}
print("%^" * 10, json.dumps(data))
r = requests.post('https://www.googleapis.com/oauth2/v3/token', data=json.dumps(data))
print("%" * 10, r.text)
return flask.redirect(redirect_url)
What am I doing wrong? And if there's another way kindly enlighten me.
Once you call oauth2_flow.fetch_token(authorization_response=flask.request.url) you just exchanged the authorization code in that response for an access token.
So you don't need to call the token endpoint, you just need to get credentials:
credentials = oauth2_flow.credentials
And finally get token and refresh_token from credentials.token and credentials.refresh_token.
Take a look on this documentation.
I hope it's clear!
I am trying to short an URL using Google API but using only the requests module.
The code looks like this:
import requests
Key = "" # found in https://developers.google.com/url-shortener/v1/getting_started#APIKey
api = "https://www.googleapis.com/urlshortener/v1/url"
target = "http://www.google.com/"
def goo_shorten_url(url=target):
payload = {'longUrl': url, "key":Key}
r = requests.post(api, params=payload)
print(r.text)
When I run goo_shorten_url it returns:
"error": {
"errors": [
{
"domain": "global",
"reason": "required",
"message": "Required",
"locationType": "parameter",
"location": "resource.longUrl"
}
],
"code": 400,
"message": "Required"
}
But the longUrl parameter is there!
What am I doing wrong?
At first, please confirm that "urlshortener api v1" is enabled at Google API Console.
Content-Type is required as a header. And please use data as a request parameter. The modified sample is as follows.
Modified sample :
import json
import requests
Key = "" # found in https://developers.google.com/url-shortener/v1/getting_started#APIKey
api = "https://www.googleapis.com/urlshortener/v1/url"
target = "http://www.google.com/"
def goo_shorten_url(url=target):
headers = {"Content-Type": "application/json"}
payload = {'longUrl': url, "key":Key}
r = requests.post(api, headers=headers, data=json.dumps(payload))
print(r.text)
If above script doesn't work, please use an access token. The scope is https://www.googleapis.com/auth/urlshortener. In the case of use of access token, the sample script is as follows.
Sample script :
import json
import requests
headers = {
"Authorization": "Bearer " + "access token",
"Content-Type": "application/json"
}
payload = {"longUrl": "http://www.google.com/"}
r = requests.post(
"https://www.googleapis.com/urlshortener/v1/url",
headers=headers,
data=json.dumps(payload)
)
print(r.text)
Result :
{
"kind": "urlshortener#url",
"id": "https://goo.gl/#####",
"longUrl": "http://www.google.com/"
}
Added 1 :
In the case of use tinyurl.com
import requests
URL = "http://www.google.com/"
r = requests.get("http://tinyurl.com/" + "api-create.php?url=" + URL)
print(r.text)
Added 2 :
How to use Python Quickstart
You can use Python Quickstart. If you don't have "google-api-python-client", please install it. After installed it, please copy paste a sample script from "Step 3: Set up the sample", and create it as a python script. Modification points are following 2 parts.
1. Scope
Before :
SCOPES = 'https://www.googleapis.com/auth/drive.metadata.readonly'
After :
SCOPES = 'https://www.googleapis.com/auth/urlshortener'
2. Script
Before :
def main():
"""Shows basic usage of the Google Drive API.
Creates a Google Drive API service object and outputs the names and IDs
for up to 10 files.
"""
credentials = get_credentials()
http = credentials.authorize(httplib2.Http())
service = discovery.build('drive', 'v3', http=http)
results = service.files().list(
pageSize=10,fields="nextPageToken, files(id, name)").execute()
items = results.get('files', [])
if not items:
print('No files found.')
else:
print('Files:')
for item in items:
print('{0} ({1})'.format(item['name'], item['id']))
After :
def main():
credentials = get_credentials()
http = credentials.authorize(httplib2.Http())
service = discovery.build('urlshortener', 'v1', http=http)
resp = service.url().insert(body={'longUrl': 'http://www.google.com/'}).execute()
print(resp)
After done the above modifications, please execute the sample script. You can get the short URL.
I am convinced that one CANNOT use ONLY requests to use google api for shorten an url.
Below I wrote the solution I ended up with,
It works, but it uses google api, which is ok, but I cannot find much documentation or examples about it (Not as much as I wanted).
To run the code remember to install google api for python first with
pip install google-api-python-client, then:
import json
from oauth2client.service_account import ServiceAccountCredentials
from apiclient.discovery import build
scopes = ['https://www.googleapis.com/auth/urlshortener']
path_to_json = "PATH_TO_JSON"
#Get the JSON file from Google Api [Website]
(https://console.developers.google.com/apis/credentials), then:
# 1. Click on Create Credentials.
# 2. Select "SERVICE ACCOUNT KEY".
# 3. Create or select a Service Account and
# 4. save the JSON file.
credentials = ServiceAccountCredentials.from_json_keyfile_name(path_to_json, scopes)
short = build("urlshortener", "v1",credentials=credentials)
request = short.url().insert(body={"longUrl":"www.google.com"})
print(request.execute())
I adapted this from Google's Manual Page.
The reason it has to be so complicated (more than I expected at first at least) is to avoid the OAuth2 authentication that requires the user (Me in this case) to press a button (to confirm that I can use my information).
As the question is not very clear this answer is divided in 4 parts.
Shortening URL Using:
1. API Key.
2. Access Token
3. Service Account
4. Simpler solution with TinyUrl.
API Key
At first, please confirm that "urlshortener api v1" is enabled at Google API Console.
Content-Type is required as a header. And please use data as a request parameter. The modified sample is as follows.
(Seems not to be working despite what the API manual says).
Modified sample :
import json
import requests
Key = "" # found in https://developers.google.com/url-shortener/v1/getting_started#APIKey
api = "https://www.googleapis.com/urlshortener/v1/url"
target = "http://www.google.com/"
def goo_shorten_url(url=target):
headers = {"Content-Type": "application/json"}
payload = {'longUrl': url, "key":Key}
r = requests.post(api, headers=headers, data=json.dumps(payload))
print(r.text)
Access Token:
If above script doesn't work, please use an access token. The scope is https://www.googleapis.com/auth/urlshortener. In the case of use of access token, the sample script is as follows.
This answer in Stackoverflow shows how to get an Access Token: Link.
Sample script :
import json
import requests
headers = {
"Authorization": "Bearer " + "access token",
"Content-Type": "application/json"
}
payload = {"longUrl": "http://www.google.com/"}
r = requests.post(
"https://www.googleapis.com/urlshortener/v1/url",
headers=headers,
data=json.dumps(payload)
)
print(r.text)
Result :
{
"kind": "urlshortener#url",
"id": "https://goo.gl/#####",
"longUrl": "http://www.google.com/"
}
Using Service Account
To avoid the user need to accept the OAuth authentication (with a pop up screen and all that) there is a solution that uses authentication from machine to machine using a Service Account (As mentioned in another proposed answer).
To run this part of the code remember to install google api for python first with pip install google-api-python-client, then:
import json
from oauth2client.service_account import ServiceAccountCredentials
from apiclient.discovery import build
scopes = ['https://www.googleapis.com/auth/urlshortener']
path_to_json = "PATH_TO_JSON"
#Get the JSON file from Google Api [Website]
(https://console.developers.google.com/apis/credentials), then:
# 1. Click on Create Credentials.
# 2. Select "SERVICE ACCOUNT KEY".
# 3. Create or select a Service Account and
# 4. save the JSON file.
credentials = ServiceAccountCredentials.from_json_keyfile_name(path_to_json, scopes)
short = build("urlshortener", "v1",credentials=credentials)
request = short.url().insert(body={"longUrl":"www.google.com"})
print(request.execute())
Adapted from Google's Manual Page.
Even simpler:
In the case of use tinyurl.com
import requests
URL = "http://www.google.com/"
r = requests.get("http://tinyurl.com/" + "api-create.php?url=" + URL)
print(r.text)