I'm using google api, googleapiclient.discovery and build.instances().list() to extract all instances information under my gcp project. But I'm having permission error.
I can run compute.instances().list() fine on local and compute.instances().list() and by using Application Default Credentials. But the default credentials doesn't work when I publish app to server, and it raise exception:
raise exceptions.DefaultCredentialsError(_HELP_MESSAGE)
2019-07-17T00:55:56.682326031Z google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials.
So I need to set credential explicitly. And I downloaded service account key as "credential.json" and tried to set GOOGLE_APPLICATION_CREDENTIALS as following
import googleapiclient.discovery
import os
os.environ["GOOGLE_APPLICATION_CREDENTIALS"]= "credential.json"
compute = googleapiclient.discovery.build('compute', 'v1')
compute.instances().list(project="projectname", zone = "us-west1-a").execute()
But I'm getting the permission error
Traceback (most recent call last):
File "lst_temp.py", line 6, in <module>
compute.instances().list(project="project-name", zone = "us-west1-a").execute()
File "/usr/local/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
return wrapped(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/googleapiclient/http.py", line 851, in execute
raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 403 when requesting https://www.googleapis.com/compute/v1/projects/project-name/zones/us-west1-a/instances?alt=json returned "Required 'compute.instances.list' permission for 'projects/projectname'">
Related
I want to list all the Google Cloud Schedule jobs within a project, but to use the ListJobsRequest() class the parent parameter is required: projects/PROJECT_ID/locations/LOCATION_ID. Since I have jobs in different locations I would like to list all jobs, is it possible to do?
I already tried projects/PROJECT_ID/locations/* and projects/PROJECT_ID/locations.
In both cases I got the following error:
Traceback (most recent call last):
File "C:\Users\user\Desktop\TI\GCP\bq-to-scheduler\main.py", line 97, in <module>
list_scheduler_jobs()
File "C:\Users\user\Desktop\TI\GCP\bq-to-scheduler\main.py", line 51, in list_scheduler_jobs
page_result = client.list_jobs(request=request)
File "C:\Users\user\Desktop\TI\GCP\bq-to-scheduler\venv\lib\site-packages\google\cloud\scheduler_v1\services\cloud_scheduler\client.py", line 548, in list_jobs
response = rpc(
File "C:\Users\user\Desktop\TI\GCP\bq-to-scheduler\venv\lib\site-packages\google\api_core\gapic_v1\method.py", line 154, in __call__
return wrapped_func(*args, **kwargs)
File "C:\Users\user\Desktop\TI\GCP\bq-to-scheduler\venv\lib\site-packages\google\api_core\retry.py", line 283, in retry_wrapped_func
return retry_target(
File "C:\Users\user\Desktop\TI\GCP\bq-to-scheduler\venv\lib\site-packages\google\api_core\retry.py", line 190, in retry_target
return target()
File "C:\Users\user\Desktop\TI\GCP\bq-to-scheduler\venv\lib\site-packages\google\api_core\grpc_helpers.py", line 52, in error_remapped_callable
raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.PermissionDenied: 403 The principal (user or service account) lacks IAM permission "cloudscheduler.jobs.list" for the resource "projects/projeto1-358102/locations/*" (or the resource may not exist).
Help?
You can list all your available locations in your project using the code below. I also included calling list_jobs() to send a request to list the available jobs on the location.
I got the code on listing location in this document but I edited the authentication to use google.auth library instead of oauth2client.client since this is already deprecated.
from googleapiclient import discovery
import google.auth
from google.cloud import scheduler_v1
def get_project_locations():
credentials, project = google.auth.default(scopes=['https://www.googleapis.com/auth/cloud-platform'])
service = discovery.build('cloudscheduler', 'v1', credentials=credentials)
name = f'projects/{project}'
loc_arr = []
request = service.projects().locations().list(name=name)
while True:
response = request.execute()
for location in response.get('locations', []):
loc_arr.append(location['labels']['cloud.googleapis.com/region'])
request = service.projects().locations().list_next(previous_request=request, previous_response=response)
if request is None:
break
return project,loc_arr
def list_jobs(project_id,location_list):
client = scheduler_v1.CloudSchedulerClient()
for location in location_list:
request = scheduler_v1.ListJobsRequest(parent = f"projects/{project_id}/locations/{location}")
page_result = client.list_jobs(request=request)
for response in page_result:
print(response.name)
print(response.http_target)
print(response.schedule)
project_id,location_list = get_project_locations()
list_jobs(project_id,location_list)
Output:
From GCP scheduler:
I'm trying to download the report from DCM by using python client library. I'm using service account and using this link code this!. I have followed all the steps to set up the service account file which is mentioned in DCM api url, but I'm getting below error while running the code.
Traceback (most recent call last):
File "/usr/bin/anaconda/envs/py35/lib/python3.5/site-packages/oauth2client/clientsecrets.py", line 86, in _validate_clientsecrets
(client_type, client_info), = clientsecrets_dict.items()
ValueError: too many values to unpack (expected 1)
I'm using below service account file to connect to the DCM api and download the reports. This is the process I'm trying to set up so that on daily basis the reports get download automatically without any manual intervention.
import argparse
import sys
from googleapiclient import discovery
import httplib2
from oauth2client import client
from oauth2client import tools
from oauth2client.service_account import ServiceAccountCredentials
# Declare command-line flags.
argparser = argparse.ArgumentParser(add_help=False)
argparser.add_argument('path_to_service_account_json_file',help='Path to the service account JSON file to use for authenticating.')
argparser.add_argument('-i','--impersonation_email',help='Google account email to impersonate.')
# The OAuth 2.0 scopes to request.
OAUTH_SCOPES = ['https://www.googleapis.com/auth/dfareporting']
def main(argv):
# Retrieve command line arguments.
parser = argparse.ArgumentParser(description=__doc__,formatter_class=argparse.RawDescriptionHelpFormatter,parents=[tools.argparser, argparser])
flags = parser.parse_args(argv[1:])
# Authenticate using the supplied service account credentials
http = authenticate_using_service_account(flags.path_to_service_account_json_file,flags.impersonation_email)
# Construct a service object via the discovery service.
service = discovery.build('dfareporting', 'v3.3', http=http)
try:
# Construct the request.
request = service.userProfiles().list()
# Execute request and print response.
response = request.execute()
for profile in response['items']:
print('Found user profile with ID %s and user name "%s".' %(profile['profileId'], profile['userName']))
except client.AccessTokenRefreshError:
print('The credentials have been revoked or expired, please re-run the ''application to re-authorize')
def authenticate_using_service_account(path_to_service_account_json_file,impersonation_email):
"""Authorizes an httplib2.Http instance using service account credentials."""
# Load the service account credentials from the specified JSON keyfile.
credentials = ServiceAccountCredentials.from_json_keyfile_name(path_to_service_account_json_file,scopes=OAUTH_SCOPES)
# Configure impersonation (if applicable).
if impersonation_email:
credentials = credentials.create_delegated(impersonation_email)
# Use the credentials to authorize an httplib2.Http instance.
http = credentials.authorize(httplib2.Http())
return http
if __name__ == "__main__":
main(sys.argv)
TypeError: cannot instantiate ctype 'EVP_MD_CTX' of unknown size
I have built an api with flask-restful and flask-jwt-extended and have correctly configured the validation passages for token expiration and invalidation. However, even though it has built the token expiration and invalid validation callbacks, api does not process correctly and reports the error: Signature has expired
On the server in the cloud, we have a Centos 7 x64 of 16gb ram, running the application using gunicorn in version 19.9.0. Using the miniconda to create the applications' python environments.
In tests in the production environment, the application complains of the expired token. However in a test environment, using Ubuntu 18.04.2, x64 with 16 gb ram, using the same settings with miniconda and gunicorn, the application has no problems executing it, returning the correct message when the token expires.
My jwt.py
from flask import Blueprint, Response, json, request
from flask_jwt_extended import (JWTManager, create_access_token,
create_refresh_token, get_jwt_identity,
jwt_required)
from app.models.core import User
from .schemas import UserSchema
from .utils import send_reponse, user_roles
def configure_jwt(app):
JWT = JWTManager(app)
#JWT.expired_token_loader
def my_expired_token_callback(expired_token):
return Response(
response=json.dumps({
"message": "Expired token"
}),
status=401,
mimetype='application/json'
)
#JWT.invalid_token_loader
def my_invalid_token_callback(invalid_token):
return Response(
response=json.dumps({
"message": "Invalid token"
}),
status=422,
mimetype='application/json'
)
Error log:
[2019-05-23 15:42:02 -0300] [3745] [ERROR] Exception on /api/company [POST]
Traceback (most recent call last):
File "/home/company/miniconda3/envs/api_ms/lib/python3.6/site-packages/flask/app.py", line 1813, in full_dispatch_request
rv = self.dispatch_request()
File "/home/company/miniconda3/envs/api_ms/lib/python3.6/site-packages/flask/app.py", line 1799, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/company/miniconda3/envs/api_ms/lib/python3.6/site-packages/flask_restful/__init__.py", line 458, in wrapper
resp = resource(*args, **kwargs)
File "/home/company/miniconda3/envs/api_ms/lib/python3.6/site-packages/flask/views.py", line 88, in view
return self.dispatch_request(*args, **kwargs)
File "/home/company/miniconda3/envs/api_ms/lib/python3.6/site-packages/flask_restful/__init__.py", line 573, in dispatch_request
resp = meth(*args, **kwargs)
File "/home/company/miniconda3/envs/api_ms/lib/python3.6/site-packages/flask_jwt_extended/view_decorators.py", line 102, in wrapper
verify_jwt_in_request()
File "/home/company/miniconda3/envs/api_ms/lib/python3.6/site-packages/flask_jwt_extended/view_decorators.py", line 31, in verify_jwt_in_request
jwt_data = _decode_jwt_from_request(request_type='access')
File "/home/company/miniconda3/envs/api_ms/lib/python3.6/site-packages/flask_jwt_extended/view_decorators.py", line 266, in _decode_jwt_from_request
decoded_token = decode_token(encoded_token, csrf_token)
File "/home/company/miniconda3/envs/api_ms/lib/python3.6/site-packages/flask_jwt_extended/utils.py", line 107, in decode_token
allow_expired=allow_expired
File "/home/company/miniconda3/envs/api_ms/lib/python3.6/site-packages/flask_jwt_extended/tokens.py", line 138, in decode_jwt
leeway=leeway, options=options)
File "/home/company/miniconda3/envs/api_ms/lib/python3.6/site-packages/jwt/api_jwt.py", line 104, in decode
self._validate_claims(payload, merged_options, **kwargs)
File "/home/company/miniconda3/envs/api_ms/lib/python3.6/site-packages/jwt/api_jwt.py", line 134, in _validate_claims
self._validate_exp(payload, now, leeway)
File "/home/company/miniconda3/envs/api_ms/lib/python3.6/site-packages/jwt/api_jwt.py", line 175, in _validate_exp
raise ExpiredSignatureError('Signature has expired')
jwt.exceptions.ExpiredSignatureError: Signature has expired
I'm trying to understand why the application is able to correctly return the token expiration message in the test environment, where in the production environment it returns the error code 500 Internal Server Error. In addition to fixing this problem in our application.
Based on this link found inside the project repository, I discovered that the problem is related to the flask configuration option called PROPAGATE_EXCEPTIONS, which must be True.
The issue in the flask-jwt-extended repository that helped me find the answer.
This comment states that Flask Restful needs to ignore JWT and JWT Extended Exceptions and provides a simple snippet that solves the issue.
Copying the code from above link,
from flask_jwt_extended.exceptions import JWTExtendedException
from jwt.exceptions import PyJWTError
class FixedApi(Api):
def error_router(self, original_handler, e):
if not isinstance(e, PyJWTError) and not isinstance(e, JWTExtendedException) and self._has_fr_route():
try:
return self.handle_error(e)
except Exception:
pass # Fall through to original handler
return original_handler(e)
from apiclient.discovery import build
from oauth2client.service_account import ServiceAccountCredentials
import json
import base64
import httplib2
f = file('tara1-553d334b8702.p12', 'rb')
key = f.read()
f.close()
credentials = ServiceAccountCredentials.from_p12_keyfile(
'googleapptara#tara1-167105.iam.gserviceaccount.com',
'tara1-554d335b8702.p12',
private_key_password='notasecret',
scopes=['https://www.googleapis.com/auth/admin.directory.user.readonly','https://www.googleapis.com/auth/admin.directory.user']
)
delegated_credentials=credentials.create_delegated('tara.gurung#orgemail.net)
http = httplib2.Http()
http = credentials.authorize(http)
DIRECOTORY = build('admin', 'directory_v1', credentials=credentials)
maxResults=10,orderBy='email').execute()
results = DIRECOTORY.users().list(domain='domainnamehere.net').execute()
users = results.get('users', [])
print users
Here ,I am trying to do the server to server authentication using the p12 security file and trying to grab all the users in the domain.
I have successfully fetched the users list by 3legs authentication,by authorizing from the browswer in the same account
But this way it's throwing me the following errors.
File "testing.py", line 41, in <module>
results = DIRECOTORY.users().list(domain='domainemailhere.net').execute()
File "/home/tara/taraproject/scripttesting/virtualenvforGapi/local/lib/python2.7/site-packages/oauth2client/_helpers.py", line 133, in positional_wrapper
return wrapped(*args, **kwargs)
File "/home/tara/taraproject/scripttesting/virtualenvforGapi/local/lib/python2.7/site-packages/googleapiclient/http.py", line 840, in execute
raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 403 when requesting https://www.googleapis.com/admin/directory/v1/users?domain=nepallink.net&alt=json returned "Not Authorized to access this resource/api">
SETUP DONE:
I have a super admin level access in the admin console.
I have also added
the scope via security>showmore>advance>manageipclient>authorize
Added the user id and scope
https://www.googleapis.com/auth/admin.directory.user.readonly
https://www.googleapis.com/auth/admin.directory.user
Added the users permission in service console and made a owner.
Admin SDK is Enabled
Where exactly am I missing the things. Why does it says I have no authority to access the resources/api
I see you are using delegated_credentials. Have you used it??
Change following line:
DIRECOTORY = build('admin', 'directory_v1', credentials=credentials)
to
DIRECOTORY = build('admin', 'directory_v1', credentials=delegated_credentials)
i want to use admin sdk directory api to create eamil account of users.
i am using google-api-python-client-1.2 library.
in folder /samples/service_account/tasks.py works for me.
but when i chance that file to list users from admin directory api it doesn't works and throws errors.
below is the code i am using.
import httplib2
import pprint
import sys
import inspect
from apiclient.discovery import build
from oauth2client.client import SignedJwtAssertionCredentials
def main(argv):
f = file('my-privatekey.p12', 'rb')
key = f.read()
f.close()
credentials = SignedJwtAssertionCredentials(
'my#developer.gserviceaccount.com',
key,
scope=['https://www.googleapis.com/auth/admin.directory.user', 'https://www.googleapis.com/auth/admin.directory.user.readonly'])
http = httplib2.Http()
http = credentials.authorize(http)
service = build("admin", "directory_v1", http)
list_of_apis = service.users().list(domain='mydomain.com').execute(http=http)
pprint.pprint(list_of_apis)
if __name__ == '__main__':
main(sys.argv)
when i run the above code i get below errors.
$python tasks.py
No handlers could be found for logger "oauth2client.util"
Traceback (most recent call last):
File "tasks.py", line 77, in <module>
main(sys.argv)
File "tasks.py", line 66, in main
list_of_apis = service.users().list(domain='messycoders.com').execute(http=http)
File "/usr/local/lib/python2.7/dist-packages/oauth2client/util.py", line 132, in positional_wrapper
return wrapped(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/apiclient/http.py", line 723, in execute
raise HttpError(resp, content, uri=self.uri) apiclient.errors.HttpError: <HttpError 403 when requesting https://www.googleapis.com/admin/directory/v1/users?domain=messycoders.com&alt=json returned "Not Authorized to access this resource/api">
Try:
credentials = SignedJwtAssertionCredentials(
'my#developer.gserviceaccount.com',
key,
sub='superadmin#mydomain.com',
scope=['https://www.googleapis.com/auth/admin.directory.user',])
You don't need both scopes, use readonly if you're doing read operations only, use the above if you're doing read and write.
sub= defines which Google Apps account the service account should impersonate to perform the directory operations, it's necessary and the account needs to have the right permissions.
Lastly, be sure that you've granted the service account's client_id access to the directory scopes you need in the Control Panel. The steps to do this are listed in the Drive documentation, just sub in the correct scope(s) for Admin Directory.