As a team we inherited an project that is using Amazon SNS to publish mobile notifications to mobile apps, this system is already active for multiple years.
The system works as following
We create a customer specific Topic
We register a new endpoint with the GCM or APN token
We noticed in older setups we had some stale endpoint data e.g. of apps that where reinstalled, uninstalled, ...
We would like to do 2 things
Cleanup that older endpoints that are not active anymore (Can we do this purely on the "status" of the endpoint?)
Actively react on when a device comes inactive - I read on how you can subscribe on "EndpointUpdated" events with an http endpoint - but it's not clear to me how can we determine that the "EndpointUpdated" is an uninstall, ... so we could clean this up?
Thanks
Python 3.6
Boto3
Need to Configure AWS Profile Account in your Local.
import boto3
def start_process(aws_profile_name):
session = boto3.Session(profile_name=aws_profile_name)
sns_client = session.client('sns')
print("Started Process")
list_of_sns = sns_client.list_topics()
list_of_topics = [x.get("TopicArn") for x in list_of_sns.get('Topics')]
for topic in list_of_topics:
try:
subscriptions = sns_client.list_subscriptions_by_topic(TopicArn=topic)
list_of_subscription = [s.get("SubscriptionArn") for s in subscriptions.get('Subscriptions')]
unsubscribe(list_of_subscription, sns_client)
sns_client.delete_topic(TopicArn=topic)
print(f"Successfully Deleted Topic - {topic}")
except Exception:
print(f"Failed to delete the sns - {topic}")
subscriptions = sns_client.list_subscriptions()
print(subscriptions)
list_of_subscription = [s.get("SubscriptionArn") for s in subscriptions.get('Subscriptions')]
unsubscribe(list_of_subscription, sns_client)
def unsubscribe(subscription_arn: list, sns_client):
for arn in subscription_arn:
try:
sns_client.unsubscribe(SubscriptionArn=arn)
print(f"Successfully unsubscribe - {arn}")
except Exception:
print(f"Failed to unsubscribe - {arn}")
print("""
Here Profile name like where access key
and secret key you stored Locally
To See
--------------------------
cat ~/.aws/creddential
To Add new One:
--------------------------
aws configure --profile <your custom profile name>
""")
profile_name = input(f"Enter the profile name (i;e 'archive'): ")
start_process(profile_name)
Related
I recently subscribed to the SimilarWeb API on the AWS Data Exchange Marketplace but have been unable to access the datasets. The reason is: send_api_asset call requires an AssetID parameter just as used in this sample notebook however, it is not available on SimilarWeb dataset page. The only available details are license, productID, OfferID, RevisionID, and DatasetID. Does anyone have an experience interacting with products on the AWS data Exchange for APIs Mareketplace, if yes, how can one connect to the endpoints or access the data sets via API call without using an AssetID ?
I have attempted:
DATA_SET_ID = '****'
REVISION_ID = '*******'
#ASSET_ID = '' #not provided
BODY = json.dumps({'body_param': 'body_param_value'})
METHOD = 'POST'
PATH = '/'
QUERY_STRING_PARAMETERS = {'param1': 'value1', 'param2': 'value2'}
response = CLIENT.send_api_asset(
DataSetId=DATA_SET_ID,
RevisionId=REVISION_ID,
#AssetId='',
Method=METHOD,
Path=PATH,
Body=BODY,
QueryStringParameters=QUERY_STRING_PARAMETERS
)
but it throws a ParamValidationError due to the missing AssetID. Any help or guidance will be appreciated.
I have read extensively on how to access GCP Gmail API using service account and have given it domain-wide authority, using the instruction here:
https://support.google.com/a/answer/162106
Here is my service account:
Here is the scopes added to the domain-wide authority. You can see that the ID matches the service account.
One thing I notice is that my GCP project is an internal project, I havent' published it or anything, yet when I added the scope, it is not showing the service account email name but the project name. Does it make any difference? Do I need to set anything here? In the OAuth Consent Screen, I see the name of the project is being defined there. I have added all same scope on this screen too, not sure if it make any difference.
Here is my code:
from google.oauth2 import service_account
from googleapiclient import discovery
credentials_file = get_credentials('gmail.json')
scopes = ['https://www.googleapis.com/auth/gmail.readonly', 'https://www.googleapis.com/auth/gmail.labels', 'https://www.googleapis.com/auth/gmail.modify']
credentials = service_account.Credentials.from_service_account_info(credentials_file, scopes=scopes)
delegated_credentials = credentials.with_subject("abc#mydomain.com")
GMAIL_SERVICE = discovery.build('gmail', 'v1', credentials=delegated_credentials)
labels = GMAIL_SERVICE.users().labels().list(userId='me').execute()
Error message:
Google.auth.exceptions.RefreshError: ('unauthorized_client: Client is
unauthorized to retrieve access tokens using this method, or client
not authorized for any of the scopes requested.', {'error':
'unauthorized_client', 'error_description': 'Client is unauthorized to
retrieve access tokens using this method, or client not authorized for
any of the scopes requested.'})
Not sure I can answer precisely on the original question (I think not), but here how things are done in cloud functions developed by me. The following particular code snippet is written/adopted for this answer, and it was not tested:
import os
import google.auth
import google.auth.iam
from google.oauth2 import service_account
from google.auth.exceptions import MutualTLSChannelError
from google.auth.transport import requests
import googleapiclient.discovery
from google.cloud import error_reporting
GMAIL_SERV_ACCOUNT = "A service account which makes the API CALL"
OAUTH_TOKEN_URI = "https://accounts.google.com/o/oauth2/token"
GMAIL_SCOPES_LIST = ["https://mail.google.com/"] # for example
GMAIL_USER = "User's email address, who's email we would like to access. abc#mydomain.com - from your question"
# inside the cloud function code:
local_credentials, project_id = google.auth.default()
local_credentials.refresh(requests.Request())
signer = google.auth.iam.Signer(requests.Request(), local_credentials, GMAIL_SERV_ACCOUNT)
delegate_credentials = service_account.Credentials(
signer, GMAIL_SERV_ACCOUNT, OAUTH_TOKEN_URI, scopes=GMAIL_SCOPES_LIST, subject=GMAIL_USER)
delegate_credentials.refresh(requests.Request())
try:
email_api_service = googleapiclient.discovery.build(
'gmail', 'v1', credentials=delegate_credentials, cache_discovery=False)
except MutualTLSChannelError as err:
# handle it somehow, for example (stupid, artificial)
ER = error_reporting.Client(service="my-app", version=os.getenv("K_REVISION", "0"))
ER.report_exception()
return 0
So, the idea is to use my (or 'local') cloud function's service account to create credentials of a dedicated service account (GMAIL_SERV_ACCOUNT - which is used in many different cloud functions running under many different 'local' service accounts); then use that 'delegate' service account to get API service access.
I don't remember if the GMAIL_SERV_ACCOUNT should have any specific IAM roles. But I think the 'local' cloud function's service account should get roles/iam.serviceAccountTokenCreator for it.
Updated:
Some clarification on the IAM role. In terraform (I use it for my CICD) for a given functional component, it looks:
# this service account is an 'external' for the given functional component,
# it is managed in another repository and terraform state file
# so we should get it at first
data "google_service_account" "gmail_srv_account" {
project = "some project id"
account_id = "actual GMAIL_SERV_ACCOUNT account"
}
# now we provide IAM role for that working with it
# where 'google_service_account.local_cf_sa' is the service account,
# under which the given cloud function is running
resource "google_service_account_iam_member" "iam_token_creator_gmail_sa" {
service_account_id = data.google_service_account.gmail_srv_account.name
role = "roles/iam.serviceAccountTokenCreator"
member = "serviceAccount:${google_service_account.local_cf_sa.email}"
depends_on = [
google_service_account.local_cf_sa,
]
}
I'm trying to get access to a particular recurring Google Calendar Meeting from a Service Account. The owner of the Calendar Meeting is a real user and created in their personal calendar (which can't be changed at this point). Sharing the entire Google Calendar of that user with the Service Account is not a viable solution.
Instead I've tried to just invite the Service Account to the meeting - so far so good.
However, I'm unable to retrieve that event with the Google Calendar API.
Basically, I'm trying something like this:
scopes = ["https://www.googleapis.com/auth/calendar.readonly"]
google_credentials = ServiceAccountCredentials.from_json_keyfile_name(CREDENTIALS_FILE, scopes)
calendar_service = build("calendar", "v3", credentials=google_credentials)
events_result = calendar_service.events().list(
# calendarId="<some-service-account>#<some-project>.iam.gserviceaccount.com",
calendarId="primary",
singleEvents=True,
).execute()
print(events_result)
events = events_result.get("items", [])
print(events)
Neither getting the calendar with id primary nor with the <some-service-account>#<some-project>.iam.gserviceaccount.com works.
Also getting the calendar list of the service account looks like that:
scopes = ["https://www.googleapis.com/auth/calendar.readonly"]
google_credentials = ServiceAccountCredentials.from_json_keyfile_name(CREDENTIALS_FILE, scopes)
calendar_service = build("calendar", "v3", credentials=google_credentials)
print(calendar_service.calendarList().list().execute())
# OUTPUTS:
# {'kind': 'calendar#calendarList', 'etag': '"<removed>"', 'nextSyncToken': '<removed>', 'items': []}
Thus, an empty items list. I'm not sure what the problem is here - or if that's not even possible.
Am I missing some permissions for the Service Account?
I'm trying to create a user using Googles Directory API and a service account. However I'm getting the error
googleapiclient.errors.HttpError: <HttpError 403 when requesting https://admin.googleapis.com/admin/directory/v1/users?alt=json returned "Not Authorized to access this resource/api". Details: "Not Authorized to access this resource/api">
I've created a service account on the Google Console and allowed Domain wide delegation. It also says the Admin SDK API is enabled for my project. However I can't seem to create a user. The documentation is confusing me slightly. Here is my implementation
def create_googleuser(content, randpass):
''' This function creates a Google Apps account for a user passing webhook contents and password as arguments '''
# Get User info from Webhook and store them in variables
firstname = get_firstname(content)
secondname = get_secondname(content)
emailaddress = firstname + "." + secondname + "#example.com"
# Connect to google API
userscope = ['https://www.googleapis.com/auth/admin.directory.user']
service_account_credentials = ('serviceaccountcredentials.json')
credentials = service_account.Credentials.from_service_account_file(service_account_credentials, scopes=userscope)
userservice = googleapiclient.discovery.build('admin', 'directory_v1', credentials=credentials)
# Create a user dictionary with user details
userinfo = {"primaryEmail": emailaddress,"name":{"givenName":firstname,"familyName":secondname},"password":randpass}
print (emailaddress)
# Create user through googleAPI
userservice.users().insert(body = userinfo).execute()
I'm thinking that my implementation is wrong rather than the permissions as the serviceaccountcredentials.json should have the correct permissions. Any suggestions?
There are two possibilities for getting this error.
If the API method requires an impersonated user to be used.
If the impersonated user has not the relevant service enabled.
Solution for case 1:
Follow the documentation to impersonate a user account.
Solution for case 2:
In the Admin console, open user information and check that the user is not suspended.
Open the "Apps" panel and check that the relevant service is "On".
May be caused by a user not having a license which allows access to the service (Cloud Identity instead of Google Workspace), or a user being in an organizational unit which has the service disabled.
Also this link might be helpful.
Thanks for the input. You were both correct to a point. Basically there were two issues. The service account user needs to be delegated domain administrator privileges that require domain admin actions, domain wide delegation isn't enough. Also the domain scope needed to be broader in the Admin console and the scope definition within the code. There is github issue open which helped here:
https://github.com/googleapis/google-api-nodejs-client/issues/1884
My working code looks like this
def create_googleuser(content, randpass):
''' This function creates a Google Apps account for a user passing webhook contents and password as arguments '''
# Get User info from Webhook and store them in variables
username = get_username(content)
firstname = get_firstname(content)
secondname = get_secondname(content)
emailaddress = firstname + "." + secondname + "#example.com"
# Connect to google API
userscope = ['https://www.googleapis.com/auth/admin.directory.user', 'https://www.googleapis.com/auth/admin.directory.user.security']
service_account_credentials = ('serviceaccountcredentials.json')
credentials = service_account.Credentials.from_service_account_file(service_account_credentials, scopes=userscope)
delegated_credentials = credentials.with_subject('domain.admin#example.com')
userservice = googleapiclient.discovery.build('admin', 'directory_v1', credentials=delegated_credentials)
# Create a user dictionary with user details
userinfo = {"primaryEmail": emailaddress,"name":{"givenName":firstname,"familyName":secondname},"password":randpass}
# Create user through googleAPI
userservice.users().insert(body = userinfo).execute()
When I do this:
session1 = boto3.session.Session(profile_name='my_profile')
iam_client = session1.client('iam')
Given that user='an_existin_iam_user', the below succeeds as I can print the accessKeyId and the accessKeySecret successfully.
responseCreateAccessKey = iam_client.create_access_key(UserName=user)
accessKeyId = responseCreateAccessKey.get('AccessKey').get('AccessKeyId')
accessKeySecret = responseCreateAccessKey.get('AccessKey').get('SecretAccessKey')
Hence, I can use the iam_client to create access credentials for other users (given that the my_profile AWS profile holds creds for an admin IAM user)
Now, with the newly created credentials above, I want to create a new IAM user. The below are the things I have tried to achieve this:
Attempt 1
temp_session = boto3.Session(aws_access_key_id=accessKeyId,aws_secret_access_key=accessKeySecret,region_name='ap-southeast-1')
temp_iam_client = temp_session.client('iam')
responseCreateUser = temp_iam_client.create_user(UserName='my-new-user')
Attempt 2
temp_iam_client = boto3.client('iam',aws_access_key_id=accessKeyId,aws_secret_access_key=accessKeySecret,region_name='ap-southeast-1')
responseCreateUser = temp_iam_client.create_user(UserName='my-new-user')
Attempt 3
temp_session = boto3.session.Session(aws_access_key_id=accessKeyId,aws_secret_access_key=accessKeySecret,region_name='ap-southeast-1')
temp_iam_client = temp_session.client('iam')
responseCreateUser = temp_iam_client.create_user(UserName='my-new-user')
Attempt 4 (as also suggested in one of the answers below)
temp_session = boto3.session.Session(aws_access_key_id=accessKeyId,aws_secret_access_key=accessKeySecret,region_name='ap-southeast-1')
temp_iam_client = temp_session.client('iam')
time.sleep(5)
responseCreateUser = temp_iam_client.create_user(UserName='my-new-user')
Each one of the above fails with the following error
An error occurred (InvalidClientTokenId) when calling the CreateUser operation: The security token included in the request is invalid.
Is there something that I am missing here?
All I want to do is create create a new user with the (as shown above) newly generated access credentials (accesskey and access secret).
It's ok if I get an AccessDeniedexception. I am expecting that for the users that do not have the right privileges. But the InvalidClientTokenId exception is something I don't understand why am I even getting it.
Also, if it helps, I am trying to do this (creation of access key and secret and then using it to create a new user) in an iterative loop for over a 100 users.
(The loop of course handles cases where access keys for the currently being processed user is already maxed out etc.)
So say for example, if I try doing the above user creation action, from the ipython console, individually for the user say, existing-test-user, alone, then I get the proper AccessDenied exception. However, when the loop reaches that user name (after iterating over the initial few users for example) then I get the InvalidClientTokenId exception.
On an additional note was going through this the get_credentials() section and I'm just wondering if it has anything to do with credentials caching ?
AWS IAM APIs employ an eventually consistent model, so delays happen during updates to IAM resources (such as creating access keys). That means that even if the function call to create an AWS Access Key succeeds, it does not necessarily mean that the AWS Access Key has completed its creation on the AWS Servers immediately.
https://docs.aws.amazon.com/IAM/latest/UserGuide/troubleshoot_general.html#troubleshoot_general_eventual-consistency
New Answer
Using backoff, we can set a retry policy for the creation of IAM users using newly created AWS access keys whilst having a timeout:
import boto3
import botocore
import backoff
#backoff.on_exception(backoff.expo, botocore.exceptions.ClientError, max_time=30)
def create_user(access_key_id, access_key_secret, username):
temp_session = boto3.Session(aws_access_key_id=access_key_id,aws_secret_access_key=access_key_secret,region_name='ap-southeast-1')
temp_iam_client = temp_session.client('iam')
temp_iam_client.create_user(UserName=username)
session = boto3.session.Session(profile_name='default')
iam_client = session.client('iam')
response_access_key = iam_client.create_access_key(UserName='user')
access_key_id = response_access_key.get('AccessKey').get('AccessKeyId')
access_key_secret = response_access_key.get('AccessKey').get('SecretAccessKey')
create_user(access_key_id, access_key_secret, 'my-new-user')
Old Answer
Adding a sleep just after creation of the AWS access key is a workaround.
import boto3
import time
session1 = boto3.session.Session(profile_name='my_profile')
iam_client = session1.client('iam')
responseCreateAccessKey = iam_client.create_access_key(UserName=user)
accessKeyId = responseCreateAccessKey.get('AccessKey').get('AccessKeyId')
accessKeySecret = responseCreateAccessKey.get('AccessKey').get('SecretAccessKey')
time.sleep(10)
temp_session = boto3.Session(aws_access_key_id=accessKeyId,aws_secret_access_key=accessKeySecret,region_name='ap-southeast-1')
temp_iam_client = temp_session.client('iam')
responseCreateUser = temp_iam_client.create_user(UserName='my-new-user')