Schedule to stop and start instance on Google Cloud Platform - Cloud Scheduler it's the Cloud Function - google-cloud-platform

I have the following need. I would like every day at 5:00 am to restart my instance, so I read the best way to automate it would be using Cloud Scheduler and Cloud Function, but I am not aware of these two GCP features.
I created two schedules in the Cloud Scheduler where my VM instance STOP at 5:00 am and another one that START at 5:10 am, but I don't know how to proceed in Cloud Function to end my process.
Could someone help me with this? hug to everyone!
See how are my project error log when trying to implement
ERROR
{ "jobName": "projects/my-project/locations/us-central1/jobs/Stop", "#type": "type.googleapis.com/google.cloud.scheduler.logging.AttemptFinished", "status": "INTERNAL", "url": "https://us-central1-my-project.cloudfunctions.net/power/stop?zone=us-central1-a&instance=my-instance", "targetType": "HTTP" }
###############
{
insertId: "1klx7n3g18eq5zs"
jsonPayload: {
jobName: "projects/my-project/locations/us-central1/jobs/Stop"
#type: "type.googleapis.com/google.cloud.scheduler.logging.AttemptFinished"
status: "INTERNAL"
url: "https://us-central1-my-project.cloudfunctions.net/power/stop?zone=us-central1-a&instance=my-instance"
targetType: "HTTP"
}
httpRequest: {
status: 500
}
resource: {
type: "cloud_scheduler_job"
labels: {
location: "us-central1"
project_id: "my-project"
job_id: "Stop"
}
}
timestamp: "2020-08-07T08:00:06.896367090Z"
severity: "ERROR"
logName: "projects/my-project/logs/cloudscheduler.googleapis.com%2Fexecutions"
receiveTimestamp: "2020-08-07T08:00:06.896367090Z"
}

I have the same use case on my project, and I use this python function in order to start and stop a vm, my function can handle the paths start and stop
from flask import Flask, request, abort
import os
import logging
app = Flask(__name__)
#app.route('/')
#app.route('/<path:path>')
def power(path=None):
#this libraries are mandatory to reach compute engine api
from googleapiclient import discovery
from oauth2client.client import GoogleCredentials
#the function will take the service account of your function
credentials = GoogleCredentials.get_application_default()
#this line is to specify the api that we gonna use, in this case compute engine
service = discovery.build('compute', 'v1', credentials=credentials, cache_discovery=False)
logging.getLogger('googleapiclient.discovery_cache').setLevel(logging.ERROR)
# Project ID for this request.
project = yourprojectID # Update placeholder value.
zone = request.args.get('zone', None)
instance = request.args.get('instance', None)
#call the method to start or stop the instance
if (request.path == "/stop"):
req = service.instances().stop(project=project, zone=zone, instance=instance)
elif (request.path == "/start"):
req = service.instances().start(project=project, zone=zone, instance=instance)
else:
abort(418)
#execute the command
response = req.execute()
print(response)
return ("OK")
if __name__ == '__main__':
app.run(port=3000, debug=True)
requirements.txt file
google-api-python-client
oauth2client
flask
Scheduler config
Create a service account with functions.invoker permission within your function
Create new Cloud scheduler job
Specify the frequency in cron format.
Specify HTTP as the target type.
Add the URL of your cloud function and method as always.
Select the token OIDC from the Auth header dropdown
Add the service account email in the Service account text box.
In audience field you must only need to write the URL of the function without any additional parameter
On cloud scheduler, I hit my function by using these URLs
https://us-central1-yourprojectID.cloudfunctions.net/power/stop?zone=us-central1-a&instance=instance-1
https://us-central1-yourprojectID.cloudfunctions.net/power/stop?zone=us-central1-a&instance=instance-1
and I used this audience
https://us-central1-yourprojectID.cloudfunctions.net/power
please replace yourprojectID in the code and in the URLs
us-central1 is the region where my function is located, power is the name of my function, us-central1-a is the zone where my instance is located and instance-1 is the name of my instance
update for java
Cloud functions supports java11 and maven but this is on beta
first you need this dependencies, please add the following lines to your pom.xml file:
<dependency>
<groupId>com.google.apis</groupId>
<artifactId>google-api-services-compute</artifactId>
<version>beta-rev20200629-1.30.10</version>
</dependency>
<dependency>
<groupId>com.google.api-client</groupId>
<artifactId>google-api-client</artifactId>
<version>1.30.10</version>
</dependency>
Cloud function to START a VM
package com.example;
import com.google.cloud.functions.HttpFunction;
import com.google.cloud.functions.HttpRequest;
import com.google.cloud.functions.HttpResponse;
import java.io.BufferedWriter;
import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
import com.google.api.client.googleapis.javanet.GoogleNetHttpTransport;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.json.JsonFactory;
import com.google.api.client.json.jackson2.JacksonFactory;
import com.google.api.services.compute.Compute;
import com.google.api.services.compute.model.Operation;
import java.io.IOException;
import java.security.GeneralSecurityException;
import java.util.Arrays;
public class Example implements HttpFunction {
#Override
public void service(HttpRequest request, HttpResponse response) throws Exception, IOException, GeneralSecurityException {
// Project ID for this request.
String project = "my-project"; // TODO: Update placeholder value.
// The name of the zone for this request.
String zone = "my-zone"; // TODO: Update placeholder value.
// Name of the instance resource to start.
String instance = "my-instance"; // TODO: Update placeholder value.
Compute computeService = createComputeService();
// you can change the method start to stop
Compute.Instances.Start request = computeService.instances().start(project, zone, instance);
Operation xresponse = xrequest.execute();
// TODO: Change code below to process the `response` object:
System.out.println(xresponse);
BufferedWriter writer = response.getWriter();
writer.write("Done");
}
public static Compute createComputeService() throws IOException, GeneralSecurityException {
HttpTransport httpTransport = GoogleNetHttpTransport.newTrustedTransport();
JsonFactory jsonFactory = JacksonFactory.getDefaultInstance();
GoogleCredential credential = GoogleCredential.getApplicationDefault();
if (credential.createScopedRequired()) {
credential =
credential.createScoped(Arrays.asList("https://www.googleapis.com/auth/cloud-platform"));
}
return new Compute.Builder(httpTransport, jsonFactory, credential)
.setApplicationName("Google-ComputeSample/0.1")
.build();
}
}
For more information you can check this document in this document are some examples in different programming languages

Related

Use the async gcloud.aio.storage to upload blob to new bucket

I'm using gcloud.aio.storage (with the fake-gcs-server emulator) and I've succesfully uploaded blobs to an existing bucket.
import os
import pytest
# Set ENV before Storage import, otherwise it gets set to prod gcs endpoint
os.environ["STORAGE_EMULATOR_HOST"] = "localhost:4443"
from gcloud.aio.storage import Storage
### WORKS OK #####
#pytest.mark.asyncio
async def test_upload_blob_to_existing_bucket() -> None:
async with Storage() as async_client:
existing_bucket_name: str = "fake_server_bucket"
await async_client.upload(existing_bucket_name, "myBlobName.json",
"blobContent")
pytest test_gcs_async_upload.py
======= 1 passed in 0.24s ======
How can I upload blobs to a new bucket ??
The below always returns me: aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found...' on the bucket
I looked at the test cases in the gcloud.aio.storage library itself, but they do not seem to do anything different from what I'm attemptying.
It's also missing the method to create_bucket(bucket) beforehand (like I could do in the synch google.cloud.storage library).
import os
import pytest
# Set ENV before Storage import, otherwise it gets set to prod gcs endpoint
os.environ["STORAGE_EMULATOR_HOST"] = "localhost:4443"
from gcloud.aio.storage import Storage
### Returns 404 notfound error ###
#pytest.mark.asyncio
async def test_upload_blob_to_new_bucket() -> None:
async with Storage() as async_client:
# async_client.create_bucket(bucket) #Cannot do this in gcloud.aio.storage lib!
new_bucket_name: str = "newbucket"
await async_client.upload(new_bucket_name, "myBlobName.json", "blobContent")
pytest test_gcs_async_upload.py
E aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found:
{"error":{"code":404,"message":"Not Found","errors":null}}\n',
url=URL('http://localhost:4443/upload/storage/v1/b/newbucket/o?
name=myBlobName.json&uploadType=media')
======= 1 failed in 0.20s ======
Thank you for the help :-)
SETUP:
python = ">=3.8,<3.11"
gcloud-aio-storage = "^7.0.1"
pytest = "^7.1.2"
pytest-asyncio = "^0.19.0"
fake-gcs-server is LATEST, TAG = 1 digest = 80e798b48cdd.
It runs as a docker instance and is prepopulated with a bucket and some blobs at startup

How can I combine methods from the EC2 resource and client API?

I'm trying to take in input for stopping and starting instances, but if I use client, it comes up with the error:
'EC2' has no attribute 'instance'
and if I use resource, it says
'ec2.Serviceresource' has no attribute 'Instance'
Is it possible to use both?
#!/usr/bin/env python3
import boto3
import botocore
import sys
print('Enter Instance id: ')
instanceIdIn=input()
ec2=boto3.resource('ec2')
ec2.Instance(instanceIdIn).stop()
stopwait=ec2.get_waiter('instance_stopped')
try:
stopwait.wait(instanceIdIn)
print('Instance Stopped. Starting Instance again.')
except botocore.exceptions.waitError as wex:
logger.error('instance not stopped')
ec2.Instance(instanceIdIn).start()
try:
logger.info('waiting for running state...')
print('Instance Running.')
except botocore.exceptions.waitError as wex2:
logger.error('instance has not been stopped')
In boto3 there's two kinds of APIs for most service - a resource-based API, which is supposed to be an abstraction of the lower level API calls that the client API provides.
You can't directly mix and match calls to these. Instead you should create a separate instance for each of those like this:
import boto3
ec2_resource = boto3.resource("ec2")
ec2_client = boto3.client("ec2")
# Now you can call the client methods on the client
# and resource classes from the resource:
my_instance = ec2_resource.Instance("instance-id")
my_waiter = ec2_client.get_waiter("instance_stopped")

How to set credentials to use Gmail API from GCE VM Command Line?

I'm trying to enable my Linux VM on GCE to access my Gmail account in order to send emails.
I've found this article, https://developers.google.com/gmail/api/quickstart/python, which reads some Gmail account information (it's useful since I just want to test my connection).
from __future__ import print_function
import pickle
import os.path
from googleapiclient.discovery import build
from google_auth_oauthlib.flow import InstalledAppFlow
from google.auth.transport.requests import Request
# If modifying these scopes, delete the file token.pickle.
SCOPES = ['https://www.googleapis.com/auth/gmail.readonly']
def main():
"""Shows basic usage of the Gmail API.
Lists the user's Gmail labels.
"""
creds = None
# The file token.pickle stores the user's access and refresh tokens, and is
# created automatically when the authorization flow completes for the first
# time.
if os.path.exists('token.pickle'):
with open('token.pickle', 'rb') as token:
creds = pickle.load(token)
# If there are no (valid) credentials available, let the user log in.
if not creds or not creds.valid:
if creds and creds.expired and creds.refresh_token:
creds.refresh(Request())
else:
flow = InstalledAppFlow.from_client_secrets_file(
'credentials.json', SCOPES)
creds = flow.run_local_server(port=0)
# Save the credentials for the next run
with open('token.pickle', 'wb') as token:
pickle.dump(creds, token)
service = build('gmail', 'v1', credentials=creds)
# Call the Gmail API
results = service.users().labels().list(userId='me').execute()
labels = results.get('labels', [])
if not labels:
print('No labels found.')
else:
print('Labels:')
for label in labels:
print(label['name'])
if __name__ == '__main__':
main()
However, I'm not sure which Credentials I need to set, as when I set and use:
Service Account: I receive the following message ValueError: Client secrets must be for a web or installed app.
OAuth client ID for Web Application type: the code runs well, however I receive the following message when trying to first authorize the application's access:
Erro 400: redirect_uri_mismatch
The redirect URI in the request, http://localhost:60443/, does not match the ones authorized for the OAuth client. To update the authorized redirect URIs, visit: https://console.developers.google.com/apis/credentials/oauthclient/${your_client_id}?project=${your_project_number}
OAuth client ID for Desktop type: the code runs well, however I receive the following message when trying to first authorize the application's access:
localhost connection has been refused
Does anyone know how is the correct setup and if I'm missing anything?
Thanks
[Nov17th]
After adding the gmail scope to my VM's scopes I ran the python script and I got the following error:
Traceback (most recent call last):
File "quickstart2.py", line 29, in <module>
main()
File "quickstart2.py", line 18, in main
results = service.users().labels().list(userId="107628233971924038182").execute()
File "/home/lucasnunesfe9/.local/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
return wrapped(*args, **kwargs)
File "/home/lucasnunesfe9/.local/lib/python3.7/site-packages/googleapiclient/http.py", line 915, in execute
raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 400 when requesting https://gmail.googleapis.com/gmail/v1/users/107628233971924038182/labels?alt=json returned "Precondition check failed.">
I checked the error HTTP link and it shows:
{
"error": {
"code": 401,
"message": "Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.",
"errors": [
{
"message": "Login Required.",
"domain": "global",
"reason": "required",
"location": "Authorization",
"locationType": "header"
}
],
"status": "UNAUTHENTICATED"
}
}
Is any manual procedure needed in a "first authorization step"?
PS: reinforcing that I have already enabled my Service Account to "G Suite Domain-wide Delegation". This action generated an OAuth 2.0 Client ID, which is being used in the python script (variable userId).
Personally, I never understood this example. I think it's too old (and even compliant Python 2.6!!)
Anyway, you can forget the first part and get a credential like this
from googleapiclient.discovery import build
import google.auth
# If modifying these scopes, delete the file token.pickle.
SCOPES = ['https://www.googleapis.com/auth/gmail.readonly']
def main():
creds, project_id = google.auth.default(scopes=SCOPES)
service = build('gmail', 'v1', credentials=creds)
# Call the Gmail API
results = service.users().labels().list(userId='me').execute()
labels = results.get('labels', [])
if not labels:
print('No labels found.')
else:
print('Labels:')
for label in labels:
print(label['name'])
if __name__ == '__main__':
main()
However, because you will use a service account to access to your GMAIL account, you have to change the userId with the ID that you want and to grand access from the GSuite admin console to the service account to have access to GMAIL API.
To achieve this, you need to grant the scope of your VM service account. Stop the VM, run this (long) command, and start your VM again. The command:
Takes the current scopes of your VM (and clean/format them)
Add the gmail scope
Set the scope to the VM (it's a BETA command)
gcloud beta compute instances set-scopes <YOUR_VM_NAME> --zone <YOUR_VM_ZONE> \
--scopes=https://www.googleapis.com/auth/gmail.readonly,\
$(gcloud compute instances describe <YOUR_VM_NAME> --zone <YOUR_VM_ZONE> \
--format json | jq -c ".serviceAccounts[0].scopes" | sed -E "s/\[(.*)\]/\1/g" | sed -E "s/\"//g")

how to use pageSize or pageToken in python code for GCP

this is the python code I got from github. running it, I got 300. But when I use gcloud to get role number, I got a total of 479 roles. I was told by the GCP support that pageSize needs to be used. where can I find documentation of how and pageSize can be used? so in my code below, where should pageSize go? or perhaps pageToken needs to be used?
(gcptest):$ gcloud iam roles list |grep name |wc -l
479
(gcptest) : $ python quickstart.py
300
def quickstart():
# [START iam_quickstart]
import os
from google.oauth2 import service_account
import googleapiclient.discovery
import pprint
# Get credentials
credentials = service_account.Credentials.from_service_account_file(
filename=os.environ['GOOGLE_APPLICATION_CREDENTIALS'],
scopes=['https://www.googleapis.com/auth/cloud-platform'])
# Create the Cloud IAM service object
service = googleapiclient.discovery.build(
'iam', 'v1', credentials=credentials)
# Call the Cloud IAM Roles API
# If using pylint, disable weak-typing warnings
# pylint: disable=no-member
response = service.roles().list().execute()
roles = response['roles']
print(type(roles))
print(len(roles))
if name == 'main':
quickstart()
You will need to write code similar to this:
roles = service.roles()
request = roles.list()
while request is not None:
role_list = request.execute()
# process each role here
for role in role_list:
print(role)
# Get next page of results
request = roles.list_next(request, role_list)
Documentation link for the list_next method
In addition of the solution of #JohnHanley, you can also add the queries parameters in parameter of your method. Like this
# Page size of 10
response = service.roles().list(pageSize=10).execute()
Here the definition of this list method

HTTP Deadline exceeded waiting for python Google Cloud Endpoints on python client localhost

I want to build a python client to talk to my python Google Cloud Endpoints API. My simple HelloWorld example is suffering from an HTTPException in the python client and I can't figure out why.
I've setup simple examples as suggested in this extremely helpful thread. The GAE Endpoints API is running on localhost:8080 with no problems - I can successfully access it in the API Explorer. Before I added the offending service = build() line, my simple client ran fine on localhost:8080.
When trying to get the client to talk to the endpoints API, I get the following error:
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/dist27/gae_override/httplib.py", line 526, in getresponse
raise HTTPException(str(e))
HTTPException: Deadline exceeded while waiting for HTTP response from URL: http://localhost:8080/_ah/api/discovery/v1/apis/helloworldendpoints/v1/rest?userIp=%3A%3A1
I've tried extending the http deadline. Not only did that not help, but such a simple first call on localhost should not be exceeding a default 5s deadline. I've also tried accessing the discovery URL directly within a browser and that works fine, too.
Here is my simple code. First the client, main.py:
import webapp2
import os
import httplib2
from apiclient.discovery import build
http = httplib2.Http()
# HTTPException happens on the following line:
# Note that I am using http, not https
service = build("helloworldendpoints", "v1", http=http,
discoveryServiceUrl=("http://localhost:8080/_ah/api/discovery/v1/apis/{api}/{apiVersion}/rest"))
# result = service.resource().method([parameters]).execute()
class MainPage(webapp2.RequestHandler):
def get(self):
self.response.headers['Content-type'] = 'text/plain'
self.response.out.write("Hey, this is working!")
app = webapp2.WSGIApplication(
[('/', MainPage)],
debug=True)
Here's the Hello World endpoint, helloworld.py:
"""Hello World API implemented using Google Cloud Endpoints.
Contains declarations of endpoint, endpoint methods,
as well as the ProtoRPC message class and container required
for endpoint method definition.
"""
import endpoints
from protorpc import messages
from protorpc import message_types
from protorpc import remote
# If the request contains path or querystring arguments,
# you cannot use a simple Message class.
# Instead, you must use a ResourceContainer class
REQUEST_CONTAINER = endpoints.ResourceContainer(
message_types.VoidMessage,
name=messages.StringField(1),
)
package = 'Hello'
class Hello(messages.Message):
"""String that stores a message."""
greeting = messages.StringField(1)
#endpoints.api(name='helloworldendpoints', version='v1')
class HelloWorldApi(remote.Service):
"""Helloworld API v1."""
#endpoints.method(message_types.VoidMessage, Hello,
path = "sayHello", http_method='GET', name = "sayHello")
def say_hello(self, request):
return Hello(greeting="Hello World")
#endpoints.method(REQUEST_CONTAINER, Hello,
path = "sayHelloByName", http_method='GET', name = "sayHelloByName")
def say_hello_by_name(self, request):
greet = "Hello {}".format(request.name)
return Hello(greeting=greet)
api = endpoints.api_server([HelloWorldApi])
Finally, here is my app.yaml file:
application: <<my web client id removed for stack overflow>>
version: 1
runtime: python27
api_version: 1
threadsafe: yes
handlers:
- url: /_ah/spi/.*
script: helloworld.api
secure: always
# catchall - must come last!
- url: /.*
script: main.app
secure: always
libraries:
- name: endpoints
version: latest
- name: webapp2
version: latest
Why am I getting an HTTP Deadline Exceeded and how to I fix it?
On your main.py you forgot to add some variables to your discovery service url string, or you just copied the code here without it. By the looks of it you were probably suppose to use the format string method.
"http://localhost:8080/_ah/api/discovery/v1/apis/{api}/{apiVersion}/rest".format(api='helloworldendpoints', apiVersion="v1")
By looking at the logs you'll probably see something like this:
INFO 2015-11-19 18:44:51,562 module.py:794] default: "GET /HTTP/1.1" 500 -
INFO 2015-11-19 18:44:51,595 module.py:794] default: "POST /_ah/spi/BackendService.getApiConfigs HTTP/1.1" 200 3109
INFO 2015-11-19 18:44:52,110 module.py:794] default: "GET /_ah/api/discovery/v1/apis/helloworldendpoints/v1/rest?userIp=127.0.0.1 HTTP/1.1" 200 3719
It's timing out first and then "working".
Move the service discovery request inside the request handler:
class MainPage(webapp2.RequestHandler):
def get(self):
service = build("helloworldendpoints", "v1",
http=http,
discoveryServiceUrl=("http://localhost:8080/_ah/api/discovery/v1/apis/{api}/{apiVersion}/rest")
.format(api='helloworldendpoints', apiVersion='v1'))