I am trying to run a scheduled query only when a table updates in big query. For this I am trying to make this python code to work in cloud functions, but it is giving me error. Would highly appreciate any help.
I am running this python code :
import time
from google.protobuf.timestamp_pb2 import Timestamp
from google.cloud import bigquery_datatransfer_v1
def runQuery (parent,requested_run_time):
client = bigquery_datatransfer_v1.DataTransferServiceClient()
projectid = '629586xxxx' # Enter your projectID here
transferid = '60cc15f8-xxxx-xxxx-8ba2-xxxxx41bc' # Enter your transferId here
parent = client.transfer_config_path(projectid, transferid)
start_time = Timestamp(seconds=int(time.time() + 10))
response = client.start_manual_transfer_runs(parent, requested_run_time=start_time)
print(response)
I get this error
start_manual_transfer_runs() got an unexpected keyword argument 'requested_run_time'
I created doing this using cloud function and it worked for me.
import time
from google.protobuf.timestamp_pb2 import Timestamp
from google.cloud import bigquery_datatransfer_v1
def runQuery(parent, requested_run_time):
client = bigquery_datatransfer_v1.DataTransferServiceClient()
PROJECT_ID = 'graphical-reach-285218'
TRANSFER_CONFIG_ID = '61adfc39-0000-206b-a7b0-089e08324288'
parent = client.project_transfer_config_path(PROJECT_ID, TRANSFER_CONFIG_ID)
start_time = bigquery_datatransfer_v1.types.Timestamp(seconds=int(time.time() + 10))
response = client.start_manual_transfer_runs(parent, requested_run_time=start_time)
print(parent)
print(response)
Replace with your project id and transfer config id. The above code will go in main.py and in requirements.txt , please keep
google-cloud-bigquery-datatransfer==1
Related
i want to run a schedule query when HTTP cloud function is triggered
This written python 3.7
import time from google.protobuf.timestamp_pb2
import Timestamp from google.cloud
import bigquery_datatransfer_v1
def runQuery (parent, requested_run_time):
client = bigquery_datatransfer_v1.DataTransferServiceClient()
projectid = '917960740952' # Enter your projectID here
transferid = '630d5269-0000-2622-98d2-3c286d4314be'
parent = client.project_transfer_config_path(projectid, transferid)
start_time = bigquery_datatransfer_v1.types.Timestamp(seconds=int(time.time() + 10))
response = client.start_manual_transfer_runs(parent, requested_run_time=start_time)
print(response)
Any help Please
I changed the parent path to be and worked with me
parent = 'projects/917960740952/locations/europe/transferConfigs/630d5269-0000-2622-98d2-3c286d4314be'
from google.cloud import bigquery_datatransfer_v1
import time
from google.protobuf.timestamp_pb2 import Timestamp
def runQuery(request):
client = bigquery_datatransfer_v1.DataTransferServiceClient()
PROJECT_ID = 'YOUR_PROJECT_NAME'
LOCATION_NAME = 'YOUR_TRANSFER_REGION'
TRANSFER_CONFIG_ID = 'YOUR_TRANSFER_ID'
parent = "projects/{0}/locations/{1}/transferConfigs/{2}".format(PROJECT_ID, LOCATION_NAME, TRANSFER_CONFIG_ID)
start_time = bigquery_datatransfer_v1.types.Timestamp(seconds=int(time.time() + 10))
response = client.start_manual_transfer_runs(parent, requested_run_time=start_time)
print(response)
I'm currently building an app which contains email sending to multiple users which i'm able to do but i want to add a functionality which schedule's an email, for instance I'm using sent_at method as you can see below:-
settings.py
EMAIL_FROM = 'EMAIL'
EMAIL_API_CLIENT ='XXXXXXXX'
views.py
import json
from sendgrid import SendGridAPIClient
from django.conf import settings
message = Mail(from_email=settings.EMAIL_FROM,
to_emails=selectedphone,
subject=subject,
html_content=editor)
message.extra_headers = {'X-SMTPAPI': json.dumps({'send_at':
FinalScheduleTime})}
sg = SendGridAPIClient(settings.EMAIL_API_CLIENT)
response = sg.send(message)
if response.status_code == 202:
emailstatus = "Accepted"
elif .....
else.....
I've also tried message.extra_headers = {'SendAt':FinalScheduleTime} but it's not working either.
Here the FinalScheduleTime is of the datetime object. The sendgrip api accepts the UNIX timestamp according to the documentation. You can check it here
Hence to convert your datetime object into unix time stamp, you can use the time module of python.
scheduleTime = int(time.mktime(FinalScheduleTime.timetuple()))
Also, replace the message.extra_headers with message.send_at.
Hence, your final code will look like:
import json
import time
from sendgrid import SendGridAPIClient
from django.conf import settings
message = Mail(from_email=settings.EMAIL_FROM,
to_emails=selectedphone,
subject=subject,
html_content=editor)
scheduleTime = int(time.mktime(FinalScheduleTime.timetuple()))
message.send_at = scheduleTime
sg = SendGridAPIClient(settings.EMAIL_API_CLIENT)
response = sg.send(message)
if response.status_code == 202:
emailstatus = "Accepted"
elif .....
else.....
This is an official blog by Twilio on Using Twilio SendGrid To Send Emails from Python Django Applications - https://www.twilio.com/blog/scheduled-emails-python-flask-twilio-sendgrid
Also here are, official docs
After writing the following code and expecting the output to be an updated database with random names, websites, etc., I get no error message and no updated database
import os
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'First_project.settings')
import django
django.setup()
import random
from first_app.models import AccessRecord,Webpage,Topic
from faker import Faker
fakegen = Faker()
topics = ['Search','Social','Marketplace','News','Games']
def add_topic():
t = Topic.object.get_or_create(top_name=random.choice(topics))[0]
t.save()
return t
def populate(N=5):
for entry in range(N):
top = add_topic()
fake_url = fakegen.url()
fake_date = fakegen.date()
fake_name = fakegen.company()
webpg = webpage.objects.get_or_create(topic=top, url=fake_ur, name=fake_name)[0]
acc_rec = AccessRecord.object.get_or_create(name=webpg,date=fake_date)[0]
if __name__ == ' __main__':
print("populate")
populate(20)
print("populating complete!")
please what do I do?
i ran the whole code in my terminal and it populated the database
Im trying to integrate the Zoho CRM v2 SDK with my Django app.
On the Django runserver, im able to get access tokens and using the refresh method and store them in the zcrm_oauthtokens.pkl file. The sdk then automatically refreshes the access token using the refresh token, so no problem here. However on my production server (heroku) im getting this error message:
2019-01-16T11:07:22.314759+00:00 app[web.1]: 2019-01-16 11:07:22,314 - Client_Library_OAUTH - ERROR - Exception occured while fetching oauthtoken from db; Exception Message::'NoneType' object has no attribute 'accessToken'
It seems to me that the tokens are being saved to file, but when the sdk try to access them it is looking for them in a DB and not the file specified in the token_persistence_path.
In my settings.py I have this:
ZOHO_CLIENT_ID = config('ZOHO_CLIENT_ID')
ZOHO_CLIENT_SECRET = config('ZOHO_CLIENT_SECRET')
ZOHO_REDIRECT_URI = config('ZOHO_REDIRECT_URI')
ZOHO_CURRENT_USER_EMAIL = 'jamesalexander#mylastwill.co.uk'
ZOHO_PATH = os.path.join(BASE_DIR, 'wills_online', 'zoho')
zoho_config = {'apiBaseUrl': "https://www.zohoapis.com",
'currentUserEmail': ZOHO_CURRENT_USER_EMAIL,
'client_id': ZOHO_CLIENT_ID,
'client_secret': ZOHO_CLIENT_SECRET,
'redirect_uri': ZOHO_REDIRECT_URI,
'token_persistence_path': ZOHO_PATH}
and in a views file I have this:
from zcrmsdk import *
import logging
from django.shortcuts import HttpResponse
from wills.models import PersonalDetails, ZoHoRecord, WillDocument
from wills_online.decorators import start_new_thread
from wills_online.settings import zoho_config
logger = logging.getLogger(__name__)
class ZohoRunOnce:
def __init__(self):
self.already_run = False
def run_once(self):
if not self.already_run:
print('zoho init run once')
ZCRMRestClient.initialize(zoho_config)
self.already_run = True
zoho_init = ZohoRunOnce()
zoho_init.run_once()
print(zoho_config['token_persistence_path'])
def zoho_callback():
return HttpResponse(200)
#start_new_thread
def zoho_personal_details(request):
""" updates or create a user account on zoho on profile completion """
personal_details_ob = PersonalDetails.objects.get(user=request.user)
zoho_ob = ZoHoRecord.objects.get(user=request.user)
try:
if zoho_ob.account:
record = ZCRMRecord.get_instance('Accounts', zoho_ob.account)
record.set_field_value('Account_Name', request.user.email)
record.set_field_value('Name', personal_details_ob.full_name)
record.set_field_value('Email', request.user.email)
record.set_field_value('Address_Line_1', personal_details_ob.address_line_1)
record.set_field_value('Address_Line_2', personal_details_ob.address_line_2)
record.set_field_value('Post_Town', personal_details_ob.post_town)
record.set_field_value('Post_Code', personal_details_ob.post_code)
record.set_field_value('Dob_Day', personal_details_ob.dob_day)
record.set_field_value('Dob_Month', personal_details_ob.dob_month)
record.set_field_value('Dob_Year', personal_details_ob.dob_year)
record.set_field_value('Gender', personal_details_ob.sex)
record.set_field_value('Marital_Status', personal_details_ob.marital_status)
record.set_field_value('Partner_Name', personal_details_ob.partner_full_name)
record.set_field_value('Partner_Gender', personal_details_ob.partner_gender)
record.set_field_value('Partner_Email', personal_details_ob.partner_email)
record.set_field_value('Children', personal_details_ob.children)
record.set_field_value('Pets', personal_details_ob.pets)
record.update()
else:
user = ZCRMUser.get_instance(name='James Alexander')
record = ZCRMRecord.get_instance('Accounts')
record.set_field_value('Account_Owner', user)
record.set_field_value('Account_Name', request.user.email)
record.set_field_value('Name', personal_details_ob.full_name)
record.set_field_value('Email', request.user.email)
record.set_field_value('Address_Line_1', personal_details_ob.address_line_1)
record.set_field_value('Address_Line_2', personal_details_ob.address_line_2)
record.set_field_value('Post_Town', personal_details_ob.post_town)
record.set_field_value('Post_Code', personal_details_ob.post_code)
record.set_field_value('Dob_Day', personal_details_ob.dob_day)
record.set_field_value('Dob_Month', personal_details_ob.dob_month)
record.set_field_value('Dob_Year', personal_details_ob.dob_year)
record.set_field_value('Gender', personal_details_ob.sex)
record.set_field_value('Marital_Status', personal_details_ob.marital_status)
record.set_field_value('Partner_Name', personal_details_ob.partner_full_name)
record.set_field_value('Partner_Gender', personal_details_ob.partner_gender)
record.set_field_value('Partner_Email', personal_details_ob.partner_email)
record.set_field_value('Children', personal_details_ob.children)
record.set_field_value('Pets', personal_details_ob.pets)
response = record.create()
# save account id to db for future updates
zoho_ob.account = response.details['id']
zoho_ob.save()
except ZCRMException as ex:
logger.log(1, ex.status_code)
logger.log(1, ex.error_message)
logger.log(1, ex.error_details)
logger.log(1, ex.error_content)
print(ex.status_code)
print(ex.error_message)
print(ex.error_content)
print(ex.error_details)
Ive tried running ZCRMRestClient.initialize(zoho_config) in settings.py, with no luck.
My method for getting the access token and refresh token, which seems to work is:
import os
import pprint
from sys import argv
import django
import requests
import zcrmsdk
from django.conf import settings
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'wills_online.settings')
django.setup()
def zoho_refresh_token(code):
""" supply a self client token from the zoho api credentials from web site """
zoho_config = {"apiBaseUrl": "https://www.zohoapis.com",
"currentUserEmail": settings.ZOHO_CURRENT_USER_EMAIL,
"client_id": settings.ZOHO_CLIENT_ID,
"client_secret": settings.ZOHO_CLIENT_SECRET,
"redirect_uri": settings.ZOHO_REDIRECT_URI,
"token_persistence_path": settings.ZOHO_PATH}
pprint.pprint(zoho_config)
print('working')
address = f'https://accounts.zoho.com/oauth/v2/token?code={code}&redirect_uri={settings.ZOHO_REDIRECT_URI}&client_id={settings.ZOHO_CLIENT_ID}&client_secret={settings.ZOHO_CLIENT_SECRET}&grant_type=authorization_code'
response = requests.post(address)
data = response.json()
pprint.pprint(data)
zcrmsdk.ZCRMRestClient.initialize(zoho_config)
oauth_client = zcrmsdk.ZohoOAuth.get_client_instance()
refresh_token = data['refresh_token']
print(type(refresh_token))
oauth_client.generate_access_token_from_refresh_token(refresh_token, settings.ZOHO_CURRENT_USER_EMAIL)
print(refresh_token)
print('finished')
if name == 'main':
zoho_refresh_token(argv[1])
This is driving me mad. Help would be greatly appreciated. This is my first post so go easy, lol.
For future reference, you will need to define persistence_handler_class and persistence_handler_path in your configuration dictionary. You will also need a handler class and a user-defined model to store the results. Sample code follows:
# settings.py
import zcrmsdk
configuration_dictionary = {
'apiBaseUrl': 'https://www.zohoapis.com',
'apiVersion': 'v2',
'currentUserEmail': ZOHO_CURRENT_USER_EMAIL,
'sandbox': 'False',
'applicationLogFilePath': '',
'client_id': ZOHO_CLIENT_ID,
'client_secret': ZOHO_CLIENT_SECRET,
'redirect_uri': ZOHO_REDIRECT_URI,
'accounts_url': 'https://accounts.zoho.com',
'access_type': 'online',
'persistence_handler_class': ZOHO_HANDLER_CLASS,
'persistence_handler_path': ZOHO_HANDLER_PATH,
}
zcrmsdk.ZCRMRestClient.initialize(configuration_dictionary)
# zoho.models.py
from django.db import models
from zcrmsdk.OAuthClient import ZohoOAuthTokens
class ZohoOAuthHandler:
#staticmethod
def get_oauthtokens(email_address):
oauth_model_instance = ZohoOAuth.objects.get(user_email=email_address)
return ZohoOAuthTokens(oauth_model_instance.refresh_token,
oauth_model_instance.access_token,
oauth_model_instance.expiry_time,
user_email=oauth_model_instance.user_email)
#staticmethod
def save_oauthtokens(oauth_token):
defaults = {
'refresh_token': oauth_token.refreshToken,
'access_token': oauth_token.accessToken,
'expiry_time': oauth_token.expiryTime,
}
ZohoOAuth.objects.update_or_create(user_email=oauth_token.userEmail, defaults=defaults)
class ZohoOAuth(models.Model):
refresh_token = models.CharField(max_length=250)
access_token = models.CharField(max_length=250)
expiry_time = models.BigIntegerField()
user_email = models.EmailField()
In this example ZOHO_HANDLER_CLASS = 'ZohoOAuthHandler' and ZOHO_HANDLER_PATH = 'zoho.models'
The first time you go to use this you will need a grant_token from https://accounts.zoho.com/developerconsole. For the scope use aaaserver.profile.READ,ZohoCRM.modules.ALL to start (see https://www.zoho.com/crm/developer/docs/api/oauth-overview.html#scopes)
Before you can use the api you'll need to run the code below in a django shell. This uses a grant token to generate your initial access and refresh tokens. Afterwards, the api should handle refreshing your access token.
grant_token = GRANT_TOKEN
import zcrmsdk
oauth_client = zcrmsdk.ZohoOAuth.get_client_instance()
oauth_tokens = oauth_client.generate_access_token(grant_token)
I'm struggling with Django, Celery, djcelery & PeriodicTasks.
I've created a task to pull a report for Adsense to generate a live stat report. Here is my task:
import datetime
import httplib2
import logging
from apiclient.discovery import build
from celery.task import PeriodicTask
from django.contrib.auth.models import User
from oauth2client.django_orm import Storage
from .models import Credential, Revenue
logger = logging.getLogger(__name__)
class GetReportTask(PeriodicTask):
run_every = datetime.timedelta(minutes=2)
def run(self, *args, **kwargs):
scraper = Scraper()
scraper.get_report()
class Scraper(object):
TODAY = datetime.date.today()
YESTERDAY = TODAY - datetime.timedelta(days=1)
def get_report(self, start_date=YESTERDAY, end_date=TODAY):
logger.info('Scraping Adsense report from {0} to {1}.'.format(
start_date, end_date))
user = User.objects.get(pk=1)
storage = Storage(Credential, 'id', user, 'credential')
credential = storage.get()
if not credential is None and credential.invalid is False:
http = httplib2.Http()
http = credential.authorize(http)
service = build('adsense', 'v1.2', http=http)
reports = service.reports()
report = reports.generate(
startDate=start_date.strftime('%Y-%m-%d'),
endDate=end_date.strftime('%Y-%m-%d'),
dimension='DATE',
metric='EARNINGS',
)
data = report.execute()
for row in data['rows']:
date = row[0]
revenue = row[1]
try:
record = Revenue.objects.get(date=date)
except Revenue.DoesNotExist:
record = Revenue()
record.date = date
record.revenue = revenue
record.save()
else:
logger.error('Invalid Adsense Credentials')
I'm using Celery & RabbitMQ. Here are my settings:
# Celery/RabbitMQ
BROKER_HOST = "localhost"
BROKER_PORT = 5672
BROKER_USER = "myuser"
BROKER_PASSWORD = "****"
BROKER_VHOST = "myvhost"
CELERYD_CONCURRENCY = 1
CELERYD_NODES = "w1"
CELERY_RESULT_BACKEND = "amqp"
CELERY_TIMEZONE = 'America/Denver'
CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'
import djcelery
djcelery.setup_loader()
On first glance everything seems to work, but after turning on the logger and watching it run I have found that it is running the task at least four times in a row - sometimes more. It also seems to be running every minute instead of every two minutes. I've tried changing the run_every to use a crontab but I get the same results.
I'm starting celerybeat using supervisor. Here is the command I use:
python manage.py celeryd -B -E -c 1
Any ideas as to why its not working as expected?
Oh, and one more thing, after the day changes, it continues to use the date range it first ran with. So as days progress it continues to get stats for the day the task started running - unless I run the task manually at some point then it changes to the date I last ran it manually. Can someone tell me why this happens?
Consider creating a separate queue with one worker process and fixed rate for this type of tasks and just add the tasks in this new queue instead of running them in directly from celerybeat. I hope that could help you to figure out what is wrong with your code, is it problem with celerybeat or your tasks are running longer than expected.
#task(queue='create_report', rate_limit='0.5/m')
def create_report():
scraper = Scraper()
scraper.get_report()
class GetReportTask(PeriodicTask):
run_every = datetime.timedelta(minutes=2)
def run(self, *args, **kwargs):
create_report.delay()
in settings.py
CELERY_ROUTES = {
'myapp.tasks.create_report': {'queue': 'create_report'},
}
start additional celery worker with that would handle tasks in your queue
celery worker -c 1 -Q create_report -n create_report.local
Problem 2. Your YESTERDAY and TODAY variables are set at class level, so within one thread they are set only once.