I have an application on django that can search for YouTube videos via the YouTube Data API v3 and translate their description using the Yandex Translate API. Often everything works fine, both services respond normally. But about once every 10 calls, the request stretches for about 20 seconds. This happens with both translation and YouTube.
I look in the developer console, in the column "Waiting (Time to receive the first byte)" just these 20 seconds. I do not understand why this is so, it is unlikely that the problem is on the side of the services, because it is observed on both. But I can't figure out what my problem is then...
I tried to set DEBUG to False, it was advised somewhere. But the problem has not gone away.
Code of functions for receiving data from YouTube:
from googleapiclient.discovery import build
from google_auth_oauthlib.flow import InstalledAppFlow
from google.auth.transport.requests import Request
import os
import pickle
SCOPES = ["https://www.googleapis.com/auth/youtube.force-ssl"]
def auth():
os.environ["OAUTHLIB_INSECURE_TRANSPORT"] = "1"
api_service_name = "youtube"
api_version = "v3"
credentials_filename = "credentials.json"
credentials = None
if os.path.exists("token.pickle"):
with open("token.pickle", "rb") as token:
credentials = pickle.load(token)
if not credentials or not credentials.valid:
if credentials and credentials.expired and credentials.refresh_token:
credentials.refresh(Request())
else:
flow = InstalledAppFlow.from_client_secrets_file(credentials_filename, SCOPES)
credentials = flow.run_local_server(port=8000)
with open("token.pickle", "wb") as token:
pickle.dump(credentials, token)
return build(api_service_name, api_version, credentials=credentials)
def search(youtube, **kwargs):
return youtube.search().list(part="snippet", **kwargs).execute()
Their call code:
yt = auth()
response = search(yt, q=request.POST['search'], maxResults=20, type='video')['items']
Function code for Yandex API:
endpoint_url = 'https://translate.api.cloud.yandex.net/translate/v2/translate'
list_texts = [str(texts[key]) for key in texts]
body = {
"targetLanguageCode": language,
"texts": list_texts,
"folderId": self.folder_id,
}
headers = {
"Content-Type": "application/json",
"Authorization": "Bearer {0}".format(self.token)
}
response = requests.post(endpoint_url, json=body, headers=headers)
Related
I am failing to send over my personal banking data via a flask webhook from the Nordigen API to Dialogflow via fulfilment as only null is being received within the Dialogflow payload:
{
"fulfillmentText": "Your text response",
"fulfillmentMessages": [
{
"payload": [
null
]
}
]
}
The webhook error message is: Webhook call failed. Error: Failed to parse webhook JSON response: Expect a map object but found: [null].
When I just send the data as a fulfillmentText I receive "fulfillmentText": null.
I have tested my webhook with postman and there - as well as locally and other webhook'esque tests - everything is fine as I receive my most recent banking data.
The overall flow is twofold and simple:
User gets the correct banking and user specific login link to a specified bank, copy & pastes it to perform banking login by query_text = 'login'.
After a successful banking login the user can fetch different kinds of banking data (like balance) by query_text = 'balance'.
I went crazy with overengineering the flask webhook as I tried out many different things like asynchronous functions, uploading my Flask app to Heroku or CORS. I have even implemented an OAuth2 process where the user would query_text = 'google auth' and initiate the OAuth2 process in a step 0) by creating OAuth2 credentials and the Flask-Dance Python package. (Even though I have hardcoded the OAuth2 redirect link but this shouldn't be an issue atm). I was even trying to trick Dialogflow by creating a small Sqlite3 db within my webhook to at least upload the data there and then use it but without success.
So my question is .. what am I missing here? Why do I receive my banking data everywhere else but not in Dialogflow. My intuition is telling me Google is blocking this data for whatever reason.
Honestly I just don't know how to continue and I would appreciate any helpful comments!
This is my Flask webhook:
from dialogflow_fulfillment import QuickReplies, WebhookClient, Payload
from flask import Flask, request, jsonify, make_response, session, render_template, redirect, url_for
from flask_cors import CORS, cross_origin
import json
from json import JSONEncoder
import os
import asyncio
import requests
import sqlite3
from app.src.handler_login import handler_login
from app.src.handler_balance import handler_balance
from app.banking_data.init_db import create_connection
from flask_dance.contrib.google import make_google_blueprint, google
from oauthlib.oauth2.rfc6749.errors import InvalidGrantError, TokenExpiredError, OAuth2Error
from google.cloud import dialogflow_v2beta1 as dialogflow
from google.oauth2 import service_account
from uuid import uuid4
from nordigen import NordigenClient
# NORDIGEN
# Credentials
secret_id="XXX"
secret_key="XXX"
# Configuration
institution_id = "XXX"
app = Flask(__name__)
# set Flask secret key
app.secret_key = os.environ.get("FLASK_SECRET_KEY", "supersekrit")
# GOOGLE API & AUTHENTICATION
app.config["GOOGLE_OAUTH_CLIENT_ID"] = "XXX"
app.config["GOOGLE_OAUTH_CLIENT_SECRET"] = "XXX"
os.environ['OAUTHLIB_INSECURE_TRANSPORT'] = "1"
os.environ['OAUTHLIB_RELAX_TOKEN_SCOPE'] = "1"
google_bp = make_google_blueprint(scope=["profile", "email"])
app.register_blueprint(google_bp, url_prefix="/login")
app.config['CORS_HEADERS'] = 'Content-Type'
cors = CORS(app, supports_credentials=True, resources={r"/webhook": {"origins": "*"}})
client = NordigenClient(
secret_id=secret_id,
secret_key=secret_key
)
client.generate_token()
# subclass JSONEncoder
class setEncoder(JSONEncoder):
def default(self, obj):
return list(obj)
#app.route("/")
def index():
if not google.authorized:
return redirect(url_for("google.login"))
try:
resp = google.get("/oauth2/v1/userinfo")
assert resp.ok, resp.text
return "You are {email} on Google".format(email=resp.json()["email"])
except (InvalidGrantError, TokenExpiredError) as e: # or maybe any OAuth2Error
return redirect(url_for("google.login"))
#app.route('/webhook', methods=['GET', 'POST', 'OPTION'])
async def webhook():
"""Handles webhook requests from Dialogflow."""
req = request.get_json(force=True)
query_text = req.get('queryResult').get('queryText')
if query_text:
if query_text == 'google auth':
if not google.authorized:
auth_link = 'MY HARD CODED GOOGLE AUTHENTICATION LINK HERE'
auth_link = {
"fulfillmentText": auth_link,
"source": 'webhook'
}
return auth_link
try:
resp = google.get("/oauth2/v1/userinfo")
assert resp.ok, resp.text
return "You are {email} on Google".format(email=resp.json()["email"])
except (InvalidGrantError, TokenExpiredError) as e: # or maybe any OAuth2Error
auth_link = 'MY HARD CODED GOOGLE AUTHENTICATION LINK HERE'
auth_link = {
"fulfillmentText": auth_link,
"source": 'webhook'
}
return auth_link
if query_text == 'login':
link = await handler_login(client, institution_id, session)
link = {
"fulfillmentText": link,
"source": 'webhook'
}
link = make_response(jsonify(link))
link.headers.add('Access-Control-Allow-Origin', '*')
return link
if query_text == 'balance':
balance = await handler_balance(client, session)
balance = {
"fulfillmentText": "Your text response",
"fulfillmentMessages": [
{
"text": {
"text": [
"Your text response"
]
}
},
{
"payload": {
balance
}
}
]
}
balance = json.dumps(balance, indent=4, cls=setEncoder)
balance = make_response(balance)
return balance
if __name__ == "__main__":
app.run(debug=True)
Here are two helper functions I have created that perform the creation of the login link the the fetching of my banking data via Nordigen:
from uuid import uuid4
async def handler_login(client, institution_id, session):
"""Handles the webhook request."""
# Initialize bank session
init = client.initialize_session(
# institution id
institution_id=institution_id,
# redirect url after successful authentication
redirect_uri="https://nordigen.com",
# additional layer of unique ID defined by you
reference_id=str(uuid4())
)
link = init.link
session["req_id"] = init.requisition_id
return link
async def handler_balance(client, session):
if "req_id" in session:
# Get account id after you have completed authorization with a bank
# requisition_id can be gathered from initialize_session response
#requisition_id = init.requisition_id
accounts = client.requisition.get_requisition_by_id(
requisition_id=session["req_id"]
)
# Get account id from the list.
account_id = accounts["accounts"][0]
#account_id = accounts["id"]
# Create account instance and provide your account id from previous step
account = client.account_api(id=account_id)
# Fetch account metadata
#meta_data = account.get_metadata()
# Fetch details
#details = account.get_details()
# Fetch balances
balance = account.get_balances()
balance = balance["balances"][0]
balance = balance["balanceAmount"]["amount"]
#balance = json.loads(balance)
# Fetch transactions
#transactions = account.get_transactions()
#agent.add(Payload({'balance': balance}))
return balance
Feel free to comment if you need any more input!
While trying to register my receiver endpoint in order to start receiving RISC indications from google, I constantly get the same reply:
403 Client Error: Forbidden for url:
https://risc.googleapis.com/v1beta/stream:update
I have created the service with the Editor Role and using the json key I created as requested on the integration guide.
This is my provisioning code I use to do that:
import json
import time
import jwt # pip install pyjwt
import requests
def make_bearer_token(credentials_file):
with open(credentials_file) as service_json:
service_account = json.load(service_json)
issuer = service_account['client_email']
subject = service_account['client_email']
private_key_id = service_account['private_key_id']
private_key = service_account['private_key']
issued_at = int(time.time())
expires_at = issued_at + 3600
payload = {'iss': issuer,
'sub': subject,
'aud': 'https://risc.googleapis.com/google.identity.risc.v1beta.RiscManagementService',
'iat': issued_at,
'exp': expires_at}
encoded = jwt.encode(payload, private_key, algorithm='RS256',
headers={'kid': private_key_id})
return encoded
def configure_event_stream(auth_token, receiver_endpoint, events_requested):
stream_update_endpoint = 'https://risc.googleapis.com/v1beta/stream:update'
headers = {'Authorization': 'Bearer {}'.format(auth_token)}
stream_cfg = {'delivery': {'delivery_method': 'https://schemas.openid.net/secevent/risc/delivery-method/push',
'url': receiver_endpoint},
'events_requested': events_requested}
response = requests.post(stream_update_endpoint, json=stream_cfg, headers=headers)
response.raise_for_status() # Raise exception for unsuccessful requests
def main():
auth_token = make_bearer_token('service_creds.json')
configure_event_stream(auth_token, 'https://MY-ENDPOINT.io',
['https://schemas.openid.net/secevent/risc/event-type/sessions-revoked',
'https://schemas.openid.net/secevent/oauth/event-type/tokens-revoked',
'https://schemas.openid.net/secevent/risc/event-type/account-disabled',
'https://schemas.openid.net/secevent/risc/event-type/account-enabled',
'https://schemas.openid.net/secevent/risc/event-type/account-purged',
'https://schemas.openid.net/secevent/risc/event-type/account-credential-change-required'])
if __name__ == "__main__":
main()
Also tested my auth token and it seems as the integration guide suggests.
Could not find 403 forbidden on the error code reference table there.
You can check for error description in the response body and match that against the possible reasons listed here!
I have a terminal that served in webbrowser with wetty. I want to authenticate the user from gitlab to let user with interaction with the terminal(It is inside docker container. When user authenticated i ll allow him to see the containers terminal).
I am trying to do OAuth 2.0 but couldn't manage to achieve.
That is what i tried.
I created an application on gitlab.
Get the code and secret and make a http call with python script.
Script directed me to login and authentication page.
I tried to get code but failed(Their is no mistake on code i think)
Now the problem starts in here. I need to get the auth code from redirected url to gain access token but couldn't figure out. I used flask library for get the code.
from flask import Flask, abort, request
from uuid import uuid4
import requests
import requests.auth
import urllib2
import urllib
CLIENT_ID = "clientid"
CLIENT_SECRET = "clientsecret"
REDIRECT_URI = "https://UnrelevantFromGitlabLink.com/console"
def user_agent():
raise NotImplementedError()
def base_headers():
return {"User-Agent": user_agent()}
app = Flask(__name__)
#app.route('/')
def homepage():
text = 'Authenticate with gitlab'
return text % make_authorization_url()
def make_authorization_url():
# Generate a random string for the state parameter
# Save it for use later to prevent xsrf attacks
state = str(uuid4())
save_created_state(state)
params = {"client_id": CLIENT_ID,
"response_type": "code",
"state": state,
"redirect_uri": REDIRECT_URI,
"scope": "api"}
url = "https://GitlapDomain/oauth/authorize?" + urllib.urlencode(params)
print get_redirected_url(url)
print(url)
return url
# Left as an exercise to the reader.
# You may want to store valid states in a database or memcache.
def save_created_state(state):
pass
def is_valid_state(state):
return True
#app.route('/console')
def reddit_callback():
print("-----------------")
error = request.args.get('error', '')
if error:
return "Error: " + error
state = request.args.get('state', '')
if not is_valid_state(state):
# Uh-oh, this request wasn't started by us!
abort(403)
code = request.args.get('code')
print(code.json())
access_token = get_token(code)
# Note: In most cases, you'll want to store the access token, in, say,
# a session for use in other parts of your web app.
return "Your gitlab username is: %s" % get_username(access_token)
def get_token(code):
client_auth = requests.auth.HTTPBasicAuth(CLIENT_ID, CLIENT_SECRET)
post_data = {"grant_type": "authorization_code",
"code": code,
"redirect_uri": REDIRECT_URI}
headers = base_headers()
response = requests.post("https://MyGitlabDomain/oauth/token",
auth=client_auth,
headers=headers,
data=post_data)
token_json = response.json()
return token_json["access_token"]
if __name__ == '__main__':
app.run(host="0.0.0.0",debug=True, port=65010)
I think my problem is on my redirect url. Because it is just an irrelevant link from GitLab and there is no API the I can make call.
If I can fire
#app.route('/console')
that line on Python my problem will probably will be solved.
I need to make correction on my Python script or different angle to solve my problem. Please help.
I was totally miss understand the concept of auth2. Main aim is to have access_token. When i corrected callback url as localhost it worked like charm.
I am aware of the HTTP Data Collector API that can be used to pull data into Azure Log analytics, my ask here is on AWS Cloudwatch data to Azure. We have Azure hosted application and an external AWS hosted Serverless Lamda functions and we want to import the logs of those 13 serverless functions into Azure. I know from the documentation and there is a python function that can be used as a AWS Lamda function and the python example is in MSFT documentation. But what I am failing to understand is what Json format that AWS cloud collector needs to create so they can send it to Azure Log Analytics. Any examples on this ? Any help on how this can be done. I have come across this blog also but that is splunk specific. https://www.splunk.com/blog/2017/02/03/how-to-easily-stream-aws-cloudwatch-logs-to-splunk.html
Hey never mind I was able to dig a little deeper and I found that in AWS I can STREAM the Logs from one Lambda to other Lambda function thru subscription. Once that was setthen all I did was consumed that and on the fly created the JSON and sent it to Azure Logs. In case if you or anyone is interested in it, following is the code:-
import json
import datetime
import hashlib
import hmac
import base64
import boto3
import datetime
import gzip
from botocore.vendored import requests
from datetime import datetime
Update the customer ID to your Log Analytics workspace ID
customer_id = "XXXXXXXYYYYYYYYYYYYZZZZZZZZZZ"
For the shared key, use either the primary or the secondary Connected Sources client authentication key
shared_key = "XXXXXXXXXXXXXXXXXXXXXXXXXX"
The log type is the name of the event that is being submitted
log_type = 'AWSLambdafuncLogReal'
json_data = [{
"slot_ID": 12345,
"ID": "5cdad72f-c848-4df0-8aaa-ffe033e75d57",
"availability_Value": 100,
"performance_Value": 6.954,
"measurement_Name": "last_one_hour",
"duration": 3600,
"warning_Threshold": 0,
"critical_Threshold": 0,
"IsActive": "true"
},
{
"slot_ID": 67890,
"ID": "b6bee458-fb65-492e-996d-61c4d7fbb942",
"availability_Value": 100,
"performance_Value": 3.379,
"measurement_Name": "last_one_hour",
"duration": 3600,
"warning_Threshold": 0,
"critical_Threshold": 0,
"IsActive": "false"
}]
#body = json.dumps(json_data)
#####################
######Functions######
#####################
Build the API signature
def build_signature(customer_id, shared_key, date, content_length, method, content_type, resource):
x_headers = 'x-ms-date:' + date
string_to_hash = method + "\n" + str(content_length) + "\n" + content_type + "\n" + x_headers + "\n" + resource
bytes_to_hash = bytes(string_to_hash, encoding="utf-8")
decoded_key = base64.b64decode(shared_key)
encoded_hash = base64.b64encode(
hmac.new(decoded_key, bytes_to_hash, digestmod=hashlib.sha256).digest()).decode()
authorization = "SharedKey {}:{}".format(customer_id,encoded_hash)
return authorization
Build and send a request to the POST API
def post_data(customer_id, shared_key, body, log_type):
method = 'POST'
content_type = 'application/json'
resource = '/api/logs'
rfc1123date = datetime.utcnow().strftime('%a, %d %b %Y %H:%M:%S GMT')
print (rfc1123date)
content_length = len(body)
signature = build_signature(customer_id, shared_key, rfc1123date, content_length, method, content_type, resource)
uri = 'https://' + customer_id + '.ods.opinsights.azure.com' + resource + '?api-version=2016-04-01'
headers = {
'content-type': content_type,
'Authorization': signature,
'Log-Type': log_type,
'x-ms-date': rfc1123date
}
response = requests.post(uri,data=body, headers=headers)
if (response.status_code >= 200 and response.status_code <= 299):
print("Accepted")
else:
print("Response code: {}".format(response.status_code))
print(response.text)
def lambda_handler(event, context):
cloudwatch_event = event["awslogs"]["data"]
decode_base64 = base64.b64decode(cloudwatch_event)
decompress_data = gzip.decompress(decode_base64)
log_data = json.loads(decompress_data)
print(log_data)
awslogdata = json.dumps(log_data)
post_data(customer_id, shared_key, awslogdata, log_type)
I am trying to take some working code and change from urlib2 to requests.
The original code provides basic login information of username, password and posts the KEY and SECRET in the header of the urllib2 request. The following code is my attempt to change to using the requests module and gain some functionality for making additional API calls. I have tried dozens of combinations and all return a code 400. Apparently, my requests code does not successfully furnish the needed information to return a 200 response and provide the needed authorization token.
## Import needed modules
import urllib2, urllib, base64
import httplib
import requests
import json
## initialize variables
KEY = "7f1xxxx-3xxx-4xxx-9a7f-8be66839dede"
SECRET = "45xxxxxx-45xxx-469a-9ae9-a7927a76cfeb"
userName = "my-email#xxx.com"
passWord = "mypassword"
URL = "https://company.com/auth/token"
token = None
sessionid = None
DATA = urllib.urlencode({"grant_type":"password",
"username":userName,
"password":passWord})
base64string = base64.encodestring('%s:%s' % (KEY, SECRET)).replace('\n', '')
request = urllib2.Request(URL, DATA)
request.add_header("Authorization", "Basic %s" % base64string)
result = urllib2.urlopen(request)
token = result.read()
print token
This returns my authorization token and all is well. I can pass the token to the authorization server and have full access to the api for interacting with the database. Below is the attempt to use requests and have the added functions it provides.
client = requests.session()
payload = {"grant_type":"password",
"username":userName,
"password":passWord,
"applicationId": KEY
}
headers = {'content-type':'application/json',
"grant_type":"password",
"username":userName,
"password":passWord,
'applicationsId': KEY,
'Authorization': base64string,
'token': token,
'sessionid': sessionid
}
response = client.post(URL, params = payload, headers=headers)
token = response.content
print token
{"error":"invalid_request"}
print response
<Response [400]>
If you want to use basic auth you should use the method from requests..
Your post should look like
response = client.post(
URL,
params = payload,
headers=headers,
auth=HTTPBasicAuth(
KEY,
SECRET
))
Somewhere in a post a contributor to another question mentioned some items actually needed to be in the body of the request not in the header. I tried various combos and the following solved the 400 response and accomplished my goals.
data = {"grant_type":"password",
"username":userName,
"password":passWord,
"applicationId": KEY
}
headers = {'Authorization': "Basic %s" % base64string,
'token': token
}
response = client.post(URL, data = data, headers=headers)
token = response.text
print token