I'm writing a fairly simply photo app using django-rest-framework for the API and django-storages for the storage engine. The front end is being written in Vue.js. I have the uploading part working, and now I'm trying to serve up the photos. As now seems obvious when the browser tries to load the images from GCS, I just get a bunch of 403 Forbidden errors. I did some reading up on this and it seems that the best practice in my case would be to generate signed urls that expire in some amount of time. I haven't been able to find a package for this, which is what I was hoping for. Short of that, it's not clear to me precisely how to do this in Django.
This is a working code in django 1.11 with python3.5.
import os
from google.oauth2 import service_account
from google.cloud import storage
class CloudStorageURLSigner(object):
#staticmethod
def get_video_signed_url(bucket_name, file_path):
creds = service_account.Credentials.from_service_account_file(
os.environ.get('GOOGLE_APPLICATION_CREDENTIALS')
)
bucket = storage.Client().get_bucket(bucket_name)
blob = bucket.blob(file_path)
signed_url = blob.generate_signed_url(
method='PUT',
expiration=1545367030, #epoch time
content_type='audio/mpeg', #change_accordingly
credentials=creds
)
return signed_url
Yes, take a look at google-cloud-storage
Installation:
pip install google-cloud-storage
Also, make sure to refer to API Documentation as you need more things.
Hope it helps!
I ended up solving this problem by using to_representation in serializers.py:
from google.cloud.storage import Blob
client = storage.Client()
bucket = client.get_bucket('myBucket')
def to_representation(self, value):
try:
blob = Blob(name=value.name, bucket=bucket)
signed_url = blob.generate_signed_url(expiration=datetime.timedelta(minutes=5))
return signed_url
except ValueError as e:
print(e)
return value
Extending #Evan Zamir's answer, instead of reassigning client and bucket you can get them from Django's default_storage (this will save time since these are already available).
This is in settings.py
from datetime import timedelta
from google.oauth2 import service_account
GS_CREDENTIALS = service_account.Credentials.from_service_account_file('credentials.json')
DEFAULT_FILE_STORAGE = "storages.backends.gcloud.GoogleCloudStorage"
GS_BUCKET_NAME = "my-bucket"
GS_EXPIRATION = timedelta(seconds=60)
In serializers.py
from django.core.files.storage import default_storage
from google.cloud.storage import Blob
from rest_framework import serializers
class SignedURLField(serializers.FileField):
def to_representation(self, value):
try:
blob = Blob(name=value.name, bucket=default_storage.bucket)
signed_url = blob.generate_signed_url(expiration=default_storage.expiration)
return signed_url
except ValueError as e:
print(e)
return value
You can use this class in your serializer like this,
class MyModelSerializer(serializers.ModelSerializer):
file = SignedURLField()
Note: Do not provide GS_DEFAULT_ACL = 'publicRead' if you want signed URLs as it creates public URLs (that do not expire)
Related
I'm trying integrate Google Tag Manager into my project. In official document, Google suggests oauth2client library. Unfortunately, this library deprecated. I used google_auth_oauthlib. I can get token and send request to Google Tag Manager API with this token. But I don't decide how can I store credentials object in Django. In oauth2client lib, there is CredentialsField for model. We can store Credentials object in this field with using DjangoORMStorage. But I can't use this deprecated library. Are there any alternative ways?
Google Tag Manager document is here
My codes here:
from google_auth_oauthlib.flow import Flow
from googleapiclient.discovery import build
FLOW = Flow.from_client_secrets_file(
settings.GOOGLE_OAUTH2_CLIENT_SECRETS_JSON,
scopes=[settings.GOOGLE_OAUTH2_SCOPE])
class GTMAuthenticateView(APIView):
def get(self,request,**kwargs):
FLOW.redirect_uri = settings.GOOGLE_OAUTH2_REDIRECT_URI
authorization_url, state = FLOW.authorization_url(access_type='offline',
include_granted_scopes='true')
return Response({'authorization_url':authorization_url,'state':state})
def post(self, request, **kwargs):
FLOW.fetch_token(code=request.data['code'])
credentials = FLOW.credentials
return Response({'token':credentials.token})
class GTMContainerView(APIView):
def get(self,request,**kwargs):
service = get_service()
containers = self.get_containers(service)
return Response({'containers':str(containers),'credentials':str(FLOW.credentials)})
#staticmethod
def get_containers(service):
account_list = service.accounts().list().execute()
container_list = []
for account in account_list['account']:
containers = service.accounts().containers().list(parent=account["path"]).execute()
for container in containers["container"]:
container["usageContext"] = container["usageContext"][0].replace("['", "").replace("']", "")
container_list.append(container)
return container_list
def get_service():
try:
credentials = FLOW.credentials
service = build('tagmanager', 'v2', credentials=credentials)
return service
except Exception as ex:
print(ex)
Just switch over to Django 2
pip install Django==2.2.12
I did it and works fine
Im trying to integrate the Zoho CRM v2 SDK with my Django app.
On the Django runserver, im able to get access tokens and using the refresh method and store them in the zcrm_oauthtokens.pkl file. The sdk then automatically refreshes the access token using the refresh token, so no problem here. However on my production server (heroku) im getting this error message:
2019-01-16T11:07:22.314759+00:00 app[web.1]: 2019-01-16 11:07:22,314 - Client_Library_OAUTH - ERROR - Exception occured while fetching oauthtoken from db; Exception Message::'NoneType' object has no attribute 'accessToken'
It seems to me that the tokens are being saved to file, but when the sdk try to access them it is looking for them in a DB and not the file specified in the token_persistence_path.
In my settings.py I have this:
ZOHO_CLIENT_ID = config('ZOHO_CLIENT_ID')
ZOHO_CLIENT_SECRET = config('ZOHO_CLIENT_SECRET')
ZOHO_REDIRECT_URI = config('ZOHO_REDIRECT_URI')
ZOHO_CURRENT_USER_EMAIL = 'jamesalexander#mylastwill.co.uk'
ZOHO_PATH = os.path.join(BASE_DIR, 'wills_online', 'zoho')
zoho_config = {'apiBaseUrl': "https://www.zohoapis.com",
'currentUserEmail': ZOHO_CURRENT_USER_EMAIL,
'client_id': ZOHO_CLIENT_ID,
'client_secret': ZOHO_CLIENT_SECRET,
'redirect_uri': ZOHO_REDIRECT_URI,
'token_persistence_path': ZOHO_PATH}
and in a views file I have this:
from zcrmsdk import *
import logging
from django.shortcuts import HttpResponse
from wills.models import PersonalDetails, ZoHoRecord, WillDocument
from wills_online.decorators import start_new_thread
from wills_online.settings import zoho_config
logger = logging.getLogger(__name__)
class ZohoRunOnce:
def __init__(self):
self.already_run = False
def run_once(self):
if not self.already_run:
print('zoho init run once')
ZCRMRestClient.initialize(zoho_config)
self.already_run = True
zoho_init = ZohoRunOnce()
zoho_init.run_once()
print(zoho_config['token_persistence_path'])
def zoho_callback():
return HttpResponse(200)
#start_new_thread
def zoho_personal_details(request):
""" updates or create a user account on zoho on profile completion """
personal_details_ob = PersonalDetails.objects.get(user=request.user)
zoho_ob = ZoHoRecord.objects.get(user=request.user)
try:
if zoho_ob.account:
record = ZCRMRecord.get_instance('Accounts', zoho_ob.account)
record.set_field_value('Account_Name', request.user.email)
record.set_field_value('Name', personal_details_ob.full_name)
record.set_field_value('Email', request.user.email)
record.set_field_value('Address_Line_1', personal_details_ob.address_line_1)
record.set_field_value('Address_Line_2', personal_details_ob.address_line_2)
record.set_field_value('Post_Town', personal_details_ob.post_town)
record.set_field_value('Post_Code', personal_details_ob.post_code)
record.set_field_value('Dob_Day', personal_details_ob.dob_day)
record.set_field_value('Dob_Month', personal_details_ob.dob_month)
record.set_field_value('Dob_Year', personal_details_ob.dob_year)
record.set_field_value('Gender', personal_details_ob.sex)
record.set_field_value('Marital_Status', personal_details_ob.marital_status)
record.set_field_value('Partner_Name', personal_details_ob.partner_full_name)
record.set_field_value('Partner_Gender', personal_details_ob.partner_gender)
record.set_field_value('Partner_Email', personal_details_ob.partner_email)
record.set_field_value('Children', personal_details_ob.children)
record.set_field_value('Pets', personal_details_ob.pets)
record.update()
else:
user = ZCRMUser.get_instance(name='James Alexander')
record = ZCRMRecord.get_instance('Accounts')
record.set_field_value('Account_Owner', user)
record.set_field_value('Account_Name', request.user.email)
record.set_field_value('Name', personal_details_ob.full_name)
record.set_field_value('Email', request.user.email)
record.set_field_value('Address_Line_1', personal_details_ob.address_line_1)
record.set_field_value('Address_Line_2', personal_details_ob.address_line_2)
record.set_field_value('Post_Town', personal_details_ob.post_town)
record.set_field_value('Post_Code', personal_details_ob.post_code)
record.set_field_value('Dob_Day', personal_details_ob.dob_day)
record.set_field_value('Dob_Month', personal_details_ob.dob_month)
record.set_field_value('Dob_Year', personal_details_ob.dob_year)
record.set_field_value('Gender', personal_details_ob.sex)
record.set_field_value('Marital_Status', personal_details_ob.marital_status)
record.set_field_value('Partner_Name', personal_details_ob.partner_full_name)
record.set_field_value('Partner_Gender', personal_details_ob.partner_gender)
record.set_field_value('Partner_Email', personal_details_ob.partner_email)
record.set_field_value('Children', personal_details_ob.children)
record.set_field_value('Pets', personal_details_ob.pets)
response = record.create()
# save account id to db for future updates
zoho_ob.account = response.details['id']
zoho_ob.save()
except ZCRMException as ex:
logger.log(1, ex.status_code)
logger.log(1, ex.error_message)
logger.log(1, ex.error_details)
logger.log(1, ex.error_content)
print(ex.status_code)
print(ex.error_message)
print(ex.error_content)
print(ex.error_details)
Ive tried running ZCRMRestClient.initialize(zoho_config) in settings.py, with no luck.
My method for getting the access token and refresh token, which seems to work is:
import os
import pprint
from sys import argv
import django
import requests
import zcrmsdk
from django.conf import settings
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'wills_online.settings')
django.setup()
def zoho_refresh_token(code):
""" supply a self client token from the zoho api credentials from web site """
zoho_config = {"apiBaseUrl": "https://www.zohoapis.com",
"currentUserEmail": settings.ZOHO_CURRENT_USER_EMAIL,
"client_id": settings.ZOHO_CLIENT_ID,
"client_secret": settings.ZOHO_CLIENT_SECRET,
"redirect_uri": settings.ZOHO_REDIRECT_URI,
"token_persistence_path": settings.ZOHO_PATH}
pprint.pprint(zoho_config)
print('working')
address = f'https://accounts.zoho.com/oauth/v2/token?code={code}&redirect_uri={settings.ZOHO_REDIRECT_URI}&client_id={settings.ZOHO_CLIENT_ID}&client_secret={settings.ZOHO_CLIENT_SECRET}&grant_type=authorization_code'
response = requests.post(address)
data = response.json()
pprint.pprint(data)
zcrmsdk.ZCRMRestClient.initialize(zoho_config)
oauth_client = zcrmsdk.ZohoOAuth.get_client_instance()
refresh_token = data['refresh_token']
print(type(refresh_token))
oauth_client.generate_access_token_from_refresh_token(refresh_token, settings.ZOHO_CURRENT_USER_EMAIL)
print(refresh_token)
print('finished')
if name == 'main':
zoho_refresh_token(argv[1])
This is driving me mad. Help would be greatly appreciated. This is my first post so go easy, lol.
For future reference, you will need to define persistence_handler_class and persistence_handler_path in your configuration dictionary. You will also need a handler class and a user-defined model to store the results. Sample code follows:
# settings.py
import zcrmsdk
configuration_dictionary = {
'apiBaseUrl': 'https://www.zohoapis.com',
'apiVersion': 'v2',
'currentUserEmail': ZOHO_CURRENT_USER_EMAIL,
'sandbox': 'False',
'applicationLogFilePath': '',
'client_id': ZOHO_CLIENT_ID,
'client_secret': ZOHO_CLIENT_SECRET,
'redirect_uri': ZOHO_REDIRECT_URI,
'accounts_url': 'https://accounts.zoho.com',
'access_type': 'online',
'persistence_handler_class': ZOHO_HANDLER_CLASS,
'persistence_handler_path': ZOHO_HANDLER_PATH,
}
zcrmsdk.ZCRMRestClient.initialize(configuration_dictionary)
# zoho.models.py
from django.db import models
from zcrmsdk.OAuthClient import ZohoOAuthTokens
class ZohoOAuthHandler:
#staticmethod
def get_oauthtokens(email_address):
oauth_model_instance = ZohoOAuth.objects.get(user_email=email_address)
return ZohoOAuthTokens(oauth_model_instance.refresh_token,
oauth_model_instance.access_token,
oauth_model_instance.expiry_time,
user_email=oauth_model_instance.user_email)
#staticmethod
def save_oauthtokens(oauth_token):
defaults = {
'refresh_token': oauth_token.refreshToken,
'access_token': oauth_token.accessToken,
'expiry_time': oauth_token.expiryTime,
}
ZohoOAuth.objects.update_or_create(user_email=oauth_token.userEmail, defaults=defaults)
class ZohoOAuth(models.Model):
refresh_token = models.CharField(max_length=250)
access_token = models.CharField(max_length=250)
expiry_time = models.BigIntegerField()
user_email = models.EmailField()
In this example ZOHO_HANDLER_CLASS = 'ZohoOAuthHandler' and ZOHO_HANDLER_PATH = 'zoho.models'
The first time you go to use this you will need a grant_token from https://accounts.zoho.com/developerconsole. For the scope use aaaserver.profile.READ,ZohoCRM.modules.ALL to start (see https://www.zoho.com/crm/developer/docs/api/oauth-overview.html#scopes)
Before you can use the api you'll need to run the code below in a django shell. This uses a grant token to generate your initial access and refresh tokens. Afterwards, the api should handle refreshing your access token.
grant_token = GRANT_TOKEN
import zcrmsdk
oauth_client = zcrmsdk.ZohoOAuth.get_client_instance()
oauth_tokens = oauth_client.generate_access_token(grant_token)
With awscli there's a credential cache in ~/.aws/cli/cache which allows me to cache credentials for a while. This is very helpful when using MFA. Does boto3 have a similar capability or do I have to explicitly cache my credentials returned from session = boto3.session.Session(profile_name='CTO:Admin')?
It is already there.
http://boto3.readthedocs.org/en/latest/guide/configuration.html#assume-role-provider
When you specify a profile that has IAM role configuration, boto3 will make an AssumeRole call to retrieve temporary credentials. Subsequent boto3 API calls will use the cached temporary credentials until they expire, in which case boto3 will automatically refresh credentials. boto3 does not write these temporary credentials to disk. This means that temporary credentials from the AssumeRole calls are only cached in memory within a single Session. All clients created from that session will share the same temporary credentials.
I created a Python library that provides this for you - see https://github.com/mixja/boto3-session-cache
Example:
import boto3_session_cache
# This returns a regular boto3 client object with the underlying session configured with local credential cache
client = boto3_session_cache.client('ecs')
ecs_clusters = client.list_clusters()
To summarise above points, a working example:
from os import path
import os
import sys
import json
import datetime
from distutils.spawn import find_executable
from botocore.exceptions import ProfileNotFound
import boto3
import botocore
def json_encoder(obj):
"""JSON encoder that formats datetimes as ISO8601 format."""
if isinstance(obj, datetime.datetime):
return obj.isoformat()
else:
return obj
class JSONFileCache(object):
"""JSON file cache.
This provides a dict like interface that stores JSON serializable
objects.
The objects are serialized to JSON and stored in a file. These
values can be retrieved at a later time.
"""
CACHE_DIR = path.expanduser(path.join('~', '.aws', 'ansible-ec2', 'cache'))
def __init__(self, working_dir=CACHE_DIR):
self._working_dir = working_dir
def __contains__(self, cache_key):
actual_key = self._convert_cache_key(cache_key)
return path.isfile(actual_key)
def __getitem__(self, cache_key):
"""Retrieve value from a cache key."""
actual_key = self._convert_cache_key(cache_key)
try:
with open(actual_key) as f:
return json.load(f)
except (OSError, ValueError, IOError):
raise KeyError(cache_key)
def __setitem__(self, cache_key, value):
full_key = self._convert_cache_key(cache_key)
try:
file_content = json.dumps(value, default=json_encoder)
except (TypeError, ValueError):
raise ValueError("Value cannot be cached, must be "
"JSON serializable: %s" % value)
if not path.isdir(self._working_dir):
os.makedirs(self._working_dir)
with os.fdopen(os.open(full_key,
os.O_WRONLY | os.O_CREAT, 0o600), 'w') as f:
f.truncate()
f.write(file_content)
def _convert_cache_key(self, cache_key):
full_path = path.join(self._working_dir, cache_key + '.json')
return full_path
session = boto3.session.Session()
try:
cred_chain = session._session.get_component('credential_provider')
except ProfileNotFound:
print "Invalid Profile"
sys.exit(1)
provider = cred_chain.get_provider('assume-role')
provider.cache = JSONFileCache()
# Do something with the session...
ec2 = session.resource('ec2')
Originally, the credential caching and automatic renewing of temporary credentials was part of the AWSCLI but this commit (and some subsequent ones) moved that functionality to botocore which means it is now available in boto3, as well.
I am having error after error trying to upload and resize images to s3 with pil and botos3 and the django default_storage. I am trying to do this on save in the admin.
here is the code:
from django.db import models
from django.forms import CheckboxSelectMultiple
import tempfile
from django.conf import settings
from django.core.files.base import ContentFile
from django.core.files.storage import default_storage as s3_storage
from django.core.cache import cache
from datetime import datetime
import Image, os
import PIL.Image as PIL
import re, os, sys, urlparse
class screenshot(models.Model):
title = models.CharField(max_length=200)
slug = models.SlugField(max_length=200)
image = models.ImageField(upload_to='screenshots')
thumbnail = models.ImageField(upload_to='screenshots-thumbs', blank=True, null=True, editable=False)
def save(self):
super(screenshot, self).save() # Call the "real" save() method
if self.image:
thumb = Image.open(self.image.path)
thumb.thumbnail(100, 100)
filename = str(self.slug)
temp_image = open(os.path.join('tmp',filename), 'w')
thumb.save(temp_image, 'JPEG')
from django.core.files import File
thumb_data = open(os.path.join('/tmp',filename), 'r')
thumb_file = File(thumb_data)
new_file.thumb.save(str(self.slug) + '.jpg', thumb_file)
def __str__(self):
return self.title
This is just one of the many ways I have tried to get it working, and I either get (2, 'No such file or directory') or some other error.
Please can someone help me to get it working. I want it to use the django backend to get the image uploaded to be resized and saved as the thumbnail and then saved. Let me know if you need to know any information. I would be happy to use the django snippet - http://djangosnippets.org/snippets/224/ but I don't know what data to feed it. I get the same IOErrors and 'no such path/filename' even though the main image is uploading to s3 fine. I have also tried things like:
myimage = open(settings.MEDIA_URL + str(self.image))
myimage_io = StringIO.StringIO()
imageresize = myimage.resize((100,100), Image.ANTIALIAS)
imageresize.save('resize_100_100_aa.jpg', 'JPEG', quality=75)
It's been 3 days of looking now so I am starting to go spare! Thanks
I had a similar problem, but in my case using sorl-thumbnail was not an option. I found that I can open an Image directly from S3BotoStorage by passing in a file descriptor instead of a path.
So instead of
thumb = Image.open(self.image.path)
use
thumb = Image.open(s3_storage.open(self.image.name))
Then you can process and save the new file locally as you were doing before.
Why don't you try sorl-thumbnail. It has the exact same interface as the default ImageField django provides and it seems like it would be a lot nicer to work with than the roll-your-own support.
Storage support
Pluggable Engine support (PIL, pgmagick)
Pluggable Key Value Store support (redis, cached db)
Pluggable Backend support
Admin integration with possibility to delete
Dummy generation
Flexible, simple syntax, generates no html
ImageField for model that deletes thumbnails
CSS style cropping options
Margin calculation for vertical positioning
I'm trying to use django, and mongoengine to provide the storage backend only with GridFS. I still have a MySQL database.
I'm running into a strange (to me) error when I'm deleting from the django admin and am wondering if I am doing something incorrectly.
my code looks like this:
# settings.py
from mongoengine import connect
connect("mongo_storage")
# models.py
from mongoengine.django.storage import GridFSStorage
class MyFile(models.Model):
name = models.CharField(max_length=50)
content = models.FileField(upload_to="appsfiles", storage=GridFSStorage())
creation_time = models.DateTimeField(auto_now_add=True)
last_update_time = models.DateTimeField(auto_now=True)
I am able to upload files just fine, but when I delete them, something seems to break and the mongo database seems to get in an unworkable state until I manually delete all FileDocument.objects. When this happens I can't upload files or delete them from the django interface.
From the stack trace I have:
/home/projects/vector/src/mongoengine/django/storage.py in _get_doc_with_name
doc = [d for d in docs if getattr(d, self.field).name == name] ...
▼ Local vars
Variable Value
_[1]
[]
d
docs
Error in formatting: cannot set options after executing query
name
u'testfile.pdf'
self
/home/projects/vector/src/mongoengine/fields.py in __getattr__
raise AttributeError
Am I using this feature incorrectly?
UPDATE:
thanks to #zeekay's answer I was able to get a working gridfs storage plugin to work. I ended up not using mongoengine at all. I put my adapted solution on github. There is a clear sample project showing how to use it. I also uploaded the project to pypi.
Another Update:
I'd highly recommend the django-storages project. It has lots of storage backed options and is used by many more people than my original proposed solution.
I think you are better off not using MongoEngine for this, I haven't had much luck with it either. Here is a drop-in replacement for mongoengine.django.storage.GridFSStorage, which works with the admin.
from django.core.files.storage import Storage
from django.conf import settings
from pymongo import Connection
from gridfs import GridFS
class GridFSStorage(Storage):
def __init__(self, host='localhost', port=27017, collection='fs'):
for s in ('host', 'port', 'collection'):
name = 'GRIDFS_' + s.upper()
if hasattr(settings, name):
setattr(self, s, getattr(settings, name))
for s, v in zip(('host', 'port', 'collection'), (host, port, collection)):
if v:
setattr(self, s, v)
self.db = Connection(host=self.host, port=self.port)[self.collection]
self.fs = GridFS(self.db)
def _save(self, name, content):
self.fs.put(content, filename=name)
return name
def _open(self, name, *args, **kwars):
return self.fs.get_last_version(filename=name)
def delete(self, name):
oid = fs.get_last_version(filename=name)._id
self.fs.delete(oid)
def exists(self, name):
return self.fs.exists({'filename': name})
def size(self, name):
return self.fs.get_last_version(filename=name).length
GRIDFS_HOST, GRIDFS_PORT and GRIDFS_COLLECTION can be defined in your settings or passed as host, port, collection keyword arguments to GridFSStorage in your model's FileField.
I referred to Django's custom storage documenation, and loosely followed this answer to a similar question.