I was wondering if any could point me in the right direction.
I'm trying to fetch twitter data and save it to a DB (with Django ORM / models).
My first approach was to create a model with all relevant information of a tweet (Status) like this:
class Tweet(models.Model):
"""
A tweet (Status) with all the respective metadata
"""
id = models.BigIntegerField()
lang = models.CharField(max_length=10)
retweet_count = models.PositiveIntegerField()
text = models.CharField(max_lenght=150)
source = ....
....
And then fetching like:
#Almost pseudocode
from monitoring.models import Tweet
status = api.get_status('xxxx') # A simple tweet status with tweepy
newtweet = Tweet(id=status.id, screen_name = status.screen_name, followers_count = status.followers_count)
newtweet.save()
I guess I could do better and easy with tweepy models factory but I can't go in the right direction by my own...any suggest? Any example/link/article would be great.
we used pickle and saved the twitter data in a variable(hard disk) if that helps you. Just use pickle in your model like this:
import pickle
filename = 'finalized_model.sav'
pickle.dump(LR_model, open(filename, 'wb'))
pickl={'vectorizer':tf_vector,'model':LR_model,'clean':clean()}
pickle.dump(pickl,open('models'+".p","wb"))
and then put this in your django apps.py:
path = os.path.join(settings.MODELS, 'models.p')
with open(path, 'rb') as pickled:
data = pickle.load(pickled)
model = data['model']
vectorizer = data['vectorizer']
given that you have declared your model in setting.py. then fetch twitter data use it in views.py
Related
I am trying to use Django ORM (or any other way using Django) to execute this query (PostgreSQL) AND send the result back to the front end in GeoJSON format.
I am using Django 2.2.15
SELECT string_agg(name, '; '), geom
FROM appname_gis_observation
where sp_order = 'order1'
GROUP BY geom;
The model looks like this (models.py)
from django.db import models
from django.contrib.gis.db import models
class gis_observation(models.Model):
name = models.CharField(max_length=100,null=True)
sp_order = models.CharField(max_length=100,null=True)
geom = models.MultiPointField(srid=4326)
So I thought this would work (views.py)
from django.core.serializers import serialize
from .models import *
from django.shortcuts import render
from django.contrib.postgres.aggregates.general import StringAgg
def show_observation(request):
results = gis_observation.objects.values('geom').filter(sp_order='order1').annotate(newname=StringAgg('name', delimiter='; '))
data_geojson = serialize('geojson', results, geometry_field='geom', fields=('newname',))
return render(request, "visualize.html", {"obs" : data_geojson})
The ORM query works fine in the Django shell but Django complains at the serialize step: AttributeError: 'dict' object has no attribute '_meta'.
Even if the serialize step worked, I suspect it would skip my annotated field (by reading other posts)
Apparently I am not the only one who met that same problem but I could not find a solution for it.
This is the solution I came up with. Frankly I'd be glad to accept another answer, so any proposal still welcome!
In the end, I built a Geojson array by looping though the result set. I guess I could as well have gone for a cursor sql query instead and skip the orm api entirely.
queryset = gis_species_observation.objects.values('geom').filter(sp_order='order1').annotate(name=StringAgg('name', delimiter='; '))
mydict = []
results = list(queryset)
for result in results:
rec = {}
rec["type"] = "Feature"
rec["geometry"] = json.loads(result["geom"].geojson)
rec["properties"] = {"name":result["name"]}
mydict.append(rec)
data_geojson = json.dumps(mydict)
return render(request, "visualize_romania.html", {"mynames" :data_geojson})
I want to check the validity of the consumer_key. I have stored one in the models.py as a CharField but how do I go on and fetch it for comparison ?
This might help understand the problem better .
Models.py
class Verification(models.Model):
consumer_key = models.CharField(max_length=30)
hidden_key = models.CharField(max_length=30 , default="")
Views.py
from .models import Verfication
id = Verification.objects.filter()
if request_key ==id.values('consumer_key'):
consumer = oauth2.Consumer( key=request_key,secret= id.values('hidden_key'))
request_key contains the oauth_consumer_key .
Try this code:
from .models import Verfication
try:
id = Verification.objects.get(consumer_key = request_key)
consumer = oauth2.Consumer(key=request_key,secret= id.hidden_key)
except Verification.DoesNotExist:
pass #token does not exist
This will try to get Verification object using request_key and call exception if request_key is not in database.
Also consumer_key should be set to unique in models using unique=True to prevent having multiple entries for same consumer_key.
I am using django-rest-framework to make an api of my data. I am making an app that takes into account user data and remove outliers from that data using Pandas. I am able to present my data on frontend using django templates but somehow I am not able to make an API containing the statistical data using django-rest-framework . Can someone explain it and please help me to rectify my errors and also provide the necessary code
Here is my code
class Data(models.Model):
Name = models.CharField(max_length=30,null=True,blank=True)
Age = models.IntegerField(null=True,blank=True)
Weight = models.FloatField(null=True,blank=True)
Height = models.FloatField(null=True,blank=True)
Sugar = models.FloatField(null=True,blank=True)
def __unicode__(self):
return self.Name
My Serializer Class
class DataSerializer(serializers.ModelSerializer):
class Meta:
model = Data
fields = '__all__'
my views.py
def my_view(request):
con = sqlite3.connect("db.sqlite3")
df = pd.read_sql_query("SELECT * from visualapp_health", con)
a = df.fillna(0)
a['x-Mean'] = abs(a['Age'] - a['Age'].mean())
a['1.96*std'] = 1.96*a['Age'].std()
a['Outlier'] = abs(a['Age'] - a['Age'].mean()) > 1.96*a['Age'].std()
con.close()
return render(request, 'visual.html', {'visual': a})
I am able to get all the Data when using Django templates but somehow I don't able to understand how to make an API of all the data using django-rest- framework.
I finally got it I used django-pandas library and it worked and there is no need to connect to the database just convert your django queryset to pandas dataframe.
I have a model
class Unit(models.Model):
date = models.DateTimeField()
name = models.CharField(max_length = 128)
Then I have a view with js jquery ui datepicker. Then I have an ajax post function with
data_send['date'] = $('#date').value;
data_send['name'] = $('#name').value;
$.post("/url_1/",data_send, function(data_recieve){
alert(data_recieve);
});
Then in a view I'm trying to save unit like
def url_1(request):
unit = Unit.objects.get(pk = 1)
unit.date = request.POST['date']
unit.name = request.POST['name']
unit.save()
return HttpResponse('ok')
Unit.name changes, but not unit.date. I use django 1.3. I have no csrf protection. I recieve 'ok' from server.
Why does the unit.date not save?
Thanks
Since django datetime field is a representation of a datetime.datetime python instance, I like to make sure that I have a valid instance before inserting into the database.
you can use the datetime module to achieve this, instead of saving unit.date as a string
from datetime import datetime
try:
valid_datetime = datetime.strptime(request.POST['date'], '%d-%m-%Y')
except ValueError:
# handle this
then you can save the valid_datetime as your unit.date
the second param of strptime is formatted using the below values, it should match your datepicker format
http://docs.python.org/library/datetime.html#strftime-strptime-behavior
i have i little problem, and that is how can serialize a django query with defer ?
I have this model :
class Evento(models.Model):
nome=models.CharField(max_length=100)
descricao=models.CharField(max_length=200,null=True)
data_inicio= models.DateTimeField()
data_fim= models.DateTimeField()
preco=models.DecimalField(max_digits=6,decimal_places=2)
consumiveis= models.CharField(max_length=5)
dress_code= models.CharField(max_length=6)
guest_list=models.CharField(max_length=15)
local = models.ForeignKey(Local)
user= models.ManyToManyField(User,null=True,blank=True)
def __unicode__(self):
return unicode('%s %s'%(self.nome,self.descricao))
my query is this :
eventos_totais = Evento.objects.defer("user").filter(data_inicio__gte=default_inicio,
data_fim__lte=default_fim)
it works fine i think (how can i check if the query has realy defer the field user ? ) but when i do:
json_serializer = serializers.get_serializer("json")()
eventos_totais = json_serializer.serialize(eventos_totais,
ensure_ascii=False,
use_natural_keys=True)
it always folow the natural keys for user and local, i need natural keys for this query because of the fields local. But i do not need the field user.
To serialize a subset of your models fields, you need to specify the fields argument to the serializers.serialize()
from django.core import serializers
data = serializers.serialize('xml', SomeModel.objects.all(), fields=('name','size'))
Ref: Django Docs