I have geojson data in the URL. Here is the URL:
https://publoccityhealthdata.s3.us-east-2.amazonaws.com/chi_boundries.geojson
I need to generate model class for Django using ogrinspect
from django.contrib.gis.db import models
Step 1: Go to manage.py folder level.
Step 2: Make sure django-contrib-gis is installed.
Step 3: Type this command to the terminal python manage.py ogrinspect https://publoccityhealthdata.s3.us-east-2.amazonaws.com/chi_boundries.geojson Neighborhoods --srid=4326 --mapping --multi
You will get following result:
class Neighborhoods(models.Model):
pri_neigh = models.CharField(max_length=0)
sec_neigh = models.CharField(max_length=0)
shape_area = models.CharField(max_length=0)
shape_len = models.CharField(max_length=0)
geom = models.MultiPolygonField(srid=4326)
# This is an auto-generated Django model module created by ogrinspect.
from django.contrib.gis.db import models
neighborhoods_mapping = {
'pri_neigh': 'pri_neigh',
'sec_neigh': 'sec_neigh',
'shape_area': 'shape_area',
'shape_len': 'shape_len',
'geom': 'MULTIPOLYGON',
}
Related
I am new to Django and my current task is to upload a xml file with 16 fields and more than 60000 rows to a database in PostgreSQL. I used Django to connect to the Database and was able to create a table in the database.
I also used XML Etree to parse the xml file. I am having trouble storing the data in the table that I created in the sql database.
This is the code that I used to parse:
import xml.etree.ElementTree as ET
def saveXML2db():
my_file = "C:/Users/Adithyas/myproject/scripts/supplier_lookup.xml"
tree = ET.parse(my_file)
root = tree.getroot()
cols = ["organization", "code", "name"]
rows = []
for i in root:
organization = i.find("organization").text
code = i.find("code").text
name = i.find("name").text
x = rows.append([organization, code, name])
data = """INSERT INTO records(organization,code,name) VALUES(%s,%s,%s)"""
x.save()
saveXML2db()
the code runs without any error, but I am unable to store the data into the table in the SQL database.
So I figured out the answer to my question and I wish to share this with you guys.
This is how I imported a xml file to PostgreSQL database using Django ORM:
First, I created a virtual environment to work with:
open command prompt in the folder you wish to run the project
py -m venv envy
envy\Scripts\activate
our virtual environment is ready to use
then,
pip install django
pip install psycopg2
django-admin startproject projectq
cd projectq
py manage.py startapp myapp
now both our project and app is created and ready to use
code . #to open Visual code
now go to settings.py in 'projectq' and add 'myapp' to INSTALLED_APPS:
INSTALLED_APPS = [
'myapp',#add myapp to the installed apps
]
now to connect our project to PostgreSQL database we have to make some changes in the DATABASES in settings.py as well:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'projectq',
'USER': 'postgres',
'PASSWORD': '1234',
}
}
change dbsqlite to the name of the database that you are using, add name of your Database, username and password
now the connection is established.
we move on to the next step
go to models.py to create our table in PostgreSQL to store our xml data:
from django.db import models
# Create your models here.
class Record(models.Model):
po_organization = models.IntegerField()
code = models.CharField(max_length = 100)
name = models.CharField(max_length=100)
address_1 = models.CharField(max_length=100 , null = True)
address_2 = models.CharField(max_length=100, null = True)
If your data has null values it's best to add null = True, to avoid errors
py manage.py makemigrations
py manage.py migrate
now the table we created should appear on the PostgreSQL database
next step is to parse our xml file and to import it to the table we created.
For that we will use Django ORM queries
open terminal in our visual code in models.py
activate virtual environment again
to use ORM query:
py manage.py shell
now add these codes to the interactive console:
>>>from myapp.models import Record
>>>import xml.etree.ElementTree as ET
>>>def data2db():
...file_dir = 'supplier_lookup.xml'
...data = ET.parse(file_dir)
...root = data.findall('record')
...for i in root:
... organization = i.find('organization').text
... code = i.find('code').text
... name = i.find('name').text
... address_1 = i.find('address_1').text
... address_2 = i.find('address_2').text
... x = Record.objects.create(organization=organization, code=code,
... name=name, address_1=address_1, address_2=address_2)
... x.save()
...
>>>data2db()
That's It. The data should be loaded into the database now.
Hope this helps.
Have you checked any python/PostgreSQL examples? Your code should have something like this (untested):
import psycopg2
def storeXmlToPostgres(xmldata):
with psycopg2.connect(host="dbhost", database="dbname", user="username", password="password") as conn:
sql = "INSERT INTO records(organization,code,name) VALUES(%s,%s,%s)"
cur = conn.cursor()
for i in xmldata:
organization = i.find("organization").text
code = i.find("code").text
name = i.find("name").text
cur.execute(sql, [organization, code, name])
I'm using Django 2.2.5 and have multiple choice fields based on enums in my models. For an unknown reason I now get a migration error when using an enum for choice field during migration:
django.db.utils.OperationalError: (1067, "Invalid default value for 'protocol'")
model.py
from django.db import models
# See class above
from .utils import NetworkProtocolList
class Networks(models.Model):
ipv4 = models.GenericIPAddressField(blank=False, null=False)
protocol = models.CharField(choices=NetworkProtocolList.choices(), max_length=20,default=NetworkProtocolList.ETH)
class Meta:
managed = True
db_table = 'networks'
utils.py
from enum import Enum
class NetworkProtocolList(Enum):
ETH = 'Ethernet'
MPLS = 'MPLS'
#classmethod
def choices(cls):
return [(key.name, key.value) for key in cls]
I issued
manage.py makemigrations
and subsequent
manage.py migrate
generated the following error:
django.db.utils.OperationalError: (1067, "Invalid default value for
'protocol'")
xxxx_auto_xxxxxxxx_xxxx.py
# Auto generated migration file
import my_.utils
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('my_app', 'yyyy_auto_yyyyyyyy_yyyy'),
]
operations = [
migrations.AddField(
model_name='networks',
name='protocol',
# Field definition here, pay attention to the default value
field=models.CharField(choices=[('ETH', 'Ethernet'), ('MPLS', 'MPLS')], default=my_app.utils.NetworkProtocolList('Ethernet'), max_length=20),
),
]
Than I edited migration file to manually set the default to a string instead of calling enum class:
xxxx_auto_xxxxxxxx_xxxx.py
# Edited migration file
import my_.utils
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('my_app', 'yyyy_auto_yyyyyyyy_yyyy'),
]
operations = [
migrations.AddField(
model_name='networks',
name='protocol',
# Field definition here, pay attention to the modified default value
field=models.CharField(choices=[('ETH', 'Ethernet'), ('MPLS', 'MPLS')], default='Ethernet', max_length=20),
),
]
Now migration works properly but I wonder why I can't define default value like before using the enum instead of a litteral string, because I have other model field for whom this has worked properly.
Is that a bug, what am I lissing here, How to set default value for a model field based on enum in Django?
Since you are using Django<3.X, Django isn't able to identify the enum value. So, use the .value property of enum class as
protocol = models.CharField(
choices=NetworkProtocolList.choices(),
max_length=20,
default=NetworkProtocolList.ETH.value # <--- change is here
)
I am trying to use Django ORM (or any other way using Django) to execute this query (PostgreSQL) AND send the result back to the front end in GeoJSON format.
I am using Django 2.2.15
SELECT string_agg(name, '; '), geom
FROM appname_gis_observation
where sp_order = 'order1'
GROUP BY geom;
The model looks like this (models.py)
from django.db import models
from django.contrib.gis.db import models
class gis_observation(models.Model):
name = models.CharField(max_length=100,null=True)
sp_order = models.CharField(max_length=100,null=True)
geom = models.MultiPointField(srid=4326)
So I thought this would work (views.py)
from django.core.serializers import serialize
from .models import *
from django.shortcuts import render
from django.contrib.postgres.aggregates.general import StringAgg
def show_observation(request):
results = gis_observation.objects.values('geom').filter(sp_order='order1').annotate(newname=StringAgg('name', delimiter='; '))
data_geojson = serialize('geojson', results, geometry_field='geom', fields=('newname',))
return render(request, "visualize.html", {"obs" : data_geojson})
The ORM query works fine in the Django shell but Django complains at the serialize step: AttributeError: 'dict' object has no attribute '_meta'.
Even if the serialize step worked, I suspect it would skip my annotated field (by reading other posts)
Apparently I am not the only one who met that same problem but I could not find a solution for it.
This is the solution I came up with. Frankly I'd be glad to accept another answer, so any proposal still welcome!
In the end, I built a Geojson array by looping though the result set. I guess I could as well have gone for a cursor sql query instead and skip the orm api entirely.
queryset = gis_species_observation.objects.values('geom').filter(sp_order='order1').annotate(name=StringAgg('name', delimiter='; '))
mydict = []
results = list(queryset)
for result in results:
rec = {}
rec["type"] = "Feature"
rec["geometry"] = json.loads(result["geom"].geojson)
rec["properties"] = {"name":result["name"]}
mydict.append(rec)
data_geojson = json.dumps(mydict)
return render(request, "visualize_romania.html", {"mynames" :data_geojson})
Is ist possible to use neomodel to make models in django?
How do I have to integrate neo4j in django? I'm using Python 3, so neo4django isn't really an option.
I'm new to both of them and at the moment I'm a little confused...
Thanks a lot! :3
Hey neomodel supports python 3 out of the box you can use it with or without django checkout the documentation here: http://neomodel.readthedocs.org/en/latest/
NEOMODEL_NEO4J_BOLT_URL = bolt://neo4j:password#localhost:7687
NEOMODEL_SIGNALS = True
NEOMODEL_FORCE_TIMEZONE = False
NEOMODEL_ENCRYPTED_CONNECTION = True
NEOMODEL_MAX_POOL_SIZE = 50
You need to add the above lines in your settings.py file to use Neo4j.
Hey I know this was asked 8 years ago, but now there's a tool called neomodel which serves this exact purpose. It can also be combined with django-neomodel which allows for neomodel to be easily integrated into your django project.
Using django-neomodel, all you need to do is specify the URL of the database in your settings.py file like so:
NEOMODEL_NEO4J_BOLT_URL = 'bolt://{username}:{password}#{HOSTorIP}'
You can easily create models in your models.py file. Here are some examples for this neomodel documentation:
from neomodel import (StructuredNode, StringProperty,
UniqueIdProperty, IntegerProperty,
RelationshipTo)
class Country(StructuredNode):
code = StringProperty(unique_index=True, required=True)
class Person(StructuredNode):
uid = UniqueIdProperty()
name = StringProperty(unique_index=True)
age = IntegerProperty(index=True, default=0)
# traverse outgoing IS_FROM relations, inflate to Country objects
country = RelationshipTo(Country, 'IS_FROM')
Then you can run python manage.py install_labels to do the equivalent of running migrations, or python manage.py clear_neo4j to clear all nodes from the database.
Nodes can be created like so:
from models import Person
john = Person(name="john", age=23).save()
frank = Person(name="frank", age=50).save()
canada = Country(code=5).save()
And relationships like so:
john.country.connect(canada)
And nodes/relationships can be retrieved as follows:
frank = Person.nodes.get(name='frank')
frank.age += 1
frank.save()
franks_country = frank.country
print(franks_country)
# {'code': 5}
I was wondering if any could point me in the right direction.
I'm trying to fetch twitter data and save it to a DB (with Django ORM / models).
My first approach was to create a model with all relevant information of a tweet (Status) like this:
class Tweet(models.Model):
"""
A tweet (Status) with all the respective metadata
"""
id = models.BigIntegerField()
lang = models.CharField(max_length=10)
retweet_count = models.PositiveIntegerField()
text = models.CharField(max_lenght=150)
source = ....
....
And then fetching like:
#Almost pseudocode
from monitoring.models import Tweet
status = api.get_status('xxxx') # A simple tweet status with tweepy
newtweet = Tweet(id=status.id, screen_name = status.screen_name, followers_count = status.followers_count)
newtweet.save()
I guess I could do better and easy with tweepy models factory but I can't go in the right direction by my own...any suggest? Any example/link/article would be great.
we used pickle and saved the twitter data in a variable(hard disk) if that helps you. Just use pickle in your model like this:
import pickle
filename = 'finalized_model.sav'
pickle.dump(LR_model, open(filename, 'wb'))
pickl={'vectorizer':tf_vector,'model':LR_model,'clean':clean()}
pickle.dump(pickl,open('models'+".p","wb"))
and then put this in your django apps.py:
path = os.path.join(settings.MODELS, 'models.p')
with open(path, 'rb') as pickled:
data = pickle.load(pickled)
model = data['model']
vectorizer = data['vectorizer']
given that you have declared your model in setting.py. then fetch twitter data use it in views.py