I am trying to use Django's db connection variable to insert a pandas dataframe to Postgres database. The code I use is
df.to_sql('forecast',connection,if_exists='append',index=False)
And I get the following error
Execution failed on sql 'SELECT name FROM sqlite_master WHERE type='table' AND name=?;': relation "sqlite_master" does not exist
LINE 1: SELECT name FROM sqlite_master WHERE type='table' AND name=?...
I think this happens because the Django connection object is not an sqlalchemy object and therefor Pandas assumes I am using sqlite. Is there any way to use .to_sql other than make another connection to the database?
It is possible to create db configuration in setting.py file
DATABASES = {
'default': env.db('DATABASE_URL_DEFAULT'),
'other': env.db('DATABASE_URL_OTHER')
}
DB_URI_DEFAULT=env.str('DATABASE_URL_DEFAULT')
DB_URI_OTHER=env.str('DATABASE_URL_OTHER')
If you want to create sql_alchemy connection you should use DB_URI_DEFAULT or DB_URI_OTHER
in the init method of the class you will use .to_sql method you should write
from you_project_app import settings
from sqlalchemy import create_engine
import pandas as pd
class Example:
def __init__(self):
self.conn_default = create_engine(settings.DB_URI_DEFAULT).connect()
And when you use .to_sql method of pandas it should be like this:
df_insert.to_sql(table, if_exists='append',index=False,con=self.conn_default)
Related
I have a Django project that utilizes multiple databases. https://docs.djangoproject.com/en/dev/topics/db/multi-db/
I perform a lot of raw queries like this:
cursor = connection.cursor()
cursor.execute("select * from my_table")
....
transaction.commit_unless_managed()
How can I specify which database to use?
Refer django docs on executing custom query directly. Specify database in your connection as given below:
from django.db import connections
cursor = connections['db_alias'].cursor()
cursor.execute("select * from my_table")
And then commit using
from django.db import transaction
transaction.commit_unless_managed(using='db_alias')
try this may be it should works.
from django.db import connections
cursor = connections[’my_db_name’].cursor()
# Your code here...
transaction.commit_unless_managed(using=’my_db_name’)
I have CSV file backed up from my django model with django-import-export,I want to restore that to my model how can i do that ?
When i want to create object for each of them i have problem with foreign keys.
id,name,address,job,KBSE
1,Hamid,3,Doctor,4311
2,Ali,7,Artist,5343
3,Reza,2,Singer,5232
See Import data workflow. Most functions can be overridden in resource subclass. If that does not help, please open issue.
You can use customized python script using pandas to load csv data into Django model.
#First define your Django Enviromental variables
import os
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "DjangoProjectName.settings")
import django
django.setup()
import pandas as pd
import numpy as np
#Import Required Django model
from djangoApp.models import * #models
# Import CSV file
df = pd.read_csv('csv_url.csv')
# Do required pre-processing on panda dataframe
# such as data cleaning, data format settings etc..
# Iterater throught panda dataframe and save data in Django model
for index, row in df.iterrows():
# create django model
samplemodelObject = SampleModel()
# Normal Fields ( Non-foreign key fields) adding
samplemodelObject.field_name01 = row['Field_01']
# Adding Foreign Key Field
samplemodelObject.field_foreignkey = ForeignKeyModel.objects.get( fk_key = row['fk_value_field']
# save data model instance
samplemodelObject.save()
samplemodelObject.clear()
You should import the table in the right order, make sure you import all the data the you depend first.
So load foreign tables and than load the current. If you don't have the foreign data and some of them where erased, you'll have to create it back.
Good luck!
I'm trying to connect to mongodb using mongoengine.
Mysql is my default database, and I have 'mongoengine.django.mongo_auth' in my installed apps. I removed the 'AUTH_USER_MODEL = 'mongo_auth.MongoUser'' due to errors about not having a default connection.
I use mongo with celery so I don't think there's a problem with the setup. This is how I am attempting to connect - the code is in views.py
from mongoengine import connect
my_connect = connect('my_db', alias='mongo')
test = test(name='a_name', desc='a desc')
test.save(using='mongo')
my_connect.connection.disconnect()
Have finally managed to sort this out:
#settings.py
from mongoengine import register_connection
register_connection(alias='default',name='db_name')
#models.py
from mongoengine import Document, StringField (etc)
class my_col(Document):
field_a = StringField()
#in your app
from mongoengine import connect
my_con = connect('db_name', alias='default')
item = my_col(field_a='something')
item.save()
my_con.disconnect()
I am using django and postgresql db to develop my project, the db has 3 schemas. And I am writing raw quires using connection string to fetch data and insert data into tables.
Example:
from django.db import connection
query = "select * from schema1.records"
cursor = connection.cursor()
cursor.execute(query)
output = cursor.fetchall()
Like way I am using in my all views.
I wanted to write test cases to my views, so i need to generate fixtures to test my views, how to I generate fixtures for schemas and tables. Please help.
Note: I did not write any models i just used raw queries in my whole project.
Run:
manage.py dumpdata help
This will show how to use loaddata to generate fixtures.
Is there a snippet or easy way to import all of my django tables when entering the prompt?
For example, usually my commands go something like this:
>>> from userprofile.models import Table
>>> Table.objects...
This way, as soon as I entered the prompt, I'd already have the tables imported. Thank you.
django-extensions adds the shell_plus command for manage.py which does exactly this.
from django.db.models import get_models
for _class in get_models():
globals()[_class.__name__] = _class
Here you end up with all installed models available globally, refering to them with their class name. Read the docs for django.db.models.get_models for more info:
Definition: get_models(self, app_mod=None, include_auto_created=False, include_deferred=False)
Docstring:
Given a module containing models, returns a list of the models.
Otherwise returns a list of all installed models.
By default, auto-created models (i.e., m2m models without an
explicit intermediate table) are not included. However, if you
specify include_auto_created=True, they will be.
By default, models created to satisfy deferred attribute
queries are not included in the list of models. However, if
you specify include_deferred, they will be.
You can do this:
>>> from django.conf import settings
>>> for app in settings.INSTALLED_APPS:
... module = __import__(app)
... globals()[module.__name__] = module
...
You'll get the fully qualified names though; userprofile.models.Table instead of Table.