I am trying to use ElasticSearch as a NoSQL database in my Django project.
The goal is to plug Kibana on the other side and be able to visualize my data there.
I do not have a choice of adding another database and using ElasticSearch only for indexing as my project is plugging to an existing infrastructure.
So I have installed the following two modules:
django-haystack
elasticsearch
I was expecting to not need SQLite (or any other database) for storage and to use ElasticSearch as a NoSQL storage (is that wrong?)
I added the Haystack connections settings to my project settings and then, looked for getting rid of "DATABASES" or replacing it with values pointing to my ElasticSearch with no success :(
Here is what I'm stuck with:
settings.py:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
}
}
HAYSTACK_CONNECTIONS = {
'default': {
'ENGINE': 'haystack.backends.elasticsearch_backend.ElasticsearchSearchEngine',
'URL': 'http://127.0.0.1:9200/',
'INDEX_NAME': 'haystack'
},
}
On all the answers I could find on StackOverflow and code snippets I would never see these DATABASES settings changed, so I assume people were using a database for storage and ElasticSearch only for their search engine/indexing which is not what I am looking for.
Is there a way I can have ElasticSearch as my storage DB (just like I would with MongoDB), not breaking the Models and interface of Django?
It's not possbile (officially) to use ElasticSearch as your django database backend (I assume that this is what you want) - currently supported backends are:
postgresql
mysql
sqlite3
oracle
The above is official list, unofficial (3rd parties) is as follows:
SAP SQL
IBM DB2
Microsoft SQL
Firebird
ODBC
Anyway, the people around django are very nice and hard working developers :) And there is package for that (you can give it a try - never used it before):
https://github.com/aparo/django-elasticsearch
One more thing - why do you need django? Is kibana not enough? You gonna make some changes in data on the ES index in your Django App? Or should it be just readonly?
Because if you want use the ES like a normal DB storage - it won't work, as you will wait each time you update/insert data about 1-2 seconds till ES index new data. This is just not the purpose of the ES.
Hope this help, happy coding.
Related
I have a website that I built using Django and deployed on Heroku. The tutorial I followed had me set up a Heroku Postgres database attached as an add-on. After Heroku's recent pricing changes the Postgres database has gone from free to $5 / month.
Maybe static isn't the correct word, but the site doesn't have user accounts, collect user information, or otherwise have any need for a database. I believe the only information that's stored in the database is login info for the Django admin site. I was able to export a .pgdmp file from Heroku of the data stored in the database but couldn't make heads or tails of the contents.
Here's the settings.py file code for the database section:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': BASE_DIR / 'db.sqlite3',
}
}
import dj_database_url
db_from_env = dj_database_url.config(conn_max_age=500)
DATABASES['default'].update(db_from_env)
My question is can I delete this add-on from the Heroku CLI or dashboard without breaking the site in production? Would I need to change anything in my settings.py file or elsewhere in my Django code first? I assume I would still be able to access the Django admin page on the development server, is that assumption correct?
Sorry for the possibly obvious question but I just don't want to break a site in production without doing some research first!
My question is can I delete this add-on from the Heroku CLI or dashboard without breaking the site in production?
We can't know for sure, but probably not.
You are using dj-database-url to set your default database. That likely means that your application is connecting to PostgreSQL via the DATABASE_URL environment variable.
SQLite does not work properly on Heroku due to its ephemeral filesystem. If you need a database, it needs to be a client-server database instead of a file-based database.
For anyone in my situation finding this in the future here's what I learned:
My initial hypothesis was correct, deleting the Postgres Heroku add-on left the site in production intact and working correctly. The only difference without the database is I can't access the Django admin page on the hosted site but I can still access this through the development server if needed.
Your site might be different than mine however, so your mileage may vary. Here's a couple things I did before making any changes to the site in production.
Check what tables/data you have in your database. Thanks to #Chris who answered in this thread and posted this link talking about the dataclips feature in Heroku. If you have tables that seem not to relate to the admin interface then you might be reliant on your database.
Consider spinning up another instance of your live site in production on Heroku. This way you can test and make changes without impacting users on the live site if something goes wrong.
I am considering using Google BigQuery as a back-end for Django but cannot be certain if this is possible, and if it is, what settings would apply.
Currently, my Django application uses Postgresql, and the code in settings.py is as follows:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': 'mydatabase',
'USER': 'mydatabaseuser',
'PASSWORD': 'mypassword',
'HOST': '127.0.0.1',
'PORT': '5432',
}
}
Ideally, I'd like to setup a database connection to Google BigQuery through settings.py and then use views and models as usual.
It's not possible, or at least not supported. You could use the API directly, but obviously you won't get any advantages of the ORM.
You should have a db motor like postgres, mysql, whatever...
The point is, this db motor is necessary to have it, because the structure works in that way.
but of course, you can invoke google cloud from librarys in django and use it as
from google.oauth2 import service_account
from google.auth.transport.requests import AuthorizedSession
from google.cloud import datastore
from google.cloud import bigquery
in my case I used to connect
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = 'OAuth2Credential.json'
for generate your .json you should go to the documentation in:
https://cloud.google.com/iam/docs/creating-managing-service-account-keys
It is possible by using SQLAlchemy with Django.
SQLAlchemy can connect to bigquery with the pybigquery driver.
See the following about how to
Configuring Django to use SQLAlchemy.
Everything is possible for sure. To create an interface would not be such a big job. But, I would just keep one note:
Bigquery is not intended to be backend database, rather it is more like data warehouse as it is defined within business intelligence discipline. This means, google will make it very hard for you to perform fast multi-user operations. As far as I can recall, update statements for example have have some thresholds.
On another hand, if this is purely for example data input, or visualisation of data, then why not. But then again, I think Azure power apps is kind of a product for it.
I have a running and working database in SQLite but multiple users are writing to it in the same time and i seem to have concurrency problems, because of the database being locked.
What do i need in order to create and access a PostgreSQL database in Python? Is there a way in which i dont install anything else than python libraries? I have a company laptop and would need 1XXXX approvals to install a third party software.
The setup is simple: The users go to the upload page on the site(built in django), uploads a xls file and the info gets extracted into a database(while being compared with info from inside the database). If 2 users try to upload files in the same time, i get database locked error.
Any help would be great, either to solve this SQLite problem or to get me started in Postgre if this should solve it. THX!
SQLite is not suitable for multiple user access since it is just a file acting as a database. To get the real power of DBMS you have to switch either MySQL or PostgreSQL which are officially supported by Django. Remember that PostgreSQL is preferred by Django for production use. Anyway you can use Oracle DB if you company already have it.
You need to setup settings.py in your project folder in order connect any database with Django as follows.
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'dbname',
'USER': 'postgres',
'PASSWORD': 'postgres',
'HOST': '',
'PORT': '',
}
}
You may need to install relevant db driver to connect it with Django.
Before committing to migrating to PostgreSQL, you may want to simply increase the busy timeout of your SQLite3 database and see if that produces acceptable results.
I am new to Django; intermediate in Python3.6. Now, I've created a basic Django app which displays index.html. Now, I want to create a very simple form in index.html which will query the database and print the results in the same form. I already have the postgresql database in my local with all the data needed; it is very simple with just one table (right now, my Django app is connected to the default sqlit3 db). Where do I go about creating the query to fetch the data and populate it?
You need to install these libraries first :
sudo apt-get install libpq-dev postgresql postgresql-contrib
Configuration
in settings.py
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': 'db_name',
'USER': 'db_user',
'PASSWORD': 'db_user_password',
'HOST': '',
'PORT': 'db_port_number',
}
}
You can refer to these two links:
For More Detailed information about setup :Tutorial for setting up application using PostgreSql
Documentation link for writing Queries
Glad you're coming on board with Django! It's a great language with plenty of useful features, you won't regret learning it.
To connect to an existing database, check out the docs here. This explains how to use your existing db to create models in your project.
I also recommend checking out the tutorials in the docs starting here. This tutorial will explain how to connect to your postgres db, as well as how to begin writing queries for your views. This is the place to start learning the basics of Django, just follow it along and you will get the hang of everything in no time
I want to use MongoDB as database backend for my Django project. Although there are many discussions on the net, I'm having troubles to integrate them well.
My goals:
use default Django (so no django-nonrel, which is still to 1.3)
integrate them so that authentication is backed by MongoDB (i.e. the
default User model) as well as the sessions thing.
if possible, still have a ORM-like query system
As I understand that, mongoengine could meet all my requirements, but I'm having troubles making it work correclty.
Docs say to ignore DATABASES setting. If I don't specify it, Django raises an error, while if I fill it, Django creates that database and does not use my MongoDB instance, even though I call connect() later in the file. When I run syncdb Django uses the other database (the one I specified in the DATABASES setting) and not MongoDB. So when I fire up MongoDB shell I can see the database is created, but the only collection is startup_log, which I never created and I suspect it's created automatically.
This might be a change in the default requirements does adding a dummy backend to your settings help?
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.dummy'
}
}