I am trying to understand how Django and Appengine work together?
First, question: Is this a good team?
Experience, what is possible and what not, would be great.
I also read some modules like auth, admin wont work.
But the article is rather old, so maybe there is an update.
And in that tutorial one has to import bforms.
What is that?
Django Module? Appengine? Python? Bigtable?
How is Bigtable different from regular SQL, MySQL?
Thanks
Regular SQL and MySQL are designed for one computer only and fail in cloud computing where you need 1,000 computers for one database. Thus the next generation databases, like bigtable, were created to distribute the data over many database servers. They are called NoSQL databases for "Not Only SQL." See http://nosql-database.org/ for a list of NoSQL databases. The google app engine apparently allows you to use the bigtable structure so you data is distributed over a dozen database servers in the cloud. So does Amazon's simple db.
Related
Can we directly use Elasticsearch as a primary database in Django? I have tried finding the solution or a way out but could not find any relevant information. Everywhere it is said that we can use Elasticsearch as a search engine over any other primary database. But as per my understanding Elasticsearch is a NoSQL Database, so there should be a way to use it as a Primary Database in a Django Project.
Please help, if someone has any idea about it.
The short answer is no.
SO already has an answer here and this is still valid: Using ElasticSearch as Database/Storage with Django
ES is not a ACID compliant
Indexing is not immediate so any kind of load would be an issue
It's very weakly consistent
Use it together with a proper database and it will help with real time searches, analytics, expensive queries etc. but treat it as derived data.
I m designing a website where security of data is an issue.
I ve read this book : https://books.agiliq.com/projects/django-multi-tenant/en/latest/index.html
I'm still thinking about the right database structure for users.
And im hesitating between shared database with isolated schemas, isolated databases in a shared app, or completely isolated tenants using docker.
As security of data is an issue, i would like to avoid to put all the users in the same table in the database or in different schemas in the same database. However i dont understand well if i should put each user in a separate database (create a database per user, sqlite for example but i dont know if it would communicate well with postgres). What is the best practice for this in terms of security?
Im wondering how these options affect database speed compared to a shared database with a shared schema, which was the basic configuration of the course i attended on django.
I dont have good knowledge on databases so your help on the performance issue would be very appreciated!
Also, if I want to do some stats and use tenants data, how difficult is it to query completely isolated tenants using docker or isolated databases, in particular if each user is a separate docker or database?
I have a working Django web application that currently uses Postgresql as the database. Moving forward I would like to perform some analytics on the data and also generate reports etc. I would like to make use of Amazon Redshift as the data warehouse for the above goals.
In order to not affect the performance of the existing django web application, I was thinking of writing a NEW Django application that essentially would leverage a READ-ONLY replica of the Postgresql database and continuously write data from read-only replicas to the Amazon Redshift. My thinking is that perhaps the NEW Django application can be used to handle some/all of the Extract, Transform and Load functions
My questions are as follows:
1. Does the Django ORM work well with Amazon Redshift? If yes, how does one handle the model schema translations? Any pointers in this regard would be greatly appreciated.
2. Is there any better alternative to achieve the goals listed above?
Thanks in advance.
This is more of an architectural question than a technological one per se.
I am currently building a business website/social network that needs to store large volumes of data and use that data to draw analytics (consumer behavior).
I am using Django and a PostgreSQL database.
Now my question is: I want to expand this architecture to include a data warehouse. The ideal would be: the operational DB would be the current Django PostgreSQL database, and the data warehouse would be something additional, preferably in a multidimensional model.
We are still in a very early phase, we are going to test with 50 users, so something primitive such as a one-column table for starters would be enough.
I would like to know if somebody has experience in this situation, and that could recommend me a framework to create a data warehouse, all while mantaining the operational DB with the Django models for ease of use (if possible).
Thank you in advance!
Here are some cool Open Source tools I used recently:
Kettle - great ETL tool, you can use this to extract the data from your operational database into your warehouse. Supports any database with a JDBC driver and makes it very easy to build e.g. a star schema.
Saiku - nice Web 2.0 frontend built on Pentaho Mondrian (MDX implementation). This allows your users to easily build complex aggregation queries (think Pivot table in Excel), and the Mondrian layer provides caching etc. to make things go fast. Try the demo here.
My answer does not necessarily apply to data warehousing. In your case I see the possibility to implement a NoSQL database solution alongside an OLTP relational storage, which in this case is PostgreSQL.
Why consider NoSQL? In addition to the obvious scalability benefits, NoSQL offer a number of advantages that probably will apply to your scenario. For instance, the flexibility of having records with different sets of fields, and key-based access.
Since you're still in "trial" stage you might find it easier to decide for a NoSQL database solution depending on your hosting provider. For instance AWS have SimpleDB, Google App Engine provide their own DataStore, etc. However there are plenty of other NoSQL solutions you can go for that have nice Python bindings.
i'm new to the nosql world, and from forums and articles that i've read: most of users try to "mix" nosql tools, for example, they use Cassandra and MongoDB together to make a "powerful system", because am beginning with MongoDB, i've downloaded the DjanMon project (am a django fan ^_^ ), of course i've downloaded the special version of django that accepts the NoSql use: Django NonRel, and i've noticed that the Setting file dont "oblige" you to use one specific NoSql solution like in Django with RDBMS where you must specify MySql or PostegreSql or other solution, so, is it possible to mix lot of (or two of course) NoSql solution using Django (for example MongoDB+Cassandra)?
There's nothing to stop you using multiple storage solutions, whether SQL or NoSQL - but the NoSQL solutions all have different architectures, data models and APIs (For example, MongoDB is a document-oriented database, whereas Cassandra is Column-oriented), so you can't usually swap one for another without some effort.
Can you clarify what you are actually trying to achieve? I.e. why are you interested in mixing these two specific solutions?