I'm building application where I need to sort many data by countries. And I decided to use separate tables for it.
For example I need to implement shop table with fields like
id
title
location
etc..
And I need separate data to localized tables like:
shops_uk
shops_us
shops_cz
etc.
But if I work as usually I need to create many entities for it like: ShopUK, ShopUS...
Question: How can I organise it with doctrine2. How to get data from localized table using 1 entity?
Related
I'm trying to build a web application where users can upload a file (specifically the MDF file format) and view the data in forms of various charts. The files can contain any number of time based signals (various numeric data types) and users may name the signals wildly.
My thought on saving the data involves 2 steps:
Maintain a master table as an index, to save such meta information as file names, who uploaded it, when, etc. Records (rows) are added each time a new file is uploaded.
Create a new table (I'll refer to this as data tables) for each file uploaded, within the table each column will be one signal (first column being timestamps).
This brings the problem that I can't pre-define the Model for the data tables because the number, name, and datatype of the fields will differ among virtually all uploaded files.
I'm aware of some libs that help to build runtime dynamic models but they're all dated and questions about them on SO basically get zero answers. So despite the effort to make it work, I'm not even sure my approach is the optimal way to do what I want to do.
I also came across this Postgres specifc model field which can take nested arrays (which I believe fits the 2-D time based signals lists). In theory I could parse the raw uploaded file and construct such an array and basically save all the data in one field. Not knowing the limit of size of data, this could also be a nightmare for the queries later on, since to create the charts it usually takes only a few columns of signals at a time, compared to a total of up to hundreds of signals.
So my question is:
Is there a better way to organize the storage of data? And how?
Any insight is greatly appreciated!
If the name, number and datatypes of the fields will differ for each user, then you do not need an ORM. What you need is a query builder or SQL string composition like Psycopg. You will be programatically creating a table for each combination of user and uploaded file (if they are different) and programtically inserting the records.
Using postgresql might be a good choice, you might also create a GIN index on the arrays to speed up queries.
However, if you are primarily working with time-series data, then using a time-series database like InfluxDB, Prometheus makes more sense.
I'm currently designing my data base using postgresql with Django and I was wondering: What is best practice - having several instances of the same model with the same value or a many to many relation ship?
Let me elaborate. Let's say I'm designing a store. The store sells items. Items can have one or many statuses (e.g. ordered, shipped, delivered, paid, pre-ordered etc.).
What would be a better practice:
Relating the items to their status via a many-to-many relationship, which will lead to one status having hundreds of thousand and later millions of relations? Will so many relations become problematic?
Or is it better for each item to have a foreignkey to their statuses? So that each status only has one item. And if I would like to query all the items that have the same status (e.g. shipped), I would have to iterate over all statuses with a common name.
What would be better, especially for the long term?
I would recommend going with a many-to-many relationship.
Hundreds of thousands or even millions of relations should not be a problem. The many-to-many relationship is stored as a table with id, item_id, status_id. SQL will be performant at querying the table either by status_id or item_id even if the table gets big. This is exactly the kind of thing it was built to handle.
Let me elaborate. Let's say I'm designing a store. The store sells
items. Items can have one or many statuses (e.g. ordered, shipped,
delivered, paid, pre-ordered etc.).
If many people will have this many itens you should use manytomany relations, better let django handle with this "third table", since this table just hold ids you can interate over them using reverse lookup, i do prefer using many to many instad of simple foreignkeys.
In your case, who you will handle when your User will hold many itens? like what if my User buy one potato and 2 bananas? you will duplicate the tuple in your User Table to tell "here he have the potato and in this second one he have the banana"? so you will be slave of Disctinct attribute while you still dirtying your main table User
...
class Item(models.Model):
...
class User(models.Model):
items = models.ManyToMany(Item)
So when i query my Item and my User will only bring attributes related to them... while if you use item inside of User Model you will have multiple instances of same user.
So instead of use User.items.all() you will use User.objects.filter(id=id)and them items = [user.item for user in User.objects.filter(id=id)]
Look how complex this get and makeing your database so dirty
I'm building a product with Zend 2 and Doctrine 2 and it requires that I have a separate table for each user to contain data unique to them. I've made an entity that defines what that table looks like but how do I change the name of the table to persist the data to, or in fact retrieve the data from, at run time?
Alternatively am I going to be better off giving each user their own database, and just changing which DB I am connecting to?
I'd question the design-choice at first. What happens if you create a new user after runtime. The table has to be created first? Furthermore, what kind of data are you storing, to me this sounds like a pretty common multi-client capabilities. Like:
tbl_clients
- id
- name
tbl_clientdata
- client_id
- data_1_value
- data_2_value
- data_n_value
If you really want to silo users data, you'd have to go the separate databases route. But that only works if each "user" is really independent of each other. Think very hard about that.
If you're building some kind of software-as-a-service, and user A and user B are just two different customers of yours, with no relationship to each other, then an N+1 database might be appropriate (one db for each of your N users, plus one "meta" database which just holds user accounts (and maybe billing-related stuff).
I've implemented something like this in ZF2/Doctrine2, and it's not terribly bad. You just create a factory for EntityManager that looks up the database information for whatever user is active, and configures the EM to connect to it. The only place it gets a bit tricky is when you find yourself writing some kind of shared job queue, where long-running workers need to switch database connections with some regularity -- but that's doable too.
Can Django support Oracle nested tables or varrays or collections in some manner? Asking just for completeness as our project is reworking the data model, attempting to move away from EAV organization, but I don't like creating a bucket load of dependent supporting tables for each main entity.
e.g.
(not the proper Oracle syntax, but gets the idea across)
Events
eventid
report_id
result_tuple (result_type_id, result_value)
anomaly_tuple(anomaly_type_id, anomaly_value)
contributing_factors_tuple(cf_type_id, cf_value)
etc,
where the can be multiple rows of the tuples for one eventid
each of these tuples can, of course exist as separate tables, but this seems to be more concise. If it 's something Django can't do, or I can't modify the model classes to do easily, then perhaps just having django create the extra tables is the way to go.
--edit--
I note that django-hstore is doing something very similar to what I want to do, but using postgresql's hstore capability. Maybe I can branch off of that for an Oracle nested table implementation. I dunno...I'm pretty new to python and django, so my reach may exceed my grasp in this case.
Querying a nested table gives you a cursor to traverse the tuples, one member of which is yet another cursor, so you can get the rows from the nested table.
I am new to Django plotform. I am trying to write a program which basically accepts a post method. The content of incoming data is storename, bookname, bookserial. That part is already implemented and works well. When I post the content such as storename=John's shopping center, bookname=Love is beatiful, bookserial=123. It creates a table and save those things into a table. But, the thing is that I want to create not just only one table for each store. Because, I can have multiple storename and each store should have its own table. When I post the storename on the fly ,it should check storename and then if it's table is created already, the bookname and bookserial should be inserted its table. If not, a new table should be created and then the incoming data is inserted the new table. The new table name should be storename as well.So, as I said, I only need to learn how to create new tables on the fly part. Could you please help me how to do that, any comments and ideas is appreciated....
An example to make it clear,
Table-1=John's shopping center
bookname=Love is beatiful
bookserial=123
Table-2= John's shopping center-2
bookname=Time is important
bookserial=456
So, the model is same for each shopping center but each of sopping center is a different table with the name of shopping center.
In the traditional sense, it is not possible to dynamically create concrete tables on the fly in django. Models have to be registered as part of the application startup, so that the ORM can properly manage all the relations. Consider what would happen if you defined a new model that set up constraints or backrefs to other models. Those other models, being classes, have already been set up and are in memory. They can no longer go through their metaclass step to wire new relations. You could easily break things.
Your options are limited to either a solution involving a few tables that can dynamicaly describe different entities, or to use a nosql backend that does not care about schemas and will let you store anything at any time.
See this question and answer for details: Django dynamic model fields
The only way to have a concrete table on the fly is if you have the django app restart itself completely in response.