With a model Book that uses MarkDown for a field called content, when I do the following query
Book.objects.filter(published=True).order_by('read')
The site becomes slow because of the content field, I think the hard work occurs when Django tries to convert these fields to python object. When I clean all the content field for each record and leave them blank, the query is pretty much faster.
In my case content field contains large text. To gain in performance, How can I fetch data by ignoring a specific field?
I want to ignore content field like:
Book.objects.filter(published=True)#.ignore_fields('content',).order_by('read')
Try to use defer:
Book.objects.defer('content').filter(published=True).order_by('read')
Related
I’m not sure where to start on this project. I know how to read the contents of the excel spreadsheet, I know how to identify the header row, I know how to loop over the contents. I believe I have the UX portion worked out but I am not sure how to process the data.
I’ve googled and only found .Net solutions but I’m looking for a ColdFusion/Lucee solution.
I have a working form allowing me to map a user's spreasheet column to my database values (this is being kept simple for this post; user does not have direct access to the database).
Now that I have my data, I'm not sure how to loop over the data results. I believe there will be several loops (an outer and an inner). Then of course I also need to loop over the file contents but I think if I can get the headings mapped out,I can figure out the remaining.
Any good links, tutorials, or guides would be greatly appreciated.
Some pseudo code might be enough to get me started.
User uploads form
System reads headers and content.
User is presented form with a list of columns from their uploaded spreadsheet to match with available database fields (eg “column1” matches “customer name”.
User submits form.
Now what?
UPDATED
Here is what the data looks like AFTER the mapping has been done in my form. The column deliiter is the ::: and within the column the ||| indicates the ID associated with the selected column value. I've included the id and the column value since I plan on displaying the mapping again as a confirmation. Having the ID saves a trip to the database.
If I understand correctly, your question is: how do you provide the user a form allowing them to map their spreadsheet columns to that of the database
Since you have their spreadsheet column names, and you have the database column names, then this problem is essentially a UI/UX problem. You need to show both lists, and allow the user to map them. I can imagine several approaches to this. My first thought would be some sort of drag/drop operation, as follows:
Create a list of boxes, one for each field in your database table, and include the field name in (or above) the box. I'll call this the db field list. Then, create another list for each column from the spreadsheet, which I'll call the spreadsheet column list. The user would drag/drop items from the spreadsheet column list to the db field list.
When a mapping has been completed by the user, you would store the column/field names in as data for the DOM element of the db field list box. Then upon submission, you would acquire the mapping data by visiting each box and adding it to an array. Then you would serialize that array into JSON and send that to your form submission handler.
This could be difficult or easy, depending on your knowledge of UI implementations using JavaScript. jQuery makes this easy (if you know jQuery). There's even a jquery UI plugin that does this: https://jqueryui.com/droppable/.
A quick search for javascript drag drop would help, and here's a few articles I found:
https://www.w3schools.com/html/html5_draganddrop.asp
https://medium.com/quick-code/simple-javascript-drag-drop-d044d8c5bed5
You would also need to submit the array of mappings using javascript. You could search for that as well, and here's an article I found:
https://codereview.stackexchange.com/questions/94493/submit-an-array-as-an-html-form-value-using-javascript
I'm trying to select all the songs in my Django database whose tag is any of those in a given list. There is a Song model, a Tag model, and a SongTag model (for the many to many relationship).
This is my attempt:
taglist = ["cool", "great"]
tags = Tag.objects.filter(name__in=taglist).values_list('id', flat=True)
song_tags = SongTag.objects.filter(tag__in=list(tags))
At this point I'm getting an error:
DatabaseError: MultiQuery does not support keys_only.
What am I getting wrong? If you can suggest a completely different approach to the problem, it would be more than welcome too!
EDIT: I should have mentioned I'm using Django on Google AppEngine with django-nonrel
You shouldn't use m2m relationship with AppEngine. NoSQL databases (and BigTable is one of them) generally don't support JOINs, and programmer is supposed to denormalize the data structure. This is a deliberate design desicion: while your database will contain redundant data, your read queries will be much simpler (no need to combine data from 3 tables), which in turn makes the design of DB server much simpler as well (of course this is made for the sake of optimization and scaling)
In your case you should probably get rid of Tag and SongTag models, and just store the tag in the Song model as a string. I of course assume that Tag model only contains id and name, if Tag in fact contains more data, you should still have Tag model. Song model in that case should contain both tag_id and tag_name. The idea, as I explained above, is to introduce redundancy for the sake of simpler queries
Please, please let the ORM build the query for you:
song_tags = SongTag.objects.filter(tag__name__in = taglist)
You should try to use only one query, so that Django also generates only one query using a join.
Something like this should work:
Song.objects.filter(tags__name__in=taglist)
You may need to change some names from this example (most likely the tags in tags__name__in), see https://docs.djangoproject.com/en/1.3/ref/models/relations/.
I have a field in a model that I want users to feel like they can write an arbitrary amount of text in. Django provides a CharField and a TextField in the models. I assume that the difference is that one of them is a char(max_length) and the other is a varchar internally.
I am tempted to use the TextField, but since it doesn't respect max_length, I am somewhat wary of someone dumping loads of data into it and DOSing my server. How should I deal with this?
Thanks!
Fields in model only represent the way data is stored in database.
You can very easily enforce maximum length in form which will validate users' input. Like this.
class InputForm(forms.Form):
text = forms.CharField(max_length=16384, widget=forms.TextArea)
...
This will make sure the maximum length user can successfully enter is 16k.
That seems simple enough, but all Django Queries seems to be 'SELECT *'
How do I build a query returning only a subset of fields ?
In Django 1.1 onwards, you can use defer('col1', 'col2') to exclude columns from the query, or only('col1', 'col2') to only get a specific set of columns. See the documentation.
values does something slightly different - it only gets the columns you specify, but it returns a list of dictionaries rather than a set of model instances.
Append a .values("column1", "column2", ...) to your query
The accepted answer advising defer and only which the docs discourage in most cases.
only use defer() when you cannot, at queryset load time, determine if you will need the extra fields or not. If you are frequently loading and using a particular subset of your data, the best choice you can make is to normalize your models and put the non-loaded data into a separate model (and database table). If the columns must stay in the one table for some reason, create a model with Meta.managed = False (see the managed attribute documentation) containing just the fields you normally need to load and use that where you might otherwise call defer(). This makes your code more explicit to the reader, is slightly faster and consumes a little less memory in the Python process.
I am using James Bennetts code (link text) to create a dynamic form. Everything is working ok but I have now come to the point where I need to save the data and have become a bit stuck. I know I can assess the data returned by the form and simply save this to the database as a string but what I'd really like to do is save what type of data it is e.g. date, integer, varchar along with the value so that when it comes to viewing the data I can do some processing on it depending on what type it is e.g. get dates greater than last week.
So my question is how do I access what database type the form element is based on what type of form element it is e.g. a django.forms.IntegerField has a database field type of int, django.forms.DateField would be a date field and django.forms.ChoiceField would be a varchar field?
Why do you want to know, what kind of database field you are using? Are you storing information from the form through raw sql? You should have some model, that you are storing information from the form and it will do all the work for you.
Maybe you could show some form code? Right now it's hard to determine, what exactly you are trying to do.
I can not understand the exact problem, so forgive me if I get things wrong.
If you are using models, then you don't need to know about database-level data types. They are defined by django according to your model fields.
However, since you are talking about dynamic forms (I've read the article), you are probably not working with models, at least not directly. In that case, it should not matter as well, because you are using form validation so, for example, you can be absolutely sure that an integer comes out of a forms.IntegerField field, unicode comes out of forms.CharField and so on.
If you are writing your database-interaction routies by hand (raw sql), then you have to map python-types to db-types yourself, for example <type 'int'> goes to a column of type integer (or something), <type 'datetime.datetime'> goes to a datetime type of column (or not, this example is arbitrary) and so on. When you are using models, django does this type of mapping for you in a database-engine-independent way.
Either way, you, yourself are defining the datatypes on the python side and you or django must also define the datatypes on the db side. The choice of those types is, at times, not an automatic 1:1 type of decision, but, rather, a design decision, based on what this data is used for in your application.
Sorry, if this makes little sense, but, I must admit, that I don't quite understand the problem behind your question.
If you're using James' code, then you don't get a Model out of the form per se, rather a list of form field elements. That means that you can't save the data as a Model instance.
I think you have two choices; bundle the whole form into a JSON object and save that into a LONGTEXT variable in your database, or save each form element into a row of the database on its own, saving it into a BLOB entry. In this case, you'll need to 'pickle' the object before saving it. If you pickle the object and save it into the database, when you retrieve it and unpickle it, you'll have all the python class information associated with the object.
Trying to make this clearer - if you have the bytes; 2009-11-28 21:34:36.516176, is this a str or a datetime object? You can't tell if it's stored in the database as a VARCHAR or LONGTEXT. Which is the core of your question - you do get object information if you save it as a pickled object though.
By extension, you could save your whole Form object into the database, either as a JSON object, or pickle the object and save that.
I'm struggling with something very similar at the moment, as I'm trying to put together a dynamic form system, and thinking of going the 'individual form field element, pickled' and then saved into the database. So I'll be watching how this question works out! :)