I want to select some x % from a mysql Django model. for example if there is 1 million device id stored into the table I want to select 20% of 1 million device id.
Can any one help me with this with optimised way to do it. I am looking to achieve this using Django ORM
if you want to use orm, i think you just request count(), and then create list of pk by using loop. and at the end you just use Model.objects.filter(pk__in=<list of pk>).
Related
I’m building an model in Azure Analysis Services. The model should contain only data for the last 3 months and is processed every day.
I have a separate dimension for date that has a relation with a fact table using a datekey. I’m using a power query to only load the last 3 months in the date dimension. In the power query to load the fact table I used Table.nestedjoin to only load the rows that have a value in the date table.
When I do this, the processing of the model takes forever. After some troubleshooting I saw that the query Analysis Services is using to retrieve data from the SQL database retrieves all rows. So, Am I correct saying AS load all data before it merge the rows? Is there a way to change this? Or is there a better way to a chief my solution?
Kind regards,
Joins are super slow in Power Query. You should avoid them if you can do it in the datasource or use normal relationships in the data model.
Also, you can setup the date dimension in DAX and dynamically populate it to contain only dates present in the FACT table.
As for the load of all the data, it could be because the data is fetched as is, and only then power query applies the transformations (the join).
You can modify the query in the Power Query Editor / Advenced Editor to add a where clause direclty in the query
I have a MySQL database and i have a Django model X with more than 1 million entries. I have to develop a web site where the user is able to search over 1 million items and select multiple items. I access these items through a ManyToMany relationship using Django forms with a MultipleChoiceField. But it's taking too much time when Django tries to load the 1 million items in a Django Form. How can i implement this efficiently in Django? The user should be able to search over the 1 million items efficiently and select multiple items.
I already tried using django-select2. This let me search, filter and select multiple items but it's still very slow.
Thank you in advance
I am developing an application in which I am explicitly using memcache with Google Appengine's NDB library. I want something like this.
1) Get 100 records from datastore and put them in memcache.
2) Now whenever user wants these records I would get these records from memcache instead of datastore.
3) I would invalidate the memcache if there is a new record in datastore and then populate the memcache with 101 records.
I am thinking of an approach like I compare the number of records in memcache and datastore and if there is a difference, I would update the memcache.
But if we see documentation of NDB, we can only get count by retrieving all the records, and this is not required as datastore query is not being avoided in this way.
Any help anyone? Or any different approach I could go with?
Thanks in advance.
Rather than relying on counts, you could give each record a creation timestamp, and keep the most recent timestamp in memcache. Then to see if there are new records you just need to check if there are any timestamps newer than that, which assuming you have an index on that field is a very quick query.
I'm new to PowerBI and am running into the following problem:
When creating a powerbi data import it doesn't import all the records from the database.
My setup is Azure SQL Database with 37138 records in a table.
Only 20001 records showing in my dashboard.
Any idea what the limitation could be?
I removed 3 varchar(max) colums. that increased my records from 2519 to 20xxx. My database contains 37138 records.
How can I retrieve all records?
You can import up to 5 million rows per table, up to 1 GB of data per dataset. But I have doubts it will be wise move. For 5 million rows you may want to consider direct query mode instead.
Please, provide more information about your import process and how you count the number of rows in both places, if you want to get an answer about that part of your question too.
I've found my problem.
I forgot I had a filter on a table which I forgot.
removed the filter and I see all the records!
I have an admin site and on each page I want to display the counts for each of 5 tables.
I understand how to get the count out of the tables, but I am looking for help as to where to perform the "count" action and a best practice for storing or caching those values to be used in the blade. There are only 5 so it's not a huge load on the database.
you can perform the count in controller. But if your db tables update frequently, I think is not so right to use cache, because if you display the cached result may not be accurate.
Laravel cache