I have a table linked and getting data from Zapier. The records come 200rows per minute. Logged-in users are supposed to be picking these data and working on them. The problem is that there has been some confusion as to know which record has been worked on and which one has not been worked on.
I have built a table to help show records which have been worked on by ticking the Treated tab Table structure.
I hope I can divide the records equally amongst the logged-in Users so as not to underwhelm or overwhelm any of the users.
Related
I'm currently trying to build a web app that would allow many users to query an external API (I cannot retrieve all the data served by this API at regular intervals to populate my PostgreSQL database for various reasons). I've read several thing about ACID and MVCC but still, I'm not sure there won't be any problem if several users are populating/reading my PostgreSQL database at the very same time. So here I'm asking for advice (I'm very new to this field)!
Let's say my users query the external API to retrieve articles. They make their search via a form, the back end gets it, queries the api, populates the database, then query the database to return some data to the front end.
Would it be okay to simply create a unique table to store the articles returned by the API when users are querying it ?
Shall I rather store the articles returned by the API and associate each of them to the user that requested it (the Article model will contain a foreign key mapping to a User model)?
Or shall I give each user a table (data isolation would be good but that sounds very inefficient)?
Thanks for your help !
Would it be okay to simply create a unique table to store the articles returned by the API when users are querying it ?
Yes. If the articles have unique keys (doi?) you could use INSERT...ON CONFLICT DO NOTHING to handle the (presumably very rare) case that an article is requested by two people nearly simultaneously.
Shall I rather store the articles returned by the API and associate each of them to the user that requested it (the Article model will contain a foreign key mapping to a User model)?
Do you want to? Is there a reason to? Do you care who requested each article? It sounds like you anticipating storing only the first person to request each article, and not every request?
Or shall I give each user a table (data isolation would be good but that sounds very inefficient)?
Right, you would be hitting the API a lot more often (assuming some large fraction of articles are requested more than once) and storing a lot of duplicates. It might not even solve the problem, if one person hits "submit" twice in a row, or has multiple tabs open, or writes a bot to hit your service in parallel.
If I go to the iCloud dashboard and search (Filter:) for a record by exact recordName, the record is found. However, if I search using any range or inexact methods (i.e. search by date range etc.), the record is not found. Also, the record is not returned inside my Objective C program when I query all records of that type. It appears to be random records that have this characteristic. I recently inserted 40 or more records of this type (via software) and this behavior is only seen in 6 of them. I have tried using the dashboard to update the affected records, in hopes that this might trigger "update" events on the backend. Nothing that I have tried fixes this!!
I should add that my data is in the Development environment, not Production.
I may have found a work around for this problem. In the iCloud dashboard, I deleted the index QUERYABLE on the recordName field (the only index on this field) and then added it back. I then waited about 5 minutes or so and this seemed to fix the problem. However, I believe that this is, perhaps, a bug in the backend iCloud server.
The following image shows a rough draft of my proposed database structure that I will develop for Django. Briefly, I have a list of ocean Buoys which have children tables of their forecast conditions and observed conditions. I'd like Users to be able to make a log of their surf sessions (surfLogs table) in which they input their location, time of surf session, and their own rating.
I'd like the program to then look in the buoysConditions table for the buoy nearest the user's logged location and time and append to the surfLog table the relevant buoyConditions. This will allow the user to keep track of what conditions work best for them (and also eventually create notifications for the user automatically).
I don't know what the name for this process of joining the tables is, so I'm having some trouble finding documentation on it. I think in SQL it's termed a join or update. How is this accomplished with Django?
Thanks!
A little background. I've been developing the core code of an application in python, and now I want to implement it as a website for the user, so I've been learning Django and have come across a problem and not sure where to go with it. I also have little experience dealing with databases
Each user would be able to populate their own list, each with the same attributes. What seems to be the solution is to create a single model defining the attributes etc..., and then the user save records to this, and at the same time very frequently changing the values of the attributes of the records they have added (maybe every 5~10 seconds or so), using filters to filter down to their user ID. Each user would add on average 4000 records to this model, so say just for 1000 users, this table would have 4 million rows, 10,000 users we get 40million rows. To me this seems it would impact the speed of content delivery a lot?
To me a faster solution would be to define the model, and then for each user to have their own instance of this table of 4000ish records. From what I'm learning this would use more memory and disk-space, but I'd rather get a faster user experience as my primary end point.
Is it just my thinking because I don't have experience with databases? Or are my concerns warranted and I should find a solution as to how to be able to do the latter?
This post asked the same question I believe, but no solution on how to achieve it. How to create one Model (table) for each user on django?
I've started a django project that will include an analytics app. I want that app to use either couchDB or mongoDB for storing data.
The initial idea was (since the client already is using Google Analytics) to once a day/week/month grab data from GA, and store store it locally as values in database. Which would ultimately build a database of entries - one entry per user per month - with summed values like
{"date":"11.2011""clicks": 21, "pageviews": 40, "n": n},
for premium users there could be one entry per user per week or even day.
The question would be:
grab analytics from GA, do a sum entries for clicks, visits etc.
or
store clicks and whatever values locally and once a month do sums for display ?
Lukasz, unless Google Analytics has really relaxed their privacy levels, you're not going to be able to access user-level records (but check out the answer here: Django saving the whole request for statistics, whats available?)
Right, old question but I've just finished the project so I'll just write what I did.
Since I didn't need concurrency and wanted more speed approach, I found that mongodb is better for that.
The final document schema that I've used is
{'date': '11.2009', 'pageviews': 40, 'clicks': 13, 'otherdata': 'that i can use as filters'}
The scope of my local analytics is monthly, so I create one entry in mongdb per user per month, and update it each day. As said just now, I update data daily, and store only summaries and averages of those.
What else. Re: Jamie's answer... The system is using GA events, so I've got access to all data that i need.
Hope someone may find it interesting.
cheers and thanks for ideas !