I built a shiny app which works with data that needs frequent update. The app doesn't change, just the data. At the beginning, i used to update my data locally, and republish my app every time the data had been updated, which i quickly found quite annoying.
I then started to store my datasets online (on arcgis online, for various reasons) so that i wouldn't need to republish my shiny app anymore, just need to handle the process of data updates.
The problem is that my app is quite slow as the datasets are very big.
Now i would like to transform my datasets as api's so that the requests coming from shiny could be more targeted.
But i don't know really know to do that. Handling the update of datasets on arcgis online through an R script was fine. But updating the same datasets as hosted feature service, i can't make it work.
Would anyone have an idea?
Or more general question, if moving away from my Arcgis online storage, what would be the best way to store data that needs frequent updates and that feed shiny apps?
You can look into caching data using 'Pins' package
Related
I know that this question can be mostly answered generally for any Web App, but because I am specifically using Shiny I figured that your answers may be considerably more useful.
I have made a relatively complex app. The data is not complex, but the user interface is.
I am storing the data in S3 using the aws.s3 package, and have built my app using golem. Because most shiny apps are used to analyse or enter some data, they usually deal with a couple of datasets, and a relational database is very useful and fast for that type of app.
However, my app is quite UI/UX extensive. Users can have their own/shared whiteboard space(s) where they drag around items. The coordinates of the items are stored in rds files in my S3 bucket, for each user. They can customise many aspects of the app just for them, font size, colours of various experimental groups (it's a research app), experimental visits that are storing pdf files, .html files and .rds files.
The .rds files stored can contain variables, lists, data.frames, reactiveValues, renderUI() objects etc.. So they are widely different.
As such I have dozens of rds files that are stored in a bucket and everytime the app loads each of these .rds files need to be read one by one in order to recreate the environment appropriate for each user. The number of files/folders in directories are queried to know how many divs need to be generated for the user to click inside their files etc..
The range of objects stored is too wide for me to use a relational database - but my app is taking at least 40 seconds to load. It is also generally slow when submitting data as well, mostly because the data entered often modified many UI elements that need to be pushed to S3 again. Because I have no background in proper Web Dev, I have no idea what is the best way to store user-related UX/UI elements and how to retrieve them seamlessly.
Could anyone please recommend me to appropriate resources for me to learn more about it?
Am I doing it completely wrong? I honestly do not know how else to store and retrieve all these R objects.
Thank you in advance for your help with the above.
I have an application that uses SQLite.swift to save a list of shopping items. It works quite well, and is deeply rooted in the app. But I've been getting asked to make a WatchOS app that has the same items on your wrist.
My issue with this, is that all the ways of syncing the data between two devices is through NSUserDefaults. But doing that would mean uprooting my entire way of saving data, so is there a way to sync my SQL database onto WatchOS by using SQLite.swift?
Yes of course: sync the database (or the shared part of it) yourself.
Read the "Communicating with the Counterpart App" chapter of https://developer.apple.com/reference/watchconnectivity/wcsession and choose the best way to transfer your information.
I am building a dashboard that will monitor production data, and am able to access this data via web services. The data changes every 1 minute, so I would like to have a page with 4 charts/gauges (the number of systems I am monitoring) that would get the data pushed to them with a successive web service call.
Can anyone suggest a good charting kit that would work well with C#? And would SignalR be a good fit here do you think? I have read that node.js and socket.io are options, but I have no experience with node yet. I would like something along the lines of DevExpress. Perhaps jquery and something on the front end works here as well? Thanks!
For such a dashboard SignalR is definitely a good fit if you already work with .NET and ASP.NET. For a web dashboard in particular, a good graphic library is Raphael, which is open source and pure JavaScript. It's simple and straight to the point, but often less is more. You can build interesting kinds of charts with it.
This project is maybe interesting for you as a sample of those 2 technologies together. If you press the skulls to raise errors, they will be triggered on a backend simulator and pushed to the dashboard using SignalR. You will notice a piechart graph there, which is done using Raphael and updates live when new errors are received.
The code of the project is here, it's a bit complex but maybe you want to have a look anyway. It's based on SignalR 1.x, but overall concepts are still the same.
I am currently trying to figure out he best practice in order to design my web services between a django administrated database (+ images) and a mobile app. My main concern is how to separate a bulk update (send every data in the database and all the files on the server) and a lighter, smaller update with only the new and / or modified objects (images or data.)
I have had access to a working code-base using a cronjob and states for each data field (new, modified, up to date) to generate either a reference data file or an update file. I find it to be very redundant and somewhat unelegant, in contradiction with the DRY spirit of Django (there are tons of lines of code, making it nearly unmaintainable.))
I find it very surprising that this aspect is almost un-documented, since web traffic is a crucial matter in mobile developpment.. Fetching everytime all the data served quickly becomes unsustainable as the database grows..
I would be very grateful for any lead or advice you could give me :-) Thx in advance !
Just have a last_modified DateTimeField in your table, and in your user's profile a last_synchronized DateTimeField. When the mobile app wants to synchronize, send the data which was modified after the last synchronization run, and update the last_synchronized field in the user's profile.
This'll be my first question on this platform. I've done lots of development using Flex, WebORB and ASP.NET. We have solved Concurrency problems with messaging (Pessimistic Concurrency Control). This works pretty good but it also makes the whole application dependent of the messaging. No messaging, no concurrency control.
I know that ASP.NET has version control in DataSets, but how would you go and use that if you are working on a RIA. It seems hard to go and store each dataset in the session of the client... So, if the Client would like need all products, I would need to store the dataset in the session of the client. When the client would change something to a product and save the product, I could then update the dataset (stored in the session) and try to save it...
Seems a lot of work and a lot of memory that will be used (because those products will be kept in the memory of the client, so the dataset needs to be kept on the server side session).
I think the most easy way would be to provide all DTO's with a version number. If the client would try to save a DTO, I could compare the version number with the one in the database.
Lieven Cardoen
This is something I've done before - as the original data was coming from an SQL Server database we just used a rowversion typed column in each DTO to determine if it had changed while the user was working on it.
At this point you can either barf on the error or try and figure out a way to merge the changes, but at least you can tell that it's changed underneath you :)