optimizing data mining from google form for online statistics for recommendation system (Flask app delpoyed on Heroku) - flask

I'm building the Flask app delpoyed on Heroku: recommendation system for the field of future study.
Now I'm stuck on the next question: for now, pupils can fill Google Form, their data adding to Google Sheets, and based on that I'm doing offline statistics. And now I want to implement live-statistics. Because of some "multiple response" questions, there is a problem to take new data from Google Sheets of form without converting data. If I will convert all responces each time, all of that will be not optimized.
What is the best way to implement live-statistics: I don't want use Heroku database and I want to use my Google Sheet. Maybe there is some way to create and update csv file online after each new response?
Any suggestions?
Thank you in advance!

As suggested by Tin, you can use the Sheet API directly; here is a useful guide to get started for Python.
However, since you mention you want to use your Sheet directly, it would be a good idea to take a look at Google Apps Script, it can be quite useful for working with Google products directly and it's easy to use.
You can try with this tutorial on how to get started with Apps Script to work with Sheets. If you have any further questions, feel free to post them separately, the community will be more than happy to help.

Related

What is a simple way to make a basic website with graph database backend for newbie?

Graph database systems like ne04j seem to becoming very popular of late. I am a newbie user who is trying to set-up a website that needs to display some data which is best displayed using a graph database (essentially i will visualize this on the website).
I am designing this website using CMS Drupal but i am open to changing to other CMS or in worst case a web application framework like Django. Can anyone suggest me a basic template, website or extension to a CMS that enable graph databases or a good way to start learning web app development using graph databases for a noob ?
Well, if you're curious about using Drupal to manage graphs, I'd recommend looking over these contributed modules:
http://drupal.org/project/graphapi
http://drupal.org/project/charts
http://drupal.org/project/views_charts
http://drupal.org/project/charts_graphs
There is http://structr.org/, a CMS built on Neo4j and very Open and capable. Other bindings, you might take a look at REST bindings and neo4j embedded bindings to get going.
If you're looking for even more alternatives for charting in Drupal, then you may want to start from the Comparison of charting module, which includes information and links about quite a lot of charting related modules in Drupal.
If you're aware of other modules not included in it, please update that docu if you can, or let me know about it.

Is there a way to extract data from Nagios as if it was a web service?

My situation: I'm working on a web monitoring dashboard that assembles informations from different applications and sources and generate graphs, info graphics and reports.
The applications I'm trying to integrate are CACTI, Nagios, and other local private monitoring tools. I had no problem to integrate these applications, except for Nagios (I don't have much experience with it).
What I want to know is if there is a way to use Nagios as a Web Service, or something similar, so I can expose some of the informations and use it to generate my own reports on my dashboard application.
Is it possible to do that without any epic effort?
thanks for reading.
Nagios 4.x starting with version 4.4 now includes CGIs for JSON output. Installing the newest version of Nagios might be the easiest way to go.
See the announcement here.
Review the slides from Nagios World Conference 2013 here.
The Check_MK Multisite GUI (Web base GUI using MK Livestatus) offers a web service mode, where you can send queries/commands as URL parameters and get the response as JSON in the body.
The trick is: Create a view in the GUI, which fits your needs. Then extract the URL of that view and add the parameter output_format=json. Now you should have the output in a parsable format.
For example, this URL should give you a JSON list of all services:
check_mk/view.py?view_name=allservices&output_format=json
You can try:
1) MK Livestatus http://mathias-kettner.de/checkmk_livestatus.html
it's not web service but it can give current data without any complicated action. All you need redirect this data.
2)status-json plugin http://exchange.nagios.org/directory/Addons/APIs/JSON/status-2Djson/details which return data in JSON format.
3)NagiosWS plugin but I wasn't able to get to work it yet. I think it can be done for Nagios 2.x
4)GroundWork Foundation plugin. I think I will try use it now.
I was able to get to work 1 and 2 solution now.
Otherwise you can use Icinga which can give you some JSON or XML output. Icinga is fork of Nagios and can be installed with saving all your nagios data and plugins. At least it written on Icinga's site =) They have some other solution like PHP lib.
Sorry, I cannot post only 2 link while I'm newbie on this site.
Best regards.
Worked for me - MK Livestatus http://mathias-kettner.de/checkmk_livestatus.html it's not web service but it can give current data without any complicated action. All you need redirect this data.

HTML5 Offline storage web framework

I am looking for a web app framework which can automatically generate an HTML5 offline storage based app, so while the users become disconnected they still can view the data which normally is stored on a server
Also currently I am using Django and it would be great if there was a framework which could pull data from Django and present that as an offline app.
From the related questions suggested by stackoverflow, while writing this question, I found one interesting link mentioning that GWT has such functionality, I would like to know more about that if possible and if it can generate an HTML5 offline app
Thanks in Advance
Rather than server-side frameworks, you should be taking a look at JavaScript frameworks.
Dojo Storage will transparently select between providers such as Google Gears, Adobe AIR or plain old HTML 5 local storage. Dojo 1.5 - dojox.storage: http://dojotoolkit.org/api/1.5/dojox/storage
There's also jQuery local storage: http://plugins.jquery.com/project/saveit
... or jStorage, which can act as a storage plugin for jQuery, Prototype or MooTools: http://www.jstorage.info/
With any of these, you should be able to use a quick little AJAX call to pull (JSON perhaps) data from your server and use one of these tools to help minimise your storage code.
You're talking about a standalone app, not a django app.
This can be done with javascript (jQuery, Sproutcore, JavascriptMVC, Pyjamas ...) or Adobe AIR, or...
Pulling data from Django is just a matter of setting up a syncing method, most probably using JSON, to fill up the browser local storage. So this is not django-specific at all.
If you want a standalone django app, this can be done if you bundle in a python desktop app django with a built-in server, that's another question
You could suggest the users to create web apps or use google gears instead... I don't know if this will fill the question, but, i'm in the same way. However, I'm developing an governamental solution who will run only for some kind of people, so, I can have a few control about the user's environment... All you need to do is to use jquery to detect if user has a live connection, or offer to the users a 'preferences' page where you define the behavior of the page itself...
Some info about offline cache: http://diveintohtml5.ep.io/offline.html
PS.: In another post in stackoverflow, I 've found another question: html5 offline caching with php driven sites... The last Post said:
HTML5 offline caching does not work to make your pages interact; it works only to make a
particular page available offline. Basically, it works on a URL-by-URL basis. If you
absolutely need offline functionality, you will be forced to make it work in JS.
Also, make sure your manifest includes all resources used by all pages.
Hope this helps!
Hope it helps!!

Web technology for a first small web project

I'm a C++ developer with basic Python skills. Here's the task, a friend of mine is running a small company and he asked me if I can make a website for him. I have no real deadline so I think it's a perfect opportunity to try sth new and do some web development.
User has to be able to add photos, change texts ect.
Do you think that Django would be an overkill for this kind of project? I have no experience with it. Perhaps I should try to customize some blog engine or Google Sites?
You may be interested in Google App Engine (http://code.google.com/appengine) which recently exhibited a rise in popularity. The application runs on Google's servers, eliminating the need to maintain Apache and worry about up-time. You basically get a Django-based solution with a data store, with an SDK which allows you to conveniently develop an application on your desktop and then upload it to appspot.com for everyone to use.
The documentation is great, and even if you eventually decide not to use it, the tutorial is excellent for getting you up to speed on Python and webapp design. There is also a codelab which contains a simple Wiki example.
If you need to set up quickly a very simple website, Wordpress can be the perfect choice.
Wordpress is born as a blogging platform, but in the newer version you can manage pages, contact form and so on. And you can find good plugins to expand its capabilities.
The administration interface of wordpress is clean and easy to use, the page or article editor is simple and powerful. Add an image in a post or in a page is easy and intuitive as in no other opensource CMS I've seen.
Django is a pretty flexible framework, it tends to scale well both up and down. It may be overkill to have to learn the whole API for just a simple site, but if you're looking to learn something new, and have some time to spare, then it's a fun platform to learn and work from. My suggestion would be to install the API and have a play around with it, read the Django book and see how you get on.
Maybe before diving into Joomla or Django a first step should be working with the technologies in a raw, pure way. Create a simple web page that excutes some server side script (python? php?) that retrieves data from a mySQL database and displays it. Throw in a little javascript too. Just so that you feel comfortable with the bare-bones fundamentals.
Then when you dive into the big frameworks and libraries, they won't seem so magical.
If you're really asking what the best tool for the job is, then you are going to have to give a little more detail than "first small web project." If what you are asking is more along the lines of you want to learn web application development and what you already know is C++ and python so what would be a good web application stack where you can leverage your current skillset, then here are some more suggestions.
Django is pretty cool. Every one has already talked about it here. As mentioned earlier, the google app engine is sort of based on Django.
Also, consider zope which is another python based web application container.
If you wanted to leverage your C++ knowledge instead, then consider Wt.
Why don't you take a look at the hello world examples for all three and decide which one looks the most interesting to you?
There are a multitude of website frameworks and kits that you could go with as opposed to starting development work from the ground up. It really depends on what kind of technology you want to go with, and how comfortable you feel with the language's potential. As mentioned above, there are things like Joomla, and If you want to stick to PhP frameworks, you could also use Drupal, which has a ton of documentation and support, and is relatively easy to understand.
If you want to venture into the Microsoft realm, you could look into DotNetNuke. It too is much like Drupal and Joomla, so it's a nice CMS framework, which you might feel more comfortable with since ASP.Net is based off of a programming language as opposed to a a scripting language, unlike Classic ASP or PhP. If you're going to go the Microsoft route, I also recommend taking a look at their BizSpark developer program - but that's more dependent on how serious your friend is on running his site as a business.
Then you also have a lot of things offered up by Google, such as their charts and visualizations -- which doesn't seems like something you're looking for, but for future reference, or cool little things like a map mashup.
It really depends on how much you're willing to learn and how much time you have and sometimes not having a rough deadline is the worst possible thing that could happen. Naturally everything starts to take longer. By the time you're done, the tech is either out of date, or you've taken so long that the business plan has changed.
using django it is easy, there is already an application for photo albums available at http://code.google.com/p/django-photologue/ it becomes with tagging too and maybe you just need to add some jquery effects to get done your site.
To get this done you just create a django project, add the applications to your settings, configure your urls and templates, and thats it.
Also i f you need a small version you can just use this code as a base of your new app.
have fun with django!!
sergio
Pylons is an excellent Python web framework based on WSGI standard. It has ported many of the features of Ruby on Rails so is great for rapid development.
I've been using it for a few years now combined with SQLAlchemy for my database layer and I've found it perfect for development of all my web sites.
It is easily good enough for production sites too.
The one downside I've found is slight complications getting it deployed on shared hosting but as long as you have shell access to your host, you should be fine.
You may try to install and customize Joomla (http://www.joomla.org/) It's full of stuff, easy to use and easy to customize also.
Everybody reaches for a framework, but, assuming this is running on Apache, why not just server side includes, and, since you have Python experience, Python CGI scripts that emit JSON for use in the UI by Javascript. By going with CGI/SSI you defer your decisions about framework and/or templating system until later, when you have more experience under your belt, but you should be able to get a reasonable amount of re-use of whatever Python code you write.
I'll cast my vote for Drupal (http://www.drupal.org)
I think LOVDbyLess might do exactly what you want. If all you need is basic blog and photo upload and a simple SNS, then check out this Ruby-on-Rails open-source thing. It's been evolving with new features and is pretty easy to set up.

Sharepoint Web Services Tutorial

I'm trying to upload documents to SharePoint using web services attaching custom metadata to the files. I've searched but have not found a good tutorial covering all these topics. Can anybody point me in the right direction?
Here's why I think I need to use web services:
I'm developing on XP and the Sharepoint object model is not remotable. This means any code which has "using Microsoft.Sharepoint" is out :-(
I'm looked into the CopyIntoItems web service but am having trouble implementing it myself. I was hoping for a clear tutorial. I've tried using the sample code from http://msdn.microsoft.com/en-us/library/copy.copy.copyintoitems.aspx , but I'm not sure what my sourceURL should be. Also, since I can't use "Microsoft.Sharepoint" references, I'm wondering what my Fields will look like? (Is this my metadata?) Also, I'm curious as to why only Website projects allow me to add a web service.
Once the file is "in" Sharepoint using that web service, I'll have to use another one to update custom columns, or metadata. Some of these are freeform text, but other must match entries in lists or lookups. I haven't found any information on this yet.
Thank you for your help!
Here is some code http://geek.hubkey.com/2007/10/upload-file-to-sharepoint-document.html
As for why it is that is the way because Microsoft wrote it that way :). Some people have written custom web services that combine them, http://www.sharepointblogs.com/ssa/archive/2006/11/30/wsuploadservice-web-service-for-uploading-documents-into-sharepoint.aspx
Using the built in web services you have to upload the file and upload CAML which contains the columns. Another option if you are using a MS-Office document is to make sure the author fills in the properties in the document then you can have those fields displayed in sharepoint.
Here is some stuff on the Sharepoint Designer - http://office.microsoft.com/en-us/sharepointdesigner/FX100487631033.aspx
Hope that helps a little.
You can link to the Sharepoint 2007 training from here: http://office.microsoft.com/en-us/training/HA102358581033.aspx
The designer I believe has a WS example in it.