I have created an interactive Bokeh chart with various widgets which allow manipulation of the data. I now want to understand what is the standard way of sharing such a plot or how do I save it for sharing.
The plot is created with the curdoc method and then output to the Bokeh server using session.show().
#create current visualization using plot p and widgets inputs
curdoc().add_root(HBox(inputs, p, width=1100))
#run the session
session = push_session(curdoc())
session.show() # open the document in a browser
session.loop_until_closed() # run forever
Does the app trigger actual python code?
If not, you might consider reworking it as a non-server standalone document (using CustomJS callbacks, for instance). That would just generate a self-contained static HTML file that you could publish or send anywhere, and have it be immediately accessible.
If your app does rely on executing actual python code to do the work, then it needs to actually be running somewhere for users to interact with it. First off, I would suggest you make a real app that runs in the server, like the ones in the demo app gallery (see also Use Case Scenarios in the User's Guide). A real server app, i.e. one you run like bokeh serve myapp.py, is definitely preferred over using bokeh.client, especially for "publishing" scenarios (it will also be simpler/less code and more performant). Then, distributing the app could mean a few things:
You give them the script and they run bokeh serve app.py locally themselves
You "deploy" the app by leaving it running on a server with a URL that is accessible to users who you want to be able to see it
Depending on how much compute the app does, and how many users you expect at a given time, the second option could be as simple as running bokeh serve app.py somewhere. But if there is heavy compute or you expect a lot of traffic, you may need more sophisticated "scale out" deployments behind a load balancer. More information is in Deployment Scenarios in the User's Guide, and of course we are happy to help wth more extended discussions on the public mailing list. Finally, I should mention that in the near future, automated scalable publishing of Bokeh applications will be available as a feature on https://anaconda.org/
Related
Is it possible to have django running on the server and one application from django inter communicating with another python process say that I developed and fetching a response from it or even make it just do a particular action?
It can be synchronous or asynchronous; I have some idea of being asynchronous where some package like hendrix, crossbar.io or even celery can be used. But I don't understand what would be the name for this inter-communication and how should I plan the architecture for this.
Going around my head I have the two following situations I'm seeking a plan to be developed:
1.
Say I have django and an e-mail sender with the python package smtp. A user making a request to a view would make django execute my python module I developed for sending an email to a particular user (with a smpt server from google/gmail). It could be synchronous or asynchronous.
OR
2
I have django (some application) and I want it to communicate with some server I maintain; say for making this server execute some code or just fetch a file (if it is an ftp server). Is this an appropriate situation to point to the term 'microservices'? Or there is another term or workaround here?
Your first solution would be called an installable python module, just like any package you install with pip. You can have this as a separate module if you need your code to be re-usable across multiple or just future projects.
Your second solution would be a microservice. This will require setting your small module as a service that could have a REST API to communicate with and make it do whatever you intend it to do.
If your question is "what is the right approach" then I would tell you it depends on your use case. If this is just some re-usable code that you don't want to repeat over and over through our project then just make it into a separate module. While if this is a service that you expect other built services will use and rely on, then just make it into a microservice. You can use a microframework such as Flask for easier and faster setup of your service. Otherwise, if it's just some code that you will use once and serves a single functionality on your application then just write it and keep it there.
There are no rules or standards on which approach should be taken. I personally judge things depending on the use-case.
Hope this helps!
I have an automation framework that uses static html pages within its project directory to perform certain aws operations such as dynamoDB scan and Aws Lambda executions. Due to some performance bottleneck in a dependent api component for the test we are trying to move the framework to an ec2 instance with Amazon linux and run the tests from there.
Since we have methods in the TestNG class that actually uses selenium web driver to spin up a browser and open up the static page in order to perform the required Aws operations I am pretty sure this test is going to run into issues.
There are two potential approach I see for solving this issue:
Implement AWSUtil classes and use necessary aws clients to replace the web dependent logics (Will require some effort and re-engineering)
Use a headless chrome browser (or any compatible one) in order to run the web dependent steps.
I am pretty sure that number 1 can be easily achieved, just a matter of time and effort. However, would love to know if there is an easy way of accomplishing #2 since this would not require any code rewrite.
We got the same issue and been successful with puppeteer,
https://github.com/GoogleChrome/puppeteer
If you don't want to install the latest version of node, you can dockerize your tests.
puppeteer can run headless or with browser.
Hope it helps.
There is no need to change anything in your tests, just the setup and the execution. The tests can run headlessly on a Continuous Integration (CI) server. There is no out-of-the-box setup since there is no display output for the browser to launch in. However with Xvfb you can launch the browser virtually. Straight from the docs:
Xvfb (short for X virtual framebuffer) is an in-memory display server for UNIX-like operating system (e.g., Linux). It enables you to run graphical applications without a display
Depending from do you want to keep Xvfb running in the background until the process is killed, there are two options:
Xvfb :99 &
export DISPLAY=:99
run-your-tests-here
or
xvfb-run run-your-tests-here
Here is a Linux tutorial. I am using this for my Docker based Jenkins setup and works like a charm, every time.
I'm developing an tool using Django for internal use at my organization. It's used to search and tag documents (using Haystack and Solr), and will be employed on different projects. My team currently has a working prototype and we want to deploy it 'in the wild.'
Our security environment is strict. Project documents are located on subfolders on a network drive, and access to these folders is restricted based on users' Windows credentials (we also have an MS SQL server that uses the same credentials). A user can only access the projects they are involved in. Since we're an exclusively Microsoft shop, if we want to deploy our app on the company intranet, we'll need to use an IIS server to deal with these permissions. No one on the team has the requisite knowledge to work with IIS, Active Directory, and our IT department is already over-extended. In short, we're not web developers and we don't have immediate access to anybody experienced.
My hacky solution is to forgo IIS entirely and have each end user run a lightweight server locally (namely, CherryPy) while each retaining access to a common project-specific database (e.g. a SQLite DB living on the network drive or a DB on the MS SQL server). In order to use the tool, they would just launch an all-in-one batch script and point their browser to 127.0.0.1:8000. I recognize how ugly this is, but I feel like it leverages the security measures already in place (note that never expect more than 10 simultaneous users on a given project). Is this a terrible idea, and if so, what's a better solution?
I've dealt with a similar situation (primary development was geared toward a normal deployment situation, but some users have a requirement to use the application on a standalone workstation). Rather than deploy web and db servers on a standalone workstation, I just run the app with the Django internal development server and a SQLite DB. I didn't use CherryPy, but hopefully this is somewhat useful to you.
My current solution makes a nice executable for users not familiar with the command line (who also have trouble remembering the URL to put in their browser) but is also relatively easy development:
Use PyInstaller to package up the Django app into single executable. Once you figure this out, don't continue to do it by hand, add it to your continuous integration system (or at least write a script).
Modify the manage.py to:
Detect if the app is frozen by PyInstaller and there are no arguments (i.e.: user executed it by double clicking it) and if so, then run execute_from_command_line(..) with arguments to start the Django development server.
Right before running the execute_from_command_line(..), pop off a thread that does a time.sleep(2) (to let the development server come up fully) and then webbrowser.open_new("http://127.0.0.1:8000").
Modify the app's settings.py to detect if frozen and change things around such as the path to the DB server, enabling the development server, etc.
A couple additional notes.
If you go with SQLite, Windows file locking on network shares may not be adequate if you have concurrent writing to the DB; concurrent readers should be fine. Additionally, since you'll have different DB files for different projects you'll have to figure out a way for the user to indicate which file to use. Maybe prompt in app, or build the same app multiple times with different settings.py files. Variety of a ways to hit this nail...
If you go with MSSQL (or any client/server DB), the app will have to know the DB credentials (which means they could be extracted by a knowledgable user). This presents a security risk that may not be acceptable. Basically, don't try to have the only layer of security within the app that the user is executing. The DB credentials used by the app that a user is executing should only have the access that the user is allowed.
If I create a HIT in the Sandbox via Mturk's GUI, is it possible to transfer it to the Production site, or do I have to re-create the HIT manually in the Production site?
In particular, is it possible, to download .input, .question and .properties for HIT created via GUI in the sandbox, in order to use them to generate the same HIT on the Production site via the CLT?
The obvious way seems to be using Mturk HIT's layouts. However, reading the doc, I don't see how/ know whether it is possible to to do this using the CLT. The doc on HITLayoutParameter requires using CreateHIT, but this is not an available command in the CLT (only have loadHITs).
I have seen other questions Creating mTurk HIT from Layout with parameters using boto and python and Create a MTurk HIT from an existing template about ways to do it with boto but I am still wondering whether that's doable with the CLT.
The live and sandbox modes are completely separate and no transfer is possible from one to the other.
You will need to implement this programmatically by storing the specs of the sandbox HIT and creating a live HIT.
Another option is to use a service like TurkPrime.com which allows you to copy HITs from sandbox to live mode
I want to run image processing algos on server which can interact easily with web apps. The image processing algos are compute heavy and wont be available in custom built libraries. Currently I am using Ruby on Rails on Heroku for my website.
What would be the best architecture to achieve this? take images from website - run image processing algo on it - display back on website
Most of my image processing code is on C/C++.
Can i call C/C++ code from Ruby on Rails directly? Is this possible on Heroku?
Or should I design a system where C/C++ code expose some APIs which can be called by Ruby on Rails server?
Heroku typically uses small virtual machine instances, so depending on just how heavy your processing is, it may not be the best choice of architecture. However, if you do use it I would do this:
Use a background task gem to do your processing. Have this running on a separate process (called a worker rather than a dyno in Heroku terminology). Delayed Job is a tried and tested solution for background tasks with a wealth of online information relating to integrating it into Heroku, but there are newer ones like Sidekiq which use the new threading system in modern versions of Ruby. They would allow everything to be done in the dyno, but I would say that it would be useful to keep all background processing away from the webserver dynos, so Delayed Job (or similar) would be fine.
As for integrating C/C++, I haven't needed to do this as yet. However, I know it is possible to create gems that integrate C or C++ code and compile natively. As long as you're using ruby rather than JRuby, I don't think Heroku should have a problem with them. There are other ways of accomplishing this, look at SO questions specifically about this topic, such as
How can I call C++ functions from within ruby
It seems that you need to create an extension, then create a gem to contain it. These links may or may not help.
http://www.rubyinside.com/how-to-create-a-ruby-extension-in-c-in-under-5-minutes-100.html
http://guides.rubygems.org/gems-with-extensions/
I recommend making a gem as I think it may be difficult to otherwise get libraries or executables on to a Heroku instance. You can store the gem in your vendor directory if you don't want to make it public.
Overall I would have the webserver upload to S3 or wherever you're storing the images (this can be done directly in the browser without using the webserver as a stepping stone with the AWS JS API. Have a look for gems to help.)
Then the webserver can request a background task to process the image.
If you're not storing them, things become a little more interesting... You'll need a database if you're using background tasks, so you could pass the image data over to the worker as a blob in the database perhaps.
I really wouldn't do all the processing just in the webserver dyno, unless you're really only hitting this thing very occasionally. With multiple users you'd hit a bottleneck very quickly.
The background process can set a flag on the image table row so the webserver can let the user know when processing is complete. (You can poll for information using JS on the upload complete screen using AJAX)
Of course, there are many other ways of accomplishing this, depending on a number of factors.
Apologies that the answer is vague, but the question is quite open-ended.
Good luck.