Dotcloud -- separating primary and secondary web requests - django

I've got a web app deployed on Dotcloud where the data on each page can be quite expensive to calculate (many seconds). I want to make initial page loads as speedy as possible by returning cached information and then hitting the server with a bunch of AJAX requests that cause the full calculations to occur. But I don't want these AJAX reqeusts to jam up initial page loads for other users, so I want them queuing separately.
I'm thinking the same Django app should be used for both servers, especially because the data model is shared. So the dotcloud.yml file would like kind of like:
www:
type: python
www-ajax:
type: python
(...)
But how can I route different URLs to each class of instances? Also, I've read about Gunicorn for long requests. These AJAX requests are long, but they don't depend on external resources, besides the DB. Is this a situation for Gunicorn, and if so, is there an easy way to integrate it into the config?

If you set it up the way you are describing in your example dotcloud.yml file, you will have two different services, with two different urls. So if you want to send stuff to the ajax service, you use the ajax url, if you want the regular one you can use the www url.
To run gunicorn you could use the python-worker user and allocate an http port for the python worker, and then have gunicorn listed on the http port. It is important to note that the python-worker doesn't have nginx in front of it like the python service, so gunicorn will need to be the one listening for the traffic directly.
So to put it together it would look something like this.
www:
type: python
approot: myapp
www-ajax:
type: python-worker
approot: myapp
ports:
www: tcp
process: gunicorn -b 0.0.0.0:$PORT_WWW yourapp:app
Your process string will most likely look different but you get the picture.
You also don't need the approot, just put it there as an example.

Related

Redirect the user to another URL using Pivotal Cloud Foundry

Is there a way to redirect the incoming URL to another URL?
EX - when user hits https://www.website1.com, user should be redirected to https://www.website2.com
CloudFoundry itself does not provide a way to redirect arbitrary requests prior to requests hitting an application (i.e. at the Gorouter layer).
You have some options though:
Perform redirects on your load balancer. If you need to redirect the request as early as possible, doing it on your load balancers would be the earliest possible place. This is good for example if you want to redirect all requests from HTTP to HTTPS. It does assume that your load balancer will allow this, and not all do.
You can redirect from an application. So in your example, the application to which you have mapped the route www.website1.com would be listening and issue the redirect response to send requests to www.website2.com.
This app could be something as simple as an nginx.conf pushed using the Nginx Buildpack, or you could develop a custom application to issue the redirects. The beauty of this is that you can map any number of routes (including wildcards) to the app and a single Nginx server, configured correctly, could issue redirects for all of them. If you are redirecting a lot of separate URLs to one central URL, this works well.
If you have an existing application on www.website1.com and cannot replace it with an Nginx app to do the redirects, you could modify your custom application to perform the redirects. You don't have to use Nginx, it is just a convenient and very low overhead way to do redirects.
You could use a route service. Route Services are able to filter requests and so you could use one to intercept requests and issue redirects for certain routes/domains.
A route service can make some sense if you have complicated routing logic or if you have many applications to which you need to apply the logic. The reason is that a route service requires a custom application to be the route service. So if you have one application and you're trying to redirect requests for that one application to somewhere else, it doesn't make sense to add a second application into the mix just to perform some redirects. Now, if you have 500 applications and they all need some sort of redirect logic, it could make a little more sense to use a route service.

Redirecting API requests in Django Rest Framework

I have a two-layer backend architecture:
a "front" server, which serves web clients. This server's codebase is shared with a 3rd party developer
a "back" server, which holds top-secret-proprietary-kick-ass-algorithms, and has a single endpoint to do its calculation
When a client sends a request to a specific endpoint in the "front" server, the server should pass the request to the "back" server. The back server then crunches some numbers, and returns the result.
One way of achieving it is to use the requests library. A simpler way would be to have the "front" server simply redirect the request to the "back" server. I'm using DRF throughout both servers.
Is redirecting an ajax request possible using DRF?
You don't even need the DRF to add a redirection to urlconf. All you need to redirect is a simple rule:
urlconf = [
url("^secret-computation/$",
RedirectView.as_view(url=settings.BACKEND_SECRET_COMPUTATION_URL))),
url("^", include(your_drf_router.urls)),
]
Of course, you may extend this to a proper DRF view, register it with the DRF's router (instead of directly adding url to urlconf), etc etc - but there isn't much sense in doing so to just return a redirect response.
However, the code above would only work for GET requests. You may subclass HttpResponseRedirect to return HTTP 307 (replacing RedirectView with your own simple view class or function), and depending on your clients, things may or may not work. If your clients are web browsers and those may include IE9 (or worse) then 307 won't help.
So, unless your clients are known to be all well-behaving (and on non-hostile networks without any weird way-too-smart proxies - you'll never believe what kinds of insanity those may do to HTTP requests), I'd suggest to actually proxy the request.
Proxying can be done either in Django - write a GenericViewSet subclass that uses requests library - or by using something in front of it, e.g. nginx or Caddy (or any other HTTP server/load balancer that you know best).
For production purposes, as you probably have a fronting webserver, I suggest to use that. This would save implementation time and also a little bit of server resources, as your "front" Django project won't even have to handle the request and keep the worker busy as it waits for the response.
For development purposes, your options may vary. If you use bare runserver then a proxy view may be your best option. If you use e.g. Docker, you may just throw in an HTTP server container in front of your Django container.
For example, I currently have a two-project setup (legacy Django 1.6 project and newer Django 1.11 project, sharing the same database) and a Caddy server in front of those, routing on per-URL basis. With a simple 9-line Caddyfile things just work:
:80
tls off
log / stdout "{common}"
proxy /foo project1:8000 {
transparent
}
proxy / project2:8000 {
transparent
}
(This is a development-mode config.) If you can have something similar, then, I guess, that would be the simplest option.

Service Worker served from different port on a local machine

I'm trying to develop a PWA for our sites. In production and staging, we serve everything from one domain. However, in development on a local machine we serve HTML from one port using Django server eg
http://localhost:8000
And the assets (including JS) using Grunt server from another port:
http://localhost:8001
The problem is that the scope of the service workers is therefore only limited to assets, which is useless, I want to offline-cache pages on the 8000-port origin.
I have somewhat been able to go around this by serving the service worker as a custom view in Django:
# urls.py
url(r'^(?P<scope>.*)sw\.js', service_worker_handler)
# views.py
def service_worker_handler(request, scope='/'):
return HttpResponse(render_to_string('assets/sw.js', {
'scope': scope,
}), content_type="application/x-javascript")
However, I do not think this is a good solution. This code sets up custom routing rules which are not necessary for production at all.
What I'm looking for is a local fix using a proxy, or something else that would let me serve the service worker with grunt like all the other assets.
I believe this resource can be of help to you: https://googlechrome.github.io/samples/service-worker/foreign-fetch/
Basically you host the Service worker on the port 8001 and the server handles it as shown in the example.
Then you fetch it from there.
The Foreign fetch is described more in details here: https://developers.google.com/web/updates/2016/09/foreign-fetch

Coherence - Cookie Session Sharing between Applications Hosted on Different Servers

Coherence - Cookie Session Sharing between Applications Hosted on Different Servers
i have some web application on different servers i need them to have shared cookie
session in browser.
i want to assign same domain to all of them with different urls.
how can i implement this?
is it actually gonna work?
i want to do it with virual host on a proxy server.
The first way that comes to mind is to create a symbolic link in your DocumentRoot to a mounted directory which exists on another server. If you do this cross-server and for each application, then no matter which server people arrive at (due to load balancing, etc.) each server has a 'complete' set as far as apache is concerned but actually you still have the different data in its respective place.
In your /html/ directory (example DocumentRoot) you would have:
application1/
application2 -> /mnt/application2/
application3 -> /mnt/application3/
Then you'd set up the mount - for example - so a df would have:
192.168.1.2:/var/www/html/application2 ... /mnt/application2
192.168.1.3:/var/www/html/application3 ... /mnt/application3
Doing it this way keeps the guy on the same site as far as apache and his browser, etc. are concerned and you are definitely using the same domain, but essentially just splitting the file system between servers based on url.

Serving protected static media from django/nginx + streaming from a 3rd party app

We want to serve protected media from django, using something similar to the django nginx x-accel-redirect setup.
The only problem is that the static files are not located on the public facing django/nginx machine, but in a internal machine that streams the file via http/rest api.
Currently we download the file on the nginx machine and serve it via nginx x-accel-redirect, but we want to optimize this part and looking for options. x-accel-redirect has known problems with files that are streamed from another source.
We are contemplating using django itself as a quasi buffer, but are open to other options as integrating something like whizzer/twisted, or maybe even having another service altogether.
What would be the best option for serving those static files and preserving security?
Use: http://www.allbuttonspressed.com/projects/django-filetransfers
Make your own Django storage backend for the internal machine's http/rest api, that returns
a File object, and pass that object to filetransfer's server_file function.
That's how I do it in Mayan EDMS https://github.com/rosarior/mayan/blob/master/apps/documents/views.py#L300
django-storages' backends could help you get started.
https://bitbucket.org/david/django-storages/wiki/Home
Update:
Django_resto appears to have an http based storage class
https://github.com/aaugustin/django-resto/blob/master/django_resto/storage.py#L62
I had success doing something similar using django-http-proxy. This assumes that the image server is at least as reliable as the django server.
Then in my urls, I simply mapped the url to the http proxy view, something like:
(r'^protected/.*$', 'httpproxy.views.proxy'),
Then configured PROXY_FORMAT accordingly.
Implement a simple one-shot signature system in the media machine, using any very thin (django is OK, as it does not need to get to the database) code layer, and x-accel-redirect in nginx.
In the auth machines, generate the correct signature only when the user is allowed to get the resource, and return a 302 to the signed media.
The signature could be time-based, expiring in a fraction of a second, so a sniffer can't use the URL again.
You could use lighttpd to handle the streaming. It has a nice module to protect resources with signatures: http://redmine.lighttpd.net/wiki/1/Docs:ModSecDownload
So I'm thinking you could have nginx just proxy to the streaming server (that's lighttpd).
It's pretty easy to cook up the signature, here's a python example: init.py#cl-27">https://bitbucket.org/ionelmc/django-secdownload-storage/src/be9b18701015/secdownload_storage/init.py#cl-27