I'm a php developer, trying to develop a website to stream on-demand music to the users.
After a lot of googling I'm confused about which kind of server or tools should I use? I've seen some like WOWZA or SHOUTCAST, but I don't know which one is the best for my needs.
I want to provide high quality audio files. So maybe I should use 320kbps mp3 format or something else but with the same quality.
I don't need live streaming. I just need on-demand streaming of the music files and the ability for the user to create his/her own playlists.
The user shouldn't be able to download the music files.
Icecast/SHOUTcast are not appropriate for your use. They take a single stream and send it to multiple connections simultaneously. They are not "on-demand" servers where each user can listen to separate content.
For your use case, you can implement something in PHP. All you are really doing is sending media files to the client. You mentioned that you wanted to keep the client from downloading those files... this is impossible. If the client can play it, the client can save it, and there is no way around this. However, there are some things you can do that prevent it from being as easy as linking to a file.
Don't store your audio in the web server's document root. All media files should be served only from your PHP scripts. This gives you control over the requests coming in. Look into readfile(). This also allows you an easy path to get off of simply loading files from disk (which you will want when you start to grow beyond 100k media files).
A URL for a media file should only work once, and for a specific user session ID. Generate these URLs on the fly, with a time limit on them. If the URL for the media file is requested by someone who doesn't have a valid session on your site, don't serve it. If the link is expired, don't serve it. This is what prevents someone from getting the URL and posting it on some message board. Only valid users of your site with current valid sessions should be able to get at your media files.
Rate-limit requests. Don't allow a user to be able to download more files at a time than is needed. If they request 100 files in 1 second, don't serve them.
Implementing all of these concepts is something I leave to you. How you do so depends your needs, and is not something that is typically done in a 5-line snippet of code.
Related
if we see big websites like youtube, google drive, facebook, cloud file download sites, etc., then we will find that every link file, video, image or whatever, then the original file link will not be seen for example videos on youtube, even if we inspect the element and see the source on the video player it isn't visible, the link is just written:
src = "https://www.youtube.com/94118230-9dbf-4207-a098-de7a7ccdf7f6"
without any real address or file extension like .mp4 or others. can anyone help explain how to engineer this and whether django can handle engineering like this?
First point: an url doesn't have to point to a file. What happens when some url is requested is up to the HTTP server serving the url. Serving files from the server's filesystem is the basic default for most HTTP servers (static sites), but that's just one of the possibilies, it can as well be executing a CGI script, delegating to a pool of long running processes (typical Python wsgi app), whatever...
Second point: files extensions are mostly cosmetic. You can have a file name without any extension directly served by Apache, and you can have an url with a filename extension that is actually served by some script or other program dynamicall generating content.
IOW, there's absolutely no relation between what the url looks like and how the response is built, and "original file link" doesn't mean anything, and the youtube url you posted IS as much of a "real address" than any other url.
can anyone help explain how to engineer this
It's impossible to answer this question, at least not with a one-size-fits-all answer. If all you want is to serve a static file without extension, just rename the file without the extension, possibly tweak your HTTP server config so that it correctly handles the case, and you're done.
whether django can handle engineering like this
Any techno that is not able to "handle engineering like this" is either prehistoric or fundamentally broken.
I strongy suggest you (seriously) read about the HTTP protocol and learn to set up and configure a standard HTTP server (Apache comes to mind). Only then will it make sense to worry about specific technos (Django or else).
I am providing sensitive username and password file to the authenticated user. I want user to download the file via file_url in template through model.
File_link = models.FileField(upload_to='SAFE_DIRECTORY_PATH')
I don't feel it safe storing it in media directory
Any suggestions keeping them safe ,web app will be generating the link.
Some security notes first.
This is probably a bad idea. Storing sensitive information in plain files is probably not the correct security approach, especially if you plan to use Django's media storage backend for doing that. It leaves all files out-in-the-open.
If however you really, really, and I mean really need to do that, you should encrypt the file first before saving in Django.
Again though, if at all possible I would recommend to store sensitive information in db. In your case of storing passwords, you can use Django techniques to store that information relatively-safely such as correctly hashing passwords via pbkdf function (e.g. pbkdf or bcrypt, etc). If users will need to download that information, you can always generate the file on the fly for them for download.
Some suggestions for uploading files.
I usually assign random filenames to the uploaded files. This way at least its more challenging for the users to guess the filenames to download them. Not very security since this relies on security by obfuscation but its better then nothing. If you need a Django field which does that automatically, you can do that by making upload_to a callable (there are also 3rd party libs for doing that such as django-auxilium although for full disclosure Im the author of that lib).
Now that files are stored with random filenames, you probably never want to provide direct download links to the users for download but instead authenticate them first and then use something like X-Accel in nginx or X-Sendfile in Apache to actually serve the file to the user. The idea being that you first authenticate user in Django. Then however instead of Django serving the file, you return a special header which nginx/apache catches which contains a filepath to the file nginx/apache should serve to the user. This way you dont have to waste resources in Django to serve the file however you still get the advantage of being able to authenticate the request. There are a number of 3rd party apps for doing that as well.
Finally to protect users from downloading the media files you can use nginx (and I imagine apache) by restricting certain parts of the media folder:
location /media/protected {
internal;
alias /var/www/files;
}
In this case nginx will refuse direct user requests to /media/protected and will only allow to serve those files via X-Accel-Redirect header sent by Django. Then all you have to configure in Django is to store files in that path to make them protected:
models.FileField(upload_to='protected/myfiles')
I was looking for a solution to serve files only to authorized users and came across this post. I think it it is top google result for "django storing and providing secure files"
As the answer is rather old I wanted to share my finding:
django-private-storage (https://pypi.org/project/django-private-storage/) seems to be a good solution to this problem.
I am using dropbox for one of my application where a user can connect their dropbox folders.
Usage is such that a user can create links among the files of a folder and many more. But the problem is the moment when I stored the file information in my application, the file media information is stored with a key expires. So obviously I wont be able to use the link next time once the expiry time is met.
One way is to generate the media information every time the user is selecting a thumbnail from my application, as I already have metadata of the file.
But is there any other way (i.e by using python client or API) that I can make a folder public when a user selects it to connect with my application.
Any help would be really appreciated.
Thanks in advance for your precious time.
I think the right thing to do is to generate a media link each time you need it.
Is there a reason you don't like that solution?
I have a site which enables the user to download certain files. However I want to keep a download count for each file so going the usual way by putting the static files on a different subdomain and then letting apache do the heavy lifting is not a way as well as HttpResponseRedirecting the user to a subdomain isn't good because then the user 'sees' the proper download url and can therefore download the file without incrementing the download count. I could just build a view which then serve()s the file however i am worried about that "big fat disclaimer". How would you/did you implement this? I am quite shure I am not the only one with that problem.
About the Platform: I am using apache and mod_wsgi.
Thank you
We've implemented a system where we needed to control download access to (largish) static files, naturally not wanting Django to serve them itself. We came up with a scheme whereby the Django app, after validating that the user was allowed to download the file (or increment a counter, in your case) we would create a randomly-named symlink to the file, which Apache had access to (be careful: make sure directory indexing is off etc), and then redirect the user to that symlink to be served by Apache.
We have a "cleanup" cronjob that cleans up symlink a minute after they're created, so if they want to download it again, they have to go through Django and have it counted again. Now, theoretically they could download it more than once in that time, but is that likely to happen? You could clean up more than every minute: Apache just needs the symlink to exist at the beginning of the download, not throughout the whole thing.
I'd be curious to know how others address this problem, as I agree with the OP that it is a common scenario.
psj's answer is definitely one viable option. Another option you should investigate is putting a reverse-proxy server in-front of apache like Perlbal which supports "X-REPROXY-URL" headers.
Once you have the reverse-proxy server in place, instead of sending the user a redirect response, you can send a response with the "X-REPROXY-URL" header set to a URL where the proxy server can access but the user can't. The proxy server will then read in the file from the location you sent in the header, and then serve it out to your client. They'll do so in an efficient way and since all your Django app server needs to send is a response with a header set, it is free to handle another request.
The easiest way to do this is to use Apache's X-Sendfile header. Just set the value of the header to the file path and Apache will send the file for you. This blog post has some more details: http://francoisgaudin.com/2011/03/13/serving-static-files-with-apache-while-controlling-access-with-django/ .
I did this with django-counter not to long ago. Lets you keep track of the counts in the admin.
http://github.com/svetlyak40wt/django-counter/
Ok, I know that serving media files through Django is a not recommended. However, I'm in a situation where I'd like to serve "static" files using fine-grained access control through Django models.
Example: I want to serve my movie library to myself over the web. I'm often travelling and I'd like to be able to view any of my movies wherever I am, provided I have internet access. So I rip my DVDs, upload them to my server and build this simple Django application coupled with some embeddable video player.
To avoid any legal repercussions, I'd like to ensure that only logged-on users with the proper permissions (i.e. myself and people living in the same household, which can, like me, access the real DVDs at their convenience), but denies it to other users (i.e. people who posted comments on my blog) and returns an HTTP 404.
Now, serving these files directly using Apache and mod_wsgi is rather troublesome because when an HTTP request for the media files (i.e. http://video.mywebsite.com/my-favorite-movie/) comes in, I need to validate against my user database that the person at the other end has the proper permissions.
Question: can I achieve this effect without serving the media files directly through a Django view? What are my options?
One thing I did think of is to write a simple script that takes a session ID and a video's slug and returns some boolean indicating if the user may (or may not) access the video file. Then, somehow request mod_wsgi to execute this script before accessing the requested URL and return an HTTP 404 if the script failed. However, I don't have a clue if this is even possible.
Edit: Posting this question clarified some of my ideas for search and I've come across mod_python's file wrapper extension. Does anyone have enough experience with that to validate that it is a viable solution?
Yes, you can hook into Django's authentication from Apache. See this how-to:
Authenticating against Django’s user database from Apache