I would like to write a program or script that will go through a website and download all MP3's. So for example, I would go to a website that has MP3's available for download. I would then run a script or start a program that would comb through the website downloading all or specified MP3's.
I don't know where to start or if an application already exists. Please help.
To download any resource from an HTTP protocol or from a website first, you have to know the url. To get the url you better have access to the code source and use a regex that start with http:// or https:// and end with .mp3. And/or follow any link in the source code and check if the mime type is audio/mp3.
Note: you can't go on a website and download any resource illegally. You have to respect the copyright.
Related
I am currently developing a feature in a Game in C++.
I provide an URL (eg:"https://ServerName/../FolderName/").
There are few files in this folder: .png, .txt, etc.
I need to download all these files into a specific location (For the Game to fetch them from that location)
I am able to download single file by providing the complete URL.
Any idea on how to download all files in a single HTTP request??
Any help is appreciated. Thanks in advance!
An HTTP request is asking for a single resource from the server via a unique URI. What you're asking for is to be able to ask for multiple resources from the server with a single URI, which means you don't understand how HTTP works. So the answer to your question is NO, and it's impossible.
I've built a gatsby website, but when I try to use it offline (by directly loading index.html into my browser), it fails to load the files in the assets folder, and links to other pages fails
running in windows:
after installing gatsby , I did the following:
gatsby new sample
cd sample
gatsby build
then I went to file explorer and opened the sample/dist directory and double clicked on index.html (Chrome is my default browser, but IE behaves the same)
the result is a half-loaded webpage that is missing the style sheets, javascript, images, and links are broken.
For instance, the "about" link on the first page points to "D:/about" vs. ".\about.html".
Is there anyway to make gatsby work to create a truly off-line website?
I've built a gatsby website, but when I try to use it offline (by directly loading index.html into my browser), it fails to load the files in the assets folder, and links to other pages fails
Gatsby will create a React app for you when it is built, and because most React apps use client-side routing, your links won't work with file:// URLs.
After installing the Gatsby CLI and building your site with gatsby build you should run gatsby serve which will serve up index.html with a static file server on your own machine.
See a similar answer about create-react-app here
Try using gatsby serve from the root of your project. Serve spins up a web server to serve your prod build.
Look it up on the Gatsby CLI docs on their site.
Gatsby isn't really set up to do that, unfortunately. It's a site generator, not page generator, and it expects living on a server. The result is that while the files are static, the navigation isn't.
Because I spent some time experimenting, this is what DOESN'T work:
Setting . as pathPrefix in gatsby-config.js. Gatsby lets you set path prefix, which it then prepends to all generated urls. Unfortunately, this prefix always gets "absolutized" somehow. . gets converted to /., for example.
Setting the absolute path of the file on disk as pathPrefix. Same as above - file:///path/to/file doesn't survive the build (results in file:/) and /path/to/file breaks the JavaScript.
Setting the pathPrefix to a bogus value like /NOTAPREFIX and search-replacing it in the generated files. Breaks the JavaScript, again.
Something I haven't tried but might get you somewhere, would be to disable the Single Page App functionality. It's possible, reportedly, (or maybe with this plugin?) but no good step-by-step instructions anywhere.
I am currently trying to set up a Virtual Machine and a Database Server for my website to launch off of. I decided to go with Amazon's AWS.
I have beeing working through This Tutorial and am at Part 3 now.
I got to the point where I started the Apache Web Server and went to my local DNS name in my browser, and successfully saw the Test Page.
I proceeded forward with Downloading "Drupal" and got to the section where it says "Configure Drupal". I am now trying to load my local DNS name in my browser again, only to get a 500 Server Error.
I'm not sure what could've went wrong, I only entered the commands in the tutorial for the whole section on "Install Drupal".
Thank you
EDIT: Also, what exactly does Drupal do? And do I need to use it?
Drupal is a CMS system, and you don't have to use it. It is only being used to provide an example for a website. Instead of installing Drupal, you can install your own site on the server if you prefer or even something like Wordpress if you're more familiar with that.
However, if you do want to follow the example your best bet for discovering the problem is to look into what the exact error is. You should be able to find more information about that in the log files located in /var/log/httpd. The error log in there should show you more details about what failed.
I'm developing a Windows intranet application in C++ which needs to download a settings file from a predefined URL hosted on a webserver on the intranet.
This file would be updated every few weeks and I need to get it only if it has changed.
To avoid unnecessary downloads I wanted to know if there is a standard HTTP method to only request the hash of the file to the webserver to prevent a full download if the file has not changed.
I'm still in the design phase and the idea would be to use CURL library on the client to download the file and Apache as a webserver, but I'm also open to other solutions.
I want to know if it is possible to download a portion of a public AWS data set and how to do it.
Specifically, I want to download a part from Common Crawl Corpus to do local tests.
It looks like you can. If you point your browser to the public URL provided by Amazon, you can see links for the whole sets and also for subsets.
You can download it using your browser or any S3 client tools or libraries.