I use youtube-dl -a filename to download the videos. However, when one URL in the list of URLs fail, the process exits, is there a way to skip the failing URL and proceeding with the remaining URLs?
The man page of youtube-dl says:
-i, --ignore-errors Continue on download errors, for example to skip unavailable
videos in a playlist
Thus:
youtube-dl -i -a filename
edit: I strongly advice you to run
youtube-dl -U
prior to any download, as the world of online videos is fast changing and updates often fix download errors. Moreover, some errors are due to content restriction and can be solved by adding login and password to the tool:
youtube-dl -u USERNAME -p PASSWORD
Related
Used command:
youtube-dl --max-filesize 30m -f m4a -o "/home/dc3014b3c6a1a23bba88b2a1fbcc1447.m4a" "https://www.youtube.com/watch?v=_Xa0ydtx8PM"
youtube-dl can't work at all for me. Error happened like these:
ERROR: Unable to download webpage: <urlopen error EOF occurred in violation of protocol (_ssl.c:618)> (caused by URLError(SSLEOFError(8, u'EOF occurred in violation of protocol (_ssl.c:618)'),))
OR
ERROR: Unable to download webpage: HTTP Error 429: Too Many Requests (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
But When I use curl command to get url content, it's ok.
curl -v https://www.youtube.com/watch?v=_Xa0ydtx8PM
How can I resolve it?
From the error message, You might want to be sure you are using the latest version of youtube-dl. you might want to update it. Am assuming you are using a *nix system. Also depending on how you first installed, there are several options for updating. here are a few options;
For manual installations:
you can simply run youtube-dl -U or, on Linux, sudo youtube-dl -U.
If you are already running on an update version, you may want to consider the below as best methods to download videos. Mind you that with the new version of youtube-dl, it automatically downloads the best version for you so you do not need to specify although you could still do this to be sure.
# Download best mp4 format available or any other best if no mp4 available
$ youtube-dl -f 'bestvideo[ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]/best'
# Download best format available but no better than 480p
$ youtube-dl -f 'bestvideo[height<=480]+bestaudio/best[height<=480]'
# Download best video only format but no bigger than 50 MB
$ youtube-dl -f 'best[filesize<50M]'
# Download best format available via direct link over HTTP/HTTPS protocol
$ youtube-dl -f '(bestvideo+bestaudio/best)[protocol^=http]'
# Download the best video format and the best audio format without merging them
$ youtube-dl -f 'bestvideo,bestaudio' -o '%(title)s.f%(format_id)s.%(ext)s'
Here is a link for reference and further instructions/support.
I hope this helps. If not, let me know. Glad to help to the end.
Just wanted to share the youtube-dl alternative (it is just a fork) that for now (September 2022) works fine and supports almost all youtube-dl features yt-dlp.
I switched to this project and now I am using the same scripts which I had have previously.
I am looking for the easiest way to download the kaggle competition data (train and test) on the virtual machine using bash to be able to train it there without uploading it on git.
Fast-forward three years later and you can use Kaggle's API using the CLI, for example:
kaggle competitions download favorita-grocery-sales-forecasting
First you need to copy your cookie information for kaggle site in a text file. There is a chrome extension which will help you to do this.
Copy the cookie information and save it as cookies.txt.
Now transfer the file to the EC2 instance using the command
scp -i /path/my-key-pair.pem /path/cookies.txt user-name#ec2-xxx-xx-xxx-x.compute-1.amazonaws.com:~
Accept the competitions rules and copy the URLs of the datasets you want to download from kaggle.com. For example the URL to download the sample_submission.csv file of Intel & MobileODT Cervical Cancer Screening competition is: https://kaggle.com/c/intel-mobileodt-cervical-cancer-screening/download/sample_submission.csv.zip
Now, from the terminal use the following command to download the dataset into the instance.
wget -x --load-cookies cookies.txt https://kaggle.com/c/intel-mobileodt-cervical-cancer-screening/download/sample_submission.csv.zip
Install CurlWget chrome extension.
start downloading your kaggle data-set. CurlWget will give you full wget command. paste this command to terminal with sudo.
Job is done.
Install cookies.txt extension on chrome and enable it.
Login to kaggle
Go to the challenge page that you want the data from
Click on cookie.txt extension on top right and it download the current page's cookie. It will download the cookies in cookies.txt file
Transfer the file to the remote service using scp or other methods
Copy the data link shown on kaggle page (right click and copy link address)
run wget -x --load-cookies cookies.txt <datalink>
I'm following tutorial on http://anantgarg.com/2013/09/25/git-ftp-for-windows/ and get error:
$ git ftp init
There are 720 files to sync:
[1 of 720] Buffered for upload '.gitignore'.
[2 of 720] Buffered for upload 'wp-config-sample.php'.
....
[24 of 720] Buffered for upload 'wp-content/plugins/woocommerce/admin/post-types/writepanels/order-item-html.php'.
Uploading ...
fatal: Could not upload files., exiting...
what could be the problem? I've set the password using quotes ( 'mypassword' ) but same error still occur
You can see output of the exact error by using the '-v' flag for super-verbose output.
The problem could be one of the following
Incorrect FTP credentials - check that the server, username, and password are all correct.
URL/server: make sure to include the protocol with the URL if not standard FTP
(sftp://ftp.domain.com:22).
Password: Try also removing the quotes around the password
The server path is not set or incorrect. Often, an FTP login takes you to the server root. You may not have write-privileges there. You'll need to set the site root path.
try setting path with in the URL argument ftp://ftp.domain.com/public_html
Try doing an 'init' via command-line arguments instead (NOT via Git config settings) and set verbose output using '-v':
git ftp init -u username -p 'password' -v ftpserverurl.com
I think you should use ftp-git, it's a GUI tool to push changed files in git repository to ftp server. And it's more error-friendly.
You can check the changed files in GUI way, and it can save your ftp connection details for future use.
I have installed Redmine and I've been playing around with a few themes. I am having trouble installing a custom logo. I add the new file in the correct folder and reference it in the correct stylesheet but when I inspect it in the browser it says "Failed to load the given URL"
I also tried making changes to the base.html.erb file and this did not show up either.
Simple changes to the stylesheet do work however.
Thanks for your help :)
Man, doing anything with Redmine customization is not easy but I finally got it. I followed this tutorial, http://www.redmine.org/projects/redmine/wiki/Howto_add_a_logo_to_your_Redmine_banner
However this did not actually work on my system so I added a few commands of my own. If these commands do not work then try adding sudo in front of them. THis will prompt you for the admin password.
Near the end, the tut tells you to do this in command line:
chown redmine:redmine /opt/redmine/public/images/logo.png
Now I am using Apache so my path would look more like this
chown apache:apache /var/www/redmine/public/images/logo.png
Anyway I tried that and it still had no effect, now the additions I was doing was either under public/images or public/themes so I chose to just target the public folder. This may be bad practice when going live but I am only working locally on a virtual machine.
Here is what worked:
chown -R apache:apache /var/www/redmine/public
chmod -R 775 apache:apache /var/www/redmine/public
The -R stands for recursive so it effects the child files too. chmod 775 allows read, write, execute permission for Owner & Group, and only read, execute permissions for Other.
Restart your server after that (the command could be slightly different depending on your setup, refer to the tut)
/etc/init.d/httpd restart
This was very confusing for me at first so let me know if anyone needs more clarification on the subject. I am using Redmine installed on a CentOS virtual box.
I'm running a django project on Centos 5.4 and serving it with httpd/mod_wsgi. I can't figure out the correct permissions for /home/website/django_project so that I don't get a 403 error.
In my httpd.conf the user and group to run httpd as is apache. The group django is set up with website and apache as members. The owner of /home/website and all subdirs is website:django, and the permissions are rwxrwx---. Right now the project works fine with the dev server, but if I try to view it through apache, I get a 403 error. chmod -R o+rx /home/website/django_project fixes the problem, but this obviously isn't a good solution.
Thanks
First, try setting the group-sticky bit on the directories:
find /home/website -type d -exec chmod g+s {} \;
Then the perms should read rwxrws---. See if this makes a difference.
If that fails, you can try to poke around as the "website" user and see what happens. Temporarily give the user "website" a home directory (not /home/website, it needs to be something else, like /var/home/website), password, and login shell, then use su - website to switch to it. Try listing the contents of /home/website and try reading files in there. Fix any problems.
Hope this helps.
P.S. I'm assuming /var/log/apache/access_log (or maybe it's /var/log/http/access_log) doesn't have anything useful.