I am trying to download just new videos. I realize I need to populate the downloads.txt with the video id's of previously downloaded files.
I have attempted exporting a .JSOn and simple powershell output but nothing is working. The .JSON .info.json file never gets created and Powershell does not record output from other softwares.
Also I am on Windows so there is no TEE and Grep command.
How can I only download new videos from a playlist and not the last 500?
So presently I run the following command in powershell to get just the Video ID's (should work in Command Prompt as well):
$YouTubeChannelUrl = Read-Host -Prompt 'Please enter the Channel URL'
C:\Video_Archive\Youtube-DL_Scripts\youtube-dl --get-id --ignore-errors $YouTubeChannelUrl
Read-Host -Prompt "Press Enter to exit"
Then I take copy the output to Notepad++ and prepend it with youtube and a space. Then I take that text and add it to the downloads.txt.
A bit of a manual process but definitely speeds up the checking for new downloads.
Related
We run S3 sync commands in a SQL Job which syncs a local directory to an S3 bucket. At times, we'll get a sync "error" with an error code of 1, or sometimes 2. The documentation lists what each code means; error code 1 provides less details and leaves more questions remaining about the problem. It simply states "One or more Amazon S3 transfer operations failed. Limited to S3 commands."
When I run a sync command in a PowerShell script, and encounter an error (i.e. a synced document being open), the window displays an error message and which specific file that is causing a problem.
How can I capture those details in my SQL job?
I have solved this problem...
Using a PowerShell script, in the AWS s3 sync command we output the results to a text file:
aws s3 sync c:\source\dir s3://target/dir/ > F:\Source\s3Inventory\SyncOutput.txt
Then read the text file contents into a string variable:
$S3Output = Get-Content -Path C:\Source\s3Inventory\SyncOutput.txt -Raw
If the $LASTEXITCODE from the sync command does not equal 0 (indicating an error), then we send an email with the results:
if ($LASTEXITCODE -ne 0)
{
#Send email containing the value of string variable $S3Output
}
This has been placed into production and we were finally able to determine which file/object was failing.
One could certainly attach the text file to the email, rather than reading the contents into a string.
Hope this helps others!
I have tried this but system tells 'wc' is not a valid command.
gsutil wc -l gs://folder/test.csv
please help me how to find no of records in a file without downloading it
i have tried this and it is working
gsutil cat gs://folder/test.csv | wc -l
Cloud Storage doesn't provide any computing resources to deal with contents of an object. The only things you can do are upload objects, download objects, or read/write metadata associated with an object. There is operation to count lines or do anything else with the contents of an object.
Your choices are to either download the object and count the lines on the client, or count the lines before uploading the object, and attaching that to metadata so that it can be easily discovered without requiring a full download.
I am using postman for API calls as I am trying to download on batch thousands of files from a database.
https://website.com/services/rest/connect/v1.4/incidents/439006/fileAttachments/?download
This creates the call but I then have to click save response -> save to a file to download the attachment.
Is this something that's possible in postman?
My IT department is very strict with downloading development environments I only have postman and R Studio. I know you can use RCURL potentially but considering I don't know how to use CURL I don't know where to start.
What I want to do is download any attachment (if it exists) for a number of keys.
And in a loop call the file:
i = o
Start loop i in n#rows(file)
i = i+
key = key(i)
https://website.com/services/rest/connect/v1.4/incidents/{key}/fileAttachments/?download
Save to file (named key).
Loop
I can't get it to work, I want a file full of thousands of downloads each called the number of the key.
You can use below command in linux system.
curl -v "https://website.com/services/rest/connect/v1.4/incidents/{key}/fileAttachments/?download"
You can use below command in linux system.
curl -v "https://website.com/services/rest/connect/v1.4/incidents/{key}/fileAttachments/?download"
Also you can try to send the request from postman and try to save that response.
Let me know if that works.
I am trying to automate a download process from a website for my data curation. It involves few searches and selecting the right file to download if the search term matches. Could you please give mes some insights or Sample code to start with?
Flow chart of process will be like:
Go to website-->Click on Advanced Search>> Click on "Title" >>Select from few dropdowns and fix the search space---> Enter the search string --->select the correct one and download the latest release.
I am tired of one problem so please make things clear to me.
Please read these following three points and help me out.
(1)
I have simply followed this https://www.wowza.com/docs/how-to-start-and-stop-live-stream-recordings-programmatically-livestreamrecordautorecord-example#documentation
I have attached my Application.xml. Now when I publish live stream name "test1" via FMLE it get recorded on server but when I run different instance of FMLE on different PC and publish live stream name "test2" it does not get record and I think it goes to previously recorded file "test1" (means no separate file being record, however there should be two files recorded test1 and test2).
Why this happenning ?
Is this com.wowza.wms.plugin.livestreamrecord.module.ModuleAutoRecordAdvancedExample for single stream recording ? means If I publish stream A B C D , it will record them in one single file ? (probably the output file will be A.mp4 as A was first published stream ?)
(2) What is this https://www.wowza.com/docs/how-to-start-and-stop-live-stream-recordings-programmatically-imediastreamactionnotify3#comments module for ?
I have implement this code in Eclipse and successfully put jar in lib folder and configured everything. Now again I am not able to record different streams with their corresponding name. Means If I publish stream1 and stream2 then desired output should be two different files (in content folder) but again I see one single file being record ?
(3) Can I use ModuleLiveStreamRecord.java ? This was in older version of WOWZA but I have properly imported required jar and tested it.
My requirement is very simple:
As soon as users start publishing, WOWZA should start live recording. If 10 users publishing live, 10 files should be generate.
Don't make things more difficult than necessary (assuming you have Wowza 4.x; if you still have 3.x then I highly recommend to upgrade for free)
Open the Engine Manager (http://your.server.com:8088)
Go to "Applications" from the top menu
Select your application from the left menu (e.g. "live")
In the setup window for this application, click the blue Edit button
Enable "Record all incoming streams"
Click "Save"
Click the orange "Restart now" button at the top
Done
Every stream that is published via this application will now automatically be recorded. The default folder for recordings is the /content folder in your Wowza installation. You can change this on the same page under "Streaming File Directory" (make sure it's a directory on your local system, unless you really well understand how Wowza works)
The filename is always the streamname + ".mp4", but when you start a new recording while the file already exists, the old file will be renamed first.
Want to control recording manually? Start publishing first, then select "Incoming streams" from the left menu and use the big red dot button behind a stream name to start recording.
If your server produces any different behavior with regards to the file (re)naming or recording, then you may need to review your Wowza setup.
I appreciate your response KBoek.
I sorted out issue but there were really debugging need if one doing custom module. I had to write custom module for live auto recording because I wanted HTTP authentication and then custom name of live recording.
thanks again