How to convert raw cookie into Netscape jar cookie and save to file? - youtube-dl

I want to pass a cookie file to youtube-dl coming from a browser extension. The extension sends cookie in raw format i.e. like VISITOR_INFO1_LIVE=_SebbjYciU0; YSC=tSqadPjjfd8; PREF=f4=4000000
But youtube-dl takes netscape jar format cookies (as far as I know).
If I put the raw text cookie in a file and pass it to --cookies=file.txt argument youtube-dl raises an exception.
I cannot manage to to convert my raw cookies to jar cookies and save to a file in the disk. I have searched for a solution but did not find any acceptable solution.

I had very similar problem to convert "curl" cookies into wget one ...
Netscape CookieJar format is straight forward see *
so I took few minutes to write a quick perl script and generate a cookiejar
here is the code (downloadable here)
#!/usr/bin/perl
# usage
# perl cookiejar.pl {{url}} '{{cookie-string}}'
use YAML::Syck qw(Dump);
my $expires = $^T + 86400;
my $path = '/';
my $url = shift;
my $cookies = shift;
printf "--- # %s at %u\n",__FILE__,$^T;
my $domain;
my $p = index($url,'://')+3;
my $l = index($url,'/',$p);
$domain = substr($url,$p,$l-$p);
my $dots = () = $domain =~ /\./g;
printf "dots: %s\n",$dots;
if ($dots > 1) {
$domain = substr($domain,index($domain,'.'));
} else {
$domain = '.'.$domain;
}
printf "domain: %s\n",$domain;
printf "url: %s\n",$url;
local *F; open F,'>','cookiejar.txt' or warn $!;
print F <<EOT;
# Netscape HTTP Cookie File
# http://curl.haxx.se/rfc/cookie_spec.html
# This is a generated file! Do not edit.
# domain: $domain
# url: $url
EOT
my #cookies = split'; ',$cookies;
printf "--- %s...\n",Dump(\#cookies);
foreach my $cookie (#cookies) {
my ($key,$value) = split('=',$cookie);
if (! $seen{$key}++) {
# domain access path sec expire cookie value
printf F "%s\t%s\t%s\t%s\t%s\t%s\t%s\n",$domain,'TRUE',$path,'FALSE',$expires,$key,$value;
}
}
close F;
print "info: cookiejar.txt created\n";
printf "cmd: wget --load-cookie-file cookiejar.txt --referer=%s -p %s\n",$domain,$url;
printf "cmd: youtube-dl --cookie cookiejar.txt -referer %s %s\n",$domain,$url;
exit $?;
1; # $Source: /my/perl/scripts/cookiejar.pl $
you run it as followed :
perl cookiejar.pl https://example.com/ 'cookie1=value1; cookie2=value2'
note:
You might need to install YAML::Syck with
cpan install YAML::Syck
or just comment out the Dump() call and the use YAML::Syck line.

Related

Edit a remote file using SSH in Perl

Below is the code snippet I have written.
This tries to fetch the server name and file name through the arguments
I am stuck at editing the remote file after the server name and corresponding file name are fetched.
Command run: server.pl sox3d1 TEST50
File server.pl:
use strict;
use warnings;
use Data::Dumper;
use XML::Simple;
my $host=$ARGV[0];
my $db_host=$ARGV[1];
my %servers;
$servers{"ser14316.local.net"}=["sox3d1","sox3d2"] ;
$servers{"ser143hn.local.net"}=["sox4d1","sox4d2"] ;
$servers{"ser14441.local.net"}=["sox6d1","sox6d2"] ;
$servers{"ser18163.local.net"}=["soxuat61","soxuat62","soxuat63"] ;
$servers{"ser1444r.local.net"}=["soxuat51","soxuat52","soxuat53"] ;
my $files=
{
db=>"/appl/$host/db_info.ref",
};
my %groups_by_host;
for my $group (keys(%servers)) {
for my $host (#{ $servers{$group} }) {
push #{ $groups_by_host{$host} }, $group;
}
}
my $server = $groups_by_host{$host}; # we get the server name via variable $host
my ( $ssh_to_server )=#$server; # This is the server to ssh
my $dest_file=$files->{"db"} ; #This will fetch the file from the remote server
print "$ssh_to_server","->","$dest_file \n";
The file in this server has to be edited and below are the steps to do that via ssh.
Search for the string $host (i.e sox3d3) in the remote file $dest_file by doing ssh to server $ssh_to_server
Replace the string TEST30 with TEST50
The file is a semicolon-separated file
File content in remote server (db_info.ref):
sox3d3 ;/appl/sox3d3/current ;TEST30 ;TEST30 ;USER_10 ;USER_30
sox4d4 ;/appl/sox4d4/current ;TEST40 ;TEST40 ;USER_20 ;USER_40
So, the 3rd and 4th columns (TEST30;TEST30) should be replaced with TEST50 as per input.
Have tried the below code to get it working
as i said i was trying for a one line and had many variables in the command , the below worked actually
Update 1:
This basically ssh's the server and changes the file accordingly
system(
'ssh' => ('-q',$ser),
'sed' => ('-i -E', qq('/$host/s#([^;]+;[^;]+;)[^;]+;[^;]+#\\1$db_host
;$db_host #') , $dest_file),
Tried something like this?
my $destination = sprintf( '%s#%s', $user, $host );
my $file = '/path/to/some/file.txt';
system(
'ssh' => $destination,
'sed' => ( '-i', 's/TEST30/TEST50/g', $file ),
);
The ssh documentation shows you can pass a command after the ssh destination. When you do this, ssh will log into the server, run that command, then log out again, instead of dropping you into the shell on the server.
So you can use that to run sed to edit the file.

powershell WebClient parse links or regex for zip file

I have a script that nearly works but I need to add in either parsing or a wildcard on the URL as the URL will change to characters I won’t know each month. I have to use New-Object System.Net.WebClient because invoke web request is blocked. So, I was thinking if anyone knows how to download the link using the characters that I will know and strip off the rest of the link.
Examples of the links below
Full-CSV-data-file-Jan19-ZIP-3633K-53821.zip
Full-CSV-data-file-Dec18-ZIP-3427K.zip
Full-CSV-data-file-Nov18-ZIP-3543K-21860.zip
So on the above links i know the latest file will have Jan19 in it and it's a zip file. The script i am using is
$currentMonthNo = get-date -format "MM"
$currentMonthName = (get-date((get-date).addmonths(-2)) -format MMM)
$currentYearNo = get-date –format yy
$url = "http://www.website.com/Full-CSV-data-file-$currentMonthName$currentYearNo-ZIP-3633K-53821.zip"
$output = "C:\Folder\Full-CSV-data-file-Jan19-ZIP-3633K-53821.zip"
$start_time = Get-Date
$wc = New-Object System.Net.WebClient
$wc.DownloadFile($url, $output)
#OR
(New-Object System.Net.WebClient).DownloadFile($url, $output)
Write-Output "Time taken: $((Get-Date).Subtract($start_time).Seconds) second(s)"
Write-Output $url
Start-Sleep -s 6

Use of uninitialized value $a in concatenation (.) or string

I am trying to remove the old files in a dir if the count is more than 3 over SSH
Kindly suggest how to resolve the issue.
Please refer the code snippet
#!/usr/bin/perl
use strict;
use warnings;
my $HOME="/opt/app/latest";
my $LIBS="${HOME}/libs";
my $LIBS_BACKUP_DIR="${HOME}/libs_backups";
my $a;
my $b;
my $c;
my $d;
my $command =qq(sudo /bin/su - jenkins -c "ssh username\#server 'my $a=ls ${LIBS_BACKUP_DIR} | wc -l;my $b=`$a`;if ($b > 3); { print " Found More than 3 back up files , removing older files..";my $c=ls -tr ${LIBS_BACKUP_DIR} | head -1;my $d=`$c`;print "Old file name $d";}else { print "No of back up files are less then 3 .";} '");
print "$command\n";
system($command);
output:
sudo /bin/su - jenkins -c "ssh username#server 'my ; =ls /opt/app/latest/libs_backups | wc -l;my ; =``;if ( > 3); { print " Found More than 3 back up files , removing older files..";my ; =ls -tr /opt/app/latest/libs_backups | head -1;my ; =``;print "Old file name ";}else { print "No of back up files are less then 3 .";} '"
Found: -c: line 0: unexpected EOF while looking for matching `''
Found: -c: line 1: syntax error: unexpected end of file
If you have three levels of escaping, you're bound to get it wrong if you do it manually. Use String::ShellQuote's shell_quote instead.
Furthermore, avoid generating code. You're bound to get it wrong! Pass the necessary information using arguments, the environment or some other channel of communication instead.
There were numerous errors in the interior Perl script on top of the fact that you tried to execute a Perl script without actually invoking perl!
#!/usr/bin/perl
use strict;
use warnings;
use String::ShellQuote qw( shell_quote );
my $HOME = "/opt/app/latest";
my $LIBS = "$HOME/libs";
my $LIBS_BACKUP_DIR = "$HOME/libs_backups";
my $perl_script = <<'__EOI__';
use strict;
use warnings;
use String::ShellQuote qw( shell_quote );
my ($LIBS_BACKUP_DIR) = #ARGV;
my $cmd = shell_quote("ls", "-tr", "--", $LIBS_BACKUP_DIR);
chomp( my #files = `$cmd` );
if (#files > 3) {
print "Found more than 3 back up files. Removing older files...\n";
print "$_\n" for #files;
} else {
print "Found three or fewer backup files.\n";
}
__EOI__
my $remote_cmd = shell_quote("perl", "-e", $perl_script, "--", $LIBS_BACKUP_DIR);
my $ssh_cmd = shell_quote("ssh", 'username#server', "--", $remote_cmd);
my $local_cmd = shell_quote("sudo", "su", "-c", $ssh_ccmd);
system($local_cmd);
I created a new file and handling the dir check and deletion logic , scp file to remote server and executing in remote server , after completion removing the file.
#!/usr/bin/perl
use strict;
use warnings;
use File::Basename;
use File::Path;
use FindBin;
use File::Copy;
my $HOME="/opt/app/test/latest";
my $LIBS_BACKUP_DIR="${HOME}/libs_backups";
my $a="ls ${LIBS_BACKUP_DIR} | wc -l";
my $b=`$a`;
my $c="ls -tr ${LIBS_BACKUP_DIR} | head -1";
my $d=`$c`;
chomp($d);
print " count : $b\n";
if ($b > 3)
{
print " Found More than 3 back up files , removing older files..\n";
print "Old file name $d\n";
my $filepath="${LIBS_BACKUP_DIR}/$d";
rmtree $filepath;
}
else
{
print "No of back up files are less then 3 .\n";
}

How can i wait until something is written to log file in my perl script

I am actually Monitoring a directory for creation of new files(.log files) these files are generated by some tool and tool writes log entries after sometime of the creation of the same file, During this time file will be empty.
and how can i wait until something is written to the log and reason being based on the log entries i will be invoking different script!,
use strict;
use warnings;
use File::Monitor;
use File::Basename;
my $script1 = "~/Desktop/parser1.pl";
my $scrip2t = "~/Desktop/parser2.pl";
my $dir = "~/Desktop/tool/logs";
sub textfile_notifier {
my ($watch_name, $event, $change) = #_;
my #new_file_paths = $change->files_created; #The change object has a property called files_created,
#which contains the names of any new files
for my $path (#new_file_paths) {
my ($base, $fname, $ext) = fileparse($path, '.log'); # $ext is "" if the '.log' extension is
# not found, otherwise it's '.log'.
if ($ext eq '.log') {
print "$path was created\n";
if(-z $path){
# i need to wait until something is written to log
}else{
my #arrr = `head -30 $path`;
foreach(#arr){
if(/Tool1/){
system("/usr/bin/perl $script1 $path \&");
}elsif(/Tool1/){
system("/usr/bin/perl $script2 $path \&");
}
}
}
}
my $monitor = File::Monitor->new();
$monitor->watch( {
name => $dir,
recurse => 1,
callback => {files_created => \&textfile_notifier}, #event => handler
} );
$monitor->scan;
while(1){
$monitor->scan;
}
Basically i am grepping some of the important information from the logs.
For such formulation of your question, something like this might help you:
use File::Tail;
# for log file $logname
my #logdata;
my $file = File::Tail->new(name => $logname, maxinterval => 1);
while (defined(my $newline = $file->read)) {
push #logdata, $newline;
# the decision to launch the script according to data in #logdata
}
Read more here
You are monitoring just the log file creation. Maybe you could use a sleep function inside the call back sub to wait for the log file been wrote. You could monitor file changes too, because some log files could be extended.

Perl file copy duplicating output

I'm attempting to write a menu driven modular perl script that will capture user input and automate the network configuration process. This script has to be able to install required Arch packages, configure AP mode, configure either DHCP or a static address for the user selected interface and give an option to enable bridging. (EDIT: The script also needs to be able to enable and configure the dhcpd service)
The part I'm stuck on right now is creating a backup of the rc.conf file, reading the file and editing the lines that need to be modified if a network interface has already been statically configured. This script is for use in ArchLinux, I did some searching around and didn't find anything that met my needs specifically.
Using generic input for
$ip = 1.1.1.1; $Bcast = 2.2.2.2; $netmask = 3.3.3.3; $GW = 4.4.4.4;
I've spent about two hours reading about file I/O and tried several things that didn't work including scrapping the multiple file IO method and using something similar to: while(<IS>){s/^interface.?=(.*)$/"interface=#if[0] \n"/;} with inputs for each of the values that need to be replaced and couldn't get it to actually do anything.
if (system ("cat","/etc/rc.conf","|","grep","interface")){
use File::Copy "cp";
$filename = "/etc/rc.conf";
$tempfile = "/etc/rc.tmp";
$bak = "/etc/rc.bak";
cp($filename,$bak);
open(IS, $filename);
open(OS, ">$tempfile");
while(<IS>){
if($_ =~ /^interface.?=(.*)$/){ print OS"interface=#if[0] \n";}
if($_ =~ /^address.?=(.*)$/){ print OS "address=$ip\n";}
if($_ =~/^netmask.?=(.*)$/){ print OS "netmask=$netmask\n";}
if($_ =~/^broadcast.?=(.*)$/){ print OS "broadcast=$Bcast\n";}
if($_ =~/^gateway.?=(.*)$/){ print OS "gateway=$GW\n"; }
else {print OS $_;}
}
close(IS); close(OS);
unlink($filename); rename($tempfile, $filename);
}
rc.conf before
#
# /etc/rc.conf - Main Configuration for Arch Linux
. /etc/archiso/functions
LOCALE_DEFAULT="en_US.UTF-8"
DAEMON_LOCALE_DEFAULT="no"
CLOCK_DEFAULT="UTC"
TIMEZONE_DEFAULT="Canada/Pacific"
KEYMAP_DEFAULT="us"
CONSOLEFONT_DEFAULT=
CONSOLEMAP_DEFAULT=
USECOLOR_DEFAULT="yes"
LOCALE="$(kernel_cmdline locale ${LOCALE_DEFAULT})"
DAEMON_LOCALE="$(kernel_cmdline daemon_locale ${DAEMON_LOCALE_DEFAULT})"
HARDWARECLOCK="$(kernel_cmdline clock ${CLOCK_DEFAULT})"
TIMEZONE="$(kernel_cmdline timezone ${TIMEZONE_DEFAULT})"
KEYMAP="$(kernel_cmdline keymap ${KEYMAP_DEFAULT})"
CONSOLEFONT="$(kernel_cmdline consolefont ${CONSOLEFONT_DEFAULT})"
CONSOLEMAP="$(kernel_cmdline consolemap ${CONSOLEMAP_DEFAULT})"
USECOLOR="$(kernel_cmdline usecolor ${USECOLOR_DEFAULT})"
MODULES=()
UDEV_TIMEOUT=30
USEDMRAID="no"
USEBTRFS="no"
USELVM="no"
HOSTNAME="archiso"
DAEMONS=(hwclock syslog-ng)
interface=eth0
address=192.168.0.99
netmask=255.255.255.0
broadcast=192.168.0.255
gateway=192.168.0.1
rc.conf after
#
# /etc/rc.conf - Main Configuration for Arch Linux
. /etc/archiso/functions
LOCALE_DEFAULT="en_US.UTF-8"
DAEMON_LOCALE_DEFAULT="no"
CLOCK_DEFAULT="UTC"
TIMEZONE_DEFAULT="Canada/Pacific"
KEYMAP_DEFAULT="us"
CONSOLEFONT_DEFAULT=
CONSOLEMAP_DEFAULT=
USECOLOR_DEFAULT="yes"
LOCALE="$(kernel_cmdline locale ${LOCALE_DEFAULT})"
DAEMON_LOCALE="$(kernel_cmdline daemon_locale ${DAEMON_LOCALE_DEFAULT})"
HARDWARECLOCK="$(kernel_cmdline clock ${CLOCK_DEFAULT})"
TIMEZONE="$(kernel_cmdline timezone ${TIMEZONE_DEFAULT})"
KEYMAP="$(kernel_cmdline keymap ${KEYMAP_DEFAULT})"
CONSOLEFONT="$(kernel_cmdline consolefont ${CONSOLEFONT_DEFAULT})"
CONSOLEMAP="$(kernel_cmdline consolemap ${CONSOLEMAP_DEFAULT})"
USECOLOR="$(kernel_cmdline usecolor ${USECOLOR_DEFAULT})"
MODULES=()
UDEV_TIMEOUT=30
USEDMRAID="no"
USEBTRFS="no"
USELVM="no"
HOSTNAME="archiso"
DAEMONS=(hwclock syslog-ng)
interface=eth0
interface=eth0
address=1.1.1.1
address=192.168.0.99
netmask=3.3.3.3
netmask=255.255.255.0
broadcast=2.2.2.2
broadcast=192.168.0.255
gateway=4.4.4.4
I am not going to comment on the wisdom of the rest of your script, but you have:
if (system ("cat","/etc/rc.conf","|","grep","interface")){
system returns 0 on success.
So, you'll enter the block only if that system call fails.
If fact, I am on a Windows system right now with no /etc/rc.conf (but cat and grep thanks to Cygwin. Running the following script:
#!/usr/bin/env perl
use strict; use warnings;
if (system ("cat","/etc/rc.conf","|","grep","interface")){
print "*** it worked! ***\n";
if ($? == -1) {
print "failed to execute: $!\n";
}
elsif ($? & 127) {
printf "child died with signal %d, %s coredump\n",
($? & 127), ($? & 128) ? 'with' : 'without';
}
else {
printf "child exited with value %d\n", $? >> 8;
}
}
produces the output:
cat: /etc/rc.conf: No such file or directory
cat: |: No such file or directory
cat: grep: No such file or directory
cat: interface: No such file or directory
*** it worked! ***
child exited with value 1
That means system returned a failure code. Now, if you want to use shell piping and redirection, you should pass system a string, not a list, and check like this:
if (system ('cat /etc/rc.conf | grep interface') == 0) {
On the other hand, I would rather not trust shells propagating exit status.
The following should point you in a better direction:
#!/usr/bin/env perl
use strict;use warnings;
my %lookup = (
eth0 => {
address => '1.1.1.1',
broadcast => '2.2.2.2',
netmask => '3.3.3.3',
gateway => '4.4.4.4',
},
wlan0 => {
address => '5.5.5.5',
broadcast => '6.6.6.6',
netmask => '7.7.7.7',
gateway => '8.8.8.8',
},
);
while (my $line = <DATA>) {
if (my ($interface) = ($line =~ /^interface=(\S+)/)) {
print $line;
if (exists $lookup{$interface}) {
$line = process_interface(\*DATA, $lookup{$interface});
redo;
}
}
else {
print $line;
}
}
sub process_interface {
my ($fh, $lookup) = #_;
my $keys = join '|', sort keys %$lookup;
while (my $line = <DATA>) {
$line =~ s/\A($keys)=.+/$1=$lookup->{$1}/
or return $line;
print $line;
}
return;
}
__DATA__
#
# /etc/rc.conf - Main Configuration for Arch Linux
. /etc/archiso/functions
# stuff
interface=eth0
address=192.168.0.99
netmask=255.255.255.0
broadcast=192.168.0.255
gateway=192.168.0.1
interface=wlan0
address=192.168.0.99
netmask=255.255.255.0
broadcast=192.168.0.255
gateway=192.168.0.1
Output:
#
# /etc/rc.conf - Main Configuration for Arch Linux
. /etc/archiso/functions
# stuff
interface=eth0
address=1.1.1.1
netmask=3.3.3.3
broadcast=2.2.2.2
gateway=4.4.4.4
interface=wlan0
address=5.5.5.5
netmask=7.7.7.7
broadcast=6.6.6.6
gateway=8.8.8.8
The problem is your if/if/if/if/if/else chain, which should be an if/elsif/elsif/elsif/elsif/else chain. The else { print OS $_ } triggers on every line that doesn't match gateway=, including the ones that match interface, address, etc.