I am trying to automate a report generation process.
I need to enter into directories having a specific pattern and then read files from them. The directories name is in pattern PYYYYMMDD001, PYYYYMMDD002 and so on. I need to enter each directory with the defined pattern and read data from each file within the directory. But I am unable to do so as I am committing a mistake while defining the pattern. Please find the command I am using
TODAY=$(date +"%m%d%Y")
cd /home/user/allFiles
for d in "P"$TODAY*
do
(cd $d && grep -o '-NEW' *_$TODAY*_GOOD* | uniq -c| sed 's/\|/ /'|awk '{print $1}' > /home/user/new/$TODAY"Report.txt" )
done
When I am trying to execute it, getting the error of P02192017* [No such file or directory]
The list of directories are - P02192017001, P02192017002, P02192017003 , P02192017004 , P02192017005, P02192017006 , P02192017007, P02192017008
Any kind of help towards this would be highly appreciated.
Related
Currently, for getting a string (here: 123456789) into some files in all my buckets I do the following:
gsutil cat -h gs://AAAA/** | grep '123456789' > 20221109.txt
And I get the name of my path file when I match, so it works, but if I do it this way, it will search among all the directories (and I have a thousand directories and thousand files, it makes so much time.
I want to filter with a date thanks to the name of the subdirectory, like:
gsutil cat -h gs://AAAA/*2022-11-09*/** | grep '123456789' > 20221109.txt
But it didn't work, and I have no clue how to solve my problem, I read a lot of answers in SO, but I don't find them.
ps: I can't use find with gsutil , so I try to make it with cat and grep with gsutil in a single command line.
Thanks in advance for your help.
Finally, I managed to get what I wanted, but it was highly illegible. I think it's possible to do better. I'm open to any improvement. Reminder, This solution avoid reading all the directory of a bucket.
1st Step :
I manage to get all the paths and the file that match my pattern of the subdirectory (like a date here):
gsutil ls gs://directory1/*2022-11-09*/** > gs_path_files_2022_11_09.txt
After that, I want to make a grep for each file and get in the output the name of the file and the line where I get my match (again in the terminal):
while read -r line; do
gsutil cat "$line" | awk -v l="'Command: gsutil cat $line | awk '/the_string_i_want_to_match_in_my_file/{print ARGV[ARGIND] ":" $0}':" '/the_string_i_want_to_match_in_my_file/{print l $0}' >> results.txt
done < test.txt
and you will get after that the command (and the name of the file ) + the line where you get your match.
Best regards
First of all, the server runs Solaris.
The context of my question is Informatica PowerCenter.
I need to list files situated in the Inbox directory. Basically, the outcome should be one file list by type of file. The different file types are distinguished by the file name. I don't want to update the script every time a new file type starts to exist so I was thinking of a parameterized shell script with the regex, the inbox directory and the file list
An example:
/Inbox/ABC.DEFGHI.PAC.AE.1236547.49566
/Inbox/ABC.DEFGHI.PAC.AE.9876543.21036
/Inbox/DEF.JKLMNO.PAC.AI.1236547.49566
... has to result in 2 list files containing the path and file name of the listed files:
/Inbox/PAC.AE.FILELIST
-->/Inbox/ABC.DEFGHI.PAC.AE.1236547.49566
-->/Inbox/ABC.DEFGHI.PAC.AE.9876543.21036
/Inbox/PAC.AI.FILELIST
-->/Inbox/DEF.JKLMNO.PAC.AI.1236547.49566
Assuming all input files follow the convention you indicate (when splitting on dots, the 3rd and 4th column determine the type), this script might do the trick:
#! /usr/bin/env bash
# First parameter or current directory
INPUTDIR=${1:-.}
# Second parameter (or first input directory if not given)
OUTPUTDIR=${2:-$INPUTDIR}
# Filter out directories
INPUTFILES=$(ls -p $INPUTDIR | grep -v "/")
echo "Input: $INPUTDIR, output: $OUTPUTDIR"
for FILE in $INPUTFILES; do
FILETYPE=$(echo $FILE | cut -d. -f3,4)
COLLECTION_FILENAME="$OUTPUTDIR/${FILETYPE:-UNKNOWN}.FILELIST"
echo "$FILE" >> $COLLECTION_FILENAME
done
Usage:
./script.sh Inbox Inbox/collections
Will read all files (not directories) from Inbox, and write the collection files to Inbox/collections. Filenames inside collections should be sorted alphabetically.
I'm very new to shell script and therefore I don't now very much about it.
I have an application, which creates a java file with a half unknown name, and now I try to write a script, which needs this name.
The known name of the file is /target/plugin-<dyn>.jar, the <dyn> part is unknown and could be nearly anything (btw it is mostly a version number with variable text parts).
Now I want to save plugin-<dyn> (without the .jar) in a variable for later use. I looked very much in the internet, but I can't find a solution.
If you need get file name without extension .jar. You can refer my bash script below:
# for loop all files in target directory that matched plugin-*.jar
for f in target/plugin-*.jar
do
# print file name without extension .jar
echo ${f%.*}
done
UPDATED:
# for loop all files in target directory that matched plugin-*.jar
for f in target/plugin-*.jar
do
# print file name without extension .jar
filename="${f##*/}" # get plugin-*.jar
echo ${filename%.*} # print plugin-* without jar
done
try this
filename="/target/plugin-<dyn>.jar"
echo ${filename} | awk -F [/.] '{print $(NF - 1)}'
echo ${filename} | sed 's/.*\/\([^/]*\)\.jar/\1/'
but if <dyn> has a slash, comma or point. it may not work
So here's the scenario. I'd like to change the following value from true to false in 100's of files in an installation but can't figure out the command and been working on this for a few days now. what i have is a simple script which looks for all instances of a file and stores the results in a file. I'm using this command to find the files I need to modify:
find /directory -type f \ ( -name 'filename' \) > file_instances.txt
Now what i'd like to do is run the following command, or a variation of it, to modify the following value:
sed 's/directoryBrowsingEnabled="false"/directoryBrowsingEnabled="true"/g' $i > $i
When i tested the above command, it had blanked out the file when it attempted to replace the string but if i run the command against a single file, the change is made correctly.
Can someone please shed some light on to this?
Thank you in advance
What has semi-worked for me is the following:
You can call sed with the -i option instead of doing > $i. You can even do a backup of the old file just in case you have a problem by adding a suffix.
sed -e 'command' -i.backup myfile.txt
This will execute command inplace on myfile.txt and save the old file in myfile.txt.backup.
EDIT:
Not using -i may indeed result in blank files, this is because unix doesn't like you to read and write at the same time (it leads to a race condition).
You can convince yourself of this by some simple cat commands:
$ echo "This is a test" > test.txt
$ cat test.txt > test.txt # This will return an error cat being smart
$ cat <test.txt >test.txt # This will blank the file, cat being not that smart
On AIX you might be missing the -i option of sed. Sad. You could make a script that moves each file to a tmp file and redirects (with sed) to the original file or try using a here-construction with vi:
cat file_instances.txt | while read file; do
vi ${file}<<END >/dev/null 2>&1
:1,$ s/directoryBrowsingEnabled="false"/directoryBrowsingEnabled="true"/g
:wq
END
done
I'm (still) not a shell-wizard, but I'm trying to find a way to create directories and files from a list of file names.
Let's take this source file (source.txt) as an example:
README.md
foo/index.html
foo/bar/README.md
foo/bar/index.html
foo/baz/README.md
I'll use this command to remove empty lines and trim useless spaces:
$ more source.txt | sed '/^$/d;s/^ *//;s/ *$//'
It will give me this list:
README.md
foo/index.html
foo/bar/README.md
foo/bar/index.html
foo/baz/README.md
Now I'm trying to loop on every line and create the related file (it it doesn't already exists), with it's parents directories.
How could I do this?
Ideally, I would put this script in an alias to quickly use it.
As always, posting a question brings me to the end of the problem...
I came to a satisfying solution, using dirname and basename in a for .. in loop:
for i in `cat source.txt | sed '/^$/d;s/^ *//;s/ *$//'`;
do mkdir -p `dirname $i`;
touch `echo $(dirname $i)$(echo "/")$(basename $i)`;
done
This one-line command will:
read the file names list
create directories
create empty files in their own directory