gcc -MM on cygwin: sometimes unix style paths - c++

I have a (legacy) bash script that parses a list of directories for source files and echos them into a makefile to create actual makerules with gcc -MM. The script works just fine, the problem is that running make on the generated makefile results in inconsistent path format in the makerules - while most makerules have mixed style paths (which is ok), some have cygwin style paths (not ok). This didn't happen on a very old (~2002 release) of cygwin.
Here is the script:
#!/bin/bash
fn_source=SourceFileList
fn_depmake=depMakefile
fn_orgMake=Makefile
fn_depIncl=Depend.incl
fn_optIncl=OptimizationCflags.incl
#
# create a Makefile to determine the dependencies
echo -n create dep-makefile ...\
#
# generate include for Optimization-File
#
echo > $fn_depIncl
echo include $fn_optIncl >> $fn_depIncl
echo >> $fn_depIncl
#
touch $fn_optIncl
#
echo > $fn_depmake
# CFLAGS options from Makefile
grep "^CFLAGS[ =]" $fn_orgMake | grep $BUILD_SPEC >> $fn_depmake
grep "^CFLAGS_AS[ =]" $fn_orgMake | grep $BUILD_SPEC >> $fn_depmake
grep "^CFLAGS_GCC[ =]" $fn_orgMake | grep $BUILD_SPEC >> $fn_depmake
echo -n generate depMakefile ...
echo >> $fn_depmake
# the default rule
echo all: >> $fn_depmake
FileList=`cat $fn_source`
incPath=""
rawIncPath=""
for fn in $FileList; do
incPath=$incPath" -I"${fn%/*}
rawIncPath=$rawIncPath" "${fn%/*}
done
# filter out redundant paths
echo $incPath | \
sed -n -e 's/ /\
/gp' | \
sort -u > _tmp
incPath=`cat _tmp`
# filter out redundant paths
echo $rawIncPath | \
sed -n -e 's/ /\
/gp' | \
sort -u > _tmp
rawIncPath=`cat _tmp`
#
# create file with include paths
#
echo LibIncludes=$incPath > LibIncludes
#
# put all source files with the same path in one gcc command
#
for path in $rawIncPath; do
sourceFiles=`grep $path/ $fn_source`
echo " "echo do $path >> $fn_depmake
echo " "gcc -MM '$(CFLAGS_GCC)' -I$path $sourceFiles \>\> _tmp >> $fn_depmake
done
# start the Makefile
echo start Makefile ...
rm _tmp; touch _tmp
make -s -f $fn_depmake
# add ./obj/xxxxx*.o to the make-rules
sed -e 's/\(^[a-zA-Z]\)/.\/obj\/\1/' < _tmp > _tmp1
# add command to the make-rules
sed -e '/\\/!a\
$(CC) $($(*F)CFLAGS) -I$(<D) $(CFLAGS) -c $< -o $#' < _tmp1 >> $fn_depIncl
# workaround for gcc generating cygdrive paths for some paths only
sed -i 's#/cygdrive/d#d:#' $fn_depIncl
rm _tmp _tmp1
rm $fn_depmake
echo done.
As you can see, I am currently using a workaround to convert those paths to mixed styled ones using sed, but I would like to know why this happens in the first place?
Example of a rule gone "wrong":
some.o: d:/some/path/A/some.cpp \
d:/some/path/A/some.h \
/cygdrive/d/some/path/B/subpathA/anotherheader.h \
/cygdrive/d/some/path/B/subpathA/yetanotherheader.h \
D:/some/path/B/subpathB/header.h
The main problem about this is that the resulting makefile is fed to a windows native make.exe, which has no idea what /cygdrive/d/path is supposed to be.
I have looked at the paths after each step, and they are all mixed style until make -s -f $fn_depmake is executed. Meaning, the occasional conversion to cygwin style paths is done by the gcc -MM command in the generated makefile.
It is always the same list of directories that get converted to /cygdrive/d/pathA/somefile.cpp style of paths. There is nothing unusual about the path, no spaces, only a-z/A-Z in name, length of the path is short, longest path is 76 characters including the filename, base path is 34 characters long.
How can I force gcc to generate mixed style paths?

I can see the same problem on my project. I'm using gmake.exe from Cygwin on Windows.
One invocation of gcc -MM outputs both Windows-style paths (C:/project/file.h) and Cygwin-style paths (/cygdrive/c/project/file.h).
I don't know how to fix that yet, but I found what triggers the change to Cygwin-style paths. This happens when an include uses a relative path. So, when a file contains #include "../file.h", then the output of gcc -MM will contain the absolute path of file.h, using a Cygwin-style path.

Related

How to find specific files in folders and do an operation in case they exist

I am having some difficulty doing a basic script in sh shell.
What I want to do is simple though:
I want to do a sh script (can also be csh) that looks through a number of folders and for each folder that contains the files I am interested in, it should do a specific operation of pasting the corresponding filename into a sh script with rdseed commands.
The script I wrote in sh shell and doesn't work is:
for dir in EV*
do
echo $dir
cd $dir
if [ -f GEFLE* = true ];
then
set dataless = gur_ini_dataless.seed
for file GEFLE*
do
echo "rdseed -d -o 2 -f "$file " -g " $dataless >> runmseed2ahGEFLE.sh
done
else
echo "File does not exists"
fi
sleep 0.5
cd ..
done
Does anyone know a solution?
Please try this... I'm adding some comments to the lines...
#!/bin/sh
for dir in EV*
do
echo $dir
cd $dir
if [ -f GEFLE* ] # true if at least one FILE named "GEFLE*" exists
then
dataless=gur_ini_dataless.seed # no `set`, no spaces
for file in GEFLE* # will match all FILES/DIRS/... that start with "GEFLE"
do
echo "rdseed -d -o 2 -f $file -g $dataless" >> runmseed2ahGEFLE.sh # vars are substituted in double quoted strings
done
else
echo "File does not exists"
fi
cd ..
done
Please note this will only look one level deep into the directories. If you need some resursion you should better use something like
for dir in `find . -type d -name 'EV*'`; do
# ...
done
The way I had put this is:
for f in `find EV* -name GEFLE* -type f`; do
echo "rdseed -d -o 2 -f ./$f -g gur_ini_dataless.seed >> ./`dirname $f`/runmseed2ahGEFLE.sh"
done

Bash copy all directory with content that matches a pattern

Is there some way to copy the directories including the contents using bash script. For example
// Suppose there are many directory inside Test in c as,
/media/test/
-- en_US
-- file1
-- file 2
-- de_DE
-- file 1
-- SUB-dir1
-- sub file 1
-- file 2
.....
.....
-- Test 1
-- testfile1
-- folder
--- more 1
............
NoW i want to copy all the directories (including sub-directory and files)
to another location which matches the pattern.
--> for example , in above case I want the directories en_US and de_DE to be copied in another
location including sub-directories and files.
So Far I have done/ find out :
1) Needed Pattern as , /b/w{2}_/w{2}/b
2) I can list all the directories as ,
$MYDIR="/media/test/"
DIRS=`ls -l $MYDIR | egrep '^d' | awk '{print $10}'`
for DIR in $DIRS
do
echo ${DIR}
done
Now I need help in combining these together so that the script can copy all the directory(including sub contents) that matches the pattern to another location.
Thanks in advance.
To selectively copy an entire directory structure to a similar directory structure, while filtering the contents, in a general way your best bet is to archive the original directory and unarchive. For instance, using GNU Tar:
$ mkdir destdir
$ tar -c /media/test/{en_US,de_DE} | tar -C destdir -x --strip-components=1
In this example, the /media/test directory structure is partially recreated under destdir, excluding the /media prefix (thanks to --strip-components=1).
The left side tar archives just the directories/paths which match the pattern that we specified. The archive is produced on that command's standard output, which is piped to the decoding tar on the right hand side. The -C tells it to change to the destination directory. It extracts the files there, removing a leading path component.
$ ls destdir
test
$ ls destdir/test
en_US de_DE
Of course, your specific example test case is quite easily handled with cp -a:
$ mkdir destdir
$ cp -a /media/test/{en_US,de_DE} destdir
If the pattern is complicated, involving multiple selections of subtree material at deeper and/or different levels of the source directory hierarchy, then you need the more general approach, if you wish to do the copy in a single batch command which just specifies source patterns.
I'm not sure about your environment, but I guess you try to do this:
cp -r src_dir/??_?? dest_dir
Here is your starter for 10:
You will have to add the extra checks and balances that you require but it should give you a flying start.
#!/bin/bash
# assumes $1 is source to search and $2 to destination to copy to
subdirs=`find $1 -name ??_?? -print`
echo $subdirs
for x in $subdirs
do
echo $x
cp -a $x $2
done
Please check if this is what you wanted. It searches for directories with format xx_yy/ab_cd/&&_$$ (2char_2char) and copies the content to a new directory .
usage : ./script.sh
cat script.sh
#!/bin/bash
MYDIR="/media/test/"
NEWDIRPATH="/media/test_new"
DIRS=`ls -l $MYDIR | grep "^d" | awk '{print $9}'`
for DIR in $DIRS
do
total_characters=`echo $DIR | wc -m`
if [ $total_characters -eq 6 ]; then
has_underscore=`echo "$DIR" | grep "_"`
if [ "$has_underscore" != "" ]; then
echo "${DIR}"
start_string_count=`echo $DIR | awk -F '_' '{print $1}' | wc -m`
end_string_count=`echo $DIR | awk -F '_' '{print $2}' | wc -m`
echo "start_string_count => $start_string_count ; end_string_count => $end_string_count"
if [ $start_string_count -eq 3 ] && [ $end_string_count -eq 3 ]; then
mkdir -p $NEWDIRPATH/"$DIR"_new
cp -r $DIR $NEWDIRPATH/"$DIR"_new
fi
fi
fi
done

Use datestring in a filename to create folder directory and move files

The script I'm trying to pull of should move files to a destination folder and place them in "year/month/" folders according to the files name which starts with YYYY-MM-DD.
Example:
2013-08-03-image_name.png -> ~/B/uploads/2013/08/2013-08-03-image_name.png
2012-01-01-image_name.png -> ~/B/uploads/2012/01/2012-01-01-image_name.png
Plan of action
(1) Set path variables
source=~/Desktop/A/
targetPath=~/Desktop/B/uploads/
(2) Perform these actions on each file in $source
cd "$source";
for i in *.png
do
# STEP 3
# STEP 4
done
(3) Step 3: Image Optimization √
(4) Step 4: File away files to directory that machtes datename
(4a) Search for datestring in filename via ^(\d{4})-(\d{2}) and create $datePath, c.f. datePath=2013/08/. I image this something like this…
awk -F … somehow put the regex here with a search and replace "-" into "/"
and save it as a variable.
(4b) Create new target directory if it doesn't exist and move files there.
targetDir=$targetPath$datePath
mkdir -p $targetDir
mv -v "$i" "$destination"
PS: Bash would be nice.
I am providing you solution for finding target path for your files in pure BASH:
f='2013-08-03-image_name.png'
targetPath=~/Desktop/B/uploads/
[[ "$f" =~ ^([0-9]{4})-([0-9]{2}) ]] && \
echo "$targetPath${BASH_REMATCH[1]}/${BASH_REMATCH[2]}/$f"
OUTPUT:
~/Desktop/B/uploads/2013/08/2013-08-03-image_name.png
I'd use find + egrep to filter, then sed to build the name of the destination directory.
cd /src
IMAGES=`find . -type f -name '*.png' -print | egrep '^./[0-9]{4}-[0-9]{2}-[0-9]{2}-.+.png$'`
for IMG in $IMAGES; do
# optimize here
DIR=`echo $IMG | sed -E 's/^\.\/([0-9]{4})-([0-9]{2})-[0-9]{2}-.+.png/\1\/\2/'`
mkdir -p /dest/$DIR
mv /src/$IMG /dest/$DIR/
done
I think you will find glob useful and might find some inspiration in this question
Here's another bash solution, without using a regex/match:
srcdir=<whatever>
destdir=<whatever>
cd "${srcdir}"
for f in *-*-*-*.png
do
{ IFS=- read y m rest
[[ -d "${destdir}/${y}/${m}" ]] || mkdir -p "${destdir}/${y}/${m}"
echo mv "${f}" "${destdir}/${y}/${m}/${f}"
} <<< "${f}"
done
The for f in ... pattern may need some adjusting, depending on what other stuff you have in your source directory...
Remove the echo from in front of mv if you're satisfied with the proposed set of commands the above produces (or just pipe the whole thing into a subshell .... | bash).

If conditions in a Makefile, inside a target

I'm trying to setup a Makefile that will search and copy some files (if-else condition) and I can't figure out what exactly is wrong with it? (thou I'm pretty sure it's because a combination of spaces/tabs written in the wrong place).
Can I get some help with it, please?
Here's what I have currently:
obj-m = linuxmon.o
KDIR = /lib/modules/$(shell uname -r)/build
UNAME := $(shell uname -m)
all:
$(info Checking if custom header is needed)
ifeq ($(UNAME), x86_64)
$(info Yes)
F1_EXISTS=$(shell [ -e /usr/include/asm/unistd_32.h ] && echo 1 || echo 0 )
ifeq ($(F1_EXISTS), 1)
$(info Copying custom header)
$(shell sed -e 's/__NR_/__NR32_/g' /usr/include/asm/unistd_32.h > unistd_32.h)
else
F2_EXISTS=$(shell [[ -e /usr/include/asm-i386/unistd.h ]] && echo 1 || echo 0 )
ifeq ($(F2_EXISTS), 1)
$(info Copying custom header)
$(shell sed -e 's/__NR_/__NR32_/g' /usr/include/asm-i386/unistd.h > unistd_32.h)
else
$(error asm/unistd_32.h and asm-386/unistd.h does not exist)
endif
endif
$(info No)
endif
#make -C $(KDIR) M=$(PWD) modules
clean:
make -C $(KDIR) M=$(PWD) clean
rm unistd_32.h
Anyways, that'll print "Yes", "Copying header" twice and then it will quit saying that sed can't read /usr/include/asm-i386/unistd.h (which of course it can't read as I'm on a x64 system).
I could say that make just isn't understanding the if/else and instead is running everything line by line.
You can simply use shell commands. If you want to suppress echoing the output, use the "#" sign. For example:
clean:
#if [ "test" = "test" ]; then\
echo "Hello world";\
fi
Note that the closing ; and \ at each line are necessary
(This is because make interpret each line as a seperate command unless it ends with \)
There are several problems here, so I'll start with my usual high-level advice: Start small and simple, add complexity a little at a time, test at every step, and never add to code that doesn't work. (I really ought to have that hotkeyed.)
You're mixing Make syntax and shell syntax in a way that is just dizzying. You should never let it get this big without testing. Let's start from the outside and work inward.
UNAME := $(shell uname -m)
all:
$(info Checking if custom header is needed)
ifeq ($(UNAME), x86_64)
... do some things to build unistd_32.h
endif
#make -C $(KDIR) M=$(PWD) modules
So you want unistd_32.h built (maybe) before you invoke the second make, you can make it a prerequisite. And since you want that only in a certain case, you can put it in a conditional:
ifeq ($(UNAME), x86_64)
all: unistd_32.h
endif
all:
#make -C $(KDIR) M=$(PWD) modules
unistd_32.h:
... do some things to build unistd_32.h
Now for building unistd_32.h:
F1_EXISTS=$(shell [ -e /usr/include/asm/unistd_32.h ] && echo 1 || echo 0 )
ifeq ($(F1_EXISTS), 1)
$(info Copying custom header)
$(shell sed -e 's/__NR_/__NR32_/g' /usr/include/asm/unistd_32.h > unistd_32.h)
else
F2_EXISTS=$(shell [[ -e /usr/include/asm-i386/unistd.h ]] && echo 1 || echo 0 )
ifeq ($(F2_EXISTS), 1)
$(info Copying custom header)
$(shell sed -e 's/__NR_/__NR32_/g' /usr/include/asm-i386/unistd.h > unistd_32.h)
else
$(error asm/unistd_32.h and asm-386/unistd.h does not exist)
endif
endif
You are trying to build unistd.h from unistd_32.h; the only trick is that unistd_32.h could be in either of two places. The simplest way to clean this up is to use a vpath directive:
vpath unistd.h /usr/include/asm /usr/include/asm-i386
unistd_32.h: unistd.h
sed -e 's/__NR_/__NR32_/g' $< > $#

Bash - CD into Untared directory with variable URL

This is the situation. I have a list of URLs that I need to extract and setup. Its all variable driven, but after I extract, I dont know what my folder will be called. I cant CD into it if I dont know what its called.
$DL_DIR = /opt/
$URL = http://nginx.org/download/nginx-1.3.3.tar.gz
$FILE=${URL##*/}
$CONFIG = "-- core"
cd "$DL_DIR"
wget $URL
tar xzf $FILE
cd <HOW DO I GO INTO IT?>
./configure "$CONFIG"
make
make install
rm $FILE
If this doesnt explain it please say. I really want to get past this problem but Im having a hard time explaining it.
Since I want this to function for any set of URL's which may have two formats like ".tar.gz" or one format ".zip" and may have .'s in the filename like "Python2.3.4" or may not "Nginx", it makes it a bit tricky.
#! /bin/bash
#
# Problem:
# find the path of the "root" folder in an archive
#
# Strategy:
# list all folders in the archive.
# sort the list to make sure the shortest path is at the top.
# print the first line
#
# Weak point:
# assumes that tar tf and unzip -l will list files in a certain way
# that is: paths ending with / and that the file-list of unzip -l
# is in the fourth column.
#
LIST_FILES=
FILE=$1
case ${FILE##*.} in
gz)
LIST_FILES="tar tf $FILE"
;;
tgz)
LIST_FILES="tar tf $FILE"
;;
zip)
LIST_FILES='unzip -l '$FILE' | awk "{print \$4}"'
;;
esac
ARCHIVE_ROOT=$(
echo $LIST_FILES | sh |\
grep '/$'|\
sort |\
head -n1
)
# we should have what we need by now, go ahead and extract the files.
if [ -d "$ARCHIVE_ROOT" ]; then
cd "$ARCHIVE_ROOT"
else
# there is no path (whoever made the archive is a jerk)
# ...or the script failed (see weak points)
exit 1
fi
If you know that there is going to be exactly one directory in $DL_DIR, then you can use:
cd `ls -m1`
Another approach would be to loop through the files of the directory:
for filename in "$DL_DIR"/*
do
echo $filename
done;
You could perform file tests and other checks as necessary.
extract_dir=$(tar -tf $FILE | cut -d/ -f1 | uniq)
cd $extract_dir
or
extract_dir=$(tar -tf $FILE | head -1 | cut -d/ -f1)
cd $extract_dir
or
ls > .dir_list_1 # save current directory listing as hidden file
tar xzf $FILE # extract the $FILE
ls > .dir_list_2 # save the directory listing after extraction...
# ...as another hidden file
# diff two lists saved in hidden files, this will help you get the created dir
# grep '>' symbol, to get the inserted line
# use head to get the dir in case there are multiple lines (not necessary)
# use cut to remove the '>' and get the actual dir name, store in extract_dir
extract_dir=$(diff .dir_list_1 .dir_list_2 | grep '>' | head -1 | cut -d' ' -f2)
# remove temporary files
rm .dir_list_*
cd $extract_dir
I'd say, strip the extension of the file with ${FILE##*.} and do the other way around with the directory name using ${FILE%.ext*}:
case ${FILE##*.} in
gz)
tar xf $FILE
cd ${FILE%.tar.gz*}
;;
tgz)
tar xf $FILE
cd ${FILE%.tgz*}
;;
zip)
unzip $FILE
cd ${FILE%.zip*}
;;
esac
Just one small problem: how do you know if the directory in the archive has the same name af the archive itself?
How about this:
rm -rf tmpdir
mkdir tmpdir && cd tmpdir || exit
wget "$URL" || exit 1
case "$(ls)" in
*.tar.gz|*.tgz)
tar xzf $(ls)
;;
*.zip)
unzip $(ls)
;;
esac
for d in $(ls -d)
do
( cd "$d" 2>/dev/null && ./configure && make && make install; )
done