Understanding a Makefile with automatic dependencies: Problems with 'sed' part - regex

I have followed this tutorial. It explains the making of Makefile which take care of dependencies. I have made the following Makefile which works according to the following directory structure:
folder--|Makefile
|src----|(all .c and .h files here)
|obj----|(all objects file are made here)
|bin----|(target is made here)
The Makefile is:
TARGET = exec
CC = gcc
CFLAGS = -g -I.
LINKER = gcc -o
LFLAGS = -I. -lm -lpthread
BINDIR = bin
OBJDIR = obj
SRCDIR = src
INTERFACE = interface
STD = -std=c99
PROGRAMSOURCES := $(wildcard $(SRCDIR)/*.c)
PROGRAMINTERFACE:= $(wildcard $(INTERFACE)/*.h)
OBJECTS := $(PROGRAMSOURCES:$(SRCDIR)/%.c=$(OBJDIR)/%.o)
$(BINDIR)/$(TARGET) : $(OBJECTS)
$(LINKER) $# $(LFLAGS) $(OBJECTS) $(STD)
#pull the dependencies to the .o files
-include $(OBJECTS:.o=.d)
#the -o $# says to put the output of the compilation in the file named on the left side of the :.
#the $< is the first item in the dependencies list. Basically the name of the .c file which is to be compiled.
$(OBJECTS) : $(OBJDIR)/%.o :$(SRCDIR)/%.c
$(CC) $(CFLAGS) -c $< -o $# $(STD)
$(CC) $(CFLAGS) -MM $< > $*.d
#mv -f $*.d $*.d.tmp #changes file name
#sed -e 's|.*:|$*.o:|' < $*.d.tmp > $*.d #Unable to understand
#sed -e 's/.*://' -e 's/\\$$//' < $*.d.tmp | fmt -1 | \
sed -e 's/^ *//' -e 's/$$/:/' >> $*.d #Unable to understand
#rm -f $*.d.tmp
.PHONY : run
run :
./$(BINDIR)/$(TARGET) ${TYPE} ${INP_FILE}
print:
#echo $(OBJECTS)
I have understood that it is trying to pre-process the temp file to generate auto dependency. What I am unable to understand is how it is being done. Here are the two lines I want explanation of:
#sed -e 's|.*:|$*.o:|' < $*.d.tmp > $*.d #Unable to understand
#sed -e 's/.*://' -e 's/\\$$//' < $*.d.tmp | fmt -1 | \
sed -e 's/^ *//' -e 's/$$/:/' >> $*.d #Unable to understand
I have never used sed before so I am having problems.
Any help appreciated.

Suppose you are building obj/foo.o from src/foo.c, so the file foo.d.tmp contains:
foo.o: src/foo.c src/foo.h src/bar.h
Now the first sed statement:
#sed -e 's|.*:|$*.o:|' < $*.d.tmp > $*.d
"Read foo.d.tmp, take everything up to a colon and change it to 'foo.o', and write the result to foo.d." So now foo.d contains:
foo.o: src/foo.c src/foo.h src/bar.h
(No change in this case.) Now the next command:
#sed -e 's/.*://' -e 's/\\$$//' < $*.d.tmp | fmt -1 | sed -e 's/^ *//' -e 's/$$/:/' >> $*.d
"Read foo.d.tmp, remove everything up to (and including) a colon, remove the trailing \ if there is one. Take the result and put each word (i.e each prerequisite) on its own line. Then for each line, remove the leading spaces, put a colon at the end, and append the result to foo.d." So now foo.d contains:
foo.o: src/foo.c src/foo.h src/bar.h
src/foo.c:
src/foo.h:
src/bar.h:
The idea is to create an empty rule for each prerequisite, so that if the code has changed and a certain prerequisite is no longer needed -- and no longer present -- but it is still listed in the old foo.d, Make will not panic at being unable to build it.

Related

Sed replace a string in the first line of a paragraph

I am trying to automate the periodic detection and elimination of files, using fdupes. I got this beautiful script:
# from here:
# https://www.techrepublic.com/blog/linux-and-open-source/how-to-remove-duplicate-files-without-wasting-time/
OUTF=rem-duplicates_2019-01.sh;
echo "#! /bin/sh" > $OUTF;
find "$#" -type f -printf "%s\n" | sort -n | uniq -d |
xargs -I## -n1 find "$#" -type f -size ##c -exec md5sum {} \; |
sort --key=1,32 | uniq -w 32 -d --all-repeated=separate |
sed -r 's/^[0-9a-f]*( )*//;s/([^a-zA-Z0-9./_-])/\\\1/g;s/(.+)/#rm \1/;' >> $OUTF;
chmod a+x $OUTF; ls -l $OUTF
This produces a file with this structure:
#! /bin/sh
#rm ./directory_a/file_a
#rm ./directory_b/file_identical_to_a
#rm ./directory_a/file_b
#rm ./directory_b/file_identical_to_b
#rm ./directory_c/another_file_identical_to_b
#rm ./directory_a/file_c
#rm ./directory_b/file_identical_to_c
#rm ./directory_c/another_file_identical_to_c
#rm ./directory_d/yet_another_file_identical_to_c
I want to remove the # tag from the first line of each paragraph to get
rm ./directory_a/file_c
#rm ./directory_b/file_identical_to_c
#rm ./directory_c/another_file_identical_to_c
#rm ./directory_d/yet_another_file_identical_to_c
I have been trying to modify the next-to-last line, with variations of things like this:
sed -r 's/^[0-9a-f]*( )*//;s/([^a-zA-Z0-9./_-])/\\\1/g;s/(.+)/#rm \1/;s/\n\n#rm/\n\nrm/;' >> $OUTF;
But cannot manage SED to recognize the (\n\n) or any other pointer I can think of to the beginning of the paragraph. What am I doing wrong?
Edit: I am unable to edit the comment, so here is the final script:
TEMPF=temp.txt;
OUTF=rem-duplic_2019-01.sh
echo "#! /bin/sh" > $TEMPF;
find "$#" -type f -printf "%s\n" | sort -n | uniq -d |
xargs -I## -n1 find "$#" -type f -size ##c -exec md5sum {} \; |
sort --key=1,32 | uniq -w 32 -d --all-repeated=separate |
sed -r 's/^[0-9a-f]*( )*//;s/([^a-zA-Z0-9./_-])/\\\1/g;s/(.+)/#rm \1/' >> $TEMPF;
awk -v a=2 '/^$/{a=2}!--a{sub(/#/,"")}1' $TEMPF > $OUTF
chmod a+x $OUTF; ls -l $OUTF
rm $TEMPF
Use awk instead:
awk '/^$/{a=1} !a--{sub(/#/,"")} 1' a=1 file
/^$/ { a = 1 } means set a to 1 if current line is a blank one,
!a-- is a shorthand for a-- == 0, following action ({ sub(/#/, "") }) removes the first # from current line,
1 means print all lines,
a=1 is required to remove # from the line after shebang (i.e 2nd line).
With sed:
sed "1n;/^#/,/^$/{ s///;}" file
You can use this too:
sed '/^$\|^#!/{N;s/#r/r/}' input.txt
feel free to add the in-place opt if you want
This might work for you (GNU sed):
sed '/^#!\|^\s*$/{n;s/.//}' file
If the current line is a shebang or an empty line, print it and remove the first character of the next line.
Just use Perl with paragraph mode
perl -00 -pe ' s/^#// '
With inputs
$ cat yozzarian.txt
#! /bin/sh
#rm ./directory_a/file_a
#rm ./directory_b/file_identical_to_a
#rm ./directory_a/file_b
#rm ./directory_b/file_identical_to_b
#rm ./directory_c/another_file_identical_to_b
#rm ./directory_a/file_c
#rm ./directory_b/file_identical_to_c
#rm ./directory_c/another_file_identical_to_c
#rm ./directory_d/yet_another_file_identical_to_c
$ perl -00 -pe ' s/^#// ' yozzarian.txt
! /bin/sh
#rm ./directory_a/file_a
#rm ./directory_b/file_identical_to_a
rm ./directory_a/file_b
#rm ./directory_b/file_identical_to_b
#rm ./directory_c/another_file_identical_to_b
rm ./directory_a/file_c
#rm ./directory_b/file_identical_to_c
#rm ./directory_c/another_file_identical_to_c
#rm ./directory_d/yet_another_file_identical_to_c
$

gcc -MM on cygwin: sometimes unix style paths

I have a (legacy) bash script that parses a list of directories for source files and echos them into a makefile to create actual makerules with gcc -MM. The script works just fine, the problem is that running make on the generated makefile results in inconsistent path format in the makerules - while most makerules have mixed style paths (which is ok), some have cygwin style paths (not ok). This didn't happen on a very old (~2002 release) of cygwin.
Here is the script:
#!/bin/bash
fn_source=SourceFileList
fn_depmake=depMakefile
fn_orgMake=Makefile
fn_depIncl=Depend.incl
fn_optIncl=OptimizationCflags.incl
#
# create a Makefile to determine the dependencies
echo -n create dep-makefile ...\
#
# generate include for Optimization-File
#
echo > $fn_depIncl
echo include $fn_optIncl >> $fn_depIncl
echo >> $fn_depIncl
#
touch $fn_optIncl
#
echo > $fn_depmake
# CFLAGS options from Makefile
grep "^CFLAGS[ =]" $fn_orgMake | grep $BUILD_SPEC >> $fn_depmake
grep "^CFLAGS_AS[ =]" $fn_orgMake | grep $BUILD_SPEC >> $fn_depmake
grep "^CFLAGS_GCC[ =]" $fn_orgMake | grep $BUILD_SPEC >> $fn_depmake
echo -n generate depMakefile ...
echo >> $fn_depmake
# the default rule
echo all: >> $fn_depmake
FileList=`cat $fn_source`
incPath=""
rawIncPath=""
for fn in $FileList; do
incPath=$incPath" -I"${fn%/*}
rawIncPath=$rawIncPath" "${fn%/*}
done
# filter out redundant paths
echo $incPath | \
sed -n -e 's/ /\
/gp' | \
sort -u > _tmp
incPath=`cat _tmp`
# filter out redundant paths
echo $rawIncPath | \
sed -n -e 's/ /\
/gp' | \
sort -u > _tmp
rawIncPath=`cat _tmp`
#
# create file with include paths
#
echo LibIncludes=$incPath > LibIncludes
#
# put all source files with the same path in one gcc command
#
for path in $rawIncPath; do
sourceFiles=`grep $path/ $fn_source`
echo " "echo do $path >> $fn_depmake
echo " "gcc -MM '$(CFLAGS_GCC)' -I$path $sourceFiles \>\> _tmp >> $fn_depmake
done
# start the Makefile
echo start Makefile ...
rm _tmp; touch _tmp
make -s -f $fn_depmake
# add ./obj/xxxxx*.o to the make-rules
sed -e 's/\(^[a-zA-Z]\)/.\/obj\/\1/' < _tmp > _tmp1
# add command to the make-rules
sed -e '/\\/!a\
$(CC) $($(*F)CFLAGS) -I$(<D) $(CFLAGS) -c $< -o $#' < _tmp1 >> $fn_depIncl
# workaround for gcc generating cygdrive paths for some paths only
sed -i 's#/cygdrive/d#d:#' $fn_depIncl
rm _tmp _tmp1
rm $fn_depmake
echo done.
As you can see, I am currently using a workaround to convert those paths to mixed styled ones using sed, but I would like to know why this happens in the first place?
Example of a rule gone "wrong":
some.o: d:/some/path/A/some.cpp \
d:/some/path/A/some.h \
/cygdrive/d/some/path/B/subpathA/anotherheader.h \
/cygdrive/d/some/path/B/subpathA/yetanotherheader.h \
D:/some/path/B/subpathB/header.h
The main problem about this is that the resulting makefile is fed to a windows native make.exe, which has no idea what /cygdrive/d/path is supposed to be.
I have looked at the paths after each step, and they are all mixed style until make -s -f $fn_depmake is executed. Meaning, the occasional conversion to cygwin style paths is done by the gcc -MM command in the generated makefile.
It is always the same list of directories that get converted to /cygdrive/d/pathA/somefile.cpp style of paths. There is nothing unusual about the path, no spaces, only a-z/A-Z in name, length of the path is short, longest path is 76 characters including the filename, base path is 34 characters long.
How can I force gcc to generate mixed style paths?
I can see the same problem on my project. I'm using gmake.exe from Cygwin on Windows.
One invocation of gcc -MM outputs both Windows-style paths (C:/project/file.h) and Cygwin-style paths (/cygdrive/c/project/file.h).
I don't know how to fix that yet, but I found what triggers the change to Cygwin-style paths. This happens when an include uses a relative path. So, when a file contains #include "../file.h", then the output of gcc -MM will contain the absolute path of file.h, using a Cygwin-style path.

Using sed to replace multiple instances in the same line before a comment

I am trying to use sed to replace all instances of a command with a variable, expect when they come after a comment or is part of another word. I have gotten close, being able to replace one instance before a comment, but not if there is more than one.
I have a test file with the line:
rm rm # rm
I want to make this read:
$RM $RM # rm
This is what I have so far:
sed -i 's/\(^\|[^[#.*]]\)\brm\b/\1$RM/' file1
Which returns:
$RM rm # rm
Any help is much appreciated. Other solutions not involving sed are welcome, but I might need some help understanding them.
Thanks!
EDIT:
This is just an example of what I am looking for. Not every line will be formatted like this, and not every line will contain a command before the comment, or vise versa. I am just looking for a solution that will also cover a situation similar to this example. Sorry for the lack of explanation. Here is a slightly better example:
"$#" #rm
# rm
rm # rm
rm
"rm "
'rm '
`rm `
{rm }
$# rm # rm
rm rm # rm
rm # rm rm
rmremovermlink
Output should be:
"$#" #rm
# rm
$RM # rm
$RM
"$RM "
'$RM '
`$RM `
{$RM }
$# $RM # rm
$RM $RM # rm
$RM # rm rm
rmremovermlink
You can use this sed command:
sed ':a;s/^\([^#]*\)\<rm\>/\1$RM/;ta;' file
# ^ ^ ^ ^ ^ ^ ^---- go to label "a" if something is replaced
# | | | | | '---- back-reference to capture group 1
# | | | '---'---- word boundaries
# | | '---- capture group 1
# | '---- replacement (only one occurrence)
# '---- defines the label "a"
You can use this perl command for this substitution:
perl -pe 's/(?<!\$)#.*$(*SKIP)(*F)|\brm\b/\$RM/g' file
"$#" #rm
# rm
$RM # rm
$RM
"$RM "
'$RM '
`$RM `
{$RM }
$# $RM # rm
$RM $RM # rm
$RM # rm rm
rmremovermlink
RegEx Demo
\$[A-Z]{2}\s\$[A-Z]{2}\s\#\s[a-z]{2}
Let me know if this is what you're looking for. It matches :
$RM $RM # rm

Makefile syntax error. Missing separator

I need to understand the syntax of a makefile prior to modify it to fit my own needs. Searching through the net I have been able to understand most of it, but the last part is giving me a syntax error:
Makefile:119: *** missing separator. Stop.
line 119 is the first one here:
rm -rf $(CLEANFILES)
.cpp.o:
$(CXX) $(CXXFLAGS) $(INCL) -c -o $# `test -f '$<' || echo '$(SRCDIR)/'`$<
.cpp.obj:
$(CXX) $(CXXFLAGS) $(INCL) -c -o $# `if test -f '$<'; then $(CYGPATH_W) '$<'; else $(CYGPATH_W) '$(SRCDIR)/$<'; fi`
.c.o:
$(CC) $(CFLAGS) $(INCL) -c -o $# `test -f '$<' || echo '$(SRCDIR)/'`$<
.c.obj:
$(CC) $(CFLAGS) $(INCL) -c -o $# `if test -f '$<'; then $(CYGPATH_W) '$<'; else $(CYGPATH_W) '$(SRCDIR)/$<'; fi`
I don't really get what's being done in these last lines, and I can't seem to find the information I need in the GNU make manual,
Thanks
Since you've cut-and-pasted from your actual Makefile it's hard to give a certain answer. If the first line you provided, rm -rf $(CLEANFILES) truly is line 119 and the makefile is valid until then, it's likely that you don't have a TAB preceding the rm -rf $(CLEANFILES). It should look like this:
clean:
rm -rf $(CLEANFILES)
.cpp.o:
$(CXX) $(CXXFLAGS) $(INCL) -c -o $# `test -f '$<' || echo '$(SRCDIR)/'`$<
.cpp.obj:
$(CXX) $(CXXFLAGS) $(INCL) -c -o $# `if test -f '$<'; then $(CYGPATH_W) '$<'; else $(CYGPATH_W) '$(SRCDIR)/$<'; fi`
.c.o:
$(CC) $(CFLAGS) $(INCL) -c -o $# `test -f '$<' || echo '$(SRCDIR)/'`$<
.c.obj:
$(CC) $(CFLAGS) $(INCL) -c -o $# `if test -f '$<'; then $(CYGPATH_W) '$<'; else $(CYGPATH_W) '$(SRCDIR)/$<'; fi`
That is, the command lines in each recipe must have a literal TAB character at the start of the line.

If conditions in a Makefile, inside a target

I'm trying to setup a Makefile that will search and copy some files (if-else condition) and I can't figure out what exactly is wrong with it? (thou I'm pretty sure it's because a combination of spaces/tabs written in the wrong place).
Can I get some help with it, please?
Here's what I have currently:
obj-m = linuxmon.o
KDIR = /lib/modules/$(shell uname -r)/build
UNAME := $(shell uname -m)
all:
$(info Checking if custom header is needed)
ifeq ($(UNAME), x86_64)
$(info Yes)
F1_EXISTS=$(shell [ -e /usr/include/asm/unistd_32.h ] && echo 1 || echo 0 )
ifeq ($(F1_EXISTS), 1)
$(info Copying custom header)
$(shell sed -e 's/__NR_/__NR32_/g' /usr/include/asm/unistd_32.h > unistd_32.h)
else
F2_EXISTS=$(shell [[ -e /usr/include/asm-i386/unistd.h ]] && echo 1 || echo 0 )
ifeq ($(F2_EXISTS), 1)
$(info Copying custom header)
$(shell sed -e 's/__NR_/__NR32_/g' /usr/include/asm-i386/unistd.h > unistd_32.h)
else
$(error asm/unistd_32.h and asm-386/unistd.h does not exist)
endif
endif
$(info No)
endif
#make -C $(KDIR) M=$(PWD) modules
clean:
make -C $(KDIR) M=$(PWD) clean
rm unistd_32.h
Anyways, that'll print "Yes", "Copying header" twice and then it will quit saying that sed can't read /usr/include/asm-i386/unistd.h (which of course it can't read as I'm on a x64 system).
I could say that make just isn't understanding the if/else and instead is running everything line by line.
You can simply use shell commands. If you want to suppress echoing the output, use the "#" sign. For example:
clean:
#if [ "test" = "test" ]; then\
echo "Hello world";\
fi
Note that the closing ; and \ at each line are necessary
(This is because make interpret each line as a seperate command unless it ends with \)
There are several problems here, so I'll start with my usual high-level advice: Start small and simple, add complexity a little at a time, test at every step, and never add to code that doesn't work. (I really ought to have that hotkeyed.)
You're mixing Make syntax and shell syntax in a way that is just dizzying. You should never let it get this big without testing. Let's start from the outside and work inward.
UNAME := $(shell uname -m)
all:
$(info Checking if custom header is needed)
ifeq ($(UNAME), x86_64)
... do some things to build unistd_32.h
endif
#make -C $(KDIR) M=$(PWD) modules
So you want unistd_32.h built (maybe) before you invoke the second make, you can make it a prerequisite. And since you want that only in a certain case, you can put it in a conditional:
ifeq ($(UNAME), x86_64)
all: unistd_32.h
endif
all:
#make -C $(KDIR) M=$(PWD) modules
unistd_32.h:
... do some things to build unistd_32.h
Now for building unistd_32.h:
F1_EXISTS=$(shell [ -e /usr/include/asm/unistd_32.h ] && echo 1 || echo 0 )
ifeq ($(F1_EXISTS), 1)
$(info Copying custom header)
$(shell sed -e 's/__NR_/__NR32_/g' /usr/include/asm/unistd_32.h > unistd_32.h)
else
F2_EXISTS=$(shell [[ -e /usr/include/asm-i386/unistd.h ]] && echo 1 || echo 0 )
ifeq ($(F2_EXISTS), 1)
$(info Copying custom header)
$(shell sed -e 's/__NR_/__NR32_/g' /usr/include/asm-i386/unistd.h > unistd_32.h)
else
$(error asm/unistd_32.h and asm-386/unistd.h does not exist)
endif
endif
You are trying to build unistd.h from unistd_32.h; the only trick is that unistd_32.h could be in either of two places. The simplest way to clean this up is to use a vpath directive:
vpath unistd.h /usr/include/asm /usr/include/asm-i386
unistd_32.h: unistd.h
sed -e 's/__NR_/__NR32_/g' $< > $#