Im back porting ffmpeg to an older version of debian.
everything is going well, but its so slow.
I am running dpkg-buildpackage -us -uc
with a debian rules file that looks like this:
#!/usr/bin/make -f
%:
dh $#
override_dh_auto_configure:
./configure
I notice, this is only processing on 1 core.
is there anything like make -j 4 that I could use to speed this up?
I've been using this guide, but i don't see anything for speeding up the build step
https://www.debian.org/doc/manuals/maint-guide/
Sure, you can use -j 4 as an argument to dpkg-buildpackage. It is documented in the man page. The relevant section is:
-jjobs Number of jobs allowed to be run simultaneously, equivalent to
the make(1) option of the same name. Will add itself to
the MAKEFLAGS environment variable, which should cause all
subsequent make invocations to inherit the option. Also adds
parallel=jobs to the DEB_BUILD_OPTIONS environment variable which
allows debian/rules files to use this information for their own
purposes. The parallel=jobs in DEB_BUILD_OPTIONS environment
variable will override the -j value if this option is given.
Related
I build the LLVM from source in release edtion.Now get the loop or cycle information is what i need.But it just can't work.
$opt -passes=print<cycles> input.ll -o /dev/null
zsh: no such file or directory: cycles
The unix command opt -passes=print<cycles> input.ll means to run the command opt -passes=print while reading from the file named cycles and writing to the file named input.ll.
Have a look at the answers to this question for more about this shell syntax.
You may want to use opt -passes='print<cycles>', that will at least not be interpreted by the shell, but I fear that won't work either. I've no idea what print<cycles> is supposed to achieve.
I want to move to make 4.2 so I can have the ability to do parallel jobs on Windows using --jobs=N.
make 4.2 is not available in binary form from SourceForge, only 3.81
So I had to build it from source - they included a VS studio project - and I was up and running.
Problem is, it appears to have a bug though it could be a bug with my makefile.
I ran make4.2 all --print-data-base --debug=vjm on my project and I got ERRORS.
What's unfortunate is that the error is NOT related to syntax of rules as this project builds without error on make 3.81 using the same options.
Here is the command it fails on.
COPY_DIR = cp -f -R
MKDIR = mkdir -p
deploy_marketProperties: deploy_themeMappingConfig
#echo Copying application data... && $(MKDIR) "C:/Users/User1/Desktop/TargetDir" && $(COPY_DIR) $(wildcard C:/Users/User1/Desktop/Source/*) "C:/Users/User1/Desktop/TargetDir"
Creates a directory
Copies bunch of files from one directory to that new directory
Used the same make call for both versions of make:
make all --print-data-base --debug=vjm
However, I get the following error when using make4.2:
Copying application data...
cp: target `C' is not a directory
makev42[3]: *** [Makefile:383: deploy_marketProperties]
The command it tried running was:
echo Copying application data... && mkdir -p "C:/Users/User1/Desktop/TargetDir" && cp -f -R C:/Users/User1/Desktop/Source/im.json "C:/Users/User1/Desktop/TargetDir"
There's actually 264 .json files but I only show one for readability.
Again, this runs fine on make v3.81 using exact same command-line options.
It appears make 4.2 has a problem parsing that huge list of files maybe??
The question
Is there something I can change with my makefiles to make it run on make v4.2?
We are highly motivated to make it work with a 4.0+ version of make. Otherwise we have to use a build tool that will cost $$.
UPDATE 1
Per the suggestion of #MadScientist, I have executed both versions of make and present the outputs.
C:\release>make381 deploy_marketProperties --debug=vjm > 381.err 2>&1
C:\release>make42 deploy_marketProperties
--debug=vjm > 42.err 2>&1
The target deploy_marketProperties has the following rule:
deploy_marketProperties:
echo Copying application data... && $(MKDIR) "C:/Users/User1/Desktop/A/Project/src/../lib/armle-v7/release/marketProperties" && $(COPY_DIR) $(wildcard C:/Users/User1/Desktop/A/Project/src/dir1/marketProperties/*) "C:/Users/User1/Desktop/A/Project/src/../lib/armle-v7/release/marketProperties"
The output of make v 3.81.
NOTE: there's actually 264 .json files and total number of characters that make up the list of .json files, including ONE white space per file, is 18251 (in case that's relevant):
GNU Make 3.81
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
This program built for Windows32
find_and_set_shell path search set default_shell = C:/QNX650/host/win32/x86/usr/bin/sh.exe
Reading makefiles...
Reading makefile `Makefile'...
Updating goal targets....
Considering target file `deploy_marketProperties'.
File `deploy_marketProperties' does not exist.
Finished prerequisites of target file `deploy_marketProperties'.
Must remake target `deploy_marketProperties'.
echo Copying application data... && mkdir -p "C:/Users/User/Desktop/A/Project/src/../lib/armle-v7/release/marketProperties" && cp -f -R C:/Users/User/Desktop/A/Project/src/fordhmi/marketProperties/2blank.json C:/Users/User/Desktop/A/Project/src/fordhmi/marketProperties/3blank.json "C:/Users/User/Desktop/A/Project/src/../lib/armle-v7/release/marketProperties"
Main thread handle = 0x000000ac
Copying application data...
Successfully remade target file `deploy_marketProperties'.
The output of make v 4.2.
NOTE: there's actually 264 .json files and total number of characters that make up the list of .json files, including ONE white space per file, is 18251 (in case that's relevant):
GNU Make 4.2
Built for Windows32
Copyright (C) 1988-2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
find_and_set_shell() path search set default_shell = C:/QNX650/host/win32/x86/usr/bin/sh.exe
Reading makefiles...
Reading makefile 'Makefile'...
Updating makefiles....
Updating goal targets....
Considering target file 'deploy_marketProperties'.
File 'deploy_marketProperties' does not exist.
Finished prerequisites of target file 'deploy_marketProperties'.
Must remake target 'deploy_marketProperties'.
echo Copying application data... && mkdir -p "C:/Users/User1/Desktop/A/Project/src/../lib/armle-v7/release/marketProperties" && cp -f -R C:/Users/User1/Desktop/A/Project/src/SubProj/marketProperties/im.json C:/Users/User1/Desktop/A/Project/src/SubProj/marketProperties/da.json "C:/Users/User1/Desktop/A/Project/src/../lib/armle-v7/release/marketProperties"
Main thread handle = 00000088
Copying application data...
cp: target `C' is not a directory
make_msvc.net2003: *** [Makefile:384: deploy_marketProperties] Error 1
Note that both make versions use the same shell:
C:\Users\User1\Desktop\A\Project\bld\armle-v7\release\subProj>where sh
C:\QNX650\host\win32\x86\usr\bin\sh.exe
C:\Users\User1\Desktop\A\Project\bld\armle-v7\release\subProj>sh --help
GNU bash, version 3.1.17(1)-release-(i686-pc-msys)
Usage: sh [GNU long option] [option] ...
sh [GNU long option] [option] script-file ...
GNU long options:
--debug
--debugger
--dump-po-strings
--dump-strings
--help
--init-file
--login
--noediting
--noprofile
--norc
--posix
--protected
--rcfile
--restricted
--verbose
--version
--wordexp
Shell options:
-irsD or -c command or -O shopt_option (invocation only)
-abefhkmnptuvxBCHP or -o option
Type `sh -c "help set"' for more information about shell options.
Type `sh -c help' for more information about shell builtin commands.
Use the `bashbug' command to report bugs.
NOTE THE msys #MadScientist; is that relevant?
UPDATE 2
I re-ran make 4.2 but I deleted all but ONE .json file in the directory and - surprise! - it works.
This is the ENTIRE output from make42 deploy_marketProperties --debug=vjm:
GNU Make 4.2
Built for Windows32
Copyright (C) 1988-2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
find_and_set_shell() path search set default_shell = C:/QNX650/host/win32/x86/usr/bin/sh.exe
Reading makefiles...
Reading makefile 'Makefile'...
Updating makefiles....
Updating goal targets....
Considering target file 'deploy_marketProperties'.
File 'deploy_marketProperties' does not exist.
Finished prerequisites of target file 'deploy_marketProperties'.
Must remake target 'deploy_marketProperties'.
echo Copying application data... && mkdir -p "C:/Users/User1/Desktop/A/Project/src/../lib/armle-v7/release/marketProperties" && cp -f -R C:/Users/User1/Desktop/A/Project/src/subProject/marketProperties/2blank.json "C:/Users/User1/Desktop/A/Project/src/../lib/armle-v7/release/marketProperties"
Main thread handle = 000000B4
Copying application data...
Successfully remade target file 'deploy_marketProperties'.
UPDATE 3
I tried it with make 4.2.1 and I got the exact same behavior as on make 4.2: fails when there are hundreds of .json files but succeeds if there less than some magic number of characters.
I got this message during one of my trials:
cp: target `C:/Users/User1/Des' is not a directory
Obviously it's getting cut-off but why?
Same shell being used and only thing different is version of make.
You should use 4.2.1, it has some important fixes (I doubt it will make a difference for this problem).
As discussed in the comments, using POSIX tools on Windows requires a specialized environment. The normal "shell" on Windows is command.com, which doesn't accept POSIX sh syntax, such as &&. The normal mkdir command on Windows doesn't accept the -p flag; that's a POSIX flag. There is no cp command on Windows.
So it's clear that your makefile is expecting to be run with some sort of POSIX environment.
Based on what you've described above, it's not clear how the problem is related to GNU make directly. GNU make is parsing the makefile and it's invoking the shell and passing along the command line properly: this can be inferred because the shell is running the echo command and running the mkdir command, and even running the cp command.
It's the cp command which is printing the message, not make. I believe you that it works with 3.81, but I suspect a non-obvious difference between the way 3.81 was compiled and the way you're building 4.2. It's even possible that there have been custom patches applied to the 3.81 version you're using, although I don't know.
To debug further I recommend the following:
First remove the # prefix from the recipe line: that's always the number one thing to do; if you can't see how make is invoking the command how can you debug?
Then run the same makefile with the two different versions of make, from the same shell/prompt (so the path to make is the only difference). Is there any difference in the printed output? If so what is it?
If not, then clearly there's something more mysterious going on. Try to see if maybe the QNX version of make is modifying the environment somehow: you can run the env command in your recipe to see it.
Maybe it's related to the length of the command line? Try temporarily moving some of the json files out of that directory so the command line is shorter.
I asked the make-w32#gnu.org mailing list and I have the answer.
To fix it, I had to uncomment the macro BATCH_MODE_ONLY_SHELL in config.h of the make source code.
/*
* If you have a shell that does not grok 'sh -c quoted-command-line'
* correctly, you need this setting. Please see below for specific
* shell support.
*/
#define BATCH_MODE_ONLY_SHELL 1
Taking quote from reply from mailing list:
What that flag means (as I understand it: I'm not that familiar with
this aspect of Windows support) is that make will never try to invoke
the shell directly passing the recipe to be run on the command line.
Instead it is being forced to always write the recipe to a temporary
file ("batch file") on your disk and invoke the shell such that it
runs the recipe in the temporary file.
Apparently your shell is able to read and execute very long command
lines from a file, which it cannot successfully process when passed in
as arguments, even though these command line arguments are not
exceeding Windows limits.
I am trying to start / stop CAN boards or to update their baudrate using SocketCAN from userspace. My tests are performed on PeakSystem and IXXAT USB-to-CAN V1/V2 boards.
My first attempt was to use visudo and to enable NOPASSWD to "ip link set ...", and then to call "sudo ip link set ..." in my C++ code.
Complete visudo line is:
%sudo ALL=(ALL:ALL) NOPASSWD: /bin/ip link set can[0123456789]* type can bitrate [0123456789]*, /bin/ip link set can[0123456789]* up, /bin/ip link set can[0123456789]* down
Then, I tried with Linux capabilities by adding capabilities to /bin/ip. That allows me to call "ip link set ..." from my C++ code which was even better.
Add capabilities command:
sudo setcap cap_net_raw,cap_net_admin+ep /bin/ip
But then I discovered libsocketcan which is a far better approach than calling command lines from C++. However when calling "can_set_bitrate" or "can_do_start", I have an error "RTNETLINK: Operation not permitted". But things are working fine when my program is launched as root. Other functions like can_get_state are working fine in userspace (actually, they are returning : 4 -> CAN_STATE_STOPPED).
I tried to adding capabilities to my program without any success ""sudo setcap cap_net_raw,cap_net_admin+ep ./myprogram".
How can I allow my program to use libsocketcan in userspace?
Thanks for your help!
sudo setcap cap_net_raw,cap_net_admin+ep your_executable is the solution. You may need to compile with -Wl,-rpath so your program finds its shared libraries (also see this answer)
I have fortran code that has been parallelized with OpenMP. I want to test my code on my PC before running on HPC. My PC has double core CPU and I work on Linux-mint. I installed gfortranmultilib and this is my script:
#!/bin/bash
### Job name
#PBS -N pme
### Keep Output and Error
#PBS -j eo
### Specify the number of nodes and thread (ppn) for your job.
#PBS -l nodes=1:ppn=2
### Switch to the working directory;
cd $PBS_O_WORKDIR
### Run:
OMP_NUM_THREADS=$PBS_NUM_PPN
export OMP_NUM_THREADS
ulimit -s unlimited
./a.out
echo 'done'
What should I do more to run my code?
OK, I changed script as suggested in answers:
#!/bin/bash
### Switch to the working directory;
cd Desktop/test
### Run:
OMP_NUM_THREADS=2
export OMP_NUM_THREADS
ulimit -s unlimited
./a.out
echo 'done'
my code and its executable file are in folder test on Desktop, so:
cd Desktop/test
is this correct?
then I compile my simple code:
implicit none
!$OMP PARALLEL
write(6,*)'hi'
!$OMP END PARALLEL
end
by command:
gfortran -fopenmp test.f
and then run by:
./a.out
but only one "hi" is printed as output. What should I do?
(and a question about this site: in situation like this I should edit my post or just add a comment?)
You don't need and probably don't want to use the script on your PC. Not even to learn how to use such a script, because these scripts are too much connected to the specifics of each supercomputer.
I use several supercomputers/clusters and I cannot just reuse the script from one at the other, because they are so much different.
On your PC you should just do:
optional, it is probably the default
export OMP_NUM_THREADS=2
to set the number of OpenMP threads to 2. Adjust if you need some other number.
cd to the working directory
cd my_working_directory
Your working directory is the directory where you have the required data or where the executable resides. In your case it seems to be the directory where a.out is.
run the damn thing
ulimit -s unlimited
./a.out
That's it.
You can also store the standard output and error output to a file
./out > out.txt 2> err.txt
to mimic the supercomputer behaviour.
The PBS variables are only set when you run the script using qsub. You probably don't have that on your PC and you probably don't want to have it either.
$PBS_O_WORKDIR is the directory where you run the qsub command, unless you set it differently by other means.
$PBS_NUM_PPN is the number you indicated in #PBS -l nodes=1:ppn=2. The queue system reads that and sets this variable for you.
The script you posted is for Portable Batch System (https://en.wikipedia.org/wiki/Portable_Batch_System) queue system. That means, that the job you want to run on the HPC infrastructure has to go first into the queue system and when the resources are available the job will run on the system.
Some of the commands (those starting with #PBS) are specific commands for this queue system. Among these commands, some allow the user to indicate the application process hierarchy (i.e. number of processes and threads). Also, keep in mind that since all the PBS commands start by # they are ignored by regular shell script execution. In the case you presented, that is given by
### Specify the number of nodes and thread (ppn) for your job.
#PBS -l nodes=1:ppn=2
which as the comment indicates it should tell the queue system that you want to run 1 process and each process will have 2 threads. The queue system is likely to pass these parameters to the process launcher (srun/mpirun/aprun/... for MPI apps in addition to OMP_NUM_THREADS for OpenMP apps).
If you want to run this job on a computer that does not have PBS queue, you should be aware at least of two things.
1) The following command
### Switch to the working directory;
cd $PBS_O_WORKDIR
will be translated into "cd" because the environment variable PBS_O_WORKDIR is only defined within the PBS job context. So, you should change this command (or execute another cd command just before the execution) in order to fix where you want to run the job.
2) Similarly for PBS_NUM_PPN environment variable,
OMP_NUM_THREADS=$PBS_NUM_PPN
export OMP_NUM_THREADS
this variable won't be defined if you don't run this within a PBS job context, so you should set OMP_NUM_THREADS to the value you want (2, according to your question) manually.
If you want your linux box environment to be like an HPC login node. You can do the following
Make sure that your compiler supports OpenMP, test a simple hello world program with OpenMP flags
Install OpenMPI on your system from your favourite package manager or download the source/binary from the website (OpenMPI Download)
I would not recommend installing cluster manager like Slurm for your experiments
After you are done, you can execute your MPI programs through the mpirun wrapper
mpirun -n <no_of_cores> <executable>
EDIT:
This is assuming that you are running this only MPI. Note that OpenMP utilizes the cores as well. If you are running MPI+OpenMP - n*OMP_NUM_THREADS=cores on a single node.
Hello I am a beginner with Linux platform therefore I am not familiar with the terminal commands.
I am writing an application on C++ and I expect it to consume a lot of processing power. So I want to make sure I am using all available cores on my device (it has 4 cores).
I am using the following to create an executable file:
gcc -o blink -l rt blink.c -l bcm2835
where bcm2835 is the library I use for I/O. So my question is, is this command is using all available cores or is there anything I can do to optimize it? I am willing to use everything available, to throw the kitchen sink if it will make this code run faster.
The -j jobs option is for make not gcc
When used with make it will cause multiple "recipes" to be executed in parallel. In this context, your gcc line is one recipe.
AFTER QUESTION REVISION
If you want your code to use multiple cores, you will need to use threads or processes. Look into pthreads.
Since you're using C++, you have this nice-enough crossplatform-enough thread library integrated for you (>=C++11).
Just make sure to add -std=c++11 so that
gcc -o blink -l rt blink.c -l bcm2835
becomes
gcc -std=c++11 -o blink -l rt blink.c -l bcm2835
Docs and basic examples at http://www.cplusplus.com/reference/thread/thread/
Nicer looking docs at http://en.cppreference.com/w/cpp/thread/thread
You still have to program what to thread on your own though.