Multilocale Chapel: make check fails (GASNet: Invalid number of nodes: -nl) - chapel

I'm trying to run Chapel in multi-locale mode with Slurm.
However, 'make check' fails.
Could someone help me with that?
I used Chapel 1.23.0. Here are the actual commands I used:
cd chapel-1.23.0/
export CHPL_HOME=$PWD
source $CHPL_HOME/util/setchplenv.bash
export CHPL_COMM=gasnet
export CHPL_LAUNCHER=slurm-srun
export CHPL_TARGET_CPU=native
make && make check
Here is the error messages I got:
== Actual Test Output (raw, with verbose) ==
srun --job-name=CHPL-hello6-tas --quiet --nodes=4 --ntasks=4 --ntasks-per-node=1 --cpus-per-task=16 --exclusive --mem=0 --kill-on-bad-exit /home/user1/.chpl/chapel-test-P4CwK/hello6-taskpar-dist_real -nl4 --printLocaleName=false -v
GASNet: Invalid number of nodes: -nl4
GASNet: Usage '/home/user1/.chpl/chapel-test-P4CwK/hello6-taskpar-dist_real <num_nodes> {program arguments}'

Assuming you're using the udp substrate with gasnet ($CHPL_HOME/util/printchplenv shows CHPL_COMM_SUBSTRATE: udp) then slurm-srun doesn't work in that particular configuration. The udp substrate requires CHPL_LAUNCHER=amudprun. From https://chapel-lang.org/docs/platforms/udp.html#using-the-udp-conduit-with-slurm, you should be able to do:
export CHPL_LAUNCHER=amudprun
export GASNET_SPAWNFN=C
export GASNET_CSPAWN_CMD="srun -N%N %C"
Note that you'll have to redo the top-level make command.
This tells Chapel to use the amudprun launcher, and then lets amudprun know how to spawn onto this system (in this case using srun instead of defaulting to using ssh)

Related

Checks not running on every file (when running -a) even though they are parsed

Summary of the problem
I've setup a repository with the checks I developed, designed to be used exclusively in an another repository.
The problem I've been facing is that when I run pre-commit run -a -v the checks don't go through every file, and time to time they even change!
A little more details
I've run the [identity][1] check and it prints every file in the repo, meaning the files are read by pre-commit (per my understanding)
When I execute pre-commit run the file in staging are correctly parsed
Why do i think that the checks don't run on every file?
I always print something to the standard output for every check, and when I run it verbosely it only prints 5 to 7 lines, meaning it checked only those.
If I try and edit a file (breaking a check) whose not on the short list, the check still passes through
Some code
The structure of the folder containing the files to be checked
./
articles/
some_name/
file.md
file.png
and_so_on.jpeg
team/
some_other_name/
file.md
propic.jpg
Summary of the .pre-commit-hooks.yaml
- id: team-check-name
name: blabla
description: blabla
entry: team-checkname
files: 'team/.*/'
types: [markdown]
language: python
- id: article-check-name
name: blabla
description: blabla
entry: article-checkname
files: 'articles/.*/'
types: [markdown]
language: python
Essentially the checks supposed to run on articles/.*/ start with article- and they all have this property: files: 'articles/.*/'.
Similarly every check supposed to run on team/.*/ starts with team- and they all have files: 'team/.*/'
Summary of the .pre-commit-config.yaml
repos:
- repo: https://github.com/repourl
rev: commit_hash
hooks:
- id: team-check-name
- id: article-check-name
But as you can see, I don't override any settings so it shouldn't interfere
the hook tools you've written are incorrect -- they only process sys.argv[1] rather than positional arguments
your code uses a lot of global variables so it's not a straightforward refactor -- typically you'd either use argparse to collect nargs='*' or loop over sys.argv[1:] (if you don't have any options)
as to why this is the convention -- it's very wasteful to start a linter process over and over to lint a single file (often times executable startup cost dwarfs the actual linting / formatting process)
disclaimer: I wrote pre-commit

Mounting ecryptfs using C++ mount function

I am trying to mount ecryptfs from within a C++ program. I can definitely mount it without it asking questions by issuing this command at the prompt:
sudo mount -t ecryptfs -o "rw,key=passphrase:passphrase_passwd=geoff,ecryptfs_cipher=aes,ecryptfs_key_bytes=32,ecryptfs_passthrough=n,ecryptfs_enable_filename_crypto=n,no_sig_cache" ~/source/ ~/target/
Note that in reality, I am passing a full canonical path in case that matters.
But from within the program I get failure with errno=EINVAL after trying by using the mount() function with the same arguments:
mount("~/source/", "~/target/", "ecryptfs", MS_NODEV, "rw,key=passphrase:passphrase_passwd=geoff,ecryptfs_cipher=aes,ecryptfs_key_bytes=32,ecryptfs_passthrough=n,ecryptfs_enable_filename_crypto=n,no_sig_cache")
The program does launch with root privileges and I have checked that I have CAP_SYS_ADMIN.
The mount() function returns -1 and sets errno to EINVAL.
Have I got the arguments correct? Is this maybe a privileges issue?
EDIT: I got it to work by executing mount externally via system(), but would still like to use the function because of reasons.
I believe this is because mount -t ecryptfs is actually calling the helper executable mount.ecryptfs, and it's processing some of the options (in particular, key=) itself. What's actually passed to the kernel is different (you can see this by looking at /proc/mounts afterward).
If you look closely at https://manpages.ubuntu.com/manpages/kinetic/en/man7/ecryptfs.7.html, key= and ecryptfs_enable_filename_crypto= are listed under "MOUNT HELPER OPTIONS" - the actual kernel module's options are ecryptfs_sig=(fekek_sig) and ecryptfs_fnek_sig=(fnek_sig).
So, if you want to bypass the helper and do the mount directly, you'd need to load the tokens into the kernel's keyring with https://man7.org/linux/man-pages/man2/keyctl.2.html and replace key= with the resulting token signatures, like mount.ecryptfs did.
It does appear that there is a libecrytpfs with functions in ecryptfs.h like ecryptfs_add_passphrase_key_to_keyring which you can (presumably, not tested) use to do this in a way matching the mount.ecryptfs

Extracting User Story from Commit Message in Jenkins

I'm using the ext-email extension to extract the User Story which is added as a commit message to include in the mail body.
This is the console output:
Commit message: "US285568"
I used the Build Log Excerpt method of the ext-email plug-in as follows:
STORY: ${BUILD_LOG_EXCERPT, start="Commit message:\ \"", end="\'"}
However, this does not match anything and I'm not able to understand why it's failing.
I couldn't find a proper documentation for this plug-in.
I used a work-around solution by triggering a helper job at the end of the current job (which contains the commit message in the console output).
I am executing the following shell code in the helper job:
result=$(curl -GET {JENKINS_IP}/jenkins/job/{UPSTREAM_JOB_NAME}/consoleFull --user "user:pass")
comm=$(grep "Commit message:" <<< "$result")
if grep -E "US[0-9]+" <<< "$comm"
then
final=$(grep -o "US[0-9]*" <<< "$comm")
else
final="<font color=\"red\">User Story not found</font>"
fi
echo FINAL=$final > env.properties
To access the FINAL variable (which contains the value of the user story) as an environment variable, I have used the EnvInject plug-in.
To access the FINAL variable outside the shell code in the job, add the "Inject environment variables" build step after the shell code and enter "env.properties" in the Properties File Path.

Does go test run unit tests concurrently?

When go test is ran it runs your files ending in _test.go by running the functions that start in the format TestXxx and use the (*t testing.T) module. I was wondering if each function in the _test.go file were ran concurrently or if it definitively ran each function separately? Does it create a go routine for each one? If it does create a go routine for each one, can I monitor the go routines in some way? Is it ever possible to do something like golibrary.GoRoutines() and get an instance for each one and monitor them some how or something like that?
Note: this question assumes you using the testing framework that comes with go (testing).
Yes, tests are executed as goroutines and, thus, executed concurrently.
However, tests do not run in parallel by default as pointed out by #jacobsa. To enable parallel execution you would have to call t.Parallel() in your test case and set GOMAXPROCS appropriately or supply -parallel N (which is set to GOMAXPROCS by default).
The simplest solution for your case when running tests in parallel would be to have a global slice for port numbers and a global atomically incremented index that serves as association between test and port from the slice. That way you control the port numbers and have one port for each test. Example:
import "sync/atomic"
var ports [...]uint64 = {10, 5, 55}
var portIndex uint32
func nextPort() uint32 {
return atomic.AddUint32(&portIndex, 1)
}
Yes, as other answers already pointed, tests inside one _test.go file are running in parallel only if you add t.Parallel() and run with -parallel tag.
BUT - by default the command go test runs in parallel tests for different packages, and default is the number of CPUs available (see go help build)
So to disable parallel execution you should run tests like this
go test -p 1 ./...
Yes if you add this :
t.Parallel() to your functions.
like this:
func TestMyFunction(t *testing.T) {
t.Parallel()
//your test code here
}
There are two aspects of "parallel tests" in go.
Testing of packages (folders) in parallel. Controlled by -p flag.
go help build:
-p n
the number of programs, such as build commands or
test binaries, that can be run in parallel.
The default is GOMAXPROCS, normally the number of CPUs available.
Testing of tests within a package. Controlled by -parallel flag, and tests need this enabled with t.Parallel().
go help testflag:
-parallel n
Allow parallel execution of test functions that call t.Parallel.
The value of this flag is the maximum number of tests to run
simultaneously; by default, it is set to the value of GOMAXPROCS.
Note that -parallel only applies within a single test binary.
The 'go test' command may run tests for different packages
in parallel as well, according to the setting of the -p flag
(see 'go help build').
For #1, you can observe this directly by having tests in multiple packages, and forcing them to be long running with a sleep. Then ps aux and grep for go running and you'll see separate processes in parallel.
Or run with -x detail flag. That pretty clearly shows what’s going on. The tests are launched once per package (e.g. given have *_test.go files in several package. Note I’m running go test -x ./... -v -count=1 |& ruby -pe 'print Time.now.strftime("[%Y-%m-%d %H:%M:%S] ")' |& tee temp.out with various configurations of -p=N -parallel=M, you can see in parallel them running:
[2022-05-09 17:22:26] $WORK/b065/understandinit.test -test.paniconexit0 -test.timeout=10m0s -test.parallel=1 -test.v=true -test.count=1
[2022-05-09 17:22:26] $WORK/b001/main.test -test.paniconexit0 -test.timeout=10m0s -test.parallel=1 -test.v=true -test.count=1
[2022-05-09 17:22:26] $WORK/b062/mainsub.test -test.paniconexit0 -test.timeout=10m0s -test.parallel=1 -test.v=true -test.count=1
You can run them concurrently by flagging the test with t.Parallel
and then run the test using the -parallel flag.
You can see other testing flags here

vimrc to detect remote connection

At the moment I have to hard code the names of servers on my vimrc in order to either make it different on the remote machine. This is done by conditional statement using hostname() function in vim. I want to make the conditional to be based on the status of remote connection and not on the hostname. So...
The first possible solution I found was using the following bash command in system():
cat /proc/$PPID/status | head -1 | cut -f2
This does not work because I use GNU screen and this will not detect my connection status properly.
The second possible solution I am exploring right now is using who am i This reliably shows whether or not remote connection has been made from which client, but I have trouble getting it working with system()
if substitute(system('who am i'), "theclient", ????, "") == ""
...
How could I get ???? to extract my client name somehow??
Even if the second solution works, allowing me to use .vimrc for many different remote machines, it is still tied to one client. I want the conditional to work in all remote session, regardless of the client name. So I am wondering, is this possible?
The following line allows me to create a variable that detects the remote connection status:
let g:remoteSession = ($STY == "")
Now you can surround the lines that you want to be ignored in the remote connection via:
if g:remoteSession
...
endif
On a side note, I do not know how expensive it is look up the environment variable compared to the global variable, but I am guessing the difference is negligible. The system call in an environment like cygwin where fork() is inefficient, it is worth doing the optimization.
Instead of adding conditional logic to a shared ~/.vimrc, you could alternatively source system-local settings. I use the following:
" Source system-specific .vimrc first.
if filereadable(expand('~/local/.vimrc'))
source ~/local/.vimrc
endif
" Stop sourcing if inclusion guard exists.
if exists('g:loaded_vimrc')
finish
endif
" Common settings of .vimrc here...
I find this more scalable than trying to maintain an ever-changing list of hostnames in a central location.