I need to run a heavy, parallelised code written in SAS studio. The code is a flow saved as .cpf file.
Say that it is structured as:
F1 -> F2 -> …->FN, with N subflows.
A macro variable in F2 is defined as:
%let MV = A1;
I need to launch this flow some 100 times by changing MV to A1, A2, ...,A100.
I was wondering whether there is a way to prepare a batch file that allows to change iteratively the value of the parameter MV and launch the execution of the entire flow.
PS. Loops are not an option due to memory issues
Thanks for any hints
Related
I have a .lua file as follows:
timeout = 3000
index = 15
function Test()
A(index, timeout)
B()
end
Test()
A and B fuctions are implemented in the c++. It will be excuted with a 'luaL_dofile(L, "test.lua");' in c++.But the timeout and the index will change at different times.
The question is how to modify the params in real time?
I'm going to write two c++ programs.First one is to sent .lua string to the sencond one. The second c++ program implemets the A and B and will dofile the lua script. But the timeout and the index will changes very often. How to do that? My solution is to parse the index and timeout string ,then write the current value to the file in the first c++ program.Any better solution?
Instead of modifying a lua script over and over to call A with different arguments, you should probably just list all arugments in a single script.
local listOfIndices = {1,5,23,124,25,}
local timeout = 3000
for _,index in ipairs(listOfIndices) do
A(index, timeout)
B()
end
Otherwise having 10000 different indices will result in 10000 file write and read operations.
If you're on Windows you might want to give this a read https://learn.microsoft.com/de-de/windows/win32/ipc/interprocess-communications?redirectedfrom=MSDN
I can think of better ways to have two programs communicate, than sending Lua scripts through files.
Also I'm not sure why you need two applications here, why not add whatever applicaton 2 does to application 1 as a library?
BACKGROUND
We have testers for our embedded GUI product and when a tester declares "test failed", sometimes it's hard for us developers to reproduce the exact problem because we don't have the exact trace of what happened.
We do currently have a logging framework but us devs have to manually input those logging statements in the code which is fine . . . except when a hard-to-reproduce bug occurs and we didn't have a log statement at the 'right' location and then when we re-build, re-run the test with the same steps, we get a different result.
ISSUE
We would like a solution wherein the compiler produces extra instrumentation code that allows us to see the exact sequence of events including, at the minimum:
function enter/exit (already provided by -finstrument-functions)
control-flow statement enter i.e. entering if/else, which case statement we jumped to
The log would look like this:
int main() entered
if-line 5 entered
else-line 10 entered
void EventLoop() entered
. . .
Some additional nice-to-haves are
Parameter values on function entry AND exit (for pass-by-reference types)
Function return value
QUESTION
Are there any gcc tools or even paid tools that can do this instrumentation automatically?
You can either use gdb for that, and you can automate that (I've got a tool for that in the works, you find it here or you can try to use gcov.
The gcov implementation is so, that it loads the latest gcov data when you start the program. But: you can manually dump and load the data. If you call __gcov_flush it will dump the current data and reset the current state. However: if you do that several times it will always merge the data, so you'd also need to rename the gcov data file then.
I just want to know that how to retrieve all the record from universe database table using universe basic subroutine.I am new in universe.
Perhaps something like this in the unibasic
OPEN "filename" to FIL ELSE STOP 201,"cannot open filename"
EXECUTE "SELECT filename"
LOOP WHILE READNEXT ID
READ REC FROM FIL,ID ELSE REC = ""
* you now have the entire row in REC
REPEAT
Can you provide more information on what you are trying to do?
Having a subroutine call return the entire contents of a UniVerse file could return a large amount of data. I would expect you would be better off only returning a subset of the items so the calling routine can process a bit at a time.
New content based on comment:
Ok, since you mentioned a type 19 file, I assume you want to read one file from the directory/folder the file points to.
In your subroutine, you can use OPEN on the type 19 file, and use the READ command to read the file. ( Note that you can also use READU, READL, MATREAD, MATREADU, or MATREADL to get the entire file in the directory/folder, depending on if/how you want to lock the item and if you want the data in a dynamic or dimensioned array. If you only need a specific attribute you can then use READV, READVL or READVU commands.
Or, since this is a type 19 file, you can use sequential reads. Open the file with OPENSEQ and read with the READSEQ or READBLK command.
There is an article and sample code on GitHub on how to execute U2 UniVerse Subroutine.
Execute Rocket MV U2 Subroutine Asynchronously using C# (async\await) and U2 Toolkit for .NET. Convert Subroutine Multi-Value Output to Json/Objects/DataTable
These sample code are based on C# (async\await), but you can use for synchronous programming as well with little code tweak.
For article:
Go to this link :
https://github.com/RocketSoftware/multivalue-lab/tree/master/U2/Demos/U2-Toolkit/AsyncAwait/Execute_Subroutine_Async
Read ‘Subroutine-Async.docx’ file.
Sample Code for this article on GitHub
Go to this link :
https://github.com/RocketSoftware/multivalue-lab/tree/master/U2/Demos/U2-Toolkit/AsyncAwait/Execute_Subroutine_Async
OPEN '',FILENAME TO F.FILE ELSE STOP
SELECT F.FILE
LOOP
READNEXT K.FILE ELSE EXIT
READ R.FILE FROM F.FILE, K.FILE ELSE NULL
PRINT R.FILE
REPEAT
PRINT "All over Red Rover"
Filename should be in quotes, i.e "MYFILE" or 'MYFILE'
The loop will repeat till all records have been read and will then exit.
The go test command has support for the -c flag, described as follows:
-c Compile the test binary to pkg.test but do not run it.
(Where pkg is the last element of the package's import path.)
As far as I understand, generating a binary like this is the way to run it interactively using GDB. However, since the test binary is created by combining the source and test files temporarily in some /tmp/ directory, this is what happens when I run list in gdb:
Loading Go Runtime support.
(gdb) list
42 github.com/<username>/<project>/_test/_testmain.go: No such file or directory.
This means I cannot happily inspect the Go source code in GDB like I'm used to. I know it is possible to force the temporary directory to stay by passing the -work flag to the go test command, but then it is still a huge hassle since the binary is not created in that directory and such. I was wondering if anyone found a clean solution to this problem.
Go 1.5 has been released, and there is still no officially sanctioned Go debugger. I haven't had much success using GDB for effectively debugging Go programs or test binaries. However, I have had success using Delve, a non-official debugger that is still undergoing development: https://github.com/derekparker/delve
To run your test code in the debugger, simply install delve:
go get -u github.com/derekparker/delve/cmd/dlv
... and then start the tests in the debugger from within your workspace:
dlv test
From the debugger prompt, you can single-step, set breakpoints, etc.
Give it a whirl!
Unfortunately, this appears to be a known issue that's not going to be fixed. See this discussion:
https://groups.google.com/forum/#!topic/golang-nuts/nIA09gp3eNU
I've seen two solutions to this problem.
1) create a .gdbinit file with a set substitute-path command to
redirect gdb to the actual location of the source. This file could be
generated by the go tool but you'd risk overwriting someone's custom
.gdbinit file and would tie the go tool to gdb which seems like a bad
idea.
2) Replace the source file paths in the executable (which are pointing
to /tmp/...) with the location they reside on disk. This is
straightforward if the real path is shorter then the /tmp/... path.
This would likely require additional support from the compiler /
linker to make this solution more generic.
It spawned this issue on the Go Google Code issue tracker, to which the decision ended up being:
https://code.google.com/p/go/issues/detail?id=2881
This is annoying, but it is the least of many annoying possibilities.
As a rule, the go tool should not be scribbling in the source
directories, which might not even be writable, and it shouldn't be
leaving files elsewhere after it exits. There is next to nothing
interesting in _testmain.go. People testing with gdb can break on
testing.Main instead.
Russ
Status: Unfortunate
So, in short, it sucks, and while you can work around it and GDB a test executable, the development team is unlikely to make it as easy as it could be for you.
I'm still new to the golang game but for what it's worth basic debugging seems to work.
The list command you're trying to work can be used so long as you're already at a breakpoint somewhere in your code. For example:
(gdb) b aws.go:54
Breakpoint 1 at 0x61841: file /Users/mat/gocode/src/github.com/stellar/deliverator/aws/aws.go, line 54.
(gdb) r
Starting program: /Users/mat/gocode/src/github.com/stellar/deliverator/aws/aws.test
[snip: some arnings about BinaryCache]
Breakpoint 1, github.com/stellar/deliverator/aws.imageIsNewer (latest=0xc2081fe2d0, ami=0xc2081fe3c0, ~r2=false)
at /Users/mat/gocode/src/github.com/stellar/deliverator/aws/aws.go:54
54 layout := "2006-01-02T15:04:05.000Z"
(gdb) list
49 func imageIsNewer(latest *ec2.Image, ami *ec2.Image) bool {
50 if latest == nil {
51 return true
52 }
53
54 layout := "2006-01-02T15:04:05.000Z"
55
56 amiCreationTime, amiErr := time.Parse(layout, *ami.CreationDate)
57 if amiErr != nil {
58 panic(amiErr)
This is just after running the following in the aws subdir of my project:
go test -c
gdb aws.test
As an additional caveat, it does seem very selective about where breakpoints can be placed. Seems like it has to be an expression but that conclusion is only via experimentation.
If you're willing to use tools besides GDB, check out godebug. To use it, first install with:
go get github.com/mailgun/godebug
Next, insert a breakpoint somewhere by adding the following statement to your code:
_ = "breakpoint"
Now run your tests with the godebug test command.
godebug test
It supports many of the parameters from the go test command.
-test.bench string
regular expression per path component to select benchmarks to run
-test.benchmem
print memory allocations for benchmarks
-test.benchtime duration
approximate run time for each benchmark (default 1s)
-test.blockprofile string
write a goroutine blocking profile to the named file after execution
-test.blockprofilerate int
if >= 0, calls runtime.SetBlockProfileRate() (default 1)
-test.count n
run tests and benchmarks n times (default 1)
-test.coverprofile string
write a coverage profile to the named file after execution
-test.cpu string
comma-separated list of number of CPUs to use for each test
-test.cpuprofile string
write a cpu profile to the named file during execution
-test.memprofile string
write a memory profile to the named file after execution
-test.memprofilerate int
if >=0, sets runtime.MemProfileRate
-test.outputdir string
directory in which to write profiles
-test.parallel int
maximum test parallelism (default 4)
-test.run string
regular expression to select tests and examples to run
-test.short
run smaller test suite to save time
-test.timeout duration
if positive, sets an aggregate time limit for all tests
-test.trace string
write an execution trace to the named file after execution
-test.v
verbose: print additional output
I'm an amateur C++ programmer trying to learn about basic shell scripting. I have a complex C++ program that currently reads in different parameter values from Parameters.h and then executes one or more simulations with each parameter value sequentially. These simulations take a long time to run. Since I have a cluster available, I'd like to effectively parallelize this job, running the simulations for each parameter value on a separate processor. I'm assuming it's easier to learn shell scripting techniques for this purpose than OpenMPI. My cluster runs on the LSF platform.
How can I write my input parameters in Bash so that they are distributed among multiple processors, each executing the program with that value? I'd like to avoid interactive submission. Ideally, I'd have the inputs in a text file that Bash reads, and I'd be passing two parameters to each job: an actual parameter value and a parameter ID.
Thanks in advance for any leads and suggestions.
my solution
GNU Parallel does look slick, but I ended up (with the help of an IT admin) writing a simple bash script that echos to screen three inputs (a treatment identifier, treatment/parameter value, and a simulation identifier):
#!/bin/bash
j=1
for treatment in cat treatments.txt; do
for experiment in cat simulations.txt; do
bsub -oo tr_${j}_sim_${experiment}_screen -eo tr_${j}_sim_${experiment}_err -q short_serial "echo \"$j $treatment $experiment\" | ./a.out"
done
let j=$j+1
done
The file treatments.txt contains a list of the values I'd like to vary, simulations.txt contains a list of all the simulation identifiers I'd like to run (currently just 1,...,s, where s is the total number of simulations I want for each treatment), and the treatments are indexed 1...j.
Maybe check out: http://www.gnu.org/software/parallel/
edit:
Or, check out the -P argument to xargs, example:
time echo {1..5} | xargs -n 1 -P 5 sleep
Say you want to run the program simulate with inputs foo, bar, baz and quux in parallel, then the simplest way is:
inputs="foo bar baz quux"
# Launch processes in the background with &
children=""
for x in $inputs; do
simulate "$x" > "$x.output" &
$children = "$children $!"
done
# Wait for each to finish
for $pid in $children; do
wait $pid
done
for x in $inputs; do
echo "simulate '$x' gave:"
cat "$x.output"
rm -f "$x.output"
done
The problem is that all simulations are launched at the same time, so if your number of inputs is much larger than your number of CPUs/cores, they may swamp the system.
My best stab at this is you background multiple instances of your program and let the OS's scheduler take over to put them on different processors. AFAIK there is no way in any shell to specify which processor a given process should run on.
Something to the effect of:
#!/bin/sh
for arg in foo bar baz; do
./your_program "$arg" &
done