I'd like to port the following command from Unix library to Jane Street's Core.Std.Unix library.
Unix.create_process exec args Unix.stdin Unix.stdout Unix.stderr
That is, I have an executable exec and arguments args and want to run the process using the same in/out/error channels as the current process.
I can get close with Core.Std.Unix.create_process ~exec:exec ~args:args, but can't figure out how to connect the stdin,stdout,stderr returned by this function from core with the file descriptors used by the current process.
You can dup2 the returned descriptors to your current descriptors, but I'm not sure that this would work.
open Core.Std
let redirect ( p : Unix.Process_info.t ) : unit =
let open Unix.Process_info in
List.iter ~f:(fun (src,dst) -> Unix.dup2 ~src ~dst) [
p.stdin, Unix.stdin;
p.stdout, Unix.stdout;
p.stderr, Unix.stderr
]
let exec prog args : unit =
let p = Unix.create_process ~prog ~args in
redirect p
But there is another solution, that maybe applicable. Consider using just Unix.system, it should work just out of box.
I'm not sure if I understand your question correctly, but if you just want to use the same channels as the current process, take a look at fork_exec in Core.Std.Unix:
val fork_exec : prog:string ->
args:string list ->
?use_path:bool ->
?env:string list ->
unit -> Core_kernel.Std.Pid.t
According to the docs,
fork_exec ~prog ~args ?use_path ?env () forks and execs prog with args in the child process, returning the child pid to the parent.
Related
I have a nextflow script with a channel for paired file inputs. I am trying to extract a substring from the file inputs to use as part of the shell call. I am trying to use Groovy's regex matching to extract the substring, but since it is based on an input value, I am having trouble executing the matching. An alternative would be to perform the regex in bash as part of the process shell call, but I am interested in figuring out how to manipulate inputs within a process anyways, as I feel it would be useful for other things too. How can I perform intermediate Groovy code with the process inputs prior to the shell call?
process alignment {
input:
val files
output:
stdout
def matcher = "${files[1][0]}" =~ /.+\/bcl2fastq_out\/([^\/]+)\/.+/
# this is the culprit, as if I hardcode the first string it works
def project = matcher.findAll()[0][1]
"""
echo ${project}
"""
}
workflow {
files = Channel
.fromFilePairs("${params.out_dir}/**{_R1,_R2}_00?.fastq.gz", checkIfExists:true, size: 2)
alignment(files)
}
when I execute this, I get the error
No such variable: files
an example input string would look like extractions/test/bcl2fastq_out/project1/example_L001_R1_001.fastq.gz where I'm trying to extract the project1 substring
As you've already discovered, you can declare variables in the script block, before the command string. For example:
process alignment {
input:
tuple val(sample), path(fastq_files)
output:
tuple val(sample), path(output_file)
script:
output_file = "${sample}.bam"
"""
>&2 echo "Aligning ${sample}..."
touch "${output_file}"
"""
}
Note that these are global (within the process scope) unless you define them using the def keyword. If you don't need these elsewhere in your process definition, like in your example, a local variable (using def) is usually preferable. If, however, you need to access these in your output declaration, for example, then they will need to be global.
Note that the fromFilePairs factory method emits a tuple, where the first element is a group key and the second element is a list of files. The problem with just using val to declare the inputs is that the files in the second element will not be localized to the working directory when your script is run. To fix this, you can just change your input definition to something like:
input:
tuple val(sample), path(fastq_files)
The problem with this approach, is that we're unable to extract the parent directory name from the localized filenames. So you will need to pass this in somehow. Usually, you could just get the parent name from the first file in the tuple, using:
params.input_dir = './path/to/files'
params.pattern = '**_R{1,2}_00?.fastq.gz'
process alignment {
debug true
input:
tuple val(sample), val(project), path(fastq_files)
"""
echo "${sample}: ${project}: ${fastq_files}"
"""
}
workflow {
Channel
.fromFilePairs( "${params.input_dir}/${params.pattern}" )
.map { sample, reads ->
def project = reads[0].parent.name
tuple( sample, project, reads )
}
.set { reads }
alignment( reads )
}
But since the glob pattern has an additional wildcard, i.e. _00?, you may not necessarily get the results you expect. For example:
$ mkdir -p path/to/files/project{1,2,3}
$ touch path/to/files/project1/sample1_R{1,2}_00{1,2,3,4}.fastq.gz
$ touch path/to/files/project2/sample2_R{1,2}_00{1,2,3,4}.fastq.gz
$ touch path/to/files/project3/sample3_R{1,2}_00{1,2,3,4}.fastq.gz
$ nextflow run main.nf
N E X T F L O W ~ version 22.04.0
Launching `main.nf` [determined_roentgen] DSL2 - revision: f80ab33ac8
executor > local (12)
[a8/9235cc] process > alignment (12) [100%] 12 of 12 ✔
sample2: project2: sample2_R1_001.fastq.gz sample2_R1_004.fastq.gz
sample1: project1: sample1_R1_003.fastq.gz sample1_R2_001.fastq.gz
sample1: project1: sample1_R1_004.fastq.gz sample1_R2_003.fastq.gz
sample3: project3: sample3_R1_001.fastq.gz sample3_R2_001.fastq.gz
sample1: project1: sample1_R1_001.fastq.gz sample1_R2_004.fastq.gz
sample1: project1: sample1_R1_002.fastq.gz sample1_R2_002.fastq.gz
sample2: project2: sample2_R1_002.fastq.gz sample2_R2_002.fastq.gz
sample2: project2: sample2_R2_001.fastq.gz sample2_R2_004.fastq.gz
sample2: project2: sample2_R1_003.fastq.gz sample2_R2_003.fastq.gz
sample3: project3: sample3_R2_002.fastq.gz sample3_R2_004.fastq.gz
sample3: project3: sample3_R1_003.fastq.gz sample3_R1_004.fastq.gz
sample3: project3: sample3_R1_002.fastq.gz sample3_R2_003.fastq.gz
Fortunately, we can supply a custom file pair grouping strategy using a closure. This uses the readPrefix helper function:
workflow {
Channel
.fromFilePairs( "${params.input_dir}/${params.pattern}" ) { file ->
prefix = Channel.readPrefix(file, params.pattern)
suffix = file.simpleName.tokenize('_').last()
"${file.parent.name}/${prefix}_${suffix}"
}
.map { key, reads ->
def (project, sample) = key.tokenize('/')
tuple( sample, project, reads )
}
.set { reads }
alignment( reads )
}
Results:
$ nextflow run main.nf
N E X T F L O W ~ version 22.04.0
Launching `main.nf` [loving_cantor] DSL2 - revision: 5a76ac712f
executor > local (12)
[f4/74edbc] process > alignment (12) [100%] 12 of 12 ✔
sample1_003: project1: sample1_R1_003.fastq.gz sample1_R2_003.fastq.gz
sample2_002: project2: sample2_R1_002.fastq.gz sample2_R2_002.fastq.gz
sample1_002: project1: sample1_R1_002.fastq.gz sample1_R2_002.fastq.gz
sample2_003: project2: sample2_R1_003.fastq.gz sample2_R2_003.fastq.gz
sample2_004: project2: sample2_R1_004.fastq.gz sample2_R2_004.fastq.gz
sample2_001: project2: sample2_R1_001.fastq.gz sample2_R2_001.fastq.gz
sample1_001: project1: sample1_R1_001.fastq.gz sample1_R2_001.fastq.gz
sample1_004: project1: sample1_R1_004.fastq.gz sample1_R2_004.fastq.gz
sample3_001: project3: sample3_R1_001.fastq.gz sample3_R2_001.fastq.gz
sample3_004: project3: sample3_R1_004.fastq.gz sample3_R2_004.fastq.gz
sample3_002: project3: sample3_R1_002.fastq.gz sample3_R2_002.fastq.gz
sample3_003: project3: sample3_R1_003.fastq.gz sample3_R2_003.fastq.gz
I figured it out, if instead of just jumping into the shell script with the triple quotes, you can start specifying the process execution script with "script:" then run Groovy using the process inputs
process alignment {
input:
val files
output:
stdout
script:
test = (files[1][0] =~ '.+/test/([^/]+)/.+').findAll()[0][1]
"""
echo $test
"""
I am currently creating a CLI application in OCaml and using Core.Command, the CLI parser included in core (v0.10), to parse the command line.
I want to have a global flag that can be used for any subcommand (like the --paginate or --git-dir flags in git for example).
For instance, I want a -debug flag so that the two following commands are valid
my-cli -debug hello world
my-cli -debug goodbye world
However, I could not find a way to do this with the Core.Command API.
Here is a simplified version what I currently have.
open Core
let initialize_logger debug =
Logs.set_reporter (Logs_fmt.reporter ());
let log_level = if debug then Logs.Debug else Logs.Info in
Logs.set_level (Some log_level)
let some_func_with_logging () =
Logs.debug (fun m -> m "the flag debug was passed!")
let hello name =
some_func_with_logging ();
Printf.printf "Hello %s!\n" name
let goodbye name =
some_func_with_logging ();
Printf.printf "Goodbye %s!\n" name
let hello_command =
let open Command.Let_syntax in
Command.basic
~summary:"says hello"
[%map_open
let name = anon ("name" %: string)
and debug = flag "debug" no_arg ~doc:"debug" in
fun () ->
initialize_logger debug;
hello name
]
let goodbye_command =
let open Command.Let_syntax in
Command.basic
~summary:"says goodbye"
[%map_open
let name = anon ("name" %: string)
and debug = flag "debug" no_arg ~doc:"debug" in
fun () ->
initialize_logger debug;
goodbye name
]
let main_command =
Command.group ~summary:"a cool CLI tool"
[ ("hello", hello_command);
("goodbye", goodbye_command);
]
let () = Command.run main_command
There are two main issues here:
the debug flag as well as the call to initialize_logger is duplicated in every subcommand
the debug flag needs to be passed after the subcommand when invoking the command: my-cli hello world -debug instead of my-cli -debug hello world
Is there a clean way to handle this with Core.Command API?
get_ue_supported_srvcc([]) ->
?SRVCC_3GPP_NONE_SUPPORT;
get_ue_supported_srvcc([#sip_contactV{extensionsP = EP} | T]) ->
case b2bLib:support_tags_to_value(EP) of
?SRVCC_3GPP_NONE_SUPPORT ->
get_ue_supported_srvcc(T);
Flag ->
Flag
end.
I want create a unit test for this function,
Here is my unit test case:
get_ue_supported_srvcc_test() ->
Contact =
[#sip_contactV{extensionsP =
[{"+sip.instance",
{quoted_string,"<urn:gsma:imei:35502406-005233-0>"}},
{"+g.3gpp.icsi-ref",
{quoted_string,"urn%3Aurn-7%3A3gpp-service.ims.icsi.mmtel"}},
"+g.3gpp.mid-call",
"+g.3gpp.srvcc-alerting",
"+g.3gpp.ps2cs-srvcc-orig-pre-alerting",
"video"]}],
?assertEqual(7, b2bAtcfLib:get_ue_supported_srvcc(Contact)).
But when I run it, I get this error:
======================== EUnit ========================
module 'b2bAtcfLib'
b2bAtcfLib_tests: get_ue_supported_srvcc_test (module 'b2bAtcfLib_tests')...*failed*
in function b2bLib:support_tags_to_value/1
called as support_tags_to_value([{"+sip.instance",{quoted_string,"<urn:gsma:imei:35502406-005233-0>"}},
{"+g.3gpp.icsi-ref",
{quoted_string,"urn%3Aurn-7%3A3gpp-service.ims.icsi.mmtel"}},
"+g.3gpp.mid-call","+g.3gpp.srvcc-alerting",
"+g.3gpp.ps2cs-srvcc-orig-pre-alerting","video"])
in call from b2bAtcfLib:get_ue_supported_srvcc/1 (src/b2bAtcfLib.erl, line 1735)
in call from b2bAtcfLib_tests:'-get_ue_supported_srvcc_test/0-fun-0-'/1 (test/unit/b2bAtcfLib_tests.erl, line 49)
in call from b2bAtcfLib_tests:get_ue_supported_srvcc_test/0
**error:undef
output:<<"">>
[done in 0.008 s]
=======================================================
The error means b2bLib:support_tags_to_value/1 is undef.
The define for this function b2bLib:support_tags_to_value:
support_tags_to_value(FieldStr) ->
lists:sum([Val || {Tag, Val} <- ?TAGLIST, lists:member(Tag, FieldStr)]).
The error is:
**error:undef
That means that the test is calling a function that's not defined. Either the module couldn't be found, or the module in question doesn't define a function with that name and arity.
The whole error message is a bit confusing. Now that we know that we got a "function undefined" error, we should be looking at this line:
in function b2bLib:support_tags_to_value/1
Even though it says that the error occurred "in" this function, this is the function that's undefined.
So either the test is run in such a way that it doesn't find the b2bLib module, or that module doesn't define a function called support_tags_to_value taking one argument. If it's the former, add -pa path/to/ebin to the Erlang command line in order to add the right directory to the code path.
I wish to use OCaml to access the Yahoo Finance API. Essentially, it will be just a bunch of HTTP requests to get quotes from Yahoo Finance.
Which module I should use?
I wish to have async HTTP requests.
There are possibilities using lwt:
ocsigen has a quite complete and a bit complex implementation
cohttp is a bit simpler but lacks some usefull parts
using opam to install:
$ opam install ocsigenserver cohttp
For instance in a toplevel:
try Topdirs.dir_directory (Sys.getenv "OCAML_TOPLEVEL_PATH") with _ -> ();;
#use "topfind";;
#thread;;
#require "ocsigenserver";;
open Lwt
(* a simple function to access the content of the response *)
let content = function
| { Ocsigen_http_frame.frame_content = Some v } ->
Ocsigen_stream.string_of_stream 100000 (Ocsigen_stream.get v)
| _ -> return ""
(* launch both requests in parallel *)
let t = Lwt_list.map_p Ocsigen_http_client.get_url
[ "http://ocsigen.org/";
"http://stackoverflow.com/" ]
(* maps the result through the content function *)
let t2 = t >>= Lwt_list.map_p content
(* launch the event loop *)
let result = Lwt_main.run t2
and using cohttp:
try Topdirs.dir_directory (Sys.getenv "OCAML_TOPLEVEL_PATH") with _ -> ();;
#use "topfind";;
#require "cohttp.lwt";;
open Lwt
(* a simple function to access the content of the response *)
let content = function
| Some (_, body) -> Cohttp_lwt_unix.Body.string_of_body body
| _ -> return ""
(* launch both requests in parallel *)
let t = Lwt_list.map_p Cohttp_lwt_unix.Client.get
(List.map Uri.of_string
[ "http://example.org/";
"http://example2.org/" ])
(* maps the result through the content function *)
let t2 = t >>= Lwt_list.map_p content
(* launch the event loop *)
let v = Lwt_main.run t2
Notice that an implementation of cohttp for jane street async library is also available
Just for the record, there is also ocurl with curl multi API support.
When I run my perl script (version: 5.6.1), I got below error:
Can't find namespace for method(16:method)
my code was:
my $ws_url = '$url';
my $ws_uri = '$uri';
my $ws_xmlns = '$xmlns';
my $soap = SOAP::Lite
-> uri($ws_uri)
-> on_action( sub { join '/s/', $ws_uri, $_[1] } )
-> proxy($ws_url);
my $method = SOAP::Data->name('Add')
->attr({xmlns => $ws_xmlns});
my #params = ( SOAP::Data->name(addParam => $myParam));
$response = $soap->call($method => #params);
then I read documentation in link:
http://docs.activestate.com/activeperl/5.8/lib/SOAP/Lite.html, which says:
Be warned, that though you have more control using this method, you should
specify namespace attribute for method explicitely, even if you made uri()
call earlier. So, if you have to have namespace on method element, instead of:
print SOAP::Lite
-> new(....)
-> uri('mynamespace') # will be ignored
-> call(SOAP::Data->name('method') => #parameters)
-> result;
do
print SOAP::Lite
-> new(....)
-> call(SOAP::Data->name('method')->attr({xmlns => 'mynamespace'})
=> #parameters)
-> result;
……….
……….
Moreover, it'll become fatal error if you try to call it with prefixed name:
print SOAP::Lite
-> new(....)
-> uri('mynamespace') # will be ignored
-> call(SOAP::Data->name('a:method') => #parameters)
-> result;
gives you:
Can't find namespace for method (a:method)
because nothing is associated with prefix 'a'.
So, I tried change my code to:
my $soap = SOAP::Lite
-> on_action( sub { join '/s/', $ws_uri, $_[1] } )
-> proxy($ws_url);
my #params = ( SOAP::Data->name(addParam => $myParam));
my $response = $soap->call(SOAP::Data->name('Add')->attr({xmlns => $ws_xmlns}) => #params)
->result;
and it still did not work..
any advice?
thanks ahead!
SOAP::Lite uses Scalar::Util. Scalar::Util has XS (ie, compiled C, non-pure-Perl) code in it.
The Perl version you are running with is 5.6.1.
The documentation link you provided points to an ActiveState library for Perl version 5.8.0. I'm going to assume that the version of SOAP::Lite you installed was compiled for use with 5.8.0, since that's the documentation version you cited.
Perl version 5.8.0 is not binary compatible with Perl 5.6.1. Modules compiled for 5.6.1 that contain XS will not run under 5.8.0. Modules compiled for 5.8.0 that contain XS code will not run under 5.6.1. In your case, it's not the module SOAP::Lite that contains the XS code, but one of its dependencies: Scalar::Util.
When you installed SOAP::Lite from the ActiveState repository for 5.8.0, PPM updated all of the module's dependencies, including Scalar::Util. In so doing, it installed a version of Scalar::Util that is not binary compatible with Perl 5.6.1.
The error you're experiencing is sufficiently wonky to support this theory, in the absence of a better one. Seems like the easiest way out of the mess would be to upgrade Perl, as well as your installed modules, and hope it doesn't break something else. ;)