Custom Keyword written in python 2.7:
#keyword("Update ${filename} with ${properties}")
def set_multiple_test_properties(self, filename, properties):
for each in values.split(","):
each = each.replace(" ", "")
key, value = each.split("=")
self.set_test_properties(filename, key, value)
When we send paremeters in a single line as shown below, its working as expected:
"Update sample.txt with "test.update=11,timeout=20,delay.seconds=10,maxUntouchedTime=10"
But when we modify the above line with a new lines (for better readability) it's not working.
Update sample.txt with "test.update = 11,
timeout=20,
delay.seconds=10,
maxUntouchedTime=10"
Any clue on this please?
I am not very sure whether it will work or not, but please try like this
Update sample.txt with "test.update = 11,
... timeout=20,
... delay.seconds=10,
... maxUntouchedTime=10"
Your approach is not working, cause the 2nd line is considered a call to a keyword (called "timeout=20,"), the 3rd another one, and so on. The 3 dots don't work cause they are "cell separators" - delimiter b/n arguments.
If you are going for readability, you can use the Catenate kw (it's in the Strings library):
${props}= Catenate SEPARATOR=${SPACE}
... test.update = 11,
... timeout=20,
... delay.seconds=10,
... maxUntouchedTime=10
, and then call your keyword with that variable:
Update sample.txt with "${props}"
btw, a) I think your keyword declaration in the decorator is without the double quotes - i.e. called like that ^ they'll be treated as part of the argument's value, b) there seems to be an error in the py method - the argument's name is "properties" while the itterator uses "values", and c) you might want to consider using named varargs (**kwargs in python, ${kwargs} in RF syntax) for this purpose (sorry, offtopic, but couldn't resist :)
Related
I've got a Node module file containing about 100 exported methods, which looks something like this:
exports.methodOne = async user_id => {
// other method contents
};
exports.methodTwo = async user_id => {
// other method contents
fooMethod();
};
exports.methodThree = async user_id => {
// other method contents
fooMethod();
};
Goal: What I'd like to do is figure out how to grab the name of any method which contains a call to fooMethod, and return the correct method names: methodTwo and methodThree. I wrote a regex which gets kinda close:
exports\.(\w+).*(\n.*?){1,}fooMethod
Problem: using my example code from above, though, it would effectively match methodOne and methodThree because it finds the first instance of export and then the first instance of fooMethod and goes on from there. Here's a regex101 example.
I suspect I could make use of lookaheads or lookbehinds, but I have little experience with those parts of regex, so any guidance would be much appreciated!
Edit: Turns out regex is poorly-suited for this type of task. #ctcherry advised using a parser, and using that as a springboard, I was able to learn about Abstract Syntax Trees (ASTs) and the recast tool which lets you traverse the tree after using various tools (acorn and others) to parse your code into tree form.
With these tools in hand, I successfully built a script to parse and traverse my node app's files, and was able to find all methods containing fooMethod as intended.
Regex isn't the best tool to tackle all the parts of this problem, ideally we could rely on something higher level, a parser.
One way to do this is to let the javascript parse itself during load and execution. If your node module doesn't include anything that would execute on its own (or at least anything that would conflict with the below), you can put this at the bottom of your module, and then run the module with node mod.js.
console.log(Object.keys(exports).filter(fn => exports[fn].toString().includes("fooMethod(")));
(In the comments below it is revealed that the above isn't possible.)
Another option would be to use a library like https://github.com/acornjs/acorn (there are other options) to write some other javascript that parses your original target javascript, then you would have a tree structure you could use to perform your matching and eventually return the function names you are after. I'm not an expert in that library so unfortunately I don't have sample code for you.
This regex matches (only) the method names that contain a call to fooMethod();
(?<=exports\.)\w+(?=[^{]+\{[^}]+fooMethod\(\)[^}]+};)
See live demo.
Assuming that all methods have their body enclosed within { and }, I would make an approach to get to the final regex like this:
First, find a regex to get the individual methods. This can be done using this regex:
exports\.(\w+)(\s|.)*?\{(\s|.)*?\}
Next, we are interested in those methods that have fooMethod in them before they close. So, look for } or fooMethod.*}, in that order. So, let us name the group searching for fooMethod as FOO and the name of the method calling it as METH. When we iterate the matches, if group FOO is present in a match, we will use the corresponding METH group, else we will reject it.
exports\.(?<METH>\w+)(\s|.)*?\{(\s|.)*?(\}|(?<FOO>fooMethod)(\s|.)*?\})
Explanation:
exports\.(?<METH>\w+): Till the method name (you have already covered this)
(\s|.)*?\{(\s|.)*?: Some code before { and after, non-greedy so that the subsequent group is given preference
(\}|(?<FOO>fooMethod)(\s|.)*?\}): This has 2 parts:
\}: Match the method close delimiter, OR
(?<FOO>fooMethod)(\s|.)*?\}): The call to fooMethod followed by optional code and method close delimiter.
Here's a JavaScript code that demostrates this:
let p = /exports\.(?<METH>\w+)(\s|.)*?\{(\s|.)*?(\}|(?<FOO>fooMethod)(\s|.)*?\})/g
let input = `exports.methodOne = async user_id => {
// other method contents
};
exports.methodTwo = async user_id => {
// other method contents
fooMethod();
};
exports.methodThree = async user_id => {
// other method contents
fooMethod();
};';`
let match = p.exec( input );
while( match !== null) {
if( match.groups.FOO !== undefined ) console.log( match.groups.METH );
match = p.exec( input )
}
I am trying to find a way of parsing sequences of related arguments, preferably using argparse.
For example:
command --global-arg --subgroup1 --arg1 --arg2 --subgroup2 --arg1 --arg3 --subgroup3 --arg4 --subcommand1 --arg1 --arg3
where --global-arg applies to the whole command, but each --subgroupN argument has sub-arguments that apply only to it (and may have the same name, such as --arg1 and --arg3 above), and where some sub-arguments are optional, so the number of sub-arguments is not constant. However, I know that each --subgroupN sub-argument set is complete either by the presence of another --subgroupN or the end of the argument list (I am not fussed if global arguments cannot appear at the end, although I imagine that is possible as long as they don't clash with sub-argument names).
The --subgroupN elements are essentially sub-commands, but I do not appear to be able to use the sub-parser ability of argparse as it slurps any following --subgroupN entries as well (and therefore barfs with unexpected arguments).
(An example of this style of argument list is used by xmlstarlet)
Are there any suggestions beyond writing my own parser? I assume I can at least leverage something out of argparse if that is the only option...
Examples
The examples below were an attempt to find a way to parse an argument structure along the following lines:
(a --name <name>|b --name <name>)+
in the first example I hoped to have --a and --b introduce a set of arguments that were processed by a subparser.
I was hoping to get something out perhaps along the lines of
Namespace(a=Namespace(name="dummya"), b=Namespace(name="dummyb"))
subparser example fails
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers()
parser_a = subparsers.add_parser("a")
parser_b = subparsers.add_parser("b")
parser_a.add_argument("--name")
parser_b.add_argument("--name")
parser.parse_args(["a", "--name", "dummy"])
> Namespace(name='dummy') (Good)
parser.parse_args(["b", "--name", "dummyb", "a", "--name", "dummya"])
> error: unrecognized arguments: a (BAD)
mutually exclusive group fails
parser = argparse.ArgumentParser()
g = parser.add_mutually_exclusive_group()
g1 = g.add_mutually_exclusive_group()
g1.add_argument("--name")
g2 = g.add_mutually_exclusive_group()
g2.add_argument("--name")
> ArgumentError: argument --name: conflicting option string(s): --name (BAD)
(I wasn't really expecting this to work, it was an attempt to see if I could have repetition of grouped arguments.)
Other than the subparser mechanism, argparse is not designed to handle groups of arguments. Other than the nargs grouping, it handles the arguments in the order that they appear in the argv list.
As I mentioned in the comments there have been earlier questions, which can probably be found by search with words like multiple. But one way or other they seek to work about the basic order-independent design of argparse.
https://stackoverflow.com/search?q=user%3A901925+[argparse]+multiple
I think the most straight forward solution is to process the sys.argv list before hand, breaking it into groups, and then passing those sublists to one or more parsers.
parse [command --global-arg],
parse [--subgroup1 --arg1 --arg2],
parse [--subgroup2 --arg1 --arg3],
parse [--subgroup3 --arg4],
parse [--subcommand1 --arg1 --arg3]
In fact the only alternative is to use that subparser 'slurp everything else' behavior to get a remainder of arguments that can be parsed again. Use parse_known_args to return a list of unknown arguments (parse_args raises an error if that list is not empty).
Using hpaulj's reply above, I came up with the following:
args = [
"--a", "--name", "dummya",
"--b", "--name", "dummyb",
"--a", "--name", "another_a", "--opt"
]
parser_globals = argparse.ArgumentParser()
parser_globals.add_argument("--test")
parser_a = argparse.ArgumentParser()
parser_a.add_argument("--name")
parser_a.add_argument("--opt", action="store_true")
parser_b = argparse.ArgumentParser()
parser_b.add_argument("--name")
command_parsers = {
"--a": parser_a,
"--b": parser_b
}
the_namespace = argparse.Namespace()
if globals is not None:
(the_namespace, rest) = parser_globals.parse_known_args(args)
subcommand_dict = vars(the_namespace)
subcommand = []
val = rest.pop()
while val:
if val in command_parsers:
the_args = command_parsers[val].parse_args(subcommand)
if val in subcommand_dict:
if "list" is not type(subcommand_dict[val]):
subcommand_dict[val] = [subcommand_dict[val]]
subcommand_dict[val].append(the_args)
else:
subcommand_dict[val] = the_args
subcommand = []
else:
subcommand.insert(0, val)
val = None if not rest else rest.pop()
I end up with:
Namespace(
--a=[
Namespace(
name='another_a',
opt=True
),
Namespace(
name='dummya',
opt=False
)
],
--b=Namespace(
name='dummyb'
),
test=None
)
which seems to serve my purposes.
I have a list of countries that i need to convert into standardized format (iso3c). Some have long names, others have 2 or 3 digit codes, and others do not display the whole country name like "Africa" instead of "South Africa". Ive done some research and come up to use countrycode package in R. However, when i tried to use "regex" R doesnt seem to recognize it. Im getting the error below:
> countrycode(data,"regex","iso3c", warn = TRUE)
Error in countrycode(data, "regex", "iso3c", :
Origin code not supported
Any other option I need to do?
Thanks!
You can view the README for the countrycode package here https://github.com/vincentarelbundock/countrycode, or you can pull up the help file in R by entering this into your R console ?countrycode::countrycode.
"regex" is not a valid 'origin' value (2nd argument in the countrycode() function). You must use one of "cowc", "cown", "eurostat", "fao", "fips105", "imf", "ioc", "iso2c", "iso3c", "iso3n", "p4_ccode", "p4_scode", "un", "wb", "wb_api2c", "wb_api3c", "wvs", "country.name", "country.name.de" (using latest version 0.19).
If you use either of the following 'origin' values, regex matching will be performed automatically: "country.name" or "country.name.de"
If you're using a custom dictionary with the new (as of version 0.19) custom_dict argument, you must set the origin_regex argument to TRUE for regex matching to occur.
In your example, this should do what you want:
countrycode(data, origin = "country.name", destination = "iso3c", warn = TRUE)
I'm running into trouble trying to call a Python function from my C++ code.
I'm trying to call the Django function call_command, with the following parameters:
call_command('test', stdout=content)
See here why. This runs the command without the stdout=content argument:
PyObject_CallFunctionObjArgs(executeFunc, PyString_FromString("test"), NULL)
I can't, for the life of me, figure out how to create this keyword.
PyDict_New();
PyDict_SetItemString(...);
Nets me a 'stdout'='content' string, instead of stdout=content.
Py_BuildValue("{s=s}", ... , ...)
Does the same thing.
I can get the content PyObject no problem if necessary, I just can't seem to correctly create the stdout keyword argument without Python putting quotes (handling it as a string) at some point.
How can I call call_command('test', stdout=content) from the C API?
PyObject *args = Py_BuildValue("(s)", "test");
PyObject *kwargs = PyBuildValue("{s:O}", "stdout", content);
PyObject_Call(executeFunc, args, kwargs);
Note that even in Python, you can call the function the same way:
args = ("test",)
kwargs = {"stdout" : content}
call_command(*args,**kwargs)
"stdout" is a string with quotes here too, but it works. You are trying to get the same syntax as you use in Python, with noquotename=noquotevalue, but the syntax is just a representation. Both {"quotedname" : noquotevalue} and noquotename=noquotevalue are two different ways to represent the same thing. They both actually represent a dictionary whose key is a string and value is an object. That is what you pass as the third argument to PyObject_Call.
Py_BuildValue("{s:O}", "stdout", content_obj)
So I'm doing something wrong in this python script, but it's becoming convoluted and I'm losing sight of what I'm doing wrong.
I want a script to go through a file, find all the function definitions, and then pull out the name, return type, and parameters of the function, and output a "doxygen" style comment like this:
/******************************************************************************/
/*!
\brief
Main function for the file
\return
The exit code for the program
*/
/******************************************************************************/
But I'm doing something wrong with the regular expression in trying to parse the parameters... Here is the script so far:
import re
import sys
f = open(sys.argv[1])
functions = []
for line in f:
match = re.search(r'([\w]+)\s+([\S]+)\(([\w+\s+\w+])+\)',line)
if line.find("\\fn") < 0:
if match:
returntype = match.group(1)
funcname = match.group(2)
print '/********************************************************************'
print " \\fn " + match.group()
print ''
print ' \\brief'
print ' Function description for ' + funcname
print ''
if len(match.groups()) > 2:
params = []
count = len(match.groups()) - 2
while count > 0:
matchingstring = match.group(count + 2)
if matchingstring.find("void") < 0:
params.append(matchingstring)
count -= 1
for parameter in params:
print " \\param " + parameter
print ' Description of ' + parameter
print ''
print ' \\return'
print ' ' + returntype
print '********************************************************************/'
print ''
Any help would be appreciated. Thanks
The grammar of C++ is far too complex to be handled by simple
regular expressions. You'll need at least a minimal parser.
I've found that for restricted cases, where I'm not concerned
with C++ in general, but only my own style, I can often get away
with a flex based tokenizer and a simple state machine. This
will fail in many cases of legal C++—for starters, of
course, if someone uses the pre-processor to modify the syntax;
but also because < can have different meanings, depending on
what precedes it names a template or not. But it's often
adequate for a specific job.
I've used a PEG parser with great success when trying to do simple format parsing. pyPeg is a very simple implementation of such a parser written in Python.
Example Python code for C++ function parser:
EDIT: Address template parameters. Tested with input from SK-logic and output is correct.
import pyPEG
from pyPEG import parseLine
import re
def symbol(): return re.compile(r"[abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ&*][\w:]+")
def type(): return symbol
def functionName(): return symbol
def templatedType(): return symbol, "<", -1, [templatedType, symbol, ","], ">"
def parameter(): return [templatedType, type], symbol
def template(): return "<", -1, [symbol, template], ">"
def function(): return [type, templatedType], functionName, -1, template, "(", -1, [",", parameter], ")" # -1 -> zero or more repetitions.
sourceCode = "std::string foobar(std::vector<int> &A, std::map<std::string, std::vector<std::string> > &B)"
results = parseLine(sourceCode, function(), [], packrat=True)
When this is executed results is:
([(u'type', [(u'symbol', 'std::string')]), (u'functionName', [(u'symbol', 'foobar')]), (u'parameter', [(u'templatedType', [(u'symbol', 'std::vector'), (u'symbol', 'int')]), (u'symbol', '&A')]), (u'parameter', [(u'templatedType', [(u'symbol', 'std::map'), (u'symbol', 'std::string'), (u'templatedType', [(u'symbol', 'std::vector'), (u'symbol', 'std::string')])]), (u'symbol', '&B')])], '')
C++ cannot really be parsed by a (sane) regular expression: they are a nightmare as soon as nesting is concerned.
There is another concern too, determining when to parse and when not to. A function may be declared:
at file scope
in a namespace
in a class
And the two last can be nested at arbitrary depths.
I would propose to use CLang here. It's a real C++ front-end with a full-featured parser and there are:
a C API, with (notably) an API to the Indexing Library
Python bindings on top of the C API
The C API and Python bindings are far from fully exposing the underlying C++ model, but for a task as simple as listing functions it should be enough.
That said, I would question the usefulness of the project: if the documentation can be generated by a simple parser, then it is redundant with the code. And redundancy is at best, useless, and worst dangerous: it introduces the potential threat of desynchronization...
If the function is tricky enough that its use requires documentation, then a developer, who knows the limitations and al, has to write this documentation.