Puppet advanced default params - if-statement

Basically I have a puppet class cassandra_manager::params. I have a ton of params set like so:
# etc...
$cluster_name = undef,
$num_tokens = 256,
$initial_token = undef,
$hinted_handoff_enabled = true,
# etc...
Now most of these ones set undef, I just handle in my template just commenting it out if it's undef.
But there are a few that I need to use some previous params, and facts to figure out the defaults. Here is one main example:
$memtable_flush_writers = undef,
I then try to set it later like this:
$min_from_cores_and_disks = min([ $::processorcount, size(split($::hard_drives, ',')) ])
if !$memtable_flush_writers {
if 0 + (0 + $min_from_cores_and_disks[0]) > 8 or (0 + $min_from_cores_and_disks[0]) < 2 {
if (0 + $min_from_cores_and_disks[0]) < 2 {
$memtable_flush_writers = 2
}
elsif (0 + $min_from_cores_and_disks[0]) > 8 {
$memtable_flush_writers = 8
}
} else {
$memtable_flush_writers = (0 + $min_from_cores_and_disks[0])
}
}
Then puppet tells me I can't set $memtable_flush_writers because all variables are immutable in puppet.
So then I changed checking for whether the variable was "false" and didn't set the variable up above, but that just told me the obvious, that $memtable_flush_writers wasn't set.
What's the best way that I can still get the functionality from the if statements above without having these errors pop up. Thanks in advance.

You have to create a new variable which you use only for declaration and setting any default value in the class definition.
class cassandra_manager::params (
$memtable_flush_writers_count = undef
){
if !$memtable_flush_writers_count {
# Whatever you want to do
$memtable_flush_writers = 'your_value'
} else {
$memtable_flush_writers = $memtable_flus_writers_count
}
}

The variables in cassandra_manager::params are typically used as default values for parameters of class cassandra_manager.
class cassandra_manager(
$cluster_name = $cassandra_manager::params::cluster_name,
$num_tokens = $cassandra_manager::params::num_tokens,
...) inherits cassandra_manager::params {
}
You override the defaults by passing such parameters.
class {
'cassandra_manager':
memtable_flush_writers => <value for this node>
}

Related

Scala function takes 2 hours with 2 million values

Would be grateful if any ideas to speed it up!
case class Pair(aa:String, bb:String)
case class OutputRow(bb:String, aa:String, bb_2:String, aa_2:String)
def startSearch(
_1_sorted: Array[Pair] ,
_2_hashmap: HashMap[String, String] ) : ArrayBuffer[OutputRow] = {
var outputTableListBuffer = ArrayBuffer[OutputRow]()
var searchComparisionFlag = false
var storeMain = Pair("0","0") //Initialize with Dummy data
var i = 0
def search(xxxx_1: Pair): Unit = {
if (searchComparisionFlag==true) {
var _2_exists = _2_hashmap.exists(_._1 == xxxx_1.aa)
if (_2_exists) {
val _2_xxxx = _2_hashmap(xxxx_1.aa)
outputTableListBuffer.append(OutputRow(storeMain.aa, storeMain.bb,_2_xxxx, xxxx_1.aa))
i = i + 1
if (i % 1000 == 0) println("In recursive search storeMain: ", storeMain)
var storePair = Pair(_2_xxxx,xxxx_1.aa)
search(storePair)
} else {
searchComparisionFlag = false
return
}
} else {
var _2_exists = _2_hashmap.exists(_._1 == xxxx_1.aa)
if (_2_exists) {
val _2_xxxx = _2_hashmap(xxxx_1.aa)
searchComparisionFlag = true
outputTableListBuffer.append(OutputRow(xxxx_1.aa, xxxx_1.bb,_2_xxxx, xxxx_1.aa))
var store = Pair(_2_xxxx,xxxx_1.aa)
search(store)
}
}
}
_1_sorted.foreach{ aa_1 =>
val store = Pair(aa_1.aa, aa_1.bb)
storeMain = store
search(store)
}
outputTableListBuffer
}
The above function takes 2 hours with 1 million values in _1_sorted and with a good 1 Million lookup in the hashmap.
Any ideas to speed this up?
This is a recursive logic function
The biggest problem is this:
_2_hashmap.exists(_._1 == xxxx_1.aa)
This is checking every single element of the hashmap on every call. Instead, use get:
_2_hashmap.get(xxxx_1.aa) match {
Some(_2_xxxx) => // Found
???
None => // Not found
???
}
Other code issues:
Don't use return
Pass flags down through recursive call rather than using global var
Use val wherever possible
Don't start variable names with _

Parsing custom structured file with PyParsing

I'm need of help parsing a custom file structured file. As you can see below is the structure. The issue is that I can't seem to parse the structure correctly, namely myOriginalFormula & myBonusType in the same group when I want them seperated for example.
AttributeDictionary SomeDictName
{
myAttributeDefinitionCategories
{
AttributeDefinitionList SomeList
{
AttributeDefinition SomeDefinitioName < uid=8972789HHDUAI7 >
{
myOriginalFormula "(A_Variable) * 10"
myBonusType FlatValue
}
AttributeDefinition UIBlankAttribute < uid=JIAIODJ7899 >
{
}
AttributeDefinition SomeOtherDefinitionName < uid=17837HHJAJ7788 >
{
myOriginalFormula 1
mySpecializations
{
Specialization "Some_Specialization 1"
{
myCondition "Something.MustEqual = 1"
myFormula 0
}
Specialization "SomeSpecialization 2"
{
myCondition "Something.MustEqual = 2"
myFormula 0.026
}
}
myBonusType FlatValue
}
AttributeDefinition SomeReal_Other_definition < uid=6768GYAG//() >
{
myOriginalFormula "(Some_Formula / Other_Variable) - 1"
myBonusType Percentage
myUINumDecimals 1
myHasAddSignUI FALSE
}
}
}
}
My try is blow. Could someone help me parse this structure correctly?
def syntaxParser():
# constants
left_bracket = Literal("{").suppress()
right_bracket = Literal("}").suppress()
semicolon = Literal(";").suppress()
space = White().suppress()
key = Word(alphanums + '_/"')
value = CharsNotIn("{};,")
# rules
assignment = Group(key + Optional(space + value))
block = Forward()
specialblock = Forward()
subblock = ZeroOrMore(Group(assignment) | specialblock)
specialblock = (
Keyword("Specialization")
+ key
+ left_bracket
+ subblock
+ right_bracket)
block << Group(
Optional(Group(key + Optional(space + value)))
+ left_bracket
+ Group(ZeroOrMore(assignment | Group(block) | Group(specialblock)))
+ right_bracket)
script = block
return script
Your definition of value is too greedy:
value = CharsNotIn("{};,")
I don't know how this works with the format you are parsing, but I get better results with this:
value = quotedString | CharsNotIn("{};,\n")

How to validate Lua table keys from C++

I am trying to use Lua for the configuration of a C++ application and am having trouble generating helpful messages when something is wrong in the configuration, not the the Lua syntax.
For example, suppose the following is a valid configuration:
foo = { a = 0, b = 'bar' }
but the user actually typed this:
foo = { a = 0, c = 'bar' }
Now, the app knows that foo can have fields a and b. It can load foo and get the value of a. It can even tell that b is not set and use a default. But I want to detect that c is present and report a warning.
Here is an extract of my attempt at that which blows up:
static void check_table(lua_State* L)
{
lua_pushnil(L);
while ( lua_next(L, -2) )
{
// key at -2 and value at -1
if ( lua_isstring(L, -2) )
{
const char* key = lua_tostring(L, -2);
// validate here; just printing key for now
cout << key << endl;
}
lua_pop(L, 1);
}
}
This works fine as long as the table is not actually an array. When I hit one of those, it dies on the second iteration with this:
...
1
PANIC: unprotected error in call to Lua API (invalid key to 'next')
which I attribute to this from the Lua reference page:
"If the value is a number, then lua_tolstring also changes the actual value in the
stack to a string. (This change confuses lua_next when lua_tolstring is applied to
keys during a table traversal.)"
Any way around this? I am open to alternate approaches. Ideally a message could be emitted like:
WARNING: conf.lua line 18: table foo does not use key 'c', ignored
(The Lua debug API doesn't give the file name and line number either, but that is a different topic.)
PS: I know, c could benign, but it could also be a typo. In a large configuration, ignoring such things could lead to hours of head scratching.
Validation will probably be much easier if written in Lua. I have something like this in mind:
local template = { a="number", b="string"}
local function validate(t)
for k,v in pairs(t) do
if template[k]==nil then
print("field "..k.." cannot be present")
elseif type(v)~=template[k] then
print("field "..k.." should be a "..template[k])
end
end
end
validate{ a = 0, b = 'bar' }
validate{ a = 0, b = 42 }
validate{ a = 0 }
validate{ a = 0, c = 'bar' }
lua_isstring is defined:
LUA_API int lua_isstring (lua_State *L, int idx) {
int t = lua_type(L, idx);
return (t == LUA_TSTRING || t == LUA_TNUMBER);
}
So instead of:
if ( lua_isstring(L, -2) )
use:
if ( lua_type(L, -2) == LUA_TSTRING )

Fuzzy Matches on dijit.form.ComboBox / dijit.form.FilteringSelect Subclass

I am trying to extend dijit.form.FilteringSelect with the requirement that all instances of it should match input regardless of where the characters are in the inputted text, and should also ignore whitespace and punctuation (mainly periods and dashes).
For example if an option is "J.P. Morgan" I would want to be able to select that option after typing "JP" or "P Morgan".
Now I know that the part about matching anywhere in the string can be accomplished by passing in queryExpr: "*${0}*" when creating the instance.
What I haven't figured out is how to make it ignore whitespace, periods, and dashes. I have an example of where I'm at here - http://jsfiddle.net/mNYw2/2/. Any help would be appreciated.
the thing to master in this case is the store fetch querystrings.. It will call a function in the attached store to pull out any matching items, so if you have a value entered in the autofilling inputfield, it will eventually end up similar to this in the code:
var query = { this.searchAttr: this.get("value") }; // this is not entirely accurate
this._fetchHandle = this.store.query(query, options);
this._fetchHandle.then( showResultsFunction );
So, when you define select, override the _setStoreAttr to make changes in the store query api
dojo.declare('CustomFilteringSelect', [FilteringSelect], {
constructor: function() {
//???
},
_setStoreAttr: function(store) {
this.inherited(arguments); // allow for comboboxmixin to modify it
// above line eventually calls this._set("store", store);
// so now, 'this' has 'store' set allready
// override here
this.store.query = function(query, options) {
// note that some (Memory) stores has no 'fetch' wrapper
};
}
});
EDIT: override queryEngine function as opposed to query function
Take a look at the file SimpleQueryEngine.js under dojo/store/util. This is essentially what filters the received Array items on the given String query from the FilteringSelect. Ok, it goes like this:
var MyEngine = function(query, options) {
// create our matching query function
switch(typeof query){
default:
throw new Error("Can not query with a " + typeof query);
case "object": case "undefined":
var queryObject = query;
query = function(object){
for(var key in queryObject){
var required = queryObject[key];
if(required && required.test){
if(!required.test(object[key])){
return false;
}
}else if(required != object[key]){
return false;
}
}
return true;
};
break;
case "string":
/// HERE is most likely where you can play with the reqexp matcher.
// named query
if(!this[query]){
throw new Error("No filter function " + query + " was found in store");
}
query = this[query];
// fall through
case "function":
// fall through
}
function execute(array){
// execute the whole query, first we filter
var results = arrayUtil.filter(array, query);
// next we sort
if(options && options.sort){
results.sort(function(a, b){
for(var sort, i=0; sort = options.sort[i]; i++){
var aValue = a[sort.attribute];
var bValue = b[sort.attribute];
if (aValue != bValue) {
return !!sort.descending == aValue > bValue ? -1 : 1;
}
}
return 0;
});
}
// now we paginate
if(options && (options.start || options.count)){
var total = results.length;
results = results.slice(options.start || 0, (options.start || 0) + (options.count || Infinity));
results.total = total;
}
return results;
}
execute.matches = query;
return execute;
};
new Store( { queryEngine: MyEngine });
when execute.matches is set on bottom of this function, what happens is, that the string gets called on each item. Each item has a property - Select.searchAttr - which is tested by RegExp like so: new RegExp(query).test(item[searchAttr]); or maybe a bit simpler to understand; item[searchAttr].matches(query);
I have no testing environment, but locate the inline comment above and start using console.debug..
Example:
Stpre.data = [
{ id:'WS', name: 'Will F. Smith' },
{ id:'RD', name:'Robert O. Dinero' },
{ id:'CP', name:'Cle O. Patra' }
];
Select.searchAttr = "name";
Select.value = "Robert Din"; // keyup->autocomplete->query
Select.query will become Select.queryExp.replace("${0]", Select.value), in your simple queryExp case, 'Robert Din'.. This will get fuzzy and it would be up to you to fill in the regular expression, here's something to start with
query = query.substr(1,query.length-2); // '*' be gone
var words = query.split(" ");
var exp = "";
dojo.forEach(words, function(word, idx) {
// check if last word
var nextWord = words[idx+1] ? words[idx+1] : null;
// postfix 'match-all-but-first-letter-of-nextWord'
exp += word + (nextWord ? "[^" + nextWord[0] + "]*" : "");
});
// exp should now be "Robert[^D]*Din";
// put back '*'
query = '*' + exp + '*';

Groovy template - code generation

I am asking for advice and opition as of the code to use with groovy templates.
All template examples on the web used a very limited logic but I simply cannot overcome that barrier and the code in my template is substantial.
Is this acceptable? What could be a better way to do this?
Thanks
Peter
The task is to generate TCL type code - specifically if then/elsif/else type contraint
if { [streq $pm_enrichment('a1') "'aaaa2'"] && [strlen $pm_enrichment('aaaa3')] &&\
[strlen $pm_enrichment('aaaa4') ] } {
set pm_enrichment('ResultAAA') 0
}
elseif { [streq $pm_enrichment('b1') "'bb2'"] && [strlen $pm_enrichment('bbb3')] &&\
[strlen $pm_enrichment('bbbb4') ] } {
set pm_enrichment('ResultBBB') 1
}
else { [streq $pm_enrichment('c1') "'cc2'"] && [strlen $pm_enrichment('ccc3')] &&\
[strlen $pm_enrichment('cccc4') ] } {
set pm_enrichment('ResultCCC') 2G
}
//////////////////////////////////////
def dataCasesClosure= {->
pos=0
arrSorted = []
mapStmt.each{arrSorted.add(it.key) }
arrSorted = arrSorted.sort()
outStr=''''''
arrSorted.each { i ->
tmpStatement = statement
tmpResultStmt = resultStmt
list=mapStmt[i]
resultList=mapResultStmt[i]
pos=0
int index = tmpStatement.indexOf(keyword);
while (index >=0){
val = list[pos].replaceAll(~'"','')
pos +=1
tmpStatement=tmpStatement.replaceFirst( ~/#/,/${val}/)
index = tmpStatement.indexOf(keyword, index+keyword.length()) ;
}
if (tmpStatement ==~/\W+$/) {
tmpStatement=tmpStatement[0..-2]
}
pos=0
index = tmpResultStmt.indexOf(keyword);
while (index >=0){
val = resultList[pos]
pos +=1
tmpResultStmt=tmpResultStmt.replaceFirst( ~/#/,/${val}/)
index = tmpResultStmt.indexOf(keyword, index+keyword.length()) ;
}
if (i==0) {
outStr= "if {${tmpStatement} } { \n\t\t ${tmpResultStmt} \n }"
} else if (i < arrSorted.size()-1 ){
outStr += "elseif {${tmpStatement} } { \n\t\t ${tmpResultStmt} \n }"
} else {
outStr += "else {${tmpStatement} } { \n\t\t ${tmpResultStmt} \n }"
}
}
outStr
} // ### dataCasesClosure
def valuesIfThenStmt= [
"statement":dataCasesClosure
]
tplIfThenStmt = '''
##############################
${statement}
'''
def engine = new SimpleTemplateEngine()
templateResult = engine.createTemplate(tplIfThenStmt).make(valuesIfThenStmt)
println templateResult.toString()
If this is all you have to generate, the template is overkill. You could have just called the dataCasesClosure directly to get its output.
Assuming it is part of a larger template then, I think it is very reasonable to use closures to produce output for a particularly complex parts, just as you have done. I have personally done this on an extreme scale with good results.