I'm executing a parallel block that has a list as an output, like this:
[
{
"Query": "SELECT * FROM some_table_1 WHERE my_partition = 20220101",
"PartitionValue": "20220101"
},
{
"Query": "SELECT * FROM some_table_2 WHERE my_partition = 20220101",
"PartitionValue": "20220101"
},
{
"Query": "SELECT * FROM some_table_3 WHERE my_partition = 20220102",
"PartitionValue": "20220102"
}
]
After the parallel block I have a Choice state with a rule that all the PartitionValue must be the same, else I end the execution with a error.
So I wrote something like this in my choice rule:
"$.[0].PartitionValue" == "$.[1].PartitionValue"
AND
"$.[0].PartitionValue" == "$.[2].PartitionValue"
AND
"$.[1].PartitionValue" == "$.[2].PartitionValue"
The thing is, when i'm trying to save the state machine it throws an error saying square brackets ']' are not valid characters.
I checked the documentation for something like "you can't access lists in choice states you st00pid' but did not find.
What am I doing wrong? Is there any other way I can achieve this validation step?
Related
I wish to abort the pipeline if the user did not select any value for Active Choice parameter for single/multi choice/string pipeline parameter.
For example I have Active Choices Reactive Parameter Named "IPAddress" of Type "Multi Select" with Groovy Script as below:
if (Location.equals("MyTown")) {
return["DDL1", "DDL2", "DDL3", "DDL4"]
} else if (Location.equals("Your Town")) {
return["DDP1", "DDP2"]
} else {
return ["Select an IP from the drop-down"]
}
Thus, once I run the pipeline i see "Select an IP from the drop-down" for IPAddress.
Now, If the user does not select anything from the dropdown the pipeline should fail & abort.
In the pipeline script I have written the below condition check which fails to check the condition despite user ignoring to select any IPAddress.
def ex(param){
currentBuild.result = 'ABORTED'
error('BAD PARAM: ' + param)
}
pipeline {
agent any
stages {
stage ("Pre-Check Parameters") {
steps {
echo "Pre-Check called in pipeline"
script {
if ("${params.IPAddress}" == null) {ex("IPAddress")}
//if ("${params.myEnv}" == null) {ex("b")}
//if ("${params.myLoc}" == null) {ex("c")}
}
}
}
}
}
Can you please suggest what could be the issue here ?
Do you have any constraint against using the input step?
def days=''
pipeline{
agent any;
stages {
stage('master'){
steps{
script {
try {
timeout(time:10, unit:'SECONDS') {
days = input message: 'Please enter the time window in number of days', ok: 'Fetch Statistics', parameters: [string(defaultValue: '90', description: 'Number of days', name: 'days', trim: true)]
}
}
catch (err){
error("No custom value has been entered for number of days.")
}
}
}
}
}
}
To determine if your string is empty you can use the method .trim(). It will remove leading and trailing spaces from your string. The two magic words are "Groovy Truth". An empty string is false in Groovy. This makes it easier to evaluate conditional expressions. Means in your case, if you use .trim() in combination with an if conditional then the Groovy Truth value of the string will be used for the evaluation.
Your pipeline should work if you change it to the following. It will check if your variable is null or empty:
script {
if (!params.IPAddress?.trim()) {
ex("IPAddress")
}
}
Before you mark this as duplicate, note that others are asking about the error Invalid column index undefined. ... or Invalid column index 5. Should be an integer in the range [0-4]. But no. Mine is "3 should be an integer in the range of [0-3]." Also, the table does work without the formatter.format() line (just no formatting).
google.charts.load('current', {'packages':[data.chartType.toLowerCase()]});
google.charts.setOnLoadCallback(function(){
var googleData = new google.visualization.DataTable();
for (var h in data.headers) {
googleData.addColumn(data.headers[h].type, data.headers[h].html);
if (data.headers[h].format) {
var formatter = new google.visualization.NumberFormat(data.headers[h].format);
console.log(data.headers[h].format);
formatter.format(googleData, h); // Errors Here
}
}
/* ... Add Rows ... Draw Chart ... */
}
The header in question looks like this:
header[3] = {
"html": "Total Amount",
"source": "total_amount",
"type": "number",
"format": {
"negativeColor": "#F05840", //orange
"negativeParens": true,
"pattern": "#,###",
"prefix": "$",
"suffix": "",
}
}
I can't figure out why it would be erroring.
Please forgive me for any typos here, I had to hand-edit the spacing and remove my company's specific info upon pasting the code here.
Edit
WhiteHat is correct in that my h variable was a string instead of an integer, and calling parseInt did remove that error. However, instead of calling parseInt on the formatter and wherever else it's needed, I got rid of my for (var h in data.headers) calls and went with the bog-standard for (var h = 0; h < data.headers.length; h++). Although more verbose with more room for typos, it's far more standardized and predictable.
I'm still having issues with GoogleCharts NumberFormatter, but that's for another round of research and questions, not this one.
make sure you're passing a number (3),
and not a string ('3'),
by using --> parseInt...
e.g.
formatter.format(googleData, parseInt(h)); // <-- here
I have an app written in C++ with 16 threads which reads from the output of wireshark/tshark. Wireshark/tshark dissects pcap files which are gsm_map signalling captures.
Mongodb is 2.6.7
The structure I need for my documents are like this:
Note "packet" is an array, it will become apparent why later.
For all who don't know TCAP, the TCAP layer is transaction-oriented, this means, all packets include:
Transaction State: begin/continue/end
Origin transaction ID (otid)
Destination transaction ID (dtid)
So for instance, you might see a transaction comprising 3 packets, which looking at the TCAP layer would be roughly this
Two packets, one "begin", one "end".
{
"_id" : ObjectId("54ccd186b8ea19c89ee8f231"),
"deleted" : "0",
"packet" : {
"datetime" : ISODate("2015-01-31T12:58:11.939Z"),
"signallingType" : "M2PA",
"opc" : "326",
"dpc" : "6406",
"transState" : "begin",
"otid" : "M2PA0400435B",
"dtid" : "",
"sccpCalling" : "523332075100",
"sccpCalled" : "523331466304",
"operation" : "mo-forwardSM (46)",
...
}
}
/* 1 */
{
"_id" : ObjectId("54ccd1a1b8ea19c89ee8f7c5"),
"deleted" : "0",
"packet" : {
"datetime" : ISODate("2015-01-31T12:58:16.788Z"),
"signallingType" : "M2PA",
"opc" : "6407",
"dpc" : "326",
"transState" : "end",
"otid" : "",
"dtid" : "M2PA0400435B",
"sccpCalling" : "523331466304",
"sccpCalled" : "523332075100",
"operation" : "Not Found",
...
}
}
Because of the network architecture, we're tracing in two (2) points, and the traffic is balanced amongst these two points. This means sometimes we see "continue"s or "end"s BEFORE a "begin". Conversely, we might see a "continue" BEFORE a "begin" or "end". In short, transactions are not ordered.
Moreover, multiple end-points are "talking" amongst themselves, and transactionIDs might get duplicated, 2 endpoints could be using the same tid and other 2 endpoints at the same time, though this doesn't happen all the time, it does happen.
Because of the later, I also need to use the SCCP layer's "calling" and "called" Global titles (like phone numbers).
Bear in mind that I don't know which way a given packet is going, so this is what I'm doing:
Whenever I get a new packet I must find whether the transaction already exists in mongodb, I'm using upsert to do this.
I do this by searching the current's packet otid or dtid in either otid or dtid of existing packets
If it does: push the new packet into the existing document.
If it doesn't: create a new document with the packet.
As an example, this is a upsert for an "end" which should find a "begin":
db.runCommand(
{
update: "packets",
updates:
[
{ q:
{ $and:
[
{
$or: [
{ "packet.otid":
{ $in: [ "M2PA042e3918" ] }
},
{ "packet.dtid":
{ $in: [ "M2PA042e3918" ] }
}
]
},
{
$or: [
{ "packet.sccpCalling":
{ $in: [ "523332075151", "523331466305" ] }
},
{ "packet.sccpCalled":
{ $in: [ "523332075151", "523331466305" ] }
}
]
}
]
},
{
$setOnInsert: {
"unique-id": "422984b6-6688-4782-9ba1-852a9fc6db3b", deleted: "0"
},
$push: {
packet: {
datetime: new Date(1422371239182),
opc: "327", dpc: "6407",
transState: "end",
otid: "", dtid: "M2PA042e3918", sccpCalling: "523332075151", ... }
}
},
upsert: true
}
],
writeConcern: { j: "1" }
}
)
Now, all of this works, until I put it in production.
It seems packets are coming way to fast and I see lots of:
"ClientCursor::staticYield Can't Unlock B/c Of Recursive Lock" Warnings
I read that we can ignore this warning, but I've found that my upserts DO NOT update the documents! It looks like there's a lock and mongodb forgets about the update. If I change the upsert to a simple insert, no packets are lost
I also read this is related to no indexes being used, I have the following index:
"3" : {
"v" : 1,
"key" : {
"packet.otid" : 1,
"packet.dtid" : 1,
"packet.sccpCalling" : 1,
"packet.sccpCalled" : 1
},
"name" : "packet.otid_1_packet.dtid_1_packet.sccpCalling_1_packet.sccpCalled_1",
"ns" : "tracer.packets"
So in conclusion:
1.- If this index is not correct, can someone please help me creating the correct index?
2.- Is it normal that mongo would NOT update a document if it finds a lock?
Thanks and regards!
David
Why are you storing all of the packets in an array? Normally in this kind of situation it's better to make each packet a document on its own; it's hard to say more without more information about your use case (or, perhaps, more knowledge of all these acronyms you're using :D). Your updates would become inserts and you would not need to do the update query. Instead, some other metadata on a packet would join related packets together so you could reconstruct a transaction or whatever you need to do.
More directly addressing your question, I would use an array field tids to store [otid, dtid] and an array field sccps to store [sccpCalling, sccpCalled], which would make your update query look like
{ "tids" : { "$in" : ["M2PA042e3918"] }, "sccps" : { "$in" : [ "523332075151", "523331466305" ] } }
and amenable to the index { "tids" : 1, "sccps" : 1 }.
I am trying to use a kind of builder pattern to build an OR query using multiple criteria depending upon the scenario. An example is
public class Stylist extends Model {
public String firstName;
public String lastName;
public String status;
...
}
I would like to search Stylist collection if the first name or last name matches a given string and also status matches another string. I am writing the query as follows:
MorphiaQuery query = Stylist.q();
if (some condition) {
query.or(query.criteria("status").equal("PendingApproval"), query.criteria("status").equal(EntityStatus.ACTIVE));
}
if (some other condition as well) {
query.or(query.criteria("firstName").containsIgnoreCase(name), query.criteria("lastName").containsIgnoreCase(name));
}
When both the conditions are met, I see that query contains only the criteria related to firstName and lastName i.e. different OR criteria are not added/appended but overwritten. It's quite different from filter criteria where all the different filter conditions are appended and you can easily build queries containing multiple AND conditions.
I can solve the problem by putting my conditions differently and building my queries differently but doesn't seem to be an elegant way. Am I doing something wrong ?
I am using Play! Framework 1.2.4 and Play Morphia module version 1.2.5a
Update
To put it more clearly, I would like to AND multiple OR queries. Concretely, in the above mentioned scenario, I would like to
I would like to search for Stylists where :
firstName or lastName contains supplied name AND
status equals ACTIVE or PENDING_APPROVAL.
I have been able to construct the query directly on Mongo shell through :
db.stylists.find({$and: [{$or : [{status: "PENDING_APPROVAL"}, {status : "ACTIVE"}]},{$or : [{firstName : { "$regex" : "test" , "$options" : "i"}}, {lastName : { "$regex" : "test" , "$options" : "i"}}]}] }).pretty();
But have not able to achieve the same through Query API methods. Here is my attempt :
Query<Stylist> query = MorphiaPlugin.ds().find(Stylist.class);
CriteriaContainer or3 = query.or(query.criteria("firstName").containsIgnoreCase(name), query.criteria("lastName").containsIgnoreCase(name));
CriteriaContainer or4 = query.or(query.criteria("status").equal("PENDING_APPROVAL"), query.criteria("status").equal("ACTIVE"));
query.and(or3, or4);
query.toString() results in following output : { "$or" : [ { "status" : "PENDING_APPROVAL"} , { "status" : "ACTIVE"}]}
Not sure, where am I missing ?
I guess there could be 2 ways to handle your case:
first, use List<Criteria>
MorphiaQuery query = Stylist.q();
List<Criteria> l = new ArrayList<Criteria>()
if (some condition) {
l.add(query.criteria("status").equals("PendingApproval");
l.add(query.criteria("status").equal(EntityStatus.ACTIVE));
}
if (some other conditional as well) {
l.add(query.criteria("firstName").containsIgnoreCase(name));
l.add(query.criteria("lastName").containsIgnoreCase(name));
}
query.or(l.toArray());
Second, use CritieriaContainer
MorphiaQuery query = Stylist.q();
CriteriaContainer cc = null;
if (some condition) {
cc = query.or(query.criteria("status").equal("PendingApproval"), query.criteria("status").equal(EntityStatus.ACTIVE));
}
if (some other condition) {
if (null != cc) query.or(cc, query.criteria("firstName").containsIgnoreCase(name), query.criteria("lastName").containsIgnoreCase(name));
else query.or(query.criteria("firstName").containsIgnoreCase(name), query.criteria("lastName").containsIgnoreCase(name));
}
I have some documents saved in a collection (called urls) that look like this:
{
payload:{
url_google.com:{
url:'google.com',
text:'search'
}
}
},
{
payload:{
url_t.co:{
url:'t.co',
text:'url shortener'
}
}
},
{
payload:{
url_facebook.com:{
url:'facebook.com',
text:'social network'
}
}
}
Using the mongo CLI, is it possible to look for subdocuments of payload that match /^url_/? And, if that's possible, would it also be possible to query on the match's subdocuments (for example, make sure text exists)?
I was thinking something like this:
db.urls.find({"payload":{"$regex":/^url_/}}).count();
But that's returning 0 results.
Any help or suggestions would be great.
Thanks,
Matt
It's not possible to query against document keys in this way. You can search for exact matches using $exists, but you cannot find key names that match a pattern.
I assume (perhaps incorrectly) that you're trying to find documents which have a URL sub-document, and that not all documents will have this? Why not push that type information down a level, something like:
{
payload: {
type: "url",
url: "Facebook.com",
...
}
}
Then you could query like:
db.foo.find({"payload.type": "url", ...})
I would also be remiss if I did not note that you shouldn't use dots (.) is key names in MongoDB. In some cases it's possible to create documents like this, but it will cause great confusions as you attempt to query into embedded documents (where Mongo uses dot as a "path separator" so to speak).
You can do it but you need to use aggregation: Aggregation is pipeline where each stage is applied to each document. You have a wide range of stages to perform various tasks.
I wrote an aggregate pipeline for this specific problem. If you don't need the count but the documents itself you might want to have a look at the $replaceRoot stage.
EDIT: This works only from Mongo v3.4.4 onwards (thanks for the hint #hwase0ng)
db.getCollection('urls').aggregate([
{
// creating a nested array with keys and values
// of the payload subdocument.
// all other fields of the original document
// are removed and only the filed arrayofkeyvalue persists
"$project": {
"arrayofkeyvalue": {
"$objectToArray": "$$ROOT.payload"
}
}
},
{
"$project": {
// extract only the keys of the array
"urlKeys": "$arrayofkeyvalue.k"
}
},
{
// merge all documents
"$group": {
// _id is mandatory and can be set
// in our case to any value
"_id": 1,
// create one big (unfortunately double
// nested) array with the keys
"urls": {
"$push": "$urlKeys"
}
}
},
{
// "explode" the array and create
// one document for each entry
"$unwind": "$urls"
},
{
// "explode" again as the arry
// is nested twice ...
"$unwind": "$urls"
},
{
// now "query" the documents
// with your regex
"$match": {
"urls": {
"$regex": /url_/
}
}
},
{
// finally count the number of
// matched documents
"$count": "count"
}
])