I have the log file of which i have pasted two rows below:
Nov 26 14:20:32 172.16.0.1 date=2014-11-26 time=14:18:37 devname=XXXXCCCFFFFF devid=XXXCCVVGFFDD logid=3454363464 type=traffic subtype=forward level=notice vd=root srcip=172.16.1.251 srcport=62032 srcintf="Combo_LAN" dstip=X.X.X.X dstport=X dstintf="wan2" sessionid=16172588 status=close user="X.X" group="Open Group" policyid=2 dstcountry="United States" srccountry="Reserved" trandisp=snat transip=X.X.X.X transport=X service=HTTP proto=6 applist="Block_Applications" duration=11 sentbyte=2377 rcvdbyte=784 sentpkt=6 rcvdpkt=7 identidx=5 utmaction=passthrough utmevent=webfilter utmsubtype=ftgd-cat urlcnt=1 hostname="tacoda.at.atwola.com" catdesc="Advertising"
Nov 26 14:20:32 172.16.0.1 date=2014-11-26 time=14:18:37 devname=XXXXCCCFFFFF devid=XXXCCVVGFFDD logid=3454363464 type=utm subtype=webfilter eventtype=ftgd_allow level=notice vd="root" policyid=2 identidx=5 sessionid=15536743 user="X.X" srcip=X.X.X.X srcport=X srcintf="Combo_LAN" dstip=X.X.X.X dstport=80 dstintf="wan2" service="http" hostname="streaming.sbismart.com" profiletype="Webfilter_Profile" profile="Open Group_Policy" status="passthrough" reqtype="direct" url="/diffusion/" sentbyte=984 rcvdbyte=202 msg="URL belongs to an allowed category in policy" method=domain class=0 cat=18 catdesc="Brokerage and Trading"
My question is i can parse the data if number of columns and order is fixed.
But, how do i parse the dynamic columns in the config file so that i don't get the _grokparsefailure?
Ruby Plugin can help you.
Here is the configuration:
input {
stdin{
}
}
filter {
ruby {
code => '
msg = event["message"]
msgIndex = msg.index("date=")
msgInsert = msg[msgIndex..-1]
msgMap = msgInsert.scan(/(\w+)=("(.*?)"|([^ ]+))/).map { |(first, second)| [first, second] }
for x in msgMap
key = x[0]
value = x[1]
event[key] = value
end
'
}
}
output {
stdout{
codec => rubydebug
}
}
First, get all the key=value pair by index the start value date=
Then map all the key,value to string array.
Use For loop to insert all the value.
I have try your logs and I can create all the correspond field with the value.
Hope this can help you
The simple answer to avoiding grokparsefailure is to provide a valid pattern that matches your input. That said, your question seems to imply that the fields are not always specified in this order. Given the examples, you should be using the "kv" filter to split these key/value pairs into fields.
Related
I am looking to split a string and put it into a set. The string to be split is an element of a tuple.
This string element in the tuple takes values such as(pitblockSet) :
{"P499,P376,P490,P366,P129,"}
{"P388,P491,P367,"}
{"P500,P377,P479,P355,"}
and so on. Each set refers to a Path Id(string name of the path)
The tuple was defined as :
tuple Path {
string id;
string source;
string dest;
{string} pitblockSet;
{string} roadPoints;
{string} dumpblockSet;
{string} others;
float dist;
};
And the above sets to be split refers to the element : {string} pitblockSet;
I now have to split the pitBlockSet. I am using the following :
{Path} Pbd={};
// Not putting the code to populate Pbd as it is irrelevant here
// there are several lines here for the purpose of creating set Pbd...
{string} splitPitBlocksPath[Pathid];
{string} splitDumpBlocksPath[Pathid];
execute {
for(var p in Pbd) {
var splitPitBlocksPath[p.id] = p.pitblockSet.split(",") ;
var splitDumpBlocksPath[p.id] = p.dumpblockSet.split(",") ;
}
}
The problem is when I execute it I get error in the above 2 lines appearing 4 times:
Scripting parser error: missing ';' or newline between statements.
I am not able to understand where I am going wrong
===============Added after Alex's Answer =============================
Thank you for the Answer again - It worked perfectly with some minor changes.
I might not have been able to explain the issue properly in the above hence adding the following. My actual code is much bigger and these are only an extract
Pbd for my case is a tuple of type {Path}. Path is described above. Pbd reads about 20,000 records from excel and each Pbd has tuple fields like id, source, dest, pitblockSet, dumpblockSet etc etc. These are all read from excel and populated into the tuple Pbd - this part is working fine. The 3 lines that I mentioned above were just for example of the Pbd.pitBlockSet for just 3 records out of the 20,000.
p.pitblockSet is a set but it contains only one string. The requirement is to break this string into a set. Like for example if p.pitblockSet has a value {"P499,P376,P490,P366,P129,"} for say p.id = "PT129" the expected result for this p.id is {"P499" "P376" "P490" "P366" "P129"}. Then say for example for p.id="PT1" the p.pitblockSet is {"P4,"} the expected result is a set with only one element like {"P4"}. As mentioned earlier there are several such records of p and the above two are just for example.
I have therefore modified the suggested code to some extent to fit into the problem. However I am still getting an issue with the split command.
{string} result[Pbd];
int MaxS=10;
execute {
for(var p in Pbd) {
var stringSet = Opl.item(p.pitblockSet,0);
var split= new Array(MaxS);
split=stringSet.split(",") ;
for(var i=0;i<=MaxS;i++) if ((split[i]!='null') && (split[i]!='')) result[p].add(split[i]);
writeln("result:", p.id, result[p]);
}
}
The Answers look like below :
result:PT1 {"P4"}
result:PT2 {"P5"}
result:PT3 {"P6"}
result:PT4 {"P7"}
result:PT5 {"P8"}
result:PT6 {"P8" "P330" "P455" "P341"}
result:PT7 {"P326"}
result:PT8 {"P327"}
result:PT9 {"P328"}
.
.
and so on
.
.
result:PT28097 {"P500" "P377" "P479" "P355"}
result:PT28098 {"P501" "P378" "P139"}
result:PT28099 {"P501" "P388" "P491" "P367"}
result:PT28100 {"P501" "P378" "P480"}
result:PT28101 {"P501" "P378" "P139"}
result:PT28102 {"P502"}
result:PT28103 {"P503"}
Unfortunately, I'm afraid you encounter a product limitation.
See: https://www.ibm.com/support/knowledgecenter/SSSA5P_12.10.0/ilog.odms.ide.help/refjsopl/html/intro.html?view=kc#1037020
Regards,
Chris.
int MaxS=10;
{string} splitDumpBlocksPath={"P499,P376,P490,P366,P129,"} union
{"P388,P491,P367,"} union
{"P500,P377,P479,P355,"};
range Pbd=0..card(splitDumpBlocksPath)-1;
{string} result[Pbd];
execute {
for(var p in Pbd) {
var stringSet=Opl.item(splitDumpBlocksPath,p);
writeln(stringSet);
var split= new Array(MaxS);
split=stringSet.split(",") ;
for(var i=0;i<=MaxS;i++) if ((split[i]!='null') && (split[i]!='')) result[p].add(split[i]);
}
writeln(result);
}
works fine and gives
P499,P376,P490,P366,P129,
P388,P491,P367,
P500,P377,P479,P355,
[{"P499" "P376" "P490" "P366" "P129"} {"P388" "P491" "P367"} {"P500" "P377"
"P479" "P355"}]
I have an application using solr that needs to be able to sort on two fields. The Solrj api is a little confusing, providing multiple different APIs.
I am using Solr 4.10.4
I have tried:
for (int i = 0; i < entry.getValue().size();) {
logger.debug("Solr({}) {}: {} {}", epName, entry.getKey(), entry
.getValue().get(i), entry.getValue().get(i + 1));
if (i == 0) {
query.setSort(new SolrQuery.SortClause(entry.getValue().get(i++), SolrQuery.ORDER.valueOf(entry.getValue().get(i++))));
} else {
query.addSort(new SolrQuery.SortClause(entry.getValue().get(i++), SolrQuery.ORDER.valueOf(entry.getValue().get(i++))));
}
}
When I look at the generated URL I only see the last SortClause sort=sequence+asc
I also tried creating a List and the setSorts SolrQuery method and that too seems to output only as single sort field, always the last one.
I was able to create the correct sort clause by generating it manually with strings.
I have tried addOrUpdateSort as well. I think I've tried most of the obvious combinations. of methods in the Solrj API.
This does work:
StringBuilder sortString = new StringBuilder();
for (int i = 0; i < entry.getValue().size();) {
if (sortString.length() > 0) {
sortString.append(",");
}
logger.debug("Solr({}) {}: {} {}", epName, entry.getKey(), entry
.getValue().get(i), entry.getValue().get(i + 1));
sortString.append(entry.getValue().get(i++)).append(" ").
append(SolrQuery.ORDER.valueOf(entry.getValue().get(i++)));
}
query.set("sort",sortString.toString());
The sort clause I want to see is: sort=is_cited+asc,sequence+asc
The solrj API seems to only output the final clause.
I suspect a bug in solrj 4.10
can you substitute setSort with addSort ie
for (int i = 0; i < entry.getValue().size();) {
logger.debug("Solr({}) {}: {} {}", epName, entry.getKey(), entry
.getValue().get(i), entry.getValue().get(i + 1));
if (i == 0) {
query.addSort(new SolrQuery.SortClause(entry.getValue().get(i++), SolrQuery.ORDER.valueOf(entry.getValue().get(i++))));
} else {
query.addSort(new SolrQuery.SortClause(entry.getValue().get(i++), SolrQuery.ORDER.valueOf(entry.getValue().get(i++))));
}
}
And let me know if this worked
Check out addOrUpdateSort()
Updates or adds a single sort field specification to the current sort
information. If the sort field already exist in the sort information map,
its position is unchanged and the sort order is set; if it does not exist,
it is appended at the end with the specified order..
#return the modified SolrQuery object, for easy chaining
#since 4.2
I have a unique situation where I need to query a mongo database to find the names of people who occur in a body of text. The query must specify the body of text and find records with values that occur in the body of text. How can I do this with a regular expression?
I need to write a query where this would match:
/Jonathan is a handsome guy/.test('Jonathan')
The problem is that the text inside "test" is the value of a mongo field, so this query must be written such that the body of text is provided as input, and it matches on names that occur within (are substrings of) the body of text.
A more concrete example:
db.test.find();
{ "_id" : ObjectId("547e9b79f2b519cd1657b21e"), "name" : "Jonathan" }
{ "_id" : ObjectId("547e9b88f2b519cd1657b21f"), "name" : "Sandy" }
db.test.find({name: { $in: [/Jonathan has the best queries/]} } );
I need to construct a query that would return "Jonathan" when provided the input "Jonathan has the best queries"
This $where may do the trick, though can be very slow:
db.test.find({$where: function() {
var mystr = '/Jonathan has the best queries/';
var patt = new RegExp(this.name);
if (patt.test(mystr)) return true;
return false;
}})
I have a JSON file that has a key value pairs as shown below
{
"parameters": "<FieldLabel Type='Something'><Label><![CDATA[Click on this number to initiate call <a href='tel:123456' parameter='DialMe,100.200.3000'>100.200.3000'>100.200.3000'>100.200.3000'>tel:1002003000'>100.200.3000</a> or<a href='tel:911'parameter='dial911,911'>911'>911'>911'>tel:911'>911</a> ]]></Label><Description><![CDATA[]]></Description></FieldLabel>"
}
I want to replace
parameter='DialMe,100.200.3000' with my-url-click='DialMe,null,null,100.200.3000'
and
parameter='dial911,911' with my-url-click='dial911,null,null,911'
before I can render it on as HTML using Angular's ng-bind-html and $sce.trustAsHtml.
The catch is the JSON has many such key value pairs and each of them has different values for the parameter like parameter=dial108,108.So normal string replacement is not possible.How shall I do it for each of them?
Try this:
var str = "<FieldLabel Type='Something'><Label><![CDATA[Click on this number to initiate call <a href='tel:123456' parameter='DialMe,100.200.3000'>100.200.3000'>100.200.3000'>100.200.3000'>tel:1002003000'>100.200.3000</a> or<a href='tel:911'parameter='dial911,911'>911'>911'>911'>tel:911'>911</a> ]]></Label><Description><![CDATA[]]></Description></FieldLabel>";
str = str.replace(/parameter=\'.*,/g, function(s){
return s.replace("parameter", "my-url-click") + "null,null,"
});
ColdFusion 8
I have a cfgrid that that is based on a query. It is not bound to a cfc function because I want a scrolling grid, not a paged grid (you must supply the page number and page size if you use BIND).. I can figure out how to make it filter on one column by using the following code, but I really need to filter on three columns...
grid.getDataSource().filter("OT_MILESTONE",t1);
Adding more to the filter string does not do the trick...it ignores anything more than the first pair of values..
so..I thought if I called a function that passes the three values and returned the query results to me, I could replace the Data Store for the grid..but I cannot figure out the syntax to get it to replace.
The returned variable for the query has the following format:
{"COLUMNS":["SEQ_KEY","ID","OT_MILESTONE"],"DATA":[[63677,"x","y"]]}
Any ideas?
have you looked at queryconvertforgrid()?
http://www.cfquickdocs.com/cf9/#queryconvertforgrid
Update: have you looked at these?
http://www.danvega.org/blog/index.cfm/2008/3/10/ColdFusion-8-Grid-Filtering
http://www.coldfusion-ria.com/Blog/index.cfm/2009/1/13/Playing-with-cfgrid--Filter-showhide-Columns-and-using-the-YUI-Buttons-library
http://cfsilence.com/blog/client/index.cfm/2007/8/9/Filtering-Records-In-An-Ajax-Grid
after much blood, sweat, tears and swearing..here's the answer, in case anyone else might need to filter a cfgrid by more than one variable:
var w1 = ColdFusion.getElementValue('wbs');
var t1 = ColdFusion.getElementValue('task');
var p1 = ColdFusion.getElementValue('project');
grid = ColdFusion.Grid.getGridObject('data');
store = grid.getDataSource();
store.clearFilter();
store.filterBy(function myfilter(record) {
var wantit = true;
if (trim(w1) != '') {
if(record.get('WBS_ID') != w1) {
wantit = false;
}}
if (trim(t1) != '') {
if(record.get('OT_MILESTONE') != t1) {
wantit = false;
}}
if (trim(p1) != '') {
if(record.get('PROJECT') != p1) {
wantit = false;
}}
return wantit;
});
ColdFusion.Grid.refresh('data',false);
you will need a JS trim function...
Make sure the column names are caps...