I want to add scritpted field in Kibana 5 to get stored proc name from message. To be able to visualize number of errors per each SP.
I have field "message" where I can see error text:
"[2017-02-03 05:04:51,087] # MyApp.Common.Server.Logging.ExceptionLogger [ERROR]: XmlWebServices Exception
User:
Name: XXXXXXXXXXXXXXXXXXXXXXX
Email: 926715#test.com
User ID: 926715 (PAID)
Web Server: PERFTESTSRV
Exception:
Type: MyApp.Common.Server.DatabaseException
Message: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
Source: MyApp.Common.Server
Database: MyDB
Cmd Type: StoredProcedure
Cmd Text: spGetData
Trans: YES
Trans Lvl: Unspecified"
Guide: https://www.elastic.co/blog/using-painless-kibana-scripted-fields
My plan is to add something like as a Painless script:
def m = /(?:Cmd\sText:\s*)[a-zA-Z]{1,}/.matcher(doc['message'].value);
if ( m.matches() ) {
return m.group(1)
} else {
return "no match"
}
And also I've tried
def tst = doc['message'].value;
if (tst != null)
{
def m = /(?:User\sID:\s*)[0-9]{1,}/.matcher(tst);
if ( m.matches() ) {
return m.group(1)
}
} else {
return "no match"
}
How I can address doc['message'].value?
When I try to do so I got error "Courier Fetch: 5 of 5 shards failed."
When I try doc['message.keyword'].value, I do not have full message inside. I do not understand where I can learn the structure of what message have inside and how I can refer it?
I assume that problem with lenght of message. It is too long to be type "keyword". It should be type "text" which is not supported by painless.
https://www.elastic.co/blog/using-painless-kibana-scripted-fields
Both Painless and Lucene expressions operate on fields stored in doc_values. So >for string data, you will need to have the string to be stored in data type >keyword. Scripted fields based on Painless also cannot operate directly on >_source.
https://www.elastic.co/guide/en/elasticsearch/reference/master/keyword.html_italic_
A field to index structured content such as email addresses, hostnames, status >codes, zip codes or tags.
If you need to index full text content such as email bodies or product >descriptions, it is likely that you should rather use a text field.
Related
In [50]: r = confluence.search(cql=f'title contains "Agent Alert - {event_name}" and label = "agent-event"')
# prints the params of the request
{'cql': 'title contains "Agent Alert - SYS_THRESHOLD_REACHED" and label = "agent-event"', 'expend': 'body.view'}
And I get this error
In [49]: r.content
Out[49]: b'{"statusCode":400,"data":{"authorized":false,"valid":true,"allowedInReadOnlyMode":true,"errors":[],"successful":false},"message":"Could not parse cql : title contains \\"Agent Alert - SYS_THRESHOLD_REACHED\\" and label = \\"agent-event\\"","reason":"Bad Request"}'
However I tried using the exact string in confluence webUI and it works.
Managed to work it out...
it doesn't seem to like contains in the API, so have to use ~
the spaces need to be replaced with + within the title search
so turned out this is what is accepted
title~"Agent+Alert+-+SYS_THRESHOLD_REACHED" and label="agent-event"
I need to parse an Apache log file and output IP, URL, and URL status code into hashes but don't know how to put the elements into a hash.
My code uses regular expressions to get the info I need from each line of the log file:
line_array = File.readlines("access_log")
line_array.each { |line| }
#regexp
md = (/^([:\d\.]+) .*\[.*\].*\"[A-Z]+ *(.+) HTTP\S*\s(\d+)/).match(line)
ip = md[1]
url = md[2]
status = md[3]
Is my current code even on the right track to be able to do this?
I need the hashes to display the item and then the frequency of said item. So if we have the hash for ip addresses it will display the ip addresses in the log file followed the the frequency of that specific ip.
Assuming your md correctly returns an array of strings which is what you appear to be expecting, then try:
line_array = File.readlines("access_log")
line_array.each { |line| }
#regexp
md = (/^([:\d\.]+) .*\[.*\].*\"[A-Z]+ *(.+) HTTP\S*\s(\d+)/).match(line)
hash = Hash.new
hash[:ip] = md[1]
hash[:url] = md[2]
hash[:status] = md[3]
This will create a hash object with the three keys:
hash
=> { ip: 'whatever is in md[1]', url: 'whatever is in md[2]', status: 'whatever is in md[3]' }
Also just to highlight, by accessing md[1] you're accessing the second element of the array, if you want the first you need md[0]
I need to send a user text input to the robot through the integrated tablet, and catch it somehow, for further processing in Choregraphe.
After reading the Aldebaran documentation about ALTabletService API, I found few methods which might be a solution to all this. The methods are ALTabletService::showInputTextDialog and ALTabletService::onInputText, but somehow I can't get them to work: they return absolutely nothing when I input some text through the tablet.
I need access to the string created when the user inputs a piece of text. Any advice how to do it?
i realized this without ALTabletService methods showInputTextDialog or onInputText
My Approach:
I made an html page with an input field and a button to send the input.
On button press I use the forceInput method from ALDialog doc via QiMessaging Javascript library. doc
I can't test it right now but this should help as a inspiration
function forceInput(input) {
QiSession(function(session) {
session.service('ALDialog').then(function(ALDialog) {
ALDialog.forceInput(input);
});
}
}
Now you can send the input to the topic.
This could be sth like "imput_from_tablet blablabla".
And in the Dialog you catch
u:(imput_from_tablet _*) $1
Where $1 should be blablabla.
Hope that helps
best regards
You can create a webpage for the tablet and package it in your application - see the documentation here; then on that webpage you can create a text input field (be careful that the bottom half of the screen will be hidden by the keyboard when the field is selected), and then use the javascript SDK to (for example) raise an ALMemory event with the inputted text value, that you can then get from Choregraphe.
I had exactly the same problem and I found this ALTabletService::onInputText method in the signal list. You can find examples how to use signals on the same page. Based on these examples I created the following script that can get a value from the input field:
import qi
import sys
def main(app):
try:
session = app.session
tabletService = session.service("ALTabletService")
tabletService.showInputTextDialog("Example dialog", "OK", "Cancel")
signal_id = 0
def callback(button_id, input_text):
if button_id == 1:
print "'OK' button is pressed."
print "Input text: " + input_text
if button_id == 0:
print "'Cancel' button is pressed"
tabletService.onInputText.disconnect(signal_id)
app.stop()
# attach the callback function to onJSEvent signal
signal_id = tabletService.onInputText.connect(callback)
print "Signal ID: {}".format(signal_id)
app.run()
except Exception, e:
print "Error was: ", e
if __name__ == "__main__":
ip = "10.0.10.254" # the IP of the robot
port = 9559
try:
connection_url = "tcp://{}:{}".format(ip, port)
app = qi.Application(url=connection_url)
app.start()
except RuntimeError:
print("Can't connect to Naoqi.")
sys.exit(1)
main(app)
I'm trying to build a chatbot using Amazon's boto3 library. Right now, I am trying to create an intent using the put_intent function. My code is as follows:
intent = lexClient.put_intent(name = 'test',
sampleUtterances = ["Who is messi?"]
)
When I try running this, I get the following exception:
botocore.errorfactory.BadRequestException: An error occurred
(BadRequestException) when calling the PutIntent operation: RelativeId
does not match Lex ARN format: intent:test2:$LATEST
Can anyone tell me what I'm doing wrong?
I got the same error when trying to have a digit in intent name field. Realized that was not allowed when trying to do the same from AWS console.
Error handling could really be more specific.
Try taking the question mark out of the utterance, that has caused me issues in the past!
You need to run GetSlotType. That will return current checksum for that slot. Put that checksum in your PutSlotType checksum. Big bang boom.
https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/LexModelBuildingService.html#getSlotType-property
var params = {
name: "AppointmentTypeValue",
checksum:'54c6ab5f-fe30-483a-a364-b76e32f6f05d',
description: "Type of dentist appointment to schedule'",
enumerationValues: [
{
value: "cleaning"
},
{
value: "whitening"
},
{
value: "root canal"
},
{
value:"punch my face"
}
]
};
For the put_intent function I faced similar issues.
At least the following 3 are worth mentioning.
Sample Utterances
There are requirements for the sample utterances:
An utterance can consist only of Unicode
characters, spaces, and valid punctuation marks. Valid punctuation
marks are: periods for abbreviations, underscores, apostrophes, and
hyphens. If there is a slot placeholder in your utterance, ensure that
it's in the {slotName} format and has spaces at both ends.
It seems like there is no error raised when calling the put_intent function with the following code.
intent = lexClient.put_intent(name = 'test',
sampleUtterances = ["Who is messi"]
)
However, if you try to add it to your bot and start building the bot it will fail.
To fix it remove the question mark at the end of you sampleUtterance.
intent = lexClient.put_intent(name = 'test',
sampleUtterances = ["Who is messi?"]
)
Prior intent version
If your intent already exists you need to add the checksum to your function call. To get the checksum of your intent you can use the get_intent function.
For example docs:
response = client.get_intent(
name='test',
version='$LATEST'
)
found_checksum = response.get('checksum')
After that you can put a new version of the intent:
intent = lexClient.put_intent(name = 'test',
sampleUtterances = ["Who is messi"],
checksum = found_checksum
)
Intent Name (correct in your case, just adding this for reference)
It seems like the name can only contain letters, underscores, and should be <=100 in length. Haven't found anything in the docs. This is just trial and error.
Calling put_intent with the following:
intent = lexClient.put_intent(name = 'test_1',
sampleUtterances = ["Who is messi"]
)
Results in the following error:
BadRequestException: An error occurred (BadRequestException) when calling the PutIntent operation: RelativeId does not match Lex ARN format: intent:test_1:$LATEST
To fix the name you can replace it to:
intent = lexClient.put_intent(name = 'test',
sampleUtterances = ["Who is messi"]
)
I'm testing a web service that returns JSON responses and I'd like to pull multiple values from the response. A typical response would contain multiple values in a list. For example:
{
"name":"#favorites",
"description":"Collection of my favorite places",
"list_id":4894636,
}
A response would contain many sections like the above example.
What I'd like to do in Jmeter is go through the JSON response and pull each section outlined above in a manner that I can tie the returned name and description as one entry to iterate over.
What I've been able to do thus far is return the name value with regular expression extractor ("name":"(.+?)") using the template $1$. I'd like to pull both name and description but can't seem to get it to work. I've tried using a regex "name":"(.+?)","description":"(.+?)" with a template of $1$$2$ without any success.
Does anyone know how I might pull multiple values using regex in this example?
You can just add (?s) to the regex to avoid line breaks.
E.g: (?s)"name":"(.+?)","description":"(.+?)"
It works for me on assertions.
It may be worth to use BeanShell scripting to process JSON response.
So if you need to get ALL the "name/description" pairs from response (for each section) you can do the following:
1. extract all the "name/description" pairs from response in loop;
2. save extracted pairs in csv-file in handy format;
3. read saved pairs from csv-file later in code - using CSV Data Set Config in loop, e.g.
JSON response processing can be implemented using BeanShell scripting (~ java) + any json-processing library (e.g. json-rpc-1.0):
- either in BeanShell Sampler or in BeanShell PostProcessor;
- all the required beanshell libs are currently provided in default
jmeter delivery;
- to use json-processing library place jar into JMETER_HOME/lib folder.
Schematically it will look like:
in case of BeanShell PostProcessor:
Thread Group
. . .
YOUR HTTP Request
BeanShell PostProcessor // added as child
. . .
in case of BeanShell Sampler:
Thread Group
. . .
YOUR HTTP Request
BeanShell Sampler // added separate sampler - after your
. . .
In this case there is no difference which one use.
You can either put the code itself into the sampler body ("Script" field) or store in external file, as shown below.
Sampler code:
import java.io.*;
import java.util.*;
import org.json.*;
import org.apache.jmeter.samplers.SampleResult;
ArrayList nodeRefs = new ArrayList();
ArrayList fileNames = new ArrayList();
String extractedList = "extracted.csv";
StringBuilder contents = new StringBuilder();
try
{
if (ctx.getPreviousResult().getResponseDataAsString().equals("")) {
Failure = true;
FailureMessage = "ERROR: Response is EMPTY.";
throw new Exception("ERROR: Response is EMPTY.");
} else {
if ((ResponseCode != null) && (ResponseCode.equals("200") == true)) {
SampleResult result = ctx.getPreviousResult();
JSONObject response = new JSONObject(result.getResponseDataAsString());
FileOutputStream fos = new FileOutputStream(System.getProperty("user.dir") + File.separator + extractedList);
if (response.has("items")) {
JSONArray items = response.getJSONArray("items");
if (items.length() != 0) {
for (int i = 0; i < items.length(); i++) {
String name = items.getJSONObject(i).getString("name");
String description = items.getJSONObject(i).getString("description");
int list_id = items.getJSONObject(i).getInt("list_id");
if (i != 0) {
contents.append("\n");
}
contents.append(name).append(",").append(description).append(",").append(list_id);
System.out.println("\t " + name + "\t\t" + description + "\t\t" + list_id);
}
}
}
byte [] buffer = contents.toString().getBytes();
fos.write(buffer);
fos.close();
} else {
Failure = true;
FailureMessage = "Failed to extract from JSON response.";
}
}
}
catch (Exception ex) {
IsSuccess = false;
log.error(ex.getMessage());
System.err.println(ex.getMessage());
}
catch (Throwable thex) {
System.err.println(thex.getMessage());
}
As well a set of links on this:
JSON in JMeter
Processing JSON Responses with JMeter and the BSF Post Processor
Upd. on 08.2017:
At the moment JMeter has set of built-in components (merged from 3rd party projects) to handle JSON without scripting:
JSON Path Extractor (contributed from ATLANTBH jmeter-components project);
JSON Extractor (contributed from UBIK Load Pack since JMeter 3.0) - see answer below.
I am assuming that JMeter uses Java-based regular expressions... This could mean no named capturing groups. Apparently, Java7 now supports them, but that doesn't necessarily mean JMeter would. For JSON that looks like this:
{
"name":"#favorites",
"description":"Collection of my favorite places",
"list_id":4894636,
}
{
"name":"#AnotherThing",
"description":"Something to fill space",
"list_id":0048265,
}
{
"name":"#SomethingElse",
"description":"Something else as an example",
"list_id":9283641,
}
...this expression:
\{\s*"name":"((?:\\"|[^"])*)",\s*"description":"((?:\\"|[^"])*)",(?:\\}|[^}])*}
...should match 3 times, capturing the "name" value into the first capturing group, and the "description" into the second capturing group, similar to the following:
1 2
--------------- ---------------------------------------
#favorites Collection of my favorite places
#AnotherThing Something to fill space
#SomethingElse Something else as an example
Importantly, this expression supports quote escaping in the value portion (and really even in the identifier name portion as well, so that the Javascript string I said, "What is your name?"! will be stored in JSON as AND parsed correctly as I said, \"What is your name?\"!
Using Ubik Load Pack plugin for JMeter which has been donated to JMeter core and is since version 3.0 available as JSON Extractor you can do it this way with following Test Plan:
namesExtractor_ULP_JSON_PostProcessor config:
descriptionExtractor_ULP_JSON_PostProcessor config:
Loop Controller to loop over results:
Counter config:
Debug Sampler showing how to use name and description in one iteration:
And here is what you get for the following JSON:
[{ "name":"#favorites", "description":"Collection of my favorite places", "list_id": 4894636 }, { "name":"#AnotherThing", "description":"Something to fill space", "list_id": 48265 }, { "name":"#SomethingElse", "description":"Something else as an example", "list_id":9283641 }]
Compared to Beanshell solution:
It is more "standard approach"
It performs much better than Beanshell code
It is more readable