access json body parameters in custom formatter of lnav log file navigator - libcurl

I'm using lnav to filter and query on top of the custom log file we have created.
As this is a custom log file, I need to create a custom format file and install it using below command to define the structure of the log entries.
lanv -i custom_format.json
Custom format JSON: https://pastebin.com/z7sCVqxd
Though this is working at the high level, I would also like to apply filters using the parameters that are present in body JSON.
The body parameter I've mentioned in the custom formatter is JSON. Not the entire log entry is JSON but only the body parameter value alone is the JSON., In the body parameter only the log level, message and other useful information are present.
Is it possible to use something like,
"level-field" : body.level,
"level" : {
"error" : "error",
"warning" : "info"
},
And also how to define other parameters from the body json to Value for defining "kind" and "identifier" or not
Thanks

So far I've analyzed I couldn't find any direct solution. I went through the documentation of lnav (here) and found out that there are few JSON functions available which we can use to extract the data from JSON based columns.
with mylog as (select jobserver, workqueue, processid, jget('{"' || col_0 || '"}','/LEVEL') level, jget('{"' || col_0 || '"}','/PREFIX') pid, jget('{"' || col_0 || '"}','/MESSAGE') message from logline) select * from mylog where processid = 358812 limit 1
This temporarily solved the problem. But I'm still looking for the support from lnav API itself.

Related

CloudWatch Metric Filter for checking JSON key exists

I'm trying to come up with a metric filter expression that filters CloudWatch Logs when a special JSON key attribute is present.
Use case is the following: the application does all kinds of logging(in JSON format) and whenever it has a special JSON key(nested JSON response from third-part service), I would like to filter it.
Example logs:
{"severity":"INFO","msg":"EVENT","event":{"key1":"value1"}}
{"severity":"INFO","msg":"FooService responded","response":{"response_code":800}}
Filter patterns that I've tried that don't work:
{ $.response }
{ $.response = *}
{ $.response = "*"}
{ $.response EXISTS }
{ $.response IS TRUE }
{ $.response NOT NULL }
{ $.response != NULL }
Expected filtering result:
{"severity":"INFO","msg":"FooService responded","response":{"response_code":800}}
{ $.response EXISTS } does the opposite of what I expect(returns the 1st line rather than then 2nd) but I'm not sure how to negate it.
Reference material: Filter and pattern syntax # CloudWatch User Guide
I haven't found a good solution.
But I did find one at least.
If you search for a key being != a specific value, it seems to do a null check on it.
So if you say:
{$.response != "something_no_one_should_have_ever_saved_this_response_as"}
Then you get all entries where response exists in your json, and where it's not your string (hopefully all of the valid entries)
Definitly not a clean solution, but it seems to be pretty functional
I don't have a solution to the task of finding records where a field exists. Indeed, the linked document in the question specifically calls this out as not supported.
but
If we simply reverse our logic this becomes a more tractable problem. Looking at your data, you want All records where there's a response key but that could also be stated as All records where there isn't an events key.
This means you could accomplish the task with {$.event NOT EXISTS}. Of course, this becomes more complicated the more types of log messages you get (I had to chain three different NOT EXISTS queries for my use case) but it does solve the problem.

You called the function 'Value Selector' with these arguments

I am in the process of implementing a LoanBroker with Mulesoft but have an error message when sending a request. I get the following error message back from Postman and Mulesoft Anypoint Studio:
ERROR 2021-06-27 15:20:51,133 [[MuleRuntime].uber.04: [loanbroker].LoanBrokerFlow_Gr7.CPU_LITE #254be3ee] [processor: LoanBrokerFlow_Gr7/processors/0; event: 7e49f560-d74a-11eb-b598-b66921dc5aa5] org.mule.runtime.core.internal.exception.OnErrorPropagateHandler:
********************************************************************************
Message: "You called the function 'Value Selector' with these arguments:
1: Binary ("" as Binary {base: "64"})
2: Name ("amount")
But it expects one of these combinations:
(Array, Name)
(Array, String)
(Date, Name)
(DateTime, Name)
(LocalDateTime, Name)
(LocalTime, Name)
(Object, Name)
(Object, String)
(Period, Name)
(Time, Name)
1| payload.amount
^^^^^^^^^^^^^^
Trace:
at main (line: 1, column: 1)" evaluating expression: "payload.amount".
Element : LoanBrokerFlow_Gr7/processors/0 # loanbroker:bi_gruppe7.xml:34 (Copy_of_setAmount)
Element DSL : <set-variable value="#[payload.amount]" doc:name="Copy_of_setAmount" doc:id="cbcca557-1a69-4cf2-80b1-64333175589d" variableName="amount"></set-variable>
Error type : MULE:EXPRESSION
FlowStack : at LoanBrokerFlow_Gr7(LoanBrokerFlow_Gr7/processors/0 # loanbroker:bi_gruppe7.xml:34 (Copy_of_setAmount))
(set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
********************************************************************************
Can anyone help me?
Thanks
This generally happens when one tries to access inner value of a payload like json but incoming payload is NOT actually a json type.
One could check the payload mediaType and then try to access the amount in order to avoid Value Selector exception.
%dw 2.0
output application/java
---
if( !isEmpty(payload) and payload.^mediaType contains "json" )
payload.amount
else
read(payload, "application/json").amount //best effort
Would recommend creating a separate dataweave file like dwl/set-amount.dwl and referencing it.
You are probably sending some body in the HTTP request from Postman but Mule doesn't know how to read it. Maybe you did not the Content-Type header in the request to let DataWeave know it is a JSON (application/json) or XML (application/XML).
Ensure you are sending the right content type.
I ran into the same situation. I know exactly the base64 is a json. So, I tried to set the MIME Type by
<set-payload value="#[payload]" doc:name="Set Payload" mimeType="application/json"/>
It works for me.

PowerBI Query WebMethod.Post returns Expression.Error: We cannot convert the value "POST" to type Function

I'm using a website that requires that their API key AND query data be submitted using Webform.Post method. I'm able to get this to work in Python, C# and I'm even able to construct and execute a cURL command which returns a usable JSON file that Excel can parse. I am also using Postman to validate my parameters and everything looks good using all these methods. However, my goal is to build a query form that I can use within Excel but I can't get past this query syntax in PowerBi Query.
For now I am doing a simple query. That query looks like this:
let
url_1 = "https://api.[SomeWebSite].com/api/v1.0/search/keyword?apiKey=blah-blah-blah",
Body_1 = {
"SearchByKeywordRequest:
{
""keyword"": ""Hex Nuts"",
""records"": 0,
""startingRecord"": 0,
""searchOptions"": Null.Type,
""searchWithYourSignUpLanguage"": Null.Type
}"
},
Source = WebMethod.Post(url_1,Body_1)
in
Source
ScreenSnip showing valid syntax
It generates the following error:
Expression.Error: We cannot convert the value "POST" to type Function.
Details:
Value=POST
Type=[Type]
ScreenSnip of Error as it appears in PowerQuery Advanced Editor
I've spend the better part of the last two days trying to find either some example using this method or documentation. The Microsoft documentation simple states the follow:
WebMethod.Post
04/15/2018
2 minutes to read
About
Specifies the POST method for HTTP.
https://learn.microsoft.com/en-us/powerquery-m/webmethod-post
This is of no help and the only posts I have found so far criticize the poster for not using GET versus POST. I would do this but it is NOT supported by the website I'm using. If someone could just please either point me to a document which explains what I am doing wrong or suggest a solution, I would be grateful.
WebMethod.Post is not a function. It is a constant text value "POST". You can send POST request with either Web.Contents or WebAction.Request function.
A simple example that posts JSON and receives JSON:
let
url = "https://example.com/api/v1.0/some-resource-path",
headers = [#"Content-Type" = "application/json"],
body = Json.FromValue([Foo = 123]),
source = Json.Document(Web.Contents(url, [Headers = headers, Content = body])),
...
Added Nov 14, 19
Request body needs to be a binary type, and included as Content field of the second parameter of Web.Contents function.
You can construct a binary JSON value using Json.FromValue function. Conversely, you can convert a binary JSON value to a corresponding M type using Json.Document function.
Note {} is list type in M language, which is similar to JSON array. [] is record type, which is similar to JSON object.
With that said, your query should be something like this,
let
url_1 = "https://api.[SomeWebSite].com/api/v1.0/search/keyword?apiKey=blah-blah-blah",
Body_1 = Json.FromValue([
SearchByKeywordRequest = [
keyword = "Hex Nuts",
records = 0,
startingRecord = 0,
searchOptions = null,
searchWithYourSignUpLanguage = null
]
]),
headers = [#"Content-Type" = "application/json"],
source = Json.Document(Web.Contents(url_1, [Headers = headers, Content = Body_1])),
...
References:
Web.Contents (https://learn.microsoft.com/en-us/powerquery-m/web-contents)
Json.FromValue (https://learn.microsoft.com/en-us/powerquery-m/json-fromvalue)
Json.Document (https://learn.microsoft.com/en-us/powerquery-m/json-document)

can we pass multiple args in Get Json Value key?

I'm for now fetching one arg value through Get Json Value key
Create Session Get_Inventory_Details ${Base_URL}
${Headers}= Create Dictionary Content-Type=application/json Authorization=bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ1c2VyTmFtZSI6InByaXRpc3dhaW4iLCJpYXQiOjE1Njc1MTQ2NDd9.o3qTPbj2KH6AgHpIf1BLV5nOlGL-fR83wSGAYyuWsBQ
${Response}= Get Request Get_Inventory_Details inventorybyOutlet/7 headers=${Headers}
Log To Console ${Response.status_code}
Log Many ${Response.content}
${actual_response}= Convert To String ${Response.status_code}
Should Be Equal ${actual_response} 200
${quantity}= Get Json Value ${Response.content} /0/currentInventory
Run Keyword If ${quantity}>10 Log Threshold Condition Pass ELSE Log Threshold Condition Fail
What can be the possible way out if I want to pass more than one arg in Get Json Value
I have tried
${quantity}= Get Json Value ${Response.content} /0/currentInventory/0/itemid
Reference to Json Data
[
{
"isInventoryOperationEnable":1,
"itemId":1,
"name":"Afghani Chicken Tikka Biryani (Heavy Eater)",
"posName":"Afghani Chicken Tikka Biryani (Heavy Eater)",
"image":"https://d30mle0t4iy75h.cloudfront.net/websiteV2/images/menuItems/AfghaniTikkaBiryani.jpg",
"typeid":2,
"isCombo":0,
"currentInventory":100,
"categoryId":1,
"categoryname":"Biryani",
"subCategoryId":1,
"sequenceInCategory":1
},
{
"isInventoryOperationEnable":1,
"itemId":3,
"name":"Chicken Tikka Biryani (Heavy Eater)",
"posName":"Chicken Tikka Biryani (Heavy Eater)",
"image":"https://d30mle0t4iy75h.cloudfront.net/websiteV2/images/menuItems/ChickenTikkaBiryani.jpg",
"typeid":2,
"isCombo":0,
"currentInventory":100,
"categoryId":1,
"categoryname":"Biryani",
"subCategoryId":1,
"sequenceInCategory":14
}
]
But RobotFramework throws error as :-
JsonPointerException: Document '' does not support indexing, must be mapping/sequence or support __getitem__
Error Message on RobotFramework
Please can anyone help in finding out a way how to pass more than 1 arg in this case
Thank You
Create Session Get_Inventory_Details ${Base_URL}
${Headers}= Create Dictionary Content-Type=application/json Authorization=bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ1c2VyTmFtZSI6InByaXRpc3dhaW4iLCJpYXQiOjE1Njc1MTQ2NDd9.o3qTPbj2KH6AgHpIf1BLV5nOlGL-fR83wSGAYyuWsBQ
${Response}= Get Request Get_Inventory_Details inventorybyOutlet/7 headers=${Headers}
Log To Console ${Response.status_code}
Log Many ${Response.content}
${actual_response}= Convert To String ${Response.status_code}
Should Be Equal ${actual_response} 200
${response_content_json}= To Json ${Response.content} #To Json is from RequestsLibrary
${quantity}= Set Variable ${response_content_json}[0][currentInventory]
Run Keyword If ${quantity}>10 Log Threshold Condition Pass ELSE Log Threshold Condition Fail
In this case, ${quantity} was 100.
Note two things:
after converting to JSON, variable values can be accessed alike dictionary type variables
To JSON keyword is from RequestsLibrary

Store Data from Postman request in variables to use in tests

Im currently trying to get used to POSTMAN and i was wondering if there is a way to store variables from my request JSON Body via Pre Request in some environment variable so ican resuse it in the tests for response value cheks
This is how my json File might look like
{
"text" : "myText",
"attachments": {
"text": "myText2",
"anotherText" : "myText3"
}
So i want to get all Values, store them in a variable before sending my request, and then test if they match the expected value in my response
(example: myText2 gets mapped to green, myText3 gets mapped to red and so on)
That would make it possible to write one test for several request
Thanks a lot!
You can write the following in your script:
let body = JSON.parse(pm.request.body);
_.forEach(body, (value, key) => pm.environment.set(key, JSON.stringify(value)));
This will set each key and it's associated value as an environment variables.
Note you'll need to JSON.parse the value in the test script before using it for testing.
For eg in your test script you'll need to do something like this:
let attachments = JSON.parse(pm.environment.get('attachments'));
pm.test('All attachments are of correct value', function () {
// ...write your test here using the `attachments` variable
});