Create a cloud scheduler job in CLI with nested json request body - google-cloud-platform

I am running into trouble trying to schedule a Workflow using Google Cloud Scheduler through the Google CLI.
In particular my workflow requires the following request body:
{
"arg1": "some_string",
"arg2": "some_other_string",
"arg3": [
{
"foo": "foo1",
"bar": "bar1"
},
{
"foo": "foo2",
"bar": "bar2"
}
]
}
In a different workflow with request body consisting only of arg1 and arg2 I was able to schedule a cloud function using the double-escaped json string format:
gcloud scheduler jobs create http <NAME> --schedule=<CRON> --uri=<URI> --message-body="{\"argument\": \"{\\\"arg1\\\":\\\"some_string\\\",\\\"arg2\\\":\\\"some_other_string\\\"}\"}" --time-zone="UTC"
With the above request body I am unclear how to do this, I tried setting the message-body as
"{\"argument\": \"{\\\"arg1\\\":\\\"some_string\\\",\\\"arg2\\\":\\\"some_other_string\\\",\\\"arg3\\\":\\\"[{\\\\\"foo\\\\\":\\\\\"foo1\\\\\",\\\\\"bar\\\\\":\\\\\"bar1\\\\\"}]\\\"}\"}"
But it didn't seem to like this and threw an "INVALID ARGUMENT" status. I've also tried a few other variations such as without quotes around the list brackets but haven't had any success.
Apologies for how ugly these strings are. Is anyone aware how to format them correctly, or better yet, a simplified way of entering the request body in the command?
Thanks in advance.
Edit: I have tried using the --message-body-from-file argument as mentioned in the comments by #john-hanley. I found it still required escape quotes to work on my simple case.
body.json
{"argument": "{\"arg1\":\"some_string\",\"arg2\":\"some_other_string\"}"}
When I tried the nested case however with no quotes around the list it did work!
body.json
{"argument": "{\"arg1\":\"some_string\",\"arg2\":\"some_other_string\", \"arg3\": [{\"foo\": \"foo1\", \"bar\": \"bar1\"},{\"foo\":\"foo2\", \"bar\": \"bar2\"}]"}

Solved by incorporating comments by #JohnHanley and unquoting the repeated field
Command:
gcloud scheduler jobs create http <NAME> --schedule=<CRON> --uri=<URI> --message-body-from-file="body.json" --time-zone="UTC"
body.json
{"argument": "{\"arg1\":\"some_string\",\"arg2\":\"some_other_string\", \"arg3\": [{\"foo\": \"foo1\", \"bar\": \"bar1\"},{\"foo\":\"foo2\", \"bar\": \"bar2\"}]"}

Related

Log entries api not retrieving log entries

I am trying to retrieve custom logs for a particular project in google-cloud. I am using this api:
https://logging.googleapis.com/v2/entries:list
as per the example given in this link.
The below is the payload:
{
"filter": "projects/projectA/logs/slow_log",
"resourceNames": [
"projects/projectA"
]
}
There is a custom log based metric called slow_log I created in that projectA, which gathers query logs from cloud-SQL database in that project. I also generated data before calling this api. I am able to see the data in stack-driver console, but unable to get it from the rest call.
Every time I run this api, I only get this response and nothing else:
"nextPageToken": "EAA4suKu3qnLwbtrSg8iDSIDCgEAKgYIgL7q8wVSBwibvMSMvhhglPDiiJzdjt_zAWocCgwI2buKhAYQlvTd2gESCAgLEMPV7ukCGAAgAQ"
Is there anything missing here?
How is it possible to pass time range in this query?
Update
Changed the request as per the comment below as gave the full path of the logs: still only the token is displayed
{
"filter": "projects/projectA/logs/cloudsql.googleapis.com%2Fmysql-slow.log",
"projectIds": [
"projectA"
],
"orderBy": "timestamp desc"
}
Also I give this command from command line:
gcloud logging read logName="projects/projectA/logs/cloudsql.googleapis.com%2Fmysql-slow.log"
then it fetches the logs in command line, so I am not sure what I am missing in the api explorer and postman where I get only nextpage token.
resourceNames, filter and orderBy are mandatory, try like this:
{
"resourceNames": [
"projects/projectA"
],
"filter": "projects/projectA/logs/cloudsql.googleapis.com%2Fmysql-slow.log",
"orderBy": "timestamp desc"
}

AWS API Gateway only allows the first element of the form data and ignores the rest

I have been trying to push data into AWS SQS using AWS API Gateway, the data I send is in the form of application/x-www-form-urlencoded.
And it looks somewhat like this:
fruits[]: apple
fruits[]: mango
fruits[]: banana
season: summer
Now when I poll the data from AWS SQS, I see only fruits[]=apple has been stored and all others are ignored.
This is my current mapping template to push in SQS:
Action=SendMessage&MessageBody=$input.body
Looks like it has multiple $input.body but if that is the case then its kinda impossible to capture random data coming in.
I am new to AWS API Gateway, thanks in advance. :D
After a lot of research and stuff, I was able to decipher this mystery.
the value of $input.body is:
fruits[]=apple&fruits[]=mango&fruits[]=banana&season=summer
Now only MessageBody is pushed in SQS, so according to my template, the resulting query string which was forming, was:
Action=SendMessage&MessageBody=fruits[]=apple&fruits[]=mango&fruits[]=banana&season=summer
only fruits[]=apple is falling under MessageBody and all other becomes separate query objects and hence were ignored.
I just had to tweak the template to:
Action=SendMessage&MessageBody=$util.urlEncode($input.body)
So the resulting query string does not include any more & or = and every thing falls under MessageBody
Edits are welcomed
try this
Request:
POST apigateway/stage/resource?query=test
{
"season": "summer",
"list": [apple,mango,banana]
}
Mapping:
#set($inputRoot = $input.path('$'))
{
"query": "$input.params('query')",
"id": "$inputRoot.season",
"list": $inputRoot.list
}
https://aws.amazon.com/blogs/compute/using-api-gateway-mapping-templates-to-handle-changes-in-your-back-end-apis/
https://docs.aws.amazon.com/apigateway/latest/developerguide/example-photos.html

Elasticsearch Update Doc String Replacement

I have some documents on my Elasticsearch. I want to update my document contents by using String Regexp.
For example, I would like to replace all http words into https words, is it possible ?
Thank You
This should get you off to a start. Check out the "Update by Query" API here. The API allows you to include the update script and search query in the same request body.
Regarding your case, an example might look like this...
POST addresses/_update_by_query
{
"script":
{
"lang": "painless",
"inline": "ctx._source.data.url = ctx._source.data.url.replace('http', 'https')"
},
"query":
{
"query_string":
{
"query": "http://*",
"analyze_wildcard": true
}
}
}
Pretty self explanatory, but script is where we do the update, and query returns the documents to update.
Painless supports regex so you're in luck, look here for some examples, and update the inline value accordingly.

How to parse mixed text and JSON log entries in AWS CloudWatch for Log Metric Filter

I am trying to parse log entries which are a mix of text and JSON. The first line is text representation and the next lines are JSON payload of the event. One of the possible examples are:
2016-07-24T21:08:07.888Z [INFO] Command completed lessonrecords-create
{
"key": "lessonrecords-create",
"correlationId": "c1c07081-3f67-4ab3-a5e2-1b3a16c87961",
"result": {
"id": "9457ce88-4e6f-4084-bbea-14fff78ce5b6",
"status": "NA",
"private": false,
"note": "Test note",
"time": "2016-02-01T01:24:00.000Z",
"updatedAt": "2016-07-24T21:08:07.879Z",
"createdAt": "2016-07-24T21:08:07.879Z",
"authorId": null,
"lessonId": null,
"groupId": null
}
}
For these records I try to define Log Metric Filter to a) match records b) select data or dimensions if possible.
According to the AWS docs JSON pattern should look like this:
{ $.key = "lessonrecords-create" }
however, it does not match anything. My guess is that because of mix text and JSON in a single log entry.
So, the questions are:
1. Is it possible to define a pattern that will match this log format?
2. Is it possible to extract dimensions, values from such a log format?
3. Help me with a pattern to do this.
If you set up the metric filter in the way that you have defined, the test will not register any matches (I have also had this issue), however when you deploy the metric filter it will still register matches (at least mine did). Just keep in mind that there is no way (as far as I am aware) to run this metric filter BACKWARDS (ie. it will only capture data from when it is created). [If you're trying to get stats on past data, you're better off using log insight queries]
I am currently experimenting with different parse statements to try and extract data (its also a mix of JSON and text), this thread MAY help you (it didn't for me) Amazon Cloudwatch Logs Insights with JSON fields .
UPDATE!
I have found a way to parse the text but its a little bit clunky. If you export your cloudwatch logs using a lamda function to SumoLogic, their search tool allows for MUCH better log manipulation and lets you parse JSON fields (if you treat the entire entry as text). SumoLogic is also really helpful because you can just extract your search results as a CSV. For my purposes, I parse the entire log message in SumoLogic, extract all the logs as a CSV and then I used regex in Python to filter through and extract the values I need.
Let's say you have the following log
2021-09-29 15:51:18,624 [main] DEBUG com.company.app.SparkResources - AUDIT : {"user":"Raspoutine","method":"GET","pathInfo":"/analysis/123"}
you can parse it like this to be able to handle the part after "AUDIT : " as a JSON
fields #message
| parse #message "* [*] * * - AUDIT : *" as timestamp, thread, logLevel, clazz, msg
| filter ispresent(msg)
| filter method = "GET" # You can use fields which are contained in the JSON String of 'msg' field. Do not use 'msg.method' but directly 'method'
The fields contained in your isolated / parsed JSON field are automatically added as fields usable in the query
You can use CloudWatch Events for such purpose(aka Subscription Filters), what you will need to do is define a cloudwatch Rule which uses an expression statement to match your logs.
Here, I will let you do all the reading:
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html
https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/Create-CloudWatch-Events-Scheduled-Rule.html
:)
Split the message into 3 fields and the 3rd field will be a valid json . I think in your case it would be
fields #timestamp, #message
| parse #message '[] * {"*"}' as field1, field2, field3
| limit 50
field3 is the valid json.
[INFO} will be the first field.
You can search JSON string representation, which is not as powerful.
For your example,
instead of { $.key = "lessonrecords-create" }
try "\"key\":\"lessonrecords-create\"".
This filter is not semantically identical to your requirement, though. It will also give events where key is not at the root of json.
you can use fluentd agent to send logs to Cloudwatch. Create custom grok pattern based on your metric filter.
Steps:
Install fluentd agent in your server
Install fluent-plugin-cloudwatch-logs plugin and fluent-plugin-grok-parser plugin
write your custom grok pattern based on your log format
please refer this blog for more information

WSO2 CEP siddhi Filter issue

I am trying to use the siddhi query langage but it seems I am misusing it.
I have some events with the following streamdef :
{ 'name':'eu.ima.stat.events', 'version':'1.1.0', 'nickName': 'Flux event Information', 'description': 'Details of Analytics Statistics', 'metaData':[ {name:'HostIP','type':'STRING'} ], 'correlationData':[ {name:'ProcessType','type':'STRING'}, {name:'Flux','type':'STRING'}, {name:'ReferenceId','type':'STRING'} ], 'payloadData':[ {'name':'Timestamp','type':'STRING'}, {'name':'EventCode','type':'STRING'}, {'name':'Type','type':'STRING'}, {'name':'EventInfo','type':'STRING'} ]}
I am just trying to filter events with the same processus value and the same flux value using a query like this one :
from myEventStream[processus == 'SomeName' and flux == 'someOtherName' ]
insert into someStream
processus, flux, timestamp
Whenever I try this, no output is generated. When I get rid of the filter
from myEventStream
insert into someStream
processus, flux, timestamp
all my events are ther in the output.
What's wrong with my query ?
I can see some spell mistakes in your query... In the filter you have used a variable name called "processus" which is not in the event stream. That is why this query does not give any output. When you are creating a bucket in WSO2 CEP, make sure that the bucket is deployed correctly in the CEP server and check in the management console.(CEP BUCKETS --> List).
On your situation. bucket will not be deployed because of the wrong configuration and also there will be error messages printed in the terminal where CEP server runs. After correcting this mistake your query will run perfectly without any issue...
Regards,
Mohan
Considering Mohan's answer,rename 'ProcessType' or change your query like this
from myEventStream[ ProcessType == 'SomeName' and flux == 'someOtherName' ]
insert into someStream
ProcessType, flux, timestamp