How to use AWS IoT Rule Query Statement to get temperature and humidity from the following MQTT payload to 'example' topic?
[
{
"timestamp": "2019-08-26T14:21:46Z",
"type": "Gateway",
"mac": "XXYYZZXXYYZZ",
"gatewayFree": 96,
"gatewayLoad": 0.26
},
{
"timestamp": "2019-08-26T14:21:46Z",
"type": "S1",
"mac": "XXYYZZXXYYXX",
"bleName": "",
"rssi": -53,
"battery": 100,
"temperature": 0.69,
"humidity": 37.28
},
{
"timestamp": "2019-08-26T14:21:46Z",
"type": "iBeacon",
"mac": "XXYYZZXXYYYY",
"bleName": "",
"ibeaconUuid": "ABCDEFGHIJKLMNOPQRSTUVWXYZ",
"ibeaconMajor": 0,
"ibeaconMinor": 0,
"rssi": -64,
"ibeaconTxPower": -59,
"battery": 0
}
]
So far, I cannot find it on AWS IoT SQL Reference.
Try this, it's working!
SELECT (SELECT temperature, humidity from input) as output FROM 'iot/topic'
I have modified the input JSON to have a key -- see below.
Here is the modified input:
{
"input": [ ... your json array elements ... ]
}
Here is the output:
{
"output": [
{},
{
"temperature": 0.69,
"humidity": 37.28
},
{}
]
}
cheers,
ram
Also try this one...
{
"sql": "SELECT * FROM 'iot/example' WHERE mac = 'XXYYZZXXYYXX'",
...
}
Cheers!
Related
Can anyone please guide me steps to create multiple triggers types one with conditional and other with scheduled trigger type in single workflow
So far I have used create_trigger function . But above requirement not sure how to address.
Can any one help here please.
I have tried with below syntax didn't work
response = client.create_trigger(
Name='two_triggers',
WorkflowName='wf_With_two_tirggers',
Type='SCHEDULED',
Schedule='cron(0 12 * * ? *)',
Actions=[
{
'JobName': 'abc_dev',
'Arguments': {
'string': 'string'
},
'Timeout': 123,
'SecurityConfiguration': 'string',
'NotificationProperty': {
'NotifyDelayAfter': 123
},
'Trigger': 'string'
},
],
Type='CONDITIONAL',
Predicate={
'Logical': 'ANY',
'Conditions': [
{
'LogicalOperator': 'EQUALS',
'JobName': 'def_dev',
'State': 'SUCCEEDED'
},
]
},
Actions=[
{
'JobName': 'ghi_dev',
'Arguments': {
'string': 'string'
},
'Timeout': 123,
'SecurityConfiguration': 'string',
'NotificationProperty': {
'NotifyDelayAfter': 123
},
'CrawlerName': 'string'
},
],
Description='string',
StartOnCreation=True,
Tags={
'string': 'string'
}
)
Below is the design workflow struggling to write code for. Tried with above code for below design using boto3 didn't work
Yes I figured out on an answer. Below is the code for design given in question
import boto3
import os
import logging
glue = boto3.client(service_name="glue", region_name='us-east-1')
response = glue.create_workflow(
Name="dual_trigger_wf")
response1 = glue.create_trigger(
Name="trigger_one_to_many",
WorkflowName="dual_trigger_wf",
Type="SCHEDULED",
Schedule="cron(0 8 * * ? *)",
Actions=[
{
"JobName": "abc",
"Arguments": {"string": "string"},
"Timeout": 123,
"SecurityConfiguration": "string",
"NotificationProperty": {"NotifyDelayAfter": 123},
},
{
"JobName": "def",
"Arguments": {"string": "string"},
"Timeout": 123,
"SecurityConfiguration": "string",
"NotificationProperty": {"NotifyDelayAfter": 123},
},
],
Description="string",
StartOnCreation=False,
)
response2 = glue.create_trigger(
Name="trigger_many_to_one",
WorkflowName="dual_trigger_wf",
Type="CONDITIONAL",
Predicate={
"Logical": "AND",
"Conditions": [
{
"LogicalOperator": "EQUALS",
"JobName": "abc",
"State": "SUCCEEDED",
},
{
"LogicalOperator": "EQUALS",
"JobName": "def",
"State": "SUCCEEDED",
},
],
},
Actions=[
{
"JobName": "ghi",
"Arguments": {"string": "string"},
"Timeout": 123,
"SecurityConfiguration": "string",
"NotificationProperty": {"NotifyDelayAfter": 123},
}
],
Description="string",
StartOnCreation=False,
)
haii, i have the awscli result describing the target group as json format
{
"TargetHealthDescriptions": [
{
"Target": {
"Id": "1.1.1.1",
"Port": 123,
"AvailabilityZone": "ap-south-1"
},
"HealthCheckPort": "123",
"TargetHealth": {
"State": "healthy"
}
},
{
"Target": {
"Id": "2.2.2.2",
"Port": 123,
"AvailabilityZone": "ap-south-1"
},
"HealthCheckPort": "123",
"TargetHealth": {
"State": "healthy"
}
}
]
}
Im trying to make an awscli script to get a result like this
[
{
"Id": "1.1.1.1",
"Port": 123,
"Health": null
},
{
"Id": "2.2.2.2",
"Port": 123,
"Health": null
}
]
I've tried several query methods but I have problems getting a null value for health, its any errors in the query ?
ex query
--query 'TargetHealthDescriptions[*].Target.{Id:Id, Port:Port, Health:TargetHealth.{state:State}}' --output json
Try the below:
--query 'TargetHealthDescriptions[*].{Id:Target.Id,Port:Target.Port,Health:TargetHealth.State}'
I am reading monitoring data through Google Timeseries api. The api is working correctly and if give alignment period=3600s it gives me the values for that time series between start and end time for any metric type.
I am calling it through Python like this:
service.projects().timeSeries().list(
name=api_args["project_name"],
filter=api_args["metric_filter"],
aggregation_alignmentPeriod=api_args["aggregation_alignment_period"],
# aggregation_crossSeriesReducer=api_args["crossSeriesReducer"],
aggregation_perSeriesAligner=api_args["perSeriesAligner"],
aggregation_groupByFields=api_args["group_by"],
interval_endTime=api_args["end_time_str"],
interval_startTime=api_args["start_time_str"],
pageSize=config.PAGE_SIZE,
pageToken=api_args["nextPageToken"]
).execute()
and in Postman:
https://monitoring.googleapis.com/v3/projects/my-project/timeSeries?pageSize=500&interval.startTime=2020-07-04T16%3A39%3A37.230000Z&aggregation.alignmentPeriod=3600s&aggregation.perSeriesAligner=ALIGN_SUM&filter=metric.type%3D%22compute.googleapis.com%2Finstance%2Fnetwork%2Freceived_bytes_count%22+&pageToken=&interval.endTime=2020-07-04T17%3A30%3A01.497Z&alt=json&aggregation.groupByFields=metric.labels.key
I face an issue here:
{
"metric": {
"labels": {
"instance_name": "insta-demo1",
"loadbalanced": "false"
},
"type": "compute.googleapis.com/instance/network/received_bytes_count"
},
"resource": {
"type": "gce_instance",
"labels": {
"instance_id": "1234343552",
"zone": "us-central1-f",
"project_id": "my-project"
}
},
"metricKind": "DELTA",
"valueType": "INT64",
"points": [
{
"interval": {
"startTime": "2020-07-04T16:30:01.497Z",
"endTime": "2020-07-04T17:30:01.497Z"
},
"value": {
"int64Value": "6720271"
}
}
]
},
{
"metric": {
"labels": {
"loadbalanced": "true",
"instance_name": "insta-demo2"
},
"type": "compute.googleapis.com/instance/network/received_bytes_count"
},
"resource": {
"type": "gce_instance",
"labels": {
"instance_id": "1234566343",
"project_id": "my-project",
"zone": "us-central1-f"
}
},
"metricKind": "DELTA",
"valueType": "INT64",
"points": [
{
"interval": {
"startTime": "2020-07-04T16:30:01.497Z",
"endTime": "2020-07-04T17:30:01.497Z"
},
"value": {
"int64Value": "579187"
}
}
]
}
],
"unit": "By". //This "By" is the value which is causing problem,
I am getting this value like "unit": "By" or "unit":"ms" or something like that at the end, Also if I don't find any data for a range I'm getting this value, as I am evaluating this response in Python I am getting key error as there is not key called "unit"
logMessage: "Key Error: ' '"
severity: "ERROR"
As the response is empty I am getting the single key called "unit". Also at the end of any response I am getting this "unit":"ms" or "unit":"by" - is there any way to prevent that unit value coming in the response?
I am new to Google Cloud APIs and Python. What can I try next?
The "unit" field expresses the kind of resource the metric is counting. For bytes, it is "By". Read this. I understand it is always returned, so there is no way of not receiving it; I recommend you to adapt your code to correctly deal with its appearance in the responses.
I'm working on ingesting metrics from Lambda into our centralized logging system. Our first idea is too costly so I'm trying to figure out if there is a way to lower the cost (instead of ingesting 3 metrics from 200 lambdas every 60s).
I've been messing around with MetricMath and have pretty much figured out what I want to do. I'd run this as a k8s cron job like thing and variabilize the start and end time.
How would this be charged? Is it the number of metrics used to perform the math or the number of values that I output?
i.e. m1 and m2 are pulling Errors and Invocations from 200 lambdas. To pull each of these individually would be 400 metrics.
In this method, would it only be 1, 3, or 401?
{
"MetricDataQueries": [
{
"Id": "m1",
"MetricStat": {
"Metric": {
"Namespace": "AWS/Lambda",
"MetricName": "Errors"
},
"Period": 300,
"Stat": "Sum",
"Unit": "Count"
},
"ReturnData": false
},
{
"Id": "m2",
"MetricStat": {
"Metric": {
"Namespace": "AWS/Lambda",
"MetricName": "Invocations"
},
"Period": 300,
"Stat": "Sum",
"Unit": "Count"
},
"ReturnData": false
},
{
"Id": "e1",
"Expression": "m1 / m2",
"Label": "ErrorRate"
}
],
"StartTime": "2020-02-25T02:00:0000",
"EndTime": "2020-02-26T02:05:0000"
}
Output:
{
"Messages": [],
"MetricDataResults": [
{
"Label": "ErrorRate",
"StatusCode": "Complete",
"Values": [
0.0045127626568890146
],
"Id": "e1",
"Timestamps": [
"2020-02-26T19:00:00Z"
]
}
]
}
Example 2:
Same principle. This is pulling the invocations by of each function by FunctionName. It then sorts them and outputs the most invoked. Any idea how many metrics this would be?
{
"MetricDataQueries": [
{
"Id": "e2",
"Expression": "SEARCH(' {AWS/Lambda,FunctionName} MetricName=`Invocations` ', 'Sum', 60)",
"ReturnData" : false
},
{
"Id": "e3",
"Expression": "SORT(e2, SUM, DESC, 1)"
}
],
"StartTime": "2020-02-26T12:00:0000",
"EndTime": "2020-02-26T12:01:0000"
}
Same question. 1 or 201 metrics?
Output:
{
"MetricDataResults": [
{
"Id": "e3",
"Timestamps": [
"2020-02-26T12:00:00Z"
],
"Label": "1 - FunctionName",
"Values": [
91.0
],
"StatusCode": "Complete"
}
],
"Messages": []
}
Billing is on metrics requested: https://aws.amazon.com/cloudwatch/pricing/
In the first example, you're requesting only 2 metrics. These metrics are aggregates of per lambda function metrics, but as far you're concerned, that's only 2 metrics and you will be billed for 2. You're not billed for the metric math, only for metrics you request.
In the second example, the number of metrics the search returns is the amount you will be billed for, 200 in your case.
I have a couchdb record structure which looks like this
[
{
"app_version": 2,
"platform": "android",
"session": {
"timestamp": "2014-08-20T00:00:00.000Z",
"session_id": "TOnNIhCNQ31LlkpEPQ7XnN1D",
"ip": "202.150.213.66",
"location": "1.30324,103.5498"
}
},
{
"app_version": 2,
"platform": "android",
"session": {
"timestamp": "2014-08-21T00:00:00.000Z",
"session_id": "TOnNIhCNQ31LlkpEPQ7XnN1D",
"ip": "202.150.213.66",
"location": "1.30324,103.5498"
}
}
{
"app_version": 2,
"platform": "ios",
"session": {
"timestamp": "2014-08-21T00:00:00.000Z",
"session_id": "TOnNIhCNQ31LlkpEPQ7XnN1D",
"ip": "202.150.213.66",
"location": "1.30324,103.5498"
}
},
{
"app_version": 1,
"platform": "ios",
"session": {
"timestamp": "2014-08-21T00:00:00.000Z",
"session_id": "TOnNIhCNQ31LlkpEPQ7XnN1D",
"ip": "202.150.213.66",
"location": "1.30324,103.5498"
}
}
]
I need to query all the records which happened between a a given number of dates and a app_version number, and I want to get the total of each by the platform.
So I wrote a map-reduce function like this;
"total": {
"map": "function(doc) {
date = doc.session.timestamp.split("T")[0];
emit([date, doc.app_version,doc.platform], 1);
}",
"reduce": "_count"
}
This gives me the output properly by grouping the records into dates.
["2014-08-20", 2, "android"] 2
["2014-08-20", 2, "ios"] 1
["2014-08-21", 2, "android"] 1
["2014-08-21", 2, "ios"] 1
But the problem comes when I try to query them using the start_key and end_key (to query by the date range)
Im sending the GET request as follows;
http://localhost/dummy_db_new/_design/views/_view/total?
start_key=["2014-08-20",2,WHAT_TO_PUT_HERE]
&end_key=["2014-08-20",2,WHAT_TO_PUT_HERE]
&group=true
I need to know what to put at the above places for it to have any platform(a string).
Oh I was able to find an answer.
The Answer was to use a wildcard. So basically I sent the request with a wildcard which will accept any platform type
http://localhost/dummy_db_new/_design/views/_view/total?
start_key=["2014-08-20",2,0]
&end_key=["2014-08-20",2,{}]
&group=true
{} means javascript object, so it will accept any JS object.