Monitoring api in Google gives "By" as response - python-2.7

I am reading monitoring data through Google Timeseries api. The api is working correctly and if give alignment period=3600s it gives me the values for that time series between start and end time for any metric type.
I am calling it through Python like this:
service.projects().timeSeries().list(
name=api_args["project_name"],
filter=api_args["metric_filter"],
aggregation_alignmentPeriod=api_args["aggregation_alignment_period"],
# aggregation_crossSeriesReducer=api_args["crossSeriesReducer"],
aggregation_perSeriesAligner=api_args["perSeriesAligner"],
aggregation_groupByFields=api_args["group_by"],
interval_endTime=api_args["end_time_str"],
interval_startTime=api_args["start_time_str"],
pageSize=config.PAGE_SIZE,
pageToken=api_args["nextPageToken"]
).execute()
and in Postman:
https://monitoring.googleapis.com/v3/projects/my-project/timeSeries?pageSize=500&interval.startTime=2020-07-04T16%3A39%3A37.230000Z&aggregation.alignmentPeriod=3600s&aggregation.perSeriesAligner=ALIGN_SUM&filter=metric.type%3D%22compute.googleapis.com%2Finstance%2Fnetwork%2Freceived_bytes_count%22+&pageToken=&interval.endTime=2020-07-04T17%3A30%3A01.497Z&alt=json&aggregation.groupByFields=metric.labels.key
I face an issue here:
{
"metric": {
"labels": {
"instance_name": "insta-demo1",
"loadbalanced": "false"
},
"type": "compute.googleapis.com/instance/network/received_bytes_count"
},
"resource": {
"type": "gce_instance",
"labels": {
"instance_id": "1234343552",
"zone": "us-central1-f",
"project_id": "my-project"
}
},
"metricKind": "DELTA",
"valueType": "INT64",
"points": [
{
"interval": {
"startTime": "2020-07-04T16:30:01.497Z",
"endTime": "2020-07-04T17:30:01.497Z"
},
"value": {
"int64Value": "6720271"
}
}
]
},
{
"metric": {
"labels": {
"loadbalanced": "true",
"instance_name": "insta-demo2"
},
"type": "compute.googleapis.com/instance/network/received_bytes_count"
},
"resource": {
"type": "gce_instance",
"labels": {
"instance_id": "1234566343",
"project_id": "my-project",
"zone": "us-central1-f"
}
},
"metricKind": "DELTA",
"valueType": "INT64",
"points": [
{
"interval": {
"startTime": "2020-07-04T16:30:01.497Z",
"endTime": "2020-07-04T17:30:01.497Z"
},
"value": {
"int64Value": "579187"
}
}
]
}
],
"unit": "By". //This "By" is the value which is causing problem,
I am getting this value like "unit": "By" or "unit":"ms" or something like that at the end, Also if I don't find any data for a range I'm getting this value, as I am evaluating this response in Python I am getting key error as there is not key called "unit"
logMessage: "Key Error: ' '"
severity: "ERROR"
As the response is empty I am getting the single key called "unit". Also at the end of any response I am getting this "unit":"ms" or "unit":"by" - is there any way to prevent that unit value coming in the response?
I am new to Google Cloud APIs and Python. What can I try next?

The "unit" field expresses the kind of resource the metric is counting. For bytes, it is "By". Read this. I understand it is always returned, so there is no way of not receiving it; I recommend you to adapt your code to correctly deal with its appearance in the responses.

Related

Twilio Studio RegEx in "Split Based On..." Widget

I'm still in my trial phase with Twilio and working on putting together a "flow" in Studio to handle my use-case properly. So far, all of the basic, plain-text steps in my flow seem to be working exactly as expected, but I'm having problems figuring out how to handle incoming media/attachments the way I want. I readily admit and acknowledge that what I've put together may not be the most effective solution, but it's what I've come up with based on the research I've done so far.
First, here is the basic breakdown of the desired steps in my flow:
I have a separate application that connects to the Twilio REST API to initiate the flow's execution. It passes a couple of parameters into the flow's flow.data namespace for inclusion in various messages sent to the end-user as a part of the flow.
Private Async Sub StartFlow(ByVal Recipient As String)
Try
Dim RefNum As String = GenerateReferenceNumber(8)
Dim NoticeParameters As New Dictionary(Of String, Object) From {
{"ref_num", RefNum},
{"contact_num", "(800) 555-1234"},
{"client_name", "Caliente Client"}
}
TwilioClient.Init(SID, Token)
Dim FlowExecution = Await ExecutionResource.CreateAsync(parameters:=NoticeParameters,
[to]:=New PhoneNumber(Recipient),
from:=MyTwilioNumber,
pathFlowSid:=NoticeFlowSID)
Catch ex As Exception
End Try
End Sub
The first step in the flow's execution is to initialize a variable in the flow.variables namespace named current_response with a value of EMPTY.
The flow's next step is to send our initial SMS contact to the end-user via a Send & Wait For Reply widget called "Notice".
When a reply is received from the user, the flow.variables.current_response value is updated by a Set Variables widget:
If the value of widgets.Notice.inbound.MediaUrl0 is not NULL, the flow.variables.current_response value is set to the URL value from that property.
If the value of widgets.Notice.inbound.MediaUrl0 is NULL, the flow.variables.current_response value is set to the value of the widgets.Notice.inbound.Body property instead.
After setting the current_response variable's value, that variable is passed on to a Split Based On... widget (named "Response_Received") that tests the flow.variables.current_response with the following Regular Expression to determine whether or not the value is a URL:
[(http(s)?):\/\/(www\.)?a-zA-Z0-9#:%._\+~#=]{2,256}\.[a-z]{2,6}\b([-a-zA-Z0-9#:%_\+.~#?&//=]*)
If the value is not a URL, the flow should pass execution on to the next Split Based On... widget (named "Other_Response") to handle certain plain-text replies with a pre-defined set of responses.
If the value is a URL, it should pass execution to a Make HTTP Request widget that will POST to my MVC controller to log the incoming file.
Depending on the end-user's reply, the Other_Response widget will then pass execution on to another Send & Wait For Reply widget that can update the flow.variables.current_response property with that widget's inbound.MediaUrl0 or inbound.Body via another Set Variables widget if they continue the conversation.
After updating the current_response variable, flow execution returns to the "Response_Received" widget with the new value (see #5).
The idea is that, if the user initially responds with something like HELP, I want to be able to send them a message with additional details about the reason we reached out to them in the first place, then allow them to send us the specific information (file) we're requesting while still retaining the values I populated into the flow from my in-house application - most specifically, a unique reference number I've generated for the contact. Even if it takes them a couple of tries, as long as the execution is valid, I want any reply within that execution to have the capability of processing an incoming file based on those same values.
In my tests, I've been able to get all of the simple, plain-text responses to work as I've designed:
I reply to the flow's initial message with one of the pre-defined messages and the flow returns the correct response and waits.
I reply to that message with another of the pre-defined messages and the flow again returns the appropriate response and waits.
I reply to that message with a plain-text message that isn't defined as a valid response and the flow correctly returns a message stating, "I'm sorry. I didn't recognize your response. Please try again."
However, if I reply to the flow's initial message with a file as I expect to have our end-users doing (I've used PDF and PNG files in my testing so far), I still get the message stating, "I'm sorry. I didn't recognize your response. Please try again." This message is not defined in the path from the split designated for replies with a URL.
Based on the results, I can only assume that there's something "wrong" with the RegEx logic that's not actually matching the URL as it should. I've tested the pattern on RegExr and it "works", identifying the URL as a match. For testing purposes, I've even included the value of the flow.variables.current_response in the message sent back to the user and it shows that the value is, in fact, the Twilio API URL of the media.
For reference, here is a slightly redacted version of the JSON definition for this flow. There are some "silly" nodes in here I've been using for testing my logic. As stated above, the only thing that seems to be failing is the RegEx matching of the MediaUrl0 value to send it to my MVC controller.
{
"description": "A New Flow",
"states": [
{
"name": "Trigger",
"type": "trigger",
"transitions": [
{
"event": "incomingMessage"
},
{
"next": "Incoming_Call",
"event": "incomingCall"
},
{
"event": "incomingConversationMessage"
},
{
"next": "Initialize_Reply",
"event": "incomingRequest"
},
{
"event": "incomingParent"
}
],
"properties": {
"offset": {
"x": 0,
"y": 0
}
}
},
{
"name": "Notice",
"type": "send-and-wait-for-reply",
"transitions": [
{
"next": "Notice_Response",
"event": "incomingMessage"
},
{
"event": "timeout"
},
{
"event": "deliveryFailure"
}
],
"properties": {
"offset": {
"x": 700,
"y": 270
},
"service": "{{trigger.message.InstanceSid}}",
"channel": "{{trigger.message.ChannelSid}}",
"from": "{{flow.channel.address}}",
"body": "Information Request from {{flow.data.client_name}}: We need some additional information from you about your recent purchase. For more information about this request, reply HELP. To stop receiving these messages from {{flow.data.client_name}}, reply STOP. Reference #{{flow.data.ref_num}}",
"media_url": "",
"timeout": "2419200"
}
},
{
"name": "Response_Received",
"type": "split-based-on",
"transitions": [
{
"next": "Other_Response",
"event": "noMatch"
},
{
"next": "Log_Document",
"event": "match",
"conditions": [
{
"friendly_name": "Document Received",
"arguments": [
"{{flow.variables.current_response}}"
],
"type": "regex",
"value": "[(http(s)?):\\/\\/(www\\.)?a-zA-Z0-9#:%._\\+~#=]{2,256}\\.[a-z]{2,6}\\b([-a-zA-Z0-9#:%_\\+.~#?&//=]*)"
}
]
}
],
"properties": {
"input": "{{flow.variables.current_response}}",
"offset": {
"x": 370,
"y": 800
}
}
},
{
"name": "Other_Response",
"type": "split-based-on",
"transitions": [
{
"next": "Unrecognized_Response",
"event": "noMatch"
},
{
"event": "match",
"conditions": [
{
"friendly_name": "More Info",
"arguments": [
"{{flow.variables.current_response}}"
],
"type": "matches_any_of",
"value": "HELP,INFO,MORE"
}
]
},
{
"next": "Log_Opt_Out",
"event": "match",
"conditions": [
{
"friendly_name": "Unsubscribe",
"arguments": [
"{{flow.variables.current_response}}"
],
"type": "matches_any_of",
"value": "STOP,QUIT,REMOVE,DELETE,CEASE,CANCEL,UNSUBSCRIBE,STOPALL,END"
}
]
},
{
"next": "Jedi_Code",
"event": "match",
"conditions": [
{
"friendly_name": "Jedi Code",
"arguments": [
"{{flow.variables.current_response}}"
],
"type": "equal_to",
"value": "JEDI"
}
]
},
{
"next": "Sith_Code",
"event": "match",
"conditions": [
{
"friendly_name": "Sith Code",
"arguments": [
"{{flow.variables.current_response}}"
],
"type": "equal_to",
"value": "SITH"
}
]
}
],
"properties": {
"input": "{{flow.variables.current_response}}",
"offset": {
"x": -130,
"y": 1020
}
}
},
{
"name": "Document_Confirmation",
"type": "send-message",
"transitions": [
{
"event": "sent"
},
{
"event": "failed"
}
],
"properties": {
"offset": {
"x": 620,
"y": 1370
},
"service": "{{trigger.message.InstanceSid}}",
"channel": "{{trigger.message.ChannelSid}}",
"from": "{{flow.channel.address}}",
"to": "{{contact.channel.address}}",
"body": "Thank you. We will update our records as soon as possible. If there are any further questions or concerns, one of our representatives will contact you to follow up.\n\nReference #{{flow.data.ref_num}}"
}
},
{
"name": "Log_Document",
"type": "make-http-request",
"transitions": [
{
"next": "Document_Confirmation",
"event": "success"
},
{
"next": "Unrecognized_Document",
"event": "failed"
}
],
"properties": {
"offset": {
"x": 880,
"y": 1020
},
"method": "POST",
"content_type": "application/json;charset=utf-8",
"url": "https://my.domain.com/sms"
}
},
{
"name": "Jedi_Code",
"type": "send-and-wait-for-reply",
"transitions": [
{
"next": "Jedi_Response",
"event": "incomingMessage"
},
{
"event": "timeout"
},
{
"event": "deliveryFailure"
}
],
"properties": {
"offset": {
"x": -260,
"y": 1340
},
"service": "{{trigger.message.InstanceSid}}",
"channel": "{{trigger.message.ChannelSid}}",
"from": "{{flow.channel.address}}",
"body": "There is no emotion, there is peace.\nThere is no ignorance, there is knowledge.\nThere is no passion, there is serenity.\nThere is no chaos, there is harmony.\nThere is no death, there is the Force.\nReference #{{flow.data.ref_num}}\nContact #{{flow.data.contact_num}}",
"media_url": "",
"timeout": "43200"
}
},
{
"name": "Sith_Code",
"type": "send-and-wait-for-reply",
"transitions": [
{
"next": "Sith_Response",
"event": "incomingMessage"
},
{
"event": "timeout"
},
{
"event": "deliveryFailure"
}
],
"properties": {
"offset": {
"x": 90,
"y": 1370
},
"service": "{{trigger.message.InstanceSid}}",
"channel": "{{trigger.message.ChannelSid}}",
"from": "{{flow.channel.address}}",
"body": "Peace is a lie. There is only Passion.\nThrough Passion, I gain Strength.\nThrough Strength, I gain Power.\nThrough Power, I gain Victory.\nThrough Victory my chains are Broken.\nThe Force shall free me.\nReference #{{flow.data.ref_num}}",
"media_url": "",
"timeout": "43200"
}
},
{
"name": "Log_Opt_Out",
"type": "make-http-request",
"transitions": [
{
"next": "Opt_Out_Success",
"event": "success"
},
{
"next": "Opt_Out_Fail",
"event": "failed"
}
],
"properties": {
"offset": {
"x": -660,
"y": 1310
},
"method": "POST",
"content_type": "application/x-www-form-urlencoded;charset=utf-8",
"url": "https://my.domain.com/sms"
}
},
{
"name": "Opt_Out_Success",
"type": "send-message",
"transitions": [
{
"event": "sent"
},
{
"event": "failed"
}
],
"properties": {
"offset": {
"x": -1060,
"y": 1320
},
"service": "{{trigger.message.InstanceSid}}",
"channel": "{{trigger.message.ChannelSid}}",
"from": "{{flow.channel.address}}",
"to": "{{contact.channel.address}}",
"body": "Thank you. Your preferences have been updated and you will no longer receive insurance notifications from {{flow.data.client_name}}. \nYou can re-enable these notifications later by texting OPT-IN to this number."
}
},
{
"name": "Opt_Out_Fail",
"type": "send-and-wait-for-reply",
"transitions": [
{
"next": "Opt_Out_Response",
"event": "incomingMessage"
},
{
"event": "timeout"
},
{
"event": "deliveryFailure"
}
],
"properties": {
"offset": {
"x": -650,
"y": 1580
},
"service": "{{trigger.message.InstanceSid}}",
"channel": "{{trigger.message.ChannelSid}}",
"from": "{{flow.channel.address}}",
"body": "We're sorry. There was a problem registering your opt-out request. You can try again or contact {{flow.data.client_name}} to update your preferences.",
"timeout": "43200"
}
},
{
"name": "Unrecognized_Response",
"type": "send-and-wait-for-reply",
"transitions": [
{
"next": "Unknown_Response",
"event": "incomingMessage"
},
{
"event": "timeout"
},
{
"event": "deliveryFailure"
}
],
"properties": {
"offset": {
"x": -490,
"y": 1030
},
"service": "{{trigger.message.InstanceSid}}",
"channel": "{{trigger.message.ChannelSid}}",
"from": "{{flow.channel.address}}",
"body": "I'm sorry. I didn't recognize your response. Please try again.\n\nResponse: {{flow.variables.current_response}}",
"timeout": "43200"
}
},
{
"name": "Initialize_Reply",
"type": "set-variables",
"transitions": [
{
"next": "Notice",
"event": "next"
}
],
"properties": {
"variables": [
{
"value": "EMPTY",
"key": "current_response"
}
],
"offset": {
"x": 390,
"y": 270
}
}
},
{
"name": "Notice_Response",
"type": "set-variables",
"transitions": [
{
"next": "Response_Received",
"event": "next"
}
],
"properties": {
"variables": [
{
"value": "{% if widgets.Notice.inbound.MediaUrl0 != null %}\n {{widgets.Notice.inbound.MediaUrl0}} \n{% else %}\n{{widgets.Notice.inbound.Body}} \n{% endif %}",
"key": "current_response"
}
],
"offset": {
"x": 390,
"y": 550
}
}
},
{
"name": "Jedi_Response",
"type": "set-variables",
"transitions": [
{
"next": "Response_Received",
"event": "next"
}
],
"properties": {
"variables": [
{
"value": "{% if widgets.Jedi_Code.inbound.MediaUrl0 != null %} {{widgets.Jedi_Code.inbound.MediaUrl0}} {% else %} {{widgets.Jedi_Code.inbound.Body}} {% endif %}",
"key": "current_response"
}
],
"offset": {
"x": -280,
"y": 1580
}
}
},
{
"name": "Sith_Response",
"type": "set-variables",
"transitions": [
{
"next": "Response_Received",
"event": "next"
}
],
"properties": {
"variables": [
{
"value": "{% if widgets.Sith_Code.inbound.MediaUrl0 != null %} {{widgets.Sith_Code.inbound.MediaUrl0}} {% else %} {{widgets.Sith_Code.inbound.Body}} {% endif %}",
"key": "current_response"
}
],
"offset": {
"x": 120,
"y": 1600
}
}
},
{
"name": "Unknown_Response",
"type": "set-variables",
"transitions": [
{
"next": "Response_Received",
"event": "next"
}
],
"properties": {
"variables": [
{
"value": "{% if widgets.Unrecognized_Response.inbound.MediaUrl0 != null %} {{widgets.Unrecognized_Response.inbound.MediaUrl0}} {% else %} {{widgets.Unrecognized_Response.inbound.Body}} {% endif %}",
"key": "current_response"
}
],
"offset": {
"x": -840,
"y": 1040
}
}
},
{
"name": "Opt_Out_Response",
"type": "set-variables",
"transitions": [
{
"next": "Response_Received",
"event": "next"
}
],
"properties": {
"variables": [
{
"value": "{% if widgets.Opt_Out_Fail.inbound.MediaUrl0 != null %} {{widgets.Opt_Out_Fail.inbound.MediaUrl0}} {% else %} {{widgets.Opt_Out_Fail.inbound.Body}} {% endif %}",
"key": "current_response"
}
],
"offset": {
"x": -1000,
"y": 1590
}
}
},
{
"name": "Unrecognized_Document",
"type": "send-and-wait-for-reply",
"transitions": [
{
"next": "Unrecognized_Document_Response",
"event": "incomingMessage"
},
{
"event": "timeout"
},
{
"event": "deliveryFailure"
}
],
"properties": {
"offset": {
"x": 970,
"y": 1330
},
"service": "{{trigger.message.InstanceSid}}",
"channel": "{{trigger.message.ChannelSid}}",
"from": "{{flow.channel.address}}",
"body": "I'm sorry. We were unable to process the document you sent. Please try again, or contact us at {{flow.data.contact_num}}",
"timeout": "604800"
}
},
{
"name": "Unrecognized_Document_Response",
"type": "set-variables",
"transitions": [
{
"next": "Response_Received",
"event": "next"
}
],
"properties": {
"variables": [
{
"value": "{% if widgets.Unrecognized_Document.inbound.MediaUrl0 != null %} {{widgets.Unrecognized_Document.inbound.MediaUrl0}} {% else %} {{widgets.Unrecognized_Document.inbound.Body}} {% endif %}",
"key": "current_response"
}
],
"offset": {
"x": 1060,
"y": 1610
}
}
},
{
"name": "Incoming_Call",
"type": "connect-call-to",
"transitions": [
{
"event": "callCompleted"
},
{
"event": "hangup"
}
],
"properties": {
"offset": {
"x": -460,
"y": -60
},
"caller_id": "{{contact.channel.address}}",
"noun": "number",
"to": "+19875551212",
"timeout": 30
}
}
],
"initial_state": "Trigger",
"flags": {
"allow_concurrent_calls": true
}
}
Also, in case it helps, here's a screenshot of the entire flow from the Studio editor:
I am continuing to work with the Twilio Support team as well to determine if there's an actual "problem" here, or if I'm simply doing it wrong (a distinct possibility). There very well may be more effective/efficient ways to achieve my goals but, in the absence of a "generic" option to validate any incoming widget (e.g., widgets.previous_widget.inbound.MediaUrl0), this is the best solution I've been able to find so far.
The main reason I'm posting my question here is not only to try and find a solution that fits my specific needs, but also to document the troubleshooting process and hopefully help anyone else who's just beginning to work with Studio and needs to implement some "complex" business rules.
UPDATE - Alternative Solution Found
So, after talking it through with Twilio Support, we still weren't able to figure out why the above method doesn't work, but we were able to determine that the current best course of action is to add a series of Split Based On... widgets into the flow at each point where I currently just have the Set Variables widgets. While it significantly increases the overall complexity of the flow itself, this method eliminates the need to use the RegEx matching I had to identify incoming URLs.
Even so, once I was finally able to properly articulate why I was trying to make this work, the representative with whom I spoke agreed that it should work and is going to continue testing and investigating the issue. In the meantime, I'm going to update my flow to add all of the extra widgets I need for my logic, but I'd still like to find out if there's some reason I'm overlooking why this is failing.
Based on my conversations with the Twilio Support team, it seems that, for whatever reason, the RegEx method I had implemented will not work. I wasn't able to get a specific reason why it fails, but it appears to be a technical limitation somewhere within Studio. In this case, to achieve my specific goals, there are basically two alternatives:
As I described in the UPDATE in the OP, I can use Split Based On... widgets at each Send & Wait For Reply point that test whether the MediaUrl0 value is empty, then use a pair of Set Variables widgets - one for text-only replies and one for replies with attachments - to pass the value(s) on to the appropriate next step in the flow.
Instead of using a RegEx match to test the value of my current_response variable, I can use the Starts With test to see if the current_response begins with "my" Twilio API URL (https://api.twilio.com/2010-04-01/Accounts/[ACCOUNT SID]/Messages).
Obviously, the former method requires a bit more complexity - several additional widgets to direct the flow to the appropriate next widget - but is less dependent on the actual value of the Twilio API URL. The only "problem" with the latter method would be that if the API version (2010-04-01) changes, I would need to update each point in my flow where it's used to reference the new version.
At this time, I've opted for the former solution and have tested the flow several times to ensure that it all seems to be working correctly. After a few "false starts", I've been able to get through the entire process with all of the correct responses and operations. Simple text replies send back the appropriate response while sending a file pushes the information over to my MVC controller for further processing.
One totally tangential side note: Because the MediaUrl0 value "masks" the original file name and strips the file extension, I needed a way to determine what kind of file was being sent. In a "regular" SMS interaction using Twilio's Programmable Messaging tools, there's a MediaContentType0 property. This value is not a selectable option in the Studio flow editor. I was able to retrieve the value of this property by manually typing it into the appropriate fields (widgets.[WIDGET_NAME].inbound.MediaContentType0), so it seems that any value that is available in the current JSON should be accessible through Studio. You just have to know what you're looking for because it isn't "documented". I hope this all helps someone else looking to implement their own Studio flow.

AWS Stepfunction, ValidationException

i got the error "The provided key element does not match the schema", while getting data from AWS dynamoDB using stepfunction.
stepfunction Defination
{
"Comment": "This is your state machine",
"StartAt": "Choice",
"States": {
"Choice": {
"Type": "Choice",
"Choices": [
{
"Variable": "$.data.Type",
"StringEquals": "GET",
"Next": "DynamoDB GetItem"
},
{
"Variable": "$.data.Type",
"StringEquals": "PUT",
"Next": "DynamoDB PutItem"
}
]
},
"DynamoDB GetItem": {
"Type": "Task",
"Resource": "arn:aws:states:::dynamodb:getItem",
"Parameters": {
"TableName": "KeshavDev",
"Key": {
"Email": {
"S": "$.Email"
}
}
},
"End": true
},
"DynamoDB PutItem": {
"Type": "Task",
"Resource": "arn:aws:states:::dynamodb:putItem",
"Parameters": {
"TableName": "KeshavDev",
"Item": {
"City": {
"S.$": "$.City"
},
"Email": {
"S.$": "$.Email"
},
"Address": {
"S.$": "$.Address"
}
}
},
"InputPath": "$.data",
"End": true
}
}
}
Input
{
"data": {
"Type": "GET",
"Email": "demo#gmail.com"
}
}
Error
{ "resourceType": "dynamodb", "resource": "getItem", "error":
"DynamoDB.AmazonDynamoDBException", "cause": "The provided key
element does not match the schema (Service: AmazonDynamoDBv2; Status
Code: 400; Error Code: ValidationException; Request ID:
a78c3d7a-ca3f-4483-b986-1735201d4ef2; Proxy: null)" }
I see some potential issues with the getItem task when compared to AWS documentation.
I think the Key field needs to be S.$ similar to what you have in your putItem task.
There is no ResultPath attribute to tell the state machine where to put the results.
Your path may not be correct, try $.data.Email
"DynamoDB GetItem": {
"Type": "Task",
"Resource": "arn:aws:states:::dynamodb:getItem",
"Parameters": {
"TableName": "KeshavDev",
"Key": {
"Email": {
"S.$": "$.data.Email"
}
}
},
"ResultPath": "$.DynamoDB",
"End": true
},
To be honest, I'm not sure if one of all of these are contributing to the validation error those are some things to experiment with.
On another note, there are some open source validators for Amazon State Language but for this case, they were not very helpful and said that your code was valid.
its working, above JD D mentoned steps and also by adding both key in step function definition.
DynamoDb have two key.
primary partition key
primary sort key

How to get the number of rows inserted using BigQuery Streaming

I am reading data from a CSV file, inserting the data to a Big Query table using insertAll() method from Streaming Insert as shown below:
InsertAllResponse response = dfsf.insertAll(InsertAllRequest.newBuilder(tableId).setRows(rows).build());
rows here is an Iterable declared like this:
Iterable<InsertAllRequest.RowToInsert> rows
Now, I am actually batching the rows to insert into a size of 500 as suggested here - link to suggestion
After all the data has been inserted, how do I count the total number of rows that were inserted?
I want to find that out and log it to log4j.
This can be done one of two ways
The BigQuery Jobs API via the getQueryResults
https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/getQueryResults
Cloud Logging, the output you want in the tableDataChange field.
Here is a sample output:
{
"protoPayload": {
"#type": "type.googleapis.com/google.cloud.audit.AuditLog",
"status": {},
"authenticationInfo": {
"principalEmail": "service_account"
},
"requestMetadata": {
"callerIp": "2600:1900:2000:1b:400::27",
"callerSuppliedUserAgent": "gl-python/3.7.1 grpc/1.22.0 gax/1.14.2 gapic/1.12.1 gccl/1.12.1,gzip(gfe)"
},
"serviceName": "bigquery.googleapis.com",
"methodName": "google.cloud.bigquery.v2.JobService.InsertJob",
"authorizationInfo": [
{
"resource": "projects/project_id/datasets/dataset/tables/table",
"permission": "bigquery.tables.updateData",
"granted": true
}
],
"resourceName": "projects/project_id/datasets/dataset/tables/table",
"metadata": {
"tableDataChange": {
"deletedRowsCount": "2",
"insertedRowsCount": "2",
"reason": "QUERY",
"jobName": "projects/PRJOECT_ID/jobs/85f19bdd-aff5-4abe-9283-9f0bc9ed3ce8"
},
"#type": "type.googleapis.com/google.cloud.audit.BigQueryAuditMetadata"
}
},
"insertId": "7x7ye390qm",
"resource": {
"type": "bigquery_dataset",
"labels": {
"project_id": "PRJOECT_ID",
"dataset_id": "dataset-id"
}
},
"timestamp": "2020-10-26T07:00:22.960735Z",
"severity": "INFO",
"logName": "projects/PRJOECT_ID/logs/cloudaudit.googleapis.com%2Fdata_access",
"receiveTimestamp": "2020-10-26T07:00:23.763159336Z"
}

Alexa smart home skill development: Device discovery not working

I am currently developing a smart home skill for my blinds, however I am unable to discover device. Is there a way for me to validate my Discovery response message? I'm thinking this is some logical error in the JSON.
I'm using a Lambda function to perform the requests to my API using node-fetch and async/await, thus I have tagged all JS function as async, this could be another potential cause of this issue. I don't get any errors in CloudWatch either.
This is the response my Lambda function is sending:
{
"event": {
"header": {
"namespace": "Alexa.Discovery",
"name": "Discover.Response",
"payloadVersion": "3",
"messageId": "0a58ace0-e6ab-47de-b6af-b600b5ab8a7a"
},
"payload": {
"endpoints": [
{
"endpointId": "com-tobisoft-rollos-1",
"manufacturerName": "tobisoft",
"description": "Office Blinds",
"friendlyName": "Office Blinds",
"displayCategories": [
"INTERIOR_BLIND"
],
"capabilities": [
{
"type": "AlexaInterface",
"interface": "Alexa.RangeController",
"instance": "Blind.Lift",
"version": "3",
"properties": {
"supported": [
{
"name": "rangeValue"
}
],
"proactivelyReported": true,
"retrievable": true
},
"capabilityResources": {
"friendlyNames": [
{
"#type": "asset",
"value": {
"assetId": "Alexa.Setting.Opening"
}
}
]
},
"configuration": {
"supportedRange": {
"minimumValue": 0,
"maximumValue": 100,
"precision": 1
},
"unitOfMeasure": "Alexa.Unit.Percent"
},
"semantics": {
"actionMappings": [
{
"#type": "ActionsToDirective",
"actions": [
"Alexa.Actions.Close"
],
"directive": {
"name": "SetRangeValue",
"payload": {
"rangeValue": 100
}
}
},
{
"#type": "ActionsToDirective",
"actions": [
"Alexa.Actions.Open"
],
"directive": {
"name": "SetRangeValue",
"payload": {
"rangeValue": 1
}
}
},
{
"#type": "ActionsToDirective",
"actions": [
"Alexa.Actions.Lower"
],
"directive": {
"name": "AdjustRangeValue",
"payload": {
"rangeValueDelta": 10,
"rangeValueDeltaDefault": false
}
}
},
{
"#type": "ActionsToDirective",
"actions": [
"Alexa.Actions.Raise"
],
"directive": {
"name": "AdjustRangeValue",
"payload": {
"rangeValueDelta": -10,
"rangeValueDeltaDefault": false
}
}
}
],
"stateMappings": [
{
"#type": "StatesToValue",
"states": [
"Alexa.States.Closed"
],
"value": 100
},
{
"#type": "StatesToRange",
"states": [
"Alexa.States.Open"
],
"range": {
"value": 0
}
}
]
}
},
{
"type": "AlexaInterface",
"interface": "Alexa",
"version": "3"
}
]
}
]
}
}
}
Thanks for any help.
Is there a way for me to validate my Discovery response message?
Yes, you could use the Alexa Smart Home Message JSON Schema. This schema can be used for message validation during skill development, it validates Smart Home skills (except the Video Skills API).
This is the response my Lambda function is sending
I've validated your response following this steps, the result: no errors found, the JSON validates against the schema. So, there's probably another thing going on. I suggest getting in touch with Alexa Developer Contact Us

Cloud API Vision Results not appearing

I'm making a request with the google vision api that appears to have worked, I get an operation number back. The problem I am having is the I am not sure how to interpret the results and nothing appeared in the output folder after running the script.
This is the script I ran
https://vision.googleapis.com/v1/files:asyncBatchAnnotate
{
"requests":[
{
"inputConfig": {
"gcsSource": {
"uri": "gs://somebucket/1.pdf"
},
"mimeType": "application/pdf"
},
"features": [
{
"type": "DOCUMENT_TEXT_DETECTION"
}
],
"outputConfig": {
"gcsDestination": {
"uri": "gs://somebucket/output/"
},
"batchSize": 1
}
}
]
}
This returns back
{
"name": "operations/8b7534d4b21b825e"
}
and when I do a lookup on the operation I get this
https://vision.googleapis.com/v1/operations/8b7534d4b21b825e
{
"name": "operations/8b7534d4b21b825e",
"metadata": {
"#type": "type.googleapis.com/google.cloud.vision.v1.OperationMetadata",
"state": "CREATED",
"createTime": "2019-01-09T21:08:57.339363096Z",
"updateTime": "2019-01-09T21:08:57.339363096Z"
}
}
However the output folder is completely empty and I am not sure what to make of the state created.
According to this answer by a Google engineer, latency in the order of minutes (~10 minutes) is somewhat expected. I’ve done some tests myself, with small files and at moments delay can be up to 25 minutes, though in some cases it is much less.
When Vision API is done processing your request, you should get a response like the one below, for the get method:
{
"name": "operations/XXXxxxxXXXX",
"metadata": {
"#type": "type.googleapis.com/google.cloud.vision.v1.OperationMetadata",
"state": "DONE",
"createTime": "2019-01-09T23:08:37.312889645Z",
"updateTime": "2019-01-09T23:08:59.169306747Z"
},
"done": true,
"response": {
"#type": "type.googleapis.com/google.cloud.vision.v1.AsyncBatchAnnotateFilesResponse",
"responses": [
{
"outputConfig": {
"gcsDestination": {
"uri": "gs://somebucket/output/"
}
}
}
]
}
}