Related
Do we need to specify the hyperparameters both in algorithmHyperParameters and algorithmHyperParameterRanges? If yes, then should we specify a single value (string as per documentation) in algorithmHyperParameters, but a range (integer in case of integer-valued hyperparameter) in algorithmHyperParameterRanges?
For example: Similar-Items recipe has an integer-valued hyperparameter item_id_hidden_dimension. If I use the following solution_config, where item_id_hidden_dimension is specified only in algorithmHyperParameterRanges and not in algorithmHyperParameters, I get the error:
An error occurred (InvalidInputException) when calling the CreateSolution operation: Provide a hyperparameter that is used in the algorithm: arn:aws:personalize:::algorithm/aws-similar-items
"solution_config": {
"algorithmHyperParameters": {},
"hpoConfig": {
"algorithmHyperParameterRanges": {
"integerHyperParameterRanges": [
{
"name": "item_id_hidden_dimension",
"minValue": 30,
"maxValue": 200
}
],
"categoricalHyperParameterRanges": [],
"continuousHyperParameterRanges": []
},
"hpoResourceConfig": {
"maxNumberOfTrainingJobs": "4",
"maxParallelTrainingJobs": "2"
}
}
}
But if I use the following solution_config, where item_id_hidden_dimension is specified both in algorithmHyperParameterRanges and in algorithmHyperParameters, I still get the same error:
An error occurred (InvalidInputException) when calling the CreateSolution operation: Provide a hyperparameter that is used in the algorithm: arn:aws:personalize:::algorithm/aws-similar-items
"solution_config": {
"algorithmHyperParameters": {
"item_id_hidden_dimension": "100"
},
"hpoConfig": {
"algorithmHyperParameterRanges": {
"integerHyperParameterRanges": [
{
"name": "item_id_hidden_dimension",
"minValue": 30,
"maxValue": 200
}
],
"categoricalHyperParameterRanges": [],
"continuousHyperParameterRanges": []
},
"hpoResourceConfig": {
"maxNumberOfTrainingJobs": "4",
"maxParallelTrainingJobs": "2"
}
}
}
This is caused by an error in the documentation. The hyperparameter names should be item_id_hidden_dim and item_metadata_hidden_dim (note they are dim and not dimension as the documentation states).
This can be confirmed by calling the DescribeRecipe API to get the algorithmArn for the Similar-Items recipe and then calling the DescribeAlgorithm API to get details on the algorithm.
import boto3
import json
personalize = boto3.client('personalize')
response = personalize.describe_recipe(recipeArn = 'arn:aws:personalize:::recipe/aws-similar-items')
print(json.dumps(response['recipe'], indent=2, default=str))
{
"name": "aws-similar-items",
"recipeArn": "arn:aws:personalize:::recipe/aws-similar-items",
"algorithmArn": "arn:aws:personalize:::algorithm/aws-similar-items",
"featureTransformationArn": "arn:aws:personalize:::feature-transformation/similar-items",
"status": "ACTIVE",
"description": "Predicts items similar to a given item based on co-occurrence of items in the user-item interactions dataset and item metadata in the item dataset.",
"creationDateTime": "2019-06-10 00:00:00+00:00",
"recipeType": "RELATED_ITEMS",
"lastUpdatedDateTime": "2022-08-17 00:25:42.935000+00:00"
}
algo_arn = response['recipe']['algorithmArn']
response = personalize.describe_algorithm(algorithmArn = algo_arn)
print(json.dumps(response['algorithm'], indent=2, default=str))
{
"name": "aws-similar-items",
"algorithmArn": "arn:aws:personalize:::algorithm/aws-similar-items",
"algorithmImage": {
"name": "Item Similarity"
},
"defaultHyperParameters": {
"item_id_hidden_dim": "100",
"item_metadata_hidden_dim": "100"
},
"defaultHyperParameterRanges": {
"integerHyperParameterRanges": [
{
"name": "item_id_hidden_dim",
"minValue": 30,
"maxValue": 200,
"isTunable": true
},
{
"name": "item_metadata_hidden_dim",
"minValue": 30,
"maxValue": 200,
"isTunable": true
}
],
"continuousHyperParameterRanges": [],
"categoricalHyperParameterRanges": []
},
"defaultResourceConfig": {
"maxNumberOfTrainingJobs": "20",
"maxParallelTrainingJobs": "5"
},
"trainingInputMode": "File",
"creationDateTime": "2019-06-10 00:00:00+00:00",
"lastUpdatedDateTime": "2022-08-17 00:24:41.307000+00:00"
}
Note the hyperparameter names in the last response above.
We will get this error fixed in the documentation ASAP.
I created a lambda function using quarkus native, however if I manually invoke the lambda function via IntelliJ using AWS plugin, I get null pointer exception.
A different behavior is observed, if I upload the resultant function.zip obtained after running mvn package -Dnative -Dquarkus.native.container-build=true command to AWS, while using the aws console by passing the S3 event json in the "Test" section, the JSON is not resolved to the S3Event object, hence if I print the event I get an empty object in console.
{
"timestamp": "2022-05-16T06:00:03.371Z",
"sequence": 46,
"loggerClassName": "org.jboss.logging.Logger",
"loggerName": "com.XX.XX.FTPDocUploaderLambdaHandler",
"level": "INFO",
"message": "{\n \"records\" : [ ]\n}",
"threadName": "Lambda Thread (NORMAL)",
"threadId": 6,
"mdc": {},
"ndc": "",
"hostName": "unknown-host.unknown-domain",
"processName": "NativeImageGeneratorRunner$JDK9Plus",
"processId": 26
}
However if the same is invoked automatically by creating an s3 trigger it works fine.
This is hindering for debugging the lambda function locally using an IDE. Any reason why the same event json if passed manually is not getting resolved by Quarkus ?
Below is the Lambda Handler :
public class FTPDocUploaderLambdaHandler implements RequestHandler<S3Event, String> {
private static final Logger LOG = Logger.getLogger(FTPDocUploaderLambdaHandler.class);
private static final String S3_OBJECT_CREATED_PUT_EVENT = "ObjectCreated:Put";
#Inject
S3Client s3;
#Inject
ObjectMapper mapper;
#Override
public String handleRequest(S3Event event, Context context) {
LOG.info("Received an s3 event");
try {
LOG.info(mapper.writerWithDefaultPrettyPrinter().writeValueAsString(event));
} catch (JsonProcessingException e) {
LOG.error("An error occurred {} ",e);
}
if (event != null && !event.getRecords().isEmpty()) {
List<String> objectKeys = event.getRecords().stream().filter(er -> er.getEventName().equalsIgnoreCase(S3_OBJECT_CREATED_PUT_EVENT)).map(r -> {
return getS3Object(r.getS3().getObject().getKey(), r.getS3().getBucket().getName());
}
).collect(Collectors.toList());
return String.join(" ,", objectKeys);
}
return "UNSUPPORTED_EVENT";
}
private String getS3Object(String objectKey, String bucketName) {
LOG.info("Fetching object from bucket " + bucketName + " with key " + objectKey);
if (StringUtils.isNotBlank(objectKey) && StringUtils.isNotBlank(bucketName)) {
GetObjectRequest request = GetObjectRequest.builder().bucket(bucketName).key(objectKey).build();
ResponseBytes<GetObjectResponse> s3ObjectResponseStream = s3.getObjectAsBytes(request);
String content = s3ObjectResponseStream.asUtf8String();
LOG.info("Object Content");
LOG.info(content);
return content;
}
LOG.info("Invalid Object Key or Bucket Name");
return "Invalid Object Key or Bucket Name";
}
}
S3 Event JSON :
{
"Records": [
{
"eventVersion": "2.1",
"eventSource": "aws:s3",
"awsRegion": "us-east-2",
"eventTime": "2019-09-03T19:37:27.192Z",
"eventName": "ObjectCreated:Put",
"userIdentity": {
"principalId": "AWS:AIDAINPONIXQXHT3IKHL2"
},
"requestParameters": {
"sourceIPAddress": "205.255.255.255"
},
"responseElements": {
"x-amz-request-id": "D82B88E5F771F645",
"x-amz-id-2": "vlR7PnpV2Ce81l0PRw6jlUpck7Jo5ZsQjryTjKlc5aLWGVHPZLj5NeC6qMa0emYBDXOo6QBU0Wo="
},
"s3": {
"s3SchemaVersion": "1.0",
"configurationId": "828aa6fc-f7b5-4305-8584-487c791949c1",
"bucket": {
"name": "DOC-EXAMPLE-BUCKET",
"ownerIdentity": {
"principalId": "A3I5XTEXAMAI3E"
},
"arn": "arn:aws:s3:::lambda-artifacts-deafc19498e3f2df"
},
"object": {
"key": "b21b84d653bb07b05b1e6b33684dc11b",
"size": 1305107,
"eTag": "b21b84d653bb07b05b1e6b33684dc11b",
"sequencer": "0C0F6F405D6ED209E1"
}
}
}
]
}
Exception :
java.lang.NullPointerException: java.lang.NullPointerException
java.lang.NullPointerException
{"errorMessage":"java.lang.NullPointerException","errorType":"java.lang.NullPointerException","stackTrace":["com.research.lambda.FTPDocUploaderLambdaHandler.handleRequest(FTPDocUploaderLambdaHandler.java:36)","com.XX.XX.FTPDocUploaderLambdaHandler.handleRequest(FTPDocUploaderLambdaHandler.java:20)"]} at com.research.lambda.FTPDocUploaderLambdaHandler.handleRequest(FTPDocUploaderLambdaHandler.java:36)
After reading through the exception, it seems it is not able to resolve #Inject attribute, hence the ObjectMapper and S3Client bean are not being injected while invoking manually.
However executing over AWS the beans are resolving but the s3Evevent is not getting resolved.
Invoking locally using sam is also not able to bind the s3Event object from JSON to the handler, below are the logs from SAM local execution :
[PS C:\Users\XX\Downloads\XX-ftp-doc-uploader> sam local invoke --template target/sam.native.yaml --event s3PutEvent.json
Invoking not.used.in.provided.runtime (provided)
Decompressing C:\Users\XX\Downloads\XX-ftp-doc-uploader\target\function.zip
Image was not found.
Removing rapid images for repo public.ecr.aws/sam/emulation-provided
Building image.........................................................................................................................................................................................................................
.....................................................
Skip pulling image and use local one: public.ecr.aws/sam/emulation-provided:rapid-1.50.0-x86_64.
Mounting C:\Users\XX\AppData\Local\Temp\tmpkbwv4bol as /var/task:ro,delegated inside runtime container
START RequestId: 00ee5bd3-6fb5-4ffe-ac29-f92b14611c32 Version: $LATEST
{
"timestamp": "2022-05-16T06:32:45.205Z",
"sequence": 42,
"loggerClassName": "org.jboss.logging.Logger",
"loggerName": "io.quarkus",
"level": "INFO",
"message": "xx-ftp-doc-uploader 1.0-SNAPSHOT native (powered by Quarkus 2.9.0.Final) started in 0.490s. ",
"threadName": "main",
"threadId": 1,
"mdc": {},
"ndc": "",
"hostName": "a6acfab4b085",
"processName": "NativeImageGeneratorRunner$JDK9Plus",
"processId": 26
}
{
"timestamp": "2022-05-16T06:32:45.233Z",
"sequence": 43,
"loggerClassName": "org.jboss.logging.Logger",
"loggerName": "io.quarkus",
"level": "INFO",
"message": "Profile prod activated. ",
"threadName": "main",
"threadId": 1,
"mdc": {},
"ndc": "",
"hostName": "a6acfab4b085",
"processName": "NativeImageGeneratorRunner$JDK9Plus",
"processId": 26
}
{
"timestamp": "2022-05-16T06:32:45.233Z",
"sequence": 44,
"loggerClassName": "org.jboss.logging.Logger",
"loggerName": "io.quarkus",
"level": "INFO",
"message": "Installed features: [amazon-lambda, amazon-s3, cdi, config-yaml]",
"threadName": "main",
"threadId": 1,
"mdc": {},
"ndc": "",
"hostName": "a6acfab4b085",
"processName": "NativeImageGeneratorRunner$JDK9Plus",
"processId": 26
}
{
"timestamp": "2022-05-16T06:32:45.29Z",
"sequence": 45,
"loggerClassName": "org.jboss.logging.Logger",
"loggerName": "com.XX.lambda.FTPDocUploaderLambdaHandler",
"level": "INFO",
"message": "Received an s3 event",
"threadName": "Lambda Thread (NORMAL)",
"threadId": 6,
"mdc": {},
"ndc": "",
"hostName": "a6acfab4b085",
"processName": "NativeImageGeneratorRunner$JDK9Plus",
"processId": 26
}
{
"timestamp": "2022-05-16T06:32:45.3Z",
"sequence": 46,
"loggerClassName": "org.jboss.logging.Logger",
"loggerName": "com.XX.lambda.FTPDocUploaderLambdaHandler",
"level": "INFO",
"message": "{\n \"records\" : [ ]\n}",
"threadName": "Lambda Thread (NORMAL)",
"threadId": 6,
"mdc": {},
"ndc": "",
"hostName": "a6acfab4b085",
"processName": "NativeImageGeneratorRunner$JDK9Plus",
"processId": 26
}
END RequestId: 00ee5bd3-6fb5-4ffe-ac29-f92b14611c32
REPORT RequestId: 00ee5bd3-6fb5-4ffe-ac29-f92b14611c32 Init Duration: 0.10 ms Duration: 928.23 ms Billed Duration: 929 ms Memory Size: 128 MB Max Memory Used: 128 MB
"UNSUPPORTED_EVENT"]
Any idea, how to resolve this to be able to debug locally ?
I am reading monitoring data through Google Timeseries api. The api is working correctly and if give alignment period=3600s it gives me the values for that time series between start and end time for any metric type.
I am calling it through Python like this:
service.projects().timeSeries().list(
name=api_args["project_name"],
filter=api_args["metric_filter"],
aggregation_alignmentPeriod=api_args["aggregation_alignment_period"],
# aggregation_crossSeriesReducer=api_args["crossSeriesReducer"],
aggregation_perSeriesAligner=api_args["perSeriesAligner"],
aggregation_groupByFields=api_args["group_by"],
interval_endTime=api_args["end_time_str"],
interval_startTime=api_args["start_time_str"],
pageSize=config.PAGE_SIZE,
pageToken=api_args["nextPageToken"]
).execute()
and in Postman:
https://monitoring.googleapis.com/v3/projects/my-project/timeSeries?pageSize=500&interval.startTime=2020-07-04T16%3A39%3A37.230000Z&aggregation.alignmentPeriod=3600s&aggregation.perSeriesAligner=ALIGN_SUM&filter=metric.type%3D%22compute.googleapis.com%2Finstance%2Fnetwork%2Freceived_bytes_count%22+&pageToken=&interval.endTime=2020-07-04T17%3A30%3A01.497Z&alt=json&aggregation.groupByFields=metric.labels.key
I face an issue here:
{
"metric": {
"labels": {
"instance_name": "insta-demo1",
"loadbalanced": "false"
},
"type": "compute.googleapis.com/instance/network/received_bytes_count"
},
"resource": {
"type": "gce_instance",
"labels": {
"instance_id": "1234343552",
"zone": "us-central1-f",
"project_id": "my-project"
}
},
"metricKind": "DELTA",
"valueType": "INT64",
"points": [
{
"interval": {
"startTime": "2020-07-04T16:30:01.497Z",
"endTime": "2020-07-04T17:30:01.497Z"
},
"value": {
"int64Value": "6720271"
}
}
]
},
{
"metric": {
"labels": {
"loadbalanced": "true",
"instance_name": "insta-demo2"
},
"type": "compute.googleapis.com/instance/network/received_bytes_count"
},
"resource": {
"type": "gce_instance",
"labels": {
"instance_id": "1234566343",
"project_id": "my-project",
"zone": "us-central1-f"
}
},
"metricKind": "DELTA",
"valueType": "INT64",
"points": [
{
"interval": {
"startTime": "2020-07-04T16:30:01.497Z",
"endTime": "2020-07-04T17:30:01.497Z"
},
"value": {
"int64Value": "579187"
}
}
]
}
],
"unit": "By". //This "By" is the value which is causing problem,
I am getting this value like "unit": "By" or "unit":"ms" or something like that at the end, Also if I don't find any data for a range I'm getting this value, as I am evaluating this response in Python I am getting key error as there is not key called "unit"
logMessage: "Key Error: ' '"
severity: "ERROR"
As the response is empty I am getting the single key called "unit". Also at the end of any response I am getting this "unit":"ms" or "unit":"by" - is there any way to prevent that unit value coming in the response?
I am new to Google Cloud APIs and Python. What can I try next?
The "unit" field expresses the kind of resource the metric is counting. For bytes, it is "By". Read this. I understand it is always returned, so there is no way of not receiving it; I recommend you to adapt your code to correctly deal with its appearance in the responses.
How to use AWS IoT Rule Query Statement to get temperature and humidity from the following MQTT payload to 'example' topic?
[
{
"timestamp": "2019-08-26T14:21:46Z",
"type": "Gateway",
"mac": "XXYYZZXXYYZZ",
"gatewayFree": 96,
"gatewayLoad": 0.26
},
{
"timestamp": "2019-08-26T14:21:46Z",
"type": "S1",
"mac": "XXYYZZXXYYXX",
"bleName": "",
"rssi": -53,
"battery": 100,
"temperature": 0.69,
"humidity": 37.28
},
{
"timestamp": "2019-08-26T14:21:46Z",
"type": "iBeacon",
"mac": "XXYYZZXXYYYY",
"bleName": "",
"ibeaconUuid": "ABCDEFGHIJKLMNOPQRSTUVWXYZ",
"ibeaconMajor": 0,
"ibeaconMinor": 0,
"rssi": -64,
"ibeaconTxPower": -59,
"battery": 0
}
]
So far, I cannot find it on AWS IoT SQL Reference.
Try this, it's working!
SELECT (SELECT temperature, humidity from input) as output FROM 'iot/topic'
I have modified the input JSON to have a key -- see below.
Here is the modified input:
{
"input": [ ... your json array elements ... ]
}
Here is the output:
{
"output": [
{},
{
"temperature": 0.69,
"humidity": 37.28
},
{}
]
}
cheers,
ram
Also try this one...
{
"sql": "SELECT * FROM 'iot/example' WHERE mac = 'XXYYZZXXYYXX'",
...
}
Cheers!
I have a OnDemand triggered webjob that I want to trigger through ADF copy activity using HTTP linked service. Here is the linked service:-
{
"name": "LS_WebJob",
"properties": {
"hubName": "yas-cdp-adf_hub",
"type": "Http",
"typeProperties": {
"url": "https://cust-app.scm.azurewebsites.net/api/triggeredwebjobs/ConsoleApplication1/run",
"authenticationType": "Basic",
"username": "$custdata-app",
"password": "**********"
}
}
}
Input Dataset
{
"name": "ZZ_Inp_Webjob",
"properties": {
"published": false,
"type": "Http",
"linkedServiceName": "LS_WebJob",
"typeProperties": {
"requestMethod": "Post",
"requestBody": "Hey Buddy"
},
"availability": {
"frequency": "Day",
"interval": 1,
"style": "StartOfInterval"
},
"external": true,
"policy": {}
}
}
Output Dataset
{
"name": "ZZ_Out_WebJob",
"properties": {
"published": false,
"type": "AzureBlob",
"linkedServiceName": "LS_ABLB",
"typeProperties": {
"fileName": "webjob.json",
"folderPath": "yc-cdp-container/Dummy/temp",
"format": {
"type": "TextFormat"
}
},
"availability": {
"frequency": "Day",
"interval": 1,
"style": "StartOfInterval"
}
}
}
Pipeline
{
"name": "ZZ-PL-WebJob",
"properties": {
"description": "This pipeline copies data from an HTTP Marina WiFi Source URL to Azure blob",
"activities": [
{
"type": "Copy",
"typeProperties": {
"source": {
"type": "HttpSource"
},
"sink": {
"type": "BlobSink",
"writeBatchSize": 0,
"writeBatchTimeout": "00:00:00"
}
},
"inputs": [
{
"name": "ZZ_Inp_Webjob"
}
],
"outputs": [
{
"name": "ZZ_Out_Webjob"
}
],
"policy": {
"timeout": "01:00:00",
"concurrency": 1
},
"scheduler": {
"frequency": "Day",
"interval": 1,
"style": "StartOfInterval"
},
"name": "WebjobSourceToAzureBlob",
"description": "Copy from an HTTP source to an Azure blob"
}
],
"start": "2017-04-10T01:00:00Z",
"end": "2017-04-10T01:00:00Z",
"isPaused": false,
"hubName": "yas-cdp-adf_hub",
"pipelineMode": "Scheduled"
}
}
My webjob is a simple C# application:-
namespace ConsoleApplication1
{
class Program
{
static void Main(string[] args)
{
Console.WriteLine("My Team Rocks!");
}
}
}
When I am executing the pipeline, the webjob is being successfully triggered. However the pipeline fails with HTTP 409 conflict error.
Copy activity encountered a user error at Source side:
ErrorCode=UserErrorFailedToReadHttpFile,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Failed
to read data from http source
file.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Net.WebException,Message=The
remote server returned an error: (409) Conflict.,Source=System,'.
Try adding in the gateway name to the linked service json. Refer to this link How to integrate a WebJob within an Azure Data Factory Pipeline .