I'm trying to upload a video from an S3 bucket to YouTube, and getting back strange output that implies a successful post, but doesn't give anything expected back. As well, I set attributes like title and description in my code, but as you can see from the output, this isn't actually being set.
Example Output:
{
"id": "-pfZ_BNH9kg",
"snippet": {
"channelId": "UCZ5AUe-rp3rXKeFS0yx4ZBA",
"title": "unknown",
"channelTitle": "Patrick Hanford",
"publishedAt": "2020-04-30T19:22:15.000Z",
"thumbnails": {
"high": {
"url": "https://i.ytimg.com/vi/-pfZ_BNH9kg/hqdefault.jpg",
"height": 360,
"width": 480
},
"default": {
"url": "https://i.ytimg.com/vi/-pfZ_BNH9kg/default.jpg",
"height": 90,
"width": 120
},
"medium": {
"url": "https://i.ytimg.com/vi/-pfZ_BNH9kg/mqdefault.jpg",
"height": 180,
"width": 320
}
},
"localized": {
"title": "unknown",
"description": ""
},
"liveBroadcastContent": "none",
"categoryId": "20",
"description": ""
},
"etag": "Dn5xIderbhAnUk5TAW0qkFFir0M/3T1YGvGo1YyaTKtTpl8JrJqWS4M",
"status": {
"embeddable": true,
"privacyStatus": "public",
"uploadStatus": "uploaded",
"publicStatsViewable": true,
"license": "youtube"
},
"kind": "youtube#video"
}
Upload Code:
def post(self, attempts=None):
TEST_VIDEO = "http://streamon-perm.s3.amazonaws.com/WPHM-48k-pl-33366.mp4"
headers = {"Content-Type": "video/mp4"}
upload_request_body = {
"snippet": {
"title": "Test Video Upload",
"description": "This is a test of uploading videos.",
"categoryId": "22",
},
"status": {
"privacyStatus": "public"
},
"fileDetails": {
"fileName": TEST_VIDEO,
"fileType": "video"
}
}
params = {
"access_token": self.google_token.get("access_token", None),
"id": self.google_token.get("id_token", None),
"part": "snippet, status"
}
extra = {
"client_id": self.client_id,
"client_secret": self.client_secret
}
google_oauth_session = OAuth2Session(
self.client_id,
token=self.google_token,
auto_refresh_url=self.token_url,
auto_refresh_kwargs=extra,
token_updater=self._save_token
)
upload_response = google_oauth_session.post(
self.video_post_url,
headers=headers,
json=upload_request_body,
params=params
)
logger.info("Response from VIDEO UPLOAD: %s", repr(upload_response.content))
return True
I have also tried downloading the file from S3 and uploading with the file directly, and I get the same result. Without proper error messages or anything to go off of, I'm really not sure what to try next. Any help is greatly appreciated.
I have also tried using requests by itself rather than using oauthlib with exactly the same result.
def post(self, attempts=None):
if attempts is None:
attempts = 0
if self.neutered:
msg = "Youtube post() disabled by ENVIRONMENT variables."
logger.info(msg)
return msg
logger.info("Youtube post() entered with attempt # %s", self.post_attempts)
if self.google_token is None:
self.google_token = self._set_google_token()
attempts += 1
self.post(attempts=attempts)
headers = {
"Content-Type": "video/mp4",
"client_id": self.client_id,
"client_secret": self.client_secret,
"Authorization": "Bearer " + self.google_token["access_token"]
}
params = {
"access_token": self.google_token.get("access_token", None),
"id": self.google_token.get("id_token", None),
"part": "snippet, status"
}
upload_request_body = {
"snippet": {
"title": "Test Video Upload",
"description": "This is a test of uploading videos from POST.",
"categoryId": "22",
},
"status": {
"privacyStatus": "public"
},
"fileDetails": {
"fileName": TEST_VIDEO,
"fileType": "video"
}
}
upload_response = requests.post(
self.video_post_url,
params=params,
headers=headers,
json=upload_request_body
)
logger.info("Response from VIDEO UPLOAD: %s", repr(upload_response.content))
return True
I have also tried downloading the file from S3 and uploading with the file directly, and I get the same result.
Your have this issue probably due to the fact that you are not actually sending the file. upload_request_body.fileDetails.fileName is not the place for the link/file. It's just a description attribute.
Have you tried an auto-generated code from https://developers.google.com/youtube/v3/code_samples/code_snippets ?
This is what you can get there:
# -*- coding: utf-8 -*-
# Sample Python code for youtube.videos.insert
# NOTES:
# 1. This sample code uploads a file and can't be executed via this interface.
# To test this code, you must run it locally using your own API credentials.
# See: https://developers.google.com/explorer-help/guides/code_samples#python
# 2. This example makes a simple upload request. We recommend that you consider
# using resumable uploads instead, particularly if you are transferring large
# files or there's a high likelihood of a network interruption or other
# transmission failure. To learn more about resumable uploads, see:
# https://developers.google.com/api-client-library/python/guide/media_upload
import os
import googleapiclient.discovery
from googleapiclient.http import MediaFileUpload
def main():
# Disable OAuthlib's HTTPS verification when running locally.
# *DO NOT* leave this option enabled in production.
os.environ["OAUTHLIB_INSECURE_TRANSPORT"] = "1"
api_service_name = "youtube"
api_version = "v3"
DEVELOPER_KEY = "YOUR_API_KEY"
youtube = googleapiclient.discovery.build(
api_service_name, api_version, developerKey = DEVELOPER_KEY)
request = youtube.videos().insert(
part="snippet,status",
body={
"fileDetails": {
"fileName": "qwer",
"fileType": "video"
},
"snippet": {
"categoryId": "22",
"description": "This is a test of uploading videos.",
"title": "Test Video Upload"
},
"status": {
"privacyStatus": "public"
}
},
# TODO: For this request to work, you must replace "YOUR_FILE"
# with a pointer to the actual file you are uploading.
media_body=MediaFileUpload("YOUR_FILE")
)
response = request.execute()
print(response)
if __name__ == "__main__":
main()
I believe it should work.
Or is there any reason not to use googleapiclient?
I'm trying to upload a video from an S3 bucket to YouTube
I doubt that you can upload files from other sites directly to Youtube. Probably you are stuck with the option of uploading files from your own server/drive. I've looked up on the Internet but all I've found is that you can't (although you could in the past). And one can imagine a lot of reasons why this is not allowed (mostly copyright but not exclusively).
Update:
Probably, that was not an exhaustive code snippet. Especially, considering that you need OAuth2.
But here is another one:
https://github.com/youtube/api-samples/blob/master/python/upload_video.py
And yet another:
https://developers.google.com/youtube/v3/guides/uploading_a_video
With OAuth2. There you can also find information on client_secrets.json.
{
"web": {
"client_id": "[[INSERT CLIENT ID HERE]]",
"client_secret": "[[INSERT CLIENT SECRET HERE]]",
"redirect_uris": [],
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://accounts.google.com/o/oauth2/token"
}
}
Also you can checkout some real life projects. For example this one: https://github.com/HA6Bots/Automatic-Youtube-Reddit-Text-To-Speech-Video-Generator-and-Uploader/tree/master/Youtube%20Bot%20Video%20Generator
Related
I am working with google cloud vision API safe search function. I am calling the API by passing in the URLs to remote images.
I noticed that the same image URL fails before 1PM PST, where it works after roughly 1PM. I have been noticing this pattern for 3 day.
Failing messages (Before 1PM) look something like this:
"error": {
"code": 14,
"message": "We can not access the URL currently. Please download the content and pass it in."
}
where as when it works correctly (after 1PM), I can get expected responses like this:
"adult": "UNLIKELY",
"spoof": "UNLIKELY",
"medical": "UNLIKELY",
"violence": "UNLIKELY",
"racy": "LIKELY"
}
Has anyone else experienced a similar issue? I am still on free plan, not sure if that is a problem.
Edit: Adding response -
{
"responses": [
{
"error": {
"code": 14,
"message": "We can not access the URL currently. Please download the content and pass it in."
}
},
{
"error": {
"code": 4,
"message": "We can not access the URL currently. Please download the content and pass it in."
}
},
{
"error": {
"code": 14,
"message": "We can not access the URL currently. Please download the content and pass it in."
}
},
{
"safeSearchAnnotation": {
"adult": "LIKELY",
"spoof": "VERY_UNLIKELY",
"medical": "UNLIKELY",
"violence": "UNLIKELY",
"racy": "VERY_LIKELY"
}
},
{
"error": {
"code": 4,
"message": "We can not access the URL currently. Please download the content and pass it in."
}
},
{
"safeSearchAnnotation": {
"adult": "UNLIKELY",
"spoof": "VERY_UNLIKELY",
"medical": "UNLIKELY",
"violence": "VERY_UNLIKELY",
"racy": "VERY_LIKELY"
}
},
{
"error": {
"code": 14,
"message": "We can not access the URL currently. Please download the content and pass it in."
}
},
{
"error": {
"code": 14,
"message": "We can not access the URL currently. Please download the content and pass it in."
}
},
{
"error": {
"code": 4,
"message": "We can not access the URL currently. Please download the content and pass it in."
}
},
{
"safeSearchAnnotation": {
"adult": "VERY_UNLIKELY",
"spoof": "VERY_UNLIKELY",
"medical": "VERY_UNLIKELY",
"violence": "VERY_UNLIKELY",
"racy": "POSSIBLE"
}
},
{
"safeSearchAnnotation": {
"adult": "POSSIBLE",
"spoof": "VERY_LIKELY",
"medical": "UNLIKELY",
"violence": "UNLIKELY",
"racy": "VERY_LIKELY"
}
},
{
"error": {
"code": 4,
"message": "We can not access the URL currently. Please download the content and pass it in."
}
},
{
"safeSearchAnnotation": {
"adult": "UNLIKELY",
"spoof": "LIKELY",
"medical": "UNLIKELY",
"violence": "UNLIKELY",
"racy": "POSSIBLE"
}
},
{
"safeSearchAnnotation": {
"adult": "POSSIBLE",
"spoof": "POSSIBLE",
"medical": "VERY_UNLIKELY",
"violence": "UNLIKELY",
"racy": "VERY_LIKELY"
}
},
{
"error": {
"code": 4,
"message": "We can not access the URL currently. Please download the content and pass it in."
}
},
{
"error": {
"code": 14,
"message": "We can not access the URL currently. Please download the content and pass it in."
}
}
]
}
200 OK
If you are using publicly available URLs you will encounter this error from time to time since there is no guarantee that the image is readily available. The host of the image might deny some request for DOS prevention. See imageUri reference for more details.
imageUri
The URI of the source image. Can be either:
A Google Cloud Storage URI of the form gs://bucket_name/object_name.
Object versioning is not supported. See Google Cloud Storage Request
URIs for more info.
A publicly-accessible image HTTP/HTTPS URL. When fetching images from
HTTP/HTTPS URLs, Google cannot guarantee that the request will be
completed. Your request may fail if the specified host denies the
request (e.g. due to request throttling or DOS prevention), or if
Google throttles requests to the site for abuse prevention. You should
not depend on externally-hosted images for production applications.
When both gcsImageUri and imageUri are specified, imageUri takes
precedence.
What I could suggest is to download the image locally or put it in cloud storage and from there you can use it as input.
I would like to trigger EMR spark job with python code through AWS Lambda after trigger the s3 event.I appreciate if any one can share the configuration/command to invoke the EMR spark job from AWS Lambda function.
Since this question is very generic, I will try to give an example code for doing this. You will have to change certain parameters based upon your actual value.
The way I generally do this is I place the main handler function in one file say named as lambda_handler.py and all the configuration and steps of the EMR in a file named as emr_configuration_and_steps.py.
Please check the code snippet below for lambda_handler.py
import boto3
import emr_configuration_and_steps
import logging
import traceback
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
formatter = logging.Formatter('%(levelname)s:%(name)s:%(message)s')
def create_emr(name):
try:
emr = boto3.client('emr')
cluster_id = emr.run_job_flow(
Name=name,
VisibleToAllUsers=emr_configuration_and_steps.visible_to_all_users,
LogUri=emr_configuration_and_steps.log_uri,
ReleaseLabel=emr_configuration_and_steps.release_label,
Applications=emr_configuration_and_steps.applications,
Tags=emr_configuration_and_steps.tags,
Instances=emr_configuration_and_steps.instances,
Steps=emr_configuration_and_steps.steps,
Configurations=emr_configuration_and_steps.configurations,
ScaleDownBehavior=emr_configuration_and_steps.scale_down_behavior,
ServiceRole=emr_configuration_and_steps.service_role,
JobFlowRole=emr_configuration_and_steps.job_flow_role
)
logger.info("EMR is created successfully")
return cluster_id['JobFlowId']
except Exception as e:
traceback.print_exc()
raise Exception(e)
def lambda_handler(event, context):
logger.info("starting the lambda function for spawning EMR")
try:
emr_cluster_id = create_emr('Name of Your EMR')
logger.info("emr_cluster_id is = " + emr_cluster_id)
except Exception as e:
logger.error("Exception at some step in the process " + str(e))
Now the second file(emr_configuration_and_steps.py) that has all the configuration would look like this.
visible_to_all_users = True
log_uri = 's3://your-s3-log-path-here/'
release_label = 'emr-5.29.0'
applications = [{'Name': 'Spark'}, {'Name': 'Hadoop'}]
tags = [
{'Key': 'Project', 'Value': 'Your-Project Name'},
{'Key': 'Service', 'Value': 'Your-Service Name'},
{'Key': 'Environment', 'Value': 'Development'}
]
instances = {
'Ec2KeyName': 'Your-key-name',
'Ec2SubnetId': 'your-subnet-name',
'InstanceFleets': [
{
"InstanceFleetType": "MASTER",
"TargetOnDemandCapacity": 1,
"TargetSpotCapacity": 0,
"InstanceTypeConfigs": [
{
"WeightedCapacity": 1,
"BidPriceAsPercentageOfOnDemandPrice": 100,
"InstanceType": "m3.xlarge"
}
],
"Name": "Master Node"
},
{
"InstanceFleetType": "CORE",
"TargetSpotCapacity": 8,
"InstanceTypeConfigs": [
{
"WeightedCapacity": 8,
"BidPriceAsPercentageOfOnDemandPrice": 50,
"InstanceType": "m3.xlarge"
}
],
"Name": "Core Node"
},
],
'KeepJobFlowAliveWhenNoSteps': False
}
steps = [
{
'Name': 'Setup Hadoop Debugging',
'ActionOnFailure': 'TERMINATE_CLUSTER',
'HadoopJarStep': {
'Jar': 'command-runner.jar',
'Args': ['state-pusher-script']
}
},
{
"Name": "Active Marker for digital panel",
"ActionOnFailure": 'TERMINATE_CLUSTER',
'HadoopJarStep': {
"Jar": "command-runner.jar",
"Args": [
"spark-submit",
"--deploy-mode",
"cluster",
"--driver-memory", "4g",
"--executor-memory", "4g",
"--executor-cores", "2",
"--class", "your-main-class-full-path-name",
"s3://your-jar-path-SNAPSHOT-jar-with-dependencies.jar"
]
}
}
]
configurations = [
{
"Classification": "spark-log4j",
"Properties": {
"log4j.logger.root": "INFO",
"log4j.logger.org": "INFO",
"log4j.logger.com": "INFO"
}
}
]
scale_down_behavior = 'TERMINATE_AT_TASK_COMPLETION'
service_role = 'EMR_DefaultRole'
job_flow_role = 'EMR_EC2_DefaultRole'
Please adjust the certain path and name according to your use case. To deploy this you need to install boto3 and package/zip these 2 files in a zip file and upload this to your lambda function. By this you should be able to spawn the EMR.
I'm trying to create token with MPGS.
I'm following this guide:
https://sample-sub.domain.mastercard.com/api/documentation/integrationGuidelines/supportedFeatures/pickAdditionalFunctionality/tokenization/tokenization.html?locale=en_US#x_tokenConfiguration
In the section "Token Operations" > "Tokenize", it says:
You can use this operation to create or update a token by storing
payment details against the token. ...
I'm posting this to help people who are frustrating like me with MPGS. I faced series of issues, and pulled my hair off many times. So here's the issues I faced and how to solve them (I'm stuck with issue #4).
Issue #1: Invalid credentials.
Fix: Make sure you're hitting the correct URL.
https://example-subdomain.mastercard.com/..
https://some.other-example.mastercard.com/..
https://MILLION-OTHER-POSSIBILITIES.mastercard.com/..
Even the documentation guide link have these same sub-domains, so make sure you're hitting the correct URL, and make sure you're following the correct documentation link.
Issue #2: Invalid parameters, or server asking for parameters although you've provided them.
Fix: If using Postman, make sure you set the parameters in "Body" > "raw" as JSON, like so:
{
"sourceOfFunds": {
"provided": {
"card": {
"expiry": {
"month": "05",
"year": "21"
},
"number": "5123456789012346"
}
},
"type": "CARD"
}
}
Issue #3: Authorization required
Fix: If using Postman, click on "Authorization", set "Type" it to Basic Auth, for "Username" set it to merchant.YOUR_MERCHANT_ID, for "Password" set it to YOUR_API_PASSWORD
Issue #4 (stuck here): Value '9999999999999999' is invalid. Card token must not be supplied
Method: PUT
URL: https://test-my.sample.gateway.mastercard.com/api/rest/version/46/merchant/MY_MERCHANT_ID/token/9999999999999999
Authorization: set correctly in Authorization tab
Body > raw:
{
"sourceOfFunds": {
"provided": {
"card": {
"expiry": {
"month": "05",
"year": "21"
},
"number": "5123456789012346"
}
},
"type": "CARD"
}
}
Response:
{
"error": {
"cause": "INVALID_REQUEST",
"explanation": "Value '9999999999999999' is invalid. Card token must not be supplied",
"field": "tokenid",
"validationType": "INVALID"
},
"result": "ERROR"
}
Q: Not sure what to do to tokenize the transaction..?! I'm stuck with issue #4.
Ok, finally figured it out. MPGS has 2 ways to create/update tokens:
Tokenization where YOU provide the token (notice: PUT method)
Tokenization where MPGS generate the token for you (notice: POST method)
They're very similar.
I got it working with the 2nd option.
Note: This is POST method !!
Method: POST
URL: https://SUBDOMAIN_YOU_SHOULD_BE_USING.mastercard.com/api/rest/version/50/merchant/YOUR_MERCHANT_ID/token
In postman, set Authorization (as described in the question, in issue #3).
Sample data to send (in postman, this should be in Body > raw):
{
"sourceOfFunds": {
"provided": {
"card": {
"expiry": {
"month": "05",
"year": "21"
},
"number": "5123456789012346"
}
},
"type": "CARD"
}
}
Sample response:
{
"repositoryId": "1000000000002",
"response": {
"gatewayCode": "BASIC_VERIFICATION_SUCCESSFUL"
},
"result": "SUCCESS",
"sourceOfFunds": {
"provided": {
"card": {
"brand": "MASTERCARD",
"expiry": "0521",
"fundingMethod": "CREDIT",
"issuer": "BANCO DEL PICHINCHA, C.A.",
"number": "512345xxxxxx2346",
"scheme": "MASTERCARD"
}
},
"type": "CARD"
},
"status": "VALID",
"token": "9717501974559694",
"usage": {
"lastUpdated": "2019-02-25T09:36:54.928Z",
"lastUpdatedBy": "1015",
"lastUsed": "2019-02-25T09:36:54.928Z"
},
"verificationStrategy": "BASIC"
}
I have completed a labelling job in AWS ground truth and started working on the notebook template for object detection.
I have 2 manifests which has 293 labeled images for birds in a train and validation set like this:
{"source-ref":"s3://XXXXXXX/Train/Blackbird_1.JPG","Bird-Label-Train":{"workerId":XXXXXXXX,"imageSource":{"s3Uri":"s3://XXXXXXX/Train/Blackbird_1.JPG"},"boxesInfo":{"annotatedResult":{"boundingBoxes":[{"width":1612,"top":841,"label":"Blackbird","left":1276,"height":757}],"inputImageProperties":{"width":3872,"height":2592}}}},"Bird-Label-Train-metadata":{"type":"groundtruth/custom","job-name":"bird-label-train","human-annotated":"yes","creation-date":"2019-01-16T17:28:23+0000"}}
Below are the parameters I am using for the notebook instance:
training_params = \
{
"AlgorithmSpecification": {
"TrainingImage": training_image, # NB. This is one of the named constants defined in the first cell.
"TrainingInputMode": "Pipe"
},
"RoleArn": role,
"OutputDataConfig": {
"S3OutputPath": s3_output_path
},
"ResourceConfig": {
"InstanceCount": 1,
"InstanceType": "ml.p3.2xlarge",
"VolumeSizeInGB": 5
},
"TrainingJobName": job_name,
"HyperParameters": { # NB. These hyperparameters are at the user's discretion and are beyond the scope of this demo.
"base_network": "resnet-50",
"use_pretrained_model": "1",
"num_classes": "1",
"mini_batch_size": "16",
"epochs": "5",
"learning_rate": "0.001",
"lr_scheduler_step": "3,6",
"lr_scheduler_factor": "0.1",
"optimizer": "rmsprop",
"momentum": "0.9",
"weight_decay": "0.0005",
"overlap_threshold": "0.5",
"nms_threshold": "0.45",
"image_shape": "300",
"label_width": "350",
"num_training_samples": str(num_training_samples)
},
"StoppingCondition": {
"MaxRuntimeInSeconds": 86400
},
"InputDataConfig": [
{
"ChannelName": "train",
"DataSource": {
"S3DataSource": {
"S3DataType": "AugmentedManifestFile", # NB. Augmented Manifest
"S3Uri": s3_train_data_path,
"S3DataDistributionType": "FullyReplicated",
"AttributeNames": ["source-ref","Bird-Label-Train"] # NB. This must correspond to the JSON field names in your augmented manifest.
}
},
"ContentType": "image/jpeg",
"RecordWrapperType": "None",
"CompressionType": "None"
},
{
"ChannelName": "validation",
"DataSource": {
"S3DataSource": {
"S3DataType": "AugmentedManifestFile", # NB. Augmented Manifest
"S3Uri": s3_validation_data_path,
"S3DataDistributionType": "FullyReplicated",
"AttributeNames": ["source-ref","Bird-Label"] # NB. This must correspond to the JSON field names in your augmented manifest.
}
},
"ContentType": "image/jpeg",
"RecordWrapperType": "None",
"CompressionType": "None"
}
]
I would end up with this being printed after running my ml.p3.2xlarge instance:
InProgress Starting
InProgress Starting
InProgress Starting
InProgress Training
Failed Failed
Followed by this error message:
'ClientError: train channel is not specified.'
Does anyone have any thoughts for how I can get this running with no errors? Any help is much apreciated!
Successful run: Below is the paramaters that were used, along with the Augmented Manifest JSON Objects for a successful run.
training_params = \
{
"AlgorithmSpecification": {
"TrainingImage": training_image, # NB. This is one of the named constants defined in the first cell.
"TrainingInputMode": "Pipe"
},
"RoleArn": role,
"OutputDataConfig": {
"S3OutputPath": s3_output_path
},
"ResourceConfig": {
"InstanceCount": 1,
"InstanceType": "ml.p3.2xlarge",
"VolumeSizeInGB": 50
},
"TrainingJobName": job_name,
"HyperParameters": { # NB. These hyperparameters are at the user's discretion and are beyond the scope of this demo.
"base_network": "resnet-50",
"use_pretrained_model": "1",
"num_classes": "3",
"mini_batch_size": "1",
"epochs": "5",
"learning_rate": "0.001",
"lr_scheduler_step": "3,6",
"lr_scheduler_factor": "0.1",
"optimizer": "rmsprop",
"momentum": "0.9",
"weight_decay": "0.0005",
"overlap_threshold": "0.5",
"nms_threshold": "0.45",
"image_shape": "300",
"label_width": "350",
"num_training_samples": str(num_training_samples)
},
"StoppingCondition": {
"MaxRuntimeInSeconds": 86400
},
"InputDataConfig": [
{
"ChannelName": "train",
"DataSource": {
"S3DataSource": {
"S3DataType": "AugmentedManifestFile", # NB. Augmented Manifest
"S3Uri": s3_train_data_path,
"S3DataDistributionType": "FullyReplicated",
"AttributeNames": attribute_names # NB. This must correspond to the JSON field names in your **TRAIN** augmented manifest.
}
},
"ContentType": "application/x-recordio",
"RecordWrapperType": "RecordIO",
"CompressionType": "None"
},
{
"ChannelName": "validation",
"DataSource": {
"S3DataSource": {
"S3DataType": "AugmentedManifestFile", # NB. Augmented Manifest
"S3Uri": s3_validation_data_path,
"S3DataDistributionType": "FullyReplicated",
"AttributeNames": ["source-ref","ValidateBird"] # NB. This must correspond to the JSON field names in your **VALIDATION** augmented manifest.
}
},
"ContentType": "application/x-recordio",
"RecordWrapperType": "RecordIO",
"CompressionType": "None"
}
]
}
Training Augmented Manifest File generated during the running of the training job
Line 1
{"source-ref":"s3://XXXXX/Train/Blackbird_1.JPG","TrainBird":{"annotations":[{"class_id":0,"width":1613,"top":840,"height":766,"left":1293}],"image_size":[{"width":3872,"depth":3,"height":2592}]},"TrainBird-metadata":{"job-name":"labeling-job/trainbird","class-map":{"0":"Blackbird"},"human-annotated":"yes","objects":[{"confidence":0.09}],"creation-date":"2019-02-09T14:21:29.829003","type":"groundtruth/object-detection"}}
Line 2
{"source-ref":"s3://xxxxx/Train/Blackbird_2.JPG","TrainBird":{"annotations":[{"class_id":0,"width":897,"top":665,"height":1601,"left":1598}],"image_size":[{"width":3872,"depth":3,"height":2592}]},"TrainBird-metadata":{"job-name":"labeling-job/trainbird","class-map":{"0":"Blackbird"},"human-annotated":"yes","objects":[{"confidence":0.09}],"creation-date":"2019-02-09T14:22:34.502274","type":"groundtruth/object-detection"}}
Line 3
{"source-ref":"s3://XXXXX/Train/Blackbird_3.JPG","TrainBird":{"annotations":[{"class_id":0,"width":1040,"top":509,"height":1695,"left":1548}],"image_size":[{"width":3872,"depth":3,"height":2592}]},"TrainBird-metadata":{"job-name":"labeling-job/trainbird","class-map":{"0":"Blackbird"},"human-annotated":"yes","objects":[{"confidence":0.09}],"creation-date":"2019-02-09T14:20:26.660164","type":"groundtruth/object-detection"}}
I then unzip the model.tar file to get the following files:hyperparams.JSON, model_algo_1-0000.params and model_algo_1-symbol
hyperparams.JSON looks like this:
{"label_width": "350", "early_stopping_min_epochs": "10", "epochs": "5", "overlap_threshold": "0.5", "lr_scheduler_factor": "0.1", "_num_kv_servers": "auto", "weight_decay": "0.0005", "mini_batch_size": "1", "use_pretrained_model": "1", "freeze_layer_pattern": "", "lr_scheduler_step": "3,6", "early_stopping": "False", "early_stopping_patience": "5", "momentum": "0.9", "num_training_samples": "11", "optimizer": "rmsprop", "_tuning_objective_metric": "", "early_stopping_tolerance": "0.0", "learning_rate": "0.001", "kv_store": "device", "nms_threshold": "0.45", "num_classes": "1", "base_network": "resnet-50", "nms_topk": "400", "_kvstore": "device", "image_shape": "300"}
Unfortunately, pipe mode with AugmentedManifestFile is not supported for the image/jpeg content type. To be able to use this feature, you will need to specify RecordWrapperType as RecordIO and ContentType as application/x-recordio.
The 'AttributeNames' parameter need to be ['source-ref', 'your label here'] in both your train and validation channel
Thank you again for your help. All of which were valid in helping me get further. Having received a response on the AWS forum pages, I finally got it working.
I understood that my JSON was slightly different to the augmented manifest training guide. Having gone back to basics, I created another labelling job, but used the 'Bounding Box' type as opposed to the 'Custom - Bounding box template'. My output matched what was expected. This ran with no errors!
As my purpose was to have multiple labels, I was able to edit the files and mapping of my output manifests, which also worked!
i.e.
{"source-ref":"s3://xxxxx/Blackbird_15.JPG","ValidateBird":{"annotations":[{"class_id":0,"width":2023,"top":665,"height":1421,"left":1312}],"image_size":[{"width":3872,"depth":3,"height":2592}]},"ValidateBird-metadata":{"job-name":"labeling-job/validatebird","class-map":{"0":"Blackbird"},"human-annotated":"yes","objects":[{"confidence":0.09}],"creation-date":"2019-02-09T14:23:51.174131","type":"groundtruth/object-detection"}}
{"source-ref":"s3://xxxx/Pigeon_19.JPG","ValidateBird":{"annotations":[{"class_id":2,"width":784,"top":634,"height":1657,"left":1306}],"image_size":[{"width":3872,"depth":3,"height":2592}]},"ValidateBird-metadata":{"job-name":"labeling-job/validatebird","class-map":{"2":"Pigeon"},"human-annotated":"yes","objects":[{"confidence":0.09}],"creation-date":"2019-02-09T14:23:51.074809","type":"groundtruth/object-detection"}}
The original mapping was 0:'Bird' for all images through the labelling job.
I've created a simple Lambda function to call a webpage, this works fine when I test it from the functions page however when trying to create a skill to call this function I end up with a "The remote endpoint could not be called, or the response it returned was invalid." error.
Lambda Function
var http = require('http');
exports.handler = function(event, context) {
console.log('start request to ' + event.url)
http.get(event.url, function(res) {
console.log("Got response: " + res.statusCode);
context.succeed();
}).on('error', function(e) {
console.log("Got error: " + e.message);
context.done(null, 'FAILURE');
});
console.log('end request to ' + event.url);
}
The Test Event code looks like this:
{
"url": "http://mywebsite.co.uk"
}
and I've added a trigger for the "Alexa Skills Kit".
The ARN for this function is showing as:
arn:aws:lambda:us-east-1:052516835015:function:CustomFunction
Alexa Skill (Developer Portal)
I've then created a skill with a simple Intent:
{
"intents": [
{
"intent": "CustomFunction"
}
]
}
and created an Utterance as:
CustomFunction execute my custom function
In the Configuration section for my skill I have selected the "AWS Lambda ARN (Amazon Resource Name)" option and entered the ARN into the box for North America.
In the Test -> Service Simulator section, I've added "execute my custom function" as the Text and this changes the Lambda Request to show:
{
"session": {
"sessionId": "SessionId.a3e8aee0-acae-4de5-85df-XXXXXXXXX",
"application": {
"applicationId": "amzn1.ask.skill.XXXXXXXXX"
},
"attributes": {},
"user": {
"userId": "amzn1.ask.account.XXXXXXXXX"
},
"new": true
},
"request": {
"type": "IntentRequest",
"requestId": "EdwRequestId.445267bd-2b4a-45ef-8566-XXXXXXXXX",
"locale": "en-GB",
"timestamp": "2016-11-27T22:54:07Z",
"intent": {
"name": "RunWOL",
"slots": {}
}
},
"version": "1.0"
}
but when I run the test I get the following error:
The remote endpoint could not be called, or the response it returned was invalid.
Does anyone have any ideas on why the skill can't connect to the function?
Thanks
The Service Simulator built into the Amazon Alexa Developer Console has known issues. Try copying the JSON generated by the Simulator and pasting it into your lambda function's test event. To access lambda's test events first find the blue 'Test' button. Next to that button select the (Actions Drop down menu) -> (Configure Test Event) -> Paste the provided JSON into the code area -> (Save and Test). Lambda's built in testing features are much more reliable than Alexa's.
If this does not solve the problem lambda's testing event returns a complete stackTrace and error codes. It becomes much easier to trouble shoot when every error isn't "The remote endpoint could not be called, or the response it returned was invalid."
{
"session": {
"sessionId": "SessionId.a3e8aee0-acae-4de5-85df-XXXXXXXXX",
"application": {
"applicationId": "amzn1.ask.skill.XXXXXXXXX"
},
"attributes": {},
"user": {
"userId": "amzn1.ask.account.XXXXXXXXX"
},
"new": true
},
"request": {
"type": "IntentRequest",
"requestId": "EdwRequestId.445267bd-2b4a-45ef-8566-XXXXXXXXX",
"locale": "en-GB",
"timestamp": "2016-11-27T22:54:07Z",
"intent": {
"name": "RunWOL",
"slots": {}
}
},
"version": "1.0"
}
​While uploading .zip, do not compress the folder into .zip.
Instead, go into the folder, select package.json, index.js and node modules & then compress them and then upload the .zip.
This error message is very broad and may imply a lot of different issues. I was getting this error and in my case it was a timeout issue. How long does that website you are pinging taking to respond? The timeout doesn't seem to be properly documented, see my original question here: Troubleshooting Amazon's Alexa Skill Kit (ASK) Lambda interaction