Related
I'm trying to create a Image Label Adjustment Job in Ground Truth and I'm having some trouble. The thing is that I have a dataset of images, in which there are pre-made bounding boxes. I have an external python script that creates the "dataset.manifest" file with the json's of each image. Here are the first four lines of that manifest file:
{"source-ref": "s3://automatic-defect-detection/LM-WNB1-M-0000126254-camera_2_0022.jpg", "bounding-box": {"image_size": [{"width": 2048, "height": 1536, "depth": 3}], "annotations": [{"class_id": 0, "width": 80, "height": 80, "top": 747, "left": 840}]}, "bounding-box-metadata": {"class-map": {"0": "KK"}, "type": "groundtruth/object-detection", "human-annotated": "yes"}}
{"source-ref": "s3://automatic-defect-detection/LM-WNB1-M-0000126259-camera_2_0028.jpg", "bounding-box": {"image_size": [{"width": 2048, "height": 1536, "depth": 3}], "annotations": [{"class_id": 0, "width": 80, "height": 80, "top": 1359, "left": 527}]}, "bounding-box-metadata": {"class-map": {"0": "KK"}, "type": "groundtruth/object-detection", "human-annotated": "yes"}}
{"source-ref": "s3://automatic-defect-detection/LM-WNB1-M-0000126256-camera_3_0006.jpg", "bounding-box": {"image_size": [{"width": 2048, "height": 1536, "depth": 3}], "annotations": [{"class_id": 3, "width": 80, "height": 80, "top": 322, "left": 1154}, {"class_id": 3, "width": 80, "height": 80, "top": 633, "left": 968}]}, "bounding-box-metadata": {"class-map": {"3": "FF"}, "type": "groundtruth/object-detection", "human-annotated": "yes"}}
{"source-ref": "s3://automatic-defect-detection/LM-WNB1-M-0000126253-camera_2_0019.jpg", "bounding-box": {"image_size": [{"width": 2048, "height": 1536, "depth": 3}], "annotations": [{"class_id": 2, "width": 80, "height": 80, "top": 428, "left": 1058}]}, "bounding-box-metadata": {"class-map": {"2": "DD"}, "type": "groundtruth/object-detection", "human-annotated": "yes"}}
Now the problem is that I'm creating private jobs in Amazon Sagemaker to try it out. I have the manifest file and the images in a S3 bucket, and it actually kinda works. So I select the input manifest, activate the "Existing-labels display options". The existing labels for the bounding boxes do not appear automatically, so I have to enter them manually (don't know why), but if I do that and try the preview before creating the adjustment job, the bounding boxes appear perfectly and I can adjust them. The thing is that, me being the only worker invited for the job, the job never apears to start working on it, and it just auto-completes. I can see later that the images are there with my pre-made bounding boxes, but the job never appears to adjust those boxes. I don't have the "Automated data labeling" option activated. Is there something missing in my manifest file?
There can be multiple reasons for this. First of all, the automated labeling option is not support for label adjustment and verification tasks. so thats ruled out.
It looks like you have not setup the adjustment job properly. Some things to check for:
Have you specified the Task timeout and Task expiration time? If these values are practically low, then the tasks would be expired even before somebody can pick them.
Have you checked the "I want to display existing labels from the dataset for this job." box? It should be checked for your case.
Does your existing label are fetched properly? If this is not fetched correctly, either you need to review your manifest file or you need to manually provide the label values(which i guess you are doing)
Since you are the only worker in the workforce. Do you have correct permissions to access the labeling task?
How many images you have? Have you set any minimum batch size while setting the label adjustment job?
I am using AWS Lex and AWS Lambda for creating a chatbot. The request and response format are as follows
Event being passed to AWS Lambda
{
"alternativeIntents": [
{
"intentName": "AMAZON.FallbackIntent",
"nluIntentConfidence": null,
"slots": {}
}
],
"botVersion": "$LATEST",
"dialogState": "ConfirmIntent",
"intentName": "OrderBeverage",
"message": "you want to order 2 pints of beer",
"messageFormat": "PlainText",
"nluIntentConfidence": {
"score": 0.92
},
"responseCard": null,
"sentimentResponse": null,
"sessionAttributes": {},
"sessionId": "2021-05-10T09:13:06.841Z-bSWmdHVL",
"slotToElicit": null,
"slots": {
"Drink": "beer",
"Quantity": "2",
"Unit": "pints"
}
}
Response Format-
{
"statusCode": 200,
"dialogAction": {
"type": "Close",
"fulfillmentState": "Fulfilled",
"message": {
"contentType": "PlainText",
"content": "Message to convey to the user. For example, Thanks, your pizza has been ordered."
}
}
}
AWS LAMBDA PYTHON IMPLEMENTATION-
import json
def lambda_handler(event, context):
# TODO implement
slots= event["slots"];
drink,qty,unit= slots["Drink"], slots["Quantity"], slots["Unit"]
retStr= "your order of "+qty+" "+unit+ " of "+drink+ " is coming right up!";
return {"dialogAction": {
"type": "Close",
"fulfillmentState": "Fulfilled",
"message": {
"contentType": "PlainText",
"content": retStr
},
}
}
The formats are in accordance with the documentation, however still getting error in processing lambda response. What is the issue?
This error occurs when the execution of the Lambda function fails and throws an error back to Amazon Lex.
I have attempted to recreate your environment using the python code shared and the test input event.
The output format that you have specified in the original post is correct. Your problem appears to lie with the input test event. The input message that you are using differs from what Lex is actually sending to your Lambda function.
Try adding some additional debugging to your Lambda function to log the event that Lex passes into it and then use the logged event as your new test event.
Ensure that you have CloudWatch logging enabled for the Lambda function so that you can view the input message in the logs.
Here's how my Lambda function looks:
import json
import logging
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
def dispatch(event):
# TODO implement
slots= event["slots"];
drink,qty,unit= slots["Drink"], slots["Quantity"], slots["Unit"]
retStr= "your order of "+qty+" "+unit+ " of "+drink+ " is coming right up!";
return {"dialogAction": {
"type": "Close",
"fulfillmentState": "Fulfilled",
"message": {
"contentType": "PlainText",
"content": retStr
},
}}
def lambda_handler(event, context):
logger.debug('event={}'.format(event))
response = dispatch(event)
logger.debug(response)
return response
Now if you test via the Lex console you will find your error in the CloudWatch logs.:
[ERROR] KeyError: 'slots' Traceback (most recent call last): File
"/var/task/lambda_function.py", line 33, in lambda_handler
response = dispatch(event) File "/var/task/lambda_function.py", line 19, in dispatch
slots= event["slots"];
Using this error trace and the logged event, you should see that slots is nested within currentIntent.
You will need to update your code to extract the slot values from the correct place.
Trust this helps you.
I'm trying to create dataset for entity recognition task in Google AutoML with their script to convert my .txt files in .jsonl and save it in Google Cloud Storage as explained in this tutorial. Data looks like (from their example - NCBI Disease Corpus):
"10021369 Identification of APC2, a homologue of the <category="Modifier">adenomatous polyposis coli tumour<\/category> suppressor . "
After uploading in GCS labels are not recognized at all. What format of data is relevant?
I'm not quite sure if <category="Modifier"> should work, but as far as I know, the right way in the Quickstart is annotating in the following way:
{"annotations": [
{"text_extraction": {"text_segment": {"end_offset": 85, "start_offset": 52}}, "display_name": "Modifier"},
{"text_extraction": {"text_segment": {"end_offset": 144, "start_offset": 103}}, "display_name": "Modifier"},
{"text_extraction": {"text_segment": {"end_offset": 391, "start_offset": 376}}, "display_name": "Modifier"},
{"text_extraction": {"text_segment": {"end_offset": 1008, "start_offset": 993}}, "display_name": "Modifier"},
{"text_extraction": {"text_segment": {"end_offset": 1137, "start_offset": 1131}}, "display_name": "SpecificDisease"}],
"text_snippet": {"content": "10021369\tIdentification of APC2, a homologue of the adenomatous polyposis coli tumour suppressor .\tThe ... APC - / - colon
carcinoma cells . Human APC2 maps to chromosome 19p13 . 3. APC and APC2 may therefore have comparable functions in development and cancer .\n "}
}
After importing the dataset, in the AutoML NL UI you will see the five annotations that are specified in the jsonl:
For more reference on the jsonl structure of the example above, you can take a look at the sample files in the Quickstart:
$ gsutil cat gs://cloud-ml-data/NL-entity/dataset.csv
TRAIN,gs://cloud-ml-data/NL-entity/train.jsonl
TEST,gs://cloud-ml-data/NL-entity/test.jsonl
$ gsutil cat gs://cloud-ml-data/NL-entity/train.jsonl
If you are using the python script for your own texts strings, you will see that it generates a csv file (dataset.csv) and jsonl files with content like:
{"text_snippet": {"content": "This is a disease\n Second line blah blabh"}, "annotations": []}
So, you will need to specify the annotations (using start_offset and the end_offset) whose manual process can be a bit overwhelm, or you can upload the CSV file in the AutoML UI and label entities interactively.
I've been searching a long time, but I can't find an answer.
I'm making a script for volumio on my Raspberry Pi
In the terminal, when I type
volumio status
I get exactly
{
"status": "pause",
"position": 0,
"title": "Boom Boom",
"artist": "France Gall",
"album": "Francegall Longbox",
"albumart": "/albumart?cacheid=614&web=France%20Gall/Francegall%20Longbox/extralarge&path=%2FUSB&metadata=false",
"uri": "/Boom Boom.flac",
"trackType": "flac",
"seek": 21192,
"duration": 138,
"samplerate": "44.1 KHz",
"bitdepth": "16 bit",
"channels": 2,
"random": true,
"repeat": null,
"repeatSingle": false,
"consume": false,
"volume": 100,
"mute": false,
"stream": "flac",
"updatedb": false,
"volatile": false,
"service": "mpd"
}
In python, I would like to store this in a dictionary
since it already has the right formatting, I thought that assigning it to a variable will make it a dictionnary right away as follows:
import subprocess, shlex
cmd = "volumio status | sed -e 's/true/True/g' -e 's/false/False/g' -e 's/null/False/g'"
cmd = shlex.split(cmd)
status = subprocess.check_output(cmd)
print status["volume"]
If what I thought was true I would get "100". Instead, I get this error :
File "return.py", line 7, in <module>
print status["volume"]
TypeError: string indices must be integers, not str
this means "status" is stored as a string. Does anybody know how I can make it a dictionary?
dict() doesn't make it, i get :
ValueError: dictionary update sequence element #0 has length 1; 2 is required
Victory! I was able to make my code work with eval()
import subprocess
status = subprocess.check_output("volumio status | sed -e 's/true/True/g' -e 's/false/False/g' -e 's/null/False/g'", shell=True)
status = eval(status)
print status["volume"]
it returns 100
I entered: "python manage.py loaddata fixtures\companies.json". It gives me an error.
What is the proper way to load the json data and then push it to the gae datastore?
Here is my model:
class Company(models.Model):
id = models.AutoField(primary_key=True);
name = models.CharField(max_length=100);
address = models.CharField(max_length=100, null=True);
city = models.CharField(max_length=100);
state = models.CharField(max_length=25);
zip = models.IntegerField();
latitude = models.CharField(max_length=20, null=True)
longitude = models.CharField(max_length=20, null=True)
phone = models.CharField(max_length=15, null=True);
website = models.CharField(max_length=50, null=True);
email = models.EmailField(max_length=50, null=True);
hiring = models.BooleanField(default=False);
approved = models.BooleanField(default=False);
date_added = models.DateTimeField(auto_now_add=True);
about_us = models.TextField(max_length=500, null=True);
Here is my fixture:
[
{
"model": "Companies.Company",
"fields": {
"id": 111,
"name": "1-800 Postcards America's Printing Company",
"address": "121 Varick Street, 7th Floor",
"city": "New York",
"state": "NY",
"zip": 10013,
"latitude": "40.724831999999999",
"longitude": "-74.00609",
"phone": "212-741-1070",
"website": "http://www.1800postcards.com",
"email": "info#1800postcards.com",
"hiring": "FALSE",
"approved": "TRUE",
"date_added": "April 30, 2012, 6:08 p.m.",
"about_us": "With over 30 years of business experience as full service offset printers and over 200 years of combined knowledge in our production department and press rooms, 1-800Postcards is your professional printer. Our state-of-the-art 125,000 square foot facility is located in the heart of New York City, with an additional web offset plant in New Jersey for our larger orders 500,000 and up. This makes it possible for us to print and mail the highest quality postcards on the market today — at the most competitive rates in the country."
}
},
{
"model": "Companies.Company",
"fields": {
"id": 222,
"name": "20x200 ",
"address": "6 Spring Street",
"city": "New York",
"state": "NY",
"zip": 10012,
"latitude": "40.721052999999998",
"longitude": "-73.994322999999994",
"phone": "212-219-0166",
"website": "http://www.20x200.com/",
"email": "hello#20x200.com",
"hiring": "TRUE",
"approved": "TRUE",
"date_added": "April 30, 2012, 6:08 p.m.",
"about_us": "(limited editions × low prices) + the internet = art for everyone. That simple formula, devised by 20x200's founder, Jen Bekman, is what launched 20x200 back in 2007. With that, we set our sights on two core goals: We want everyone to collect art, and we want to enable an economy that allows more artists to make a living by making work. More so, we wanted something as awesome as collecting art to be fun. That's right: fun!"
}
},
{
"model": "Companies.Company",
"fields": {
"id": 333,
"name": "Adafruit",
"address": "80 Nassau st #4C",
"city": "New York",
"state": "NY",
"zip": 10038,
"latitude": "40.709938000000001",
"longitude": "-74.007987999999997",
"phone": "646-248-7822",
"website": "http://www.adafruit.com/",
"email": "N/A",
"hiring": "TRUE",
"approved": "TRUE",
"date_added": "April 30, 2012, 6:08 p.m.",
"about_us": "We pride ourselves on having great prices, the best customer service, support and fast shipping. We hope we can assist you on your journey of learning – At Adafruit we celebrate “Engineering Happiness”! Adafruit is a 100% Woman-Owned Small Business."
}
}
]
Here is the error I get:
Problem installing fixture 'fixtures\companies.json': Traceback (most recent call last):
File "C:\Users\Joe\Desktop\My School Stuff\Spring 2012\CS 473\ny-tech\ny-tech\django\core\management\commands\loaddata.py", line 169, in handle
for obj in objects:
File "C:\Users\Joe\Desktop\My School Stuff\Spring 2012\CS 473\ny-tech\ny-tech\django\core\serializers\json.py", line 35, in Deserializer
for obj in PythonDeserializer(simplejson.load(stream), **options):
File "C:\Program Files (x86)\Google\google_appengine\lib\simplejson\simplejson\__init__.py", line 332, in load
use_decimal=use_decimal, **kw)
File "C:\Program Files (x86)\Google\google_appengine\lib\simplejson\simplejson\__init__.py", line 388, in loads
return _default_decoder.decode(s)
File "C:\Program Files (x86)\Google\google_appengine\lib\simplejson\simplejson\decoder.py", line 402, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Program Files (x86)\Google\google_appengine\lib\simplejson\simplejson\decoder.py", line 418, in raw_decode
obj, end = self.scan_once(s, idx)
File "C:\Program Files (x86)\Google\google_appengine\lib\simplejson\simplejson\scanner.py", line 71, in scan_once
return _scan_once(string, idx)
File "C:\Program Files (x86)\Google\google_appengine\lib\simplejson\simplejson\scanner.py", line 44, in _scan_once
return parse_array((string, idx + 1), _scan_once)
File "C:\Program Files (x86)\Google\google_appengine\lib\simplejson\simplejson\decoder.py", line 284, in JSONArray
value, end = scan_once(s, end)
File "C:\Program Files (x86)\Google\google_appengine\lib\simplejson\simplejson\scanner.py", line 42, in _scan_once
_scan_once, object_hook, object_pairs_hook, memo)
File "C:\Program Files (x86)\Google\google_appengine\lib\simplejson\simplejson\decoder.py", line 230, in JSONObject
value, end = scan_once(s, end)
File "C:\Program Files (x86)\Google\google_appengine\lib\simplejson\simplejson\scanner.py", line 42, in _scan_once
_scan_once, object_hook, object_pairs_hook, memo)
File "C:\Program Files (x86)\Google\google_appengine\lib\simplejson\simplejson\decoder.py", line 230, in JSONObject
value, end = scan_once(s, end)
File "C:\Program Files (x86)\Google\google_appengine\lib\simplejson\simplejson\scanner.py", line 39, in _scan_once
return parse_string(string, idx + 1, encoding, strict)
File "C:\Program Files (x86)\Google\google_appengine\lib\simplejson\simplejson\decoder.py", line 121, in py_scanstring
content = unicode(content, encoding)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x97 in position 484: invalid start byte
Exception AttributeError: "'NoneType' object has no attribute 'mkstemp'" in <bound method DatastoreFileStub.__del__ of <google.appengine.api.datastore_file_stub
.DatastoreFileStub object at 0x02930F10>> ignored
This type of error can occur with invalid JSON. The easiest thing to do is to make sure your JSON validates with JSON Lint, fix any errors it finds, and try the upload again.