I have uploaded 22 images and tagged with 2 tags.
But when I click start training I get an "Invalid arguments of request".
The images have been uploaded through the interface. I had to manually create the bucket.
What can have gone wrong? I have attached a screenshot.
This is likely the same error as in Google AutoML training error
Each image is assigned for one of TRAIN, VALIDATION, and TEST set. You have enough labeled images, but not enough images assigned for VALIDATION or TEST. Adding images for these two sets should solve this issue.
The best way of adding images to specified sets is importing a CSV file.
According to Preparing your training data
If you use csv file to import data, you need to assign the content to specific data set. (You need to upload your file to GCS first)
Ex.
TRAIN, gs://{YOUR_BUCKET_NAME}/{OBJECT_NAME_1}, {LABLE}
VALIDATION, gs://{YOUR_BUCKET_NAME}/{OBJECT_NAME_2}, {LABLE}
TEST, gs://{YOUR_BUCKET_NAME}/{OBJECT_NAME_3}, {LABLE}
or do not assign the data set, AutoML automatically places it in one of the three sets
gs://{YOUR_BUCKET_NAME}/{OBJECT_NAME_1}, {LABLE}
gs://{YOUR_BUCKET_NAME}/{OBJECT_NAME_2}, {LABLE}
gs://{YOUR_BUCKET_NAME}/{OBJECT_NAME_3}, {LABLE}
Related
I'm trying to do a quick PoC on the AWS Rekognition custom labels feature. I'd like to try using it for object detection.
I've had a couple of attempts at setting it up using only tools in the AWS Console. I'm using images imported from the Rekognition bucket in S3, then I added bounding boxes using the tools in the Rekognition console.
All of my images are marked up with bounding boxes, no whole image labels have been used. I have 9 labels, all of which appear in at least 1 drawing.
I've ensured my images are less than 4096x4096 in size (which is mentioned on this AWS forums thread as a possible cause of this issue.
When I attempt to train my model I get the "The manifest file contains too many invalid data objects" error.
What could be wrong here? An error message complaining about the format of a file I didn't create manually, that I can't see or edit isn't exactly intuitive.
Is there a way to reference other collections from a specific collection. For example, if I have a file upload collection (something that uploads a file), I want to be able to use that from other collections. How would I reference the file upload?
Here's an example of what I'm talking about.
I have a collection where a file is uploaded and a calculation needs to be performed. The test or collection would go something like this where each step is a POST, GET, etc
Upload and run calculation:
Generate a token
make call
copy/save token value
Upload specific file (these would be 3 individual requests)
Upload file
Monitor upload status
Return ID of file uploaded
run calculation
use ID to pass as parameter
pass other values to set up calculation
monitor run
validate results
In another collection I need to validate uploaded files metadata is correct. Not directly related to the one above, but has some similarities
Generate a token
make call
copy/save token value
Upload specific file (these would be 3 individual requests)
Upload file
Monitor upload status
Get final result and return ID of file uploaded
Get me
validate metadata is correct.
Steps 1 and 2 are common functionality, there would be no difference there. How could I extract those two steps as modular components or functionality so I can reference them from any collection?
For additional clarity, we use ReadyAPI and are able to do 'Run Test Case' which can obviously run another test case. We've separated the functionality of token and file uploads into it's own test case and use it as a modular component. I'd like to achieve something similar with Postman.
Unfortunately Postman collections are working a little bit different.
But you can Merge your two collections to a single one, and execute it as one single collections.
I have a DynamoDB table filled with nice data. I use Datapipeline to extract this to S3 and it generates a folder with 3 files.
1) "139xx-x911-407x-83xx-06x5x659xx16" that contains all DB data in this format:
{"TimeStamp":{"s":"1539699960"},"SystemID":{"n":"1001"},"AccMin":{"n":"497"},"AccMax":{"n":"509"},"CustomerID":{"n":"10001"},"SensorID":{"n":"101"}}
2) "manifest"
{"name":"DynamoDB-export","version":3,
entries: [
{"url":"s3://cxxxx/2018-10-18-15-25-02/139xx-x911-407x-83xx-06x5x659xx16","mandatory":true}
]}
3) "_SUCCESS" No data inside.
I then go to SageMaker -> Training Jobs -> Create Training Job. Here I fill in everything to create a Random Cut Forest model, and point it towards the above data (I have tried both manifest file and the bigger data-file.
The training fails with error:
"ClientError: No data was found. Please make sure training data is
provided."
What am I doing wrong?
Thank you for your interest in SageMaker.
The manifest is optional, but if provided it should conform to the schema described at https://docs.aws.amazon.com/sagemaker/latest/dg/API_S3DataSource.html . Also, RandomCutForest does not support input data in JSON format. Only protobuf and CSV are supported, see https://docs.aws.amazon.com/sagemaker/latest/dg/randomcutforest.html
In order to get training working you have to convert input data to CSV or protobuf format and set content_type value appropriately. If you want to use a manifest file, then S3 location should point to that file and context has to be fixed to conform the schema. You can however remove the manifest and point S3 location to s3://bucket/path/to/data/.
I hope this helps.
Regards,
Yury
I've submitted a training job to the cloud using the RESTful API and see in the console logs that it completed successfully. In order to deploy the model and use it for predictions I have saved the final model using tf.train.Saver().save() (according to the how-to guide).
When running locally, I can find the graph files (export-* and export-*.meta) in the working directory. When running on the cloud however, I don't know where they end up. The API doesn't seem to have a parameter for specifying this, it's not in the bucket with the trainer app, and I can't find any temporary buckets on the cloud storage created by the job.
When you set up your Cloud ML environment you set up a bucket for this purpose. Have you looked in there?
https://cloud.google.com/ml/docs/how-tos/getting-set-up
Edit (for future record): As Robert mentioned in comments, you'll want to pass the output location to the job as an argument. Couple of things to be mindful of:
Use a unique output location per job, so one job doesn't clobber over the outputs of another.
The recommendation is to specify the parent output path, and use it to contain the exported model in a subpath called 'model', as well as organizing other outputs like checkpoints and summaries within that path. That makes it easier to manage all the outputs.
While not required, I'll also suggest staging the training code in a packages subpath of the output, which helps correlate the source with the outputs it produces.
Finally(!), also keep in mind when you use hyperparameter tuning, you'll need to append the trial id to the output path for outputs produced by individual runs.
I'm having problems when uploading lots of files in Django. The context is the following: I've a spreadsheet with one or more columns being image filenames; those images are being uploaded through an form with input type=file and the option multiple.
With few lines - say 70, everything goes fine. But with more lines, and consequently more images, there's a IOError happening in random positions.
I've checked several questions about file/image upload in Django but couldn't find any that is related to my problem.
The model I'm using is the Product model of LFS (www.getlfs.com). We are developing a system that is based on LFS and to facilitate the creation of dozens of products in batch we wrote some views and templates to receive the main product properties through a spreadsheet. Each line is a product and the columns are the desired properties.
LFS uses a custom class ImageWithThumbsField(ImageField) to store the product's image and when saving the product instance (got from the spreadsheet), all thumbnails are generated. This is a time (cpu) consuming task, and my initial guess is that for some reason the temporary file is deleted before all processing had occurred.
Is there a way to keep these uploaded files for more time? Any other approach suggested to be able to process hundreds of uploaded files? Any hints on what can be happening?
Hope you can understand my question. I can post code if need.
Links to relevant portions of LFS code:
where thumbnails are generated:
https://github.com/diefenbach/django-lfs/blob/master/lfs/core/fields/thumbs.py
product model
https://github.com/diefenbach/django-lfs/blob/master/lfs/catalog/models.py
Thanks in advance!
It sounds like you are running out of memory. When django processess uploads, until the form is validated all of the files are either:
kept in memory inside the python/wsgi process/worker. (Usual mode of op for runserver)
In this case, you are uploading enough photos to fill up the process memory and running out of space. This will be non-deterministic as to where the IOError happens as you can imagine (GC Dependent).
Temporarily stored in /tmp/ (usual setup of apache)
In this case, the webserver's ramfs is full of images that have not yet been written to disk. In this case it should IOError arround the same place.
In either case, you should not be bulk uploading images in this way anyway. Apache/Django is not designed for it. Try uploading a single product/image per request/response, and all your problems will go away.