I am new to AWS SageMaker and i am using this technology for building and training the machine learning models. I have now developed a docker image which contains our custom code for tensorflow. I would like to upload this custom docker image to AWS SageMaker and make use of it.
I have searched various links but could not find proper information on how to upload our own custom docker image.
Can you please suggest me the recommended links regarding the process of uploading our own docker image to AWS SageMaker?
In order to work with sagemaker, you have to push your container to ECR. The most important thing is that the container must be "adapted" to be complaint to what sagemaker requires, but everything is described here. In addition if you want to take a look to an example, here is mine.. where I use my container with TF Object Detection API in AWS Sagemaker.
Related
I have a time series model and I've forecasted using SARIMAX. I'm very new to AWS. I want to deploy this on AWS Sagemaker but SARIMAX is not available directly on AWS sagemaker.
So the one method I read about was to create docker image and push it to ECR. As I don't know docker I'm searching if there are any other methods to do this. Is this the only way?
Can you please tell me if there are any alternatives of how can I do it in sagemaker? Or if I could use lambda?
I have a working setup in AWS that looks something like:
The point is that whenever a file is uploaded to S3, it will trigger a lambda that will trigger a Codebuild project. The codebuild project is then based on a docker image (Stored at ECR) and needs to run a few bash command, mainly executing python files in the docker image. That works really well actually.
The files in S3 are updated approximately once a day and each execution in codebuild takes around 4 minutes.
I got the question why I am not using fargate/SageMaker (the scripts are basicly machine learning retraining and predictions). I was just thinking about if there would be any advantages in using Fargate and/or SageMaker for this? Is it e.g. possible to use Fargate and execute bash commands inside the container when triggered?
IIUC, You're wondering the diffenerce between CodeBuild and Fargate/SageMaker
Price
Calcute the price of these three products using the links below.
Pricing Fargate
Pricing SageMaker
Pricing CodeBuild
As your said, you're using the docker image as the main training tools, so maybe the Fargateis more suitable for your scenario.
Is there any way to load containers stored in docker hub registry in Amazon Sagemaker.
According to some documentation, it should be possible, but I have not been able to find any relevan example or guide for it.
While you can use any registry when working with Docker on a SageMaker notebook, as of this writing other SageMaker components presently only support images from Amazon ECR repositories.
I have created a local image based on CentOS with a specific application running on it. Running the application locally on my machine works fine. I want to take this image and upload it to AWS (preferably to an ElasticBeanstalk instance but I can work with other types).
I cannot upload the image to a image repository like docker hub or other type The application is by a third party and they gave us special permission to make the image and run it on AWS but we can not place the image in a repository.
Is there a way to just export my docker image (I have the saved TAR file) and upload it to AWS and have them use it directly? I've searched and searched and not found anything showing how to do that, all I have found indicate you have to have it in a repository or upload the code to AWS and have it build the image.
Thank you
As AWS CodeBuild provides "Docker" curated images (http://docs.aws.amazon.com/codebuild/latest/userguide/build-env-ref.html),
I'd like to reuse one of those to build my custom CodeBuild Image (http://docs.aws.amazon.com/codebuild/latest/userguide/sample-docker-custom-image.html).
I don't know the CodeBuild Registry Url.
How to reference those images (FROM) ?
I could rebuild myself the image I need (https://github.com/aws/aws-codebuild-docker-images) and host it in a private ECR but it's wasting time and storage.
CodeBuild's internal Docker repository used for Curated Images is not publicly accessible. You should build your own base image from the official Dockerfiles.
The AWS CodeBuild provides also a console where you can use these images. I think they use Amazon EC2 Repository to provide these internally, like the one you can use. but there isn't a public registry (or I already haven't found).
You can find these images to create Docker machines using the build spec defined by Amazon
http://docs.aws.amazon.com/codebuild/latest/userguide/build-spec-ref.html
http://docs.aws.amazon.com/codebuild/latest/userguide/sample-docker.html
I found this public image but is not official:
https://hub.docker.com/r/sandorzelei/aws-codebuild/
So you can use it as follows:
FROM sandorzelei/aws-codebuild