currently trying to process a number of images simultaneously using custom labels via postman. I'm a business client with AWS and have been on hold for over 30 minutes to speak with an engineer but because AWS customer sucks I'm asking the community if they can help. Rather than analyze an image one at a time, is there away to analyze images all at once? Any help would be great, really need it at this time.
Nick
I don't think there is a direct API or SDK by AWS for asynchronous image processing with custom labels.
But the right workaround here can be introducing an event-based architecture yourself.
You can upload images in batch to S3 and configure S3 events to send the event notification to an SNS topic.
You can have your API subscribed to this S3 topic which takes in the object name and bucket name. And then within the API, you have the logic to use custom labels and store results in a Database like DynamoDB. This way, you can process images asynchronously.
Just make sure you have the right inference hours configured so you don't flood your systems thus making them unavailable
Hope this process solves your problem
You can achieve this by using a batch processing solution published by AWS.
Please refer this blog for the solution: https://aws.amazon.com/blogs/machine-learning/batch-image-processing-with-amazon-rekognition-custom-labels/
Also, the solution can be deployed from github where it is published as a AWS Sample: https://github.com/aws-samples/amazon-rekognition-custom-labels-batch-processing. If you are in a region for which the deployment button is not provided, please raise a issue.
Alternatively, you can deploy this solution using SAM. The solution is developed as a AWS Serverless Application Model. So it can be deployed using sam with the following steps:
Install the sam cli - https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html
Download the code repository on your local machine
from within the folder execute the following steps. The folder name is referrenced as sam-app in the below example.
a. #Step 1 - Build your application
i. cd sam-app
ii. sam build
b. #Step 2 - Deploy your application
i. sam deploy --guided
Related
I am trying to understand the correct way to setup my project on AWS so that I ultimately get the possibility to have CI/CD on the lambda functions. And also to ingrain good practices.
My application is quite simple : an API that calls lambda functions based on users' requests.
I have deployed the application using AWS SAM. For that, I used a SAM template that was using local paths to the lambda functions' code and that created the necessary AWS ressources (API Gateway and Lambda). It was necessary to use local paths for the lambda functions because the way SAM works does not allow using existing S3 buckets for S3 events trigger (see here) and I deploy a Lambda function that is watching the S3 bucket to see any updated code to trigger lambda updates.
Now what I have to do is to push my Lambda code on Github. And have a way that Github pushes the lambda functions' code from github to the created S3 bucket during the SAM deploy and the correct prefix. Now what I would like is a way to automatically to that upon Github push.
What is the preferred way to achieve that ? I could not find clear information in AWS documentation. Also, if you see a clear flaw in my process don't hesitate to point it out.
What you're looking to do is a standard CI/CD pipeline.
The steps of your pipeline will be (more or less): Pull code from GitHub -> Build/Package -> Deploy
You want this pipeline to be triggered upon a push to GitHub, this can be done by setting up a Webhook which will then trigger the pipeline.
Last two steps are supported by SAM which I think you have already implemented before, so will be a matter of triggering the same from the pipeline.
These capabilities are supported by most CI/CD tools, if you want to keep everything in AWS you could use CodePipeline which also supports GitHub integration. Nevertheless, Jenkins is perfectly fine and suitable for your use case as well.
There are a lot of ways you can do it. So would depend eventually on how you decide to do it and what tools you are comfortable with. If you want to use native AWS tools, then Codepipeline is what might be useful.
You can use CDK for that
https://aws.amazon.com/blogs/developer/cdk-pipelines-continuous-delivery-for-aws-cdk-applications/
If you are not familiar with CDK and would prefer cloudformation, then this can get you started.
https://docs.aws.amazon.com/codepipeline/latest/userguide/tutorials-github-gitclone.html
I would like to automate setting up the collection of AWS Application Load Balancer logs using Sumo Logic as documented here:
https://help.sumologic.com/07Sumo-Logic-Apps/01Amazon_and_AWS/AWS_Elastic_Load_Balancer_-_Application/01_Collect_Logs_for_the_AWS_Elastic_Load_Balancer_Application_App
This involves creating a bucket, creating a Sumo Logic hosted collector with an S3 source, taking the URL of the collector source provided by Sumo Logic and then creating an SNS Topic with an HTTP subscription where the subscription URL is the one provided by the Sumo Logic source.
The issue with this is that the SumoLogic source URL is not known at synthesis time. The Bucket must be deployed, then the Sumlogic things created, then the SNS topic created.
As best I can figure, I will have to do this through separate invocations of CDK using separate stacks, which is slower. One stack to create the bucket. After deploying that stack, use the Sumo Logic api to create or affirm prior creation of the Sumo Logic hosted collector and source, another CDK deploy to create the SNS topic and HTTP subscription.
I was just wondering if anyone knew of a better way to do this, perhaps some sort of deploy time hook that could be used.
There are two ways(which I know of) in which you can automate the collection of AWS Application Load Balancer.
Using CloudFormation
Sumo Logic have a template that creates the Collection process for AWS Application Load Balancer which is part of the AWS Observability Solution. You can fork the repository and can create your own CloudFormation template after removing resources you do not require.
Sumo Logic also have a Serverless Application which auto enable Access logging for existing and new (which are created after application installation) load balancer. Example template which uses the application.
Using Terraform
As mentioned by Grzegorz, you can create a terraform script also.
Disclaimer: Currently employed by Sumo Logic.
You could try using a Custom Resource SDK Call to trigger a lambda that does what you want.
https://docs.aws.amazon.com/cdk/api/latest/docs/#aws-cdk_custom-resources.AwsSdkCall.html
(I know this is not a perfect answer as it suggests to use another tool, yet I believe it fulfills the needs expressed in the question)
How about using Terraform?
sumologic_s3_source in Terraform is able to create the source at Sumo AND output its URL for other uses within Terraform - e.g. to set up AWS resources.
The docs on this even mention URL being one of the returned values:
url - The HTTP endpoint to use with SNS to notify Sumo Logic of new
files.
Disclaimer: I am currently employed by Sumo Logic.
I'am trying to make an Android application which can send images taken by a camera on an Android tablet to AWS Rekognition. It's the intention that pictures will be directly send to the AWS Rekognition service without the need of an S3 bucket. The picture itself don't need to be saved in the cloud. Only the face metadata need to be stored on AWS in a collection. Afterwards the ultimate goal is for a person to be able to capture his face again, and AWS says there is a match with a previous face in the collection.
There is a lot of information on the internet. But most of the times AWS suggest the Amplify framework. And I don't really know if this is necessary in such simple case.
I have already done all the steps in the AWS CLI (and these work) but I don't succeed to do these steps into Android studio. Below I describe the steps that I have done in the AWS CLI. I would do these steps in Android Studio, but I'm not a pro in programming this language.
(I have already made a collection within the AWS CLI.)
First I index a face which can be found by AWS into a picture. In my AWS CLI code I use S3 as an example. It should be the intention that I can send the picture directly to AWS Rekognition. This action only need to be done if someone specially push a button. So taking a picture and sending it to AWS Rekognition to index the face in a specific collection.
aws rekognition index-faces --image '{"S3Object":{"Bucket":"bucketName","Name":"picture1.jpg"}}' --collection-id "collectionName" --max-faces 1 --quality-filter "AUTO" --detection-attributes "DEFAULT" --external-image-id "picture1.jpg"
Then when a user push another button it need to take a picture again and it need to be send to AWS Rekognition to search the collection by the image which was sent. I have already succeeded this with the following AWS CLI code. It should also be the intention to send the picture directly to AWS without the need of S3. AWS returns a match with a face which is already in the collection.
aws rekognition search-faces-by-image --image '{"S3Object":{"Bucket":"bucketName","Name":"picture.jpg"}}' --collection-id "collectionName"
Again I'am not a professional in Android studio, so it would be very nice if someone haves a quite easy solution. It would also be very nice if someone can tell me if the Amplify framework is really necessary.
Thanks in advance!
You don't have to use Amplify, you can use Rekognition through the AWS Java SDK.
To achieve this same functionality you're getting with the CLI, you can first index the face(s) in the collection using IndexFacesRequest, or you can forego this and create the collection manually over the CLI if this is a one-time operation.
To search the collection's faces by image, you would simply need to modify the following code snippet to pass in a byte-64 encoded image instead of the S3 URL. Full documentation for the searchFacesByImage() method here.
AmazonRekognition client = AmazonRekognitionClientBuilder.standard().build();
SearchFacesByImageRequest request = new SearchFacesByImageRequest().withCollectionId("myphotos")
.withImage(new Image().withS3Object(new S3Object().withBucket("mybucket").withName("myphoto"))).withMaxFaces(5).withFaceMatchThreshold(95f);
SearchFacesByImageResult response = client.searchFacesByImage(request);
For example say I build a workflow that uses 10 lambda functions that trigger each other and are triggered by a dynamodb table and an S3 bucket.
Is there any AWS tool that tracks how these triggers are tying together so I can easily visualize the whole workflow I’ve created?
Bang on, few months ago, I too was in a similar situation for my distributed architecture running on AWS.
So far, I have found the following options as possibilities. I'm still figuring out which is more suitable. But, hope this information helps you.
1. AWS-Native option :: Engineer your Lambda code to trigger Cloudwatch custom-metrics for any important events from within the code. Later, you may use Cloudwatch dashboard to visualize them.
2. Non-AWS options :: There are several of them, but all of them require you to engineer your code with their respective libraries / packages to transmit the needed information. Some of them support ASYNC invocations, so it shouldn't keep your master lambdas in the waiting state for log tracing.
IOPipe
Epsagon
3. Mix of AWS & Non-AWS :: This is a more traditional approach to our problem. You log events to Cloudwatch Logs (like how Lambda does it out of the box), "ingest" these logs into popular log management and analysis SaaS tooling to make sense between these logs via "pattern-matching" and other proprietary techniques.
Splunk Cloud
Datadog
All the best! Keep me posted how it is going.
cheers,
ram
If you use CloudFormation you can visualize the resource relations with CloudFormation Designer. However, if you don't have the resources in a CloudFormation stack, you can create one from all the existing resources.
Some images which is already uploaded on AWS S3 bucket and of course there is a lot of image. I want to edit and replace those images and I want to do it on AWS server, Here I want to use aws lambda.
I already can do my job from my local pc. But it takes a very long time. So I want to do it on server.
Is it possible?
Unfortunately directly editing file in S3 is not supported Check out the thread. To overcome the situation, you need to download the file locally in server/local machine, then edit it and re-upload it again to s3 bucket. Also you can enable versions
For node js you can use Jimp
For java: ImageIO
For python: Pillow
or you can use any technology to edit it and later upload it using aws-sdk.
For lambda function you can use serverless framework - https://serverless.com/
I have made youtube videos long back. This is related to how get started with aws-lambda and serverless
https://www.youtube.com/watch?v=uXZCNnzSMkI
You can trigger a Lambda using the AWS SDK.
Write a Lambda to process a single image and deploy it.
Then locally use the AWS SDK to list the images in the bucket and invoke the Lambda (asynchronously) for each file using invoke. I would also save somewhere which files have been processed so you can continue if something fails.
Note that the default limit for Lambda is 1000 concurrent executions, so to avoid reaching the limit you can send messages to an SQS queue (which then triggers the Lambda) or just retry when invoke throws an error.