Ours is a Spring-Boot based application. For integration with AWS SNS and SQS, we have couple of options:
Use Spring-Cloud-AWS
Use AWS-SDK-Java 2
I wanted to know if there is any advantage in using one or the other.
When I ask AWS guys, they tell me that AWS SDK gets updated regularly and integration with SNS and SQS is not difficult. Hence, there is no need to integrate with Spring-Cloud-AWS.
I tried searching on gitter channel for Spring-Cloud and could not find any relevant information. Documentation does state that I can update the AWS-SDK version. Documentation does not state any compelling reason for not using AWS-SDK directly.
If anyone has some insights, please share.
From the AWS Spring Team:
"From now on, our efforts focus on 3.0 - based on AWS SDK 2.0."
So, if you need AWS SDK 2.0, you probably want to go directly with the SDK.
https://spring.io/blog/2021/03/17/spring-cloud-aws-2-3-is-now-available
For more on what's new on AWS Java SDK 2.0:
https://aws.amazon.com/blogs/developer/aws-sdk-for-java-2-0-developer-preview/
The main advantage over the AWS Java SDK is the Spring style convenience and head start we get by using the Spring project. As per the project documentation (https://cloud.spring.io/spring-cloud-aws/reference/html/##using-amazon-web-services)
Using the SDK, application developers still have to integrate the SDK
into their application with a considerable amount of infrastructure
related code. Spring Cloud AWS provides application developers already
integrated Spring-based modules to consume services and avoid
infrastructure related code as much as possible.
I am trying to achieve automatic instrumentation of all calls made by AWS SDKs for Java using X-Ray.
The X-Ray SDK for Java automatically instruments all AWS SDK clients when you include the AWS SDK Instrumentor submodule in your build dependencies.
(from the documentation)
I have added these to my POM
aws-xray-recorder-sdk-core
aws-xray-recorder-sdk-aws-sdk
aws-xray-recorder-sdk-spring
aws-xray-recorder-sdk-aws-sdk-instrumentor
and am using e.g. aws-java-sdk-ssm and aws-java-sdk-sqs.
I expected to only have to add the X-Ray packages to my POM and provide adequate IAM policies.
However, when I start my application I get exceptions such as these:
com.amazonaws.xray.exceptions.SegmentNotFoundException: Failed to begin subsegment named 'AWSSimpleSystemsManagement': segment cannot be found.
I tried wrapping the SSM call in a manual segment and so that worked but then immediately the next call from another AWS SDK throws a similar exception.
How do I achieve the automatic instrumentation mentioned in the documentation? Am I misunderstanding something?
It depends on how you make AWS SDK calls in your application. If you have added X-Ray servlet to your spring application per https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-java-filters.html then each time your application receives a request, the X-Ray servlet filter will open a segment and store it in the thread serving that request. Any AWS SDK calls you make as part of that request/response cycle will pick up that segment as the parent.
The error you got means the the X-Ray instrumentor tries to record the AWS API call to a subsegment but it cannot find a parent (which request this call belongs to).
Depending on your use case you might want to explicitly instrument certain AWS SDK clients and left others plain, if some of those clients are making calls in a background worker.
I have just started learning AWS Simple workflow service and wrote a workflow using AWS Flow framework for Java. I am able to execute the workflow successfully from eclipse. But for my requirement, I need to execute it from my web application back-end which I am planning to write using Nodejs. I found AWS Restful service for SWF but not sure if it will work with flow framework. So please help me in taking the right approach.
So basically my question is how can I execute workflow starter and workers from web back-end?
You can use A Node.js library for accessing Amazon Simple Workflow. But this library is pretty low level comparing to AWS Flow Framework making writing complex workflows really non trivial.
You might consider only starting workflows and implementing activities using Javascript and implementing workflow decider using Java AWS Flow Framework.
What should I do to profile worflows exposed as Windows Workflow services? Which tool did you use?
I have tried to use dotTrace (jetBrains): I can see data in the profiling snapshot, but it seems I cannot see methods called by workflows.
Depending on the information you want to get out of it you can use AppFabric. Once installed you can go into IIS and set monitoring to "Troubleshooting" and get back pretty much everything the workflow has done.
I'm quite interested in beginning some development using Amazon SQS, perhaps SimpleDB too, my question is this, are there any open source solutions that mimic the functionality, just for the purposes of development. I've already encountered the Eucalyptus project (http://open.eucalyptus.com) for creating an EC-esque cloud.
I've not had any success with google, I suspect it's because the cost of entry is so inexpensive, but still, does anyone know of anything like this?
For SQS I wrote ElasticMQ, which you can run either embedded (it's written in Scala, so runs on the JVM) or stand-alone. It has both persistent and in-memory modes, the first being good for dev, second for testing.
If you need a test double for more than just SQS, you can try LocalStack.
To simulate SQS, it internally uses ElasticMQ mentioned by adamw.
You can start LocalStack via Docker, for example, and it will start the following services:
API Gateway at http://localhost:4567
Kinesis at http://localhost:4568
DynamoDB at http://localhost:4569
DynamoDB Streams at http://localhost:4570
Elasticsearch at http://localhost:4571
S3 at http://localhost:4572
Firehose at http://localhost:4573
Lambda at http://localhost:4574
SNS at http://localhost:4575
SQS at http://localhost:4576
Redshift at http://localhost:4577
ES (Elasticsearch Service) at http://localhost:4578
SES at http://localhost:4579
Route53 at http://localhost:4580
CloudFormation at http://localhost:4581
CloudWatch at http://localhost:4582
SSM at http://localhost:4583
Some of the Amazon SDKs have "mock" mode, which is:
The mock service is an alternate way
to use the sample code. The service
doesn't call AWS, but instead returns
a set response that you can modify to
suit your needs (the XML response
files are in the Mock directory). The
mock service makes it easy for you to
test how your application handles
different responses.
For SQS, it appears the Perl and PHP SDKs have mock mode. I know that the .NET SDK for Amazon RDS also has the mock mode.
The Java SDK doesn't contain mock implementations:
The client mock implementations have been removed. Instead, developers
are encouraged to use more flexible and full featured mock libraries,
such as EasyMock, jMock
If the SDK you will be using doesn't have the mock mode available, you could probably create your own similar type of thing which returns the preconfigured responses instead of actually hitting up the service.
See here for more info
GoAws - https://github.com/p4tin/goaws - was just released as beta. (disclaimer - I am the developer).
Regarding the Java SDK, it does no longer contain mock implementations:
The client mock implementations have been removed. Instead, developers
are encouraged to use more flexible and full featured mock libraries,
such as EasyMock, jMock
If you are in .NET or Mono you can try Stratosphere. It has local implementations that mimic SimpleDB, SQS and S3. For SimpleDB mock implementation it uses SQLite, for SQS and S3 it stores messages/objects in file system.
if you need to simulate SNS as well as SQS you can check out: Yopa