I have a client with a guaranteed execution timeout setting here (which can be configured per request)
https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/AmazonWebServiceRequest.html#setSdkClientExecutionTimeout-int-
But I cannot find an equivalent for the SDKV2, sync or async.
I was wondering if anyone in SO or AWS would know about this. Is this an intentional feature drop? Or am I missing some other setting.
Found the solution : https://github.com/aws/aws-sdk-java-v2/pull/657#pullrequestreview-799397170. This is for async clients and at client level. if you want to do request level (for async clients), use the orComplete functionality of completable future instead. https://docs.oracle.com/javase/9/docs/api/java/util/concurrent/CompletableFuture.html#orTimeout-long-java.util.concurrent.TimeUnit-
Related
Has anyone used the sns_publish_operator?
https://airflow.apache.org/docs/stable/_modules/airflow/contrib/operators/sns_publish_operator.html
I am quite new to airflow and am having some issues around setting up the architecture correctly.
I have set up a simple DAG with a data quality check task. Basically, if the dataset fails the data quality checks, I'd like to send an SNS notification. If it passes the data quality checks, I'd like it to reframe from sending an email.
There does not seem to be as much online help in this realm as I thought. Any resources or general tips would be much appreciated.
This question is a bit older, but maybe this helps someone still.
First addressing the SnsPublishOperator: You will need to setup an Airflow connection to AWS. There is multiple ways to do that. The easiest is probably using the Web UI. Go there to Admin->Connections->[+] (add new record). Then you set it up as 'Amazon Webservices Connection' Conn Type. Login and Password are the AWS key and key secret. Finally, you also have to provide the region, where you SNS topic is seated in the 'Extras' section:
{"region_name": "us-east-1"}
Now, you can use the Operator in you code, by providing it also the new connection's conn_id:
my_sns_task = SnsPublishOperator(
task_id='task_name',
target_arn='your_sns_topic_arn',
message='your_message',
aws_conn_id=conn_id
)
Theoretically, that operator also has a "subject" parameter, but I receive an error from the component, when I try to set it.
As for incorporating that operator into your DAG, a possible way would be to have the task, that evaluates your data, fail, if the checks to do not pass, and have the sns task be triggered on failure:
my_sns_task = SnsPublishOperator(
task_id='task_name',
target_arn='your_sns_topic_arn',
message='your_message',
aws_conn_id=conn_id,
trigger_rule='one_failed'
)
my_sns_task.set_upstream(datacheck_task)
I have a serverless backend running on lambda. The runtime usually varies betweeen 40-250s which is over the apigateway max allowed runtime (29s). As such I think my only option is to resort to asynchronous processing. I get the idea behind it, but help online seems sparse and I'd like to know if there are any best practices out there? Or what would be the simplest way for me to get around this timeout problem–using asynchronous processing or other?
It really depends on your use case. But probably an asynchronous approach is best fitted for this scenario given that it's not usually a good idea from the calling side of your API to wait 250 seconds to get the reply back (probably that's why the 29s limitation on API Gateway).
Asynchronous simply means that you will be replying back from Lambda saying that you received the request and you are going to work on it but it will be available only later.
Then, you will be changing the logic on the client side, too, to check back after some time or perform some checks in a loop until the requested resource is ready.
Depending on what work needs to be done you could create an S3 bucket on the fly and reply back to the client with an S3 presigned URL. Then your worker will upload their results to the S3 bucket and the client will poll that bucket for the results until they are present.
I would like to be disable a web service inbound-gateway based on a database flag.
I have achieved this with other inbound adapters (e.g. file) by setting the auto-start attribute based on a SPEL expression which gets the database value.
Any advice on a good way to achieve this? I do not see an auto-start attribute on the ws:inbound-gateway.
Well, looks like it is a bug there around Lifecycle and in case of stopped state we should return to the client something like HttpStatus.SERVICE_UNAVAILABLE.
Please, raise a JIRA ticket on the matter and we will take care about that soon.
I see that <int-http:inbound-gateway> has similar problem, even if auto-startup is exposed there.
As a workaround I see something like ChannelInterceptor on the request-channel for you <int-ws:inbound-gateway>, which checks some variable and throws some exception (NoEndpointFoundException ?) from the preSend() implementation to notify WS client that the service isn't available.
I have a requirement to count the jetty transactions and measure the time it took to process the request and get back the response using JMX for our monitoring system.
I am using Jetty 8.1.7 and I can’t seem to find a proper way to do this. I basically need to identify when request is sent (due to Jetty Async approach this is triggered from thread A) and when the response is complete (as the oncompleteResponse is done in another thread).
I usually use ThreadLocal for such state in other areas I need similar functionality, but obviously this won’t work here.
Any ideas how to overcome?
To use jetty's async requests you basically have to subclass ContentExchange and override its methods. So you can add an extra field to it which would contain a timestamp of when the request was sent, and use it later in your onResponseComplete() method to measure the processing time. If you need to know the time when your request was actually sent to the server instead of when it was created you can override the onRequestCommitted() and onRequestComplete() methods.
I have a situation where I need my API to have a call for triggering a service-side event, no information (besides authentication) is needed from the client, and nothing needs to be returned by the server. Since this doesn't fit well into the standard CRUD/Resource interaction, should I take this as an indicator that I'm doing something wrong, or is there a RESTful design pattern to deal with these conditions?
Your client can just:
POST /trigger
To which the server would respond with a 202 Accepted.
That way your request can still contain the appropriate authentication headers, and the API can be extended in the future if you need the client to supply an entity, or need to return a response with information about how to query the event status.
There's nothing "non-RESTful" about what you're trying to do here; REST principles don't have to correlate to CRUD operations on resources.
The spec for 202 says:
The entity returned with this response SHOULD include an indication of
the request's current status and either a pointer to a status monitor
or some estimate of when the user can expect the request to be
fulfilled.
You aren't obliged to send anything in the response, given the "SHOULD" in the definition.
REST defines the nature of the communication between the client and server. In this case, I think the issues is there is no information to transfer.
Is there any reason the client needs to initiate this at all? I'd say your server-side event should be entirely self-contained within the server. Perhaps kick it off periodically with a cron call?