GCP Pub/Sub: Acknowledgement deadline doesn't work when reading messages - google-cloud-platform

Creating a subscription on GCP Pub Sub with "Acknowledgement deadline" set as its max value (600 seconds), my java client in spring boot, keeps receiving messages every 60 seconds in case the task is still running.
pub/sub subscription
We have a simple consumer very similar to this:
https://spring.io/guides/gs/messaging-gcp-pubsub/
We need to perform operations which can run for some minutes each, but even setting 600 seconds, if the task hasn't finished after 60 seconds, the message arrives again.
Doesn't someone experienced something similar?
Thanks
Update:
These are the main dependencies:
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.1.10.RELEASE</version>
</parent>
<spring-cloud-gcp.version>1.2.6.RELEASE</spring-cloud-gcp.version>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-gcp-starter-pubsub</artifactId>
</dependency>
And this is the setup in the consumer (it's exactly as the spring boot guide linked above explains):
#Bean
public PubSubInboundChannelAdapter messageChannelAdapter(
#Qualifier("pubsubInputChannel") MessageChannel inputChannel,
PubSubTemplate pubSubTemplate) {
LOGGER.info("pubsubInputChannel");
PubSubInboundChannelAdapter adapter =
new PubSubInboundChannelAdapter(pubSubTemplate, subscription);
adapter.setOutputChannel(inputChannel);
adapter.setAckMode(AckMode.AUTO_ACK);
return adapter;
}
#Bean
public MessageChannel pubsubInputChannel() {
return new DirectChannel();
}
#Bean
#ServiceActivator(inputChannel = "pubsubInputChannel")
public MessageHandler messageReceiver() {
return message -> {
String json = new String((byte[]) message.getPayload());
LOGGER.info("Message arrived! Payload: " + json);
//Main code here
//Operations might take some minutes to be finished
};
}

Related

PoolingHttpClientConnectionManager - TTL Constructor & related issues

I was creating HttpComponentsMessageSender bean as below
#Bean
public HttpComponentsMessageSender reservationHttpComponent() {
HttpComponentsMessageSender httpComponentsMessageSender = new HttpComponentsMessageSender();
httpComponentsMessageSender.setConnectionTimeout(reservationConnectionTimeOut);
httpComponentsMessageSender.setReadTimeout(reservationReadTimeOut);
return httpComponentsMessageSender;
}
Here I was getting Intermittent Read Time Out Issue - Like if I try for 1st time after a 30 mins break I was getting read time out, afterwards all further transactions are successful. If again take 30 mins break then again 1st transaction failed with read time out issue and then all further are successful...
I tried fixing as below code -
#Bean
public HttpComponentsMessageSender reservationHttpComponent() {
RequestConfig requestBuilder = RequestConfig.custom()
.setSocketTimeout(reservationReadTimeOut)
.setConnectionRequestTimeout(reservationConnectionTimeOut)
.setConnectTimeout(reservationConnectionTimeOut)
.setCircularRedirectsAllowed(false)
.build();
org.apache.http.client.HttpClient httpClient = HttpClientBuilder.create()
.setConnectionManager(getConnManager())
.addInterceptorFirst(new HttpComponentsMessageSender.RemoveSoapHeadersInterceptor())
.setDefaultRequestConfig(requestBuilder)
.build();
HttpComponentsMessageSender messageSender = new HttpComponentsMessageSender();
messageSender.setHttpClient(httpClient);
return messageSender;
}
private PoolingHttpClientConnectionManager getConnManager() {
PoolingHttpClientConnectionManager connectionManager = new PoolingHttpClientConnectionManager();
connectionManager.setDefaultMaxPerRoute(connectionManagerDefaultMaxPerRoute);
connectionManager.setMaxTotal(connectionManagerMaxTotal);
connectionManager.setDefaultSocketConfig(SocketConfig.custom()
.setSoTimeout(reservationReadTimeOut).build());
return connectionManager;
}
Above has resolved the Read time out issue but then I have started getting below issue-
message:Error Occurred While Retrieving Reservation - I/O error: Timeout waiting for connection from pool; nested exception is org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
I am looking for the fix and then I tried initializing PoolingHttpClientConnectionManager with TTL -
PoolingHttpClientConnectionManager connectionManager = new PoolingHttpClientConnectionManager(1, TimeUnit.MINUTES)
Then I ran multiple performance jmx script where I have fired 70 - 80 TPS on my service that have above code for external dependency and all looks good.
However I am not sure if having TTL in PoolingHttpClientConnectionManager constructor is the very appropriate solution.. So here I am looking for suggestions on if this solution can cause any further issue or any this else could that be a better approach than this.

Azure Webjob, KeyVault Configuration extension, Socket Error

Need some help to determine if this is a bug in my code or in the config kevault extensions.
I have a netcore console based webjob. all working fine until a few weeks ago when we stated getting occasional startup errors which were Socket Error 10060 - Socket timed out or "A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond"
These were all related to loading configuration layers (app settings, env, command line and keyvault). The errors stemmed from the keyvault once the build was executed on the hostbuilder.
I initially added the retry policy with the default HttpStatusCodeErrorDetectionStrategy and an exponential back-off but this is not executing.
finally I added my own retry policy with my own detection strategy (see below). Still not being fired.
I have stripped down the code to a hello world like example and included the messages from the webjob.
Here is the code summary:
Main
public static async Task<int> Main(string[] args)
{
var host = CreateHostBuilder(args)
.UseConsoleLifetime()
.Build();
using var serviceScope = host.Services.CreateScope();
var services = serviceScope.ServiceProvider;
//**stripped down to logging just for debug
var loggerFactory = host.Services.GetRequiredService<ILoggerFactory>();
var logger = loggerFactory.CreateLogger("Main");
logger.LogDebug("Hello Test App Started OK. Exiting.");
//**Normally lots of service calls go here to do real work**
return 0;
}
HostBuilder - why hostbuilder? We use lots of components that are built for webapi and webapps so it was convenient to use a similar services model.
public static IHostBuilder CreateHostBuilder(string[] args)
{
var host = Host
.CreateDefaultBuilder(args)
.ConfigureAppConfiguration((ctx, config) =>
{
//override with keyvault
var azureServiceTokenProvider = new AzureServiceTokenProvider(); //this is awesome - it will use MSI or Visual Studio connection
var keyVaultClient = new KeyVaultClient(new KeyVaultClient.AuthenticationCallback(azureServiceTokenProvider.KeyVaultTokenCallback));
var retryPolicy = new RetryPolicy<ServerErrorDetectionStrategy>(
new ExponentialBackoffRetryStrategy(
retryCount: 5,
minBackoff: TimeSpan.FromSeconds(1.0),
maxBackoff: TimeSpan.FromSeconds(16.0),
deltaBackoff: TimeSpan.FromSeconds(2.0)
)
);
retryPolicy.Retrying += RetryPolicy_Retrying;
keyVaultClient.SetRetryPolicy(retryPolicy);
var prebuiltConfig = config.Build();
config.AddAzureKeyVault(prebuiltConfig.GetSection("KeyVaultSettings").GetValue<string>("KeyVaultUri"), keyVaultClient, new DefaultKeyVaultSecretManager());
config.AddCommandLine(args);
})
.ConfigureLogging((ctx, loggingBuilder) => //note - this is run AFTER app configuration - whatever the order it is in.
{
loggingBuilder.ClearProviders();
loggingBuilder
.AddConsole()
.AddDebug()
.AddApplicationInsightsWebJobs(config => config.InstrumentationKey = ctx.Configuration["APPINSIGHTS_INSTRUMENTATIONKEY"]);
})
.ConfigureServices((ctx, services) =>
{
services
.AddApplicationInsightsTelemetry();
services
.AddOptions();
});
return host;
}
Event - this is never fired.
private static void RetryPolicy_Retrying(object sender, RetryingEventArgs e)
{
Console.WriteLine($"Retrying, count = {e.CurrentRetryCount}, Last Exception={e.LastException}, Delay={e.Delay}");
}
Retry Policy - only fires for the non-MSI attempt to contact the keyvault.
public class ServerErrorDetectionStrategy : ITransientErrorDetectionStrategy
{
public bool IsTransient(Exception ex)
{
if (ex != null)
{
Console.WriteLine($"Exception {ex.Message} received, {ex.GetType()?.FullName}");
HttpRequestWithStatusException httpException;
if ((httpException = ex as HttpRequestWithStatusException) != null)
{
switch(httpException.StatusCode)
{
case HttpStatusCode.RequestTimeout:
case HttpStatusCode.GatewayTimeout:
case HttpStatusCode.InternalServerError:
case HttpStatusCode.ServiceUnavailable:
return true;
}
}
SocketException socketException;
if((socketException = (ex as SocketException)) != null)
{
Console.WriteLine($"Exception {socketException.Message} received, Error Code: {socketException.ErrorCode}, SocketErrorCode: {socketException.SocketErrorCode}");
if (socketException.SocketErrorCode == SocketError.TimedOut)
{
return true;
}
}
}
return false;
}
}
WebJob Output
[SYS INFO] Status changed to Initializing
[SYS INFO] Run script 'run.cmd' with script host - 'WindowsScriptHost'
[SYS INFO] Status changed to Running
[INFO]
[INFO] D:\local\Temp\jobs\triggered\HelloWebJob\42wj5ipx.ukj>dotnet HelloWebJob.dll
[INFO] Exception Response status code indicates server error: 401 (Unauthorized). received, Microsoft.Rest.TransientFaultHandling.HttpRequestWithStatusException
[INFO] Exception A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. received, System.Net.Http.HttpRequestException
[ERR ] Unhandled exception. System.Net.Http.HttpRequestException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
[ERR ] ---> System.Net.Sockets.SocketException (10060): A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
[ERR ] at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
[ERR ] --- End of inner exception stack trace ---
[ERR ] at Microsoft.Rest.RetryDelegatingHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
[ERR ] at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
[ERR ] at Microsoft.Azure.KeyVault.KeyVaultClient.GetSecretWithHttpMessagesAsync(String vaultBaseUrl, String secretName, String secretVersion, Dictionary`2 customHeaders, CancellationToken cancellationToken)
[ERR ] at Microsoft.Azure.KeyVault.KeyVaultClientExtensions.GetSecretAsync(IKeyVaultClient operations, String secretIdentifier, CancellationToken cancellationToken)
[ERR ] at Microsoft.Extensions.Configuration.AzureKeyVault.AzureKeyVaultConfigurationProvider.LoadAsync()
[ERR ] at Microsoft.Extensions.Configuration.AzureKeyVault.AzureKeyVaultConfigurationProvider.Load()
[ERR ] at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers)
[ERR ] at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
[ERR ] at Microsoft.Extensions.Hosting.HostBuilder.BuildAppConfiguration()
[ERR ] at Microsoft.Extensions.Hosting.HostBuilder.Build()
[ERR ] at HelloWebJob.Program.Main(String[] args) in C:\Users\mark\Source\Repos\HelloWebJob\HelloWebJob\Program.cs:line 21
[ERR ] at HelloWebJob.Program.<Main>(String[] args)
[SYS INFO] Status changed to Failed
[SYS ERR ] Job failed due to exit code -532462766
This is an issue in the KV connectivity which is identified by the PG. Below is an official statement from Product Group:
The Microsoft Azure App Service Team has identified an issue with the
Key Vault references for App Service and Azure Functions feature
related to intermittent failure to resolve references at runtime.
Engineers identified a regression in the system that reduced the
performance and availability of our scale unit’s ability to retrieve
key vault references at runtime. A patch has been written and deployed
to our fleet of VMs to mitigate this issue.
We are continuously taking steps to improve the Azure Web App service
and our processes to ensure such incidents do not occur in the future,
and in this case, it includes (but is not limited to): Improving
detection and testing of performance and availability of the Key Vault
App Setting References feature Improvements to our platform to ensure
high availability of this feature at runtime. We apologize for any
inconvenience.
For almost everyone, updating packages to the new Microsoft.Azure packages has mitigated this issue, so trying those would be my first suggestion.
Thanks #HarshitaSingh-MSFT, makes sense though I searched for this when I had the problem and couldn't find it.
As a work around, I added some basic retry code to the startup.
Main looks like this for now:
public static async Task<int> Main(string[] args)
{
IHost host = null;
int retries = 5;
while (true)
{
try
{
Console.WriteLine("Building Host...");
var hostBuilder = CreateHostBuilder(args)
.UseConsoleLifetime();
host = hostBuilder.Build();
break;
}
catch (HttpRequestException hEx)
{
Console.WriteLine($"HTTP Exception in host builder. {hEx.Message}, Name:{hEx.GetType().Name}");
SocketException se;
if ((se = hEx.InnerException as SocketException) != null)
{
if (se.SocketErrorCode == SocketError.TimedOut)
{
Console.WriteLine($"Socket error in host builder. Retrying...");
if (retries > 0)
{
retries--;
await Task.Delay(5000);
host?.Dispose();
}
else
{
throw;
}
}
else
{
throw;
}
}
}
}
using var serviceScope = host.Services.CreateScope();
var services = serviceScope.ServiceProvider;
var transferService = services.GetRequiredService<IRunPinTransfer>();
var result = await transferService.ProcessAsync();
return result;
}

Amazon AppConfig from Spring Boot

How can I access configurations from aws appconfig, in my spring boot application?
Since appconfig is a new service, is there any java sdk that we can use, cos i dont see anything for appconfig yet in https://github.com/aws/aws-sdk-java/tree/master/src/samples
Here's how I've integrated AWS AppConfig into my Spring Boot project.
First, let’s make sure we have this dependency in our pom.xml:
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-appconfig</artifactId>
<version>1.12.134</version>
</dependency>
Next, let’s create a simple configuration class of our own AWS AppConfig Client:
#Configuration
public class AwsAppConfiguration {
private static final Logger LOGGER = LoggerFactory.getLogger(AwsAppConfiguration.class);
private final AmazonAppConfig appConfig;
private final GetConfigurationRequest request;
public AwsAppConfiguration() {
appConfig = AmazonAppConfigClient.builder().build();
request = new GetConfigurationRequest();
request.setClientId("clientId");
request.setApplication("FeatureProperties");
request.setConfiguration("JsonProperties");
request.setEnvironment("dev");
}
public JSONObject getConfiguration() throws UnsupportedEncodingException {
GetConfigurationResult result = appConfig.getConfiguration(request);
String message = String.format("contentType: %s", result.getContentType());
LOGGER.info(message);
if (!Objects.equals("application/json", result.getContentType())) {
throw new IllegalStateException("config is expected to be JSON");
}
String content = new String(result.getContent().array(), "ASCII");
return new JSONObject(content).getJSONObject("feature");
}
}
Lastly, let’s create a scheduled task that polls the configuration from AWS AppConfig:
#Configuration
#EnableScheduling
public class AwsAppConfigScheduledTask {
private static final Logger LOGGER = LoggerFactory.getLogger(AwsAppConfigScheduledTask.class);
#Autowired
private FeatureProperties featureProperties;
#Autowired
private AwsAppConfiguration appConfiguration;
#Scheduled(fixedRate = 5000)
public void pollConfiguration() throws UnsupportedEncodingException {
LOGGER.info("polls configuration from aws app config");
JSONObject externalizedConfig = appConfiguration.getConfiguration();
featureProperties.setEnabled(externalizedConfig.getBoolean("enabled"));
featureProperties.setLimit(externalizedConfig.getInt("limit"));
}
}
I came across this question, as I was also trying to figure out how to best integrate AWS AppConfig into Spring Boot.
Here's an article I created. You can visit it here: https://levelup.gitconnected.com/create-features-toggles-using-aws-appconfig-in-spring-boot-7454b122bf91
Also, the source code is available on github: https://github.com/emyasa/medium-articles/tree/master/aws-spring-boot/app-config
First I added dependencies
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>appconfig</artifactId>
<version>2.18.41</version>
</dependency>
<!-- https://mvnrepository.com/artifact/software.amazon.awssdk/appconfigdata -->
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>appconfigdata</artifactId>
<version>2.19.4</version>
</dependency>
Then build the client
client = AppConfigDataClient.builder()
.credentialsProvider(() -> AwsBasicCredentials.create("<your id>", "your secret key"))
.region(Region.<your region>)
.build();
Use the client to start the configuration session
StartConfigurationSessionRequest startConfigurationSessionRequest = StartConfigurationSessionRequest.builder()
.applicationIdentifier("<your application id>")
.environmentIdentifier("your environment id")
.configurationProfileIdentifier("your config id")
.build();
Get the session token in the beginning and use it for the initial call.
String sessionToken = client.startConfigurationSession(startConfigurationSessionRequest).initialConfigurationToken();
GetLatestConfigurationRequest latestConfigurationRequest = GetLatestConfigurationRequest.builder()
.configurationToken(sessionToken)
.build();
GetLatestConfigurationResponse latestConfigurationResponse = client.getLatestConfiguration(latestConfigurationRequest);
String response = latestConfigurationResponse.configuration().asUtf8String();
You can use the next token available in the response to make the next call. The token can be cached as required.
2023, use aws-java-sdk-appconfigdata
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-appconfigdata</artifactId>
<version>1.12.394</version>
</dependency>
example:
#Slf4j
#Configuration
#EnableScheduling
public class AWSAppConfig {
private String token;
private final AWSAppConfigData client;
public AWSAppConfig() {
log.info("init app config");
var client = AWSAppConfigDataClient.builder().build();
var request = new StartConfigurationSessionRequest();
request.setEnvironmentIdentifier("prod");
request.setApplicationIdentifier("my-app");
request.setConfigurationProfileIdentifier("my-config");
request.setRequiredMinimumPollIntervalInSeconds(15);
var result = client.startConfigurationSession(request);
this.client = client;
this.token = result.getInitialConfigurationToken();
}
#Scheduled(fixedRate = 20000)
public void pollConfiguration() {
var request = new GetLatestConfigurationRequest();
request.setConfigurationToken(token);
var result = client.getLatestConfiguration(request);
this.token = result.getNextPollConfigurationToken();
var configuration = StandardCharsets.UTF_8.decode(result.getConfiguration()).toString();
log.info("content type: {}", result.getContentType());
log.info("configuration: {}", configuration);
}
}

SQSlistener not receiving messages

I am able to send messages to SQS queue from my springboot but not able to receive using sqslistener annotation, can someone help?
public void send(String message) {
queueMessagingTemplate.convertAndSend("test-queue", MessageBuilder.withPayload(message).build());
}
#SqsListener(value = "test-queue", deletionPolicy = SqsMessageDeletionPolicy.NEVER)
public void receive(String message)
{
System.out.println("message: " + message);
}
I have verified send by goign to AWS console, i can see my messages in queue, but they are not coming to receive method.
config:
#Bean
public AmazonSQSAsyncClient amazonSQSAsyncClient()
{
AmazonSQSAsyncClient amazonSQSAsyncClient= new AmazonSQSAsyncClient(amazonAWSCredentials());
if (!StringUtils.isEmpty(amazonSqsEndpoint)) {
amazonSQSAsyncClient.setEndpoint(amazonSqsEndpoint);
}
}
#Bean
public SimpleMessageListenerContainerFactory simpleMessageListenerContainerFactory() {
SimpleMessageListenerContainerFactory msgListenerContainerFactory = new SimpleMessageListenerContainerFactory();
msgListenerContainerFactory.setAmazonSqs(amazonSQSAsyncClient());
return msgListenerContainerFactory;
}
#Bean
public QueueMessagingTemplate queueMessagingTemplate(AmazonSQSAsync amazonSqs) {
return new QueueMessagingTemplate(amazonSQSAsyncClient());
}
#Bean
public BasicAWSCredentials amazonAWSCredentials() {
return new BasicAWSCredentials(amazonAWSAccessKey, amazonAWSSecretKey);
}
In my case i was missing an annotation #EnableSqs in config class
Turns out i had a typo in the queue names in SQS console and my code, my bad.
Though it is late, it may help someone. In my case, I used gradle with these configuration
implementation "io.awspring.cloud:spring-cloud-starter-aws-messaging:2.3.0"
implementation "io.awspring.cloud:spring-cloud-aws-dependencies:2.3.0"
It doesn't work , below works perfectly for me:
compile "io.awspring.cloud:spring-cloud-starter-aws-messaging:2.3.0"
compile "io.awspring.cloud:spring-cloud-aws-dependencies:2.3.0"
In my case, the wrong region is specified. For some reason, sending didn't complain & pushed the message to the queue, but the listener is not getting called. Fixing the region in the application.yml file solved the issue.
In my case QueueMessageHandler bean wasn't initialized.
In the documentation they are using maven dependency:
<dependency>
<groupId>io.awspring.cloud</groupId>
<artifactId>spring-cloud-aws-messaging</artifactId>
<version>{spring-cloud-version}</version>
</dependency>
But in my case I'm using gradle, and also require dependency:
implementation(platform("io.awspring.cloud:spring-cloud-aws-dependencies:2.4.2"))
implementation("io.awspring.cloud:spring-cloud-aws-messaging")
implementation("io.awspring.cloud:spring-cloud-aws-autoconfigure")
Or you could define QueueMessageHandler in your own config file
#Bean
public QueueMessageHandler queueMessageHandler(QueueMessageHandlerFactory factory) {
return factory.createQueueMessageHandler();
}

Camel exchange expired via jetty continuation

Is there a possibility in Apache Camel to register a handler for managing exchanges that cannot be written to jetty endpoint http response because continuation timeout has been reached?
I'll just add my notes on that because I made it available in my project by modifying CamelContinuationServlet in the if (continuation.isExpired()) block like this
if (continuation.isExpired()) {
String id = (String) continuation.getAttribute(EXCHANGE_ATTRIBUTE_ID);
// remember this id as expired
expiredExchanges.put(id, id);
log.warn("Continuation expired of exchangeId: {}", id);
consumer.getBinding().doWriteExceptionResponse(new TimeoutException(), response);
return;
}
in combination with a custom HttpBinding called ErrorHandlingHttpBinding in my code like this
public class ErrorHandlingHttpBinding extends DefaultHttpBinding {
#Override
public void doWriteExceptionResponse(Throwable exception, HttpServletResponse response) throws IOException {
if (exception instanceof TimeoutException) {
response.setStatus(HttpServletResponse.SC_GATEWAY_TIMEOUT);
response.getWriter().write("Continuation timed out...");
} else {
super.doWriteExceptionResponse(exception, response);
}
}
}
registered as spring bean with id="errorHandlingHttpBinding" and referred in the component string as jetty:http://localhost:21010/?useContinuation=true&continuationTimeout=1&httpBindingRef=errorHandlingHttpBinding.
No this is not possible. Maybe you need to set a higher timeout if you have some slow processing exchanges.
You are welcome to dive in the Jetty APIs to see if you can find a hook for such a onTimeout event and see what it takes to support that in camel-jetty.