Geode/GemFire/PCF/SCDF error - user not authorized for DATA:WRITE / DATA:READ permission - cloud-foundry

Question

We are using Spring Cloud Data Flow to build stream pipe. The spring-cloud-dataflow-server version is 2.8.3.
The out of box sink module gemfire is provided by official site: https://docs.spring.io/spring-cloud-dataflow/docs/2.8.3/reference/htmlsingle/#applications
Here is the source code of this module: https://github.com/spring-attic/gemfire/tree/v2.1.4.RELEASE
Recently the server end enabled the Gemfire security Authorization feature. On client end we set username/password in SCDF stream definition. But when the data sink to Gemfire, we got user not authorized for DATA:WRITE / DATA:READ error. I attached the details at the end.
The problem is, gemfire server end already granted client's user DATA READ/WRITE permission, and this gemfire sink module can write data into database. But meanwhile we are keep getting this error.
According to the Spring Project Version Compatibility Matrix:
https://github.com/spring-projects/spring-boot-data-geode/wiki/Spring-Boot-for-Apache-Geode-and-VMware-Tanzu-GemFire-Version-Compatibility-Matrix
We tried all different Apache Geode version, but all of them got the same error.
Is there any way to handle this issue?
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] org.springframework.messaging.MessageHandlingException: nested exception is org.springframework.messaging.MessageHandlingException: error occurred in message handler [messageHandler]; nested exception is org.springframework.dao.DataAccessResourceFailureException: remote server on 93aed963-4624-4e01-6227-954e(23:loner):47226:c35a0e8b: org.apache.geode.security.NotAuthorizedException: user not authorized for DATA:WRITE:WriteTest; nested exception is org.apache.geode.cache.client.ServerOperationException: remote server on 93aed963-4624-4e01-6227-954e(23:loner):47226:c35a0e8b: org.apache.geode.security.NotAuthorizedException: user not authorized for DATA:WRITE:WriteTest, failedMessage=GenericMessage [payload=PDX[4548420,__GEMFIRE_JSON]{read_datetime=2022-03-15T00:52:40:722Z}, headers={id=8ef8d368-87a7-addc-1074-06bb58043933, timestamp=1647305561161}]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at org.springframework.integration.handler.MethodInvokingMessageProcessor.processMessage(MethodInvokingMessageProcessor.java:109) ~[spring-integration-core-5.1.7.RELEASE.jar!/:5.1.7.RELEASE]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at org.springframework.integration.handler.ServiceActivatingHandler.handleRequestMessage(ServiceActivatingHandler.java:93) ~[spring-integration-core-5.1.7.RELEASE.jar!/:5.1.7.RELEASE]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:123) ~[spring-integration-core-5.1.7.RELEASE.jar!/:5.1.7.RELEASE]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:169) ~[spring-integration-core-5.1.7.RELEASE.jar!/:5.1.7.RELEASE]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:115) ~[spring-integration-core-5.1.7.RELEASE.jar!/:5.1.7.RELEASE]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:132) ~[spring-integration-core-5.1.7.RELEASE.jar!/:5.1.7.RELEASE]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:105) ~[spring-integration-core-5.1.7.RELEASE.jar!/:5.1.7.RELEASE]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:73) ~[spring-integration-core-5.1.7.RELEASE.jar!/:5.1.7.RELEASE]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:453) ~[spring-integration-core-5.1.7.RELEASE.jar!/:5.1.7.RELEASE]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:401) ~[spring-integration-core-5.1.7.RELEASE.jar!/:5.1.7.RELEASE]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:187) ~[spring-messaging-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:166) ~[spring-messaging-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:47) ~[spring-messaging-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:109) ~[spring-messaging-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at org.springframework.integration.endpoint.MessageProducerSupport.sendMessage(MessageProducerSupport.java:205) ~[spring-integration-core-5.1.7.RELEASE.jar!/:5.1.7.RELEASE]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter.sendMessageIfAny(KafkaMessageDrivenChannelAdapter.java:369) ~[spring-integration-kafka-3.1.0.RELEASE.jar!/:3.1.0.RELEASE]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter.access$400(KafkaMessageDrivenChannelAdapter.java:74) ~[spring-integration-kafka-3.1.0.RELEASE.jar!/:3.1.0.RELEASE]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter$IntegrationRecordMessageListener.onMessage(KafkaMessageDrivenChannelAdapter.java:431) ~[spring-integration-kafka-3.1.0.RELEASE.jar!/:3.1.0.RELEASE]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter$IntegrationRecordMessageListener.onMessage(KafkaMessageDrivenChannelAdapter.java:402) ~[spring-integration-kafka-3.1.0.RELEASE.jar!/:3.1.0.RELEASE]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeOnMessage(KafkaMessageListenerContainer.java:1316) [spring-kafka-2.2.12.RELEASE.jar!/:2.2.12.RELEASE]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeOnMessage(KafkaMessageListenerContainer.java:1299) [spring-kafka-2.2.12.RELEASE.jar!/:2.2.12.RELEASE]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeRecordListener(KafkaMessageListenerContainer.java:1259) [spring-kafka-2.2.12.RELEASE.jar!/:2.2.12.RELEASE]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeWithRecords(KafkaMessageListenerContainer.java:1240) [spring-kafka-2.2.12.RELEASE.jar!/:2.2.12.RELEASE]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListener(KafkaMessageListenerContainer.java:1155) [spring-kafka-2.2.12.RELEASE.jar!/:2.2.12.RELEASE]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeListener(KafkaMessageListenerContainer.java:965) [spring-kafka-2.2.12.RELEASE.jar!/:2.2.12.RELEASE]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:772) [spring-kafka-2.2.12.RELEASE.jar!/:2.2.12.RELEASE]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:705) [spring-kafka-2.2.12.RELEASE.jar!/:2.2.12.RELEASE]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_242]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_242]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_242]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] Caused by: org.springframework.messaging.MessageHandlingException: error occurred in message handler [messageHandler]; nested exception is org.springframework.dao.DataAccessResourceFailureException: remote server on 93aed963-4624-4e01-6227-954e(23:loner):47226:c35a0e8b: org.apache.geode.security.NotAuthorizedException: user not authorized for DATA:WRITE:WriteTest; nested exception is org.apache.geode.cache.client.ServerOperationException: remote server on 93aed963-4624-4e01-6227-954e(23:loner):47226:c35a0e8b: org.apache.geode.security.NotAuthorizedException: user not authorized for DATA:WRITE:WriteTest
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at org.springframework.integration.support.utils.IntegrationUtils.wrapInHandlingExceptionIfNecessary(IntegrationUtils.java:189) ~[spring-integration-core-5.1.7.RELEASE.jar!/:5.1.7.RELEASE]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:186) ~[spring-integration-core-5.1.7.RELEASE.jar!/:5.1.7.RELEASE]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at org.springframework.cloud.stream.app.gemfire.sink.GemfireSinkHandler.handle(GemfireSinkHandler.java:65) ~[spring-cloud-starter-stream-sink-gemfire-2.1.6.RELEASE.jar!/:2.1.6.RELEASE]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at sun.reflect.GeneratedMethodAccessor121.invoke(Unknown Source) ~[na:na]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_242]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_242]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:171) ~[spring-messaging-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:120) ~[spring-messaging-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at org.springframework.integration.handler.support.MessagingMethodInvokerHelper$HandlerMethod.invoke(MessagingMethodInvokerHelper.java:1115) ~[spring-integration-core-5.1.7.RELEASE.jar!/:5.1.7.RELEASE]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at org.springframework.integration.handler.support.MessagingMethodInvokerHelper.invokeHandlerMethod(MessagingMethodInvokerHelper.java:624) ~[spring-integration-core-5.1.7.RELEASE.jar!/:5.1.7.RELEASE]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at org.springframework.integration.handler.support.MessagingMethodInvokerHelper.processInternal(MessagingMethodInvokerHelper.java:491) ~[spring-integration-core-5.1.7.RELEASE.jar!/:5.1.7.RELEASE]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at org.springframework.integration.handler.support.MessagingMethodInvokerHelper.process(MessagingMethodInvokerHelper.java:362) ~[spring-integration-core-5.1.7.RELEASE.jar!/:5.1.7.RELEASE]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] at org.springframework.integration.handler.MethodInvokingMessageProcessor.processMessage(MethodInvokingMessageProcessor.java:106) ~[spring-integration-core-5.1.7.RELEASE.jar!/:5.1.7.RELEASE]
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] ... 29 common frames omitted
2022-03-14T20:52:41.232-04:00 [APP/PROC/WEB/0] [OUT] Caused by: org.springframework.dao.DataAccessResourceFailureException: remote server on 93aed963-4624-4e01-6227-954e(23:loner):47226:c35a0e8b: org.apache.geode.security.NotAuthorizedException: user not authorized for DATA:WRITE:WriteTest;

Answer 1

Technically, using Spring Boot for Apache Geode [equally for GemFire] (SBDG), particularly when running in a Pivotal CloudFoundry (PCF) environment connecting your Spring [Boot] app, or in your case, Spring Cloud Data Flow (SCDF) app, to a Pivotal Cloud Cache (PCC) service instance (i.e. GemFire in PCF), then SBDG will automatically connect, authenticate and authorize your application once your Spring [Boot] app has been pushed up to PCF.
NOTE: Pivotal CloudFoundry (PCF) is now known as VMware Tanzu Appliation Service (TAS) and Pivotal Cloud Cache (PCC) is now known as VMware Tanzu GemFire for VMS.
Of course, this assumes that the PCF/PCC environment, and specifically, the VCAP environment variables, were setup and configured properly when the PCC service instance was provisioned.
If you are not using Spring Boot for Apache Geode, then there is no "automatic" inspection of the PCF/PCC environment (VCAP env vars) and therefore, you become responsible for handling connections, auth, etc.
SBDG was specifically designed to handle these concerns across environments and provides auto-configuration to handle connections, auth and other concerns when a Spring Boot app is pushed up to PCF connected to PCC.
More details can be found in the documentation.
Additionally, the Getting Started Sample walks a user through building a Spring Boot app using Apache Geode in a local context, then switching to a non-managed client/server topology locally, and finally pushing and running the app in a managed context like PCF, connecting (and authenticating) with PCC.
All of this requires SBDG though.
I am not certain that SCDF uses SBDG under the hood. It may simply only use Spring Data for Apache Geode (SDG), in which case, you may need to swap out the SDG dependency for SBDG.
There is most likely other work involved in this process as well since it is unclear to me what specific GemFire/Geode objects (e.g. a cache instance) SCDF creates on your behalf when using SCDF (sources/sinks) that may conflict with the auto-configuration provided in and by SBDG.
For instance, if SCDF creates a cache instance (i.e. ClientCache) for you, then it will override the SBDG auto-configuration that automatically creates a ClientCache instance]3, by default. If this is the case, then once again, you become responsible for security (auth) since security must be configured before an GemFire/Geode cache instance (e.g. ClientCache) is created.
NOTE: This is a GemFire/Geode requirement, not a Spring requirement.
Therefore, SBDG's auto-configuration arrangement is very deliberate in its precedence and ordering when being applied. If the SBDG auto-configuration is explicitly overridden either by you, or implicitly by another framework (e.g. SCDF), then you become responsible for knowing the expectations (internals) of GemFire/Geode configuration.
On the other hand, if you are certain SBDG is in the application classpath and being used properly, then perhaps this problem stems from the app using the wrong assigned user.
If your environment is rather complex, declaring multiple users with different sets of assigned permissions, then maybe your app needs to be run with a different user assignment, in which case, you should review this particular section of the documentation.
As always you should make sure you Spring [Boot | CDF] application runs correctly in a local, non-managed environment with the similar setup and configuration before running remotely in a managed environment like PCF.
The goals of SBDG have always been clear and SBDG is tested and proven to this effect.
Please share as many specifics (code, configuration, etc) as you can here in order for us to be able to triage this problem correctly.