Infinispan RemoveCache Error with Concurrency - concurrency

Question

In my application i use infinispan as distributed cache.
I work with 3 application server running: wildfly 9.2. On each of them i job is exectuted and its work is just to validate some cache items. If the validation fails the job will remove the cache as it's not valid any more.
The removing code is quite simple:
if (somecondition){
cacheManager.removeCache(sessionCacheName);
}
I realized that when all three server are running (so there are 3 jobs that concurrently execute the romove operation) i get systematically this exception:
19:43:00,005 WARN [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (OOB-20,ws-7-aor-57542) ISPN000220: Problems un-marshalling remote command from byte buffer
a.lang.NullPointerException
at org.infinispan.commands.RemoteCommandsFactory.fromStream(RemoteCommandsFactory.java:219)
at org.infinispan.marshall.exts.ReplicableCommandExternalizer.fromStream(ReplicableCommandExternalizer.java:107)
at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:155)
at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:65)
at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:436)
at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:227)
at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:153)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:354)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:134)
at org.infinispan.marshall.core.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:101)
at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:80)
at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:28)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:298)
at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460)
at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377)
at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:250)
at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:675)
at org.jgroups.JChannel.up(JChannel.java:739)
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1029)
at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:383)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:394)
at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1042)
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:435)
at org.jgroups.protocols.pbcast.NAKACK2.deliver(NAKACK2.java:961)
at org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:843)
at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:618)
at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:200)
at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:297)
at org.jgroups.protocols.MERGE3.up(MERGE3.java:288)
at org.jgroups.protocols.Discovery.up(Discovery.java:291)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1577)
at org.jgroups.protocols.TP$MyHandler.run(TP.java:1796)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
on one server while this one:
19:44:00,199 ERROR [stderr] (DefaultQuartzScheduler_Worker-1) Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from ws-7-aor-36158, see cause for remote stack trace
19:44:00,200 ERROR [stderr] (DefaultQuartzScheduler_Worker-1) at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:46)
19:44:00,211 ERROR [stderr] (DefaultQuartzScheduler_Worker-1) at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:71)
19:44:00,211 ERROR [stderr] (DefaultQuartzScheduler_Worker-1) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:586)
19:44:00,212 ERROR [stderr] (DefaultQuartzScheduler_Worker-1) at org.infinispan.manager.DefaultCacheManager.removeCache(DefaultCacheManager.java:492)
on the other 2.
This error disappear when there is only 1 application server instance running.
So it's clearly related to the concurrence.
What am i missing?

Answer 1

The removeCache() method was only intended as an admin operation to be called from a JMX/RHQ console, so concurrent calls weren't much of a concern.
The good news is that concurrent calls will work in Infinispan 8.1+/WildFly 10, which include the fix for ISPN-5756.