Actor Cluster Slowness when Large size messages produced by a ShardRegion proxy - akka

Question

We implemented Akka clustering with cluster sharding for a use case.
When we doing load testing for that, We created 1000 entity actors in a Node by cluster Sharding.(Cluster Node)
And we sends messages to that entity actors from a Proxy Node (Shard Region Proxy on other Node).
What we done
Using
akka.cluster.shard.remember.entities=ddata
akka {
remote {
netty.tcp {
hostname = "x.x.x.x"
port = 255x
}
}
}
akka.cluster {
sharding{
remember-entities = on
remember-entities-store = ddata
distributed-data.durable.keys = []
}
}
Created a Dispatcher thread with 1000 threads and assigned that to Entity actors.(Which is on Cluster Node).
Created a java program which spawn 100 threads and each thread produce message to 10 actors sequentially one by one by the ShardRegion Proxy from Proxy node to Cluster Node.
For each message we wait for acknowledgement from the Entity Actor to the sender thread.Thereafter only next message will be produced.
So at a time 100 parallel messages can be fired.
When i produce 10KB messages with this 100 Parallel threads to 1000 Entity Actors we getting the acknowledgement from Entity actor pretty fast.like <40 ms
But when i sending 100KB messages like the same the acknowledgement making 150 to even 200ms delay for each messages.
I know huge message will take more time than small messages.
As i read some blogs and others questions similar like this. They are saying to increase
akka {
remote {
netty.tcp {
# Sets the send buffer size of the Sockets,
# set to 0b for platform default
send-buffer-size = 2MiB
# Sets the receive buffer size of the Sockets,
# set to 0b for platform default
receive-buffer-size = 2MiB
}
}
this configurations.
Even after increased this config from 200KB,2MB,10MB,20MB there is no performance gain.
I put some debug log on Endpoint Writer Actor and saw a strange thing, even i have a buffer size 2MB when huge no of messages send to shard Region the Buffer in Endpoint writer is increasing but it writing into the Association-handle one by one.I'm getting logger for each message write to Association Handle(Same Association handle Object Id on each write).
Then is it sequential???
Then how the send and receive buffer used in this cases.?
Some one said increasing Shard count will help.Even after increasing there is no performance gain.
Is that any miss configuration i done or any Configuration i missed?.
NOTE:
Cluster Node have 1000 Entity Actors which split into 3 Shards.
Proxy Node which have 100 parallel threads which produce messages to the Cluster Node.