sending EventData to Azure Eventhub larger than 256k - azure-eventhub

Question

I need to send some data to eventhub, but it is not going through because the size is too big. Is there a way to compress the data or some command that chunks up the data and joins it in event hub?
I am using the following java-code to send:
EventData sendEvent = new EventData(payloadBytes);
EventHubClient ehClient = EventHubClient.createFromConnectionStringSync(connStr.toString());
ehClient.sendSync(sendEvent);
What are my options if the payloadBytes is too big?

Answer 1

Another approach would be to switch from basic to standard tier, which allows you to send events up to size 1MB. Microsoft docs: Quotas and limits - basic vs standard tiers.

Answer 2

You can try to compress the message you send. According to the docs You can edit the properties map inside the EventData :
var eventData = new EventData(....);
eventData.getProperties().put("Compression","GZip");
However, their solution didn't work for me and the messages size didnt decrease. Therefore I gziped the data by myself before adding it to the batch :
private EventData compressData(Data data) throws IOException {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
GZIPOutputStream gos = new GZIPOutputStream(baos);
gos.write(new DefaultJsonSerializer().serializeToBytes(data)); # worth to create the seriazlier outside of this functions scope..
gos.flush();
gos.close();
return new EventData(baos.toByteArray());
}

Answer 3

Could you describe what is your use case and what sort of data are you going to publish, it helps find a right solution. Here are some common options I used in my projects.
Some concerns to keep in mind when designing system that need message broker or event streaming:
Event hub is designed for event streaming as it is described here. It scale well when you have huge amount of events.
256KB limit per message is way more than what is usually needed to transfer events, which are usually text based.
In my project I had two different use cases when 256K was not enough, here is how I solved it:
We need to publish internal event of a our Monolithic system out which later can be consumed by our Microservices outside. most of the time the messages were small but sometime they go slightly bigger than 256K and fix was easy we compress them before publish, and unzip them at the receiver. you can find a good sample here.
In the second scenario messages were way bigger than 256K which compression was not enough, what I did was first create a blob with that content and then publish the event with a reference to that blob, then your event receiver can get the content from blob, no matter how big it is.
It should cover most of the use cases but let me know more about your problem if it is not useful for you.