Gstreamer gst_buffer_pool_acquire_buffer function is slow on ARM - gstreamer

Question

My Gstreamer version is 1.17, cross compiled using instructions from here.
Here is my gstreamer pipeline,
appsrc name=framesrc0 do-timestamp=true format=time ! video/x-raw,width=640,height=480,framerate=30/1,format=NV12 ! queue ! x264enc ! queue ! h264parse ! mpegtsmux ! filesink name=mysink location=./myfile.ts
I feed NV12 frames to appsrc using the below function (6404801.5 = 460800 bytes)
bool BelGst::FeedData0(uint8_t *buf, uint32_t len)
{
GstFlowReturn ret;
GstBuffer *buffer;
GstMapInfo info;
timespec ts_beg, ts_end;
uint32_t time_ms;
clock_gettime(CLOCK_MONOTONIC, &ts_beg);
ret = gst_buffer_pool_acquire_buffer (pool0, &buffer, NULL);
if (G_UNLIKELY (ret != GST_FLOW_OK)) {
cout << "BufferPool pool0 failed" << endl;
return FALSE;
}
clock_gettime(CLOCK_MONOTONIC, &ts_end);
time_ms = (ts_end.tv_sec - ts_beg.tv_sec)*1000 + (ts_end.tv_nsec - ts_beg.tv_nsec) / 1e6;
cout << "Buffer pool acquire time = " << time_ms << "ms" << endl;
/* Set its timestamp and duration */
GST_BUFFER_TIMESTAMP(buffer) = timestamp0;
GST_BUFFER_DURATION(buffer) = gst_util_uint64_scale(1, GST_SECOND, 30);
GST_BUFFER_OFFSET(buffer) = offset0++;
timestamp0 += GST_BUFFER_DURATION(buffer);
gst_buffer_map(buffer, &info, GST_MAP_WRITE);
memcpy(info.data, buf, len);
gst_buffer_unmap(buffer, &info);
g_signal_emit_by_name(app_source0, "push-buffer", buffer, &ret);
gst_buffer_unref(buffer);
return TRUE;
}
I've setup the buffer pool as shown below,
void BufferPoolSetup(GstBufferPool *&pool)
{
GstStructure *config;
int size, min, max;
GstCaps *caps;
pool = gst_buffer_pool_new();
/* get config structure */
config = gst_buffer_pool_get_config(pool);
size = 640*480*1.5;
min = 1;
max = 4;
caps = gst_caps_from_string("video/x-raw");
/* set caps, size, minimum and maximum buffers in the pool */
gst_buffer_pool_config_set_params (config, caps, size, min, max);
gst_caps_unref(caps);
gst_buffer_pool_set_config (pool, config);
/* and activate */
gst_buffer_pool_set_active (pool, TRUE);
return;
}
When I run the pipeline, I see that the function gst_buffer_pool_acquire_buffer is taking somewhere between 20ms to 60ms. Could someone point if there is something wrong in my approach? Am I missing something?