Optimizing custom IDCT calculation - c++

Question

For an image format I have to decode a jpeg like compression. The steps are pretty standard, however the color coding and the produced IDCT values seem to be slightly different (I think, I am no expert). This is why using a standard JPEG library produced invalid results it seems and I had to implement it myself.
Now I am not really happy with the performance. After optimizing the naive approach by doing some caching of data that does not change I need ~150ms to decode a 1024x1024 image. Of that 120ms is spent in the idct function which looks like this:
float idctHelper(const int16_t *inBlock, int32_t u, int32_t v, int32_t blockWidth, int32_t blockHeight) {
glm::vec<4, float, glm::packed_lowp> vec1{};
glm::vec<4, float, glm::packed_lowp> vec2{};
glm::vec<4, float, glm::packed_lowp> vec3{};
float result = 0.0f;
for (auto y = 0; y < blockHeight; ++y) {
for (auto x = 0; x < blockWidth; x += 4) {
const auto idx = (v * 8 + u) * 64 + y * 8 + x;
vec1 = glm::vec<4, float, glm::packed_lowp>(inBlock[y * blockWidth + x], inBlock[y * blockWidth + x + 1], inBlock[y * blockWidth + x + 2], inBlock[y * blockWidth + x + 3]);
vec2 = glm::vec<4, float, glm::packed_lowp>(idctLookup[idx], idctLookup[idx + 1], idctLookup[idx + 2], idctLookup[idx + 3]);
vec3 = vec1 * vec2;
result += vec3.x + vec3.y + vec3.z + vec3.w;
}
}
return result;
}
template<typename T, typename U = T>
U clamp(T value, T min, T max) {
return static_cast<U>(std::min<T>(std::max<T>(value, min), max));
}
void idct(int16_t *outBlock, int16_t *inBlock, bool isLuminance, int32_t blockWidth = 8, int32_t blockHeight = 8) {
for (auto y = 0; y < blockHeight; ++y) {
for (auto x = 0; x < blockWidth; ++x) {
auto value = static_cast<int16_t>(std::round(
0.25f * idctHelper(inBlock, x, y, blockWidth, blockHeight)));
if (isLuminance) {
value = clamp<int16_t>(static_cast<int16_t>(value + 128), 0, 255);
} else {
value = clamp<int16_t>(value, -256, 255);
}
outBlock[y * blockWidth + x] = value;
}
}
}
As you can see I have tried to use (potentially) SIMD and low precision operations to speed up the idctHelper but now I am kinda lost on where I can continue optimizing this. I would really appreciate any idea or hint.