rtpvp8depay + rtpvp8pay appear to introduce artifacts on Janus Gateway - gstreamer

Question

A VP8 stream comes from Janus Videoroom plugin with restreaming to 10002/10004 locally. From there, it's picked up with the following gstreamer pipeline:
gst-launch-1.0 -v udpsrc \
caps="application/x-rtp,media=(string)video,encoding-name=(string)VP8,payload=100" \
address=127.0.0.1 port=10004 ! \
rtpvp8depay ! rtpvp8pay ! \
udpsink host=127.0.0.1 port=5004
and sent to Streaming plugin. As you can see, no transcoding here, just depayloading and payloading. resulting video breaks down into artifacts on some keyframes, approximately once in 10 keyframes, only to be fixed on next keyframe.
if i remove depay and pay, simply forwarding on rtp level, to get this
gst-launch-1.0 -v udpsrc \
caps="application/x-rtp,media=(string)video,encoding-name=(string)VP8,payload=100" \
address=127.0.0.1 port=10004 ! \
udpsink host=127.0.0.1 port=5004
it never happens.
i understand this is not the Janus issue but a gstreamer issue. but maybe anyone has an idea what could be the problem? this has been very reliably tested, the problem is easy to reproduce in the former case and never happens in the latter.
Of course, the goal of what i am doing is transcoding, and there was a lot more in the setup and the pipeline before i boiled it down to this level. Reproduced on Janus installed on a fresh Ubuntu 18.04 machine with all out-of-the-box settings.
update:
export GST_DEBUG="rtp*:4";
revealed this error message which drops out each time artifacts appear:
rtpbasedepayload gstrtpbasedepayload.c:473:gst_rtp_base_depayload_handle_buffer:
<rtpvp8depay0> 12 <= 100, dropping old packet
with the number which is "12" fluctuating being typically between 5 and 12.

Answer 1

This was the fix:
rtpjitterbuffer latency=50 !
before rtpvp8depay.
Logically, the order of packets is the same by that point as the one that came through the internet between the sending browser and Janus. If we don't depay+pay, it goes the same way to the receiving browser that's connected to Streaming plugin, and it has it's own jitter buffer so able to fix order, but if we do depay+pay here, there is no buffer so these packets are dropped, resuling in broken frames.
And yes, i got back transcoding and all the rest of my pipeline and all of other bells and whistles that were around, and it still works fine.