gcp PubSub getting "processing delay was high" - google-cloud-platform

Question

This is our set up, we have one topic and this topic has two subscriptions v1 and v2 with exact identical settings, both are pull subscription with 10 sec ack deadline.
both subscription v1 and v2 goes to separate dedicated dataflow where v2's data flow is more optimized but pretty much doing same thing.
Problem is that every now and then we see below warning messages and backlog starts build up in v2 subscription only and v1 shows little to no backlog.
08:53:56.000 ${MESSAGE_ID} Pubsub processing delay was high at 72 sec.
Dataflow log in v2 shows nothing obvious except above messages. In fact v2 dataflow cpu usage is lower than v1 so I cant make any sense out of this.
Questions:
What causes processing delay and how can I fix it?
Why isn't v1 subscription getting same warnings?
Updated at 2017/01/17
As suggested via #ben it seems that ParDo filtering operation we do right after PubSub read is hitting unexpectedly high latency. But considering getClassroomIds is a simple java list I'm not sure how I can tackle this problem. One question is that is coder we have applied to pubsub lazy? Is unzipping and deserializing we have defined in coder applied when ProcessContext#element() is called?
def processElement(c: DoFn[Entity, Entity]#ProcessContext) = {
val startTime = System.currentTimeMillis()
val entity = c.element()
if (!entity.getClassroomIds.isEmpty) {
c.output(entity)
}
val latencyMs = System.currentTimeMillis() - startTime
if (latencyMs > 1000) {
// We see this warning messages during the load spike
log.warn(s"latency breached 1 second threshold: $latencyMs ms")
}
}

Answer 1

The timing you mention isn't accurately accounting for the time spent in that step. In particular, due to the fusion optimization it reflects all of the ParDos after the filtering operation.
If you have a pipeline which looks like:
ReadFromPubSub -> ParDo(Filter) -> ParDo(Expensive) -> ParDo(Write)
Both Expensive and Write execute on each element coming out of the Filter before the call to c.output returns. This is further a problem because it is fused into the elements coming out of PubSub.
The easiest solution is probably to perform a Reshuffle:
pipeline
.apply(PubSubIO.read(...))
.apply(keyEachElement by their value and Void)
.apply(new Reshuffle<E, Void>())
.apply(move the key back to the element)
// rest of the pipeline
Note that using the Reshuffle instead of a regular GroupByKey has nice properties since it triggers faster than any normally trigger will fire.