streaming machine learning pipeline with apache beam - google-cloud-platform

Question

I'm using Beam to build a streaming ML pipeline that's very similar to Zeitgeist system mentioned in the original MillWheel paper. However, I'm have difficulty in using trained models to do online predictions (the blue vertical arrow "Models" in Figure 1. of the paper).
It seems that Zeitgeist models are updated incrementally each time (?) when a new windowed counter comes in. However, the specific model I'm using doesn't support incremental/online training, so I need to use some trigger/windowing to train the model in batches.
During prediction, I don't know how to align windows of features and batch-trained models.
using side input (see below) can make the pipeline run, but it stucks at the prediction step waiting for the model to be materialized.
CoGroupByKey does not work because the windows of features and ensembledModels are not the same.
I also want to do model ensemble, which makes things even more complicated.
Here's a rough sketch of what I have
// Correspond to "Window Counter" in MillWheel paper
final PCollection<Feature> features = pipeline
.apply(PubsubIO.read(...))
.apply(...some windowing...)
.apply(ParDo.of(new FeatureExtractor()));
// Correspond to "Model Calculator" in MillWheel paper
final PCollection<Iterable<Model>> ensembledModels = features
.apply(...some windowing and data triggers...)
.apply(new ModelTrainer()));
// Correspond to "Spike/Dip Detector" in MillWheel paper
// The pipeline stucks at here when run
final PCollection<Score> score = features
.apply(ParDo.of(new Predictor()).withSideInput(ensembledModels));