Unit Testing Flink Jobs with new Data Source API - unit-testing

Question

in the past i unit tested the flink jobs by writing the job with pluggable Sources/Sink and then mocked them via simple Source-/SinkFunctions. Like this:
public class Example {
private static SourceFunction<String> someSource;
private static SourceFunction<String> someOtherSource;
private static SinkFunction<String> someSink;
Example(
SourceFunction<String> someSource,
SourceFunction<String> someOtherSource,
SinkFunction<String> someSink
) {
this.someSource = someSource;
this.someOtherSource = someOtherSource;
this.someSink = someSink;
}
void build(StreamExecutionEnvironment env) {
/*
... build your logic here ...
*/
}
public static void main(String[] args) {
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
Example(
new FlinkKafkaConsumer<String>(...),
new FlinkKafkaConsumer<String>(...),
new FlinkKafkaProducer<String>(...)
).build(env);
env.execute();
}
}
This way i could easily test the whole job by just exchange the real KafkaSinks & Sources with custom Sink-/SourceFunctions.
The new DataSources are much more complex for simply implementing it for test cases.
Even if i would implement it it would end in a generic hell to make this injectable at the constructor.
So i was wondering whats the best approach is to unit test the whole job without bring up e.g. a complete Kafka cluster.
Are there any ideas or solutions ?

Answer 1

You could get part way there by building something based on the NumberSequenceSource, followed by a map.
The DataGeneratorSource described in FLIP-238 is meant to fill this need, and it will be released as part of 1.16. (I believe it is self-contained, so you could copy it and start using it now.)
An alternative approach to using a pluggable Sink is to use DataStream#executeAndCollect():
DataStream<Integer> stream = env.fromElements(1, 2, 3);
try (CloseableIterator<Integer> results = stream.executeAndCollect()) {
assertThat(results).containsInAnyOrder(1, 2, 3);
}