How to create a Dataflow Template without running code? - google-cloud-platform

Question

I am creating a Dataflow template that runs perfectly if I pass all required parameters but my use case is to create a Generic Template where I can pass parameters at Runtime.
All my options are ValueProvider but still, it is trying to execute the code while creating the template and that gives an error. Is there any way I can create a Dataflow template without executing code?
I have also faced one weird issue when I use create template command along with all the parameters then it creates the template successfully but at that time all parameters get hard coded in the code and if I pass the new parameter value while running it does not change the value with the template.
Is it the correct behavior of the Dataflow template?
Command without passing all parameters :
mvn compile exec:java -D"exec.mainClass"="org.example.Main" -D"exec.args"="--runner=DataflowRunner --project=<project> --stagingLocation=gs://test-bucket/staging_4 --templateLocation=gs://test-bucket/templates/my_template --region=asia-south1 " -P dataflow-runner -X

Answer 1

After designing my code multiple times, I got to know what mistake I was doing.
What I got to know is that your code should be designed in such a way that it will not ask for any value for generating the Graph except Project Id, Stage Path, and Template Path. if it is asking then check the below practices and design accordingly.
Following are the practice you should follow if you are creating Custom Template.
1.Declare all the options as ValueProvider so that it will be calculated at Run Time and the Template creation process will skip it.
2.Do not assign the ValueProvider value to the normal variable as below.
String Query = options.getQuery().toString();
PCollection<TableRow> rows = p.apply("Read From Source", JdbcIO.<TableRow>read()
.withQuery(Query)
3.Try to pass ValueProvider value directly to the method. For example above code snippet should be like below
PCollection<TableRow> rows = p.apply("Read From Source", JdbcIO.<TableRow>read()
.withQuery(options.getQuery())
4.Do not use options only inside any one particular block of code which is never accessible during Template creation. As below example.
Boolean Check = false;
if (options.getA().isAccessible()){
Check = true;
}
if (Check == true){
//Use of Options.getB().
}
In the above case if you are not pass --a option while creating a template then the Check value will always false and a particular below block of code will not be executable. If we are not using --b out of Check block then after the creation of Template options b will be unknown to Template and it will give an error if you pass it at Run time.
5.If you have use case where you need to manipulate data of input options passed then you should use NestedValueProvider instead of converting ValueProvider into a Normal variable such as String and manipulate data.
Fo example, avoid the below practice.
String Query = String.format("SELECT * FROM %s", options.getTableName().toString());
PCollection<TableRow> rows = p.apply("Read From Source", JdbcIO.<TableRow>read()
.withQuery(Query)
Instead implement same logic as below:
ValueProvider<String> Query = NestedValueProvider.of(options.getTableName(),
new SerializableFunction<String, String>() {
#Override
public String apply(TranslatorInput<String, String> input)
{
return String.format("SELECT * FROM %s", input);;
}
});;
PCollection<TableRow> rows = p.apply("Read From Source", JdbcIO.<TableRow>read()
.withQuery(Query)

Answer 2

Use FlexTemplates instead. Documentation can be found here.