How to do branch grafting on separate tasks - ducttape

Question

I want to write a ducttape task that grafts the outputs of tasks belonging to different branches. For example, take a scenario where you have a training and test data. You want to use the training data to develop a model and you want to evaluate the model on a test set.
Here's a dummy example. prep is run on each of the data sets, but trainModel is only to be executed on training. The goal is to have applyModel use the results from trainModel with the training set and to evaluate it on the test set.
task prep
< input=$corpus
> output
{
# Do some common preprocessing on all corpora
cp $input $output
}
task trainModel
< input=$output#prep
> trainedModel
{
echo "Trained..." > $trainedModel
cat $input >> $trainedModel
}
task applyModel
< trainedModel=$trainedModel#trainModel[Set:Train]
< testInput=$output#prep[Set:Test]
> out
{
diff $file1 $file2 > $out
}
plan DoIt {
# reach prep via (Set: *)
reach trainModel via (Set: Train)
reach applyModel via (Set: *)
}
global
{
train=train.txt
test=test.txt
corpus=(Set: Train=$train Test=$test)
}
However, if you look at the execution plan:
ducttape 0.3
by Jonathan Clark
Loading workflow version history...
Have 0 previous workflow versions
Finding hyperpaths contained in plan...
Found 4 vertices implied by realization plan DoIt (Clause 1 of 2)
Found 4 vertices implied by realization plan DoIt (Clause 2 of 2)
Union of all planned vertices has size 5
Checking for completed tasks
Finding packages...
Found 0 packages
Checking for already built packages (if this takes a long time, consider switching to a local-disk git clone instead of a remote repository)...
Checking inputs...
Work plan (depth-first traversal):
RUN: ./prep/Baseline.baseline (Set.Train)
RUN: ./trainModel/Baseline.baseline (Set.Train)
RUN: ./prep/Set.Test (Set.Test)
RUN: ./applyModel/Baseline.baseline (Baseline.baseline)
RUN: ./trainModel/Set.Test (Set.Test)
you see that trainModel will be executed twice, both on the training and test data. How can I configure the work plan to train only on one or more training sets, and evaluate on one or more test sets?
(I realize it might be better in this example to break the $corpus variable into sub-branches, but in more complex scripts this problem still exists.)

Answer 1

This is exactly the sort of scenario that branches are aimed at. The issue in the sample above is that the graft for the training data is applied to late -- since you want to run training for only the training data set, you should apply the graft at the training task.
You also mentioned the use case of having multiple test sets, so I've included an example of that as well, which makes use of nested branch points.
To keep things clear, I renamed your original branch point "Set" to "Section" (i.e. train vs test data sections) and introduced a new "TestSet" branch point.
task prep
< input=$corpus
> output
{
# Do some common preprocessing on all corpora
cp $input $output
}
task trainModel
< input=$output#prep[Section:Train] <----- EDITED
> trainedModel
{
echo "Trained..." > $trainedModel
cat $input >> $trainedModel
}
task applyModel
< trainedModel=$trainedModel#trainModel <----- EDITED
< testInput=$output#prep[Section:Test]
> out
{
diff $file1 $file2 > $out
}
plan DoIt {
reach applyModel via (TestSet: *) <----- EDITED
}
global
{
train=train.txt
test_a=test_a.txt
test_b=test_b.txt
corpus=(Section:
Train=$train
Test=(TestSet: A=$test_a B=$test_b) <----- EDITED
)
}