Input/Output for EmrActivity - AWS Data Pipeline - amazon-web-services

Question

I'm trying to use Data Pipeline to run a Spark Application. How can I access the input / output I specify (S3DataNode) for the EmrActivity, inside my Spark application?
My question is similar to this - https://forums.aws.amazon.com/message.jspa?messageID=507877
Earlier I used to pass the input and output as arguments to the Spark application in steps.
Thanks

Answer 1

I ran across the same question. There's very limited documentation around this. This is my understanding:
You specify the input and output for the EmrActivity. This will create the dependencies between the data nodes and the activity.
In the EmrActivity, you can reference the input sources like this: #{input.directoryPath},#{output.directoryPath}
Example:
...
{
"name": "Input Data Node",
"id": "inputDataNode",
"type": "S3DataNode",
"directoryPath": "s3://my/raw/data/path"
},
{
"name": "transform",
"id": "transform",
"type": "EmrActivity",
"step": [
"s3://us-east-1.elasticmapreduce/libs/script-runner/script-runner.jar,s3://my/transform/script.sh,#{input.directoryPath},#{output.directoryPath}"
],
"runsOn": {
"ref": "emrcluster"
},
"input": {
"ref": "inputDataNode"
},
"output": {
"ref": "outputDataNode"
}
},
{
"name": "Output Data Node",
"id": "outputDataNode",
"type": "S3DataNode",
"directoryPath": "s3://path/to/output/"
},
...