How to make Spark session read all the files recursively? - regex

Question

Displaying the directories under which JSON files are stored:
$ tree -d try/
try/
├── 10thOct_logs1
├── 11thOct
│   └── logs2
└── Oct
└── 12th
└── logs3
Task is to read all logs using SparkSession.
Is there an elegant way to read through all the files in directories and then sub-directories recursively?
Few commands that I tried are prone to cause unintentional exclusion.
spark.read.json("file:///var/foo/try/<exp>")
+----------+---+-----+-------+
| <exp> -> | * | */* | */*/* |
+----------+---+-----+-------+
| logs1 | y | y | n |
| logs2 | n | y | y |
| logs3 | n | n | y |
+----------+---+-----+-------+
You can see in the above table that none of the three expressions matches all the directories (located at 3 different depths) at the same time. Frankly speaking, I wasn't expecting the exclusion of 10thOct_logs1 while using the third expression */*/*.
This makes me conclude that whatever files or directories path match against the expression following last / is considered as an exact match, and everything else is ignored.

Answer 1

Update
A new option was introduced in Spark 3 to read from nested folder recursiveFileLookup :
spark.read.option("recursiveFileLookup", "true").json("file:///var/foo/try")
For older versions, alternatively, you can use Hadoop listFiles to list recursively all the file paths and then pass them to Spark read:
import org.apache.hadoop.fs.{Path}
val conf = sc.hadoopConfiguration
// get all file paths
val fromFolder = new Path("file:///var/foo/try/")
val logfiles = fromFolder.getFileSystem(conf).listFiles(fromFolder, true)
var files = Seq[String]()
while (logfiles.hasNext) {
// one can filter here some specific files
files = files :+ logfiles.next().getPath().toString
}
// read multiple paths
val df = spark.read.csv(files: _*)
df.select(input_file_name()).distinct().show(false)
+-------------------------------------+
|input_file_name() |
+-------------------------------------+
|file:///var/foo/try/11thOct/log2.csv |
|file:///var/foo/try/10thOct_logs1.csv|
|file:///var/foo/try/Oct/12th/log3.csv|
+-------------------------------------+

Answer 2

Unfortunately hadoop globs do not support recursive globs. See Querying the Filesystem#File Patterns
There is an option to list multiple globs for each dir level.
{a,b} alternation Matches either expression a or b
You have to be careful not to match same file twice, otherwise it will appear as duplicate.
spark.read.json("./try/{*logs*,*/*logs*,*/*/*logs*}")
You can also load multiple dataframes and union them
val dfs = List(
spark.read.json("./try/*logs*"),
spark.read.json("./try/*/*logs*"),
spark.read.json("./try/*/*/*logs*")
)
val df = dfs.reduce(_ union _)