Simultaneously read Snowflake and s3 using Spark - amazon-web-services

Question

I tried to read data from s3 and snowflake simultaneously using spark and put it into snowflake after processing(join Operation).
During the tests, I found that each gives the same result but different performance.
(The second attempt was made for logs.)
v1 : Read from s3 and snowflake respectively, process "Join" operation and save snowflake
1st try : It took 2 hours, 46 minutes.
2nd try : It took 1 hour, 12 minutes.
v2 : Write the data B read from snowflake in s3, read it again, and save snowflake after "Join" operation
1st try : It took 5 minutes.
2nd try : It took 5 minutes.
v3 : Read from s3 and snowflake respectively, write the result to s3 and read it again, and save snowflake
1st try : It took 1h 11mins.
2nd try : It took 50 minutes.
Question. Why does each test perform differently?
I think v1 should be the fastest in that snowflake spark-connector is supported.
Here is test details and time information elapsed for each line,
Common config
Data A - stored in s3 : 20.2GB
Data B - stored in snowflake(internal table) : 7KB(195 rows)
spark config : r5.4xlarge(4) on-demand
snowflake warehouse : 3x-large
aws emr version : emr-5.31.0
spark version : 2.4.6
snowflake version : spark-snowflake_2.11
Test(using aws EMR)
v1 : Read from s3 and snowflake respectively, process "Join" operation and save snowflake
Result : OOM occurs after 2 hours ((Exit status 137.)
Change spark config by aws guide ".set("spark.sql.shuffle.partitions", "500") & resultDF.repartition(500).cache(): It took 2h 46mins.
object snowflake_v1 {
val spark_config = new SparkConf()
.set("fs.s3.maxConnections", "5000")
.set("spark.sql.broadcastTimeout", "1200")
.set("spark.sql.shuffle.partitions", "500")
.set("spark.network.timeout", "10000000")
.set("spark.executor.heartbeatInterval", "1000000")
val spark = SparkSession
.builder()
.master("yarn")
.config(spark_config)
.appName("snowflake")
.getOrCreate()
def main(args: Array[String]): Unit = {
//parquet at snowflake_v1.scala:27
val Adf =
spark.read
.parquet(
"s3://"
)
.cache()
var sfOptions = Map.apply(
"sfURL" -> "XXX",
"sfUser" -> "XXX",
"sfPassword" -> "XXX",
"sfDatabase" -> "XXX",
"sfSchema" -> "XXX",
"sfWarehouse" -> "XXX"
)
val Bdf: DataFrame = spark.sqlContext.read
.format(SNOWFLAKE_SOURCE_NAME)
.options(sfOptions)
.option("dbtable", "t_B")
.load()
val resDF =
Adf.join(Bdf, Seq("cnty"), "leftouter").cache()
val newDF = resultDF.repartition(500).cache() //If not, OOM occurs
newDF .write
.format(SNOWFLAKE_SOURCE_NAME)
.options(sfOptions)
.option("dbtable", "t_result_spark_v1")
.option("parallelism", "8")
.mode(SaveMode.Overwrite)
.save()
}
}
v2 : Write the data B read from snowflake in s3, read it again, and save snowflake after "Join" operation
Result : It took 5mins.
object snowflake_v2 {
val spark_config = "same as v1"
val spark = "same as v1"
def main(args: Array[String]): Unit = {
// parquet at snowflake_v2.scala:25
val Adf = "same as v1"
var sfOptions = "same as v1"
val Bdf: DataFrame = "same as v1"
//parquet at snowflake_v2.scala:53
Bdf.write
.mode(SaveMode.Overwrite)
.parquet("s3://..b")
//parquet at snowflake_v2.scala:56
val Bdf2=
spark.read.parquet("s3://..b")
val resDF =
Adf.join(Bdf2, Seq("cnty"), "leftouter").cache()
resDF .write
.format(SNOWFLAKE_SOURCE_NAME)
.options(sfOptions)
.option("dbtable", "t_result_spark_v2")
.option("parallelism", "8")
.mode(SaveMode.Overwrite)
.save()
}
}
v3 : Read from s3 and snowflake respectively, write the result to s3 and read it again, and save snowflake
Result : It took 30mins, but duplicated rows occur in s3 and snowflake
Before writing the result in s3, it takes 1h 11mins if I print the count so that the actual operation proceeds.(No duplication)
object snowflake_v3 {
val spark_config = "same as v1"
val spark = "same as v1"
def main(args: Array[String]): Unit = {
//parquet at snowflake_v3.scala:25
val Adf = "same as v1"
var sfOptions = "same as v1"
val Bdf: DataFrame = "same as v1"
val resDF =
Adf.join(Bdf, Seq("cnty"), "leftouter")
println("resDF count")
//count at snowflake_v3.scala:54
println(resDF.count) //If not, duplicated rows occur
//parquet at snowflake_v3.scala:58
resDF.write
.mode(SaveMode.Overwrite)
.parquet("s3://../temp_result")
//parquet at snowflake_v3.scala:65
val resDF2 =
spark.read.parquet("s3://../temp_result")
resDF2 .write
.format(SNOWFLAKE_SOURCE_NAME)
.options(sfOptions)
.option("dbtable", "t_result_spark_v3")
.option("parallelism", "8")
.mode(SaveMode.Overwrite)
.save()
}
}