Unit Testing Large SQL Queries in Databricks with Spark Scala [closed] - unit-testing

Question

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 11 hours ago.
Improve this question
I am trying to write some unit tests to ensure that data within our ETL pipeline is as expected. I would have generally expected unit tests to be a standard practice, but currently, at my place of work, the devs are not expected to write any tests (Bad I know).
We are ingesting data from on-prem source tables into Azure. We are then pulling the data from Azure into Databricks and Transforming that data into essentially a new table. The new table has data from various different source tables, which is then transferred across into a SaaS product.
As previously mentioned, there are currently no tests to ensure this SQL query is doing what it should do, but I am trying to change that. However, due to the nature of the project and how it has been developed, I feel like it has made testing at a unit level very difficult.
I have made fairly good progress with writing some tests that check that the source data is correctly being transformed into the new table, but I keep asking myself do I need to do more? It's difficult working in a place where noone seems to care about quality, but I do.
As an example, please see the below query, this is pulling the recently transferred source data and building that into a new table with a number of joins, and in my opinion, is becoming very messy:
DROP TABLE IF EXISTS platform.student;
CREATE OR REPLACE TABLE platform.student
(
STUDENT_ID string,
ULN string,
DOB string,
ETHNICITY string,
SEXID string,
DIFFLEARN1 string,
DIFFLEARN2 string,
DOMICILE string,
PARENTS_ED string,
SOCIO_EC string,
OVERSEAS string,
APPSHIB_ID string,
VLE_ID string,
HUSID string,
USERNAME string,
LAST_NAME string,
FIRST_NAME string,
ADDRESS_LINE_1 string,
ADDRESS_LINE_2 string,
ADDRESS_LINE_3 string,
ADDRESS_LINE_4 string,
POSTCODE string,
PRIMARY_EMAIL_ADDRESS string,
PERSONAL_EMAIL_ADDRESS string,
HOME_PHONE string,
MOBILE_PHONE string,
PHOTO_URL string,
ENTRY_POSTCODE string,
CARELEAVER string,
ADJUSTMENT_PLAN string,
COURSE_REGISTRATION_DATE string,
PROVIDED_AT string
);
insert into platform.student
(
STUDENT_ID,
ULN,
DOB,
ETHNICITY,
SEXID,
DIFFLEARN1,
DIFFLEARN2,
DOMICILE,
PARENTS_ED,
SOCIO_EC,
OVERSEAS,
APPSHIB_ID,
VLE_ID,
HUSID,
USERNAME,
LAST_NAME,
FIRST_NAME,
ADDRESS_LINE_1,
ADDRESS_LINE_2,
ADDRESS_LINE_3,
ADDRESS_LINE_4,
POSTCODE,
PRIMARY_EMAIL_ADDRESS, --University Email
PERSONAL_EMAIL_ADDRESS,
HOME_PHONE,
MOBILE_PHONE,
PHOTO_URL,
ENTRY_POSTCODE,
CARELEAVER,
ADJUSTMENT_PLAN,
COURSE_REGISTRATION_DATE,
PROVIDED_AT
)
select
STUDENT_ID,
ULN,
DOB,
ETHNICITY,
SEXID,
DIFFLEARN1,
DIFFLEARN2,
DOMICILE,
PARENTS_ED,
SOCIO_EC,
OVERSEAS,
APPSHIB_ID,
VLE_ID,
HUSID,
USERNAME,
LAST_NAME,
FIRST_NAME,
ADDRESS_LINE_1,
ADDRESS_LINE_2,
ADDRESS_LINE_3,
ADDRESS_LINE_4,
POSTCODE,
PRIMARY_EMAIL_ADDRESS, --University Email
PERSONAL_EMAIL_ADDRESS,
HOME_PHONE,
MOBILE_PHONE,
PHOTO_URL,
ENTRY_POSTCODE,
CARELEAVER,
ADJUSTMENT_PLAN,
COURSE_REGISTRATION_DATE,
PROVIDED_AT
from
(
SELECT DISTINCT
spriden_id AS STUDENT_ID,
case when skbspin_uln is null then 'NULL' else skbspin_uln end AS ULN,
DATE_FORMAT(spbpers_birth_date, 'yyyy-MM-dd') AS DOB,
stvethn_desc AS ETHNICITY,
'NULL' AS DIFFLEARN1,
'NULL' AS DIFFLEARN2,
'NULL' AS DOMICILE,
'NULL' AS PARENTS_ED,
'NULL' AS SOCIO_EC,
CASE WHEN sgbstdn_resd_code in ('1', '4', 'H') THEN '1'
WHEN sgbstdn_resd_code in ('5', '9', 'F', '7' ) THEN '3'
ELSE '99' END AS OVERSEAS,
CONCAT(syraccs_username, '#salford.ac.uk') AS APPSHIB_ID,
LOWER(syraccs_username) AS VLE_ID,
case when skbspin_husid is null then 'NULL' else skbspin_husid end AS HUSID,
syraccs_username AS USERNAME,
spriden_last_name AS LAST_NAME,
spriden_first_name AS FIRST_NAME,
tt.spraddr_street_line1 AS ADDRESS_LINE_1,
tt.spraddr_street_line2 AS ADDRESS_LINE_2,
tt.spraddr_street_line3 AS ADDRESS_LINE_3,
tt.spraddr_city AS ADDRESS_LINE_4,
tt.spraddr_zip AS POSTCODE,
goremal.goremal_email_address AS PRIMARY_EMAIL_ADDRESS, --University Email
personal_goremal.goremal_email_address AS PERSONAL_EMAIL_ADDRESS,
pr_sprtele.sprtele_intl_access AS HOME_PHONE,
mob_sprtele.sprtele_intl_access AS MOBILE_PHONE,
CONCAT('<URL>', spriden_id, '.png') AS PHOTO_URL,
'NULL' AS CARELEAVER,
PR_Address.spraddr_zip AS ENTRY_POSTCODE,
CASE
WHEN rap.student_id IS NULL THEN 'No'
ELSE 'Yes'
END AS ADJUSTMENT_PLAN,
spbpers_sex AS SEXID,
sfbetrm_ests_date AS COURSE_REGISTRATION_DATE,
'NULL' AS PROVIDED_AT,
row_number() OVER(PARTITION BY spriden_id ORDER BY skrsain_ucas_app_date desc) AS APPLICANT_ORDER_NUMBER
FROM global_temp.spriden
JOIN global_temp.spbpers ON spbpers_pidm = spriden_pidm
LEFT JOIN global_temp.spraddr tt ON spraddr_pidm = spriden_pidm
AND spraddr_atyp_code = 'TT'
AND spraddr_to_date IS NULL
AND spraddr_status_ind IS NULL
JOIN global_temp.skbspin ON skbspin_pidm = spriden_pidm
JOIN global_temp.skrsain ain1 ON ain1.skrsain_pidm = spriden_pidm
LEFT JOIN
(select distinct spraddr_pidm,spraddr_zip
from
global_temp.spraddr s1
join global_temp.skrsain on spraddr_pidm=skrsain_pidm
where
spraddr_atyp_code='PR'
and
spraddr_to_date is null
and
spraddr_seqno =(select max(spraddr_seqno)
from global_temp.spraddr s2
where s2.spraddr_pidm = s1.spraddr_pidm
and s2.SPRADDR_ATYP_CODE ='PR')) PR_Address
ON PR_Address.spraddr_pidm = spriden_pidm
JOIN(
SELECT MAX (ain2.skrsain_activity_date) AS skrsain_activity_date, ain2.skrsain_pidm
FROM global_temp.skrsain ain2
GROUP BY ain2.skrsain_pidm) ain3
ON ain3.skrsain_pidm = ain1.skrsain_pidm
AND ain3.skrsain_activity_date = ain1.skrsain_activity_date
JOIN global_temp.syraccs on syraccs_pidm = spriden_pidm
LEFT JOIN global_temp.stvethn on stvethn_code = spbpers_ethn_code
JOIN global_temp.sgbstdn stdn1 on sgbstdn_pidm = spriden_pidm
JOIN global_temp.sfbetrm ON sfbetrm_pidm = stdn1.sgbstdn_pidm AND sfbetrm_term_code = stdn1.sgbstdn_term_code_eff
LEFT JOIN global_temp.V_Maximiser_RAP_Status rap ON rap.student_id = spriden_id
JOIN global_temp.shrdgmr dgmr ON dgmr.shrdgmr_pidm = sgbstdn_pidm
AND dgmr.shrdgmr_program = sgbstdn_program_1
LEFT JOIN (SELECT goremal_pidm, goremal_email_address, goremal_emal_code, goremal_status_ind, goremal_activity_date
FROM global_temp.goremal mal
WHERE goremal_emal_code = 1
AND goremal_status_ind = 'A'
AND goremal_activity_date = (SELECT max(goremal_activity_date)
FROM global_temp.goremal g
WHERE goremal_pidm = mal.goremal_pidm
AND goremal_emal_code = 1
AND goremal_status_ind = 'A')) goremal
ON goremal_pidm = stdn1.sgbstdn_pidm
LEFT JOIN (SELECT goremal_pidm, goremal_email_address, goremal_emal_code, goremal_status_ind, goremal_activity_date
FROM global_temp.goremal mal
WHERE goremal_emal_code = 2
AND goremal_status_ind = 'A'
AND goremal_activity_date = (SELECT max(goremal_activity_date)
FROM global_temp.goremal g
WHERE goremal_pidm = mal.goremal_pidm
AND goremal_emal_code = 2
AND goremal_status_ind = 'A')) personal_goremal
ON personal_goremal.goremal_pidm = stdn1.sgbstdn_pidm
LEFT JOIN (SELECT DISTINCT sprtele_pidm,sprtele_intl_access, sprtele_tele_code, sprtele_status_ind,sprtele_seqno
FROM global_temp.sprtele s
WHERE sprtele_tele_code = 'MOB'
AND sprtele_status_ind IS NULL
AND sprtele_seqno = (SELECT MAX(sprtele_seqno)
FROM global_temp.sprtele s2
WHERE s2.sprtele_pidm = s.sprtele_pidm
AND sprtele_tele_code = 'MOB'
AND sprtele_status_ind IS NULL)) mob_sprtele
ON sprtele_pidm = stdn1.sgbstdn_pidm
LEFT JOIN (SELECT DISTINCT sprtele_pidm,sprtele_intl_access, sprtele_tele_code, sprtele_status_ind,sprtele_seqno
FROM global_temp.sprtele s
WHERE sprtele_tele_code = 'PR'
AND sprtele_status_ind IS NULL
AND sprtele_seqno = (SELECT MAX(sprtele_seqno)
FROM global_temp.sprtele s2
WHERE s2.sprtele_pidm = s.sprtele_pidm
AND sprtele_tele_code = 'PR'
AND sprtele_status_ind IS NULL)) pr_sprtele
ON pr_sprtele.sprtele_pidm = stdn1.sgbstdn_pidm
WHERE spriden_change_ind IS NULL
AND spriden_entity_ind = 'P'
--AND sfbetrm_ests_code IN ('RE', 'RS', 'RP', 'WU','EL')
AND (sfbetrm_ests_code IN ('RE', 'RS', 'RP', 'WU')
OR sfbetrm_ests_code = 'EL' AND substr(sgbstdn_blck_code, -1,1) > 1) -- include EL's on previous years except first year
AND stdn1.sgbstdn_program_1 IN (SELECT DISTINCT skrspri_program
FROM global_temp.skrspri
WHERE nvl(skrspri_frnchact,1) <> 3)
AND stdn1.sgbstdn_term_code_eff = (SELECT MAX(stdn2.sgbstdn_term_code_eff)
FROM global_temp.sgbstdn stdn2
WHERE stdn1.sgbstdn_pidm = stdn2.sgbstdn_pidm)
-- Previous 2 years data only
AND SUBSTR(stdn1.sgbstdn_term_code_eff,1,4) >= (
SELECT DISTINCT SUBSTR(stvterm_code,1,4) - 1 stvterm_code
FROM global_temp.stvterm
WHERE CURRENT_DATE BETWEEN stvterm_start_date AND stvterm_end_date)
AND dgmr.shrdgmr_degs_code <> 'AW'
AND dgmr.shrdgmr_seq_no =
(
SELECT MAX(dgmr2.shrdgmr_seq_no)
FROM global_temp.shrdgmr dgmr2
WHERE dgmr.shrdgmr_pidm = dgmr2.shrdgmr_pidm
AND dgmr.shrdgmr_program = dgmr2.shrdgmr_program)
AND EXISTS -- must have student record in last 2 years
(SELECT sgbstdn_pidm
FROM global_temp.sgbstdn stdn2
WHERE substr(stdn2.sgbstdn_term_code_eff,1,4) >= (
SELECT DISTINCT SUBSTR(stvterm_code,1,4) - 1 stvterm_code
FROM global_temp.stvterm
WHERE CURRENT_DATE BETWEEN stvterm_start_date AND stvterm_end_date)
AND stdn1.sgbstdn_pidm = stdn2.sgbstdn_pidm)
) a
where
APPLICANT_ORDER_NUMBER = 1
Tests, I have written so far (As an example), but these are more data validation checks?
test("Test - Date of Birth - Banner & Extracts Match") {
var dateOfBirth = spark.sql("""
SELECT distinct spriden.SPRIDEN_ID, spriden.SPRIDEN_PIDM, SPBPERS.SPBPERS_BIRTH_DATE, s.DOB
FROM global_temp.spriden spriden
JOIN global_temp.SPBPERS spbpers on spbpers.SPBPERS_PIDM = spriden.spriden_pidm
JOIN global_temp.V_SAL_EA_STUDENT s on s.STUDENT_ID = spriden.spriden_id
""")
dateOfBirth.as[DateOfBirth].collect().foreach(record => {
if (!dataMatches(s"${record.SPBPERS_BIRTH_DATE}", s"${record.DOB}")) {
println("Error between DOB records for student " + s"${record.SPRIDEN_ID}")
} else {
assert(s"${record.SPBPERS_BIRTH_DATE}" === s"${record.DOB}")
}
})
}
Has anyone ever unit tested large SQL queries like this? I have generally always wrote unit tests for units of code, which generally should be written in a way to do a specific job, which can then easily be identified and tested. I am not really looking for specific answers, but just ideally need someone who has done some thing similiar to give me some hints/tips. All of the articles I have found, just do not seem to do what I need them to.
To unit test, this, do I need to Mock out all of the ingested source data? But even if I did that (Which is a huge job) I would not know what the best way to run that is. If we were running that data against methods/functions then I could easily create tests but it will be being run against the SQL query, right?
Some articles I have found:
Scenario1
Databricks Unit Testing
Please get back to me if you have any information, or if you need any more info, please let me know :)
Thanks!