Casting not working correctly in Amazon Athena (Presto)? - amazon-web-services

Question

I have a doctor license registry dataset which includes the total_submitted_charge_amount for each doctor as well as the number of entitlements with medicare & medicaid . I used the query from the answer suggested below:
with datamart AS
(SELECT npi,
provider_last_name,
provider_first_name,
provider_mid_initial,
provider_address_1,
provider_address_2,
provider_city,
provider_zipcode,
provider_state_code,
provider_country_code,
provider_type,
number_of_services,
CASE
WHEN REPLACE(num_entitlement_medicare_medicaid,',', '') ='' THEN
null
ELSE CAST(REPLACE(num_entitlement_medicare_medicaid,',', '') AS DECIMAL)
END AS medicare_medicaid_entitlement,
CASE
WHEN REPLACE(total_submitted_charge_amount,',', '') ='' THEN
null
ELSE CAST(REPLACE(num_entitlement_medicare_medicaid,',', '') AS DECIMAL)
END AS total_submitted_charge_amount
FROM cmsaggregatepayment2017)
SELECT *
FROM datamart
ORDER BY total_submitted_charge_amount DESC
Unfortunately I get the error
INVALID_CAST_ARGUMENT: Cannot cast VARCHAR '' to DECIMAL(38, 0)
This query ran against the aggregatepayment_data_2017 database, unless qualified by the query. Please post the error message on our forum or contact customer support with Query Id: be01d1e8-dc4d-4c75-a648-428dcb6be3a5." I have tried Decimal, Real, Big int and nothing works for casting num_entitlement_medicare_medicaid. Below is a screenshot of how the data looks like:
Can someone please suggest how to rephrase this query?

Answer 1

Instead of putting cast/replace in your queries, you could convert the data into a new table with 'clean' data:
CREATE TABLE clean_table
WITH (format='Parquet', external_location='s3://my_bucket/clean_data/')
AS
SELECT
npi,
provider_last_name,
provider_first_name,
...
CASE WHEN REPLACE(num_entitlement_medicare_medicaid,',', '') ='' THEN null
ELSE CAST(REPLACE(num_entitlement_medicare_medicaid,',', '') AS DECIMAL)
END AS medicare_medicaid_entitlement,
CASE WHEN REPLACE(total_submitted_charge_amount,',', '') ='' THEN null
ELSE CAST(REPLACE(num_entitlement_medicare_medicaid,',', '') AS DECIMAL)
END AS total_submitted_charge_amount
FROM cmsaggregatepayment2017
You can the SELECT ... FROM clean_table without having to do any conversions.
In data warehousing, this type of process is known as ETL (Extract, Transform, Load). The cleaning process is the 'transform' to convert the data into a more useful format.
See: CREATE TABLE AS - Amazon Athena

Answer 2

You might want to try try_cast() in presto. This version works on coercion. If there is any error it avoids it and moves to the next item.
Documention: https://prestodb.io/docs/current/functions/conversion.html

Answer 3

The reason you are getting error is you have blank value(but it is not null) in the column and we cannot cast varchar '' as decimal. You can probably use case statement. Also as per the data set column num_entitlement_medicare_medicaid has comma ',' in it which you are not replacing.
SELECT npi,
case
when REPLACE(num_entitlement_medicare_medicaid,'[^A-Za-z0-9.]', '') ='' then null
else CAST(REPLACE(num_entitlement_medicare_medicaid,'[^0-9.]', '') AS DECIMAL)
end as medicare_medicaid_entitlement,
case
when REPLACE(total_submitted_charge_amount,'[^A-Za-z0-9.]', '') ='' then null
else CAST(REPLACE(total_submitted_charge_amount,'[^0-9.]', '') AS DECIMAL)
end as total_submitted_charge_amount
FROM cmsaggregatepayment2017