How to CTAS with external location as csv.gz - amazon-web-services

Question

I have close to 90 GB of data that needs to be uploaded to an S3 bucket with a specific naming convention.
If I use CTAS query with external_location it does not give me the option to give the file a specific name. Additionally with format csv is not an option.
CREATE TABLE ctas_csv_partitioned
WITH (
format = 'TEXTFILE',
external_location = 's3://my_athena_results/ctas_csv_partitioned/',
partitioned_by = ARRAY['key1']
)
AS SELECT name1, address1, comment1, key1
FROM tables1
I want to upload the output file so it look as sample_file.csv.gz
What is the easiest way to go about this?

Answer 1

Unfortunately, there is no way to specify neither file name nor the extension for it with Athena alone. Moreover, files created with CTAS query won't have any file extension at all. However, you can rename files directly with CLI for S3.
aws s3 ls s3://path/to/external/location/ --recursive \
| awk '{cmd="aws s3 mv s3://path/to/external/location/"$4 " s3://path/to/external/location/"$4".csv.gz"; system(cmd)}'
Just have tried this snippet and everything worked fine. However, sometimes an empty file s3://path/to/external/location/.csv.gz would also got created. Note I didn't include --recursive option for aws s3 mv since it would also produce weird results.
As far as format field is concerned, then you simply need to add field_delimiter=',' into WITH clause.
CREATE TABLE ctas_csv_partitioned
WITH (
format = 'TEXTFILE',
field_delimiter=','
external_location = 's3://my_athena_results/ctas_csv_partitioned/',
partitioned_by = ARRAY['key1']
)
AS SELECT name1, address1, comment1, key1
FROM tables1