AWS Athena. Can't use CSV manifest as a location - amazon-web-services

Question

I'm trying to create an external table in Athena, the problem is that the s3 bucket has different files in the same folder so I can't use the folder as a location.
I can't modify the path of the s3 files but I have a CSV manifest, I was trying to use it as a location but Athena didn't allow me to do that.
CREATE EXTERNAL TABLE `my_DB`.`my_external_table`(
column1 string,
column2 string
)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
WITH SERDEPROPERTIES (
'separatorChar' = ',',
'quoteChar' = '\"',
'escapeChar' = '\\'
)
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
's3://mys3bucket/tables/my_table.csvmanifest'
TBLPROPERTIES (
'has_encrypted_data'='false',
'skip.header.line.count'='1')
Any ideas to use my manifest? or another way to solve this without Athena? The goal of using Athena was to avoid getting all the data from the CSVs since I only need few records

Answer 1

You'll need to make a couple changes to your CREATE TABLE statement:
use 'org.apache.hadoop.hive.ql.io.SymlinkTextInputFormat' as your INPUTFORMAT
Ensure you're pointing to a folder with your LOCATION statement
So your statement would look like:
CREATE EXTERNAL TABLE `my_DB`.`my_external_table`(
column1 string,
column2 string
)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
WITH SERDEPROPERTIES (
'separatorChar' = ',',
'quoteChar' = '\"',
'escapeChar' = '\\'
)
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.SymlinkTextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
's3://mys3bucket/tables/my_table/'
And s3://mys3bucket/tables/my_table/ would have a single file in it with the S3 paths of the CSV files you want query - one path per line. I'm unsure if the skip.header.line.count setting will operate on the manifest file itself or the CSV files so you will have to test.
Alternatively, if you have a limited number of files, you could use S3 Select to query for specific columns in those files, one at a time. Using the AWS CLI, the command to extract the 2nd column would look something like:
aws s3api select-object-content \
--bucket mys3bucket \
--key path/to/your.csv.gz \
--expression "select _2 from s3object limit 100" \
--expression-type SQL \
--input-serialization '{"CSV": {}, "CompressionType": "GZIP"}' \
--output-serialization '{"CSV":{}}' \
sample.csv
(Disclaimer: AWS employee)