How to create parquet partitioned table - amazon-athena

Question

Here is a sample create table statement that works as expected.
CREATE EXTERNAL TABLE default.reviews(
marketplace varchar(10),
customer_id varchar(15),
review_id varchar(15),
product_id varchar(25),
product_parent varchar(15),
product_title varchar(50),
star_rating int,
helpful_votes int,
total_votes int,
vine varchar(5),
verified_purchase varchar(5),
review_headline varchar(25),
review_body varchar(1024),
review_date date,
year int)
PARTITIONED BY (
product_category varchar(25))
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
LOCATION
's3://amazon-reviews-pds/parquet/';
When I repair the table, I get an error:
MSCK REPAIR TABLE default.reviews
Partitions not in metastore: reviews:product_category=Apparel reviews:product_category=Automotive
If the Partition is not in metastore, how do I get a count of 3.5 million?
SELECT
COUNT(*)
FROM
"default"."reviews"
WHERE
product_category='Automotive'
-- OUTPUT
3516476
How do I make sure that all the records are correctly read and available?
How was this parquet partitioned table was created? I am asking because I have a csv table that I will like to partition exactly the same way.

Answer 1

Concept of partitioning is used in Athena only to restrict which "directories" should be scanned for data.
Since MSCK REPAIR TABLE command failed, no partitions were created. Therefore, WHERE product_category='Automotive'
doesn't have any affect and I'd say that 3516476 is the total number of rows in all csv files under s3://amazon-reviews-pds/parquet/.
Note, MSCK REPAIR TABLE would only work if "folder" structure on AWS S3 adheres HIVE convention:
s3://amazon-reviews-pds/parquet/
|
├── product_category=Apparel
│ ├── file_1.csv
│ | ...
│ └── file_N.csv
|
├── product_category=Automotive
│ ├── file_1.csv
│ | ...
│ └── file_M.csv
In order to ensure that all records are correctly read you have to make sure that table definition is correct.
In order to ensure that all records are available you have to make sure that LOCATION points to the root
"directory" of where all files are located on S3.
If you have a huge csv file with columns col_1, col_2, col_3, col_4 and you want to partition it by col_4 you would
need to use CTAS query statements, however, keep in mind
limitations of such statements.
Alternatively, if you already have multiple csv files each of which corresponds to a single value from col_4, then
simply upload them onto S3 in a way mentioned above. Then you should use combination of the following DDL statements:
-- FIRST STATMENT
CREATE EXTERNAL TABLE `my_database`.`my_table`(
`col_1` string,
`col_2` string,
`col_3` string,
)
PARTITIONED BY (
`col_4` string)
ROW FORMAT SERDE
-- CHANGE AS APPROPRIATE
'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
STORED AS INPUTFORMAT
-- CHANGE AS APPROPRIATE
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'
OUTPUTFORMAT
-- CHANGE AS APPROPRIATE
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
LOCATION
's3://amazon-reviews-pds/parquet/';
-- SECOND STATEMENT
MSCK REPAIR TABLE `my_database`.`my_table`