AWS Athena | CSV vs Parquet | size of data scanned - amazon-web-services

Question

TLDR : Athena: select top 10 scans more data for parquet format, than csv format. Shouldn't it be the other way round?
I am using Athena(V1) to query the following two datasets (same data but two different file formats):
Format
Size
Athena DB name
Athena table name
dataset description
CSV
91.3 MB
nycitytaxi
data
nycity taxi trip, present in a public s3 bucket
Parquet
19.4 MB
nycitytaxi
aws_glue_result_xxxx
same data as above converted to parquet - through a Glue Crawler job - and stored in one of my S3 buckets
Now I am executing the following query on both the tables :
select lpep_pickup_datetime, lpep_dropoff_datetime
from nycitytaxi.<table_name>
limit 10
On executing this query on the csv based table (table_name: data), Athena console shows it scanned 721.96 KB of data.
On executing this query on the parquet based table (table_name : aws_glue_result_xxxx), Athena console shows it scanned 10.9 MB of data.
Shouldn't Athena be scanning way less data for the parquet based table, since parquet is columnar based, as opposed to row based storage for CSV ?

Answer 1

It is due to your specific query.
select lpep_pickup_datetime, lpep_dropoff_datetime
from nycitytaxi.<table_name>
limit 10
In row based formats like CSV, all data is stored row wise. Which means as soon as you say, select any 10 rows, it can just start reading the csv file from the beginning and select the first 10 rows, resulting in very low data scan.
In columnar data formats like parquet, the records are stored column wise. Let us assume the data has three columns, say id, name, number. This means, all of id values will be stored together, all name values will be stored together and all number values will be stored together. So when you run the query, select 10 rows in parquet, i will have to scan for 10 values in each column which are present in different storage locations. Which means I will have to scan more.
More on parquet pros and cons here.