AWS WAF Log Partition Projection: Partitions not in metastore - amazon-athena

Question

Im trying to set up a table in Athena with partition projection.
My logs are in the format s3://bucket/folder/year/month/day/hour and then a json file inside that.
I have tried creating the table with partition projection as follows:
CREATE EXTERNAL TABLE `waf_logs_webacl1`(
`timestamp` bigint,
`formatversion` int,
`webaclid` string,
`terminatingruleid` string,
`terminatingruletype` string,
`action` string,
`terminatingrulematchdetails` array<
struct<
conditiontype:string,
location:string,
matcheddata:array<string>
>
>,
`httpsourcename` string,
`httpsourceid` string,
`rulegrouplist` array<
struct<
rulegroupid:string,
terminatingrule:struct<
ruleid:string,
action:string,
rulematchdetails:string
>,
nonterminatingmatchingrules:array<
struct<
ruleid:string,
action:string,
rulematchdetails:array<
struct<
conditiontype:string,
location:string,
matcheddata:array<string>
>
>
>
>,
excludedrules:array<
struct<
ruleid:string,
exclusiontype:string
>
>
>
>,
`ratebasedrulelist` array<
struct<
ratebasedruleid:string,
limitkey:string,
maxrateallowed:int
>
>,
`nonterminatingmatchingrules` array<
struct<
ruleid:string,
action:string
>
>,
`requestheadersinserted` string,
`responsecodesent` string,
`httprequest` struct<
clientip:string,
country:string,
headers:array<
struct<
name:string,
value:string
>
>,
uri:string,
args:string,
httpversion:string,
httpmethod:string,
requestid:string
>,
`labels` array<
struct<
name:string
>
>
)
PARTITIONED BY
(
day STRING
)
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION 's3://bucket/folder/'
TBLPROPERTIES
(
"projection.enabled" = "true",
"projection.day.type" = "date",
"projection.day.range" = "2021/01/01,NOW",
"projection.day.format" = "yyyy/MM/dd/HH",
"projection.day.interval" = "1",
"projection.day.interval.unit" = "YEARS",
"storage.location.template" = "s3://bucket/folder/${year}/${month}/${day}/${hour}/"
)
It gets created successfully but when I load all the partitions in it gives me the error
Partitions not in metastore: waf_logs_webacl1:2021/05/16/23 waf_logs_webacl1:2021/05/17/00 waf_logs_webacl1:2021/05/17/01 waf_logs_webacl1:2021/05/17/02 waf_logs_webacl1:2021/05/17/03 etc
I have also tried with the storage.location.template being s3://bucket/folder/ and s3://bucket/folder/${year}/ and get the same error when loading partitions. Please help thanks.

Answer 1

When you use partition projection you don't need to load partitions, the partitions will be found at query execution time.
The problem with your table is that you have one partition key, day, but you say to Athena that the data is stored in a directory structure containing /${year}/${month}/${day}/${hour}/, i.e. four partition keys.
Either you need to create the table with all four partition keys and configure partition projection for them (e.g. projection.year.type, etc.) or you need to remove the undefined keys from the storage location template.
I think the right course of action is the former, since that's how the data is organized. There is an example in the Athena docs that you should be able to use as a starting point here: https://docs.aws.amazon.com/athena/latest/ug/partition-projection-kinesis-firehose-example.html (edit: this page has been removed from the documentation and there is no updated version)