Searching Athena AWS WAF logs by http header - amazon-web-services

Question

I've set up AWS WAF Logging to S3, and created an Athena table as described in the documentation ( https://docs.aws.amazon.com/athena/latest/ug/waf-logs.html )
However, http headers are stored as an array<struct<name: string, value: string>> not a map (for valid reasons). I'd like to write a query like
select headers.user-agent, headers.if-none-match from waf_logs where something;
if-none-match may or may not be present in the headers list.
using CROSS JOIN UNNEST(httprequest.headers) doesn't work, as this will create multiple rows. Using map notation doesn't work, as its a array of struct, not a map.
There are a lot of pages on the internet about how to set up the table, but not so many with worked examples of real-life queries, and none that I can find about how to query by nested attributes.
I'd really appreciate any suggestions. Thanks!

Answer 1

Here's a WAF query that should do the trick for request headers:
WITH waf_data AS (
SELECT
waf.action as action,
waf.httprequest.clientip as clientip,
waf.httprequest.country as country,
map_agg(f.name, f.value) AS kv
FROM "waf_logs" waf,
UNNEST(waf.httprequest.headers) AS t(f)
GROUP BY 1, 2, 3
)
SELECT
waf_data.action,
waf_data.clientip,
waf_data.country,
waf_data.kv['Host'] AS host,
waf_data.kv['User-Agent'] as UserAgent,
waf_data.kv['Cookie'] as cookie
FROM waf_data
WHERE waf_data.kv['Host'] like 'waf_alb.us-east-2.elb.amazonaws.com'
LIMIT 10;

Answer 2

I used the following to extract the HTTP header values (here Host field by name):
SELECT action, header.value as hostname, clientip, timestamp
from (
SELECT
httprequest.clientip as clientip,
action,
timestamp,
httprequest.headers as headers
FROM waf_logs
)
cross join unnest(headers) as c(header)
where lower(header.name) = 'host'
I used this post to extract the array of row which is formatted like array(row("name" varchar,"value" varchar)) in the waf logs database