Automatically update athena partition - MSCK Repair - amazon-web-services

Question

I have a inventory Bucket - inside the bucket - I have 6 Folders.
In Athena for each 6 folders - i have 6 tables in athena.
Now i have to update the paritions - as and when a file is dropped into any one of the 6 folders.
How do i write multiple sql (6 Sql) in one lambda for s3 event trigger.
import boto3
def lambda_handler(event, context):
bucket_name = 'some_bucket'
client = boto3.client('athena')
config = {
'OutputLocation': 's3://' + bucket_name + '/',
'EncryptionConfiguration': {'EncryptionOption': 'SSE_S3'}
}
# Query Execution Parameters
sql = 'MSCK REPAIR TABLE some_database.some_table'
context = {'Database': 'some_database'}
client.start_query_execution(QueryString = sql,
QueryExecutionContext = context,
ResultConfiguration = config)
Database is same ; however i have 6 different tables. I have to update all 6 tables.

Answer 1

First I would check the key of the dropped file and only update the table that points to the prefix where the file was dropped. E.g. if your folders and tables are prefix0, prefix1, prefix2, etc. and the dropped file has the key prefix1/some-file you update only the table with the location prefix1. There is no need to update the other tables, their data hasn't changed.
However, I would suggest not using MSCK REPAIR TABLE for this. That command is terrible in almost every possible way. It's wildly inefficient and its performance becomes worse and worse as you add more objects to your table's prefix. It doesn't look like you wait for it to complete in your Lambda, so at least you're not paying for its inefficiency, but there are much better ways to add partitions.
You can use the Glue APIs directly (under the hoods Athena tables are tables in the Glue catalog), but that is actually a bit complicated to show since you need to specify a lot of metadata (a downside of the Glue APIs).
I would suggest that instead of the MSCK REPAIR TABLE … call you do ALTER TABLE ADD PARTITION …:
Change the line
sql = 'MSCK REPAIR TABLE some_database.some_table'
to
sql = 'ALTER TABLE some_database.some_table ADD IF NOT EXISTS PARTITION (…) LOCATION \'s3://…\''
The parts where it says … you will have to extract from the object's key. If your keys look like s3://some-bucket/pk0=foo/pk1=bar/object.gz and your table has the partition keys pk0 and pk1 the SQL would look like this:
ALTER TABLE some_database.some_table
ADD IF NOT EXISTS
PARTITION (pk0 = 'foo', pk1 = 'bar') LOCATION 's3://some-bucket/pk0=foo/pk1=bar/'