AWS Glue Job hangs from time to time - amazon-web-services

Question

I've got an AWS Glue job, which reads data from 22 MySQL tables, transforms it using queries and fills in 8 MySQL tables in different schema.
These are fairly easy queries - few joins that execute in maximum few seconds.
All tables combined have around 1.5 mln records. I run this job every 4 hours incrementally - amount of records inserted each time is between 100 and 5000. Example of one of the parts of the script, containing the query:
# SOURCE TABLES
item_node1647201451763 = glueContext.create_dynamic_frame.from_catalog(
database="db-" + args['ENVIRONMENT'],
table_name="item",
transformation_ctx="chopdb2item_node1647201451763",
)
inventory_node1647206574114 = glueContext.create_dynamic_frame.from_catalog(
database="bi-db-" + args['BIDB_ENVIRONMENT'],
table_name="inventory"
)
# [...OTHER SOURCE TABLE DECLARATIONS HERE...]
# SQL
SqlQuery12 = """
select distinct r.id receipt_id,
i.total value_total,
i.qty quantity,
d2.id receipt_upload_date_id,
i.text item_name,
i.brand item_brand,
i.gtin item_gtin,
i.ioi,
cp.client_promotion_id,
inv.inventory_id
from receipt r
join item i on i.receipt_id = r.id
join verdict v on r.verdict_id = v.id and v.awardable = 1
join account a on v.account_id = a.id
join offer o on v.offer_id = o.id
join date_dimension d2
on d2.occurence_date = DATE(r.upload_time)
left join client_promotion cp
on cp.client_key = a.client_key and
cp.promotion_key = o.promotion_key
left join inventory inv on inv.inventory_gtin = i.gtin
"""
extractitemfacttabledata_node1647205714873 = sparkSqlQuery(
glueContext,
query=SqlQuery12,
mapping={
"item": item_node1647201451763,
"receipt": receipt_node1647477767036,
"verdict": verdict_without_bookmark,
"account": account_without_bookmark,
"offer": offer_without_bookmark,
"date_dimension": date_dimension_node1649721691167,
"client_promotion": client_promotion_node1647198052897,
"inventory": inventory_node1647206574114,
},
transformation_ctx="extractitemfacttabledata_node1647205714873",
)
# WRITING BACK TO MYSQL DATABASE
itemfacttableinwarehouse_node1647210647655 = glueContext.write_from_options(
frame_or_dfc=extractitemfacttabledata_node1647205714873,
connection_type="mysql",
connection_options={
"url": f"{args['DB_HOST_CM']}:{args['DB_PORT_CM']}/chopbidb?serverTimezone=UTC&useSSL=false",
"user": args['DB_USERNAME_CM'],
"password": args['DB_PASSWORD_CM'],
"dbtable": "bidb.item", # THIS IS DIFFERENT SCHEMA THAN THE SOURCE ITEM TABLE
"bulksize": 1
}
)
Structure of the file is:
ALL source tables declaration
ALL transformations (SQL queries)
ALL writes (inserting back to MySQL database)
My problem is, that from time to time, the job hangs and runs indefinitely. I set timeout to 8h and it's in progress for 8h and then cancelled.
I installed Apache Spark history UI and tried to analyze the logs.
The results are as follows:
90% of the time this is the query quoted above, but sometimes it's successful and the next one fails. Looks like it might fail when trying to insert the data.
Usually it happens when there is very little load on the system (in the middle of the night).
What I tried:
in first version of the script I inserted the data using glueContext.write_dynamic_frame.from_catalog. I thought it might be the issue with bulk inserts, which are causing the deadlocks in database, but changing it to write_from_options with bulksize = 1 did not help
increased resources in RDS -> did not help
moving insert to this particular table further in the script usually also resulted in it's failing