BigQuery - BATCH mode queries randomly failing with resources exceeded - google-cloud-platform

Question

A part of our application submits 100s of BQ queries to build an aggregated dashboard view. Due to the large number of queries, we submit all queries with BATCH priority.
In last 24 hours we've had quite a few queries (more than 300) fail due to resources exceeded. This is odd as some of the failed queries save results to an output table specifically to avoid this type of issue. We spot checked a few others and are able to run them without issue in the BQ web UI.
Sample job ID (no dest table): job_GPghMyEBvlYFBGBHI5Szv0HMIpMY
Query:
SELECT order_sub.value as value, count(*) as count
FROM
(
SELECT order_id, drop_ship as value
FROM (TABLE_QUERY(d34f20170905T212849Zc4abca, 'table_id IN ("order_detail_2018")'))
WHERE order_date BETWEEN TIMESTAMP("2018-01-01") AND TIMESTAMP("2018-08-31")
) order_sub
JOIN EACH
(
SELECT customer_id, order_id, sequence
FROM temp_30_days.query_q6109c34f20180918T151206Ze52003_buyers
) cust_sub
ON order_sub.order_id = cust_sub.order_id
GROUP BY value
ORDER BY value
Error Message:
"status": {
"state": "DONE",
"errorResult": {
"reason": "resourcesExceeded",
"message": "Resources exceeded during query execution."
},
"errors": [
{
"reason": "resourcesExceeded",
"message": "Resources exceeded during query execution."
}
]
}
Sample job ID (with dest table): job_2YsCt-cxql5TP0jklb1aRENzINdZ
Query:
SELECT
customer_id, sequence
FROM
(
SELECT
customer_id,
sequence,
random,
MAX(random) OVER (PARTITION BY customer_id) AS max_value
FROM
(
SELECT customer_id, sequence, RAND() as random
FROM (TABLE_QUERY(d34f20170905T212849Zc4abca, 'table_id IN ("cust_char_2015", "cust_char_2016", "cust_char_2017")'))
WHERE end_date >= TIMESTAMP("2016-12-31")
AND effective_date <= TIMESTAMP ("2017-08-31")
AND (frequency_end = "98")
)
)
WHERE max_value = random
GROUP EACH BY
customer_id, sequence
Error Message:
"status": {
"errorResult": {
"message": "Resources exceeded during query execution.",
"reason": "resourcesExceeded"
},
"errors": [
{
"message": "Resources exceeded during query execution.",
"reason": "resourcesExceeded"
}
],
"state": "DONE"
}
FWIW status.cloud.google.com shows green lights for BigQuery, is anyone else experiencing similar issues?