In Redshift query plan Network is taking 3 hours - amazon-web-services

Question

In Redshift one of the query is taking 3 hours to execute while analyzing its query plan it seems the network is taking all the time. How I could troubleshoot and resolve this problem.
Below is my query execution plan:
QUERY PLAN: Filter: ((to_date((productrequestdate)::text, 'YYYY-MM-DD'::text) <= '2022-01-31'::date) AND (to_date((productrequestdate)::text, 'YYYY-MM-DD'::text) >= '2020-01-01'::date) AND ((mstrclientid)::text = 'GSKUS'::text) AND (quantityrequested >= 0))
QUERY PLAN: -> XN Seq Scan on brsit_sample_transparency (cost=0.00..0.30 rows=1 width=4980)
QUERY PLAN: Filter: ((to_date((productrequestdate)::text, 'YYYY-MM-DD'::text) <= '2022-01-31'::date) AND (to_date((productrequestdate)::text, 'YYYY-MM-DD'::text) >= '2020-01-01'::date) AND ((mstrclientid)::text = 'GSKUS'::text) AND (quantityrequested >= 0))
QUERY PLAN: -> XN Seq Scan on verri_sample_transparency (cost=0.00..0.30 rows=1 width=4980)
QUERY PLAN: Filter: ((to_date((productrequestdate)::text, 'YYYY-MM-DD'::text) <= '2022-01-31'::date) AND (to_date((productrequestdate)::text, 'YYYY-MM-DD'::text) >= '2020-01-01'::date) AND (quantityrequested >= 0) AND ((mstrclientid)::text = 'GSKUS'::text))
QUERY PLAN: -> XN Seq Scan on gskus_sample_transparency (cost=0.00..33348.33 rows=5558 width=993)
QUERY PLAN: -> XN Multi Scan (cost=0.00..33404.53 rows=5560 width=4980)
QUERY PLAN: -> XN Subquery Scan bi_sample_transparency_view (cost=0.00..33460.13 rows=5560 width=1488)
QUERY PLAN: Sort Key: productndc10
QUERY PLAN: -> XN Sort (cost=1000000033805.99..1000000033819.89 rows=5560 width=1488)
QUERY PLAN: Send to leader
QUERY PLAN: -> XN Network (cost=1000000033805.99..1000000033819.89 rows=5560 width=1488)
QUERY PLAN: Merge Key: productndc10
QUERY PLAN:XN Merge (cost=1000000033805.99..1000000033819.89 rows=5560 width=1488)

Answer 1

As you say this is a problematic step in the plan (network transfer before SORT which isn't really a plan step but an activity that needs to be performed). With only 5560 rows being reported it doesn't seem like this should be a ton of data but your column count is high and I don't know the sizes of these columns. It could be there is a lot of data moving even for this limited number of rows. Or it could be that the reported number of rows is not indicative of the number of rows being moved in the network activity which can happen but this would need to be a huge difference. You can look at stl_dist for this query to see exactly how much data (bytes) is being moved.
Another possibility here is that your query was a victim not the culprit. You see Redshift is a cluster and clusters are connected by networks and these networks are common infrastructure for all queries running on the cluster. If there was a really bad query running during this window which browned out the internode network (bandwidth hog) then you query was caught up in this traffic jam. Does your query run normally most of the time but just went slow this time? What was the cluster activity like at this time? Were other queries impacted? I've debugged plenty of "slow" queries that were victims. That said it is always good in a clustered database like Redshift to not transfer excessive amounts of data on the network due to its clustered nature.
If you want to debug this query further (it is the culprit) then the query text, stl_dist information, and explain plan could shine some more light on the situation.