Filling in missing dates Redshift - amazon-web-services

Question

I have a table that looks like this:
Account Value Last_Day_in_Month
ABC 7 2018-06-30
ABC 12 2018-06-30
ABC 3 2018-08-31
FGH 57 2019-01-31
FGH 13 2019-03-31
FGH 127 2019-03-31
For each account, I need to fill in the missing dates corresponding to the last day in each month such that the resulting table just fills in the value from the last month (you'll notice two additional rows)
Account Value Last_Day_in_Month
ABC 7 2018-06-30
ABC 12 2018-06-30
ABC 12 2018-07-31
ABC 3 2018-08-31
FGH 57 2019-01-31
FGH 57 2019-02-28
FGH 13 2019-03-31
FGH 127 2019-03-31
I have many accounts each of them with different start and stop times (Last_Day_in_Month) so I only need to fill in the missing months between the min and max months for each account. Because I may have multiple values corresponding to one single month end date per account, my current solution is to use a lead with a case statement that adds a single day and a date table that contains only the last day of each month and perform a cross join. But, I think it's messy and I'm sure there's a better way that I'm not aware of. Here is my current solution...
select
*,
lead(Last_Day_in_Month,1)over (
partition by Account
order by Last_Day_in_Month
) as intermed2,
case
when intermed2 = Last_Day_in_Month
then dateadd('day',1, intermed2)
else intermed2
end as next_last_day
from table
cross join dates
where dates.date_actual >= table.Last_Day_in_Month
and dates.date_actual < table.next_last_day
Any suggestions are appreciated.

Answer 1

What you are doing is fine for reasonable numbers of rows. One thing I'd recommend for clarity is changing from a cross join to a right join with an ON clause. The query planner should see right through what you have and plan an efficient query so just a nit.
There are a number of other ways to do this and you can find examples by searching "gaps and islands" in stack overflow. The biggest feedback I have is about creating additional rows. What you are doing is making new rows for the missing months which is fine for reasonably small tables because they don't get super large when you add rows. For example if you have a table with 100 billion rows and you have an average gap size of 2, you will be creating a result with 300 billion rows. Making this much data will never be fast or efficient. So you say you have "many accounts", how many is many?
If the amount of data can fit in memory or you are doing this operation only once in a while then creating rows will work ok. If this is being done as part of ongoing queries and the data created will be large then I'd rethink why you need to create data to get your queries to execute. In general Redshift stores very large sets of data and multiplying out (cross join) these rows by some other factor (dates) will result in very slow queries. If the the intent is to pared this data down to some smaller result you will want to find a way to create this result without making such a large intermediate dataset.