Redshift - SQL - Cumulative average for grouped results - amazon-web-services

Question

I currently have a table like so:
Date
customer_id
sales
1/1
1
1
1/1
1
1
1/1
1
1
1/1
2
1
1/2
2
3
1/2
2
1
1/2
1
2
1/2
1
1
1/3
1
2
1/3
2
2
1/3
2
3
1/3
2
3
This eventually gets aggregated by the customer_id to get total_sales like so:
customer_id
total_sales
1
8
2
13
I then calculate one metric based off of this table, average_sales, which is defined as:
sum(total_sales) / count(distinct customer_id)
This would result in average_sales of 10.5 based on the information above.
However, I need to find a way to calculate this average but for each day on a cumulative basis like so:
Date 1/1 would be sum(total sales) for 1/1 / count(distinct customer_ids) for 1/1
Date 1/2 would be sum(total sales) for 1/1-1/2 / count(distinct customer_ids) for 1/1-1/2
Date 1/3 would be sum(total sales) for 1/1-1/3 / count(distinct customer_ids) for 1/1-1/3
The final day(1/3) should be equal to the overall average metric of 10.5.
Final table should look like this:
Date
average_sales
1/1
2 (4/2)
1/2
5.5 (11/2)
1/3
10.5 (21/2)
I've tried multiple things thus far with grouping/window functions but can't seem to get the right numbers. Any help would be greatly appreciated!

Answer 1

The main problem is that you can't use COUNT(DISTINCT) with a window.
But, there's a hacky way to calculate it anyway.
Work out the first month each customer id appears
Rank the customers in order of when they appeared
MAX(customer_rank) is then the number of customers seen to date
This gives...
WITH
check_first_date AS
(
SELECT
*,
MIN(date_id) OVER (PARTITION BY cust_id) AS cust_id_first_date
FROM
example
),
rank_customers_by_time AS
(
SELECT
*,
DENSE_RANK() OVER (ORDER BY cust_id_first_date, cust_id) AS cust_rank
FROM
check_first_date
)
SELECT
date_id,
MAX(MAX(cust_rank)) OVER (ORDER BY date_id) AS customers_to_date,
SUM(SUM(sales)) OVER (ORDER BY date_id) AS sales_to_date
FROM
rank_customers_by_time
GROUP BY
date_id
ORDER BY
date_id
Then you can divide one by the other.
There are other ways to do the count-distinct over time, such as using correlated sub-queries. I suspect (I haven't tested) that it's even slower though.
SELECT
date_id,
(
SELECT COUNT(DISTINCT lookup.cust_id)
FROM example AS lookup
WHERE lookup.date_id <= example.date_id
)
AS customers_to_date,
SUM(SUM(sales)) OVER (ORDER BY date_id) AS sales_to_date
FROM
example
GROUP BY
date_id
ORDER BY
date_id
Here is a demo (using postgresql, as the clostest approximation to redshift) with slightly different data to show that it works even when customer id's appear 'out of order'.
https://dbfiddle.uk/?rdbms=postgres_9.6&fiddle=a5a37f3337e42123424c5cf1dbfe0152
EDIT: An even shorter (faster?) version with windows
For each customer_id, identify which is their first row (implicitly requires the rows to have a unique id).
The sum up the number of first rows that have occurred to date...
WITH
check_first_occurrence AS
(
SELECT
*,
MIN(id) OVER (PARTITION BY cust_id) AS cust_id_first_id
FROM
example
)
SELECT
date_id,
SUM(SUM(CASE WHEN id = cust_id_first_id THEN 1 ELSE 0 END)) OVER (ORDER BY date_id) AS customers_to_date,
SUM(SUM(sales )) OVER (ORDER BY date_id) AS sales_to_date
FROM
check_first_occurrence
GROUP BY
date_id
ORDER BY
date_id
https://dbfiddle.uk/?rdbms=postgres_9.6&fiddle=94e5fb624a89170aaf819e2b3ccd01d6
This version should be significantly more friendly to RedShift's horizontal scaling.
Assuming, for example, you distribute by customer and sort by date