How to find average time to load data from S3 into Redshift - amazon-web-services

Question

I have more than 8 schemas and 200+ tables and data is loaded by CSV files in different schema.
I want to to know the SQL script for how to find average time to load the data from S3 into Redshift for all 200 tables.

Answer 1

You can examine the STL System Tables for Logging to discover how long queries took to run.
You'd probably need to parse the Query text to discover which tables were loaded, but you could use the historical load times to calculate a typical load time for each table.
Some particularly useful tables are:
STL_QUERY_METRICS: Contains metrics information, such as the number of rows processed, CPU usage, input/output, and disk use, for queries that have completed running in user-defined query queues (service classes).
STL_QUERY: Returns execution information about a database query.
STL_LOAD_COMMITS: This table records the progress of each data file as it is loaded into a database table.

Answer 2

Run this query to find out how fast your COPY queries are working.
select q.starttime, s.query, substring(q.querytxt,1,120) as querytxt,
s.n_files, size_mb, s.time_seconds,
s.size_mb/decode(s.time_seconds,0,1,s.time_seconds) as mb_per_s
from (select query, count(*) as n_files,
sum(transfer_size/(1024*1024)) as size_MB, (max(end_Time) -
min(start_Time))/(1000000) as time_seconds , max(end_time) as end_time
from stl_s3client where http_method = 'GET' and query > 0
and transfer_time > 0 group by query ) as s
LEFT JOIN stl_Query as q on q.query = s.query
where s.end_Time >= dateadd(day, -7, current_Date)
order by s.time_Seconds desc, size_mb desc, s.end_time desc
limit 50;
Once you find out how many mb/s you're pushing through from S3 you can roughly determine how long it will take each file based on the size.

Answer 3

There's a smart way to do it. You ought to have an ETL script that migrates data from S3 to Redshift.
Assuming that you have a shell script, just capture the time stamp before the ETL logic starts for that table (let's call that start), capture another timestamp after the ETL logic ends for that table (let's call that end) and take the difference towards the end of the script:
#!bin/sh
.
.
.
start=$(date +%s) #capture start time
#ETL Logic
[find the right csv on S3]
[check for duplicates, whether the file has already been loaded etc]
[run your ETL logic, logging to make sure that file has been processes on s3]
[copy that table to Redshift, log again to make sure that table has been copied]
[error logging, trigger emails, SMS, slack alerts etc]
[ ... ]
end=$(date +%s) #Capture end time
duration=$((end-start)) #Difference (time taken by the script to execute)
echo "duration is $duration"
PS: The duration will be in seconds and you can maintain a log file, entry to a DB table etc. The timestamp will be in epoc and you can use functions (depending upon where you're logging) like:
sec_to_time($duration) --for MySQL
SELECT (TIMESTAMP 'epoch' + 1511680982 * INTERVAL '1 Second ')AS mytimestamp -- for Amazon Redshift (and then take the difference of the two instances in epoch).