Stop Iterative API Pagination Requests When Data "Caught Up"? - powerbi

Question

I've created a minimal reproducible example below for testing, using the free paginated API endpoint https://api.instantwebtools.net/v1/passenger?page=0&size=10 for anyone to use. This endpoint is perfect because it closely resembles my real use case in what I'm looking for.
I'm using an iterative loop process to update page=0 with page=i in the URL endpoint during the iteration to get multiple pages of data. The response of each API call contains a page of 10 records with no primary key or index of any kind.
What I'm looking for is every time I run this query/code, I want to stop the request iteration if we start seeing duplicate results that are already in the previously saved table. My use case makes thousands of API calls, and the latest information is always retrieved first, so I'm looking to just add the latest records from the API to the top of my table, and once we start receiving data that's already in the table, we can stop requesting as we don't have to make the remaining calls anymore.
Specifically, I need a way to determine if all 10 records returned from the current query are already in the previously stored records, and stop the iteration if they are. Since there is no primary key or index of any kind, I think a good way to do this would be to compare the row values.
A pseudo-workflow would look something like:
Run the code once to retrieve and store some records.
Run the code again.
Upon retrieval of the current response, compare the records in the response with the records previously saved in the table:
If the response records are NOT in the table yet, add them to the top of the existing dataset and continue to the next iteration
If the response records ARE in the table, stop the iteration.
An idea I have here to mitigate a double for loop, because this would be helpful to my use case, is to:
Check how many records are in the current response (this example API will return 10 all the time, however my use case is generally between 48-52, it changes randomly, so make this dynamic)
Iterate through the currently saved table with a step size == the length of the response, and check if the current response's "chunk" of records appears in the table (so if we start at j=0in the table, and the response has 9 records in it, compare rows j:j+(9-1) from the table with the current response rows, and if they match, we can end the iteration).
This would be especially helpful for me because sometimes records are retroactively added to some previous datetimes, so to just stop iteration when only one duplicate row is found would not be enough. Being able to compare a chunk of roughly 50 rows for equality in the table, I can THEN assume that I can stop making any more requests as that could mark the end of any preceding changes.
Here's the code to get you started, this performs the iterative requesting process successfully, but need to work in the "checking for duplicate row chunks" logic:
let
// Define some variables
_current_page = 0,
_base_url = "https://api.instantwebtools.net/v1/passenger?page=",
_end_url = "&size=10",
_start_url = _base_url & Number.ToText(_current_page) & _end_url,
_iterations = 5,
// Define a function to retrieve each page's response from the API
FnGetOnePage = (input_url) as record =>
let
// Random sleep function re: API limits
sleep = Function.InvokeAfter( () => 2 + 2, #duration(0,0,0,1)),
// Get the JSON response of the current page
Source = Json.Document(Web.Contents(input_url)),
// Return the "data"
res = [Data=Source]
in
res,
// Iterate through the _iterations, gathering each page's data
GeneratedList = List.Generate(
()=>[i=0, res = FnGetOnePage(_start_url)],
each [i]<_iterations and [res][Data]<>null,
each [i=[i]+1, res = FnGetOnePage(_base_url & Number.ToText(i) & _end_url)],
each [res][Data]),
// Unpack the data
#"Converted to Table" = Table.FromRecords(GeneratedList),
#"Expanded data" = Table.ExpandListColumn(#"Converted to Table", "data"),
#"Expanded data1" = Table.ExpandRecordColumn(#"Expanded data", "data", {"_id", "name", "trips", "airline", "__v"}, {"data._id", "data.name", "data.trips", "data.airline", "data.__v"}),
#"Expanded data.airline" = Table.ExpandListColumn(#"Expanded data1", "data.airline"),
#"Expanded data.airline1" = Table.ExpandRecordColumn(#"Expanded data.airline", "data.airline", {"id", "name", "country", "logo", "slogan", "head_quaters", "website", "established"}, {"data.airline.id", "data.airline.name", "data.airline.country", "data.airline.logo", "data.airline.slogan", "data.airline.head_quaters", "data.airline.website", "data.airline.established"}),
#"Changed Type" = Table.TransformColumnTypes(#"Expanded data.airline1",{{"totalPassengers", Int64.Type}, {"totalPages", Int64.Type}, {"data._id", type text}, {"data.name", type text}, {"data.trips", Int64.Type}, {"data.airline.id", Int64.Type}, {"data.airline.name", type text}, {"data.airline.country", type text}, {"data.airline.logo", type text}, {"data.airline.slogan", type text}, {"data.airline.head_quaters", type text}, {"data.airline.website", type text}, {"data.airline.established", Int64.Type}, {"data.__v", Int64.Type}})
in
#"Changed Type"