Paging through rest api results? - sas

Question

I am using the following code to pull a json file from a url :
options NOQUOTELENMAX;
filename usage "/folders/myfolders/sasuser.v94/usage.json";
%let AccessKey = reallylongstring;
proc http
url="https://a.url"
method="GET" out=usage;
headers
"Authorization"="Bearer &AccessKey.";
run;
libname usage json "/folders/myfolders/sasuser.v94/usage.json";
data usage;
set usage.data;
run;
proc print data=usage noobs;
run;
however now the results are returning more that 1000 results and i need to check for nextLink property somehow?
in .net I could use something like this:
$usagerest = Invoke-Restmethod -url $usageurl -header $authheaders -method get
while ($null -ne $usageRest.nextLink) {
$usageRest = Invoke-Restmethod -uri $usagerest.nextlink -headers $authheaders -method get
}
is something like this possible with proc http in sas?
heres a tree view of the actual json if of help?
so far i have tried a quick dirty version:
libname usage1 JSON fileref=resp1;
data x;
set usage1.root;
call symputx('nextLink',nextLink);
run;
proc http
url="%superq(nextLink)"
method="GET" out=resp2;
headers
"Authorization"="Bearer &AccessKey.";
run;
libname usage2 JSON fileref=resp2;
data y;
set usage2.root;
call symputx('nextLink',nextLink);
run;
proc http
url="%superq(nextLink)"
method="GET" out=resp3;
headers
"Authorization"="Bearer &AccessKey.";
run;
libname usage3 JSON fileref=resp3;
data z;
set usage3.root;
call symputx('nextLink',nextLink);
run;
Libraries overview:
usage1.data sample:
work.x sample:
Thanks

Answer 1

For the context of the resolved macro variable needing to be double quoted, such as url= option value, place the %superq inside double quotes -- "%superq(<macro-var-name>)"
Try
…
url = "%superq(nextLink)"
If the link continues to be troublesome you can try
url = "%qsysfunc(urlencode(%superq(nextLink)))"
Invoking a macro once per row in a control data set
There are numerous ways, such as
Stack the 1,000 invocations via data _null_; and call execute(…
Proc SQL select nextLink into :link1- to create 1,000 macro variables
Open the data set with %let ds=%sysfunc(open(… and loop over the 1,000 rows — %do %while (%sysfunc(fetch(&ds) = 0))
Sample code for second bullet item:
This sample passes a macro variable from the 'looping' macro to the 'worker' macro. When macro variable name is passed, the macro var does not have to be resolved and quoted for passage into the worker. Instead the work gets the macro var (aka symbol name) and has the superq resolve it in quoted manner for use in source code generation. Essentially, passing the macro var is akin to a traditional language's pass-by-reference concept.
data have;
do linknum = 1 to 25;
link = cats("place=", byte(64+linknum),'&extra="zoom=',linknum,'"&key=MYAPIKEY');
output;
end;
run;
%macro processLink(link_mvar=);
%put url="%sysfunc(urlencode(%superq(&link_mvar)))";
%mend;
%macro processLinks (data=);
proc sql;
reset noprint;
select link into :link1- from &data;
quit;
%local i;
%do i = 1 %to &sqlobs;
%* pass name of macro variable to macro;
%processLink (link_mvar=link&i);
%end;
%mend;
%processLinks(data=have)

Answer 2

The question seems clearer. The process can be macro-ized to repeatedly call Proc HTTP inside a %do loop. Each page of data can be appended to a data set that will grow with each page retrieved.
Untested
%macro getThatPagedJsonData (url=, accesskey=, out=);
%local page guard;
%let page = 1;
%Let guard = 50; * just in case - prevent infinite/excessive looping during development/testing;
filename response temp;
%do %until (%length(%superq(url)) eq 0 or &page > &guard);
* clear libname, releasing any locks on json repsonse file;
libname page;
* prior response will be over written;
proc http url="%superq(url)" method="GET" out=response;
headers "Authorization"="Bearer &AccessKey.";
run;
* magic json engine;
libname page JSON fileref=response;
if &page = 1 %then %do;
* first page starts the output data set;
data &out;
set page.data;
run;
%end;
%else %do;
* append subsequent pages of data;
proc append base=&out data=page.data;
run;
%end;
* track number of pages processed;
%let page = %eval (&page + 1);
* reset url for %until test;
%let url=;
* fetch the nextlink as the url for next iteration of %do %until;
* might need error handling here when last page has no nextlink;
data _null_;
set page.root;
call symput('url', trim(nextlink));
run;
%end;
%mend;
%getThatPagedJsonData (url=...., accesskey=...., out=serviceAggreements);