I'm trying to automate a scheduled event in SAS so that when the SAS program opens it first runs a pass through query that creates a local table. I want this query to keep running in a loop until the observation count of that table is >0. The idea is that I need to wait for a confirmation from the data source that the table is ready before I can run additional code to the server which I would do after the macro. I would also like it to sleep for 10 minutes each time the table shows no observations.
My ODBC connection...
proc sql;
connect to odbc(datasrc="SSDM");
EXECUTE ( Create Volatile Table DataReady as
(
SQL...
) WITH DATA ON COMMIT PRESERVE ROWS;) by ODBC;
CREATE TABLE DataReady AS SELECT * FROM CONNECTION TO ODBC ( SELECT * FROM DataReady );
DISCONNECT FROM odbc;
quit;
and also include a sleep function...
data _null_;
rc=SLEEP(600);
run;
Break it down into the key component parts
set initial number of observations counter to 0
Execute your queries
Check number of obs
If zero, sleep then loop back to #2
4 requires macro code (hence the macro wrapper LOOPER below), 2 & 3 are PROC / DATASTEP, and 1 could be either.
%MACRO LOOPER ;
%LET OBS = 0 ; /* #1 */
%DO %WHILE (&OBS = 0) ; /* run everything inside this %DO %WHILE loop whilst 0 obs - #4 */
/* put your existing SQL code here - #2 */
proc sql ; select count(1) into :OBS from DataReady ; quit ; /* #3 */
%IF &OBS = 0 %THEN %LET S = %SYSFUNC(sleep(10,60)) ; /* sleep for 10 minutes if 0 obs */
%END ;
%MEND ;
%LOOPER ;
Related
I have a yearly table which on a monthly basis will append data from another table. However, I need to check the max date on the monthly table before appending it. If the max date on monthly is same as YTD, then do not append else append. How can I achieve this in SAS.
I tried using append but don't know how to check the dates before appending.
You can use a macro.
First, insert the last year/month id on a variable and preper a macro to execute the append operation:
proc sql;
select max(yourDtcolumn) into : var
from yourTable;
quit;
%macro append;
proc sql;
insert into yourtable
select * from sourcetable;
quit;
%mend;
then, verify if the variable are the same:
%macro verify;
%if &var > &curMonth %then %do;
%append;
%end;
%mend;
finally, you call the macro to execute:
%verify;
I'd skip the check, and simply stack the existing data (excluding the current month, if it exists) and the new data.
/* Get period from new data */
proc sql ;
select min(period) into :LATEST
from new ;
quit ;
/* Append new to master, and save back master */
data perm.master ;
set perm.master (where=(period < &LATEST))
new ;
run ;
I need to query a large table in a server (REMOTE_TBL) using the SAS pass-through facility. In order to make the query shorter, I want to send a list of IDs extracted from a local table (LOCAL_TBL).
My first step is to get the IDs into a variable called id_list using an INTO statement:
select distinct ID into: id_list separated by ',' from WORK.LOCAL_TBL
Then I pass these IDs to the pass-through query:
PROC SQL;
CONNECT TO sybaseiq AS dbcon
(host="name.cl" server=alias db=iws user=sas_user password=XXXXXX);
create table WANT as
select * from connection to dbcon(
select *
from dbo.REMOTE_TBL
where ID in (&id_list)
);
QUIT;
The code runs fine except that I get the following message:
The length of the value of the macro variable exceeds the maximum length
Is there an easier way to send the selected ID's to the pass-through query?
Is there a way to store the selected ID's in two or more variables?
Store the values into multiple macro variables and then store the names of the macro variables into another macro variable.
So this code will make a series of macro variables named M1, M2, .... and then set ID_LIST to &M1,&M2....
data _null_;
length list $20200 mlist $20000;
do until(eof or length(list)>20000);
set LOCAL_TBL end=eof;
list=catx(',',list,id);
end;
call symputx(cats('m',_n_),list);
mlist=catx(',',mlist,cats('&m',_n_));
if eof then call symputx('id_list',mlist);
run;
Then when you expand ID_LIST the macro processor will expand all of the individual Mx macro variables. This little data step will create a couple of example macro variables to demonstrate the idea.
data _null_;
call symputx('m1','a,b,c');
call symputx('m2','d,e,f');
call symputx('id_list','&m1,&m2');
run;
Results:
70 %put ID_LIST=%superq(id_list);
ID_LIST=&m1,&m2
71 %put ID_LIST=&id_list;
ID_LIST=a,b,c,d,e,f
You are passing many data values that appear in your IN (…) clause. The number of values allowed varies by data base; some may limit to 250 values per clause and the length of a statement might have limitations. If the macro variable creates a list of values 20,000 characters long, the remote side might not like that.
When dealing with a lookup of perhaps > 100 values, take some time first to communicate your need to the DB admin for creating temporary tables. When you have such rights, your queries will be more efficient remote side.
… upload id values to #myidlist …
create table WANT as
select * from connection to dbcon(
select *
from dbo.REMOTE_TBL
where ID in (select id from #myidlist)
);
QUIT;
If you can't get the proper permissions, you would have to chop up the id list into pieces and have a macro create a series of ORed IN searches.
1=0
OR ID IN ( … list-values-1 … )
…
OR ID IN ( … list-values-N … )
For example:
data have;
do id = 1 to 44;
output;
end;
run;
%let IDS_PER_MACVAR = 10; * <---------- make as large as you want until error happens again;
* populated the macro vars holding the chopped up ID list;
data _null_;
length macvar $20; retain macvar;
length macval $32000; retain macval;
set have end=end;
if mod(_n_-1, &IDS_PER_MACVAR) = 0 then do;
if not missing(macval) then call symput(macvar, trim(macval));
call symputx ('VARCOUNT', group);
group + 1;
macvar = cats('idlist',group);
macval = '';
end;
macval = catx(',',macval,id);
if end then do;
if not missing(macval) then call symput(macvar, trim(macval));
call symputx ('MVARCOUNT', group);
end;
run;
* macro that assembles the chopped up bits as a series of ORd INs;
%macro id_in_ors (N=,NAME=);
%local i;
1 = 0
%do i = 1 %to &N;
OR ID IN (&&&NAME.&i)
%end;
%mend;
* use %put to get a sneak peek at what will be passed through;
%put %id_in_ors(N=&MVARCOUNT,NAME=IDLIST);
* actual sql with pass through;
...
create table WANT as
select * from connection to dbcon(
select *
from dbo.REMOTE_TBL
where ( %ID_IN_ORS(N=&MVARCOUNT,NAME=IDLIST) ) %* <--- idlist piecewise ors ;
);
...
I suggest that you first save all the distinct values into a table, and then (again using proc sql + into) load the values into a few stand-alone macrovariables, reading the table several times in a few sets; indeed they have to be mutually exclusive yet jointly exhaustive.
Do you have access to and CREATE privileges in the DB where your dbo.REMOTE_TBL resides? If so you might also think about copying your WORK.LOCAL_TBL into a temporary table in the DB and run an inner join right there.
Another option - write out the query to a temporary file and then %include it. No macro logic needed!
proc sort
data = WORK.LOCAL_TBL(keep = ID)
out = distinct_ids
nodupkey;
run;
data _null_;
set distinct_ids end = eof;
file "%sysfunc(pathname(work))/temp.sas";
if _n_ = 1 then put "PROC SQL;
CONNECT TO sybaseiq AS dbcon
(host=""name.cl"" server=alias db=iws user=sas_user password=XXXXXX);
create table WANT as
select * from connection to dbcon(
select *
from dbo.REMOTE_TBL
where ID in (" #;
put ID #;
if not(eof) then put "," #;
if eof then put ");QUIT;" #;
put;
run;
/*Use nosource2 to avoid cluttering the log*/
%include "%sysfunc(pathname(work))/temp.sas" /nosource2;
How can I check if the values are obtained by this condition?
data ad01(keep=str);
length str $1024;
set Address(where=(Type_="1"));
///if resultat not null do something
run;
As #Quentin suggests this becomes a macro question. one way to do this is capturing count into macro variable.
/* an example where count is zero*/
proc sql noprint;
select count(*) into :cnt from sashelp.class
where name = 'unknown';
%put &cnt;
/* Here datastep gets aborted if count = 0 */
data class;
if &cnt = 0 then abort;
set sashelp.class;
run;
/* above query works when there is count gt 0 see the example below*/
proc sql noprint;
select count(*) into :cnt from sashelp.class
where trim(name) = 'Alfred';
%put &cnt;
data class;
if &cnt = 0 then abort;
set sashelp.class (where =(trim(name) = 'Alfred'));
run;
I’m assuming you mean if there are any records that meet your WHERE condition you want to process them somehow.
You can do that with just:
data ad01(keep=str);
length str $1024;
set Address(where=(Type_="1"));
*do something;
run;
If the dataset has no records with TYPE_=“1” the step will end when the SET statement executes and hits the (logical) end of file mark. If there are records that meet the condition, then any statements after the SET statement will be executed.
If by “do something” you mean execute additional steps (not just statements) this could become a macro language question. Where you would want a macro with an %IF statement, something like:
%if %anyobs(Address(where=(Type_=“1”))) %then %do;
*data steps and PROC steps and whatever here;
%end;
For examples of function-style %anyobs macros see e.g. http://www2.sas.com/proceedings/sugi26/p095-26.pdf or https://www.devenezia.com/downloads/sas/macros/index.php?m=anyobs or http://www.datasavantconsulting.com/roland/Spectre/utilmacros/nlobs.sas
If you want to do something when the SET statement return 0 observations you have to do it BEFORE the SET. With 0 obs the SET returns and data step ends. Consider this code where an observation is output when the SET returns 0 obs.
data class;
if 0 then set sashelp.class;
if _n_ eq 1 and eof then do;
name = 'NODATA';
output;
end;
set sashelp.class(where=(sex='X')) end=eof;
run;
proc print; run;
i have a data set with multiple attributes and each attribute has 10-15 rows each in the master table. i wish to use a do loop on the data set which would allow me to extract outputs for each attribute seperately. my concern is how to automate the selection of attribute in the do loop once the previous attribute's output is extracted??
thanks in advance.
I'm not completely sure what you're asking to do, but I can hopefully show the basic ideas of a do loop.
%macro YOUR_MACRO();
%let YOUR_VARIABLE = 1 2 3 ...; /*This could be whatever you want to split up from your master table*/
%let NUM_VAR = 3; /*Change this to the number of YOUR_VARIABLEs listed*/
%do i = 1 %to &NUM_VAR. %by 1;
%let LOOP_VAR = %scan(&YOUR_VARIABLE., %i.);
/*This do i = 1 starts your loop at 1 and goes up by 1 until your NUM_VAR is reached*/
proc sql;
create table TABLE_&LOOP_VAR. as /*Creates a specific table for each variable*/
select *
from MASTER_TABLE
where COLUMN_NAME = &LOOP_VAR. /*Splits up your table by a certain attribute equaling the loop variable*/
;
quit;
%end;
%mend;
%YOUR_MACRO(); /*Runs your loop*/
This is the basic structure and should give a little help. You can also just scan your master table for each variable name then separate it by that without having to type each one out.
Is there a way to check how many observations are in a SAS data set at runtime OR to detect when you've reached the last observation in a DATA step?
I can't seem to find anything on the web for this seemingly simple problem. Thanks!
The nobs= option to a set statement can give you the number of observations. When the data step is compiled, the header portion of the input datasets are scanned, so you don't even have to execute the set statement in order to get the number of observations. For instance, the following reports 2 as expected:
/* a test data set with two observations and no vars */
data two;
output;
output;
run;
data _null_;
if 0 then set two nobs=nobs;
put nobs=;
run;
/* on log
nobs=2
*/
The end= option sets a flag when the last observation (for the set statement) is read in.
A SAS data set, however, can be a SAS data file or a SAS view. In the case of the latter, the number of observations may not be known either at compile time or at execution time.
data subclass/view=subclass;
set sashelp.class;
where sex = symget("sex");
run;
%let sex=F;
data girls;
set subclass end=end nobs=nobs;
put name= nobs= end=;
run;
/* on log
Name=Alice nobs=9.0071993E15 end=0
Name=Barbara nobs=9.0071993E15 end=0
Name=Carol nobs=9.0071993E15 end=0
Name=Jane nobs=9.0071993E15 end=0
Name=Janet nobs=9.0071993E15 end=0
Name=Joyce nobs=9.0071993E15 end=0
Name=Judy nobs=9.0071993E15 end=0
Name=Louise nobs=9.0071993E15 end=0
Name=Mary nobs=9.0071993E15 end=1
*/
You can also use %sysfunc(attrn( dataset, nlobs)) though it is limited to SAS data sets (i.e. not data views). Credit for the macro to this SUGI paper, which also give great information regarding good macro design.
You can get all sorts of other character and numeric information on a SAS data set.
See the documentation on attrn and attrc.
%macro numobs (data=&syslast ) ;
/* --------------------------------------------
Return number of obs as a function
--------------------------------------------
*/
%local dsid nobs rc;
%let data = &data ; /* force evaluation of &SYSLAST */
%let dsid=%sysfunc(open(&data));
%if &dsid > 0 %then
%do ;
%let nobs=%sysfunc(attrn(&dsid,nlobs));
%let rc=%sysfunc(close(&dsid));
%end ;
%else
%let nobs = -1 ;
&nobs
%mend numobs;
Find the number of observations in a SAS data set:
proc sql noprint;
select count(*) into: nobs
from sashelp.class
;
quit;
data _null_;
put "&nobs";
run;
The SQL portion counts the number of observaions, and stores the number in a macro variable called "nobs".
The data step puts the number for display, but you can use the macro variable like any other.
Performing a certain action when the last observation is processed:
data _null_;
set sashelp.class end=eof;
if eof then do;
put name= _n_=;
end;
run;
The "end" option to the "set" statement defines a variable (here "eof" for end-of-file) that is set to 1 when the last observation is processed. You can then test the value of the variable, and perform actions when its value is 1. For more info, see the documentation for the "set" statement.
data hold;
set input_data end=last;
.
.
.
if last then do;
.
.
.
end;
run;