With PROC SQL it's possible to connect to a Database (Db2 in my case) and execute Inserts, Deletes, etc.
If one such process causes no modifications to the target Table you will see a note like this in the Log:
NOTE: No data found/modified.
So it's clear that SAS checks for this after every such step.
Can I access this Information during the execution of a Program other than by parsing the Log on the fly?
Perhaps some sort of automatic Macrovariable/Dataset that stores the Status of the last step?
EDIT: I'm using Pass Thru SQL with EXECUTE-Statements.
Check the automatic macro variables PROC SQL creates after remote execution.
SQLXMSG contains descriptive information and the DBMS-specific return
code for the error that is returned by the pass-through facility.
Note: Because the value of the SQLXMSG macro variable can contain
special characters (such as &, %, /, *, and ;), use the %SUPERQ macro
function when printing the following value: %put %superq(sqlxmsg); For
information about the %SUPERQ function, see SAS Macro Language:
Reference.
SQLXRC contains the DBMS-specific return code that is returned by the
pass-through facility.
Related
I was using the following code to analyze data:
set taq.cq_&yyyymmdd:;
by symbol date time NOTSORTED ex;
There are are thousands of datasets I am running the code on in the unit of days. When &yyyymmdd only specifies one dataset (for one day. for example, 20130102), it works. However, when I try to run it for multiple datasets (for example, 201301:), SAS returns the following errors:
BY NOTSORTED/NOBYSORTED cannot be used with SET statement when
more than one data set is specified.
If I cannot use NOTSORTED here, what is an equivalent statement that I could use?
My understanding of the keyword NOTSORTED is that you use it when the data is not sorted yet. Therefore, do I need to sort it first? How to do it?
I am also confused by the number of variables that NOTSORTED is referencing. Does it only have an effect on "time", or it has effect on "symbol, data, time"?
Many thanks!
UPDATE#2:
The rest of the process immediately following the set statement is: (pseudo code as i don't have the permission to post the original code)
Data _quotes;
SET STATEMENT HERE
Change the name of a variable in the dataset (Variable name is EXN).
last.EXN in a if statement. If the condition is satisfied, label EXN.
Drop some variables.
Run;
DATA NEWDATASET (sortedby= SYMBOL DATE TIME index=(SYMBOL)
label="WRDS-TAQ NBBO Data");
SET _quotes;
by symbol date time;
....
Run;
NOTSORTED means that SAS can assume the sort order in the data is correct, so it may not have explicitly gone through a PROC SORT but it is in logical order as listed in the BY statement.
All variables in the BY statement are included in the NOTSORTED option. Given that I suspect you fully don't understand BY group processing.
It's usually a bit dangerous to use, especially if you don't understand BY group processing. If your data is in the same group but not adjacent it won't work properly and will not produce an error. The correct workaround depends on your processes to be honest.
I would suggest reviewing the documentation regarding BY group processing. It's quite in depth and has lots of samples to illustrate the different type of calculations.
http://support.sas.com/documentation/cdl/en/lrcon/69852/HTML/default/viewer.htm#n138da4gme3zb7n1nifpfhqv7clq.htm
NOTSORTED is often used in example posts to either avoid a sort or when using a custom sort that's difficult to implement in other ways. Explicitly sorting will remove this issue but you may also be misunderstanding how SAS processes data when you have a SET statement with a BY statement. I believe this is called interleaving.
http://support.sas.com/documentation/cdl/en/lrcon/69852/HTML/default/viewer.htm#n1tgk0uanvisvon1r26lc036k0w7.htm
I suspect that the NOTSORTED keyword is being using to find groups for observations with the same value for the EX variable within the same symbol,date,time. If you only need to find the FIRST then you can use the LAG() function to calculate the FIRST.EX flag.
data want;
set taq.cq_&yyyymmdd:;
by symbol date time;
first_ex = first.time or ex ne lag(ex);
Otherwise then perhaps you want to convert the process to data step views and then set the views together.
data work.view_cq_20130102 / view=work.view_cq_20130102;
set taq.cq_20130102;
by symbol date time ex NOTSORTED;
...
run;
...
data want ;
set work.view_cq_201301: ;
by symbol date time;
...
This code executes fine when Run as a SAS program:
%MyMacro(foo_val, bar_val, bat_val);
I have created a table using:
DATA analyses;
input title : $32. weight : $32. response : $32.;
datalines;
foo1 bar1 bat1
foo2 bar2 bat2
;
I want to execute MyMacro once for each row of the analyses table.
The following code appears to only pass the string values title, weight and response (rather than the data values foo1 etc.) to my macro (tested with calls to the %put command) :
DATA _NULL_ ;
set analyses;
%MyMacro(title, weight, response);
RUN;
How can I invoke the macro once per record of the analyses table whilst passing data values as arguments to the macro? The intention is to actually run this for a very large number of analyses so the solution must scale appropriately to many more records in the analyses table.
This in part depends on what your macro is doing. If we assume that your macro is doing something that is intended to be run outside of a data step (ie, it's not just assigning a data step variable), then you have several options.
CALL EXECUTE
PROC SQL: SELECT INTO macro variable
Write macro calls into an %INCLUDE file
DOSUBL
CALL EXECUTE has already been explained, and is a good option for some cases. It has some downsides, however, particularly with macro timing, that requires some extra care to protect in some cases - particularly when you are creating macro variables inside your macro. Quentin in his comments shows a way to get around this (adding %NRSTR to the call), but I find that I prefer to only use CALL EXECUTE when there's an advantage to doing so over the other methods - particularly, if I want to use SAS data step techniques (such as FIRST or LAST, for example, or some form of looping) in creating my macro calls, or when I have to do things in a data step anyway and can avoid the overhead of reading the file another time. If I'm just writing a data step like yours above - data something, set something, call execute, run - I wouldn't use it.
PROC SQL SELECT INTO is typically what I use for list processing (which is largely what this is). I like SQL's simplicity a bit better when doing things that aren't too complicated; for example, you can get just one version of each macro call easily with DISTINCT without having to explicitly write a proc sort nodupkey or use first/last processing. It also has the advantage for debugging that you can write all of your macro calls to your results window (if you don't add noprint), which is a bit easier to read than the log for me if I'm trying to see why my calls didn't get generated properly (and doesn't take any extra PUT statements).
proc sql;
select catx(',','%macro(',arg1,arg2,arg3)||')'
into :mvarlist separated by ' '
from dataset;
quit;
&mvarlist.
That runs them quite simply, and has no timing issues (As you're just writing a bunch of macro calls out).
The main downside to this method is that you have a maximum of 64k characters in a macro variable, so if you're writing a huge number of these you'll run into that. In that case use CALL EXECUTE or %INCLUDE files.
%INCLUDE files are largely useful either as replacement for SELECT INTO when the call is over the character limit, or if you find it useful to have a text file to look at with your calls (if you're running this in batch mode for example, this could be easier to get to and/or parse than log or listing output). You just write your calls out to a file, and then %INCLUDE that file.
filename myfile temp; *or a real file if you want to look at it.;
data _null_;
set dataset;
file myfile;
length str $200;
str=catx(',','%macro(',arg1,arg2,arg3)||')';
put str;
run;
%include myfile;
I don't really use this much anymore, but it's a common technique used particularly by older SAS programmers so good to know.
DOSUBL is a relatively new method, and to some extent can be used to replace CALL EXECUTE as its default behavior is typically closer to what you expect intuitively than CALL EXECUTE's. The doc page has really the best example for how this works differently; basically, it fixes the timing issue by letting each separate call look import and export the macro variables from/to the calling environment, meaning that each iteration of DOSUBL is run at a distinct time versus CALL EXECUTE where everything is run in one bunch and the macro environment is 'fixed' (ie, any reference to a macro variable is fixed at run time, unless you escape it messily with %NRSTR).
One more thing worth mentioning is RUN_MACRO, a part of the FCMP language. That allows you to completely run a macro and import its contents back to the data step, which is an interesting option in some cases (for example, you could wrap a call around a PROC SQL that selected a count of something, and then import that to the dataset as a variable, all in one datastep). It's applicable if you're doing this for the purpose of calling a macro to assign a data step variable, not to run a process that does things that don't need to be imported into the data step, but it's something worth considering if you do want that data back all in the dataset that called the process.
You could use CALL EXECUTE:
data _null_;
set analyses;
call execute('%nrstr(%MyMacro('||title||','||weight||','||response||'))');
run;
You can put the variables values into macrovariables and then call your %MyMacro many times (the number of obs in your dataset) with the macrovariables as argument:
Data :
DATA analyses;
input title : $32. weight : $32. response : $32.;
datalines;
foo1 bar1 bat1
foo2 bar2 bat2
;
run;
Code to run macro :
data _NULL_;
set analyses end=fine;
call symput("ARGUMENT"||compress(_N_),catx(",",title,weight,response));
if fine then call symput("NLOOPS",compress(_N_));
run;
%*PUT &ARGUMENT1;
%*PUT &ARGUMENT2;
%MACRO MAIN;
%DO L=1 %TO &NLOOPS;
%MyMacro(&&ARGUMENT&L);
%END;
%MEND;
%MAIN;
I have written two different SAS EG programs under the same project. The first one is to estimate a parameter, say A, which will be used in the second program. At the moment, once first program finish running, I manually set the parameter
%let A = 0.1;
in the second program. I just wonder is there a way to declare A as a global variable once it is calculated in the first program and then my second program can use it without the manual input?
Thanks.
I'll post a data step, and use CALL SYMPUTX so you can assign the variable a global scope as well.
data _null_; *does not generate a data set;
set have;
call symputx('A', variable_name, 'G');
run;
Documentation for call symputx is here:
http://support.sas.com/documentation/cdl/en/lrdict/64316/HTML/default/viewer.htm#a002295697.htm
Global macro variables persist for an entire SAS session (unless you delete them). This means that in one EG session, if you run your %LET statement in the first program, it will be created in the session's global macro symbol table. It will then be available to be used in any other program that runs during that same SAS session.
A SAS session persists in EG until you close EG, or manually disconnect from the SAS server.
As a separate option beyond the scope of EG, if you were to run 2 separate programs in non-interactive mode a good way of sharing variables (or parameters of any kind) is to store them in a file and read them in at the start of the next program.
Using call symputx to assign a global variable will then allow these to persist throughout your new program.
Using SAS, I often want to perform an action on each row of a dataset. To do so, I use a command I found on a tutorial : call execute(). As I'm not very familiar with SAS environment, I tend to use macro-functions to do anything I don't know how to and execute them with call execute().
However I have difficulties understanding how macro-language works exactly in SAS. I know that all macro references are resolved first, which provides a base-SAS code which is then executed (or am I already wrong ?). But I don't understand how it applies with call execute().
Let's consider the following code :
%macro prog1; /* %prog1 defines a macrovariable mv1 */
%global mv1;
data _null_;
call symputx("mv1","plop");
run;
%mend;
%macro prog2(var); /* prog2 puts it on the log */
%put PUT &var;
%mend;
%macro prog_glob; /* prog_glob executes prog 1 then prog2 */
%prog1;
%prog2(&mv1);
%mend;
I know there is no need for three macros here but this is a minimal version of my real code, which has this structure.
Now if I execute prog_glob :
%prog_glob;
I get PUT plop on the log, as expected.
But if I use it with call execute() (even if there is no need for loop here) :
data _null_;
mac=%NRSTR("%prog_glob");
call execute(mac);
run;
I get only PUT.
There is no error suggesting that the macrovariable is not defined so the %global statement worked. But somehow, prog2 was executed before prog1 base part (at least I think so) and mv1 is not defined yet.
My questions are :
Is my interpretation correct ?
Why does the result change when I use call execute ?
Depending on the precedent question answer, how should I fix it or is there a more convenient way to loop trough a column values ?
EDIT : My original code intends to rename the variables of several tables, which I have listed in a dataset. For each listed table, I want the following algorithm executed :
prog1 : store a list with all variables in a macrovariable (this is where I define mv equivalent)
prog2 : add a common suffix to these variables names
There is probably a more clever way to do this. Again, I'm not so familiar with SAS and I tend to over-use macros. If you want to show me a better way to do this, I'd be happy to chat but I don't expect you guys to rewrite all my code so an alternative to call execute would be enough for me to be grateful ! :)
Let's have a look at the documentation
http://support.sas.com/documentation/cdl/en/mcrolref/61885/HTML/default/viewer.htm#a000543697.htm
If an EXECUTE routine argument is a macro invocation or resolves to one, the macro executes immediately. However, any SAS statements produced by the EXECUTE routine do not execute until after the step boundary has been passed.
Note: Because macro references execute immediately and SAS statements do not execute until after a step boundary, you cannot use CALL EXECUTE to invoke a macro that contains references for macro variables that are created by CALL SYMPUT in that macro. See Interfaces with the Macro Facility, for an example.
This means, if you call it via call execute:
macro statements are executed immediately - those are:
1.1. first in %prog1: %global mv1; - so mv1 is defined but empty, no Apparent... warning
1.2. SAS statements from %prog1 are still deferred
Now %prog2 - here's only macro statement %PUT which puts (still empty) &mv1 variable. That what you see in the log
Now all the what gets executed immediately has been done. The data step which contains call execute ends.
SAS statements deferred from call execute are now being executed:
4.1. the dataset from prog1 sets the value for mv1.
And that's all :-)
EDIT: regarding your edit: try looking at this http://support.sas.com/kb/48/674.html
data _null_;
mac='%nrstr(%prog_glob)';
call execute(mac);
run;
or, more plainly, as you would see in the documentation...
data _null_;
call execute('%nrstr(%prog_glob)');
run;
or
%let prog=%nrstr(%prog_glob);
data _null_;
mac="&prog.";
call execute(mac);
run;
or, and I wouldn't really recommend this one, but you could also manually concatenate the macro quotes
data _null_;
mac=cats('01'x,'%prog_glob','02'x);
call execute(mac);
run;
The way you are running it, the macro statements get execute at run time and the data step is execute after the calling data step completes. You're not properly using %NRSTR for this context, as described by the documentation. You need to pass the macro, along with the quoting as text to the call routine.
This code executes fine when Run as a SAS program:
%MyMacro(foo_val, bar_val, bat_val);
I have created a table using:
DATA analyses;
input title : $32. weight : $32. response : $32.;
datalines;
foo1 bar1 bat1
foo2 bar2 bat2
;
I want to execute MyMacro once for each row of the analyses table.
The following code appears to only pass the string values title, weight and response (rather than the data values foo1 etc.) to my macro (tested with calls to the %put command) :
DATA _NULL_ ;
set analyses;
%MyMacro(title, weight, response);
RUN;
How can I invoke the macro once per record of the analyses table whilst passing data values as arguments to the macro? The intention is to actually run this for a very large number of analyses so the solution must scale appropriately to many more records in the analyses table.
This in part depends on what your macro is doing. If we assume that your macro is doing something that is intended to be run outside of a data step (ie, it's not just assigning a data step variable), then you have several options.
CALL EXECUTE
PROC SQL: SELECT INTO macro variable
Write macro calls into an %INCLUDE file
DOSUBL
CALL EXECUTE has already been explained, and is a good option for some cases. It has some downsides, however, particularly with macro timing, that requires some extra care to protect in some cases - particularly when you are creating macro variables inside your macro. Quentin in his comments shows a way to get around this (adding %NRSTR to the call), but I find that I prefer to only use CALL EXECUTE when there's an advantage to doing so over the other methods - particularly, if I want to use SAS data step techniques (such as FIRST or LAST, for example, or some form of looping) in creating my macro calls, or when I have to do things in a data step anyway and can avoid the overhead of reading the file another time. If I'm just writing a data step like yours above - data something, set something, call execute, run - I wouldn't use it.
PROC SQL SELECT INTO is typically what I use for list processing (which is largely what this is). I like SQL's simplicity a bit better when doing things that aren't too complicated; for example, you can get just one version of each macro call easily with DISTINCT without having to explicitly write a proc sort nodupkey or use first/last processing. It also has the advantage for debugging that you can write all of your macro calls to your results window (if you don't add noprint), which is a bit easier to read than the log for me if I'm trying to see why my calls didn't get generated properly (and doesn't take any extra PUT statements).
proc sql;
select catx(',','%macro(',arg1,arg2,arg3)||')'
into :mvarlist separated by ' '
from dataset;
quit;
&mvarlist.
That runs them quite simply, and has no timing issues (As you're just writing a bunch of macro calls out).
The main downside to this method is that you have a maximum of 64k characters in a macro variable, so if you're writing a huge number of these you'll run into that. In that case use CALL EXECUTE or %INCLUDE files.
%INCLUDE files are largely useful either as replacement for SELECT INTO when the call is over the character limit, or if you find it useful to have a text file to look at with your calls (if you're running this in batch mode for example, this could be easier to get to and/or parse than log or listing output). You just write your calls out to a file, and then %INCLUDE that file.
filename myfile temp; *or a real file if you want to look at it.;
data _null_;
set dataset;
file myfile;
length str $200;
str=catx(',','%macro(',arg1,arg2,arg3)||')';
put str;
run;
%include myfile;
I don't really use this much anymore, but it's a common technique used particularly by older SAS programmers so good to know.
DOSUBL is a relatively new method, and to some extent can be used to replace CALL EXECUTE as its default behavior is typically closer to what you expect intuitively than CALL EXECUTE's. The doc page has really the best example for how this works differently; basically, it fixes the timing issue by letting each separate call look import and export the macro variables from/to the calling environment, meaning that each iteration of DOSUBL is run at a distinct time versus CALL EXECUTE where everything is run in one bunch and the macro environment is 'fixed' (ie, any reference to a macro variable is fixed at run time, unless you escape it messily with %NRSTR).
One more thing worth mentioning is RUN_MACRO, a part of the FCMP language. That allows you to completely run a macro and import its contents back to the data step, which is an interesting option in some cases (for example, you could wrap a call around a PROC SQL that selected a count of something, and then import that to the dataset as a variable, all in one datastep). It's applicable if you're doing this for the purpose of calling a macro to assign a data step variable, not to run a process that does things that don't need to be imported into the data step, but it's something worth considering if you do want that data back all in the dataset that called the process.
You could use CALL EXECUTE:
data _null_;
set analyses;
call execute('%nrstr(%MyMacro('||title||','||weight||','||response||'))');
run;
You can put the variables values into macrovariables and then call your %MyMacro many times (the number of obs in your dataset) with the macrovariables as argument:
Data :
DATA analyses;
input title : $32. weight : $32. response : $32.;
datalines;
foo1 bar1 bat1
foo2 bar2 bat2
;
run;
Code to run macro :
data _NULL_;
set analyses end=fine;
call symput("ARGUMENT"||compress(_N_),catx(",",title,weight,response));
if fine then call symput("NLOOPS",compress(_N_));
run;
%*PUT &ARGUMENT1;
%*PUT &ARGUMENT2;
%MACRO MAIN;
%DO L=1 %TO &NLOOPS;
%MyMacro(&&ARGUMENT&L);
%END;
%MEND;
%MAIN;