SAS Data Integration Studio: problems with sending mail (log in attach) - sas

I'm an inexperienced user of SAS Data Integration Studio.
I would like to receive an e-mail information about the job status. Using "Status Handling" and action "Send Email" (from job options) is not satisfying to me. First of all - (propably?) i can't attach log to this e-mail.
In SAS Enterprise Guide I developed and tested code for sending email with log in attach and some useful informations (datetime, list of errors) in mail body:
%let mail = "test#test.com";
%let path_error = /home/ ... .log;
filename msg email
to = (&maile)
subject = "SAS Message Test"
attach = "&path_error.";
data _null_;
dttm = put(datetime(),nldatm.);
infile "&path_error.";
input;
file msg;
if _n_ = 1 then do;
put "Date time: " dttm;
put;
put "Full log in attach.";
put "There are some ERRORs and WARNINGs:";
put;
end;
if substr(_infile_,1,5) = "ERROR" then
put _infile_;
if substr(_infile_,1,7) = "WARNING" then
put _infile_;
run;
This code works fine - I get complete mail with list of errors and warnings. In log I can see:
NOTE: 268 records were read from the infile "[...].log"
But how to implement this code (especially reading by infile statement) in DIS?
I have modified the job options:
Precode - puts the log into an external file; log name contains jobname and datetime:
%let path = /home/[...]/log_&etls_jobName._%sysfunc(datetime(), datetime.).log;
proc printto log="&path.";
run;
Postcode - I used code from Enterprise Guide:
%let address = "test#test.com";
%let message = problems with &etls_jobName;
filename sendMail email
to= (&address)
subject= "ETL Process problem: &etls_jobName."
attach= "&path.";
options nosyntaxcheck;
data _null_;
dttm = put(datetime(),nldatm.);
infile "&path.";
input;
file sendMail;
if _n_ = 1 then do;
put "Date time: " dttm;
put;
put "Full log in attach.";
put "There are some ERRORs and WARNINGs:";
put;
end;
if substr(_infile_,1,5) = "ERROR" then
put _infile_;
if substr(_infile_,1,7) = "WARNING" then
put _infile_;
run;
In effect I get e-mail with log in attach, but empty body. In attached log I can see:
NOTE: 0 records were read from the infile
I've got some questions:
Why 0 records???
When I delete input; statement from Postcode and run job I get e-mail with "Date time / Full log in attach / There are some ERRORs and WARNINGs" in body. Why they are deleted where input works in code?
E-mail is not sending without options nosyntaxcheck; in code. Why?
Thanks for the answers.
Regards,
MichaƂ

The normal way for a data step to stop is when it reads past the end of the input (either raw data being read or datasets being accessed).
So move your _N_=1 block to BEFORE the INPUT statement. You might want to change the wording of the message about the errors and warnings. Or wait to write until after you know there are records to be read.
%let address = "test#test.com";
%let message = problems with &etls_jobName;
filename sendMail email
to=(&address)
subject="ETL Process problem: &etls_jobName."
attach="&path."
;
options nosyntaxcheck;
data _null_;
dttm = put(datetime(),nldatm.);
file sendMail;
if _n_ = 1 then do;
put "Date time: " dttm;
put;
put "Full log in attach.";
put "There are some ERRORs and WARNINGs:";
put;
end;
infile "&path.";
input;
if _n_ = 1 then do;
put "There are some ERRORs and WARNINGs:";
put;
end;
if substr(_infile_,1,5) = "ERROR" then put _infile_;
if substr(_infile_,1,7) = "WARNING" then put _infile_;
run;

Related

ERROR: Open failure for (zip file) during attempt to create a local file

My coworker and I have three zip files, representing three iterations of a monthly download from CMS of the NPPES Data Dissemination (March, April, and May). We use the following code to extract what we need from the newest zip file and create a fairly compact dataset.
PROC IMPORT OUT=NPI_Layout
DATAFILE= "&dir./NPI File Layout.xlsx"
DBMS=XLSX REPLACE;
SHEET="Sheet1";
RUN;
options compress = yes;
data npi_layout;
set npi_layout;
length infmt fmt inpt $60. lbl $200.;
if type = 'NUMBER' then do;
infmt = 'informat '||compress(field)||' '||compress(length)||'.;';
fmt = 'format '||compress(field)||' '||compress(length)||'.;';
inpt = compress(field);
end;
else if type = 'VARCHAR' then do;
infmt = 'informat '||compress(field)||' $'||compress(length)||'.;';
fmt = 'format '||compress(field)||' $'||compress(length)||'.;';;
inpt = compress(field)||' $';
end;
else if type = 'DATE' then do;
infmt = 'informat '||compress(field)||' mmddyy10.;';
fmt = 'format '||compress(field)||' date9.;';
inpt = compress(field);
end;
lbl = 'label '||compress(field)||" = '"||trim(label)||"';";
run;
proc sql noprint;
select infmt
,fmt
,inpt
,lbl
into :infmt1 -
,:fmt1 -
,:inpt1 -
,:lbl1 -
from npi_layout;
quit;
%macro loop;
%let infmt_stmnt = ;
%let fmt_stmnt = ;
%let inpt_stmnt = input;
%let lbl_stmnt = ;
%do i = 1 %to &sqlobs;
%let infmt_stmnt = &infmt_stmnt &&infmt&i;
%let fmt_stmnt = &fmt_stmnt &&fmt&i;
%let inpt_stmnt = &inpt_stmnt &&inpt&i;
%let lbl_stmnt = &lbl_stmnt &&lbl&i;
%end;
data npi.npi;
%let _EFIERR_ = 0; /* set the ERROR detection macro variable */
infile inzip(npidata_pfile_20050523-20180513.csv)
delimiter = ',' MISSOVER DSD lrecl = 32767 firstobs = 2;* obs = 10000;
&infmt_stmnt;
&fmt_stmnt;
&inpt_stmnt;
&lbl_stmnt;
run;
%mend loop;
%loop;
When we run the above code on the file from March, we get a successful output. However, when we try to run it on the April and May downloads, we get the following error:
Error in Log
ERROR: Open failure for
*dir*/NPI/Downloads/NPPES_Data_Dissemination_May_2018.zip
during attempt to create a local file handle.
Google only returns a single result, which indicates that it's an error that pops up when a filename (or path, presumably) is wrong. We've double-checked the path and filename multiple times, and it's all correct (and, obviously, the code works on the March file). Additionally, if I change the code so it's trying to pull a non-existent .csv from the zip file, it gives me a different error about that file not existing within the zip, so it's clearly seeing the zip file in the first place. We're not really sure what's going on; any advice?
(The data is sourced from http://download.cms.gov/nppes/NPI_Files.html, if you want to check the file for yourself.)
Did you try adding quotes around the member name?
infile inzip("npidata_pfile_20050523-20180513.csv") ...
Saw the same ERROR message on Windows 10 64-bit with plenty of RAM and disk space.
The Windows internals used by the ZIP engine is likely dealing with streams which involves file handles. So I would suspect the ZIP engine is trying to allocate too much RAM or too large an intermediary file for dealing with extracting the 6GB "npidata_pfile_20050523-20180513.csv".
Submit the issue to SAS Support -- There might be some session settings that would let the engine work against the file. If not, you'll have to extract the file outside SAS.
How large were the April and May pfile sizes?

SAS %errmail macro

Although I have the error handling:
filename myfile email
to=&e_mail.
subject= "Error: &number."
type="text/plain";
%macro errmail;
%if &syserr ne 0 %then %do;
options obs= max replace nosyntaxcheck;
data _null_;
file myfile;
put;
put 'ERROR';
put "&syserrortext";
put 'check a log'
run;
%abort cancel;
%end;
%mend errmail;
when I have the error in the proc export:
(&number. is the table in the work)
proc export data=&number.
outfile="/usr/local/backup/&number./&number._%SYSFUNC(TODAY(),DATE9.).txt"
replace
dbms=dlm;
delimiter=';';
run;
%errmail;
ERROR: Physical file does not exist, /usr/local/backup/2116/2116_13MAY2016.txt.
NOTE: The SAS System stopped processing this step because of errors.
NOTE: There were 1 observations read from the data set WORK.2116.
NOTE: DATA statement used (Total process time):
real time 0.01 seconds
cpu time 0.01 seconds
SAS dosen't check a macro and doesn't send an e-mail. I have an error in the path, so I understand the error, but I need an e-mail about that and I want that SAS stops processing because of macro.
Adding some options to the macro can fix it?
this is how I send out my email from SAS
not required to create a file on the system
filename outbox email "email#address.com";
data _null_;
file outbox
to=("email#address.com")
cc=("email#address.com")
subject="SUBJECT OF EMAIL "
/* attach=("C:\sas\results.out" "C:\sas\code.sas") */ ; /* incase you need to attach a file */
put 'Hello! ';
put ' ';
put 'Thank you & let me know if there is anything,';
put "&macrovar."; /*in case you need to put some detail in the email from macro variables make sure to use double quote for that */
put 'SIGNATURE ME';
run;

SAS Data Step if statement not working

With the below code the email always gets sent. 1 obviously does not equal 0, but yet it still runs. I have tried removing the do part but still get the same issue.
data _null_;
set TestTable;
if 1 = 0 then do;
file sendit email
to=("email#gmail.com")
subject="Some Subject Line";
end;
run;
While the file statement is considered an executable statement (and thus should not be executed when behind a false if statement), that is not really entirely true. SAS sees the file statement during compilation, and knows that it needs to create a file to write to - and thus, it's somewhat compile-time. That's what is happening here - SAS creates the file (in this case, the email) as a result of the compiler's activity, then doesn't actually populate it with anything, but still has an email at the end of the day.
The same thing happens with any other file - like so:
data _null_;
set sashelp.class;
if 0 then do;
file "c:\temp\test_non_zero.txt";
put name $;
end;
run;
A blank file is created by that code.
If you need to conditionally send emails, I would recommend wrapping your email code in a macro, and then calling that macro using call execute or similar from the dataset. Like so:
%macro write_email(parameters);
data _null_;
file sendit email
to=("email#gmail.com")
subject="Some Subject Line";
run;
%mend write_email;
data _null_;
set TestTable;
if 0 then do;
call execute('%write_email(parameters)');
end;
run;
Use email directives to abort the message.
http://support.sas.com/documentation/cdl/en/lrdict/64316/HTML/default/viewer.htm#a002058232.htm
data _null_;
file sendit email to=("email#gmail.com") subject="Some Subject Line";
if nobs=0 then put '!EM_ABORT!';
set TestTable nobs=nobs;
....
run;

A macro to send a email

my code to send mail using sas data step. here trying to create a code to send mail.
FILENAME outbox EMAIL ("***********");
DATA _NULL_;
FILE outbox
TO=("************")
FROM=("***********")
SUBJECT=("Example of a SAS E-mail" );
/* ATTACH=(""); */
PUT " ";
PUT "Hello Boss,";
PUT " ";
PUT "Attached are the Daily Operational Reports.";
PUT " ";
PUT "rrt";
RUN;
I don't tend to include email parameters in the datastep itself, but rather in fileref. I have tested the code below with my email address and it worked.
As I said in the comments, you simply need to wrap your datastep in a macro if you want it to run within macro.
You can have positional or keyword parameters. See this source for detailed info. I have used keyword parameters in this example. you call the macro by specifying the keyword and then the value like I showed the last line. If you don't put anything, then the keyword gets ignored.
By the way, the empty keywords result in initialised local macro variables. Let statements like ( %LET FROM=FROM="&FROM") are there to add a string "FROM=" to the beginning of "FROM" macro variable so the filename syntax would be complete. If macro variable such as "FROM" would be passed as is, it would resolve to email address that filename woudn't know what to do with.
%MACRO send_email (TO=,FROM=,SUBJECT=,ATTACHMENT=,BODY=);
%IF &TO ne %THEN
%LET TO="&TO";
%IF &FROM ne %THEN
%LET FROM=FROM="&FROM";
%IF &SUBJECT ne %THEN
%LET SUBJECT=SUBJECT="&SUBJECT";
%IF &ATTACHMENT ne %THEN
%LET ATTACHMENT=ATTACHMENT="&ATTACHMENT";
%IF &BODY ne %THEN
%LET BODY="&BODY";
FILENAME outbox EMAIL &TO &FROM &SUBJECT &ATTACHMENT;
DATA _NULL_;
FILE outbox;
PUT &BODY;
RUN;
%MEND;
%send_email(email=example#example.com);
Peeyush,
Try the code sample below. Make sure to enter your SMTP email host details and the email addresses to and from. You can make the email addresses macro variables if you want additional automation:
%macro emailM;
OPTIONS emailsys = SMTP emailhost = Youremail.Host.net emailport= 25;
FILENAME Mailbox EMAIL 'You#something.com';
attach=("\\somewhere\something\File.xls");
DATA _NULL_;
FILE MailBox TO=("somebody#something.com" "somebody2#something.com")
Subject="Example of a SAS E-mail";
PUT "Hello Boss,";
PUT " ";
PUT "Attached are the Daily Operational Reports.";
PUT " ";
PUT "Thank you";
RUN;
%mend emailM;
%emailM;

Repeatedly running a data step if URL fails to load

I am creating a dataset using filename URL web submissions. However, in some instaces I keep getting '502' responses from the server. To get around this I would like to use some conditional logic inside a macro. I'm most of the way there, but I cant quite get the end bit to work. The idea is that the macro, which is nested within other nested macros will keep trying this one submission until it gets a dataset that doesn't have 0 observations then move on:
%macro test_exst;
filename loader url "http://finance.yahoo.com/d/quotes.csv?s=&svar1.+&svar2.+&svar3.+&svar4.+&svar5.+&svar6.+&svar7.+&svar8.+&svar9.+&svar10.+
&svar11.+&svar12.+&svar13.+&svar14.+&svar15.+&svar16.+&svar17.+&svar18.+&svar19.+&svar20.+
&svar21.+&svar22.+&svar23.+&svar24.+&svar25.+&svar26.+&svar27.+&svar28.+&svar29.+&svar30.+
&svar31.+&svar32.+&svar33.+&svar34.+&svar35.+&svar36.+&svar37.+&svar38.+&svar39.+&svar40.+
&svar41.+&svar42.+&svar43.+&svar44.+&svar45.+&svar46.+&svar47.+&svar48.+&svar49.+&svar50.+
&svar51.+&svar52.+&svar53.+&svar54.+&svar55.+&svar56.+&svar57.+&svar58.+&svar59.+&svar60.+
&svar61.+&svar62.+&svar63.+&svar64.+&svar65.+&svar66.+&svar67.+&svar68.+&svar69.+&svar70.+
&svar71.+&svar72.+&svar73.+&svar74.+&svar75.+&svar76.+&svar77.+&svar78.+&svar79.+&svar80.+
&svar81.+&svar82.+&svar83.+&svar84.+&svar85.+&svar86.+&svar87.+&svar88.+&svar89.+&svar90.+
&svar91.+&svar92.+&svar93.+&svar94.+&svar95.+&svar96.+&svar97.+&svar98.+&svar99.+&svar100.+
&svar101.+&svar102.+&svar103.+&svar104.+&svar105.+&svar106.+&svar107.+&svar108.+&svar109.+&svar110.+
&svar111.+&svar112.+&svar113.+&svar114.+&svar115.+&svar116.+&svar117.+&svar118.+&svar119.+&svar120.+
&svar121.+&svar122.+&svar123.+&svar124.+&svar125.+&svar126.+&svar127.+&svar128.+&svar129.+&svar130.+
&svar131.+&svar132.+&svar133.+&svar134.+&svar135.+&svar136.+&svar137.+&svar138.+&svar139.+&svar140.+
&svar141.+&svar142.+&svar143.+&svar144.+&svar145.+&svar146.+&svar147.+&svar148.+&svar149.+&svar150.+
&svar151.+&svar152.+&svar153.+&svar154.+&svar155.+&svar156.+&svar157.+&svar158.+&svar159.+&svar160.+
&svar161.+&svar162.+&svar163.+&svar164.+&svar165.+&svar166.+&svar167.+&svar168.+&svar169.+&svar170.+
&svar171.+&svar172.+&svar173.+&svar174.+&svar175.+&svar176.+&svar177.+&svar178.+&svar179.+&svar180.+
&svar181.+&svar182.+&svar183.+&svar184.+&svar185.+&svar186.+&svar187.+&svar188.+&svar189.+&svar190.+
&svar191.+&svar192.+&svar193.+&svar194.+&svar195.+&svar196.+&svar197.+&svar198.+&svar199.+&svar200.
&f=&&fvar&a." DEBUG ;
/* data step based on filename url above goes here, each pass will give 500 metrics x 1 symbol dataset*/
%put create dataset from csv submission;
data temp_&I._&&fvar&a.;
infile loader length=len MISSOVER /*delimiter = ','*/;
/* input record $varying8192. len; */
input record $varying30. len;
format record $30.;
informat record $30.;
run;
data _null_;
dsid=open("temp_&I._&&fvar&a.");
obs=attrn(dsid,"nobs");
put "number of observations = " obs;
if obs = 0 then stop;
else;
filename loader url "http://finance.yahoo.com/d/quotes.csv?s=&svar1.+&svar2.+&svar3.+&svar4.+&svar5.+&svar6.+&svar7.+&svar8.+&svar9.+&svar10.+
&svar11.+&svar12.+&svar13.+&svar14.+&svar15.+&svar16.+&svar17.+&svar18.+&svar19.+&svar20.+
&svar21.+&svar22.+&svar23.+&svar24.+&svar25.+&svar26.+&svar27.+&svar28.+&svar29.+&svar30.+
&svar31.+&svar32.+&svar33.+&svar34.+&svar35.+&svar36.+&svar37.+&svar38.+&svar39.+&svar40.+
&svar41.+&svar42.+&svar43.+&svar44.+&svar45.+&svar46.+&svar47.+&svar48.+&svar49.+&svar50.+
&svar51.+&svar52.+&svar53.+&svar54.+&svar55.+&svar56.+&svar57.+&svar58.+&svar59.+&svar60.+
&svar61.+&svar62.+&svar63.+&svar64.+&svar65.+&svar66.+&svar67.+&svar68.+&svar69.+&svar70.+
&svar71.+&svar72.+&svar73.+&svar74.+&svar75.+&svar76.+&svar77.+&svar78.+&svar79.+&svar80.+
&svar81.+&svar82.+&svar83.+&svar84.+&svar85.+&svar86.+&svar87.+&svar88.+&svar89.+&svar90.+
&svar91.+&svar92.+&svar93.+&svar94.+&svar95.+&svar96.+&svar97.+&svar98.+&svar99.+&svar100.+
&svar101.+&svar102.+&svar103.+&svar104.+&svar105.+&svar106.+&svar107.+&svar108.+&svar109.+&svar110.+
&svar111.+&svar112.+&svar113.+&svar114.+&svar115.+&svar116.+&svar117.+&svar118.+&svar119.+&svar120.+
&svar121.+&svar122.+&svar123.+&svar124.+&svar125.+&svar126.+&svar127.+&svar128.+&svar129.+&svar130.+
&svar131.+&svar132.+&svar133.+&svar134.+&svar135.+&svar136.+&svar137.+&svar138.+&svar139.+&svar140.+
&svar141.+&svar142.+&svar143.+&svar144.+&svar145.+&svar146.+&svar147.+&svar148.+&svar149.+&svar150.+
&svar151.+&svar152.+&svar153.+&svar154.+&svar155.+&svar156.+&svar157.+&svar158.+&svar159.+&svar160.+
&svar161.+&svar162.+&svar163.+&svar164.+&svar165.+&svar166.+&svar167.+&svar168.+&svar169.+&svar170.+
&svar171.+&svar172.+&svar173.+&svar174.+&svar175.+&svar176.+&svar177.+&svar178.+&svar179.+&svar180.+
&svar181.+&svar182.+&svar183.+&svar184.+&svar185.+&svar186.+&svar187.+&svar188.+&svar189.+&svar190.+
&svar191.+&svar192.+&svar193.+&svar194.+&svar195.+&svar196.+&svar197.+&svar198.+&svar199.+&svar200.
&f=&&fvar&a." DEBUG ;
data temp_&I._&&fvar&a.;
infile loader length=len MISSOVER /*delimiter = ','*/;
/* input record $varying8192. len; */
input record $varying30. len;
format record $30.;
informat record $30.;
run;
run;
%mend;
%test_exst;
The idea here is try URL submission, create dataset from it, check number of obs is not zero. If its not end the macro. If it is resubmit the same filename URL then create the dataset from it again. Keep doing this until the server respond, the exit the macro and most on to the rest of the code.
I haven't got as far as running this code in anger yet. I'm guessing the filename URL will work fine, but it is the fact that the code is attempting to create a dataset within a data null step right at the end that is making it fall over. Any ideas?
Thanks
Without getting into the specifics of your project, a good way to approach this generally is with recursion.
%macro test_me(iter);
%let iter=%eval(&iter.+1);
data my_data;
infile myfilename;
input stuff;
call symputx("obscount",_n_);
run;
%if &obscount=1 and &iter. < 10 %then %do;
%put Iteration &iter. failed, trying again;
%test_me(&iter.);
%end;
%mend test_me;
%test_me(0);
It checks to see if it worked, and if it did not work, it calls itself again, with a maximum iteration count to make sure you don't end up in infinite loop land if the server is down or somesuch. You also might put a delay in there if the server has a maximum frequency of calls or any other rules the API requires you to follow.