I wonder if I can see data going to an output file during each iteration while debugging. Using DEBUG option allows me to see what is going on during each iteration but I can’t see which data got written to an output file during each iteration.
I think it is important to know as some programs include more complex options like looping, retaining, output, # or ## etc.
Thanks,
Yulia
I would suggest you amend your data step code to include an extra variable indicating which part of the data step each bit of output came from. You don't have to include this in the output dataset, but next to each put statement you can include a corresponding putlog statement.
Related
I downloaded daily MODIS DATA LEVEL 3 data for a few months from https://disc.gsfc.nasa.gov/datasets. The filenames are of the form MCD06COSP_M3_MODIS.A2006001.061.2020181145945 but the files do not contain any time dimension. Hence when I use ncecat to concatenate various files, the date information is missing in the resulting file. I want to know how to add the time information in the combined dataset.
Your commands look correct. Good job crafting them. Not sure why it's not working. Possibly the input files are HDF4 format (do they have a .hdf suffix?) and your NCO is not HDF4-enabled. Try to download the files in netCDF3 or netCDF4 format and your commands above should work. If that's not what's wrong, then examine the output files in each step of your procedure and identify which step produces the unintended results and then narrow your question. Good luck.
I have some problem to find out if what I want to do is possible or not on SAS EG 7.15.
Context
We have splitted a quite big project into several flow process which have each one a program to define a %macro statement and a final process flow which contain a program call "EXECUTION" to call all those program in the row in this way :
Flow_Process_1/program:
%macro TheFirstOne;
some code
%mend TheFirstOne;
some other process flow
Flow_Process_123/EXECUTION:
%TheFirstOne;
%TheSecondOne;
%TheThirdOne;...
The advantage is that all is splitted and easy to execute just by a click on the first process flow -> SHIFT+Click on the last one and everything is going to be executed. Also you have a unique log file of all the different program in one. And everything works perfectly.
The Problem
The problem is that we have a quite long execution (in time) that we wanted to split into parallel process which is not really hard to set with the "Allow Parallel execution on the same server" option and this feature also work like a charm when you try to execute some program which work on their own.
However, with the way we are working, the first process (thread i would say) knows about our %macro definition and works perfectly, but the second or any other one don't know about them and give error like :
WARNING: Apparent invocation of macro xxxxxxx not resolved.
since we never execute the program before which define this macro before.
My attempt
I tried to use the Autoexec feature but I have to copy the definition of every macro (5000 + lines) inside if that it works for every parallel process. I also looked into SAS code to run a flow process in a program but it looks like it does not exist, or I did not find it.
I'm quite sure that senior in SAS will say that is not the way we should use SAS EG, but we did that way and I just would like to know if there is a solution to let know to other process the definition of all %macro define previously.
I have the following code that is running inside a macro. When it is run in interactive mode, it runs absolutely fine, no errors or warning. That was the case for last two year.
The same code has now been deployed in batch mode and it generates a warning WARNING: Apparent symbolic reference FIRSTRECCOUNT not resolved. and no value assigned to macro variable.
My question is, does anyone have any ideas why batch mode and interactive mode would behave differently?
Here some more information:
The dataset is being created and it is in work library.
The dataset does get opened by data step.
`firstreccount' doesn't get initialiased anywhere else in the program
I have search sas community. There is a topic here, but I don't have the same errors in batch initilisation as described in the answer.
Detailed information on the warning but it doesn't explain by it would work in interactive mode, but not in batch mode.
.
1735 %LET FIRSTSET = work.dataset1;
1744 DATA _NULL_;
1745 IF 0 THEN
1746 SET &FIRSTSET NOBS=X;
1747 CALL SYMPUT('FIRSTRECCOUNT' ,X);
1748 STOP;
1749 RUN;
1755 DATA _NULL_;
1756 IF 0 THEN
1757 SET &SECONDSET NOBS=X;
1758 CALL SYMPUT('SECONDRECOUNT' ,X);
1759 STOP;
1760 RUN;
WARNING: Apparent symbolic reference FIRSTRECCOUNT not resolved.
Update:
So I have attempted to replicate the error by copying the code with warning into a separate scheduled flow, but it didn't cause any errors at all.
By the way, the original job was deployed from SAS DI studio. I have checked all lines in user written code nodes and made sure that the length was within 80 characters as recommended by #RawFocus, #RobertPentridge, but it didn't solve the issue.
As recomended by #data_null_ I have checked VALIDVARNAME and it was different between interactive (value of "any") and batch mode (value of "V7") but changing these hasn't made any difference.
I have rewritted the logic to get the number of observations by calling attr for an open dataset. This eliminated the warning, but program would still fail with warning popping out in different places. It made me think Robert Partridge is correct. At the same time, I got an error that a macro not being resolved. The macro was inserted by DI studio to collect performance MI even that the job wasn't meant to be collecting MI. This made me think that SAS DI studio is not generating code correctly when deploying it, so I manually edited the deployed code to remove offending macro call and I also spotted that there was one line of code with MD5 function that was too long on one line because of a number of parameters being passed to it, so I inserted some white space. And finally the problem was fixed!!
I still need to do something about the job because when it will get redeployed from SAS DI, it will generate the same errors again. I don't have time to look into this further at the moment.
Conclusion: what you write in SAS DI and what gets deployed could be slightly different which could cause syntax parse to throw errors in random places. So I will mark Robert's answer as correct because it got me closer to solving the problem then any other answer.
The problem could be happening above the code snippet you pasted. The parser got into a funk earlier, and ended up issuing warning about code that is perfectly fine.
Check to make sure that no code within a macro is longer that ~160 chars on a single line. I try to keep my code well below that but long lines of code can run fine interactively and fail in batch - particularly when inside of a macro.
I expect your program has some small error above that does not cause SAS to go into syntax check mode when run interactively but does cause SAS to set obs to 0 and enter syntax check mode when run in batch.
One possibility is the limit (in batch mode) of the length of a line in your submitted SAS program:
See: http://support.sas.com/kb/15/883.html
Which version of SAS are you running?
When working is the base sas client, if a user accidentially forgets a parathesis, quote, semi-colon, or end-comment that will sometimes cause any additional code they try running to fail. If I'm trying to help someone experiencing one of these errors, how can I clear the current buffer and start fresh without opening a brand new session?
I've always used the one documented on support.sas.com
http://support.sas.com/documentation/cdl/en/mcrolref/61885/HTML/default/viewer.htm#a001302436.htm
*'; *"; *); */; %mend; run;
Submit that until you get the log message :
ERROR: No matching %MACRO statement for this %MEND statement.
There is no true best way, as it depends on too many things. Presumably the user will know what might have gone wrong, and can use that information; for example, are we possibly missing a %mend somewhere that caused us to get locked into a macro?
The EG method is one of the better ones. At the start of every EG process (ie, every time you hit 'Run' on a program or section of program), it runs:
;*';*";*/;quit;run;
That:
Semicolon ends any statement that hadn't ended
Comment singlequote, and comment doublequote, end any trailing quotes (both trailed by ; )
Star slash ends any block comments
Quit and run end any procedures that had been left open.
EG doesn't include %mend; because it would generate a warning, but you could include that if you didn't mind the warning and wanted to catch one of the common issues. If you do include it, put it up front - macros will not properly deal with those hanging quotes, and you need to be out of them for sure first.
I'm using pyuno to read an excel spreadsheet (running on linux.) Many cells have formulas referring to addins that are, obviously, not available. However the cell values are what I want.
But when I load and read the sheet, it seems those formulas are being evaluated and thus the values are being overwritten with errors.
I've tried several things, none of which have worked:
set flags AutomaticCalculation=False, MacroExecutionMode=NEVER_EXECUTE in the call to desktop.loadComponentFromURL
call document.enableAutomaticCalculation(False) on the loaded document
Any suggestions?
If formluas aren't a matter, you might circumvent the problem by processing a copy of your spreadsheet in which only the values (not the formulas) are present.
To achieve this quickly, select the whole sheet content, copy, special paste; then remove everything except "value". Save to a new file (make sure you don't overwrite the original file or every formula will be lost!). Your script should then be able to process this file.
This is an ugly solution, as there must be a way to do it programmaticaly.
Calc does not yet support using the cached results after loading the document. Libreoffice Calc does now use cached results for xls documents. The results are also stored in ods but are ignored while loading the document and the formula result is evaluated by compiling and interpreting the saved formula.
There are some plans to add this for ods and xlsx too but there are many ods producers out there writting incorrect results in the file. So till now the only solution is to have a second version of the document only saving the results (or implementing it inside calc).