In particular, what, if any, are the substantial changes or extensions in the programming language that gives it functionality beyond PROC TABULATE?
Or is it the case that the programming languages in Proc Tabulate and TPL Tables ( from QQQ Software ) are pretty close to the same?
I was really surprised to hear about TPL Tables, and it's predecessor, the Table Producing Language from the US Department of Labor in the 1970s. After all these years, I had never heard of it. Turns out, two commercial descendants of the Table Producing Language are the SAS PROC and TPL Tables.
Has anyone worked with both? Why are TPL Tables so unknown?
Robert
You are correct, both TABULATE and QQQ TPL Tables are descendants of the US Bureau of Labor Statistics TPL. According to this thread, the developers of TPL/PCL at the Bureau of Labor Statistics eventually left BLS and started QQQ.
This SAS article is a good read regarding TABULATE. According to the article, TABULATE, which was introduced in the 80s, originally borrowed much of its syntax and features from BLS TPL while addressing some of its shortcomings, though the specific shortcomings addressed are not mentioned.
What, if any, are the substantial changes or extensions in the programming language that give it functionality beyond PROC TABULATE?
The features of QQQ TPL Tables have evolved over time, as have the features of TABULATE. I've found no information to suggest that ongoing TABULATE development kept abreast of QQQ TPL features, so the two systems are now likely too different to compare effectively. As a SAS product, TABULATE is intended to integrate with other SAS technologies, such as ODS. TPL probably integrates with other QQQ technologies.
Although, just based on documentation, something that TPL (v7+) can do that TABULATE (as of v9.4) cannot is perform statistical hypothesis tests, e.g. t-tests, chi-squared tests, and ANOVA. But in SAS you have other, likely more flexible, options to get these.
If you're looking to integrate one or the other into your development cycle, I recommend choosing the one that best fits your current system. If you're already using SAS, stick with TABULATE.
Why is TPL Tables so unknown?
Who knows. It's still in use by the BLS and a few others, apparently. But SAS is such a giant in the field that it tends to overshadow its competition.
Related
What procs are easy to learn and essential for SAS programming? I have learned several like proc print, sort, freq, format, univariate, anova, glm, import, transpose. What ones should I learn next?
Welcome to Stack Overflow (and SAS). The procedures that AlanC mention are all important.
Probably your best bet is to pick up a copy of The little SAS book and learn the data processing as well as the analysis procedures. I have used many versions of it for years and students like it. SAS changes at a glacial pace. So, if money is tight, pick up an older edition.
You have already hit many of the main procedures. Focus on data processing with data step and PROC SQL. SQL is its own language and is extremely useful with or without SAS. Also do not neglect ODS. SAS can make very beautiful output and the aesthetics matter when you are showing your portfolio.
If you want to be a professional SAS programmer you will need to learn macro, to automate tasks, and also the intermediate to advanced magic that Ron Cody writes about. Get comfortable with the language then work on converting your code into macro. Along the way be sure to check Cody's data cleaning book
SAS has several forms it uses to create output data sets from within a procedure. It is not always clear whether or not a particular procedure can generate a data set and, if it seems to be able to, it's not always clear how.
Off the top of my head, here are some examples of how widely the syntax can differ.
Example 1
proc sort data = sashelp.baseball out = baseball_sorted;
by
league
division
;
run;
Example 2
proc means noprint data = baseball_sorted;
by
league
division
;
var nHits;
output
out = baseball_avg_hits (drop = _TYPE_ _FREQ_)
mean = mean_hits
;
run;
Example 3
ods exclude all;
ods output
statistics = baseball_statistics
equality = baseball_ftest
;
proc ttest data = baseball_sorted;
class league;
var nHits;
run;
ods exclude none;
Example 4
The PROC ANOVA OUTSTAT= option.
It seems almost as if SAS has implemented each of these willy-nilly. Is the SAS syntax dictating how to create a data set directed by some consistent approach I am not seeing or is it truly capricious and arbitrary?
For PROC code, the syntax for outputting data is often specific to that procedure, which often feels willy-nilly. (Your examples 1, 2, 4) I think PROC developers are given a lot of freedom, and remember that many of these PROCS are 30+ years old.
The great thing about the Output Delivery System (ODS, your example 3) is it provides a single syntax for outputting data, regardless of the procedure. So you can use the ODS OUTPUT statement with (almost?) any PROC. The names and structures of the output objects will of course vary between PROCs. So if you are looking for a consistent approach, I would focus on using ODS OUTPUT. ODS was added in V7 (I think).
It would be interesting to try to find an example of an output dataset which could be made by a PROC but could not be made by ODS OUTPUT. I hope there aren't any. If that is the case, you could consider the range of OUTPUT statements/options within PROCs as legacy code.
Agree with Quentin. You have to remember that there are SAS systems out there running code written in the 80s. SAS would have a huge headache if they forced every team to rewrite all the procedures and then forced their customers to change all their code. SAS has been around since the 60s and the organic growth of the syntax is to be expected.
FWIW, having an OUT= statement makes sense on things with no graphical output. I.E. PROC SORT or PROC TRANSPOSE.
The way I see it there are four main ways to specify the output data sets.
In the PROC statement you may be able to specify some type of output statements or options, such as OUT= OUTEST=.
In the main statement of the procedure, ie MODEL/TABLE can have options that allow for output. ie PROC FREQ has an OUT= on the TABLE statement.
An explicit OUTPUT statement within a procedure. These are typically from older procedures. ie PROC MEANS
ODS tables which are relatively newer method, more frequently used these days since the format aligns with what you'd expect to see.
Yes, there are multiple places to check, but fortunately the SAS documentation for procedures is relatively clear with the options and how to use/specify the outputs.
If I've missed anything that seems different post in the comments and I can update this.
PS. Although SAS is definitely bad, trying to navigate different packages/modules in Python to export an XLSX file isn't straight forward either. Some packages support some options others don't. I've given up on asking why these days and just accept it as peculiarities of the different languages at this point.
I have a time series dataset with serious serial correlation problem, so I adopted Prais-Winsten estimator with iterated estimates to fix that. I did the regressions in Stata with the following command:
prais depvar indepvar indepvar2, vce(robust) rhotype(regress)
My colleague wanted to reproduce my results in SAS, so she used the following:
proc autoreg data=DATA;
model depvar = indepvar indepvar2/nlag=1 iter itprint method=YW;
run;
For the different specifications we ran, some of them roughly match, while others do not. Also I noticed that for each regression specification, Stata has many more iterations than SAS. I wonder if there is something wrong with my (or my colleague's) code.
Update
Inspired by Joe's comment, I modified my SAS code.
/*Iterated Estimation*/
proc autoreg data=DATA;
model depvar = indepvar indepvar2/nlag=1 itprint method=ITYW;
run;
/*Twostep Estimation*/
proc autoreg data=DATA;
model depvar = indepvar indepvar2/nlag=1 itprint method=YW;
run;
I have a few suggestions. Note that I'm not a real statistician and am not familiar with the specific estimators here, so this is just a quick read of the docs.
First off, the most likely issue is that it looks like SAS uses the OLS variance estimation method. That is, in your Stata code, you have vce(robust), which is in contrast to what I read SAS as using, the equivalent of vce(ols). See this page in the docs which explains how SAS does the Y-W method of autoregression, compared to this doc page that explains how Stata does it.
Second, you probably should not specify method=YW. SAS distinguishes between the simple Y-W estimation ("two-step" method) and iterated Y-W estimation. method=ITYW is what you want. You specify iter, so it may well be that you're getting this anyway as SAS tends to be smart about those sorts of things, but it's good to verify.
I would suggest actually turning the iterations off to begin with - have both do the two-step method (Stata option twostep, SAS by removing the iter request and specifying method=YW or no method specification). See how well they match there. Once you can get those to match, then move on to iterated; it's possible SAS has a different cutoff than Stata and may well not iterate past that.
I'd also suggest trying this with only one independent and dependent variable pair first, as it's possible the two programs handle things differently when you add in a second independent variable. Always start simple and then add complexity.
is it possible to show the mathemetical formular / concept behind the analysis done with SAS Enterprise?
Assuming SAS would calculate a correlation between a list of numbers -- is it possible to see what exactly SAS did from a mathematical perspective?
It is not possible to ask SAS for the mathematical formula, no. You can check the documentation; for example, this page gives many of the 'elemantary statistics' formulas (like variance, UCLM, etc.)
If you need the formula behind something more complex that you can't find online, contact your SAS Support rep, and they may be able to put you in contact with the developer of that particular proc - like if you need to know some particular to how PROC GLM does something.
You can ask SAS to give you the SAS code that it ran if you executed a task (in most cases it's available by clicking on the task node), in many cases, but that would be something like proc freq; tables a*b; run;, not a mathematical formula per se.
I'm trying to create a table using SAS 9.3 that shows information on current and past projects. For current projects, I want to show whether they've met various criteria ("yes", "no", OR "n/a"). In the same table, I want to show summary information of past projects (i.e. how many projects met the criteria, how many did not, and how many were n/a). Having one table to show current projects and one table to show past projects is easy. I'm struggling to show them together in a single table. Using proc tabulate, my code looks like this:
proc tabulate data = projects order=formatted missing;
class project;
var dt criteria1 criteria2 criteria3;
table
(dt=”Start Date)"*min=''*f=year_date.)
(criteria1="Criteria 1")*sum=''*f=ans.
(criteria2="Criteria 2")*sum=''*f=ans.
(criteria3="Criteria 3")*sum=''*f=ans.
,(project='');
format project $project_label.;
run;
The values for each criteria are 1 for yes, 0 for no, and . for n/a. The year format distinguishes current from past projects and the ans format shows "yes" for 1 and "no" for 0. This works for the the current projects. It also gives me the total number of past projects with "yes" answers. What I don't know how to do is the break-out for past projects showing no and n/a. (I'm also in trouble if there sum of past projects is 1 or 0 because the format would replace those with 'yes' or 'no.'
Any suggestions?
Thanks.
Brandon
Edit: I'll try to add some sample data that looks reasonable...
Criteria ActiveProject1 ActiveProject2 Past_Projects
Criteria1 yes no 5/10/5
Criteria2 yes yes 7/9/4
Criteria3 no yes 2/15/3
While I can't visualize what you're trying to do, one suggestion I would have is to use the ODS DOCUMENT and PROC DOCUMENT facility, or PROC REPORT.
You can in this way build your two separate tables that you like, then use PROC DOCUMENT to put them together so they show up in one place. This might suffice for what you're aiming to do.
If it doesn't, then PROC REPORT is probably more apt than PROC TABULATE when you are in some places summarizing and in other places not, if that's what you're trying to do. It allows limited data step functionality along with the summarization elements of the tabulation procs. I can't suggest a specific example because I don't understand what you're doing, but it may be the superior choice.