Proc Append Duplication (SAS) - sas

I was working on a SAS problem where I need to append the data. The data run is successful but it creates duplicates every time I run the program.
Please check my code and screenshot of the table:
Question: Create a new file "Total_Sales" by appending data file "Hyundai" with the file first created in problem 3.
/*Problem 3*/:
data avik1.var1;
length uniqueid $50 Manufacturer $ 50 Model $20 Sales_in_thousands 8 _4_year_resale_value 8 Price_in_thousands 8;
retain uniqueid Manufacturer Model Latest_Launch Sales_in_thousands _4_year_resale_value Price_in_thousands;
set avik1.conc(drop= Vehicle_type Engine_size Horsepower Wheelbase Width Length Curb_weight Fuel_capacity Fuel_efficiency );
informat Latest_Launch date9.;
format Latest_Launch ddmmyy10.;
run;
proc print data = avik1.var1;
run;
/* Data To be Appended */
data avik1.hyundai;
length uniqueid $ 50 Manufacturer $ 50 Model $20 Sales_in_thousands 8 _4_year_resale_value 8;
informat Latest_Launch date7. ;
format Latest_Launch ddmmyy10.;
input Manufacturer $ Model $ Sales_in_thousands _4_year_resale_value Latest_Launch;
uniqueid=(Model||Manufacturer);
cards;
Hyundai Tuscon 16.919 16.36 2Feb12
Hyundai i45 39.384 19.875 3Jun11
Hyundai Verna 14.114 18.225 4Jan12
Hyundai Terracan 8.558 29.775 10Mar11
;
run;
Proc Print data = avik1.hyundai;
run;
Now I used the following code to append:
data avik1.total_sales;
set avik1.var1 avik1.hyundai;
proc append base=avik1.var1 new=avik1.hyundai force;
run;
proc print data= avik1.total_sales;
run;
The program runs but gets me duplicates which you can check in the image
Screenshot in Yellow Mark Shows Duplicates
I am new to SAS really appreciate your response and solution to this problem. Also please tell me why this is happening.
Thanks!

Did you run it twice? I'm guessing but that could be the reason you see duplicates. I'll try to explain.
In your append code here, you are creating the new dataset total_sales by combining var1 and hyundai:
data avik1.total_sales;
set avik1.var1 avik1.hyundai;
In the below code, you are not creating a new dataset, you are expanding var1 by adding the records from hyundai.
proc append base=avik1.var1 new=avik1.hyundai force;
run;
If you ran this proc append and then ran the first data step again, you will have duplicates of all hyundai records because you are taking the EXPANDED var1 and re-adding the hyundai records.
So the point is, to answer the original question, the proc append procedure is totally unnecessary. You achieved it with just the data step.

Related

Output the dropped/excluded observation in Proc GLIMMIX - SAS

When I run a proc glimmix in SAS, sometimes it drops observations.
How do I get the set of dropped/excluded observations or maybe the set of included observations so that I can identify the dropped set?
My current Proc GLIMMX code is as follows-
%LET EST=inputf.aarefestimates;
%LET MODEL_VAR3 = age Male Yearc2010 HOSPST
Hx_CTSURG Cardiogenic_Shock COPD MCANCER DIABETES;
data work.refmodel;
set inputf.readmref;
Yearc2010 = YEAR - 2010;
run;
PROC GLIMMIX DATA = work.refmodel NOCLPRINT MAXLMMUPDATE=100;
CLASS hospid HOSPST(ref="xx");
ODS OUTPUT PARAMETERESTIMATES = &est (KEEP=EFFECT ESTIMATE STDERR);
MODEL RADM30 = &MODEL_VAR3 /Dist=b LINK=LOGIT SOLUTION;
XBETA=_XBETA_;
LINP=_LINP_;
RANDOM INTERCEPT/SUBJECT= hospid SOLUTION;
OUTPUT OUT = inputf.aar
PRED(BLUP ILINK)=PREDPROB PRED(NOBLUP ILINK)=EXPPROB;
ID XBETA LINP hospst hospid Visitlink Key RADM30;
NLOPTIONS TECH=NRRIDG;
run;
Thank you in advance!
It drops records with missing values in any variable you're using in the model, in a CLASS, BY, MODEL, RANDOM statement. So you can check for missing among those variables to see what you get. Usually the output data set will also indicate this by not having predictions for the records that are not used.
You can run the code below.
*create fake data;
data heart;set sashelp.heart; ;run;
*Logistic Regression model, ageCHDdiag is missing ;
proc logistic data=heart;
class sex / param=ref;
model status(event='Dead') = ageCHDdiag height weight diastolic;
*generate output data;
output out=want p=pred;
run;
*explicitly flag records as included;
data included;
set want;
if missing(pred) then include='N'; else include='Y';
run;
*check that Y equals total obs included above;
proc freq data=included;
table include;
run;
The output will show:
The LOGISTIC Procedure
Model Information
Data Set WORK.HEART
Response Variable Status
Number of Response Levels 2
Model binary logit
Optimization Technique Fisher's scoring
Number of Observations Read 5209
Number of Observations Used 1446
And then the PROC FREQ will show:
The FREQ Procedure
Cumulative Cumulative
include Frequency Percent Frequency Percent
ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ
N 3763 72.24 3763 72.24
Y 1446 27.76 5209 100.00
And 1,446 records are included in both of the data sets.
I think I answered my question.
The code line -
OUTPUT OUT = inputf.aar
gives the output of the model. This table includes all the observations used in the proc statement. So I can match the data in this table to my input table and find the observations that get dropped.
#REEZA - I already looked for missing values for all the columns in the data. Was not able to identify the records there are getting dropped by only identifying the no. of records with missing values. Thanks for the suggestion though.

Value labels to be created using data from another data set

I am having two data sets. The first data set has airport codes (JFK, LGA, EWR) in a variable 'airport'. The second dataset has the list of all major airports in the world. This dataset has two variables 'faa' holding the FAA Code (like JFG, LGA, EWR) and 'name' holding the actual name of the airport (John. F Kennedy, Le Guardia etc.).
My requirement is to create value labels for in the first data set, so that instead of airport code, the actual name of the airport comes up. I know I can use custom formats to achieve this. But can I write SAS code which can read the unique airport codes, then get the names from another data set and create a value label automatically?
PS: Other wise, the only option I see is to use MS Excel to get the unique list of FAA codes in dataset 1, and then use VLOOKUP to get the names of the airports. And then create one custom format by listing each unique FAA code and the airport name.
I think "value label" is SPSS terminology. Looks like you want to create a format. Just use your lookup table to create an input dataset for PROC FORMAT.
So if your second table looks like this:
data table2;
length FAA $4 Name $40 ;
input FAA Name $40. ;
cards;
JFK John F. Kennedy (NYC)
LGA Laguardia (NYC)
EWR Newark (NJ)
;
You can use this code to convert it into a dataset that PROC FORMAT can use to create a format.
data fmt ;
fmtname='$FAA';
hlo=' ';
set table2 (rename=(faa=start name=label));
run;
proc format cntlin=fmt lib=work.formats;
run;
Now you can use that format with your other data.
proc freq data=table1 ;
tables airport ;
format airport faa. ;
run;
Firstly, consider if it is really a format what is needed. For example, you may just do a left join to retrieve the column (airport) name from table2 (FAA-Name table).
Anyway, I believe the following macro does the trick:
Create auxiliary tables:
data have1;
input airport $;
datalines;
a
d
e
;
run;
data have2;
input faa $ name $;
datalines;
a aaaa
b bbbb
c cccc
d dddd
;
run;
Macro to create Format:
%macro create_format;
*count number of faa;
proc sql noprint;
select distinct count(faa) into:n
from have2;
quit;
*create macro variables for each faa and name;
proc sql noprint;
select faa, name
into:faa1-:faa%left(&n),:name1-:name%left(&n)
from have2;
quit;
*create format;
proc format;
value $airport
%do i=1 %to &n;
"&faa%left(&i)" = "&name%left(&i)"
%end;
other = "Unknown FAA code";
run;
%mend create_format;
%create_format;
Apply format:
data want;
set have1;
format airport $airport.;
run;

How to retrieve SAS overwritten sashelp dataset?

by mistake i overwrote the sashelp.class dataset instead of creating a new one setting the data from sashelp.class.
I need the initial stage so that i might work with exam's questions.
Is there any way to make the dataset return to its initial stage? By downloading or changing the file on my PC?
Thanks!
Run the below to get your sashelp dataset back:
data sashelp.class ;
attrib
Name length= $8
Sex length= $1
Age length= 8
Height length= 8
Weight length= 8
;
infile cards dsd delimiter=',';
input
Name :$char.
Sex :$char.
Age
Height
Weight
;
datalines4;
Alfred,M,14,69,112.5
Alice,F,13,56.5,84
Barbara,F,13,65.3,98
Carol,F,14,62.8,102.5
Henry,M,14,63.5,102.5
James,M,12,57.3,83
Jane,F,12,59.8,84.5
Janet,F,15,62.5,112.5
Jeffrey,M,13,62.5,84
John,M,12,59,99.5
Joyce,F,11,51.3,50.5
Judy,F,14,64.3,90
Louise,F,12,56.3,77
Mary,F,15,66.5,112
Philip,M,16,72,150
Robert,M,12,64.8,128
Ronald,M,15,67,133
Thomas,M,11,57.5,85
William,M,15,66.5,112
;;;;
run;
These datalines were generated using %mp_ds2cards() - a macro I contributed to the open source MacroCore library, available here: https://github.com/sasjs/core

How to create a new variable in SAS by extracting part of the value of an existing numeric variable?

I have two datasets in SAS that I would like to merge, but they have no common variables. One dataset has a "subject_id" variable, while the other has a "mom_subject_id" variable. Both of these variables are 9-digit codes that have just 3 digits in the middle of the code with common meaning, and that's what I need to match the two datasets on when I merge them.
What I'd like to do is create a new common variable in each dataset that is just the 3 digits from within the subject ID. Those 3 digits will always be in the same location within the 9-digit subject ID, so I'm wondering if there's a way to extract those 3 digits from the variable to make a new variable.
Thanks!
SQL(using sample data from Data Step code):
proc sql;
create table want2 as
select a.subject_id, a.other, b.mom_subject_id, b.misc
from have1 a JOIN have2 b
on(substr(a.subject_id,4,3)=substr(b.mom_subject_id,4,3));
quit;
Data Step:
data have1;
length subject_id $9;
input subject_id $ other $;
datalines;
abc001def other1
abc002def other2
abc003def other3
abc004def other4
abc005def other5
;
data have2;
length mom_subject_id $9;
input mom_subject_id $ misc $;
datalines;
ghi001jkl misc1
ghi003jkl misc3
ghi005jkl misc5
;
data have1;
length id $3;
set have1;
id=substr(subject_id,4,3);
run;
data have2;
length id $3;
set have2;
id=substr(mom_subject_id,4,3);
run;
Proc sort data=have1;
by id;
run;
Proc sort data=have2;
by id;
run;
data work.want;
merge have1(in=a) have2(in=b);
by id;
run;
an alternative would be to use
proc sql
and then use a join and the substr() just as explained above, if you are comfortable with sql
Assuming that your "subject_id" variable is a number then the substr function wont work as sas will try convert the number to a string. But by default it pads some paces on the left of the number.
You can use the modulus function mod(input, base) which returns the remainder when input is divided by base.
/*First get rid of the last 3 digits*/
temp_var = floor( subject_id / 1000);
/* then get the next three digits that we want*/
id = mod(temp_var ,1000);
Or in one line:
id = mod(floor(subject_id / 1000), 1000);
Then you can continue with sorting the new data sets by id and then merging.

SAS - Find value in a column and display that value in excel export

I am trying to basically do this :
I have a frequency query running on a data set which will output the result in excel.
I also want to add a column to the output in which the value will be based on what is listed in a particular cell or a particular column.
How would I go about this? (*very new sas user)
Without hearing more information, I assume what you're trying to do is save the output of your proc freq and then manipulate it further with a data step.
Simple example of this:
data beer;
length firstname favbrand $20.;
input firstname $
favbrand $;
datalines;
John bud
Steve dogfishhead
Jason coors
Anna anchorsteam
Bob bud
Dan bud
;
run;
proc freq data=beer;
table favbrand / out=freqout;
run;
data beerstat(keep=favbrand status);
set freqout;
* create a new column called "status" based on the count column ;
if (count >=2) then status="popular";
else status = "hipster";
run;
* instead of proc print you can send your output to excel with proc export ;
proc print data=beerstat;
run;