Multiple To clauses in Data step - sas

I have a data step where I have a few columns that need tied to one other column.
I have tried using multiple "from" statements and " to" statements and a couple other permutations of that, but nothing seems to do the trick. The code looks something like this:
data analyze;
set css_email_analysis;
from = bill_account_number;
to = customer_number;
output;
from = bill_account_number;
to = email_addr;
output;
from = bill_account_number;
to = e_customer_nm;
output;
run;
I would like to see two columns showing bill accounts in the "from" column, and the other values in the "to", but instead I get a bill account and its customer number, with some "..."'s for the other values.

Issue
This is most likely because SAS has two datatypes and the first time the to variable is set up, it has the value of customer_number. At your second to statement you attempt to set to to have the value of email_addr. Assuming email_addr is a character variable, two things can happen here:
Customer_number is a number - to has already been set up as a number, so SAS cannot force to to become a character, an error like this may appear:
NOTE: Invalid numeric data, 'me#mywebsite.com' , at line 15 column 8. to=.
ERROR=1 N=1
Customer_number is a character - to has been set up as a character, but without explicitly defining its length, if it happens to be shorter than the value of email_addr then the email address will be truncated. SAS will not show an error if this happens:
Code:
data _NULL_;
to = 'hiya';
to = 'me#mydomain.com';
put to=;
run;
short=me#m
to is set with a length of 4, and SAS does not expand it to fit the new data.
Detail
The thing to bear in mind here is how SAS works behind the scenes.
The data statement sets up an output location
The set statement adds the variables from first observation of the dataset specified to a space in memory called the PDV, inheriting lengths and data types.
PDV:
bill_account_number|customer_number|email_addr|e_customer_nm
===================================================================
010101 | 758|me#my.com |John Smith
The to statement adds another variable inheriting the characteristics of customer_number
PDV:
bill_account_number|customer_number|email_addr|e_customer_nm|to
===================================================================
010101 | 758|me#my.com |John Smith |758
(to is either char length 3 or a numeric)
Subsequent to statements will not alter the characteristics of the variable and SAS will continue processing
PDV (if customer_number is character = TRUNCATION):
bill_account_number|customer_number|email_addr|e_customer_nm|to
===================================================================
010101 | 758|me#my.com |John Smith |me#
PDV (if customer_number is numeric = DATA ERROR, to set to missing):
bill_account_number|customer_number|email_addr|e_customer_nm|to
===================================================================
010101 | 758|me#my.com |John Smith |.
Resolution
To resolve this issue it's probably easiest to set the length and type of to before your first to statement:
data analyze;
set css_email_analysis;
from = bill_account_number;
length to $200;
to = customer_number;
output;
...
You may get messages like this, where SAS has converted data on your behalf:
NOTE: Numeric values have been converted to character
values at the places given by: (Line):(Column).
27:8
N.B. it's not necessary to explicitly define the length and type of from, because as far as I can see, you only ever get the values for this variable from one variable in the source dataset. You could also achieve this with a rename if you don't need to keep the bill_account_number variable:
rename bill_account_number = from;

Related

How to convert text field with formatted currency to numeric field type in Postgres?

I have a table that has a text field which has formatted strings that represent money.
For example, it will have values like this, but also have "bad" invalid data as well
$5.55
$100050.44
over 10,000
$550
my money
570.00
I want to convert this to a numeric field but maintain the actual numbers that can be retained, and for any that can't , convert to null.
I was using this function originally which did convert clean numbers (numbers that didn't have any formatting). The issue was that it would not convert $5.55 as an example and set this to null.
CREATE OR REPLACE FUNCTION public.cast_text_to_numeric(
v_input text)
RETURNS numeric
LANGUAGE 'plpgsql'
COST 100
VOLATILE
AS $BODY$
declare v_output numeric default null;
begin
begin
v_output := v_input::numeric;
exception when others then return null;
end;
return v_output;
end;
$BODY$;
I then created a simple update statement which removes the all non digit characters, but keeps the period.
update public.numbertesting set field_1=regexp_replace(field_1,'[^\w.]','','g')
and if I run this statement, it correctly converts the text data to numeric and maintains the number:
alter table public.numbertesting
alter column field_1 type numeric
using field_1::numeric
But I need to use the function in order to properly discard any bad data and set those values to null.
Even after I run the clean up to set the text value to say 5.55
my "cast_text_to_numeric" function STILL sets this to null ? I don't understand why this sets it to null, but the above statement correctly converts it to a proper number.
How can I fix my cast_text_to_numeric function to properly convert values such as 5.55 , etc?
I'm ok with disgarding (setting to NULL) any values that don't end up with numbers and a period. The regular expression will strip out all other characters... and if there happens to be two numbers in the text field, with the script, they would be combined into one (spaces are removed) and I'm good with that.
In the example of data above, after conversion, the end result in numeric field would be:
5.55
100050.44
null
550
null
570.00
FYI, I am on Postgres 11 right now

SAS format char

First i have created this table
data rmlib.tableXML;
input XMLCol1 $ 1-10 XMLCol2 $ 11-20 XMLCol3 $ 21-30 XMLCol4 $ 31-40 XMLCol5 $ 41-50 XMLCol6 $ 51-60;
datalines;
| AAAAA A||AABAAAAA|| BAAAAA|| AAAAAA||AAAAAAA ||AAAA |
;
run;
I want to clean, concatenate and export. I have written the following code
data rmlib.tableXML_LARGO;
file CleanXML lrecl=90000;
set rmlib.tableXML;
array XMLCol{6} ;
array bits{6};
array sqlvars{6};
do i = 1 to 6;
*bits{i}=%largo(XMLCol{i})-2;
%let bits =input(%largo(XMLCol{i})-2,comma16.5);
sqlvars{i} = substr(XMLCol{i},2,&bits.);
put sqlvars{i} &char10.. #;
end;
run;
the macro largo count how many characters i have
%macro largo(num);
length(put(&num.,32500.))
%mend;
What i need is instead of have char10, i would like that this number(10) would be the length, of each string, so to have something like
put sqlvars{i} &char&bits.. #;
I don't know if it possible but i can't do it.
I would like to see something like
AAAAA AAABAAAAA BAAAAA AAAAAAAAAAAAA AAAA
It is important to me to keep the spaces(this is only an example of an extract of a xml extract). In addition I will change (for example) "B" for "XPM", so the size will change after cleaning the text, that it what i need to be flexible in the char
Thank you for your time
Julen
I'm still not quite sure what you want to achieve, but if you want to combine the text from multiple varriables into one variable, then you could do something along the lines:
proc sql;
select name into :names separated by '||'
from dictionary.columns
where 1=1
and upcase(libname)='YOURLIBNAME'
and upcase(memname)='YOURTABLENAME';
quit;
data work.testing;
length resultvar $ 32000;
set YOURLIBNAME.YOURTABLENAME;
resultvar = &names;
resultvar2 = compress(resultvar,'|');
run;
Wasn't able to test this, but this should work if you replace YOURLIBNAME and YOURTABLENAME with your respective tables. I'm not 100% sure if the compress will preserve the spaces in the text.. But I think it should.
The format $VARYING. <length-variable> is a good candidate for solving this output problem.
On the presumption of having a number of variables whose values are vertical-bar bounded and wanting to output to a file the concatenation of the values without the bounding bars.
data have;
file "c:\temp\want.txt" lrecl=9000;
length xmlcol1-xmlcol6 $100;
array values xmlcol1-xmlcol6 ;
xmlcol1 = '| A |';
xmlcol2 = '|A BB|';
xmlcol3 = '|A BB|';
xmlcol4 = '|A BBXC|';
xmlcol5 = '|DD |';
xmlcol6 = '| ZZZ |';
do index = 1 to dim(values);
value = substr(values[index], 2); * ignore presumed opening vertical bar;
value_length = length(value)-1; * length with still presumed closing vertical bar excluded;
put value $varying. value_length #; * send to file the value excluding the presumed closing vertical bar;
end;
run;
You have some coding errors in that is making it difficult to understand what you want to do.
Your %largo() macro doesn't make any sense. There is no format 32500.. The only reason it would run in your code is because you are trying to apply the format to a character variable instead of a number. So SAS will automatically convert to use the $32500. instead.
The %LET statement that you have hidden in the middle of your data step will execute BEFORE the data step runs. So it would be less confusing to move it before the data step.
So replacing the call to %largo() your macro variable BITS will contain this text.
%let bits =input(length(put(XMLCol{i},32500.))-2,comma16.5);
Which you then use inside a line of code. So that line will end up being this SAS code.
sqlvars{i} = substr(XMLCol{i},2,input(length(put(XMLCol{i},$32500.))-2,comma16.5));
Which seems to me to be a really roundabout way to do this:
sqlvars{i} = substr(XMLCol{i},2,length(XMLCol{i})-2);
Since SAS stores character variables as fixed length, it will pad the value stored. So what you need to do is to remember the length so that you can use it later when you write out the value. So perhaps you should just create another array of numeric variables where you can store the lengths.
sqllen{i} = length(XMLCol{i})-2;
sqlvars{i} = substr(XMLCol{i},2,sqllen{i});

How to use proc format with the number of lines?

I have a table like this :
|Num | Label
-----------------------
1|1 | a thing
2|2 | another thing
3|3 | something else
4|4 | whatever
I want to replace my values of my label column by something more generic for example the first two lines : label One, the two next ones label Two ...
|Num | Label
-----------------------
1|1 | label One
2|2 | label One
3|3 | label Two
4|4 | label Two
How can I do that using proc format procedure ? I was wondering if I can use either the number of lines or another column like Num.
I need to do something like this :
proc format;
value label_f
low-2 = "label One"
3-high = "label Two"
;
run;
But I want to specify the number of the line or the value of the Num column.
You could do what you are describing using the words format. You could swap out num for _N_ in the ceil function below in order to use the observation number instead of the value of num (if they are not always equal):
data have;
length num 8 label $20;
infile datalines dlm='|';
input num label $;
datalines;
1|a thing
2|another thing
3|something else
4|whatever
5|whatever else
6|so many things
;
run;
data want;
set have;
label=catx(' ','label',propcase(put(ceil(num/2),words.)));
run;
Although this answer is probably a bit too specific to your example and it may not apply in your actual context.
Gatsby:
It sounds like you want to format NUM instead of LABEL.
Where you want the use the 'generic' representation defined by your format simply place a FORMAT statement in the Proc being used:
PROC PRINT data=have;
format num label_f.;
RUN;
If you want both num and generic, you will need to add a new column to the data for use during processing. This can be done with a view:
data have_view / view=have_view;
set have;
num_replicate1 = num;
attrib num_replicate1 format=label_f. label='Generic';
num_replacement = put (num,label_f.);
attrib num_replacement label='Generic'; %* no format because the value is the formatted value of the original num;
run;
PROC PRINT data=have_view;
var num num_replicate1 num_replacement;
RUN;
If you want a the 'generic' representation of the NUM column to be used in by-processing as a grouping variable, you have several scenarios:
know apriori the generic representation is by-group clustered
use a view and process with BY or BY ... NOTSORTED if clusters are not in sort order
force ordering for use with by-group processing
use an ordered SQL view containing the replicate and process with BY
add a replicate variable to the data set, sort by the formatted value and process with BY
A direct backmap from label to num to generic is possible only if the label is known to be unique, or you know apriori the transformation backmap-num + num-map is unique.
Proc FORMAT also has a special value construct [format] that can be used to map different ranges of values according to different formatting rules. The other range can also map to a different format that itself has an other range that maps to yet another different format. The SAS format engine will log an error if you happen to define a recursive loop using this advanced kind of format mapping.
propaedeutics
One of my favorite Dorfman words.
Format does not replace underlying values. Format is a map from the underlying data value to a rendered representation. The map can be 1:1, many:1. The MultiLabel Format (MLF) feature of the format system can even perform 1:many and many:many mappings in procedures many MLF enabled procedures (which is most of them)
To replace an underlying value with it's formatted version you need to use the PUT, PUTC or PUTN functions. The PUT functions always outputs a character value.
character ⇒ PUT ⇒ character [ FILE / PUT ]
numeric ⇒ PUT ⇒ character [ FILE / PUT ]
There is no guarantee a mapped value will mapped to the same value, it depends on the format.
INFORMATs are similar to FORMATs, however the target value depend on the in format type
character ⇒ INPUT ⇒ character [ INFILE / INPUT ]
numeric ⇒ INPUT ⇒ character
character ⇒ INPUT ⇒ numeric [ INFILE / INPUT ]
numeric ⇒ INPUT ⇒ numeric
Custom formats are created with Proc FORMAT. The construction of a format is specified by either the VALUE statement, or the CNTLIN= option. CNTLIN lets you create formats directly from data and avoids really large VALUE statements that are hand-entered or code-generated (via say macro)
Data-centric 'formatting' performs the mapping through a left-join. This is prevalent in SQL data bases. Left-joins in SAS can be done through SQL, DATA Step MERGE BY and FORMAT application. 1:1 left-joins can also be done via Hash object SET POINT=

How to add elements to datalines dynamically

I am trying to create Dataset using the Dataline option based on the data that the user input. Is there a way to add the values in Dataline dynamically in stored process? If not how do I go about doing this?
EDIT: I am getting input from user as an array of numbers. I want to add few more fields to form my dataset. So in short, the dataset i am trying to create is a combination of array elements from the user and some more data based on these input numbers.
User inputs: 1234, 2345, 3456
Dataset:
number | text | id
1234 | "Something 1" | 1
2345 | "Something 2" | 2
3456 | "Something 3" | 3
Datalines/cards shouldn't be used in any sort of production system. Among other things, they're illegal in an %include, which is often used in production systems.
I believe the default in a stored procedure is to return macro variables (based on the name given to the field in the input form). Either directly assign that to a dataset variable, or if it's a lot of text that you want to parse, write the macro variable out to a text file and read it in. More information for the proper way to handle this for your case might be available with more information in the question.
Given the edits, and assuming you get the value 1234, 2345, 3456 in a macro variable &number, you can do something like this:
data want;
_numvar = "&number.";
do _t = 1 to countc(_numvar,",")+1;
number = scan(_numvar,_t,",");
text = catx(" ","Something",substr(number,1,1);
id = input(substr(number,1,1));
output;
keep number text id;
end;
run;
I don't know how you're constructing text and id so I just made something up there.

SAS infile messy format of variable lengths

I have a messy file, where some of the columns are tab delimitated and some are comma.
My problem with the data set is reading the files with variable lengths
12 Stephen Cole, 33, Columbia, MO
5 Dave Anderson, 25*, Concord, OH
The first column is a ID (tab) the the name (comma) age (comma), active (presence of an asterisk after age), home (tab)
The * after the age indicates if they are inactive.
All the names start at column #19, but everything after that is variable lengths and column starts.
I want to read into a format where I finally get.
ID Name Age Active Home
12 Stephen Cole 33 Active Columbia, MO
5 Dave Anderson 25 Inactive Concord, OH
Thus far I have:
data marathon;
infile 'c:/file.txt' dlm=',' pad firstobs=12;
input #3 ID 3. #19 Name $CHAR13.;
Then I get stuck on how to read the rest. I am mostly thrown with how to read the asterisk next to the age as its own column. If I had that understood, I think I can handle the rest.
You have a couple of issues. First, you need to use delimited input, specifically you need to combine comma and tab into one set of delimiters - one way is shown below. Second, you have two fields that are nontrivial; the one with the asterisk needs to be parsed afterwards (I use compress to keep specifically digits in the first line, and to keep specifically asterisks in the second line). You also need to read city/state in separate fields and combine them together (I use catx).
data want;
infile "c:\temp\test.dat" dlm='092C'x;
input
id
name :$50.
age_active $
home_city :$25.
home_st $
;
age=input(compress(age_active,,'kd'),best.);
active = ifc(compress(age_active,'*','k')='*','Active','Inactive');
home = catx(', ',home_city,home_st);
run;
Watch your lengths, I suggest reasonable ones given my past experience but you could see longer names or cities easily.