I have two data sets having the same content but one is in tab-delimited format, and the other is in space-delimited format.
Space-Delimited
Tab_Delimited
I have three questions which I could not figure them out and would like to ask for help. Any suggestions would be highly appreciated.
First, I used the TextWrangler to open these two data sets, and I feel that the space-delimited data set means that the data sets are separated by spaces and the observations each row are in the same position.
On the other hand, my understanding for tab-delimited data set was that the data sets which are separated by blanks and the blanks might not be necessary the same widths for each rows of the variables. Was my understanding correct? I am having trouble distinguishing them.
Second, I was printing out the snowfall dataset as mentioned above from row number 5 to row number 122, and the "T" values in the dataset has to
be converted to 0.
My code for the space-delimited file of the snowfall data was as below,
and my question was about its LOG. There were many warnings about "T" but I did not receive any errors.
LOG
Should I be concerned about the warnings here mentioning
"invalid data for month(i) in line..."
* Trying Space-Delimited data set;
OPTIONS Errors=200;
DATA SASWEEK.SnowSpace;
DROP i MyTot diff;
INFILE "&dirLSB.RochesterSnowfallSpace.txt" FIRSTOBS= 2 OBS= 122;
INPUT Season $ Sep Oct Nov Dec Jan Feb Mar Apr May Total ;
ARRAY Month(10) Sep -- Total;
DO i = 1 TO 10 ;
IF Month(i) = . THEN Month(i) = 0 ;
MyTot = sum (of Sep -- May);
diff = round (MyTot-Total, 3);
IF diff ne 0 THEN PUT "**ERROR" MyTot= Total= diff= ;
END;
PROC PRINT DATA=sasweek.snowspace;
TITLE "Rochester Snowfall in Space-Delimited format";
RUN;
One of my professors suggested I should have made the monthly snowfall as "character". So the "T"s would not incur a warning in the LOG. I am not sure whether I should try it this way.
Lastly, I tried to use "Proc Import" for the same data set but in xls file.
The data set is as the link
And my code is as follows:
* Trying Excel file ;
OPTIONS ERRORS=200;
OPTIONS MSGLEVEL=i;
PROC IMPORT OUT=SASWEEK.SNOWxls
DATAFILE= "&dirLSB.RochesterSnowfall.xls" DBMS=xls;
GETNAMES= no;
RANGE= "Sheet1$a5:k122" ;
PROC PRINT DATA= SASWEEK.SNOWxls;
TITLE "Rochester Snowfall in xls format";
RUN;
I received the error in the LOG saved as the HTML
I still printed out a part of the dataset but the variable names were messed up and the output was not complete.
Any ideas?
Thank you all for your reading and thanks for any help:)
The DATA step with INPUT statement might be the best place to start.
WARNINGs are fine, unless the goal is to have no warnings.
The data file can be cleanly read by creating an input environment built for it:
Custom informat zeroT converts T(text) to 0(number). Prevents warnings.
INFILE
DLM='0920'x specifying either tab or space may be delimiting data file values.
INPUT
Wrap fields Sep to Total in parenthesis ( ) to indicate grouped input
Wrap informat specifiers in parenthesis ( ) that are applied over grouped variables
: list input modifier that advances input parsing to next non-blank and reads until next character is blank.
Sample Code
proc format;
invalue zeroT 'T'=0 other=[best12.];
run;
data have;
infile snowdata firstobs=2 dlm='0920'x;
INPUT Season $ (Sep Oct Nov Dec Jan Feb Mar Apr May Total) (10 * :zeroT.) ;
run;
Sample Data (from SP text viewer)
filename snowdata "%TEMP%\roc_snowfalls.txt";
* create local sample data file, text copied from sharepoint viewer;
data _null_;
file snowdata;
input;
put _infile_;
datalines;
Season Sep Oct Nov Dec Jan Feb Mar Apr May Total
1884-85 0 T 1 27.1 22.2 17 3.5 19.5 T 90.3
1885-86 0 1.7 8.2 8.4 16.9 16 6.5 7 0 64.7
1886-87 0 T 22.2 12.5 12 18.4 6.3 1.2 0 72.6
1887-88 0 0.2 2.2 9.3 21.3 4.1 13.2 0.4 0 50.7
1888-89 0 T 4 15.5 17.8 22 17.5 5.4 0 82.2
1889-90 0 T 5.7 6.1 20.2 14.8 19 T 0 65.8
1890-91 0 0 2.1 29.2 16.1 24.6 12.2 0.3 0.1 84.6
1891-92 0 0.1 9.7 4.7 26.4 10.3 25.1 0.8 T 77.1
1892-93 0 T 14 19.2 15.9 29.8 8.1 9.6 0 96.6
1893-94 0 0.5 6.1 27.6 20 29.5 5.4 13.3 0 102.4
1894-95 0 T 11.1 22.1 26.5 23.6 9.5 0.6 0 93.4
1895-96 0 1.5 5.9 8.7 22.5 39.1 45.1 1 0 123.8
1896-97 0 T 5.5 13.9 20.1 13.7 8.1 5.2 0 66.5
1897-98 0 0 10.1 18.4 32.1 26.8 1.2 2.4 0 91
1898-99 0 T 10.6 27 16.6 16.3 21.2 4.3 T 96
1899-00 T T 1.3 21.5 24.7 28.5 54 1.3 0 131.3
1900-01 0 0 17 20.3 29.8 36.9 13.7 23.8 T 141.5
1901-02 0 0.1 14.1 14.5 23.8 23 1.2 2.3 T 79
1902-03 0 0.1 4.1 27.7 18.1 15.6 2.4 0.3 0 68.3
1903-04 0 0.6 4.4 16.1 27.2 17.2 10.7 19.5 T 95.7
1904-05 0 0.2 2.1 15.8 27.5 15.2 7 0.5 0 68.3
1905-06 0 T 4 8.4 7.6 8 15.2 1.1 0 44.3
1906-07 0 5 5.7 18.7 11.7 15.7 3.1 2.5 1.3 63.7
1907-08 0 0 2.2 11.6 16.5 19.8 7.9 6.3 3 67.3
1908-09 0 0.5 4.6 10 22.5 6.1 9.7 9.8 3.3 66.5
1909-10 0 T 1.7 14.6 22 42.7 3.4 0.5 0 84.9
1910-11 0 2.2 15.7 29.8 9.5 30 13.5 4.7 2 107.4
1911-12 0 0 6.5 7.5 21.5 10.8 8.8 6.9 T 62
1912-13 0 0 7.2 6.9 10 18.6 15.2 1.3 0 59.2
1913-14 0 0.2 0.3 14.4 15.1 21.6 27.9 7.2 0 86.7
1914-15 0 0.8 4.7 16.1 22.9 9.8 6 0.5 0 60.8
1915-16 0 0 3.4 14.8 8.5 35.7 43.8 0.7 0 106.9
1916-17 0 0 11.7 24.9 22.7 16.7 14.6 2.3 T 92.9
1917-18 0 T 7.9 29.7 17.2 12.7 10.5 1.3 0 79.3
run;
I am trying to scrape time series data using pandas DataFrame for Python 2.7 from the web page (http://owww.met.hu/eghajlat/eghajlati_adatsorok/bp/Navig/202_EN.htm). Could somebody please help me how I can write the code. Thanks!
I tried my code as follows:
html =urllib.urlopen("http://owww.met.hu/eghajlat/eghajlati_adatsorok/bp/Navig/202_EN.htm");
text= html.read();
df=pd.DataFrame(index=datum, columns=['m_ta','m_tax','m_taxd', 'm_tan','m_tand'])
But it doesn't give anything. Here I want to display the table as it is.
You can use BeautifulSoup for parsing all font tags, then split column a, set_index from column idx and rename_axis to None - remove index name:
import pandas as pd
import urllib
from bs4 import BeautifulSoup
html = urllib.urlopen("http://owww.met.hu/eghajlat/eghajlati_adatsorok/bp/Navig/202_EN.htm");
soup = BeautifulSoup(html)
#print soup
fontTags = soup.findAll('font')
#print fontTags
#get text from tags fonts
li = [x.text for x in soup.findAll('font')]
#remove first 13 tags, before not contain necessary data
df = pd.DataFrame(li[13:], columns=['a'])
#split data by arbitrary whitspace
df = df.a.str.split(r'\s+', expand=True)
#set column names
df.columns = columns=['idx','m_ta','m_tax','m_taxd', 'm_tan','m_tand']
#convert column idx to period
df['idx'] = pd.to_datetime(df['idx']).dt.to_period('M')
#convert columns to datetime
df['m_taxd'] = pd.to_datetime(df['m_taxd'])
df['m_tand'] = pd.to_datetime(df['m_tand'])
#set column idx to index, remove index name
df = df.set_index('idx').rename_axis(None)
print df
m_ta m_tax m_taxd m_tan m_tand
1901-01 -4.7 5.0 1901-01-23 -12.2 1901-01-10
1901-02 -2.1 3.5 1901-02-06 -7.9 1901-02-15
1901-03 5.8 13.5 1901-03-20 0.6 1901-03-01
1901-04 11.6 18.2 1901-04-10 7.4 1901-04-23
1901-05 16.8 22.5 1901-05-31 12.2 1901-05-05
1901-06 21.0 24.8 1901-06-03 14.6 1901-06-17
1901-07 22.4 27.4 1901-07-30 16.9 1901-07-04
1901-08 20.7 25.9 1901-08-01 14.7 1901-08-29
1901-09 15.9 19.9 1901-09-01 11.8 1901-09-09
1901-10 12.6 17.9 1901-10-04 8.3 1901-10-31
1901-11 4.7 11.1 1901-11-14 -0.2 1901-11-26
1901-12 4.2 8.4 1901-12-22 -1.4 1901-12-07
1902-01 3.4 7.5 1902-01-25 -2.2 1902-01-15
1902-02 2.8 6.6 1902-02-09 -2.8 1902-02-06
1902-03 5.3 13.3 1902-03-22 -3.5 1902-03-13
1902-04 10.5 15.8 1902-04-21 6.1 1902-04-08
1902-05 12.5 20.6 1902-05-31 8.5 1902-05-10
1902-06 18.5 23.8 1902-06-30 14.4 1902-06-19
1902-07 20.2 25.2 1902-07-01 15.5 1902-07-03
1902-08 21.1 25.4 1902-08-07 14.7 1902-08-13
1902-09 16.1 23.8 1902-09-05 9.5 1902-09-24
1902-10 10.8 15.4 1902-10-12 4.9 1902-10-25
1902-11 2.4 9.1 1902-11-01 -4.2 1902-11-18
1902-12 -3.1 7.2 1902-12-27 -17.6 1902-12-15
1903-01 -0.5 8.3 1903-01-11 -11.5 1903-01-23
1903-02 4.6 13.4 1903-02-23 -2.7 1903-02-17
1903-03 9.0 16.1 1903-03-28 4.9 1903-03-09
1903-04 9.0 16.5 1903-04-29 2.6 1903-04-19
1903-05 16.4 21.2 1903-05-03 11.3 1903-05-19
1903-06 19.0 23.1 1903-06-03 15.6 1903-06-07
... ... ... ... ... ...
1998-07 22.5 30.7 1998-07-23 15.0 1998-07-09
1998-08 22.3 30.5 1998-08-03 14.8 1998-08-29
1998-09 16.0 21.0 1998-09-12 10.4 1998-09-14
1998-10 11.9 17.2 1998-10-07 8.2 1998-10-27
1998-11 3.8 8.4 1998-11-05 -1.6 1998-11-21
1998-12 -1.6 6.2 1998-12-14 -8.2 1998-12-26
1999-01 0.6 4.7 1999-01-15 -4.8 1999-01-31
1999-02 1.5 6.9 1999-02-05 -4.8 1999-02-01
1999-03 8.2 15.5 1999-03-31 3.0 1999-03-16
1999-04 13.1 17.1 1999-04-16 6.1 1999-04-18
1999-05 17.2 25.2 1999-05-31 11.1 1999-05-06
1999-06 19.8 24.4 1999-06-07 12.2 1999-06-22
1999-07 22.3 28.0 1999-07-06 16.3 1999-07-23
1999-08 20.6 26.7 1999-08-09 17.3 1999-08-23
1999-09 19.3 22.9 1999-09-26 15.0 1999-09-02
1999-10 11.5 19.0 1999-10-03 5.7 1999-10-18
1999-11 3.9 12.6 1999-11-04 -2.2 1999-11-21
1999-12 1.3 6.4 1999-12-13 -8.1 1999-12-25
2000-01 -0.7 8.7 2000-01-31 -6.6 2000-01-25
2000-02 4.5 10.2 2000-02-01 -0.1 2000-02-23
2000-03 6.7 11.6 2000-03-09 0.6 2000-03-17
2000-04 14.8 22.1 2000-04-21 5.8 2000-04-09
2000-05 18.7 23.9 2000-05-27 12.3 2000-05-22
2000-06 21.9 29.3 2000-06-14 15.4 2000-06-17
2000-07 20.3 26.6 2000-07-03 14.0 2000-07-16
2000-08 23.8 29.7 2000-08-20 18.5 2000-08-31
2000-09 16.1 21.5 2000-09-14 12.7 2000-09-24
2000-10 14.1 18.7 2000-10-04 8.0 2000-10-23
2000-11 9.0 14.9 2000-11-15 3.7 2000-11-30
2000-12 3.0 9.4 2000-12-14 -6.8 2000-12-24
[1200 rows x 5 columns]
i got this data frame
x1 x2 x3
1 2.5 2.8 1.4
2 2.1 1.9 2.3
3 1.7 2.2 4.4
4 2.4 3.8 3.7
5 4.3 4.4 4.1
6 4.2 4.9 2.4
7 2.7 1.5 2.5
8 2.8 3.3 4.9
9 3.5 2.3 2.9
10 4.1 2.8 2.2
so i need to check for every row a condition and apply a function to this row so that the value of this function would be in the fourth column or in the external vector. i.e. if min_value_of_row < thrshld then min(row) else mean(row)
How would one do that?
A bit late, but I was looking for something similar. Firstly I would create two columns with min and mean values of each row with:
df['min'] = df.min(axis=1)
and
df['mean'] = df.mean(axis=1)
then build a function:
def f(x):
thr = 2
if x['min'] <= thr:
x = x['min']
else:
x = x['mean']
return x
and apply it to the dataframe row-wise (axis=1):
df['value'] = df.apply(f, axis=1)
this returns:
x1 x2 x3 value
1 2.5 2.8 1.4 1.400
2 2.1 1.9 2.3 1.900
3 1.7 2.2 4.4 1.700
4 2.4 3.8 3.7 3.075
5 4.3 4.4 4.1 4.225
6 4.2 4.9 2.4 3.475
7 2.7 1.5 2.5 1.500
8 2.8 3.3 4.9 3.450
9 3.5 2.3 2.9 2.750
10 4.1 2.8 2.2 2.825
I have two data frames as such:
df1 = pd.DataFrame({ 'pressure' : [42,42,42,42,42,42,42,36,36,36,36,36,36,36],
'load' : [350,350,350,350,350,350,350,700,700,700,700,700,700,700],
'speed' : [70,60,50,40,30,20,10,70,60,50,40,30,20,10],
'lforce' : [3.6,3.5,3.3,3.2,3.1,3.1,2.9,7.7,7.3,7.0,6.8,6.5,6.4,6.1],
'rforce' : [3.4,3.2,3.1,3.0,2.9,2.8,2.7,7.6,7.2,6.9,6.6,6.3,6.2,5.9]
}).set_index(['pressure','load','speed'])
df2 = pd.DataFrame({ 'pressure' : [47,47,47,47,47,47,47],
'load' : [20,20,20,20,20,20,20],
'speed' : [70,60,50,40,30,20,10],
'lforce' : [2.5,2.1,1.9,1.7,1.5,1.3,1.2],
'rforce' : [2.8,2.6,2.4,2.2,2.0,1.8,1.7]
}).set_index(['pressure','load','speed'])
Formatted:
>>> df1
lforce rforce
pressure load speed
42 350 70 3.6 3.4
60 3.5 3.2
50 3.3 3.1
40 3.2 3.0
30 3.1 2.9
20 3.1 2.8
10 2.9 2.7
36 700 70 7.7 7.6
60 7.3 7.2
50 7.0 6.9
40 6.8 6.6
30 6.5 6.3
20 6.4 6.2
10 6.1 5.9
>>> df2
lforce rforce
pressure load speed
47 20 70 2.5 2.8
60 2.1 2.6
50 1.9 2.4
40 1.7 2.2
30 1.5 2.0
20 1.3 1.8
10 1.2 1.7
I would like to subtract df2 from df1 on the lforce and rforce columns for each speed to get the resulting data frame df3.
My problem is that I need to ignore the pressure and load in df2 during the subtraction, but retain the originals from df1.
Desired result:
>>> df3
lforce rforce
pressure load speed
42 350 70 1.1 0.6
60 1.3 0.6
50 1.4 0.7
40 1.5 0.8
30 1.6 0.9
20 1.7 1.0
10 1.7 1.0
36 700 70 5.2 4.8
60 5.1 4.6
50 5.1 4.4
40 5.1 4.4
30 5.0 4.3
20 5.0 4.3
10 4.9 4.2
df1.sub(df2.reset_index([0, 1], drop=True), level=2)
output:
lforce rforce
pressure load speed
42 350 70 1.1 0.6
60 1.4 0.6
50 1.4 0.7
40 1.5 0.8
30 1.6 0.9
20 1.8 1.0
10 1.7 1.0
36 700 70 5.2 4.8
60 5.2 4.6
50 5.1 4.5
40 5.1 4.4
30 5.0 4.3
20 5.1 4.4
10 4.9 4.2
May be somehing like this:
>>> df3 = df1.reset_index(level=[0,1])
>>> df4 = df2.reset_index(level=[0,1])
>>> df4['pressure'] = 0
>>> df4['load'] = 0
>>> df3 - df4
pressure load lforce rforce
speed
10 42 350 1.7 1.0
10 36 700 4.9 4.2
20 42 350 1.8 1.0
20 36 700 5.1 4.4
30 42 350 1.6 0.9
30 36 700 5.0 4.3
40 42 350 1.5 0.8
40 36 700 5.1 4.4
50 42 350 1.4 0.7
50 36 700 5.1 4.5
60 42 350 1.4 0.6
60 36 700 5.2 4.6
70 42 350 1.1 0.6
70 36 700 5.2 4.8
Now you just have to move pressure and load back to index
Is this what you're looking for?
d1 = df1.reset_index(['pressure','load'])
d2 = df2.reset_index(['pressure','load'])
r0 = d1.merge(d2, left_index=True, right_index=True)
r1 = r0.set_index(['pressure_x','load_x'], drop=False)
r1['lforce'] = r1.lforce_x - r1.lforce_y
r1['rforce'] = r1.rforce_x - r1.rforce_y
df3 = r1[['lforce','rforce']]
df3
How do I read a data file one row at a time in SAS?
Say, I have 3 lines of data
1.0 3.0 5.6 7.8
2.3 4.9
3.2 5.3 6.8 7.5 3.9 4.1
I have to read each line in a different variable. I want the data to look like.
A 1.0
A 3.0
A 5.6
A 7.8
B 2.3
B 4.9
C 3.2
C 5.3
C 6.8
C 7.5
C 3.9
C 4.1
I tried a bunch of things.
If it has a variable name before every data point, following code works fine
INPUT group $ x ##;
I can't figure out how to go about this. Can someone please guide me on this?
Thanks
i think this will produce almost exactly the result you want. you could apply a format to the Group variable.
data orig;
infile datalines missover pad;
format Group 4. Value 4.1;
Group = _n_;
do until (Value eq .);
input value #;
if value ne . then output;
else return;
end;
datalines;
1.0 3.0 5.6 7.8
2.3 4.9
3.2 5.3 6.8 7.5 3.9 4.1
run;
proc print; run;
/*
Obs Group Value
1 1 1.0
2 1 3.0
3 1 5.6
4 1 7.8
5 2 2.3
6 2 4.9
7 3 3.2
8 3 5.3
9 3 6.8
10 3 7.5
11 3 3.9
12 3 4.1 */