I have a data set that looks as follows in a CSV file:
Date Sample
01-AUG-09 Sample 1
02-Aug-09 Sample 2
etc...
When I use Pandas, I read in the file with the following code:
in_file = pd.read_csv('File Name.csv', parse_dates = True)
However, it is not recognizing the date column properly. Does anybody know if the Pandas date parser can recognize dates that are in DD-MMM-YY format?
The following worked for me
I suspect yours is probably much simpler to parse because they are many tab separated? (I did an exact width parsing which is not trivial)
In [41]: df = pd.read_fwf(StringIO(data),widths=[9,13],parse_dates=True,index_col=0,names=['sample'],header=None,skiprows=1)
In [42]: df
Out[42]:
sample
2009-08-01 Sample 1
2009-08-02 Sample 2
Tab separated is much simpler
In [43]: data2 = """Data\tSample\n01-AUG-09\tSample 1\n02-Aug-09\tSample 2\n"""
In [44]: read_csv(StringIO(data2),sep='\t',parse_dates=True,index_col=0)
Out[44]:
Sample
Data
2009-08-01 Sample 1
2009-08-02 Sample 2
Related
I am new to python. I have a csv file which has cleaned tweets. I want to create a bag of words of these tweets.
I have the following code but its not working correctly.
import pandas as pd
from sklearn import svm
from sklearn.feature_extraction.text import CountVectorizer
data = pd.read_csv(open("Twidb11.csv"), sep=' ')
count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(data.Text)
count_vect.vocabulary_
Error:
.ParserError: Error tokenizing data. C error: Expected 19 fields in
line 5, saw 22
It's duplicated i think. U can see answer here. There are a lot of answers and comments.
So, solution can be:
data = pd.read_csv('Twidb11.csv', error_bad_lines=False)
Or:
df = pandas.read_csv(fileName, sep='delimiter', header=None)
"In the code above, sep defines your delimiter and header=None tells pandas that your source data has no row for headers / column titles. Thus saith the docs: "If file contains no header row, then you should explicitly pass header=None". In this instance, pandas automatically creates whole-number indeces for each field {0,1,2,...}."
I have seen this question asked on Stack Overflow multiple times before, however, what I have not seen is anyone either ask nor answer the question properly here. So, here is my question: I have a .csv file named selected.csv, with three columns - Date (LT), AQI and Raw Conc. The date is in the format dd-mm-yyyy hh:mm. I want to convert the format of the date to yyyy-mm-dd, thereafter, saving the corrected data with three columns - Date, AQI and Raw Conc. as corrected.csv. I have tried the code typed below, but to no avail.
import csv
from datetime import datetime
output_file = open(r"C:\Users\Win-8.1\Desktop\delhi\corrected.csv", "wb")
fieldnames = ['Date', 'AQI' , 'Raw Conc.']
writer = csv.DictWriter(output_file, fieldnames = fieldnames)
writer.writeheader()
with open(r"C:\Users\Win-8.1\Desktop\delhi\selected.csv") as csvfile:
reader = csv.DictReader(csvfile)
for row in reader:
output_row = {}
output_row['Date'] = datetime.datetime.strptime(row['Date (LT)'], '%d-%m-%Y %H:%M').strftime('%Y-%m-%d')
output_row['AQI'] = row['AQI']
output_row['Raw Conc.'] = row['Raw Conc.']
writer.writerow(output_row)
output_file.close()
output_row['Date'] = datetime.strptime(row['Date (LT)'], '%d-%m-%Y %H:%M').strftime('%Y-%m-%d')
You are using also %H:%M when you mentioned the format is DD-MM-YYYY. Maybe you didn't intend this?
Say I have a csv file that looks like the following with the first column containing frequencies and the second column containing the power level (dBm).
Frequency | dBm
1 -11.43
2.3 -51.32
2.5 -12.11
2.8 -11.21
3.1 -73.22
3.2 -21.13
I only want to read in the data sets of this file that have a (dBm) value between -13 and -10. Therefore, in this example I only want the data sets (1, -11.43)(2.5, -12.11)(2.8, -11.21) to be read into my program variables x1 and y1. Could someone give me some help in how I could do this?
You can just use the csv library and check if each meets your criteria.
Something like this should work on your file:
with open('file.csv') as csvfile:
reader = csv.reader(csvfile,delimiter=' ')
reader.next()
reader.next()
for row in reader:
a = [float(i) for i in row if i!='']
if a[1]>=-13 and a[1]<=-1:
print (a[0],a[1])
Edit: If you're working with table data I would suggest trying out Pandas, it's really helpful in these situations.
I am very new to programming and am working with Python. For a work project I am trying to read several .csv files, convert them to data frames, concatenate some of the fields into one for a column header, and then append all of the dataframes into one big DataFrame. I have searched extensively in StackOverflow as well as in other resources but I have not been able to find an answer. Here is the code I have thus far along with some abbreviated output:
import pandas as pd
import glob
# Read a directory of files to a list
csvlist = []
for f in glob.glob("AssayCerts/*"):
csvlist.append(f)
csvlist
['AssayCerts/CH09051590.csv', 'AssayCerts/CH09051591.csv', 'AssayCerts/CH14158806.csv', 'AssayCerts/CH14162453.csv', 'AssayCerts/CH14186004.csv']
# Read .csv files and convert to DataFrames
dflist = []
for csv in csvlist:
df = pd.read_csv(filename, header = None, skiprows = 7)
dflist.append(df)
dflist
[ 0 1 2 3 4 5 \
0 NaN Au-AA23 ME-ICP41 ME-ICP41 ME-ICP41 ME-ICP41
1 SAMPLE Au Ag Al As B
2 DESCRIPTION ppm ppm % ppm ppm
#concatenates the cells in the first three rows of the last dataframe; need to apply this to all of the dataframes.
for df in dflist:
column_names = df.apply(lambda x: str(x[1]) + '-'+str(x[2])+' - '+str(x[0]),axis=0)
column_names
0 SAMPLE-DESCRIPTION - nan
1 Au-ppm - Au-AA23
2 Ag-ppm - ME-ICP41
3 Al-% - ME-ICP41
I am unable to apply the last operation across all of the DataFrames. It seems I can only get it to apply to the last DataFrame in my list. Once I get past this point I will have to append all of the DataFrames to form one large DataFrame.
As Andy Hayden mentions in his comment, the reason your loop only appears to work on the last DataFrame is that you just keep assigning the result of df.apply( ... ) to column_names, which gets written over each time. So at the end of the loop, column_names always contains the results from the last DataFrame in the list.
But you also have some other problems in your code. In the loop that begins for csv in csvlist:, you never actually reference csv - you just reference filename, which doesn't appear to be defined. And dflist just appears to have one DataFrame in it anyway.
As written in your problem, the code doesn't appear to work. I'd advise posting the real code that you're using, and only what's relevant to your problem (i.e. if building csvlist is working for you, then you don't need to show it to us).
Suppose i have tabular column as below.Now i want to extract the column wise data.I tried extracting data by creating a list.But it is extracting the first row correctly but from second row onwards there is space i.e under CEN/4.Now my code considers zeroth column has 5.0001e-1 form second row,it starts reading from there. How to extract the data correctly coulmn wise.output is scrambled.
0 1 25 CEN/4 -5.000000E-01 -3.607026E+04 -5.747796E+03 -8.912796E+02 -88.3178
5.000000E-01 3.607026E+04 5.747796E+03 8.912796E+02 1.6822
27 -5.000000E-01 -3.641444E+04 -5.783247E+03 -8.912796E+02 -88.3347
5.000000E-01 3.641444E+04 5.783247E+03 8.912796E+02 1.6653
28 -5.000000E-01 -3.641444E+04 -5.712346E+03 -8.912796E+02 -88.3386
5.000000E-01 3.641444E+04 5.712346E+03 8.912796E+02
my code is :
f1=open('newdata1.txt','w')
L = []
for index, line in enumerate(open('Trial_1.txt','r')):
#print index
if index < 0: #skip first 5 lines
continue
else:
line =line.split()
L.append('%s\t%s\t %s\n' %(line[0], line[1],line[2]))
f1.writelines(L)
f1.close()
my output looks like this:
0 1 CEN/4 -5.000000E-01 -5.120107E+04
5.000000E-01 5.120107E+04 1.028093E+04 5.979930E+03 8.1461
i want columnar data as it is in the file.How to do that.I am a bgeinner
its hard to tell from the way the input data is presented in your question, but Im guessing your file is using tabs to separate columns, in any case, consider using python csv module with the relevant delimiter like:
import csv
with open('input.csv') as f_in, open('newdata1', 'w') as f_out:
reader = csv.reader(f_in, delimiter='\t')
writer = csv.writer(f_out, delimiter='\t')
for row in reader:
writer.writerow(row)
see python csv module documentation for further details