I have a CSV file and I am trying to populate them to a sqlite database. I have no error message and it works perfectly fine but loads only the last line of the file.
MD= MD()
database = options.get('database')
filename = options.get('filename')
dataReader = csv.reader(open(filename))
for row in dataReader:
if row[0] != 'ID':
bb= 1 if row[3] == 'YES' else 0
pro = 'YES' if row[4] == 'Pro' else 'NO'
MD.id = row[0]
MD.mol = row[1]
MD.phase = row[2]
MD.warning = black_box
MD.pro = pro
MD.status = Type.objects.get(description=row[5])
MD.name = row[6]
MD.stem = row[7]
MD.year = row[8]
MD.iname = row[9]
MD.iyear = row[10]
print row[1], row[2],row[3],row[4],row[5],row[6], row[0]
MD.save()
But the print statement prints all the lines in the CSV file. I have no idea what happens. Thanks
You are only creating one MD instance outside of the for loop, then saving to that same instance on each iteration of the loop. You need to create a new MD() instance for every iteration (per line of the file) if you want to create and save a new record for each line. This is why you are only saving the last line - you are over-writing the pre-existing instance you created. Good luck.
You are saving always the same object. Try with this:
Put the MD= MD() inside the for:
... # The same here
for row in dataReader:
if row[0] != 'ID':
MD= MD()
... # The same here
MD.id = row[0]
MD.mol = row[1]
...
MD.save()
Related
I'm relatively new to python (2.7) and need help looping through 2 CSV files. The first (outer loop) file is the row I want to write if certain conditions are met with the second (inner loop) file.
import csv
f = open('../CI Working Copy.csv')
with open('../first.csv', 'wb') as n:
theWriter = csv.writer(n)
csv_f = csv.reader(f)
g = open('../second.csv')
csv_g = csv.reader(g)
for row in csv_f:
cbd = row[3]
ced = row[4]
rbd = row[5]
red = row[6]
ciCn = row[10]
for iRow in csv_g:
cn = iRow[0]
startDate = iRow[1]
endDate = iRow[2]
iId = iRow[3]
writeRow = 'false'
if ciCn == cn:
if (cbd == startDate and ced == endDate) or (rbd == startDate and red == endDate):
theWriter.writerow(row)
g.close()
f.close()
It makes it into the second (inner loop) file, but never returns to the outer loop. I only need to write the row from the first file.
For each row of the first csv file, you consume all the second file, so you need to go back on the beginning of the second file on each iteration.
The solution is:
for row in csv_f:
g.seek(0) #go at the start of the second file
for iRow in csv_g:
do_smth(iRow,row)
g.close()
Hello I was hoping someone could help me with my college coursework, I have an issue with my code. I keep running into a memory error with my data export.
Is there any way I can reduce the memory that is being used or is there a different approach I can take?
For the course work I am given a file of 300 records about customer orders from a CSV file and then I have to export the Friday records to a new CSV file. Also I am required to print the most popular method for customer's orders and the total money raised from the orders but I have an easy plan for that.
This is my first time working with CSV so I'm not sure how to do it. When I run the program it tends to crash instantly or stop responding. Once it appeared with 'MEMORY ERROR' however that is all it appeared with. I'm using a college provided computer so I am not sure on the exact specs but I know it runs 4GB of memory.
defining count occurences predefined function
def countOccurences(target,array):
counter = 0
for element in array:
if element == target:
counter= counter + 1
print counter
return counter
creating user defined functions for the program
dataInput function used for collecting data from provided file
def dataInput():
import csv
recordArray = []
customerArray = []
f = open('E:\Portable Python 2.7.6.1\Choral Shield Data File(CSV).csv')
csv_f = csv.reader(f)
for row in csv_f:
customerArray.append(row[0])
ticketID = row[1]
day, area = datasplit(ticketID)
customerArray.append(day)
customerArray.append(area)
customerArray.append(row[2])
customerArray.append(row[3])
recordArray.append(customerArray)
f.close
return recordArray
def datasplit(variable):
day = variable[0]
area = variable[1]
return day,area
def dataProcessing(recordArray):
methodArray = []
wed_thursCost = 5
friCost = 10
record = 0
while record < 300:
method = recordArray[record][4]
methodArray.append(method)
record = record+1
school = countOccurences('S',methodArray)
website = countOccurences('W',methodArray)
if school > website:
school = True
elif school < website:
website = True
dayArray = []
record = 0
while record < 300:
day = recordArray[record][1]
dayArray.append(day)
record = record + 1
fridays = countOccurences('F',dayArray)
wednesdays = countOccurences('W',dayArray)
thursdays = countOccurences('T', dayArray)
totalFriCost = fridays * friCost
totalWedCost = wednesdays * wed_thursCost
totalThurCost = thursdays * wed_thursCost
totalCost = totalFriCost + totalWedCost + totalThurCost
return totalCost,school,website
My first attempt to writing to a csv file
def dataExport(recordArray):
import csv
fridayRecords = []
record = 0
customerIDArray = []
ticketIDArray = []
numberArray = []
methodArray = []
record = 0
while record < 300:
if recordArray[record][1] == 'F':
fridayRecords.append(recordArray[record])
record = record + 1
with open('\Courswork output.csv',"wb") as f:
writer = csv.writer(f)
for record in fridayRecords:
writer.writerows(fridayRecords)
f.close
My second attempt at writing to the CSV file
def write_file(recordArray): # write selected records to a new csv file
CustomerID = []
TicketID = []
Number = []
Method = []
counter = 0
while counter < 300:
if recordArray[counter][2] == 'F':
CustomerID.append(recordArray[counter][0])
TicketID.append(recordArray[counter][1]+recordArray[counter[2]])
Number.append(recordArray[counter][3])
Method.append(recordArray[counter][4])
fridayRecords = [] # a list to contain the lists before writing to file
for x in range(len(CustomerID)):
one_record = CustomerID[x],TicketID[x],Number[x],Method[x]
fridayRecords.append(one_record)
#open file for writing
with open("sample_output.csv", "wb") as f:
#create the csv writer object
writer = csv.writer(f)
#write one row (item) of data at a time
writer.writerows(recordArray)
f.close
counter = counter + 1
#Main Program
recordArray = dataInput()
totalCost,school,website = dataProcessing(recordArray)
write_file(recordArray)
In the function write_file(recordArray) in your second attempt the counter variable counter in the first while loop is never updated so the loop continues for ever until you run out of memory.
I am using a csv iterator to go through a csv file, which contains data associated with time. I am sure the csv file is correct. I am using Jupyter, iPython notebook in python 3.x
When I try to iterate on the first row by using .next() method, I have an AttributeError: 'module' object has no attribute 'next'.
My code is divided in two parts, one part containing function and imports, one part calling them. The function I have a problem with is:
def get_durations(csvfile):
try:
csvfile = iter(csvfile)
except TypeError, te:
print csvfile, 'is not iterable'
print "data is", repr(csvfile)
first_time = csvfile.next()[5]
first_time = (first_time.replace(" ", ""));
for row in csvfile:
last_time = row[5]
last_time = (last_time.replace(" ", ""))
first_time = datetime.datetime.strptime(first_time, "%H:%M:%S")
last_time = datetime.datetime.strptime(last_time, "%H:%M:%S")
return first_time.replace(second = 0), last_time.replace(second = 0)
I make the call to the function here:
for el_mousefile in mousefiles:
os.chdir(el_mousefile)
print "data is", repr(csvfile)
csvfile = csv.reader(open("mouse.csv", "rU"), delimiter=';', quoting=csv.QUOTE_NONE)
print "data is", repr(csvfile)
try:
csvfile = iter(csvfile)
except TypeError, te:
print csvfile, 'is not iterable'
first_time, last_time = get_durations(csv)
I get this output when trying to run the program:
data is <_csv.reader object at 0x000000000A388D08>
data is <_csv.reader object at 0x000000000A388948>
module 'csv' from 'C:\Users\**\AppData\Local\Continuum\Anaconda\lib\csv.pyc' is not iterable
data is module 'csv' from 'C:\Users\**\AppData\Local\Continuum\Anaconda\lib\csv.pyc'
96------>first_time = csvfile.next()[5]
97 first_time = (first_time.replace(" ", ""));
98 for row in csvfile:
AttributeError: 'module' object has no attribute 'next'
I don't understand how can my csv be iterable in the second part, but then when being passed to the function, it is not iterable anymore which is responsible for my error.
first_time, last_time = get_durations(csv)
You are passing in the csv module, not a file.
I want to add a date column to data exported to csv file. Luckily the files are identified by the date they represent. However, I can't get the column to fill up with the appropriate user inputted value. Here's the code I have so far:
def read_file():
user_input = raw_input("Please put cost folder with date in form Costmm.dd: ")
path = r'C:\\Users\\CP\\documents\\' + user_input
allFiles = glob.glob(path + '/*.csv')
frame = pd.DataFrame()
frame['Date'] = pd.Series()
frame['Date'] = frame['Date'].astype(object).fillna(user_input)
list = []
for file in allFiles:
df = pd.read_csv(file,index_col=None,header=0)
list.append(df)
frame =pd.concat(list,ignore_index=True)
frame['Date'] = pd.Series()
return(frame)
Any and all help is greatly appreciated!
If I understand what you're after the following should work:
import datetime as dt
def read_file():
user_input = raw_input("Please put cost folder with date in form Costmm.dd: ")
path = r'C:\\Users\\CP\\documents\\' + user_input
allFiles = glob.glob(path + '/*.csv')
l = []
for file in allFiles:
df = pd.read_csv(file,index_col=None,header=0)
l.append(df)
frame =pd.concat(l,ignore_index=True)
frame['Date'] = dt.datetime.strptime(user_input, '%m.%d')
return frame
A few notes:
frame = pd.DataFrame()
frame['Date'] = pd.Series()
frame['Date'] = frame['Date'].astype(object).fillna(user_input)
The above was redundant as you overwrite this with the result of pd.concat anyway.
Don't user list for a variable name of type list use something like df_list.
To convert a string to datetime you can use dt.datetime.strptime and pass the format string.
I am trying to use web2py to build an app. I have a simple print function that a user submits a key word . The string or int key word is directed to an sqlite db to retrieve a row and output the data. I need to know
1. how to use the print on html.
2. How to split the string...so far i did the list:string
Here is my code:
def first():
form = SQLFORM.factory(Field('visitor_name',
label = 'Please Type Your keyword here!',
requires= [IS_NOT_EMPTY(), IS_LOWER(),'list:string']))
form.element('input[type=submit]')['_onclick'] = "return \
confirm('Are you sure you want to submit:');"
if form.process().accepted:
session.visitor_name = form.vars.visitor_name
redirect(URL('main'))
return dict(form=form)
def main():
while True:
name = request.vars.visitor_name or redirect(URL('first'))
name2 = name[:]
for item in name2:break
name3 = ' '.join(name2)
import sqlite3
id = 0
location = ""
conn = sqlite3.connect("keywords.db")
c = conn.cursor()
c.execute('select * from kmedicals')
records = c.fetchall()
for record in records:
id = record[0]
location = record[15]
if id == name3:
print name3.capitalize(),':' '\n',location
break
sys.exit()
return dict(name=name)
my view...default/main.html:
{{extend 'layout.html'}}
{{=name}}