Assigning variable from csv file based on header using python - python-2.7

I have a csv file that has a single record in it that I need to assign variables to and run through a python script. My date looks like the line below. Has header.
"CompanyName","Contact","Street","CityZip","Store","DateRec","apples","appQuan","oranges","orgQuan","peaches","peaQuan","pumpkins","pumQuan","Receive",0
American Grocers","Allison Smith","456 1st. Street","Podunk, California 00990","Store 135 Order","05/14/2015",1,10,0,4,1,4,2,0
Each value needs to be assigned a variable
1st position, "American Grocers" = CompanyName
2nd position, "Allison Smith" = Contact
3rd position = Street, etc.
After the date it gets tricky. The last 11 values are related to each other and get saved to a key.
If value 7 = 1, then variable 7 = "apples" and variable 8 = 10, else skip values 7 and 8 and go to 9
If value 9 = 1, then variable 9 = "oranges" and value 10 = the variable in position 10 (4), else skip values 9 and 10 and go to 11
If value 11 = 1, then variable 11 = "peaches" and value 12 = the variable in position 10 (4), else skip values 11 and 12 and go to 13
If value 13 = 1, then variable 13 = "pumkins" and value 13 = the variable in position 13 (2), else skip values 13 and 14
If value 15 = 1 then variable 15 = "Delivery", else variable = "Pick up"
thus python would assign the following:
CompanyName = "American Grocers"
Contact = "Allison Smith"
Street = "456 1st. Street"
CityZip = "Podunk, California 00990"
Store = "Store 135 Order"
OrderDate (does not need to be date type) = "05/14/2015"
orderList = {"apples" : 10, "peaches" : 4, "pumpkins" : 2}
Recieve = "Pick up"
I need to manipulate these variables further along in the script.
I have the following code to which outputs the data to its corresponding header information.
import csv
MY_FILE = "C:\\tests\\DataRequestsData\\qryFruit.csv"
def parse(raw_file, delimiter):
opened_file = open(raw_file)
csv_data = csv.reader(opened_file, delimiter=delimiter)
parsed_data = []
fields = csv_data.next()
for row in csv_data:
parsed_data.append(dict(zip(fields, row)))
opened_file.close()
return parsed_data
def main():
new_data = parse(MY_FILE, ",")
print new_data
if __name__ == "__main__":
main()
Output looks like this. (I am not sure why the output is not in the same order as the file ...)
[{'DateRec': '05/14/2015', 'orgQuan': '4', 'CompanyName': 'American Grocers', 'appQuan': '10', 'peaQuan': '4', 'oranges': '0', 'peaches': '1', 'Contact': 'Allison Smith', 'CityZip': 'Podunk, California 00990', 'pumpkins': '2', 'apples': '1', 'pumQuan': '0', 'Store': 'Store 135 Order', 'Street': '456 1st. Street'}]
I do not know how to take this and get the variables assigned as listed above. Suggestions? Using python 2.7

I am not sure why the output is not in the same order as the file ...
In a Python dictionary, entries are displayed an arbitrary order.
Below is the general contour of how to parse the program. The detailed logic: "if this field is this do this otherwise do that" is something that I hope you can do on your own. The specifics are way too long, detailed, and specific to be of interest of value for anyone else and I'm guessing that is why there hasn't been much interest in this.
import csv
MY_FILE = "C:\\tests\\DataRequestsData\\qryFruit.csv"
def parse(raw_file, delimiter):
parsed_data = []
with open(raw_file) as opened_file:
rec = {}
csv_data = csv.reader(opened_file, delimiter=delimiter)
fields = csv_data.next()
for row in csv_data:
for i, val in enumerate(row[0:6]):
rec[fields[i]] = val
# This part below is too specific, long, and complicated
# that it is doubtful filling this out in detail will be use
# or interest to anyone else on stackoverflow. But to give
# you an idea of how to proceed...
if row[6] == '1':
rec[fields[6]] = 'apples'
rec[fields[7]] = 10
else:
# continue
pass
# ...
parsed_data.append(rec)
return parsed_data
def main():
new_data = parse(MY_FILE, ",")
print new_data
if __name__ == "__main__":
main()

I finally accomplished what I was looking for with the following code: Thank you to all who offered help. Your ideas spurred me onto the tangents I needed.
import csv
MY_FILE = csv.reader(open("C:\\tests\\DataRequestsData\\qryFruit.csv", "rb"))
for row in MY_FILE:
CompanyName, Contact, Street, CityZip, Store, DateRec, apples, appQuan, oranges, orgQuan, peaches, peaQuan, pumpkins, pumQuan, Receive = row
s='{'
if apples == "1":
s = s + '"apples"' + ":" + appQuan
if oranges == "1":
s = s + '", "oranges"' + ":" + orgQuan
if peaches == "1":
s = s + '", "peaches"' + ":" + peaQuan
if pumpkins == "1":
s = s + '", "pumpkins"' + ":" + pumQuan
s = s + '}'
if Receive == "0":
Receive = "Pick up"
else:
Receive = "Deliver"

Related

Q: Python3 - If Statements for changing list lengths

I am attempting to analyze data sets as lists of differing lengths. I am calling lines (rows) of my data set one by one to be analyzed by my function. I want the function to still be run properly regardless of the length of the list.
My Code:
f = open('DataSet.txt')
for line in iter(f):
remove_blanks = ['']
entries = line.split()
''.join([i for i in entries if i not in remove_blanks])
trash = (entries[0], entries[1])
time = int(entries[2])
column = [int(v) for v in entries[3:]]
def myFun():
print(entries)
print_string = ''
if column[0] == 100:
if column[1] >= 250 and column[2] == 300:
if len(column) >= 9:
digit = [chr(x) for x in column[4:9]]
print_string = ('code: ' + ''.join(str(digit[l]) for l in range(5)) + ' ')
if len(column) >= 13:
optional_digit = [chr(d) for d in column[9:13]]
for m in range(0, 4):
print_string += 'Optional Field: ' + optional_digit[m] + ''
else:
print_string += 'No Optional Field '
pass
pass
print(print_string)
print('')
myFun()
f.close()
What is happening is if the length of a line of my data is not long enough (i.e. the list ends at column[6]), I get the error:
line 17, in function
print('Code: ' + digit[l])
IndexError: list index out of range
I want it to still print Code: #number #number #number #number and leave any non-existent columns as blanks when it is printed so that one line may print as Code: ABC9 and the next print as Code: AB if there are differing list lengths.
Please help! :)
Well, just make sure you're not looping over a list longer than available:
print_string = 'code: ' + ''.join(str(digit[l]) for l in range(min(5, len(digit)))) + ' '
or better:
print_string = "code {} ".format("".join(str(dig) for dig in digit[:5]))
Although I have a feeling you're over-complicating this.

summing up a column in a csv file based on user search

I have the following csv file:
data.cvs
school,students,teachers,subs
us-school1,10,2,0
us-school2,20,4,2
uk-school1,10,2,0
de-school1,10,3,1
de-school1,15,3,3
I am trying to have a user search for the school country (us or uk, or de)
and then sum up the corresponding column. (e.g. sum all students in us-* etc.)
So far i am able to search using the raw_input and display column contents corresponding to the country, appreciate if someone can give me some pointers on how i can achive this.
desired output:
Country: us
Total students: 30
Total teachers: 6
Total subs: 2
--
import csv
import re
search = raw_input('Enter school (e.g. us: ')
with open('data.csv') as csvfile:
reader = csv.DictReader(csvfile)
for row in reader:
school = row['school']
students = row['students']
teachers = row['teachers']
sub = row['subs']
if re.match(search, schools) is not None:
print students
That's relatively easy to do - all you need is a dict to hold group your countries, and then just add together all of the values:
import collections
import csv
result = {} # store the results
with open("data.csv", "rb") as f: # open our file
reader = csv.DictReader(f) # use csv.DictReader for convenience
for row in reader:
country = row.pop("school")[:2] # get our country
result[country] = result.get(country, collections.defaultdict(int)) # country group
for column in row: # loop through all other columns
result[country][column] += int(row[column]) # add them together
# Now you can use or print your result by country:
for country in result:
print("Country: {}".format(country))
print("Total students: {}".format(result[country].get("students", 0)))
print("Total teachers: {}".format(result[country].get("teachers", 0)))
print("Total subs: {}\n".format(result[country].get("subs", 0)))
This is also universal as you can add additional number columns (e.g. janitors :D) and it will happily sum them together, but keep in mind that it works only with integers (if you want floats, replace the references to int with float) and it expects that every field except school is a number.
Your problem could be solved with something like this:
import csv
search = raw_input('Enter school (e.g. us: ')
with open('data.csv') as csvfile:
reader = csv.DictReader(csvfile)
result_countrys = {}
for row in reader:
students = int(row['students'])
teachers = int(row['teachers'])
subs = int(row['subs'])
subs = row['subs']
country = school[: 2]
if country in result_countrys:
count = result_countrys[country]
count['students'] = count['students'] + students
count['teachers'] = count['teachers'] + teachers
count['subs'] = count['subs'] + subs
else :
dic = {}
dic['students'] = students
dic['teachers'] = teachers
dic['subs'] = subs
result_countrys[country] = dic
for k, v in result_countrys[search].iteritems():
print("country " + str(search) + " has " + str(v) + " " + str(k))
I tryed out with this set of values:
reader = [{'school': 'us-school1', 'students': 20, 'teachers': 6, 'subs': 2}, {'school': 'us-school2', 'students': 20, 'teachers': 6, 'subs': 2}, {'school': 'uk-school1', 'students': 20, 'teachers': 6, 'subs': 2}]
and the result is:
Enter school (e.g. us): us
country us has 30 students
country us has 6 teachers
country us has 2 subs

Python 2.7. Being able to fill in variable name instead of index number

I can't seem to find how to, in Python 2.7, fill in a variable name instead of an index. I only find answers for other programming languages.
The file that is imported contains multiple columns, but I only want to add columns defined and asked for in the Query function to Data_BM. So how can IlocX be seen as an index.
Even if there is an easier way, I would just like to know how to put a variable on an index position. Thank you in advance.
# function for the datafile name
def datafilename():
print "Type the data filename, for example: myfile.csv : \n"
dataname = raw_input()
print dataname, 'Correct? [Y/N]'
yn = raw_input()
if yn == 'Y' or yn == 'y':
return dataname
else:
datafilename()
# function for the column numbers
def query(var):
print 'Type column number for',var,':'
loc = raw_input()
try:
loc = int(loc)
except:
loc = loc
print loc, 'Correct? [Y/N]'
yn = raw_input()
if yn == 'Y' or yn == 'y':
if type(loc) == int:
return loc
elif loc == 'NA' or loc == 'na':
return loc
else:
print 'Please type column as integer value'
query(var)
else:
query(var)
#Ask for column locations
loc1 = query("X")
loc2 = query("Y")
loc3 = query("Z")
loc4 = query("A")
loc5 = query("B")
Iloc1 = int(loc1-1)
Iloc2 = int(loc2-1)
Iloc3 = int(loc3-1)
Iloc4 = int(loc4-1)
Iloc5 = int(loc5-1)
# Asking for data file name
dataname = datafilename()
#Get data
Data_BM = pd.read_csv(dataname)
Data_BM = Data_BM[Iloc1,Iloc2,Iloc3,Iloc4,Iloc5]
I updated my file, I forgot to put in the query function. The user is asked to fill in the column number for for example X. The user types 1, which is loc1. Iloc 1 = 0. So 0 is the indexnumber. How can I have Iloc1 be the indexnumber in Data_BM = Data_BM[Iloc1,Iloc2,Iloc3,Iloc4,Iloc5] and the same accounts for the others. The reason I want this is because the columns might change but I want to make it easy for the user.
Short answer is no,
Long answer is why?
If you have a list of variables like a1, a2, a3, a4, ..., an, make a list of them, or a dictionary if you want to access it by strings.
The closest thing you can do is:
vars = {}
for i in range(10):
vars["Ilock"+str(i)] = something()
vars["Ilock4"] = 42
But it's ugly and not convenient.

Creating a if/else that appends data from mult. scraped pages if counts differ?

I"m trying to scrape Oregon teacher licensure information that looks like this or this(this is publicly available data)
This is my code:
for t in range(0,2): #Refers to txt file with ids
address = 'http://www.tspc.oregon.gov/lookup_application/LDisplay_Individual.asp?id=' + lines2[t]
page = requests.get(address)
tree = html.fromstring(page.text)
count = 0
for license_row in tree.xpath(".//tr[td[1] = 'License Type']/following-sibling::tr[1]"):
license_data = license_row.xpath(".//td/text()")
count = count + 1
if count==1:
ltest1.append(license_data)
if count==2:
ltest2.append(license_data)
if count==3:
ltest3.append(license_data)
with open('teacher_lic.csv', 'wb') as pensionfile:
writer = csv.writer(pensionfile, delimiter="," )
writer.writerow(["Name", "Lic1", "Lic2", "Lic3"])
pen = zip(lname, ltest1, ltest2, ltest3)
for penlist in pen:
writer.writerow(list(penlist))
The problem occurs when this happens: teacher A has 13 licenses and Teacher B has 2. In A my total count = 13 and B = 2. When I get to Teacher B and count equal to 3, I want to say, "if count==3 then ltest3.append(licensure_data) else if count==3 and license_data=='' then license3.append('')" but since there's no count==3 in B there's no way to tell it to append an empty set.
I'd want the output to look like this:
Is there a way to do this? I might be approaching this completely wrong so if someone can point me in another direction, that would be helpful as well.
There's probably a more elegant way to do this but this managed to work pretty well.
I created some blank spaces to fill in when Teacher A has 13 licenses and Teacher B has 2. There were some errors that resulted when the license_row.xpath got to the count==3 in Teacher B. I exploited these errors to create the ltest3.append('').
for t in range(0, 2): #Each txt file contains differing amounts
address = 'http://www.tspc.oregon.gov/lookup_application/LDisplay_Individual.asp?id=' + lines2[t]
page = requests.get(address)
tree = html.fromstring(page.text)
count = 0
test = tree.xpath(".//tr[td[1] = 'License Type']/following-sibling::tr[1]")
difference = 15 - len(test)
for i in range(0, difference):
test.append('')
for license_row in test:
count = count + 1
try:
license_data = license_row.xpath(".//td/text()")
except NameError:
license_data = ''
if license_data=='' and count==1:
ltest1.append('')
if license_data=='' and count==2:
ltest2.append('')
if license_data=='' and count==3:
ltest3.append('')
except AttributeError:
license_data = ''
if count==1 and True:
print "True"
if count==1:
ltest1.append(license_data)
if count==2 and True:
print "True"
if count==2:
ltest2.append(license_data)
if count==3 and True:
print "True"
if count==3:
ltest3.append(license_data)
del license_data
for endorse_row in tree.xpath(".//tr[td = 'Endorsements']/following-sibling::tr"):
endorse_data = endorse_row.xpath(".//td/text()")
lendorse1.append(endorse_data)

''str' object does not support item assignment' python 2

So this is my code it is strying tomease car registration plates, start times and end times (In the complete code it would be printed at the bottom).
data = str(list)
sdata = str(list)
edata = str(list)
current = 0
repeats = input ('How many cars do you want to measure?')
def main():
global current
print (current)
print ''
print ''
print '---------------------------------------'
print '---------------------------------------'
print 'Enter the registration number.'
data[current] = raw_input(' ')
print 'Enter the time it passed Camera 1. In this form HH:MM:SS'
sdata[current] = raw_input(' ')
print 'Enter the time it passed Camera 2. In this form HH:MM:SS'
edata[current] = raw_input (' ')
print '---------------------------------------'
print''
print''
print''
print 'The Registration Number is :'
print data[current]
print''
print 'The Start Time Is:'
print sdata[current]
print''
print 'The End Time Is:'
print edata[current]
print''
print''
raw_input('Press enter to confirm.')
print'---------------------------------------'
d = d + 1
s = s + 1
a = a + 1
current = current = 1
while current < repeats:
main()
When I run it and it gets to:
data[current] = raw_input(' ')
I get the error message 'TypeError: 'str' object does not support item assignment'
Thank you in advance for the help. :D
The error is clear. str object does not support item assignment
Strings in python are immutable. You have converted the data to a string when you do
data = str(list)
So, by
data["current"] = raw_input()
you are trying to assign some value to a string, which is not supported in python.
If you want data to be a list,
data = list()
or
data = []
will help, thus preventing the error
Dont use str during assignment
data = str(list)
sdata = str(list)
edata = str(list)
Instead use
data = []
sdata = []
edata = []
and later while printing use str if u want
print str(data[current])
as aswin said its immutable so dont complex it