ValueError: I/O operation - python-2.7

I see following error when executing this python code. What is issue here?
I have used "sys.stdout.close()" still I see these errors.
#! /usr/bin/python
import sys
a = [ 10, 12, 13, 14]
sys.stdout=open("file.txt","w")
print("++++++++")
print("***xyz***")
print("++++++++")
sys.stdout.close()
for i in a:
print i
Output:
Traceback (most recent call last):
File "./test3.py", line 10, in <module>
print i
ValueError: I/O operation on closed file`

You are trying to write to stdout (your file) after closing it. At line 8 you close the file, and at line 10 you call print.
If you want to write the list a to the file you should close it after the for loop.

Consider using with open because you don't have to worry about closing it. If your list needs to be a list then consider pickling it instead of writing it to a file. Pickling serializes your data.
#!python3
# import module
from os import system
import pickle
# clear the screan
system('cls')
a = [ 10, 12, 13, 14]
# write a list to file, but it has to be written as a string
with open('file.txt', 'w') as wf:
wf.write(str(a))
# when you open your file up, the data is a string
with open('file.txt', 'r') as fp:
for item in fp:
print(item)
print(type(item))
# if you want to retain your data as a list, then pickle it
output = open('file.pkl', 'wb')
pickle.dump(a, output)
output.close()
# open up a pickled file
pkl_file = open('file.pkl', 'rb')
data = pickle.load(pkl_file)
print(data)
print(type(data))
pkl_file.close()

Related

IOError: [Errno 22] invalid mode ('rb') or filename

I am a beginner at python and trying to use a very simple shutil module (shutil.copy) to copy databases from multiple folders into a backup folder. I am getting the error below. Any help is appreciated.
# importing os module
import os
#import time module
import time
import datetime
# importing shutil module
import shutil
now = datetime.datetime.now()
timestamp = str(now.strftime("%Y%m%d_%H%M%S"))
source5 = "F:/SHARED/SOP/PRE GO LIVE/TEST CASES & SCENARIOS/MASTER/PRE_GO_LIVE_MASTER.accdb"
dest5 = "F:/SHARED/SOP/SB/Python/Destination/PRE_GO_LIVE_MASTER.accdb_"+timestamp+".accdb"
print("Before copying ")
DB5 = shutil.copy(source5,dest5)
print("After DATABASE has been copied")
Error:
Traceback (most recent call last):
File "C:\Users\sbasava1\Desktop\Python\Final_Attempt.py", line 101, in <module>
DB5 = shutil.copy(source5,dest5)
File "C:\Python27\lib\shutil.py", line 119, in copy
copyfile(src, dst)
File "C:\Python27\lib\shutil.py", line 82, in copyfile
with open(src, 'rb') as fsrc:
IOError: [Errno 22] invalid mode ('rb') or filename:
Check your file path, use double backslashes \\ or a single forward slash / or make your string a raw string r"...".
# Original path - wouldn't work
path = "c:\location\directory"
# Raw string - would work
path = r"c:\location\directory"
# Double slashes - would work
path = "c:\\location\\directory"
# Forward slashes - would work
path = "c:/location/directory"
Consider learning about string literals
If this didn't help leave me a comment, it would also help to see the/part of the code you are working with!
Edit: Running your script didn't give me an issue:
Check if the directory you are trying to create a file in actually exists

Memory error even though RAM is free

I am merging files together in 4 folders. Within those 4 folders I am merging 80 .dbf files together each of which is 35 megabytes. I am using the following code:
import os
import pandas as pd
from simpledbf import Dbf5
list1=[]
folders=r'F:\dbf_tables'
out=r'F:\merged'
if not os.path.isdir(out):
os.mkdir(out)
for folder in os.listdir(folders):
if not os.path.isdir(os.path.join(out,folder)):
os.mkdir(os.path.join(out,folder))
for f in os.listdir(os.path.join(folders,folder)):
if '.xml' not in f:
if '.cpg' not in f:
table=Dbf5(os.path.join(folders,folder,f))
df=table.to_dataframe()
list1.append(df)
dfs = reduce(lambda left,right: pd.merge(left,right,on=['POINTID'],how='outer',),list1)
dfs.to_csv(os.path.join(out,folder,'combined.csv'), index=False)
almost immediately after running the code I receive this error:
Traceback (most recent call last):
File "<ipython-input-1-77eb6fd0cda7>", line 1, in <module>
runfile('F:/python codes/prelim_codes/raster_to_point.py', wdir='F:/python codes/prelim_codes')
File "C:\Users\spotter\AppData\Local\Continuum\Anaconda_64\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 714, in runfile
execfile(filename, namespace)
File "C:\Users\spotter\AppData\Local\Continuum\Anaconda_64\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 74, in execfile
exec(compile(scripttext, filename, 'exec'), glob, loc)
File "F:/python codes/prelim_codes/raster_to_point.py", line 66, in <module>
dfs = reduce(lambda left,right: pd.merge(left,right,on=['POINTID'],how='outer',),list1)
File "F:/python codes/prelim_codes/raster_to_point.py", line 66, in <lambda>
dfs = reduce(lambda left,right: pd.merge(left,right,on=['POINTID'],how='outer',),list1)
File "C:\Users\spotter\AppData\Local\Continuum\Anaconda_64\lib\site-packages\pandas\tools\merge.py", line 39, in merge
return op.get_result()
File "C:\Users\spotter\AppData\Local\Continuum\Anaconda_64\lib\site-packages\pandas\tools\merge.py", line 217, in get_result
join_index, left_indexer, right_indexer = self._get_join_info()
File "C:\Users\spotter\AppData\Local\Continuum\Anaconda_64\lib\site-packages\pandas\tools\merge.py", line 353, in _get_join_info
sort=self.sort, how=self.how)
File "C:\Users\spotter\AppData\Local\Continuum\Anaconda_64\lib\site-packages\pandas\tools\merge.py", line 559, in _get_join_indexers
return join_func(lkey, rkey, count, **kwargs)
File "pandas\src\join.pyx", line 160, in pandas.algos.full_outer_join (pandas\algos.c:61256)
MemoryError
but only 30% of my memory is being used, which is pretty much the baseline.
EDIT:
I picked out only 2 files and tried the merge using:
merge=pd.merge(df1,df2, on=['POINTID'], how='outer')
and still get a memory error, something weird is going on.
When I run the same thing in 32-bit Anaconda I get ValueError: negative dimensions are not allowed
EDIT:
The entire problem stemmed from the solution give here:
Value Error: negative dimensions are not allowed when merging
EDITED based on comment:
Try this (it's enough to use only one if statement with logical and conditions):
import os
import pandas as pd
from simpledbf import Dbf5
folders = r'F:\dbf_tables'
out = r'F:\merged'
if not os.path.isdir(out):
os.mkdir(out)
for folder in os.listdir(folders):
if not os.path.isdir(os.path.join(out, folder)):
os.mkdir(os.path.join(out, folder))
# Initialize empty dataframe by folders
dfs = pd.DataFrame(columns=['POINTID'])
for f in os.listdir(os.path.join(folders, folder)):
if ('.xml' not in f) and ('.cpg' not in f):
table = Dbf5(os.path.join(folders, folder, f))
df = table.to_dataframe()
# Merge actual dataframe to result dataframe
dfs = dfs.merge(df, on=['POINTID'], how='outer')
# Save results by folder
dfs.to_csv(os.path.join(out, folder, 'combined.csv'), index=False)

csv files in python

i am working on a machine leaning project and here is my code
import csv
import numpy as np
import string
from sklearn.ensemble import RandomForestRegressor
def main():
alchemy_category_set = {}
#read train data
train = []
target = []
with open("/media/halawa/93B77F681EC1B4D2/GUC/Semster 8/CSEN 1022 Machine Learning/2/train.csv", 'rb') as csvfile:
reader = csv.reader(csvfile, delimiter=',')
reader.next() #skip the header
for row in reader:
line = row[3:len(row)-1]
train.append(line)
target.append(row[len(row)-1])
if row[3] not in alchemy_category_set:
alchemy_category_set[row[3]] = len(alchemy_category_set)
#read valid data
valid = []
valid_index = []
with open("/media/halawa/93B77F681EC1B4D2/GUC/Semster 8/CSEN 1022 Machine Learning/2/test.csv", 'rb') as csvfile:
reader = csv.reader(csvfile, delimiter=',')
reader.next() #skip the header
for row in reader:
line = row[3:len(row)]
valid.append(line)
valid_index.append(row[1])
if row[3] not in alchemy_category_set:
alchemy_category_set[row[3]] = len(alchemy_category_set)
if __name__=="__main__":
main()
the reading of the test.csv is not working although it is working with the traing,csv , when i run it gives me
/usr/bin/python2.7 /home/halawa/PycharmProjects/ML/train.py
Traceback (most recent call last):
File "/home/halawa/PycharmProjects/ML/train.py", line 68, in <module>
main()
File "/home/halawa/PycharmProjects/ML/train.py", line 26, in main
reader.next() #skip the header
StopIteration
Process finished with exit code 1
the problem is with reading the csv file , any help would be appreciated .
I think you just forgot indentation after opening test file. Namely, after with open line the next 8 lines (each of these lines) should be indented with 2 more space .
By the way, it is highly recommended to indent with 4 spaces, not just 2.
And it should be consistent in your file

IndexError: too many indices on Python

Using PyDev with an eclipse environment for Python 2.7 on OSX. Trying to count the element in the array and sum up the elements in the array. Getting an error on the index.
import numpy as np
import os
import sys
csv_file_object = fileName = os.path.join('train.csv')
print('Directory separator on your platform ({}): {}'.format(sys.platform, os.sep))
data=[]
for row in csv_file_object:
data.append(row)
data = np.array(data)
number_passengers = np.size(data[0::,0].astype(np.float))
number_survived = np.sum(data[0::,0].astype(np.float))
proportion_survivors = number_survived / number_passengers
Traceback (most recent call last):
File "/Users/scdavis6/Documents/Kaggle/Titanic1.py", line 14, in <module>
number_passengers = np.size(data[0::,0].astype(np.float))
IndexError: too many indices
Let me know if I can provide additional information.
Thank you.
Update:
I made the edits, but got another error about the module not being callable:
Traceback (most recent call last):
File "/Users/scdavis6/Documents/Kaggle/Titanic1.py", line 5, in <module>
csv_file_object = fileName = os.path('train.csv')
TypeError: 'module' object is not callable
Update:
I changed os.path('train.csv') to os.path.join('train.csv'), but got another error about not finding the .csv file.
Traceback (most recent call last):
File "/Users/scdavis6/Documents/Kaggle/Titanic1.py", line 9, in <module>
with open(fileName) as f:
IOError: [Errno 2] No such file or directory: 'train.csv'
Here's the absolute path for the .csv file and the python scripts.
import os
os.path.abspath("/Users/scdavis6/Desktop/train.csv")
'/Users/scdavis6/Desktop/train.csv'
import os
os.path.abspath("/Users/scdavis6/Documents/Kaggle/Titanic1.py")
'/Users/scdavis6/Documents/Kaggle/Titanic1.py'
Assuming that this is your actual code, the problem is that you never open the file. Your csv_file_object is still just the fileName, and thus your data is made up of the characters of that file name, resulting in a 1D numpy array.
Instead, you should open the file and create a csv.reader for it.
import csv
with open(fileName) as f:
reader = csv.reader(f)
data=[]
for row in reader:
data.append(row)
data = np.array(data)
Or shorter: data = np.array([row for row in csv.reader(f)])
Update: The new error you are getting is probably due to you accidentally changing
os.path.join('train.csv') to os.path('train.csv'), i.e., instead of calling the join function from the os.path module, you are (trying to) call the module itself.
Update: It seems your train.csv file is not in the same directory as your Python script, thus the script won't find the file if you just use the filename. You have to use the absolute path together with the filename:
fileName = os.path.join('/Users/scdavis6/Desktop', 'train.csv')
Or just fileName = '/Users/scdavis6/Desktop/train.csv'. Alternatively, move your train.csv file to the same directory as your Python script. This might indeed be the better and more robust option, unless you are using this file in multiple scripts in different directories.

Django-Adaptors TypeError: "delimiter" must be an 1-character string

I am getting an error when trying to import some data into my model. The error I'm getting is TypeError: complaining about the delimiter I'm using.
Below is my model for the CSV import, I am using the default delimiter suggested by the documentation.
class SkuCsvModel(CsvModel):
sku_num = models.CharField()
sku_category = models.ForeignKey(SkuCategory)
short_desc = models.CharField()
class Meta:
delimiter = ";"
dbModel = Sku
The CSV file I'm trying to use is below:
1365400;9;3/8 BALL VALVE
1401901;9;BRASS ELBOW
1406300;9;HOSE BARB, NPT
The code I'm testing in the manage.py shell is:
>>> from core.models import SkuCsvModel
>>> my_csv_list = SkuCsvModel.import_data(data = open("labconco.csv"))
And finally the error I'm getting is:
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "E:\bin\Python27\lib\site-packages\adaptor\model.py", line 197, in import_data
return importer.import_data(data)
File "E:\bin\Python27\lib\site-packages\adaptor\model.py", line 466, in import_data
for line in csv.reader(data, delimiter=self.delimiter):
TypeError: "delimiter" must be an 1-character string
So I've been fiddling around with the django-adaptor tools, and this error is coming from the import_data() method of the CsvImporter, when I try and put a delimiter directly into the csv.reader(data, delimiter=';') this works fine and I'm able to see the file correctly. But no matter how I try and enter this import_data method sending in a ';' will generate an error.
Look at the snippet below. If I provide an integer as a delimiter, it fails with the same exception as in your example. If I provide a semicolon as a delimiter to csv.reader it works. This is basically what is done in model.CsvImporter.import_data() as you already found out.
>>> import csv
>>> import StringIO
>>> io = StringIO.StringIO('name;surname\nsascha;gottfried')
>>> for line in csv.reader(io, delimiter=10):
... print line
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: "delimiter" must be an 1-character string
>>> for line in csv.reader(io, delimiter=';'):
... print line
...
['name', 'surname']
['sascha', 'gottfried']
To debug the situation dump the current value of 'self.delimiter' to the console or similar before it will be passed as a delimiter to csv.reader(). It must be a different value and/or type than ';'. Looking at the code of django-adaptors you can validate your django-adaptors model definition with this base class method as well to debug. This call should print out what you defined as Meta.delimiter.
>>> from core.models import SkuCsvModel
>>> SkuCsvModel.has_class_delimiter()
But it is pretty fine to omit a delimiter definition and call 'import_from_file' on the model. Make sure there is no class delimiter defined. If so the importer runs a CSV sniff to detect the delimiter from the file you passed. If you provide the file you mentioned, the sniffer will detect a ';' and sets self.delimiter.
>>> from core.models import SkuCsvModel
>>> SkuCsvModel.has_class_delimiter()
None
>>> my_csv_list = SkuCsvModel.import_from_file(file = open("labconco.csv"))