So trying to finish a very simple script that has given me a unbelievably hard time. It's supposed to iterate through specified directories and open all text files in them and append them all with the same specified string.
The issue is it's not doing anything to the files at all. Using print to test my logic I've replaced lines 10 and 11 with print f (the write and close functions), and get the following output:
<open file '/Users/russellculver/documents/testfolder/.DS_Store', mode 'a+' at
So I think it is storing the correct files in the f variable for the write function, however I am not familiar with how Mac's handle DS_STORE or the exact role it plays in temporary location tracking.
Here is the actual script:
import os
x = raw_input("Enter the directory path here: ")
def rootdir(x):
for dirpaths, dirnames, files in os.walk(x):
for filename in files:
try:
with open(os.path.join(dirpaths, filename), 'a+') as f:
f.write('new string content')
f.close()
except:
print "Directory empty or unable to open file."
return x
rootdir(x)
And the exact return in Terminal after execution:
Enter the directory path here: /Users/russellculver/documents/testfolder
Exit status: 0
logout
[Process completed]
Yet nothing written to the .txt files in the provided directory.
The way the indentation is in the question, you return from the function right after writing the first file; either of the for-loops never finish. Which is relatively easy to surmise from the fact that you only get one output file printed.
Since you're not doing anything with the result of the rootdir function, I would just remove the return statement entirely.
An aside: there is no need to use f.close() when you open a file with the with statement: it will automatically be closed (even upon an exception). That is in fact what the with statement was introduced for (see the pep on context managers if necessary).
To be complete, here's the function the way I would have (roughly) written it:
def rootdir(x):
for dirpaths, dirnames, files in os.walk(x):
for filename in files:
path = os.path.join(dirpaths, filename)
try:
with open(path, 'a+') as f:
f.write('new string content')
except (IOError, OSError) as exc:
print "Directory empty or unable to open file:", path
(Note that I'm catching only the relevant I/O errors; any other exceptions (though unlikely) will not be caught, as they are likely not to be related to non-existing/unwritable file.)
Return was indented wrong, ending the iteration after a single loop. Wasn't even necessary so was removed entirely.
Related
I am trying to find situations when os.path.isfile() returns false. I have the following code which checks if the content is not a file (should have used isdir()) and does structured zip.
for fpath in glob(os.path.join(local_dir, glob_wildcard)):
dirpath, filename = os.path.split(fpath)
try:
if not os.path.isfile(fpath):
//code for structured zipping and move
else:
//move the file as is
This was working fine until now but recently found some of the files are getting zipped, than getting moved. One cause was, I have found another instance running at the same time which stole the file and isfile() for this instance returned false. So, it created an empty zip file.
Are there any other scenarios when it will return false? like file getting created or copied etc.
I am learning about exceptions and so performing some file operations and testing various parts of code that can possibly generate exceptions while working with files in Python. I am executing this Python 2.7 code on Canopy.
#!/usr/bin/python
import os
try:
fp = open('testfile', 'r')
except IOError:
print 'File not opened successfully'
else:
print 'File opened successfully'
try:
fp.write('Hello!')
except IOError:
print 'Write not allowed on this file'
else:
print 'Write successful'
try:
fp.close()
except IOError:
print 'File not closed properly'
else:
print 'File closed successfully'
finally:
if os.path.exists(fp.name):
os.remove(fp.name)
When I execute this code, I get the following output:
File not opened properly
NameErrorTraceback (most recent call last)
/home/sr/Python/tcs.py in ()
--> 185 if os.path.exists(fp.name)
NameError: name 'fp' is not defined
But if I change the access mode of file to 'w', Then everything seems to work properly with the correct output as:
File opened successfully
Write successful
File closed successfully
I cannot understand why the 'r' mode is not making the file open properly and thus the fp file object is not created. Please help me figure the problem out.
P.S.: Also I would like to know if there is a better way of implementing the same thing. But this is optional.
Explanation
The error combined with your printout should be pretty self-explanatory: the variable fp does not exist if you can't open the file.
The mode 'r' indicates that you want to open the file for reading. You can not read something that is not there, so you end up going to the finally block in your code after processing the IOError. But the error occurs before fp was set, so there is no variable fp, hence the error. [Solutions below]
The mode 'w' indicates that you want to open for writing, but from scratch. There is also an 'a' mode to append if the file already exists. You can write to a non-existent file just fine, so your code does not fail. In fact, if the file did exist in 'w' mode, it would be trucated and any previous contents would be lost.
Try creating an empty file and running with mode 'r'. You should get an exception that prints 'Write not allowed on this file'. That is because, as your error message correctly indicates, writing to a file opened in read mode is not allowed.
Improvements
There are two major improvements that you can make to your code. One is fixing the logical flaws, the other is a major stylistic improvement using with statements.
You have two major logic errors. The first is in the outermost finally block that you already saw. The simplest fix is moving the contents of the finally block into the else, since you don't have any action to take if the file was not opened. Another solution is to refer to the file name you are trying to open in the first place. For example, you could store the file name into a variable and use that:
filename = 'testfile'
try:
fp = open(filename, 'r')
...
finally:
if os.path.exists(filename):
os.remove(filename)
The second major logic error is that you do not close the file if your write fails. Notice that you call fp.close() only in the else clause of your try block. If should instead appear in a finally block. The print statement should of course stay in the else. Change
else:
print 'Write successful'
try:
fp.close()
...
to
else:
print 'Write successful'
finally:
try:
fp.close()
...
The entire code can be improved stylistically by using with blocks to manage your file operations. The obvious way to do this is as follows:
fname = 'testfile'
with open(fname, 'r') as fp:
fp.write('Hello!')
if os.path.exists(fname):
os.remove(fname)
You will not get as many detailed messages when things fail, but overall, this code is cleaner, shorter and more robust than what you have. It guarantees that the file will be closed whether or not an exception occurred anywhere along the way. If you need the detailed error output that you currently have, keep using the current try blocks. The reason that most people will prefer with is that any error that occurs will have a detailed desciption and a line number that it happened on, so you basically get all the necessary information with a lot less work.
Here are some additional resources to help you understand with and context managers:
Understanding Python's "with" statement (from effbot.org)
Official documentation for with
SO 1, 2, 3, 4
I have the following code in python:
import os
class suppress_stdout_stderr(object):
'''
A context manager for doing a "deep suppression" of stdout and stderr in
Python, i.e. will suppress all print, even if the print originates in a
compiled C/Fortran sub-function.
This will not suppress raised exceptions, since exceptions are printed
to stderr just before a script exits, and after the context manager has
exited (at least, I think that is why it lets exceptions through).
'''
def __init__(self):
# Open a pair of null files
self.null_fds = [os.open(os.devnull,os.O_RDWR) for x in range(2)]
# Save the actual stdout (1) and stderr (2) file descriptors.
self.save_fds = (os.dup(1), os.dup(2))
def __enter__(self):
# Assign the null pointers to stdout and stderr.
os.dup2(self.null_fds[0],1)
os.dup2(self.null_fds[1],2)
def __exit__(self, *_):
# Re-assign the real stdout/stderr back to (1) and (2)
os.dup2(self.save_fds[0],1)
os.dup2(self.save_fds[1],2)
# Close the null files
os.close(self.null_fds[0])
os.close(self.null_fds[1])
for i in range(10**6):
with suppress_stdout_stderr():
print 'plop'
if i % 50 == 0:
print i
it fails at 5100 on OSX with OSError: [Errno 24] Too many open files. I'm wondering why and if there is a solution to close the file descriptor. I'm looking for a solution for a context manager which closes stdout and stderr.
I executed your code on a Linux machine and got the same error but at a different number of iterations.
I added the following two lines in the __exit__(self, *_) function of your class:
os.close(self.save_fds[0])
os.close(self.save_fds[1])
With this change I do not get an error and the script returns successfully. I assume that the duplicated file descriptors stored in self.save_fds are kept open if you don't close them with os.close(fds) and so you get the too many files open error.
Anyway my console printed "plop", but maybe this depends on my platform.
Let me know if it works :)
This script is an improvement on the previously posted one but it is
still giving me an error of "Failed to execute (CADToGeodatabase)"
It is able to iterate through the directories and subdirectories, list
the dwg files, create the geodatabase but not able to populate it with
the feature datasets and feature classes due to the error!. Please help!
import os, os.path, arcpy
from arcpy import env
env.workspace = "J:/2010"
# Set workspace and variables
gdb = r"C:\data\2010.gdb"
arcpy.env.workspace = gdb
# Create a FileGDB for the fds
arcpy.CreateFileGDB_management("C:/data", "2010.gdb")
reference_scale = "1500"
for root, dirs, files in os.walk("J:/2010/"):
for file in files:
if file.endswith('.dwg'):
print "current file is: " + file
outDS = arcpy.ValidateTableName(os.path.splitext("d" +
os.path.basename(file))[0])
arcpy.CADToGeodatabase_conversion(file, gdb, outDS, reference_scale)
The line saying def recursive_file_gen(r"J:\2010"): looks strange to me. I don't think you can put a literal string there. I am surprised that this runs at all. Maybe you meant to do something like def recursive_file_gen(directory=r"J:\2010"): or simply def recursive_file_gen():.
Also, I think the line saying yield os.path.join(root, file) needs to be indented more to be considered inside of the inner for loop.
I don't see specifically what is causing the script to work only in one subdirectory. I will need more details about what is happening.
EDIT: I didn't notice that the recursive_file_gen function is not being used at all. I don't know what is causing the problem. I think someone who is more familiar with arcpy would be more helpful to you.
I'm running a program that does processing on a file.
I want to be able to supply the program with several files, and by attaching to it with gdb, I want to get a memory dump at a certain point in the code for each of the files. I want the dump for each file to go to a file with the same filename as the input file (maybe after formatting it a little, say adding a suffix)
So suppose I have a function called HereIsTheFileName(char* filename), and another function called DumpThisMemoryRegion(void* startAddr, void* endAddr), I want to do something like the following:
To get the file name to an environment variable:
break HereIsTheFileName
commands 1
set $filename = malloc(strlen(filename) + 1)
call memcpy($filename, filename, strlen(filename) + 1)
end
Then to dump the memory to the filename I saved earlier:
break DumpThisMemoryRegion
commands 2
append binary memory "%s.memory"%$filename startAddr endAddr
end
(I would even settle for the filename as it is, without formatting, if that turns out to be the difficult part)
However, I couldn't get gdb to accept anything except an exlicit file name for the append/dump commands. when I ran "append binary memory $filename ..." I got the output in the file "/workdir/$filename".
Is there any way to make gdb choose the file name at runtime?
Thanks!
I don't know how to make append accept a runtime filename, but you can always cheat a bit by writing the whole thing to a file and then sourcing that file, using logging.
By putting this in your ~/.gdbinit
define reallyappend
printf "using gdbtmp.log to dump memory to file %s\n", $arg0
set logging file gdbtmp.log
set logging overwrite on
set logging redirect on
set logging on
printf "append binary memory %s 0x%x 0x%x", $arg0, $arg1, $arg2
set logging off
set logging redirect off
set logging overwrite off
source gdbtmp.log
end
you can use the function reallyappend instead, for example with
(gdb) set $filename = "somethingruntimegenerated"
(gdb) reallyappend $filename startAddr endAddr
I don't know if logging works ok in an "commands" environment, but you can give it a shot at least.
Yeah, you can't use a variable here for the filename argument.
The best suggestion I can offer is to write a script that will
set all the breakpoints and set up the "append" commands, and
use text editing or awk and sed to set up the filenames in the
script.