I have the following function that gives me the list of files(complete path) in a given list of directories:
from os import walk
from os.path import join
# Returns a list of all the files in the list of directories passed
def get_files(directories = get_template_directories()):
files = []
for directory in directories:
for dir, dirnames, filenames in walk(directory):
for filename in filenames:
file_name = join(dir, filename)
files.append(file_name)
return files
I'am adding some files to the template directories in Django. But this function always return the same list of files even though some are added/deleted in the run time. These changes are reflected only when I do a server restart. Is that because of some caching that os.walk() performs or is it required that we need to restart the server after adding/removing some files ?
It is not django problem, your behaviour is result of python interpreter specific:
Default arguments may be provided as plain values or as the result of a function call, but this latter technique need a very big warning. Default values evaluated once at start application and never else.
I' m sure this code will solve your problem:
def get_files(directories = None):
if not directories:
directories = get_template_directories()
files = []
for directory in directories:
for dir, dirnames, filenames in walk(directory):
for filename in filenames:
file_name = join(dir, filename)
files.append(file_name)
return files
You can find same questions on Stackoverflow Default Values for function parameters in Python
Related
I'm using Django and django-storages[azure] as backend.
But i can't to find out to list dir the files instead I have to use snippets like this:
block_blob_service.list_blobs(container_name='media', prefix=folderPath)]
this is not working:
listdir(absoluteFolderPath)
In storages/backend/azure_storage.py I found this part:
def listdir(self, path=''):
"""
Return directories and files for a given path.
Leave the path empty to list the root.
Order of dirs and files is undefined.
"""
files = []
dirs = set()
for name in self.list_all(path):
n = name[len(path):]
if '/' in n:
dirs.add(n.split('/', 1)[0])
else:
files.append(n)
return list(dirs), files
But how can I use it?
regards
Christopher.
Assuming you have set DEFAULT_FILE_STORAGE you could access it through storage
from django.core.files.storage import default_storage
dirs = default_storage.listdir(absoluteFolderPath)
The API for listdir returns a tuple of two lists, the first being the directories (folders) in the given path, the second being the files.
The correct way to unpack this would be to assign two variables during the call.
Examples:
from django.core.files.storage import default_storage
# list the contents of the root directory of the storage
directories, files = default_storage.listdir('')
# list the contents of a directory named 'images' inside 'media'
directories, files = default_storage.listdir('media/images')
# now you can iterate through just the files
for filename in files:
print(filename)
I'm using django-storages with S3, but Azure looks to work exactly the same.
I have looked extensively on this site and I can't see an example that fits the bill. I have 4 directories each of which contains a number of files and another directory called 'Superseded'. I am trying to write a script that will move all files in each folder into the 'Superseded' folder but I'm not having any luck.
import os, shutil
source = r'U:\Data\All\Python_Test\Exports\GLA'
dest = r'U:\Data\All\Python_Test\Exports\Superseded'
listofFiles = os.listdir(source)
for f in listofFiles:
fullPath = source + "/" + f
shutil.move(fullPath, dest)
I can only get this to work for one directory and even then only when I've made the destination directory outside of the GLA directory if that makes sense.
I know there is a a os.path.isfile() module so that I can only move the files but I can't seem to get it to work. Does anybody have any ideas?
This works for me:
import os
#from:
# https://stackoverflow.com/questions/1158076/implement-touch-using-python
# I use this to create some empty file to move around later
def touch(fname, times=None):
fhandle = open(fname, 'a')
try:
os.utime(fname, times)
finally:
fhandle.close()
# this function is only to create the folders and files to be moved
def create_files_in_known_folders():
nameList=["source_dir_{:02d}".format(x) for x in range(4)]
for name in nameList:
path=os.path.expanduser(os.path.join("~",name))
if not os.path.exists(path):
os.mkdir(path)
ssPath=os.path.join(path,"superseded")
if not os.path.exists(ssPath):
os.mkdir(ssPath)
for i in range(3):
filename="{}_{:02d}.dat".format(name,i)
filepath=os.path.join(path, filename)
if not os.path.exists(filepath):
touch(filepath)
# THIS is actually the function doing what the OP asked for
# there many details that can be tweaked
def move_from_known_to_dest():
# here my given names from above
nameList=["source_dir_{:02d}".format(x) for x in range(4)]
# and my destination path
destPath=os.path.expanduser(os.path.join("~","dest"))
# not interested in files that are in subfolders
# if those would exist change to os.walk and
# exclude the destination folder with according if...:
for name in nameList:
path=os.path.expanduser(os.path.join("~",name))
dirList=os.listdir(path)
print path
for fileName in dirList:
filePath=os.path.join(path, fileName)
print filePath
if os.path.isfile(filePath):
destPath=os.path.join(path,"superseded",fileName)
print destPath
#alternatively you can chose to 1) overwrite ()might not work 2)delete first 3) not copy
# another option is to check for existence and if
# present add a number to the dest-file-name
# use while loop to check for first non-present number
assert not os.path.exists(destPath), "file {} already exits".format(destPath)
#https://stackoverflow.com/questions/8858008/how-to-move-a-file-in-python
os.rename( filePath, destPath)
if __name__=="__main__":
create_files_in_known_folders()
#break here and check that filestructure and files have been created
move_from_known_to_dest()
But, think carefully what to do if the file already exits in your destination folder.
os.walk might also be something you want to look at.
Implementing several options for the copy behaviour may look like this:
import warnings
#from:
# https://stackoverflow.com/questions/2187269/python-print-only-the-message-on-warnings
formatwarning_orig = warnings.formatwarning
warnings.formatwarning = lambda message, category, filename, lineno, line=None: \
formatwarning_orig(message, category, filename, lineno, line='')
def move_from_known_to_dest_extra(behaviour='overwrite'):
assert behaviour in ['overwrite','leave','accumulate'], "unknown behaviour: {}".format(behaviour)
nameList=["source_dir_{:02d}".format(x) for x in range(4)]
destPath=os.path.expanduser(os.path.join("~","dest"))
for name in nameList:
path=os.path.expanduser(os.path.join("~",name))
dirList=os.listdir(path)
for fileName in dirList:
filePath=os.path.join(path, fileName)
if os.path.isfile(filePath):
destPath=os.path.join(path,"superseded",fileName)
# simplest case...does not exist so copy
if not os.path.exists(destPath):
os.rename( filePath, destPath)
else:
if behaviour=='leave':
warnings.warn( "Warning! Not copying file: {}; file {} already exists!".format(filePath, destPath))
elif behaviour =='overwrite':
os.remove(destPath)
# documentation states:
# On Windows, if dst already exists, OSError will be raised even if it is a file.
os.rename( filePath, destPath)
warnings.warn( "Warning!Overwriting file: {}.".format(destPath))
elif behaviour=='accumulate': #redundant but OK
addPost=0
while True:
newDestPath=destPath+"{:04d}".format(addPost)
if not os.path.exists(newDestPath):
break
addPost+=1
assert addPost < 10000, "Clean up the mess!"
os.rename( filePath, newDestPath)
else:
assert 0, "Unknown copy behaviour requested."
Additionally one might check for file permissions as, e.g., os.remove() may raise an exception. In this case, however, I assume that permissions are properly set by the OP.
I have my python code that runs a C++ code, which takes files in another folder as input.
I have my codes in folder A, and the input files are in folder B, and I have been trying this:
path = 'C:/pathToInputFiles'
dirs = os.listdir(path)
for path in dirs:
proc = subprocess.Popen([fullPathtoCppCode, inputFiles])
However, I keep receiving WindowsError: [Error 2] The system cannot find the file specified
The only way it works is when I put the C++ executable file in the same folder of the input files, which I am avoiding to do.
How can I make python reads the file path properly?
Try using os.path.join after your for statement.
path = os.path.join(directory, filename)
for example
def test(directory):
for filename in os.listdir(directory):
filename = os.path.join(directory, filename)
proc = subprocess.Popen([fullPathtoCppcode, inputFiles])
The code below searches within a directory for any PDFs and for each one it finds it moves into the corresponding folder which has '_folder' appended.
Could it be expressed in simpler terms? It's practically unreadable. Also if it can't find the folder, it destroys the PDF!
import os
import shutil
for root, dirs, files in os.walk(folder_path_variable):
for file1 in files:
if file1.endswith('.pdf') and not file1.startswith('.'):
filenamepath = os.path.join(root, file1)
name_of_file = file1.split('-')[0]
folderDest = filenamepath.split('/')[:9]
folderDest = '/'.join(folderDest)
folderDest = folderDest + '/' + name_of_file + '_folder'
shutil.move(filenamepath2, folderDest)
Really I want to traverse the same directory after constructing the variable name_of_file and if that variable is in a folder name, it performs the move. However I came across issues trying to nest another for loop...
I would try something like this:
for root, dirs, files in os.walk(folder_path_variable):
for filename in files:
if filename.endswith('.pdf') and not filename.startswith('.'):
filepath = os.path.join(root, filename)
filename_prefix = filename.split('-')[0]
dest_dir = os.path.join(root, filename_prefix + '_folder')
if not os.path.isdir(dest_dir):
os.mkdir(dest_dir)
os.rename(filepath, os.path.join(dest_dir, filename))
The answer by John Zwinck is correct, except it contains a bug where if the destination folder already exists, a folder within that folder is created and the pdf is moved to that location. I have fixed this by adding a 'break' statement within the inner for loop (for filename in files).
The code below now executes correctly. Looks for folder named as the pdf's first few characters (taking the prefix split at '-') with '_folder' at the tail, if it exists the pdf is moved into it. If it doesn't, one is created with the prefix name and '_folder' and pdf is moved into it.
for root, dirs, files in os.walk(folder_path_variable):
for filename in files:
if filename.endswith('.pdf') and not filename.startswith('.'):
filepath = os.path.join(root, filename)
filename_prefix = filename.split('-')[0]
dest_dir = os.path.join(root, filename_prefix + '_folder')
if not os.path.isdir(dest_dir):
os.mkdir(dest_dir)
os.rename(filepath, os.path.join(dest_dir, filename))
break
I've got a python script that deletes an entire directory and its subfolders, and I'd like to print out the number of files and folders removed. Currently, I have found some code from a different question posed 2010, but the answer I receive back is 16... If I right-click on the the folder it states that theres 152 files, 72 folders...
The code I currently have for checking the directory;
import os, getpass
user = getpass.getuser()
copyof = 'Copy of ' + user
directory = "C:/Documents and Settings/" + user
print len([item for item in os.listdir(directory)])
How can I extend this to show the same number of files and folders that there actually are?
To perform recursive search you may use os.walk.
os.walk(top, topdown=True, onerror=None, followlinks=False)
Generate the file names in a directory tree by walking the tree either top-down or bottom-up. For each directory in the tree rooted at
directory top (including top itself), it yields a 3-tuple (dirpath,
dirnames, filenames).
Sample usage:
import os
dir_count = 0
file_count = 0
for _, dirs, files in os.walk(dir_to_list_recursively):
dir_count += len(dirs)
file_count += len(files)
I was able to solve this issue by using the following code by octoback (copied directly);
import os
cpt = sum([len(files) for r, d, files in os.walk("G:\CS\PYTHONPROJECTS")])