IsADirectoryError: [Errno 21] Is a directory: '/app' when deploying on Heroku - flask

I've been having trouble deploying my Flask app on Heroku. My current problem is Heroku can't locate the files in my static folder. After trying suggestions on SO, I am now getting this error:
with open(file_dir, encoding="utf-8") as f:
IsADirectoryError: [Errno 21] Is a directory: '/app'
Here's one of the suggestions I found to help Heroku find the encouragement.txt file:
# Absolute path to the file
file_dir = os.path.dirname(os.path.abspath('encouragement.txt'))
# Absolute path to this file's root directory
parent_dir = os.path.join(file_dir, os.pardir)
app_dir = os.path.join(parent_dir, '/app')
The '/app' was added because one of the posts mentioned that Heroku had an app directory as the parent directory, so instead of 'static/encouragement.txt', it would be 'app/static/encouragement.txt'. The same error was given with and without joining '/app'.
Here's where the error came from:
#app.route("/")
def index():
# Read data from the file as str
with open(file_dir, encoding="utf-8") as f:
messages = f.read()
I tried using heroku bash command to check the file system structure and confirm the app directory but the command executed and returned blank.
My files are organized as follow:
MyProject
> static
> encouragement.txt
app.py
Not sure if it's needed, but I'm using windows. Thank you.
Update:
I tried adding index.php and composer.json as a couple tutorials suggested. They didn't work for the txt files; not sure if they helped with the other static files though. Not the best solution, but I ended up just copying and pasting all the txt content to my app.py.

I ran into this issue today after doing a bit of "clean-up" to my environment variables/configs. The issue is you're trying to open a directory itself as if it were a file, which causes the error. I recommend the answer in this article (as shown).
import os
# path to your file
file_name = r'/app/static/encouragement.txt'
print(os.path.isfile(file_name))
with open(file_name, 'r', encoding='utf-8') as f:
lines = f.readlines()
print(lines)

Related

Django static files templatetags.static

I am trying to open a json file in python. It is in a directory called json in static directory of my app. I configured the static files as per the given documentation.
But I get this error on opening files in python using open() function
def send(request):
file = static('accounts/json/credentials.json')
f = open(file, "r")
return HttpResponse("click here <a href='" + file + "'> asd</a>")
The above code generates a FileNotFoundError
No such file or directory: '/static/accounts/json/credentials.json'
I also used this code
def send(request):
file = static('accounts/json/credentials.json')
return HttpResponse("click here <a href='" + file + "'> asd</a>")
The above code successfully gave me the response. And On clicking the link asd it is working fine and opens the JSON file.
I have googled a bit and also went through few stackoverflow questions but not sure what the actual error is, Can anyone help me in finding out.
Thanks
Presuming static is the helper function from the static templatetags, it gives you the URL path to STATIC_URL. It doesn't give you the file path. That is set in STATIC_ROOT and you will need to join it yourself:
path = os.path.join(settings.STATIC_ROOT, 'accounts/json/credentials.json')
f = open(path, r)
Note though it doesn't make sense to to try and put this file into the text of a link; your second snippet is correct if you want to do that.
In your settings.py file there should be a variable called BASE_DIR defined. You need to prepend that on your file name. Something like:
file = static(settings.BASE_DIR + '/accounts/json/credentials.json')

How to download a snappy.parquet file from s3 using Boto in Python

I'm new to this, and trying to download a snappy.parquet file from Amazon s3 I can later convert to CSV file.
I tried working with the following example I've found online, and I get an empty folder. can anyone please help me?
import boto
import sys, os
from boto.s3.key import Key
from boto.exception import S3ResponseError
DOWNLOAD_LOCATION_PATH =""
BUCKET_NAME = ""
AWS_ACCESS_KEY_ID= ""
AWS_ACCESS_SECRET_KEY = ""
conn = boto.connect_s3(AWS_ACCESS_KEY_ID, AWS_ACCESS_SECRET_KEY)
bucket = conn.get_bucket(BUCKET_NAME)
#goto through the list of files
bucket_list = bucket.list()
for l in bucket_list:
key_string = str(l.key)
s3_path = DOWNLOAD_LOCATION_PATH + key_string
try:
print ("Current File is ", s3_path)
l.get_contents_to_filename(s3_path)
except (OSError, S3ResponseError) as e:
pass
# check if the file has been downloaded locally
if not os.path.exists(s3_path):
try:
os.makedirs(s3_path)
except OSError as exc:
# let guard againts race conditions
import errno
if exc.errno != errno.EEXIST:
raise
The script you are using appears to recursively download the contents of the specified S3 bucket (BUCKET_NAME) to the specified local directory (DOWNLOAD_LOCATION_PATH). FWIW, I notice this script looks like it comes from here.
The "Current File is ..." output line should show you the progress of these files being written. One problem you might be having is due to this line:
s3_path = DOWNLOAD_LOCATION_PATH + key_string
If you had specified DOWNLOAD_LOCATION_PATH at the top as a directory without a trailing '/' character, e.g. like this:
DOWNLOAD_LOCATION_PATH = '/tmp/my_dir'
then the files being downloaded would be written not underneath the /tmp/my_dir directory, but directly in /tmp/ with a my_dir prefix on each filename! You can fix this by changing this line to:
s3_path = os.path.join(DOWNLOAD_LOCATION_PATH, key_string)
Other than that, the script appears to work alright. You may want to add this line at the very top:
from __future__ import print_function
if you are still using Python 2.x, otherwise the print output will look a bit odd (print will think you are printing a 2-Tuple).
Your question also makes it sound like you really only want/need to download a single file from the bucket -- if so, this isn't really a great script to be using, since it's downloading everything.

I don't understand how this "os.join" function is working? I am getting errors constantly and no reading on os functions is helping me

Here's the code
sys.path.append( "../tools/" )
from parse_out_email_text import parseOutText #(its just another .py file that has a function I wrote)
from_sara = open("from_sara.txt", "r")
from_chris = open("from_chris.txt", "r")
from_data = []
word_data = []
temp_counter = 0
for name, from_person in [("sara", from_sara), ("chris", from_chris)]:
for path in from_person:
### only look at first 200 emails when developing
### once everything is working, remove this line to run over full dataset
temp_counter += 1
if temp_counter < 200:
path = os.path.join('..', path[:-1]) #(THIS IS THE PART I CAN'T GET MY HEAD AROUND)
print path
email = open(path, "r")
email.close()
print "emails processed"
from_sara.close()
from_chris.close()
When I run this, it gives me an error as shown below:
Traceback (most recent call last):
..\maildir/bailey-s/deleted_items/101.
File "C:/Users/AmitSingh/Desktop/Data/Udacity/Naya_attempt/vectorize_text.py", line 47, in <module>
email = open(path, "r")
IOError: [Errno 2] No such file or directory: '..\\maildir/bailey-s/deleted_items/101.'
I don't even have this """'..\maildir/bailey-s/deleted_items/101.'""" directory path on my laptop, I tried to change the path by replacing the '..' in the code by the actual path name to the folder where I keep all the files, and nothing changes.
path = os.path.join('..', path[:-1])
This code is part of an online course on machine learning and I have been stuck at this point for 3 hours now. Any help would be really appreciated.
(P.S. This is not a homework question and there are no grades attached to this, its a free online course)
your test data is not there so it cannot find it. you should run start-up code again and make sure the necessary maildir are all there.
Go to tools inside your udacity project directory and run startup.py.
It is about 400 Mb so sit back and relax!
I know this is extremely late, but I found this post after having the exact same problem.
All the answers that I found here and on other sites, even the issue requests in the original github, were just "run startup.py" I already did that. However, it was telling me:
Traceback (most recent call last):
File "K:\documents\Udacity\Mini-Projects\ud120-projects\text_learning\vectorize_text.py", line 48, in <module>
email = open(path, "r")
FileNotFoundError: [Errno 2] No such file or directory: '..\\maildir/bailey-s/deleted_items/101.'
Just like yours. I then found where this file was located and it was indeed on my computer
I added 'tools' to the os.path.join() line as you can see here:
for name, from_person in [("sara", from_sara), ("chris", from_chris)]:
for path in from_person:
### only look at first 200 emails when developing
### once everything is working, remove this line to run over full dataset
temp_counter += 1
if temp_counter < 200:
#path = os.path.join('..', path[:-1]) <---original
path = os.path.join('..','tools', path[:-1])
print(path)
email = open(path, "r")
This worked for me finally. So, I hope it helps anyone else that stumbles on this problem in the future.
Also, I noticed on some examples I found of other repos of the lessons. Their 'tools' folder was named 'utils'.
Here is an example, this is a repo that someone tweaked to use jupyter notebooks to run the lessons So, use the one that you have.
In your Udacity course folder, first go to tools directory, check if you have maildir folder present and if it has got subfolders in it, if they are present then go back to text_learning/vectorize_text.py, find this line of code path = os.path.join('..', path[:-1]), change it to path = os.path.join('../tools/', path[:-1]),
On terminal, cd text_learning , then python vectorize_text.py, this should solve the issue.
If this does not solve the issue, then Go to tools inside your udacity project directory and run startup.py. Wait till the process is complete
Repeat step 1.

Django: No such file or directory

I have a process that scans a tape library and looks for media that has expired, so they can be removed and reused before sending the tapes to an offsite vault. (We have some 7 day policies that never make it offsite.) This process takes around 20 minutes to run, so I didn't want it to run on-demand when loading/refreshing the page. Rather, I set up a django-cron job (I know I could have done this in Linux cron, but wanted the project to be as self-contained as possible) to run the scan, and creates a file in /tmp. I've verified that this works -- the file exists in /tmp from this morning's execution. The problem I'm having is that now I want to display a list of those expired (scratch) media on my web page, but the script is saying that it can't find the file. When the file was created, I use the absolute filename "/tmp/scratch.2015-11-13.out" (for example), but here's the error I get in the browser:
IOError at /
[Errno 2] No such file or directory: '/tmp/corpscratch.2015-11-13.out'
My assumption is that this is a "web root" issue, but I just can't figure it out. I tried copying the file to the /static/ and /media/ directories configured in django, and even in the django root directory, and the project root directory, but nothing seems to work. When it says it cant' find /tmp/file, where is it really looking?
def sample():
""" Just testing """
today = datetime.date.today() #format 2015-11-31
inputfile = "/tmp/corpscratch.%s.out" % str(today)
with open(inputfile) as fh: # This is the line reporting the error
lines = [line.strip('\n') for line in fh]
print(lines)
The print statement was used for testing in the shell (which works, I might add), but the browser gives an error.
And the file does exist:
$ ls /tmp/corpscratch.2015-11-13.out
/tmp/corpscratch.2015-11-13.out
Thanks.
Edit: was mistaken, doesn't work in python shell either. Was thinking of a previous issue.
Use this instead:
today = datetime.datetime.today().date()
inputfile = "/tmp/corpscratch.%s.out" % str(today)
Or:
today = datetime.datetime.today().strftime('%Y-%m-%d')
inputfile = "/tmp/corpscratch.%s.out" % today # No need to use str()
See the difference:
>>> str(datetime.datetime.today().date())
'2015-11-13'
>>> str(datetime.datetime.today())
'2015-11-13 15:56:19.578569'
I ended up finding this elsewhere:
today = datetime.date.today() #format 2015-11-31
inputfilename = "tmp/corpscratch.%s.out" % str(today)
inputfile = os.path.join(settings.PROJECT_ROOT, inputfilename)
With settings.py containing the following:
PROJECT_ROOT = os.path.abspath(os.path.dirname(__file__))
Completely resolved my issues.

Zipping a file from a directory and placing it in another Directory

I am trying to set up a program to put my minecraft server world into a zip and place it into another directory on another drive (/media/500gb/MinecraftWorldBackups)
But I keep getting this error
Although the folder doesn't contain a folder or file called 'h'
What do I need to do to fix this I believe it is due to file and folder?
#!/usr/bin/env python
import time, zipfile
while True:
FileName = 'MinecraftBackup_' + str(int(time.time()))
Path = '/home/bertie/Desktop/FeedTheBeastServer/world/'
print(FileName)
Zip = zipfile.ZipFile('/media/500gb/MinecraftWorldBackups/'+FileName+'.zip','w')
for each in Path:
print(each)
try: Zip.write(Path + each)
except IOError: None
Zip.Close()
print('Done')
time.sleep(60)