Dockerfile COPY command doesn't make my file available with I run the image - dockerfile

I am using Linux Ubuntu 20.04, Pycharm Pro, Python 3.9, Docker (installed a couple weeks ago, don't remember ver).
I have a Python project in the path (
/home/crusty.user/PythonProjects/NoLegals
There are 4 files in this path: main.py, utils.py, NoLegals_Config.csv, Dockerfile
The csv file acts as a config to tell the python project which parts of the research to do, or not to. It reads a line, with a Y or N. Pretty simple. It works great in Linux and in windows.
From the path above, I run sudo docker build -t nolegals .
Everything runs successfully.
When I try to run the Dockerfile (sudo docker run nolegals) it fails when it gets to the csv file with the error:
FileNotFoundError: [Errno 2] No such file or directory: '/home/crusty.user/PythonProjects/NoLegals/NoLegals_Config.csv'
In my Dockerfile I have:
WORKDIR /NoLegals
Further down I have:
COPY NoLegals_Config.csv /
COPY main.py /
COPY utils.py /
--There's a bunch of otherstuff for setting up the environment, libraries, etc. all of which runs successfully on the build. Also, I don't get a failure regarding the path of the csv file during build. I've been digging around and I've learned that it might having something to do with not being able to find the csv file within the Docker image when it builds, but it finds the main.py and utils.py just fine. There is a line of code in the Python main.py file that points to the location of the csv file dynamically as a suggestion to fix the problem but this too has failed. The path the error prints is also the correct path to the csv file.
#this works in linux, just not in the Dockerfile
filename = r'NoLegals_Config.csv
filepath = os.path.join(os.getcwd(), filename)
print(filepath)
I've tried LOTS of different things in that COPY NoLegals_config.csv / line, but to no avail. I appreciate any suggestions.
I've tried various forms of the COPY. Previous to the one listed was using the syntax:
COPY <source-path> <destination-Path> COPY NoLegals_Config.csv / COPY <full path of source> </NoLegals/NoLegals_Config.csv>
I've tried some other things that I can't recall.

What I ended up doing was fully qualifying the COPY statements.
WORKDIR /NoLegals
COPY NoLegals_Config.csv /NoLegals/NoLegals_Config.csv
COPY main.py /NoLegals/main.py
COPY utils.py /NoLegals/utils.py
CMD ["python"."/NoLegals/main.py"]
Is there a simpler way I could have written this?

Related

Adding a JSON data file to GCP Dataflow package and using it inside the code

I want to add a JSON file to my Dataflow (Apache Beam) package and use it inside the code.
I've seen several questions on Stack Overflow with different answers, and I tried the recommended approach with a MANIFEST.in and adding data_files to the setup.py file. But nothing I tried works for me.
Here is my current setup.
(I have mapping.json in both the common folder and the root folder for testing purposes.)
MANIFEST.in
recursive-include common *.json
setup.py
import setuptools
setuptools.setup(
packages=setuptools.find_packages(),
data_files=[
("common", ["mapping.json"])
],
include_package_data=True,
install_requires=[
'apache-beam[gcp]==2.31.0',
'python-dateutil==2.8.1'
],
)
Using the file inside common/config.py
import json
from pathlib import Path
def _load_category_theme_mapping(file_name):
path = Path(__file__).parent / file_name
with path.open('r', encoding='utf-8') as file:
return json.load(file)
mapping = _load_category_theme_mapping("mapping.json")
I'm using Flex Templates to run my Dataflow job and I copy the common folder to the target common folder.
When I run the Dataflow job with this setup, it just throws an error.
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.7/site-packages/common/category_theme_mapping.json'
I tried moving the .json file outside of the common folder (into the root folder) and changed the code (and the Dockerfile) accordingly to read from the base folder.
Then I changed the setup.py file to have the data_files to (".", ["mapping.json"] and MANIFEST.in to have include *.json, but it still fails.
I also tried without having a MANIFEST.in, but then the launcher fails without any informative log.
Any idea what I'm doing wrong?

How to make Dockerfile COPY work for windows absolute paths?

How do I make my COPY command work for absolute paths on Windows? I tried git-bash, cmd and powershell consoles to build with docker build -t custom-maven-image .
# Dockerfile
FROM maven:3-openjdk-11-slim
# these are three versions of copy command I tried
COPY C:/Users/myuser/.m2 /root/.m2
COPY /C/Users/myuser/.m2 /root/.m2
COPY /c/Users/myuser/.m2 /root/.m2
What I get is an error:
...
#5 ERROR: "/C/Users/myuser/.m2" not found: not found
UPDATE:
Thx #Jeremy for bugs references and now I see that docs clearly says:
COPY obeys the following rules:
The path must be inside the context of the build; you cannot
COPY ../something /something, because the first step of a docker build
is to send the context directory (and subdirectories) to the docker
daemon.
All the resources need to be in the dir that you run the build, i.e. where your Dockerfile is. You cant use an absolute path from elsewhere, think of it from the build perspective - where is the build happening - in the Dockerfile? It can run commands but those resources need to be public.
https://github.com/moby/moby/issues/4592
https://github.com/docker/compose/issues/4857

how can i properly copy this file to another directory in Ubuntu 14.0

cp -i /usr/local/lib/python2.7/dist-packages/django/template/admin/base_site.html /usr/local/lib/python2.7/dist-packages/django/template/admin
I am trying to 'CUSTOMIZE MY PROJECT TEMPLATE' from the Django 1.8 POLLS tutorial
https://docs.djangoproject.com/en/1.8/intro/tutorial02/
but no avail, i am having a hard time trying to copy the "base_site.html" file to the "django/contrib/admin/templates"
help me learn!
You may misunderstand the tutorial, It said: copy it from the default directory into your custom directory, and make changes, it seems that you command above is trying to duplicate the file.
first, find your Django source file location:
python -c "
import sys
sys.path = sys.path[1:]
import django
print(django.__path__)"
you will get a link, go deep into the folder, find that template, then copy the base_site.html to your project's template folder.

Python install module from repo v. locally

I am trying to install a homegrown package that will be used , but python package installation is still a bit of a quagmire for me, and I haven't gotten this to work.
I created a package using setup.py sdist, which I uploaded to a repository
I am trying to install my package on another machine. I tried three methods, each time on an entirely clean machine. But none are doing what I want them to.
Method 1
easy_install http://mysite/mypkg.zip
RESULT: mypkg.egg gets added to \Python27\Lib\site-packages. But none of my folder structure is there
Method 2
pip install http://mysite/mypkg.zip
RESULT: two folders, mypkg and mypkg-1.0-py2.7.egg-info, get added to Python27\Lib\site-packages. All of the files seem to be there. But when I got to import or run nosetests on the folder, I get all sorts of import errors that reference mypkg modules. I have played with PATH and PYTHONPATH to get all variations of including the folder, but nothing has worked.
Method 3
download .zip
extract locally
add folder to PATH
run easy_install . in the local dir
RESULT: unpacks pkg locally. When I run nosetests on this folder, everything runs as expected.
Thing is, I don't want each user to have to do all of the steps in Method 3. I will eventually be running nosetests in a .bat file that does various things with the output. I don't want every user to have to modify the .bat file to indicate where the testsuite is located. Which is why Python27\Lib\site-packages appealed to me.
Any insight as to why these three methods behave so differently would be very helpful!

Do I need to update the path when installing a module?

So I'm trying to install a module, specifically xlutils. I've read through the resources that I've linked at the bottom, but none of those resources have allowed me to successfully install and import the module. I'm running Windows 8 and using Python 2.7.
I downloaded the .tar.gz file containing xlutils, and unpacked it to C:\Python which was then a .tar file, so I unpacked that to the same folder. This created a folder, xlutils, which looked like it contained what I need. I also read somewhere that these should be stored in site-packages, so I moved it there.
But when I run import commands, they don't work, just tell me the module couldn't be found. When I look at the path browser, it doesn't see the folder, but I'm certain it's in there. That leads me to wonder, do I need to do something to manually update what the path browser can view?
Note that I've also already tried going to the command line, navigating to the folder containing the module, and typing python setup.py install but that just tells me that the term "python" is not found. In general, my command line always does this though. Usually I have to type .\python instead to run Python from the command line, but I also tried doing that here (i.e navigating to the folder and typing .\python setup.py install but it still says the same thing).
Also note that I can import numpy and scipy just fine, and I can see them in the path browser--not sure why those work while this one doesn't.
Resources I've already read but hasn't solved my problem:
(... Well, I tried providing the resources I've already viewed, but can't so many links with such low reputation. But basically, I've read the first ten links on a Google search and two or three past Stack questions and answers.)
Solutions I see:
You can use the absolute path C:\Python27\python.exe setup.py install
You can add the Python directory C:\Python27\ to your path variable before running python setup.py install