While submittng a job on gogle cloud ML, I am getting an error where main training python file i.e. task.py is not able to import a function from a python script in util folder.
Generally, we write :
from util.xyz import abc
this is not getting called in the main task.py
It could be due to one of two reasons:
(1) You may be missing an __init__.py in your module directory (or util subdirectory)
(2) You may need to import the module as trainer.util.xyz (assuming your overall module is called trainer)
Related
I am running the following code:
pyinstaller --onefile main.py
main.py looks like:
import sys
import os
sys.path.append(r'C:\Model\Utilities')
from import_pythonpkg import *
......
import_pythonpkg.py looks like:
from astroML.density_estimation import EmpiricalDistribution
import calendar
import collections
from collections import Counter, OrderedDict, defaultdict
import csv
....
By running the pyinstaller on main.py, main.exe file is created successfully.
But when I run main.exe it gives error with astroML. If I move astroML to main.py from import_pythonpkg.py, there is no error with astroML. Now I get error with csv.
i.e. if I change my main.py to look as:
import sys
from astroML.density_estimation import EmpiricalDistribution
import os
sys.path.append(r'C:\Model\Utilities')
from import_pythonpkg import *
......
The astroML error is no longer present when I run main.exe.
There is no error with import calendar line in import_pythonpkg.py at all.
I am not sure how to handle this random error with packages when running main.exe after pyinstaller run.
import_pythonpkg is located at r'C:\Model\Utilities'
Edit:
Error with main.exe looks as following even though the original main.py runs fine. Pyinstaller was even able to let me create the main.exe without error.
Traceback (most recent call last):
File "main.py", line 8, in <module>
File "C:\Model\Utilities\import_pythonpkg.py", line 1, in <module>
from astroML.density_estimation import EmpiricalDistribution
ImportError: No module named astroML.density_estimation
[29180] Failed to execute script main
I believe PyInstaller is not seeing import_pythonpkg. In my experience, when adding to the path or dealing with external modules and dlls, PyInstaller will not go searching for that, you have to explicitly tell it to do so. It will compile down to an .exe properly because it just ignores it, but then won't run. Check to see if there are any warnings about missing packages or modules when you run your PyInstaller command.
But how to fix it...If indeed this is the issue (which I am not sure that it is) you can try 3 things:
1) move that package into your working directory and avoid using sys.path.append. Then compile with PyInstaller to so see if this works, then you know the issue is that pyinstaller is failing to find import_pythonpkg. You can stop there if this works.
2) explicitly tell PyInstaller to look there. You can use the hidden-import tag when compiling with PyInstaller to let it know (give it the full pathname).
--hidden-import=modulename
for more info, check here: How to properly create a pyinstaller hook, or maybe hidden import?
3) If you use the spec file that PyInstaller creates, you can try adding a variable call pathex to tell PyInstaller to search there for things:
block_cipher = None
a = Analysis(['minimal.py'],
pathex=['C:\\Program Files (x86)\\Windows Kits\\10\\example_directory'],
binaries=None,
datas=None,
hiddenimports=['path_to_import', 'path_to_second_import'],
hookspath=None,
runtime_hooks=None,
excludes=None,
cipher=block_cipher)
pyz = PYZ(a.pure, a.zipped_data,
cipher=block_cipher)
exe = EXE(pyz,... )
coll = COLLECT(...)
for more information on spec files: https://pyinstaller.readthedocs.io/en/stable/spec-files.html
(notice you can also add hiddenimports here)
This answer may also prove helpful: PyInstaller - no module named
It is about to module which loaeded on your computer. If your IDE is different from your environment, you have to load same modules on your device via pip. Check the modules on CMD screen and complete the missing modules.
Sometimes you must load the modules all IDEs on your device. In my case, there were two IDEs (pycharm and anaconda). I used pycharm but pyinstaller used anaconda's modules so i unistalled anaconda and tried again. now it works..
I am making some tests to learn using Cython. I am using pyximport to import a Cython module into a Python script and I found the following problem:
If I execute the script while working in one directory, building the module fails. The error message says:
from . import Pipeline
ImportError: Building module mymodule failed: ["ImportError: Building module _laplace failed: ['ImportError: cannot import name Pipeline\n']\n"]
On the contrary, if I copy my Python script and the Cython module in a subfolder, everything works fine.
Can anyone tell me why this is happening?
I am learning python and am trying to run the code on this github project.
https://github.com/rcfbanalysis/rcfbscraper
When I try to run a command such as the following
python "C:\Python\rcfbscraper-master\ESPN_Scraper\ESPN_Scraper\spiders\espnSpider.py"
I get the error No Module named ESPN_Scraper.items.
This is the offending line
from ESPN_Scraper.items import GameItem
From what I can tell is ESPN_Scraper items.py is not in your sys path. one quick work around for this would be to put the ESPN_Scraper directory in the same directory as your espnspider.py file.
Take a look at the docs https://docs.python.org/2/tutorial/modules.html section 6.1.2
I'm working on a large open source Python project, which has modules used by both the project and other projects. The goal is to move some of these modules out to a new "library" project that can then be imported by the original project and other projects.
To make this transition smooth, the thought was to copy the modules over to the new project, and have the original project then use the new import. However, to allow other project to have time to migrate later, the thought was to have the original module redirect the import.
For example, the usage is like this in repo 'neutron' (other projects could do the same):
cat neutron/consumer.py
from neutron.redirected import X
print(X)
The in the new 'neutron_lib' project created, the module looks like this (the same as what the original was in project 'neutron'):
cat ../neutron-lib/neutron_lib/redirected.py
X = 5
In the 'neutron' project, I'm trying to do this as the redirect module:
cat neutron/redirected.py
import neutron_lib.redirected
import sys
sys.modules['neutron.redirected'] = neutron_lib.redirected
When I run pylint, it gives these errors:
************* Module neutron.redirected
E: 1, 0: No name 'redirected' in module 'neutron_lib' (no-name-in-module)
************* Module neutron.consumer
E: 1, 0: No name 'X' in module 'neutron.redirected' (no-name-in-module)
If I run this, it runs fine, and consumer.py prints '5'. If I use ipython and load consumer.py, I can see 'X' in dir() output.
Any idea why I'm getting this pylint error? Is it a false error? Is there a way to override it?
Looks like, when running under tox, I can add the following to .pylintrc to hide the errors/warnings
no-name-in-module
nonstandard-exception
When I run pylint it passes now, as does running the Unit tests. Just wish I understood why I'm getting these errors/warnings though.
Here's my code configuration:
__init__py
./module_1/foo.py
./module_1/__init__.py
./module_2/bar.py
./module_2/__init__.py
My goal is to be able to import the foo module from bar.py, but I'm getting quite confused on how to do it.
Something such as:
from ..module_1.foo import *
Will raise the following error:
ValueError: Attempted relative import in non-package
EDIT:
Ideally I'd like to be able to run my script in the following fashion:
python ./module1/foo.py
You haven't shown how you are invoking the script, but you need to ensure that your scripts are actually packages in your python path. That's basically what the error message is telling you, you were trying to import a "non-package". You probably don't have your top-level in the python path. For example ...
If your top-level module is called app and your configuration is
<path-to-app>/app/__init__py
<path-to-app>/app/module_1/foo.py
<path-to-app>/app/module_1/__init__.py
<path-to-app>/app/module_2/bar.py
<path-to-app>/app/module_2/__init__.py
You can run your script as follows.
cd <path-to-app>
PYTHONPATH=$PWD python -m app.module_2.bar
Works for me.