I'm trying to use HDFS sensor in airflow. But am getting the below error
[2020-06-30 07:22:57,636] {dagbag.py:246} ERROR - Failed to import: /airflow/dags/test_s3_file.py
Traceback (most recent call last):
File "/airflow/airflow-staging/lib/python3.6/site-packages/airflow/models/dagbag.py", line 243, in process_file
m = imp.load_source(mod_name, filepath)
File "/airflow/airflow-staging/lib64/python3.6/imp.py", line 172, in load_source
module = _load(spec)
File "<frozen importlib._bootstrap>", line 684, in _load
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/airflow/dags/test_s3_file.py", line 3, in <module>
from S3HDFSObjectSensor import S3HDFSObjectSensor
File "/airflow/plugins/S3HDFSObjectSensor.py", line 6, in <module>
from airflow.hooks.hdfs_hook import HDFSHook
File "/airflow/airflow-staging/lib/python3.6/site-packages/airflow/hooks/hdfs_hook.py", line 24, in <module>
from snakebite.client import Client, HAClient, Namenode, AutoConfigClient
File "/airflow/airflow-staging/lib/python3.6/site-packages/snakebite/client.py", line 1473
baseTime = min(time * (1L << retries), cap);
^
SyntaxError: invalid syntax
From google, found that we will have this issue if we are using python3. Could you please let me know if there is any work around for this other than using python2.
from airflow.plugins.S3HDFSObjectSensor import S3HDFSObjectSensor
t1 = S3HDFSObjectSensor(
task_id='t1',
poke_interval=sensor_poke_interval,
timeout=sensor_timeout,
object_path=check_file_in_hdfs,
trigger_rule='one_failed',
dag=dag)
Below is my AIRFLOW_HOME directory structure
airflow
+-- dags
+-- plugins
+-- S3HDFSObjectSensor.py
Thanks
It worked after installing 'pip3 install snakebite-py3'
Related
I'm running my django-app-api server on docker and I'm stuck at a particular place. The server doesnt spawn due to the following error :
TypeError: init() got an unexpected keyword argument 'providing_args'
I'm getting the following error trace while running my django-api docker container using docker-compose
myproject-api-webserver | 2022-09-23 11:09:26,477 myprojectnetwork.settings INFO ALLOWED_HOSTS environment variable ignored.
myproject-api-webserver | Traceback (most recent call last):
myproject-api-webserver | File "manage.py", line 25, in <module>
myproject-api-webserver | execute_from_command_line(sys.argv)
myproject-api-webserver | File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line
myproject-api-webserver | utility.execute()
myproject-api-webserver | File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 420, in execute
myproject-api-webserver | django.setup()
myproject-api-webserver | File "/usr/local/lib/python3.8/site-packages/django/__init__.py", line 24, in setup
myproject-api-webserver | apps.populate(settings.INSTALLED_APPS)
myproject-api-webserver | File "/usr/local/lib/python3.8/site-packages/django/apps/registry.py", line 91, in populate
myproject-api-webserver | app_config = AppConfig.create(entry)
myproject-api-webserver | File "/usr/local/lib/python3.8/site-packages/django/apps/config.py", line 193, in create
myproject-api-webserver | import_module(entry)
myproject-api-webserver | File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
myproject-api-webserver | return _bootstrap._gcd_import(name[level:], package, level)
myproject-api-webserver | File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
myproject-api-webserver | File "<frozen importlib._bootstrap>", line 991, in _find_and_load
myproject-api-webserver | File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
myproject-api-webserver | File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
myproject-api-webserver | File "<frozen importlib._bootstrap_external>", line 843, in exec_module
myproject-api-webserver | File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
myproject-api-webserver | File "/usr/local/lib/python3.8/site-packages/cacheops/__init__.py", line 8, in <module>
myproject-api-webserver | from .query import *
myproject-api-webserver | File "/usr/local/lib/python3.8/site-packages/cacheops/query.py", line 31, in <module>
myproject-api-webserver | from .invalidation import invalidate_obj, invalidate_dict, no_invalidation
myproject-api-webserver | File "/usr/local/lib/python3.8/site-packages/cacheops/invalidation.py", line 10, in <module>
myproject-api-webserver | from .signals import cache_invalidated
myproject-api-webserver | File "/usr/local/lib/python3.8/site-packages/cacheops/signals.py", line 3, in <module>
myproject-api-webserver | cache_read = django.dispatch.Signal(providing_args=["func", "hit"])
myproject-api-webserver | TypeError: __init__() got an unexpected keyword argument 'providing_args'
My Django version==4.1.1 & Django CacheOps==6.0.0
I've mentioned the Django & Django CacheOps since I believe this could be due to version mismatch and subsequent changes that have to made, but I do not know where to make the changes.
Also note that , the moment I upgrade to django-cacheops==6.1.0 (latest) , i get the following error which I have posted as a different question.
Django rest-api - attributeerror: 'str' object has no attribute '_meta'
It really looks like the cacheops package in version 6.0.0 still passes a keyword argument providing_args that was apparently supported up to Django 2.2 (docs), but definitely no longer in the current Django 4.1 (docs). Cacheops fixed this on 2021-02-19 (github commit), so you definitely need a version newer than that.
I am trying to run the standard django migration commands, ex., python3 manage.py makemigrations, and continuously get a syntax error on module import with sphinxsearch.
I know it's deprecated, but unfortunately I am working on small additions to the site and cannot move to elasticsearch yet.
Python version : 3.5.3
System : Debian (AWS server)
Sphinx version: 3.5.4
sphinxsearch version: 0.1
Error message:
Traceback (most recent call last):
File "manage.py", line 13, in <module>
execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.5/dist-packages/django/core/management/__init__.py", line 371, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python3.5/dist-packages/django/core/management/__init__.py", line 347, in execute
django.setup()
File "/usr/local/lib/python3.5/dist-packages/django/__init__.py", line 24, in setup
apps.populate(settings.INSTALLED_APPS)
File "/usr/local/lib/python3.5/dist-packages/django/apps/registry.py", line 89, in populate
app_config = AppConfig.create(entry)
File "/usr/local/lib/python3.5/dist-packages/django/apps/config.py", line 90, in create
module = import_module(entry)
File "/usr/lib/python3.5/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 986, in _gcd_import
File "<frozen importlib._bootstrap>", line 969, in _find_and_load
File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 669, in exec_module
File "<frozen importlib._bootstrap_external>", line 775, in get_code
File "<frozen importlib._bootstrap_external>", line 735, in source_to_code
File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
File "/usr/local/lib/python3.5/dist-packages/sphinxsearch/__init__.py", line 75
SPH_ATTR_MULTI = 0X40000000L
^
SyntaxError: invalid syntax
I've tried updating the system and have changed sphinx version, but I received the same error, just referencing a different file. Please note that the ini.py file is showing as blank when I navigate to it manually.
The error is caused by the trailing L in SPH_ATTR_MULTI = 0X40000000L and this is a sign that this code is meant to work with python 2, not python 3. In python 2 the trailing L is a indication that the value is type long instead integer and it is used for large numbers, but in python 3 integer and long were unified, so the trailing L is giving an SyntaxError.
Use python 2.7 if sphinx is absolutely necessary.
I was able to remove the sphinxsearch module as it wasn't necessary, install the sphinx 4.0 release.
That gave me a context error with "str = f'{item}'" format, so I changed it to "str = "% ..." % (item)" format manually within base.py ( in /usr/local/lib/python3.5/dist-packages/sphinxsearch/base.py) and it worked properly with python 3.5
It seems its just that a weird mid-tier version is necessary.
So I want to build a Docker image with PyTorch Lightning that can be used with AWS lambda. However, when the function is invoked it raises an OS Error, that claims it uses a Read-only file system and wandb.py wants to write something.
I tried these things:
Overwrite the wandb.py file of pytroch lightning, that it does not init wandb --> Raises error
Execute a python script in Dockerfile, that the files are created on docker build and exist, when invoking the lambda function --> Same OS error
Does someone know a way to skip the wandb.py?
This is the error message:
START RequestId: ddae284d-4f32-4dc6-8160-d1fa62ba9772 Version: $LATEST
OpenBLAS WARNING - could not determine the L2 cache size on this system, assuming 256k
[ERROR] OSError: [Errno 30] Read-only file system: '/home/sbx_user1051'
Traceback (most recent call last):
File "/var/lang/lib/python3.8/imp.py", line 234, in load_module
return load_source(name, filename, file)
File "/var/lang/lib/python3.8/imp.py", line 171, in load_source
module = _load(spec)
File "<frozen importlib._bootstrap>", line 702, in _load
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 783, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/var/task/inference.py", line 5, in <module>
import pytorch_lightning as pl
File "/var/lang/lib/python3.8/site-packages/pytorch_lightning/__init__.py", line 63, in <module>
from pytorch_lightning.callbacks import Callback
File "/var/lang/lib/python3.8/site-packages/pytorch_lightning/callbacks/__init__.py", line 25, in <module>
from pytorch_lightning.callbacks.swa import StochasticWeightAveraging
File "/var/lang/lib/python3.8/site-packages/pytorch_lightning/callbacks/swa.py", line 26, in <module>
from pytorch_lightning.trainer.optimizers import _get_default_scheduler_config
File "/var/lang/lib/python3.8/site-packages/pytorch_lightning/trainer/__init__.py", line 18, in <module>
from pytorch_lightning.trainer.trainer import Trainer
File "/var/lang/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 30, in <module>
from pytorch_lightning.loggers import LightningLoggerBase
File "/var/lang/lib/python3.8/site-packages/pytorch_lightning/loggers/__init__.py", line 31, in <module>
from pytorch_lightning.loggers.wandb import _WANDB_AVAILABLE, WandbLogger # noqa: F401
File "/var/lang/lib/python3.8/site-packages/pytorch_lightning/loggers/wandb.py", line 34, in <module>
import wandb
File "/var/lang/lib/python3.8/site-packages/wandb/__init__.py", line 131, in <module>
api = InternalApi()
File "/var/lang/lib/python3.8/site-packages/wandb/apis/internal.py", line 17, in __init__
self.api = InternalApi(*args, **kwargs)
File "/var/lang/lib/python3.8/site-packages/wandb/sdk/internal/internal_api.py", line 73, in __init__
self._settings = Settings(
File "/var/lang/lib/python3.8/site-packages/wandb/old/settings.py", line 25, in __init__
self._global_settings.read([Settings._global_path()])
File "/var/lang/lib/python3.8/site-packages/wandb/old/settings.py", line 105, in _global_path
util.mkdir_exists_ok(config_dir)
File "/var/lang/lib/python3.8/site-packages/wandb/util.py", line 687, in mkdir_exists_ok
os.makedirs(path)
File "/var/lang/lib/python3.8/os.py", line 213, in makedirs
makedirs(head, exist_ok=exist_ok)
File "/var/lang/lib/python3.8/os.py", line 213, in makedirs
makedirs(head, exist_ok=exist_ok)
File "/var/lang/lib/python3.8/os.py", line 223, in makedirs
mkdir(name, mode)
END RequestId: ddae284d-4f32-4dc6-8160-d1fa62ba9772
REPORT RequestId: ddae284d-4f32-4dc6-8160-d1fa62ba9772 Duration: 27000.33 ms Billed Duration: 27001 ms Memory Size: 10240 MB Max Memory Used: 241 MB
Unknown application error occurred
You need to make sure you have write access somewhere.
Then you can use wandb environment variables to modify the default location where files are saved locally, in particular look at WANDB_DIR, WANDB_CONFIG_DIR and WANDB_CACHE_DIR.
The target is to create a pip package of a Django project.
The goal is to make the web application quickly installable anywhere just by doing pip install.
The use cases are 2 that I can think of:
Manually start the application using a console_scripts like client-web-up that should be the equivalent to type manage.py runserver.
This is what I'm focused on right now.
create some sort of script to generate configuration files for apache, nginx etc... for later.
Steps taken
Created a main folder client_web for my pip package.
Moved all my project into client_web folder.
Created a setup.py as shown below.
Edited manage.py file so its content can be executed by an function named entry().
Publish the pip package which downloads and installs correctly.
from setuptools import setup
setup(
name="client-web",
...,
version="2018.08.03.3",
packages=[
"client_web",
"client_web.controller",
"client_web.measurement",
"client_web.webapp",
"client_web.controller.migrations",
"client_web.measurement.migrations",
],
include_package_data=True,
install_requires=[
...
],
entry_points={
"console_scripts": [
"client-web-up=client_web.manage:entry"
]
}
)
The problem
If i type the following in the terminal:
$ client-web-up
Traceback (most recent call last):
File "/usr/local/bin/client-web-up", line 26, in <module>
sys.exit(entry())
File "/usr/local/lib/python3.7/site-packages/client_web/manage.py", line 33, in entry
main();
File "/usr/local/lib/python3.7/site-packages/client_web/manage.py", line 30, in main
execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 377, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 352, in execute
django.setup()
File "/usr/local/lib/python3.7/site-packages/django/__init__.py", line 24, in setup
apps.populate(settings.INSTALLED_APPS)
File "/usr/local/lib/python3.7/site-packages/django/apps/registry.py", line 89, in populate
app_config = AppConfig.create(entry)
File "/usr/local/lib/python3.7/site-packages/django/apps/config.py", line 116, in create
mod = import_module(mod_path)
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'webapp'
But if I do:
$ pip3 show client-web
Name: client-web
Version: 2018.8.3.3
Location: /usr/local/lib/python3.7/site-packages
Requires: ...
Required-by: ...
$ python3 /usr/local/lib/python3.7/site-packages/client_web/manage.py
Available subcommands:
[auth]
changepassword
createsuperuser
[contenttypes]
remove_stale_contenttypes
[django]
...
Could someone help me find a way to make the console_script work?
solved by adding a sys.path like so:
sys.path.append(os.path.dirname(os.path.realpath(__file__)))
I am trying to create documentation using Sphinx in PyCharm and it does not work. I already tried using the terminal command make html and it seems to work just fine. When I try to run Sphinx task in PyCharm it raises this error:
Traceback (most recent call last):
File "/Applications/PyCharm.app/Contents/helpers/rest_runners/sphinx_runner.py", line 5, in <module>
from sphinx import cmdline
File "/Users/wtekimam/PycharmProjects/doctor/myvenv/lib/python3.6/site-packages/sphinx/cmdline.py", line 24, in <module>
from sphinx.application import Sphinx
File "/Users/wtekimam/PycharmProjects/doctor/myvenv/lib/python3.6/site-packages/sphinx/application.py", line 33, in <module>
from sphinx.environment import BuildEnvironment
File "/Users/wtekimam/PycharmProjects/doctor/myvenv/lib/python3.6/site-packages/sphinx/environment/__init__.py", line 45, in <module>
from sphinx.util.websupport import is_commentable
File "/Users/wtekimam/PycharmProjects/doctor/myvenv/lib/python3.6/site-packages/sphinx/util/websupport.py", line 11, in <module>
from sphinxcontrib.websupport.utils import is_commentable # NOQA
File "/Users/wtekimam/PycharmProjects/doctor/myvenv/lib/python3.6/site-packages/sphinxcontrib/websupport/__init__.py", line 13, in <module>
__import__('pkg_resources').declare_namespace(__name__)
File "<frozen importlib._bootstrap>", line 961, in _find_and_load
File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 646, in _load_unlocked
File "<frozen importlib._bootstrap>", line 616, in _load_backward_compatible
File "/Users/wtekimam/PycharmProjects/doctor/myvenv/lib/python3.6/site-packages/setuptools-28.8.0-py3.6.egg/pkg_resources/__init__.py", line 3017, in <module>
File "/Users/wtekimam/PycharmProjects/doctor/myvenv/lib/python3.6/site-packages/setuptools-28.8.0-py3.6.egg/pkg_resources/__init__.py", line 3003, in _call_aside
File "/Users/wtekimam/PycharmProjects/doctor/myvenv/lib/python3.6/site-packages/setuptools-28.8.0-py3.6.egg/pkg_resources/__init__.py", line 3045, in _initialize_master_working_set
File "/Users/wtekimam/PycharmProjects/doctor/myvenv/lib/python3.6/site-packages/setuptools-28.8.0-py3.6.egg/pkg_resources/__init__.py", line 2577, in activate
File "/Users/wtekimam/PycharmProjects/doctor/myvenv/lib/python3.6/site-packages/setuptools-28.8.0-py3.6.egg/pkg_resources/__init__.py", line 2151, in declare_namespace
File "/Users/wtekimam/PycharmProjects/doctor/myvenv/lib/python3.6/site-packages/setuptools-28.8.0-py3.6.egg/pkg_resources/__init__.py", line 2091, in _handle_ns
File "/Users/wtekimam/PycharmProjects/doctor/myvenv/lib/python3.6/site-packages/setuptools-28.8.0-py3.6.egg/pkg_resources/__init__.py", line 2120, in _rebuild_mod_path
AttributeError: '_NamespacePath' object has no attribute 'sort'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Applications/PyCharm.app/Contents/helpers/rest_runners/sphinx_runner.py", line 7, in <module>
raise NameError("Cannot find sphinx in selected interpreter.")
NameError: Cannot find sphinx in selected interpreter.
note: I use virtual environment and I already configure PyCharm to use the same virtual environment that I use to run make html in terminal
There are two requirements:
Install Sphinx into your project's interpreter.
Create a run configuration for a Sphinx task with the appropriate settings as specified in PyCharm's documentation.
Here's a typical configuration with screenshot.
Command: html
Input: path/to/reST/source/files
Output: path/to/reST/build/files
Python Interpreter: Project Default (or whatever you select)
Working directory: path/to/conf.py
The solution is to update all packages that are used in the interpreter.
In particular, update the setuptools package.
This worked for me, but I do not know why, I am sorry. None of the answers above helped.