syntax error when importing django-sphinxsearch - django

I am trying to run the standard django migration commands, ex., python3 manage.py makemigrations, and continuously get a syntax error on module import with sphinxsearch.
I know it's deprecated, but unfortunately I am working on small additions to the site and cannot move to elasticsearch yet.
Python version : 3.5.3
System : Debian (AWS server)
Sphinx version: 3.5.4
sphinxsearch version: 0.1
Error message:
Traceback (most recent call last):
File "manage.py", line 13, in <module>
execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.5/dist-packages/django/core/management/__init__.py", line 371, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python3.5/dist-packages/django/core/management/__init__.py", line 347, in execute
django.setup()
File "/usr/local/lib/python3.5/dist-packages/django/__init__.py", line 24, in setup
apps.populate(settings.INSTALLED_APPS)
File "/usr/local/lib/python3.5/dist-packages/django/apps/registry.py", line 89, in populate
app_config = AppConfig.create(entry)
File "/usr/local/lib/python3.5/dist-packages/django/apps/config.py", line 90, in create
module = import_module(entry)
File "/usr/lib/python3.5/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 986, in _gcd_import
File "<frozen importlib._bootstrap>", line 969, in _find_and_load
File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 669, in exec_module
File "<frozen importlib._bootstrap_external>", line 775, in get_code
File "<frozen importlib._bootstrap_external>", line 735, in source_to_code
File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
File "/usr/local/lib/python3.5/dist-packages/sphinxsearch/__init__.py", line 75
SPH_ATTR_MULTI = 0X40000000L
^
SyntaxError: invalid syntax
I've tried updating the system and have changed sphinx version, but I received the same error, just referencing a different file. Please note that the ini.py file is showing as blank when I navigate to it manually.

The error is caused by the trailing L in SPH_ATTR_MULTI = 0X40000000L and this is a sign that this code is meant to work with python 2, not python 3. In python 2 the trailing L is a indication that the value is type long instead integer and it is used for large numbers, but in python 3 integer and long were unified, so the trailing L is giving an SyntaxError.
Use python 2.7 if sphinx is absolutely necessary.

I was able to remove the sphinxsearch module as it wasn't necessary, install the sphinx 4.0 release.
That gave me a context error with "str = f'{item}'" format, so I changed it to "str = "% ..." % (item)" format manually within base.py ( in /usr/local/lib/python3.5/dist-packages/sphinxsearch/base.py) and it worked properly with python 3.5
It seems its just that a weird mid-tier version is necessary.

Related

Fixtures are not working after Django squashmigrations

I just squashed a lot of migrations in Django 4.0.x.
Now when running the migrations I get the following error:
File "/opt/hostedtoolcache/Python/3.9.10/x64/lib/python3.9/site-packages/Django-4.0.3-py3.9.egg/django/db/migrations/loader.py", line 120, in load_disk
migration_module = import_module(migration_path)
File "/opt/hostedtoolcache/Python/3.9.10/x64/lib/python3.9/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 846, in exec_module
File "<frozen importlib._bootstrap_external>", line 983, in get_code
File "<frozen importlib._bootstrap_external>", line 913, in source_to_code
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "/home/runner/work/backend/backend/general/migrations/0001_squashed_0019_merge_0018_auto_20220226_2311_0018_auto_20220228_2343.py", line 42
code=general.migrations.0004_initial_general.Migration.load_data,
^
SyntaxError: invalid decimal literal
The according code lines in the migration file is:
migrations.RunPython(
code=general.migrations.0004_initial_general.Migration.load_data,
),
I am totally lost here.
As I could find in this link https://stackoverflow.com/a/59813844/17881270
Python identifiers can not start with a number.
Hope that could help you. Rename them in migration table and that directory and your squash.
Or maybe you can use getattr function.
Look at the top of your squashed migration file - you should find a comment that reads something like this:
# Functions from the following migrations need manual copying.
# Move them and any dependencies into this file, then update the
# RunPython operations to refer to the local versions:
# general.migrations.0004_initial_general
You should also have seen a notice printed to your console when you generated the migrations, alerting you that manual modification of the generated file is needed.
This is happening because you have defined some custom migration functions in that migration, which Django was not able to migrate automatically. You need to manually copy the load_data function from 0004_initial_general.py into this squashed migration file, and then replace the reference to it in the RunPython call.

Hdfs hook dependency on snakebite throwing error

I'm trying to use HDFS sensor in airflow. But am getting the below error
[2020-06-30 07:22:57,636] {dagbag.py:246} ERROR - Failed to import: /airflow/dags/test_s3_file.py
Traceback (most recent call last):
File "/airflow/airflow-staging/lib/python3.6/site-packages/airflow/models/dagbag.py", line 243, in process_file
m = imp.load_source(mod_name, filepath)
File "/airflow/airflow-staging/lib64/python3.6/imp.py", line 172, in load_source
module = _load(spec)
File "<frozen importlib._bootstrap>", line 684, in _load
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/airflow/dags/test_s3_file.py", line 3, in <module>
from S3HDFSObjectSensor import S3HDFSObjectSensor
File "/airflow/plugins/S3HDFSObjectSensor.py", line 6, in <module>
from airflow.hooks.hdfs_hook import HDFSHook
File "/airflow/airflow-staging/lib/python3.6/site-packages/airflow/hooks/hdfs_hook.py", line 24, in <module>
from snakebite.client import Client, HAClient, Namenode, AutoConfigClient
File "/airflow/airflow-staging/lib/python3.6/site-packages/snakebite/client.py", line 1473
baseTime = min(time * (1L << retries), cap);
^
SyntaxError: invalid syntax
From google, found that we will have this issue if we are using python3. Could you please let me know if there is any work around for this other than using python2.
from airflow.plugins.S3HDFSObjectSensor import S3HDFSObjectSensor
t1 = S3HDFSObjectSensor(
task_id='t1',
poke_interval=sensor_poke_interval,
timeout=sensor_timeout,
object_path=check_file_in_hdfs,
trigger_rule='one_failed',
dag=dag)
Below is my AIRFLOW_HOME directory structure
airflow
+-- dags
+-- plugins
+-- S3HDFSObjectSensor.py
Thanks
It worked after installing 'pip3 install snakebite-py3'

pip install locustio ==0.8a2 is not installing?

I got an issue on 'core' module in not find in django project. To overcome from this problem, i got to know that i have to install 'locustio ==0.8a2' so i type in windowshell
pip install locustio ==0.8a2
but it is not installing.
And here is the initial problem(django):
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "C:\Users\Papul\Desktop\sandeep\myproject\myproject\urls.py", line 6, in <module>
from core.views import SignUpView
ModuleNotFoundError: No module named 'core'
here is the snapshot:[1] https://i.stack.imgur.com/00tGl.png [2]: https://i.stack.imgur.com/PXO5u.png
"No module named core", it looks like could not find your django package. please show "pip list ", by the way, why not use some IDE to run your project.(e.g. pycharm )
You should install locust package it's not locustio anymore:
Locust package has moved from 'locustio' to 'locust'. Please update your reference (or pin your version to 0.14.6 if you dont want to update to 1.0)
pip install locust
To resolve, install the missing modules:
pip install core
before you run
pip install locustio==0.8a2

Pip package a django project

The target is to create a pip package of a Django project.
The goal is to make the web application quickly installable anywhere just by doing pip install.
The use cases are 2 that I can think of:
Manually start the application using a console_scripts like client-web-up that should be the equivalent to type manage.py runserver.
This is what I'm focused on right now.
create some sort of script to generate configuration files for apache, nginx etc... for later.
Steps taken
Created a main folder client_web for my pip package.
Moved all my project into client_web folder.
Created a setup.py as shown below.
Edited manage.py file so its content can be executed by an function named entry().
Publish the pip package which downloads and installs correctly.
from setuptools import setup
setup(
name="client-web",
...,
version="2018.08.03.3",
packages=[
"client_web",
"client_web.controller",
"client_web.measurement",
"client_web.webapp",
"client_web.controller.migrations",
"client_web.measurement.migrations",
],
include_package_data=True,
install_requires=[
...
],
entry_points={
"console_scripts": [
"client-web-up=client_web.manage:entry"
]
}
)
The problem
If i type the following in the terminal:
$ client-web-up
Traceback (most recent call last):
File "/usr/local/bin/client-web-up", line 26, in <module>
sys.exit(entry())
File "/usr/local/lib/python3.7/site-packages/client_web/manage.py", line 33, in entry
main();
File "/usr/local/lib/python3.7/site-packages/client_web/manage.py", line 30, in main
execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 377, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 352, in execute
django.setup()
File "/usr/local/lib/python3.7/site-packages/django/__init__.py", line 24, in setup
apps.populate(settings.INSTALLED_APPS)
File "/usr/local/lib/python3.7/site-packages/django/apps/registry.py", line 89, in populate
app_config = AppConfig.create(entry)
File "/usr/local/lib/python3.7/site-packages/django/apps/config.py", line 116, in create
mod = import_module(mod_path)
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'webapp'
But if I do:
$ pip3 show client-web
Name: client-web
Version: 2018.8.3.3
Location: /usr/local/lib/python3.7/site-packages
Requires: ...
Required-by: ...
$ python3 /usr/local/lib/python3.7/site-packages/client_web/manage.py
Available subcommands:
[auth]
changepassword
createsuperuser
[contenttypes]
remove_stale_contenttypes
[django]
...
Could someone help me find a way to make the console_script work?
solved by adding a sys.path like so:
sys.path.append(os.path.dirname(os.path.realpath(__file__)))

Django and Jenkins multiple errors on execute shell

I am trying to implement continuous integration for a python-django app. I installed Jenkins on a vps and created a job which pulls my bitbucket code.
Now I am trying to execute the tests from the tests dir, but something is failing within my execute shell in Jenkins.
Here is the shell console output:
7 - Link the cms_core app in the project using a symbolic link
/var/lib/jenkins/workspace/ed1cms/test_cms_core
total 16
drwxr-xr-x 3 jenkins jenkins 4096 Jul 5 16:12 .
drwxr-xr-x 5 jenkins jenkins 4096 Jul 5 16:12 ..
lrwxrwxrwx 1 jenkins jenkins 16 Jul 5 16:12 cms_core -> ../core/cms_core
-rwxr-xr-x 1 jenkins jenkins 811 Jul 5 16:12 manage.py
drwxr-xr-x 2 jenkins jenkins 4096 Jul 5 16:12 test_cms_core
/tmp/jenkins7283490794872735254.sh: line 28: prinf: command not found
Collecting cms_core
Could not find a version that satisfies the requirement cms_core (from versions: )
No matching distribution found for cms_core
8 - Finally, we can run the tests through django-jenkins
Traceback (most recent call last):
File "./manage.py", line 22, in <module>
execute_from_command_line(sys.argv)
File "/var/lib/jenkins/.virtualenvs/tmp-8ca50c09c788e0c/lib/python3.6/site-packages/django/core/management/__init__.py", line 363, in execute_from_command_line
utility.execute()
File "/var/lib/jenkins/.virtualenvs/tmp-8ca50c09c788e0c/lib/python3.6/site-packages/django/core/management/__init__.py", line 307, in execute
settings.INSTALLED_APPS
File "/var/lib/jenkins/.virtualenvs/tmp-8ca50c09c788e0c/lib/python3.6/site-packages/django/conf/__init__.py", line 56, in __getattr__
self._setup(name)
File "/var/lib/jenkins/.virtualenvs/tmp-8ca50c09c788e0c/lib/python3.6/site-packages/django/conf/__init__.py", line 41, in _setup
self._wrapped = Settings(settings_module)
File "/var/lib/jenkins/.virtualenvs/tmp-8ca50c09c788e0c/lib/python3.6/site-packages/django/conf/__init__.py", line 110, in __init__
mod = importlib.import_module(self.SETTINGS_MODULE)
File "/var/lib/jenkins/.virtualenvs/tmp-8ca50c09c788e0c/lib/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'cms_core'
9 - Deactivate virtualenv so it is destroyed
Removing temporary environment: tmp-8ca50c09c788e0c
Removing tmp-8ca50c09c788e0c...
Finished: SUCCESS
the 7,8,9 are steps I have written in the shell with printf so I could follow whats going on step by step. Until step 7 there are no errors taking place.
Here is my execute shell script:
#!/bin/bash
printf "\n\n1 - Set-up some things so we can use virtualenvwrapper\n\n"
source ~/.bash_profile
printf "\n\n2 - Create a temporary virtual environment for Python\n\n"
mktmpenv -p /opt/python-versions/3.6.3/bin/python3
printf "\n\n3 - We need to go back to the path we initially were, not in the virtual env path\n\n"
cd -
printf "\n\n4 - Install requirements to be able to create a Django project and run tests\n\n"
pip install -r cms_core/tests/requirements.txt
printf "\n\n5 - Create the Django project\n\n"
rm -rf test_cms_core
django-admin startproject test_cms_core
printf "\n\n6 - Go in the project folder\n\n"
cd test_cms_core
printf "\n\n7 - Link the cms_core app in the project using a symbolic link\n\n"
ln -s ../core/cms_core .
pwd
ls -la
printf "\n\n8 - Finally, we can run the tests through django-jenkins\n\n"
./manage.py jenkins --settings=cms_core.tests.settings
printf "\n\n9 - Deactivate virtualenv so it is destroyed\n\n"
deactivate
It seems that you have a typo somewhere in your code. Just search for 'prinf', maybe a t is missing.
Your module cms_core is not installed or can not be found. Maybe you are just in the wrong folder to use your settings for django? Or check your virtualenv with pip list if all requirements are installed correct.