Problem with migrating a django server (syntax error) - django

I am trying to make a django server for a sociometric badge (https://github.com/HumanDynamics/openbadge-server) for our university's project. The code (and the whole badge) has been done by someone else and I have not myself changed anything, I am simply trying to get it to work. I am able to build the server but when trying to migrate or create a superuser, I get a syntax error. I've been trying to troubleshoot it by myself but I have very limited knowledge of python, django and Ubuntu so I'm probably missing something.
The error implies an error in the line 44 of /usr/local/lib/python2.7/site-packages/pkgconf/init.py but I cannot find the file so I cannot check it. In fact, the whole site-packages folder is empty so I wonder if I have installed modules in a wrong way? The code is written in python2.7 (Which I cannot change as it is not my code) so I also wonder if the python2.7 being EOL could cause issues? It has already broken some parts, mainly how to get some of the dependencies.
The code and docker files used in this project can be found here: https://github.com/HumanDynamics/openbadge-server
The dependency versions should be fine, the Django version should be compatible with Python2.7 and same for other modules. I've tried changing the versions around but to no avail. Down here is the requirement texts
Django==1.8.4
Fabric==1.10.2
django-grappelli==2.7.1
simplejson==3.8.0
MarkupSafe==0.23
django-pipeline==1.5.4
djangorestframework==3.2.3
djangorestframework-expiring-authtoken==0.1.1
pytz==2015.7
python-dateutil==2.5.3
jsonfield==1.0.3
django-controlcenter===0.2.6
# Configuration
django-environ==0.4.1
# Python-PostgreSQL Database Adapter
psycopg2==2.7.3.2
# Unicode slugification
awesome-slugify==1.6.5
# Import and export using the admin tool
# Using tablib 0.12.1. Newer versions break the import-export add-on
tablib==0.12.1
django-import-export==1.0.0
coverage==4.3.1
django-coverage-plugin==1.3.1
Sphinx==1.5.1
django-extensions==1.7.5
Werkzeug==0.11.15
django-test-plus==1.0.16
factory-boy==2.8.1
django-debug-toolbar==1.6
# improved REPL
ipdb==0.10.1
pytest-django==3.1.2
pytest-sugar==0.8.0
This is the error. From my limited knowledge, I'd gather that it doesn't find the files from /usr/local/lib/python2.7/site-packages/ but it is completely empty, the dependencies are either installed locally or to dist-packages. Someone earlier said that it was a python3 problem but nothing should be Python3. Could it also be a docker version problem if the build somehow installs wrong things?
Starting openbadge-server_postgres_1 ... done
Postgres is up - continuing...
Traceback (most recent call last):
File "manage.py", line 23, in <module>
execute_from_command_line(sys.argv)
File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 338, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 312, in execute
django.setup()
File "/usr/local/lib/python2.7/site-packages/django/__init__.py", line 18, in setup
apps.populate(settings.INSTALLED_APPS)
File "/usr/local/lib/python2.7/site-packages/django/apps/registry.py", line 85, in populate
app_config = AppConfig.create(entry)
File "/usr/local/lib/python2.7/site-packages/django/apps/config.py", line 86, in create
module = import_module(entry)
File "/usr/local/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/usr/local/lib/python2.7/site-packages/controlcenter/__init__.py", line 1, in <module>
from .dashboards import Dashboard # NOQA
File "/usr/local/lib/python2.7/site-packages/controlcenter/dashboards.py", line 10, in <module>
from . import app_settings
File "/usr/local/lib/python2.7/site-packages/controlcenter/app_settings.py", line 1, in <module>
from pkgconf import Conf
File "/usr/local/lib/python2.7/site-packages/pkgconf/__init__.py", line 44
class Conf(metaclass=ConfMeta):
^
SyntaxError: invalid syntax
I will provide information to best of my abilities.

Fixed the issue by adding django-pkgconf==0.3.0 to the requirements. While the requirements did not have the package at all, it was still installed (and used) through other packages and it installed version 0.4.0 which does not support Python 2.7.
Weird thing is that I could not find a trace where it was installed if I installed it without having it on requirements so even when I manually installed 0.3.0, it would still use 0.4.0 (despite seemingly not having it installed) so in order to get it to work it had to be installed through the docker build.

Related

gcloud version 306.0.0 causing "No module named 'urllib2'" errors

After updating gcloud from version 290.0.1 to version 306.0.0, I'm getting an error when I run a gsutil cp command:
Traceback (most recent call last):
File "/usr/lib/google-cloud-sdk/platform/gsutil/gsutil", line 21, in <module>
gsutil.RunMain()
File "/usr/lib/google-cloud-sdk/platform/gsutil/gsutil.py", line 122, in RunMain
import gslib.__main__
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 53, in <module>
import boto
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/__init__.py", line 1216, in <module>
boto.plugin.load_plugins(config)
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/plugin.py", line 93, in load_plugins
_import_module(file)
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/plugin.py", line 75, in _import_module
return imp.load_module(name, file, filename, data)
File "/usr/lib/python3.6/imp.py", line 235, in load_module
return load_source(name, filename, file)
File "/usr/lib/python3.6/imp.py", line 172, in load_source
module = _load(spec)
File "/usr/share/google/boto/boto_plugins/compute_auth.py", line 18, in <module>
import urllib2
ModuleNotFoundError: No module named 'urllib2'
Following the downgrade instructions at https://cloud.google.com/sdk/docs/downloads-apt-get#downgrading_cloud_sdk_versions temporarily fixes the issue:
sudo apt-get update && sudo apt-get install google-cloud-sdk=290.0.1-0
But I'd like to know how to get this working with the latest version.
I have installed the version 306.0.0 and I ran a gcloud cp command, but I didn't face the issue. For this reason, checking for causes for the error ModuleNotFoundError: No module named 'urllib2', it seems that they are always related to a Python library that isn't working correctly - as you can check in this two examples here and here.
However, in further searches, this plugin usually is used within Compute Engine and startup scripts to VMs with Python, more specific relating to the file compute_auth.py - which in the message seems to be related to the error - and as you can check for more information here about this file.
Considering that, the new version of Cloud SDK bring some updates to Compute Engine that could be causing the error. In case you are indeed, using Python within your applications, I would give it a try the solution from this case here, that would be to update the file compute_auth.py, changing the line import urllib2 toimport urllib.request as urllib2.
In case this doesn't fix, raising a bug within Google's Issue Tracker will be the best option, for a further investigation.
I had a similar case. In my case, Travis CI/CD was giving the below error. What I did is add the below script to my .travis.yml file before_script section.
Error:
Traceback (most recent call last):
635 File "/usr/lib/google-cloud-sdk/platform/gsutil/gsutil", line 21, in <module>
636 gsutil.RunMain()
637 File "/usr/lib/google-cloud-sdk/platform/gsutil/gsutil.py", line 121, in RunMain
638 import gslib.__main__
639 File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 83, in <module>
640 import httplib2
641ModuleNotFoundError: No module named 'httplib2'
642error Command failed with exit code 1
Fix:
before_script:
- pip install httplib2 crcmod

Django: "django.contrib.gis.gdal.prototypes.ds" crashes with "attributeerror: symbol not found"

Running the latest Django (3.0.4) I am now getting this:
File "[...]site-packages/django/contrib/gis/gdal/driver.py", line 5, in <module>
from django.contrib.gis.gdal.prototypes import ds as vcapi, raster as rcapi
File "[...]site-packages/django/contrib/gis/gdal/prototypes/ds.py", line 71, in <module>
get_field_as_integer64 = int64_output(lgdal.OGR_F_GetFieldAsInteger64, [c_void_p, c_int])
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ctypes/__init__.py", line 361, in __getattr__
func = self.__getitem__(name)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ctypes/__init__.py", line 366, in __getitem__
func = self._FuncPtr((name_or_ordinal, self))
AttributeError: dlsym(0x101667060, OGR_F_GetFieldAsInteger64): symbol not found
My Observations:
I have [ellided] the path names which refer to Django in my virtual-environment. I observe that the module that is actually crashing is in /Library/Frameworks/Python.framework/ ... this on Macintosh OS/X Catalina.
The version of Python that is installed on this computer is 3.6.4. (The version on the hosting service is considerably older ... 3.4.)
Is this, as I suspect, a problem with the Python installation on this computer? (The most recent version that can be installed here is 3.7.7.)
EDIT: NO, it isn't!
After installing Django 3.0.4 on a Linux host I got this:
AttributeError: /usr/local/lib/libgdal.so.1: undefined symbol: OGR_F_GetFieldAsInteger64
Now, once again this is a system-wide library that's being called, but I now suspect that the root cause problem is actually in this line:
File "[...]/lib/python3.6/site-packages/django/contrib/gis/gdal/prototypes/ds.py", line 71, in <module>
get_field_as_integer64 = int64_output(lgdal.OGR_F_GetFieldAsInteger64, [c_void_p, c_int])
I am now rolling Django versions backward to see if I can make the problem go away and if so "answer my own question."
I had to walk back all the way to Django 2.1 before the problem finally went away.
libgdal version 1.6 is installed on this computer's virtual-environment and I don't know if it is practical nor possible for me to change it. ("gcc" is installed here, "configure" is not.)

ImportError: No module named time

I'm trying to get a Google app up and running on my local machine, however, am facing an issue when running the setup scripts. The script errors out and tells me that there is no module time and seems to be breaking in the google-cloud-sdk....
Things I've tried:
Importing time in Python (it works)
Trying this to no avail: https://apple.stackexchange.com/questions/96308/python-installation-messed-up
Traceback (most recent call last):
File "/Users/kennethryan/Projects/go-edu-store/y/google-cloud-sdk/platform/google_appengine/_python_runtime.py", line 83, in <module>
_run_file(__file__, globals())
File "/Users/kennethryan/Projects/go-edu-store/y/google-cloud-sdk/platform/google_appengine/_python_runtime.py", line 79, in _run_file
execfile(_PATHS.script_file(script_name), globals_)
File "/Users/kennethryan/Projects/go-edu-store/y/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/runtime.py", line 175, in <module>
main()
File "/Users/kennethryan/Projects/go-edu-store/y/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/runtime.py", line 155, in main
sandbox.enable_sandbox(config)
File "/Users/kennethryan/Projects/go-edu-store/y/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/sandbox.py", line 183, in enable_sandbox
__import__('%s.threading' % dist27.__name__)
File "/Users/kennethryan/Projects/go-edu-store/y/google-cloud-sdk/platform/google_appengine/google/appengine/dist27/threading.py", line 13, in <module>
from time import time as _time, sleep as _sleep
File "/Users/kennethryan/Projects/go-edu-store/y/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/sandbox.py", line 984, in load_module
raise ImportError('No module named %s' % fullname)
ImportError: No module named time
Here is my current $PATH:
/Users/kennethryan/Projects/go-edu-store/y/google-cloud-sdk/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
Seeing as this is an inactive year old issue, we can assume that updating the Google Cloud tools to their latest versions by running 'gcloud components update' will fix this.
Also ensuring that you are using the Python installation provided by GCloud, and that there are no conflicting 'CLOUDSDK_PYTHON' environment variables should prevent this.
If this issue is seen again in the future, it is recommended to directly report this to the Google Public Issue Tracker so that this can be properly handled and triaged to the GCloud engineering team.
In my case, I resolved this problem by setting
export PYTHONPATH=$PYTHONPATH:/usr/lib64/python2.7/lib-dynload/ where timemodule.so file is located.

Adding libraries to django nonrel

I've a project in Django which I'm trying to port to Django-nonrel so that I can upload it to Google app Engine. I've installed django-nonrel and other required libraries by going through the http://djangoappengine.readthedocs.org/en/latest/installation.html
namely: django-nonrel
djangoappengine
djangotoolbox
django-autoload
django-dbindexer
that is by downloading their zip files and placing them in my app directory.
So, my app directory is:
>
<app>/autoload
<app>/dbindexer
<app>/django
<app>/djangoappengine
<app>/djangotoolbox
I also have django in my project directory and have started the project by:
PYTHONPATH=. python django/bin/django-admin.py startproject \
--name=app.yaml --template=djangoappengine/conf/project_template app
If I am adding an external library with pip and adding it to the INSTALLED_APPS of my app's settings.py , it is not recognised by my django-nonrel which is pretty obvious considering the fact that django-nonrel is not installed on my system. It gives me the following error
Traceback (most recent call last):
File "/usr/local/google_appengine/google/appengine/tools/devappserver2/module.py", line 1390, in _warmup
request_type=instance.READY_REQUEST)
File "/usr/local/google_appengine/google/appengine/tools/devappserver2/module.py", line 884, in _handle_request
environ, wrapped_start_response)
File "/usr/local/google_appengine/google/appengine/tools/devappserver2/request_rewriter.py", line 314, in _rewriter_middleware
response_body = iter(application(environ, wrapped_start_response))
File "/usr/local/google_appengine/google/appengine/tools/devappserver2/module.py", line 1297, in _handle_script_request
request_type)
File "/usr/local/google_appengine/google/appengine/tools/devappserver2/module.py", line 1262, in _handle_instance_request
request_type)
File "/usr/local/google_appengine/google/appengine/tools/devappserver2/instance.py", line 371, in handle
raise CannotAcceptRequests('Instance has been quit')
CannotAcceptRequests: Instance has been quit
(nonrel)apurva#apurva-HP-ProBook-6470b:~/project/flogin$ python manage.py runserver
INFO 2015-08-11 16:06:54,606 sdk_update_checker.py:229] Checking for updates to the SDK.
INFO 2015-08-11 16:06:55,511 sdk_update_checker.py:257] The SDK is up to date.
INFO 2015-08-11 16:06:55,633 api_server.py:205] Starting API server at: http://localhost:60055
INFO 2015-08-11 16:06:55,847 dispatcher.py:197] Starting module "default" running at: http://127.0.0.1:8080
INFO 2015-08-11 16:06:55,847 admin_server.py:118] Starting admin server at: http://localhost:8000
INFO 2015-08-11 16:06:58,966 __init__.py:52] Validating models...
ERROR 2015-08-11 16:06:59,045 wsgi.py:263]
Traceback (most recent call last):
File "/usr/local/google_appengine/google/appengine/runtime/wsgi.py", line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/usr/local/google_appengine/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "/usr/local/google_appengine/google/appengine/runtime/wsgi.py", line 96, in LoadObject
__import__(cumulative_path)
File "/home/apurva/project/flogin/djangoappengine/main/__init__.py", line 66, in <module>
validate_models()
File "/home/apurva/project/flogin/djangoappengine/main/__init__.py", line 55, in validate_models
num_errors = get_validation_errors(s, None)
File "/home/apurva/project/flogin/django/core/management/validation.py", line 34, in get_validation_errors
for (app_name, error) in get_app_errors().items():
File "/home/apurva/project/flogin/django/db/models/loading.py", line 196, in get_app_errors
self._populate()
File "/home/apurva/project/flogin/django/db/models/loading.py", line 75, in _populate
self.load_app(app_name, True)
File "/home/apurva/project/flogin/django/db/models/loading.py", line 97, in load_app
app_module = import_module(app_name)
File "/home/apurva/project/flogin/django/utils/importlib.py", line 42, in import_module
__import__(name)
ImportError: No module named oauth2_provider
However, I'm unsure on how to add external libraries to my project. So that my django-nonrel recognises it. I've also tried google's documentation on how to this i.e.
Adding Third-party Packages to the Application
You can add any third-party library to your application, as long as it
is implemented in "pure Python" (no C extensions) and otherwise
functions in the App Engine runtime environment. The easiest way to
manage this is with a ./lib directory.
Create a directory named lib in your application root directory:
mkdir lib To tell your app how to find libraries in this directory,
create (or modify) a file named appengine_config.py in the root of
your project, then add these lines:
from google.appengine.ext import vendor
£ Add any libraries installed in the "lib" folder. vendor.add('lib') Use pip with the -t lib flag to install libraries in this directory:
$ pip install -t lib gcloud Note: pip version 6.0.0 or higher is
required for vendor to work properly.
Tip: the appengine_config.py above assumes that the current working
directory is where the lib folder is located. In some cases, such as
unit tests, the current working directory can be different. To avoid
errors, you can explicity pass in the full path to the lib folder
using
vendor.add(os.path.join(os.path.dirname(os.path.realpath(file)),
'lib'))
didn't work either.
So I had a very similar dilemma. Here is how I solved it:
Followed Google's instructions noted above, using pip and a ./lib directory. Make sure you have an updated version of pip:
sudo pip install --upgrade pip
Then, because of pkg_resources issues, I did this:
pip install -t lib setuptools
That was necessary, I am just not sure if that was the right place to install setuptools or not. It certainly worked, though.
Then, I launched the local development server like this, in the project directory:
PYTHONPATH=lib ./manage.py runserver
I hope that works for you!

Backwards migration from Django 1.4 to Django 1.3

Sometimes my stupid, reckless and daredevil programming gut gets me to some dangerous places:
I've started a Django 1.4 application with sqlite3 and then moved to mysql, no big deal there. But then I realized that my models fit well with a NoSql model and decided to try MongoDB with django-nonrel wich is a fork of Django 1.3 with support for non-relational databases. the version 1.4 still not ready to use yet.
So, I've branched my repo, created my virtualenv and pipinstalled django-nonrel but when I run ./manage.py shell I got this message
Traceback (most recent call last):
File "./manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/Users/marcoslhc/Documents/Proyectos/fontcase/mongoBE/lib/python2.7/site-packages/django/core/management/__init__.py", line 429, in execute_from_command_line
utility.execute()
File "/Users/marcoslhc/Documents/Proyectos/fontcase/mongoBE/lib/python2.7/site-packages/django/core/management/__init__.py", line 379, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Users/marcoslhc/Documents/Proyectos/fontcase/mongoBE/lib/python2.7/site-packages/django/core/management/__init__.py", line 252, in fetch_command
app_name = get_commands()[subcommand]
File "/Users/marcoslhc/Documents/Proyectos/fontcase/mongoBE/lib/python2.7/site-packages/django/core/management/__init__.py", line 101, in get_commands
apps = settings.INSTALLED_APPS
File "/Users/marcoslhc/Documents/Proyectos/fontcase/mongoBE/lib/python2.7/site-packages/django/utils/functional.py", line 276, in __getattr__
self._setup()
File "/Users/marcoslhc/Documents/Proyectos/fontcase/mongoBE/lib/python2.7/site-packages/django/conf/__init__.py", line 42, in _setup
self._wrapped = Settings(settings_module)
File "/Users/marcoslhc/Documents/Proyectos/fontcase/mongoBE/lib/python2.7/site-packages/django/conf/__init__.py", line 139, in __init__
logging_config_func(self.LOGGING)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/config.py", line 776, in dictConfig
dictConfigClass(config).configure()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/config.py", line 562, in configure
'filter %r: %s' % (name, e))
ValueError: Unable to configure filter 'require_debug_false': Cannot resolve 'django.utils.log.RequireDebugFalse': No module named RequireDebugFalse
RequireDebugFalse is new in Django 1.4 (see here and here) and I guess some other underlying magic new in 1.4 would not work either in this new installation. I was wondering if I can migrate back my application to Django 1.3 without doing django-admin.py startproject or django-admin.py startapp.
You can do that, although you probably want to leave the door open to going forward again.
The way I've handled this in the past is, if at all possible, to not go making massive changes to your code. Instead, do what you've already done -- run the code and wait for it to blow up. Identify what the source of the problem is and then do one of the following:
Create a stub that supplies the needed functionality (or at least quietly does nothing),
Back-port the functionality onto your own personal copy of 1.3, or
Monkey-patch the functionality onto an unmodified 1.3. (My recommendation)
Unless you have totally glommed onto some unique new feature in 1.4, you'd be surprised how quickly you can come up with a useable environment. I had one project where the monkey-patch file was only about 100 lines long to retrofit about a dozen features onto an older version.
If you haven't used django 1.4 stuff in your project you could create a 1.3 project and copy the logging settings from there. Then you should be fine.