How to create admin user in django tests.py - django

I'm trying to create an admin user as part of my tests.py to check on persmissions.
UPDATE:
The tests.py is standard format that subclasses TestCase and the code below is called in the setUp() function.
I can create a normal user but not an admin user. If I try this:
self.adminuser = User.objects.create_user('admin', 'admin#test.com', 'pass')
self.adminuser.save()
self.adminuser.is_staff = True
self.adminuser.save()
OR
self.adminuser = User.objects.create_superuser('admin', 'admin#test.com', 'pass')
self.adminuser.save()
I get:
Warning: Data truncated for column 'name' at row 1
If I remove the is_staff line all is well (except I can't do my test!)
Do I have to load admin users as fixtures?
UserProfile is defined as follows:
class UserProfile(models.Model):
user = models.ForeignKey(User, unique=True)
organisation = models.ForeignKey(Organisation, null=True, blank=True)
telephone = models.CharField(max_length=20, null=True, blank=True)
and full error traceback is:
Traceback (most recent call last):
File "/usr/lib/python2.4/site-packages/Django-1.1.1-py2.4.egg/django/test/testcases.py", line 242, in __call__
self._pre_setup()
File "/usr/lib/python2.4/site-packages/Django-1.1.1-py2.4.egg/django/test/testcases.py", line 217, in _pre_setup
self._fixture_setup()
File "/usr/lib/python2.4/site-packages/Django-1.1.1-py2.4.egg/django/test/testcases.py", line 440, in _fixture_setup
return super(TestCase, self)._fixture_setup()
File "/usr/lib/python2.4/site-packages/Django-1.1.1-py2.4.egg/django/test/testcases.py", line 222, in _fixture_setup
call_command('flush', verbosity=0, interactive=False)
File "/usr/lib/python2.4/site-packages/Django-1.1.1-py2.4.egg/django/core/management/__init__.py", line 166, in call_command
return klass.execute(*args, **defaults)
File "/usr/lib/python2.4/site-packages/Django-1.1.1-py2.4.egg/django/core/management/base.py", line 222, in execute
output = self.handle(*args, **options)
File "/usr/lib/python2.4/site-packages/Django-1.1.1-py2.4.egg/django/core/management/base.py", line 351, in handle
return self.handle_noargs(**options)
File "/usr/lib/python2.4/site-packages/Django-1.1.1-py2.4.egg/django/core/management/commands/flush.py", line 61, in handle_noargs
emit_post_sync_signal(models.get_models(), verbosity, interactive)
File "/usr/lib/python2.4/site-packages/Django-1.1.1-py2.4.egg/django/core/management/sql.py", line 205, in emit_post_sync_signal
interactive=interactive)
File "/usr/lib/python2.4/site-packages/Django-1.1.1-py2.4.egg/django/dispatch/dispatcher.py", line 166, in send
response = receiver(signal=self, sender=sender, **named)
File "/usr/lib/python2.4/site-packages/Django-1.1.1-py2.4.egg/django/contrib/auth/management/__init__.py", line 28, in create_permissions
defaults={'name': name, 'content_type': ctype})
File "/usr/lib/python2.4/site-packages/Django-1.1.1-py2.4.egg/django/db/models/manager.py", line 123, in get_or_create
return self.get_query_set().get_or_create(**kwargs)
File "/usr/lib/python2.4/site-packages/Django-1.1.1-py2.4.egg/django/db/models/query.py", line 335, in get_or_create
obj.save(force_insert=True)
File "/usr/lib/python2.4/site-packages/Django-1.1.1-py2.4.egg/django/db/models/base.py", line 410, in save
self.save_base(force_insert=force_insert, force_update=force_update)
File "/usr/lib/python2.4/site-packages/Django-1.1.1-py2.4.egg/django/db/models/base.py", line 495, in save_base
result = manager._insert(values, return_id=update_pk)
File "/usr/lib/python2.4/site-packages/Django-1.1.1-py2.4.egg/django/db/models/manager.py", line 177, in _insert
return insert_query(self.model, values, **kwargs)
File "/usr/lib/python2.4/site-packages/Django-1.1.1-py2.4.egg/django/db/models/query.py", line 1087, in insert_query
return query.execute_sql(return_id)
File "/usr/lib/python2.4/site-packages/Django-1.1.1-py2.4.egg/django/db/models/sql/subqueries.py", line 320, in execute_sql
cursor = super(InsertQuery, self).execute_sql(None)
File "/usr/lib/python2.4/site-packages/Django-1.1.1-py2.4.egg/django/db/models/sql/query.py", line 2369, in execute_sql
cursor.execute(sql, params)
File "/usr/lib/python2.4/site-packages/Django-1.1.1-py2.4.egg/django/db/backends/mysql/base.py", line 84, in execute
return self.cursor.execute(query, args)
File "build/bdist.linux-x86_64/egg/MySQLdb/cursors.py", line 175, in execute
File "build/bdist.linux-x86_64/egg/MySQLdb/cursors.py", line 89, in _warning_check
File "/usr/lib64/python2.4/warnings.py", line 61, in warn
warn_explicit(message, category, filename, lineno, module, registry)
File "/usr/lib64/python2.4/warnings.py", line 96, in warn_explicit
raise message
Warning: Data truncated for column 'name' at row 1
The answer seems to be that you can't create an admin user in setUp but you can in any other function so if you want an admin user in testing, use a fixture!

I'd use the built-in create_superuser and log the user in before making any requests. The following should work:
from django.contrib.auth.models import User
from django.test.client import Client
# store the password to login later
password = 'mypassword'
my_admin = User.objects.create_superuser('myuser', 'myemail#test.com', password)
c = Client()
# You'll need to log him in before you can send requests through the client
c.login(username=my_admin.username, password=password)
# tests go here

Update 2
Executed the snippet to create the superuser from within a test case (subclass of django.test.TestCase). Everything went fine. Also created and saved an instance of UserProfile with user = self.adminuser. That too worked.
Update
This line is interesting:
File "/usr/lib/python2.4/site-packages/Django-1.1.1-py2.4.egg/django/contrib/auth/management/__init__.py", line 28, in create_permissions
defaults={'name': name, 'content_type': ctype})
Looks like execution fails when creating permissions.
Original Answer
Warning: Data truncated for column 'name' at row 1
Strange. I tried this from the Django shell and it worked for me. I am using Postgresql 8.3 and Django 1.2.1 on Ubuntu Jaunty. Can you give more details about which version of Django/database are you using?
Also User does not have a name attribute. Can you double check if you are using auth.User?
Do I have to load admin users as fixtures?
You don't have to. But if you are creating this admin user solely for testing purposes then it would be a good idea to add a Fixture. That is what I do in my projects.

Related

Django ValueError: Cannot serialize: <User: username>

When I'm trying to make migrations of my models in Django, I keep getting the same error, even after I've commented out all the changes:
(.venv) C:\Users\jezdo\venv\chat\chat_proj>python manage.py makemigrations chat
Migrations for 'chat':
chat\migrations\0002_alter_customusergroup_custom_group_name_and_more.py
- Alter field custom_group_name on customusergroup
- Alter field users on customusergroup
Traceback (most recent call last):
File "C:\Users\jezdo\.venv\chat\chat_proj\manage.py", line 22, in <module>
main()
File "C:\Users\jezdo\.venv\chat\chat_proj\manage.py", line 18, in main
execute_from_command_line(sys.argv)
File "C:\Users\jezdo\.venv\lib\site-packages\django\core\management\__init__.py", line 446, in execute_from_command_line
utility.execute()
File "C:\Users\jezdo\.venv\lib\site-packages\django\core\management\__init__.py", line 440, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "C:\Users\jezdo\.venv\lib\site-packages\django\core\management\base.py", line 402, in run_from_argv
self.execute(*args, **cmd_options)
File "C:\Users\jezdo\.venv\lib\site-packages\django\core\management\base.py", line 448, in execute
output = self.handle(*args, **options)
File "C:\Users\jezdo\.venv\lib\site-packages\django\core\management\base.py", line 96, in wrapped
res = handle_func(*args, **kwargs)
File "C:\Users\jezdo\.venv\lib\site-packages\django\core\management\commands\makemigrations.py", line 239, in handle
self.write_migration_files(changes)
File "C:\Users\jezdo\.venv\lib\site-packages\django\core\management\commands\makemigrations.py", line 278, in write_migration_files
migration_string = writer.as_string()
File "C:\Users\jezdo\.venv\lib\site-packages\django\db\migrations\writer.py", line 141, in as_string
operation_string, operation_imports = OperationWriter(operation).serialize()
File "C:\Users\jezdo\.venv\lib\site-packages\django\db\migrations\writer.py", line 99, in serialize
_write(arg_name, arg_value)
File "C:\Users\jezdo\.venv\lib\site-packages\django\db\migrations\writer.py", line 63, in _write
arg_string, arg_imports = MigrationWriter.serialize(_arg_value)
File "C:\Users\jezdo\.venv\lib\site-packages\django\db\migrations\writer.py", line 282, in serialize
return serializer_factory(value).serialize()
File "C:\Users\jezdo\.venv\lib\site-packages\django\db\migrations\serializer.py", line 221, in serialize
return self.serialize_deconstructed(path, args, kwargs)
File "C:\Users\jezdo\.venv\lib\site-packages\django\db\migrations\serializer.py", line 99, in serialize_deconstructed
arg_string, arg_imports = serializer_factory(arg).serialize()
File "C:\Users\jezdo\.venv\lib\site-packages\django\db\migrations\serializer.py", line 50, in serialize
item_string, item_imports = serializer_factory(item).serialize()
File "C:\Users\jezdo\.venv\lib\site-packages\django\db\migrations\serializer.py", line 50, in serialize
item_string, item_imports = serializer_factory(item).serialize()
File "C:\Users\jezdo\.venv\lib\site-packages\django\db\migrations\serializer.py", line 386, in serializer_factory
raise ValueError(
ValueError: Cannot serialize: <User: jezdo>
There are some values Django cannot serialize into migration files.
For more, see https://docs.djangoproject.com/en/4.1/topics/migrations/#migration-serializing
my model in models.py:
class CustomUserGroup(models.Model):
custom_group_name = models.CharField(max_length=50, unique=True)
users= MultiSelectField(max_length=100,choices=users_list,unique=False)
class Meta:
verbose_name_plural = 'Custom Groups'
ordering = ['custom_group_name']
def __unicode__(self):
return self.custom_group_name
I wanted to create another model similar to CustomUserGroup byt with a different "users" field:
class CustomUserGroup2(models.Model):
custom_group_name = models.CharField(max_length=50, unique=True)
users= models.ManyToManyField(User)
class Meta:
verbose_name_plural = 'Custom Groups'
ordering = ['custom_group_name']
def __unicode__(self):
return self.custom_group_name
but couldn't makemigrations due to the described error. Now I cannot make any migrations whatsoever, even after having deleted the CustomUserGroup2 class.
I'm using Python 3.10.4 and Django 4.1.6.

How to specify SQLAlchemy ForeignKey with specific schema name?

I have a Flask / Flask-User app that I am trying to convert from Python 3.6 to Flask / Flask-Security-Too under Python 3.10. There have been a lot of changes in the world since 3.6 and I am playing whack-a-mole trying to figure them out. In this particular case I am redoing the database tables since nothing from the old app needs to transfer to the new version. I am using the latest version of PyCharm and all of the latest versions of the libraries that Flask and Flask-Security-Too require.
Things to note: the target database is an MS SQL Server instance. There is an existing "database" that I must use and a specific "schema name" under which any tables I create must live.
I have successfully run flask db init and am trying to run the first flask db migrate but get the error:
(venv) PS C:\Users\me\PycharmProjects\MyProject\UserInterface\MyApp> flask db migrate
INFO [alembic.runtime.migration] Context impl MSSQLImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
Traceback (most recent call last):
File "C:\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\Users\me\PycharmProjects\MyProject\UserInterface\venv\Scripts\flask.exe\__main__.py", line 7, in <module>
File "C:\Users\me\PycharmProjects\MyProject\UserInterface\venv\lib\site-packages\flask\cli.py", line 988, in main
cli.main()
File "C:\Users\me\PycharmProjects\MyProject\UserInterface\venv\lib\site-packages\flask\cli.py", line 579, in main
return super().main(*args, **kwargs)
File "C:\Users\me\PycharmProjects\MyProject\UserInterface\venv\lib\site-packages\click\core.py", line 1055, in main
rv = self.invoke(ctx)
File "C:\Users\me\PycharmProjects\MyProject\UserInterface\venv\lib\site-packages\click\core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "C:\Users\me\PycharmProjects\MyProject\UserInterface\venv\lib\site-packages\click\core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "C:\Users\me\PycharmProjects\MyProject\UserInterface\venv\lib\site-packages\click\core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "C:\Users\me\PycharmProjects\MyProject\UserInterface\venv\lib\site-packages\click\core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "C:\Users\me\PycharmProjects\MyProject\UserInterface\venv\lib\site-packages\click\decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "C:\Users\me\PycharmProjects\MyProject\UserInterface\venv\lib\site-packages\flask\cli.py", line 427, in decorator
return __ctx.invoke(f, *args, **kwargs)
File "C:\Users\me\PycharmProjects\MyProject\UserInterface\venv\lib\site-packages\click\core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "C:\Users\me\PycharmProjects\MyProject\UserInterface\venv\lib\site-packages\flask_migrate\cli.py", line 104, in migrate
_migrate(directory, message, sql, head, splice, branch_label, version_path,
File "C:\Users\me\PycharmProjects\MyProject\UserInterface\venv\lib\site-packages\flask_migrate\__init__.py", line 98, in wrapped
f(*args, **kwargs)
File "C:\Users\me\PycharmProjects\MyProject\UserInterface\venv\lib\site-packages\flask_migrate\__init__.py", line 155, in migrate
command.revision(config, message, autogenerate=True, sql=sql,
File "C:\Users\me\PycharmProjects\MyProject\UserInterface\venv\lib\site-packages\alembic\command.py", line 229, in revision
script_directory.run_env()
File "C:\Users\me\PycharmProjects\MyProject\UserInterface\venv\lib\site-packages\alembic\script\base.py", line 569, in run_env
util.load_python_file(self.dir, "env.py")
File "C:\Users\me\PycharmProjects\MyProject\UserInterface\venv\lib\site-packages\alembic\util\pyfiles.py", line 94, in load_python_file
module = load_module_py(module_id, path)
File "C:\Users\me\PycharmProjects\MyProject\UserInterface\venv\lib\site-packages\alembic\util\pyfiles.py", line 110, in load_module_py
spec.loader.exec_module(module) # type: ignore
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "C:\Users\me\PycharmProjects\MyProject\UserInterface\MyApp\migrations\env.py", line 104, in <module>
run_migrations_online()
File "C:\Users\me\PycharmProjects\MyProject\UserInterface\MyApp\migrations\env.py", line 98, in run_migrations_online
context.run_migrations()
File "<string>", line 8, in run_migrations
File "C:\Users\me\PycharmProjects\MyProject\UserInterface\venv\lib\site-packages\alembic\runtime\environment.py", line 853, in run_migrations
self.get_context().run_migrations(**kw)
File "C:\Users\me\PycharmProjects\MyProject\UserInterface\venv\lib\site-packages\alembic\runtime\migration.py", line 611, in run_migrations
for step in self._migrations_fn(heads, self):
File "C:\Users\me\PycharmProjects\MyProject\UserInterface\venv\lib\site-packages\alembic\command.py", line 205, in retrieve_migrations
revision_context.run_autogenerate(rev, context)
File "C:\Users\me\PycharmProjects\MyProject\UserInterface\venv\lib\site-packages\alembic\autogenerate\api.py", line 526, in run_autogenerate
self._run_environment(rev, migration_context, True)
File "C:\Users\me\PycharmProjects\MyProject\UserInterface\venv\lib\site-packages\alembic\autogenerate\api.py", line 573, in _run_environment
compare._populate_migration_script(
File "C:\Users\me\PycharmProjects\MyProject\UserInterface\venv\lib\site-packages\alembic\autogenerate\compare.py", line 55, in _populate_migration_script
_produce_net_changes(autogen_context, upgrade_ops)
File "C:\Users\me\PycharmProjects\MyProject\UserInterface\venv\lib\site-packages\alembic\autogenerate\compare.py", line 89, in _produce_net_changes
comparators.dispatch("schema", autogen_context.dialect.name)(
File "C:\Users\me\PycharmProjects\MyProject\UserInterface\venv\lib\site-packages\alembic\util\langhelpers.py", line 267, in go
fn(*arg, **kw)
File "C:\Users\me\PycharmProjects\MyProject\UserInterface\venv\lib\site-packages\alembic\autogenerate\compare.py", line 125, in _autogen_for_tables
[(table.schema, table.name) for table in autogen_context.sorted_tables]
File "C:\Users\me\PycharmProjects\MyProject\UserInterface\venv\lib\site-packages\sqlalchemy\util\langhelpers.py", line 1113, in __get__
obj.__dict__[self.__name__] = result = self.fget(obj)
File "C:\Users\me\PycharmProjects\MyProject\UserInterface\venv\lib\site-packages\alembic\autogenerate\api.py", line 443, in sorted_tables
result.extend(m.sorted_tables)
File "C:\Users\me\PycharmProjects\MyProject\UserInterface\venv\lib\site-packages\sqlalchemy\sql\schema.py", line 4697, in sorted_tables
return ddl.sort_tables(
File "C:\Users\me\PycharmProjects\MyProject\UserInterface\venv\lib\site-packages\sqlalchemy\sql\ddl.py", line 1213, in sort_tables
for (t, fkcs) in sort_tables_and_constraints(
File "C:\Users\me\PycharmProjects\MyProject\UserInterface\venv\lib\site-packages\sqlalchemy\sql\ddl.py", line 1291, in sort_tables_and_constraints
dependent_on = fkc.referred_table
File "C:\Users\me\PycharmProjects\MyProject\UserInterface\venv\lib\site-packages\sqlalchemy\sql\schema.py", line 3808, in referred_table
return self.elements[0].column.table
File "C:\Users\me\PycharmProjects\MyProject\UserInterface\venv\lib\site-packages\sqlalchemy\util\langhelpers.py", line 1113, in __get__
obj.__dict__[self.__name__] = result = self.fget(obj)
File "C:\Users\me\PycharmProjects\MyProject\UserInterface\venv\lib\site-packages\sqlalchemy\sql\schema.py", line 2507, in column
raise exc.NoReferencedTableError(
sqlalchemy.exc.NoReferencedTableError: Foreign key associated with column 'roles_users.role_id' could not find table 'role' with which to generate a foreign key to target column 'id'
The User and Role models look like (pruned for length):
### USER
#
import uuid
from flask_security import UserMixin
from sqlalchemy import Column, Integer, String, Boolean, Unicode, DateTime, func
from sqlalchemy.orm import relationship, backref
from app.database import Base
class User(Base, UserMixin):
__tablename__ = 'user'
__table_args__ = {"schema": "ExistingSchemaName"}
id = Column(Integer,primary_key=True)
email = Column(Unicode(255),nullable=False,server_default=u'',unique=True)
...
# Flask-Security-Too fields
fs_uniquifier = Column(String(255), unique=True, nullable=False)
confirmed_at = Column(DateTime())
last_login_at = Column(DateTime())
current_login_at = Column(DateTime())
last_login_ip = Column(String(100))
current_login_ip = Column(String(100))
login_count = Column(Integer)
# Relationships
child_roles = relationship('Role',back_populates='parent_user')
import uuid
### ROLE
#
from flask_security import RoleMixin
from sqlalchemy import Column, Integer, String, ForeignKey
from sqlalchemy.orm import relationship
from app.database import Base
class Role(Base, RoleMixin):
__tablename__ = 'role'
__table_args__ = {"schema": "ExistingSchemaName"}
id = Column(Integer(),primary_key=True)
name = Column(String(64),nullable=False,server_default=u'',unique=True)
description = Column(String(255),nullable=False,server_default=u'')
# Have also tried adding schema name to ForeignKey without success, as in:
# user_id = Column(Integer,ForeignKey("ExistingSchemaName.user.id"))
user_id = Column(Integer,ForeignKey("user.id"))
parent_user = relationship("User",back_populates="child_roles")
The app.database.py file looks like:
from .settings import SQLALCHEMY_DATABASE_URI
from sqlalchemy import create_engine
from sqlalchemy.orm import scoped_session, sessionmaker
from sqlalchemy.ext.declarative import declarative_base
engine = create_engine(SQLALCHEMY_DATABASE_URI)
db = scoped_session(
sessionmaker(
autocommit=False,
autoflush=False,
bind=engine
)
)
Base = declarative_base()
Base.query = db.query_property()
def init_db():
import app.models
Base.metadata.create_all(bind=engine)
I think that I have set the one-to-many relationships correctly but I do not know what I am missing.

Why is Django's factory-boy trying to create a factory when I'm not asking it to?

I'm trying to use Django's factory-bot module to create factories for my models. I'm also using pytest. I have created these factories ...
import factory
from maps.models import CoopType, Coop
from address.models import AddressField
from phonenumber_field.modelfields import PhoneNumberField
from address.models import State, Country, Locality
class CountryFactory(factory.DjangoModelFactory):
"""
Define Country Factory
"""
class Meta:
model = Country
name = "Narnia"
code = "NN"
class StateFactory(factory.DjangoModelFactory):
"""
Define State Factory
"""
class Meta:
model = State
name = "Narnia"
code = "NN"
country = CountryFactory()
class LocalityFactory(factory.DjangoModelFactory):
"""
Define Locality Factory
"""
class Meta:
model = Locality
name = "Narnia"
postal_code = "60605"
state = StateFactory()
class AddressFactory(factory.DjangoModelFactory):
"""
Define Address Factory
"""
class Meta:
model = Address
street_number = "123"
route = "Rd"
raw = "123 Fake Rd"
formatted = "123 Fake Rd."
latitude = 87.1234
longitude = -100.12342
locality = LocalityFactory()
class CoopTypeFactory(factory.DjangoModelFactory):
"""
Define Coop Type Factory
"""
class Meta:
model = CoopType
I have a very simple test file thus far. It only has one test ...
import pytest
from django.test import TestCase
from .factories import CoopTypeFactory, CoopFactory
class ModelTests(TestCase):
#classmethod
def setUpTestData(cls):
print("setUpTestData: Run once to set up non-modified data for all class methods.")
#management.call_command('loaddata', 'test_data.yaml', verbosity=0)
pass
def setUp(self):
print("setUp: Run once for every test method to setup clean data.")
#management.call_command('flush', verbosity=0, interactive=False)
pass
#pytest.mark.django_db
def test_coop_type_create(self):
""" Test customer model """ # create customer model instance
coop_type = CoopTypeFactory(name="Test Coop Type Name")
assert coop_type.name == "Test Coop Type Name"
But when I run my test, it dies complaining about a duplicate key for a factory that I'm not even creating.
davea$ python manage.py test --settings=maps.test_settings
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
E
======================================================================
ERROR: tests.test_models (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: tests.test_models
Traceback (most recent call last):
File "/Users/davea/Documents/workspace/chicommons/maps/web
...
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/loader.py", line 434, in _find_test_path
module = self._get_module_from_name(name)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/loader.py", line 375, in _get_module_from_name
__import__(name)
File "/Users/davea/Documents/workspace/chicommons/maps/web/tests/test_models.py", line 3, in <module>
from .factories import CoopTypeFactory, CoopFactory
File "/Users/davea/Documents/workspace/chicommons/maps/web/tests/factories.py", line 19, in <module>
class StateFactory(factory.DjangoModelFactory):
File "/Users/davea/Documents/workspace/chicommons/maps/web/tests/factories.py", line 28, in StateFactory
country = CountryFactory()
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/factory/base.py", line 46, in __call__
return cls.create(**kwargs)
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/factory/base.py", line 564, in create
return cls._generate(enums.CREATE_STRATEGY, kwargs)
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/factory/django.py", line 141, in _generate
return super(DjangoModelFactory, cls)._generate(strategy, params)
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/factory/base.py", line 501, in _generate
return step.build()
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/factory/builder.py", line 279, in build
kwargs=kwargs,
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/factory/base.py", line 315, in instantiate
return self.factory._create(model, *args, **kwargs)
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/factory/django.py", line 185, in _create
return manager.create(*args, **kwargs)
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/django/db/models/query.py", line 417, in create
obj.save(force_insert=True, using=self.db)
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/django/db/models/base.py", line 729, in save
force_update=force_update, update_fields=update_fields)
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/django/db/models/base.py", line 759, in save_base
updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/django/db/models/base.py", line 842, in _save_table
result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/django/db/models/base.py", line 880, in _do_insert
using=using, raw=raw)
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/django/db/models/query.py", line 1125, in _insert
return query.get_compiler(using=using).execute_sql(return_id)
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1280, in execute_sql
cursor.execute(sql, params)
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/django/db/backends/utils.py", line 68, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/django/db/backends/utils.py", line 85, in _execute
return self.cursor.execute(sql, params)
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/django/db/backends/utils.py", line 85, in _execute
return self.cursor.execute(sql, params)
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/django/db/backends/mysql/base.py", line 71, in execute
return self.cursor.execute(query, args)
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/MySQLdb/cursors.py", line 209, in execute
res = self._query(query)
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/MySQLdb/cursors.py", line 315, in _query
db.query(q)
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/MySQLdb/connections.py", line 239, in query
_mysql.connection.query(self, query)
django.db.utils.IntegrityError: (1062, "Duplicate entry 'Narnia' for key 'name'")
----------------------------------------------------------------------
Ran 1 test in 0.000s
What am I doing wrong? Why is Django trying to create a factory for something I'm not creating a factory for?
You should use factory.SubFactory for defining foreign key relationships
class LocalityFactory(factory.DjangoModelFactory):
...
state = factory.SubFactory(StateFactory)
Calling the factories in your classes is creating instances

Django Haystack update index faster

I've been using Django Haystack for a while now and it's great! I have a rather heavy site with data which needs to be updated from time to time (15 to 30 mins).
When using the python manage.py update_index it takes lots of time to update the data. Is there a way to speed this up? Or maybe update only changed data if possible..
I'm currently using Django Haystack 1.2.7 with Solr as backend and Django 1.4.
Thanks!!!
EDIT:
Yes I've tried reading that part of the documentation but what I really need is a way to speed the indexing up. Maybe update only recent data instead of updating all. I've found get_updated_field but don't know how to use it. In the documentation it's only mentioned why it's used but no real examples are shown.
EDIT 2:
start = DateTimeField(model_attr='start', null=True, faceted=True, --HERE?--)
EDIT 3:
Ok i've implemented the solution bellow but when i tried rebuild_index (with 45000 data) it almost crashed my computer. After 10 mins of waiting an error appeared:
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 443, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 382, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 196, in run_from_argv
self.execute(*args, **options.__dict__)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 232, in execute
output = self.handle(*args, **options)
File "/usr/local/lib/python2.7/dist-packages/haystack/management/commands/rebuild_index.py", line 16, in handle
call_command('update_index', **options)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 150, in call_command
return klass.execute(*args, **defaults)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 232, in execute
output = self.handle(*args, **options)
File "/usr/local/lib/python2.7/dist-packages/haystack/management/commands/update_index.py", line 193, in handle
return super(Command, self).handle(*apps, **options)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 304, in handle
app_output = self.handle_app(app, **options)
File "/usr/local/lib/python2.7/dist-packages/haystack/management/commands/update_index.py", line 229, in handle_app
do_update(index, qs, start, end, total, self.verbosity)
File "/usr/local/lib/python2.7/dist-packages/haystack/management/commands/update_index.py", line 109, in do_update
index.backend.update(index, current_qs)
File "/usr/local/lib/python2.7/dist-packages/haystack/backends/solr_backend.py", line 73, in update
self.conn.add(docs, commit=commit, boost=index.get_field_weights())
File "/usr/local/lib/python2.7/dist-packages/pysolr.py", line 686, in add
m = ET.tostring(message, encoding='utf-8')
File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1127, in tostring
ElementTree(element).write(file, encoding, method=method)
File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 821, in write
serialize(write, self._root, encoding, qnames, namespaces)
File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 940, in _serialize_xml
_serialize_xml(write, e, encoding, qnames, None)
File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 940, in _serialize_xml
_serialize_xml(write, e, encoding, qnames, None)
File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 915, in _serialize_xml
write("<" + tag)
MemoryError
get_updated_field should return a string that contains the name of the attribute on the model that contains the date that the model was updated (haystack docs). A DateField with auto_now=True would be ideal for that (Django docs).
For example, my UserProfile model has a field named updated
models.py
class UserProfile(models.Model):
user = models.ForeignKey(User)
# lots of other fields snipped
updated = models.DateTimeField(auto_now=True)
search_indexes.py
class UserProfileIndex(SearchIndex):
text = CharField(document=True, use_template=True)
user = CharField(model_attr='user')
user_fullname = CharField(model_attr='user__get_full_name')
def get_model(self):
return UserProfile
def get_updated_field(self):
return "updated"
Then when I run ./manage.py update_index --age=10 it only indexes the user profiles updated in the last 10 hours.

Django Model SyncDB fail on the choices attribute due to a DB error

Here is a little error I get when trying to use syncdb on my django project.
Error:
Traceback (most recent call last):
File "manage.py", line 11, in <module>
execute_manager(settings)
File "/usr/local/lib/python2.6/dist-packages/django/core/management/__init__.py", line 438, in execute_manager
utility.execute()
File "/usr/local/lib/python2.6/dist-packages/django/core/management/__init__.py", line 379, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python2.6/dist-packages/django/core/management/base.py", line 191, in run_from_argv
self.execute(*args, **options.__dict__)
File "/usr/local/lib/python2.6/dist-packages/django/core/management/base.py", line 219, in execute
self.validate()
File "/usr/local/lib/python2.6/dist-packages/django/core/management/base.py", line 249, in validate
num_errors = get_validation_errors(s, app)
File "/usr/local/lib/python2.6/dist-packages/django/core/management/validation.py", line 28, in get_validation_errors
for (app_name, error) in get_app_errors().items():
File "/usr/local/lib/python2.6/dist-packages/django/db/models/loading.py", line 146, in get_app_errors
self._populate()
File "/usr/local/lib/python2.6/dist-packages/django/db/models/loading.py", line 61, in _populate
self.load_app(app_name, True)
File "/usr/local/lib/python2.6/dist-packages/django/db/models/loading.py", line 78, in load_app
models = import_module('.models', app_name)
File "/usr/local/lib/python2.6/dist-packages/django/utils/importlib.py", line 35, in import_module
__import__(name)
File "/opt/admin-site/adminsite/cfadmin/models.py", line 111, in <module>
class RequestLog(models.Model):
File "/opt/admin-site/adminsite/cfadmin/models.py", line 117, in RequestLog
profile = models.PositiveIntegerField(null=True,blank=True, choices=get_profiles()) # It cannot be a foreign key this is not on the same DB
File "/opt/admin-site/adminsite/cfadmin/models.py", line 107, in get_profiles
cache.set('profiles_choices', profiles_choices, 3600)
File "/usr/local/lib/python2.6/dist-packages/django/core/cache/backends/locmem.py", line 83, in set
self._set(key, pickle.dumps(value), timeout)
File "/usr/lib/python2.6/copy_reg.py", line 84, in _reduce_ex
dict = getstate()
File "/usr/local/lib/python2.6/dist-packages/django/db/models/query.py", line 61, in __getstate__
len(self)
File "/usr/local/lib/python2.6/dist-packages/django/db/models/query.py", line 81, in __len__
self._result_cache = list(self.iterator())
File "/usr/local/lib/python2.6/dist-packages/django/db/models/query.py", line 947, in iterator
for row in self.query.get_compiler(self.db).results_iter():
File "/usr/local/lib/python2.6/dist-packages/django/db/models/sql/compiler.py", line 672, in results_iter
for rows in self.execute_sql(MULTI):
File "/usr/local/lib/python2.6/dist-packages/django/db/models/sql/compiler.py", line 727, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python2.6/dist-packages/django/db/backends/postgresql_psycopg2/base.py", line 44, in execute
return self.cursor.execute(query, args)
django.db.utils.DatabaseError: relation "cfadmin_profile" does not exist
LINE 1: ...dmin_profile"."id", "cfadmin_profile"."name" FROM "cfadmin_p...
The models:
class Profile(models.Model):
name = models.CharField(max_length=256)
categories = models.ManyToManyField(Category)
def __unicode__(self):
return self.name
class Meta():
ordering = ["name"]
def get_profiles():
profiles_choices = cache.get('profiles_choices')
if profiles_choices == None:
try:
profiles_choices = Profile.objects.values_list('id','name')
cache.set('profiles_choices', profiles_choices, 3600)
except:
logging.info("Failed to retrieve the profile choices.")
pass
return profiles_choices
class Log(models.Model):
domain = models.CharField(max_length=512)
profile = models.PositiveIntegerField(null=True,blank=True, choices=get_profiles()) # this is where I get the error on Syncdb
def __unicode__(self):
return self.domain
class Meta():
ordering = ["domain"]
So if I am syncing a new project I will get the error that I mentioned earlier.
If I remove the call to my function get_profiles() in the model choices, it will synchronize with no error.
But I don't understand why I'm still getting the error even do I put a try catch within the function.
So in case of an error I would expect that it continues but instead it's completely aborting the syncdb.
Is there anyway to achieve what I'm trying to without removing the function and the put it back?
Thank you!
From the django doc " ... if you find yourself hacking choices to be dynamic, you're probably better off using a proper database table with a ForeignKey. choices is meant for static data that doesn't change much, if ever."
So you are probably better of changing your profile field in the Log model:
profile = models.ForeignKey(Profile, null=True,blank=True)
This is not the way to get choices for a model. Even if you fix your circular dependency problem, you will still have the issue that the get_choices() function will be called when the server process starts, and won't change in the meantime. Unlike default, I don't believe choices can be a callable.