I am using Flask-SQLAlchemy and I need to create a session without auto-flushing for an operation. However, the default scoped session that is created by Flask-SQLAlchemy which is accessed using db.session has auto-flushing turned on.
I'm doing bulk updates for 100k rows, and the auto-flushing is causing severe performance issues.
I tried creating a new session with auto-flushing turned off using:
from flask_sqlalchemy import SignallingSession
db_session = SignallingSession(db, autoflush=False)
And then I tried creating a new row using:
from flask_sqlalchemy import SQLAlchemy, BaseQuery
db = SQLAlchemy(app)
class Tool(db.Model):
id = db.Column(db.Integer, primary_key=True)
title = db.Column(db.String(1024), nullable=True)
add_tool = Tool()
add_tool.title = title
db_session.add(add_tool)
However, doing this throws the following exception:
Traceback (most recent call last):
File "app", line 182, in <module>
called for last item
import_tools('x.json', 1)
File "app", line 173, in import_tools
x.add_to_database()
File "app", line 126, in add_to_database
title=metadata['title'], commit=True)
File "app", line 150, in transfer
self.db_session.add(import_tool)
File ".virtualenvs/project/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 1492, in add
self._save_or_update_state(state)
File ".virtualenvs/project/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 1504, in _save_or_update_state
self._save_or_update_impl(state)
File ".virtualenvs/project/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 1759, in _save_or_update_impl
self._save_impl(state)
File ".virtualenvs/project/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 1731, in _save_impl
self._attach(state)
File ".virtualenvs/project/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 1849, in _attach
state.session_id, self.hash_key))
sqlalchemy.exc.InvalidRequestError: Object '<Tool at 0x11361f4d0>' is already attached to session '1' (this is '2')
What I believe is happening is that when the object is created, it is attached to the global scoped session that is provided by Flask-SQLAlchemy, while I want it to be attached to the session that I created. I tried looking through the code and documentation for Flask-SQLAlchemy and SQLAlchemy, but I couldn't find either the source of the problem or the solution to it.
Is there some way I can specify which session to use when the object is created using the model class? Or is there something totally different that I am missing which would be causing this problem?
SQLAlchemy sessions provide a no_autoflush context manager. This will suspend any flushes until after you exit the block.
model1 = Model(name='spam')
db.session.add(model1) # This will flush
with db.session.no_autoflush:
model2 = Model()
db.session.add(model2) # This will not
model2.name = 'eggs'
db.session.commit()
Related
Am trying to load some generated data into Django without disrupting the existing data in the site. What I have:
Saved the data as a valid JSON (validated here).
The JSON format matches the Django documentation. In previous attempts I also aligned it to the Django documentation here (slightly different field order, the result was the same).
Output errors I'm receiving are very generic and not helpful, even with verbosity=3.
The Error Prompt
Operations to perform:
Apply all migrations: workoutprogrammes
Running migrations:
Applying workoutprogrammes.0005_auto_20220415_2021...Loading '/Users/Robert/Desktop/Projects/Powerlifts/src/workoutprogrammes/fixtures/datafile_2' fixtures...
Checking '/Users/Robert/Desktop/Projects/Powerlifts/src/workoutprogrammes/fixtures' for fixtures...
Installing json fixture 'datafile_2' from '/Users/Robert/Desktop/Projects/Powerlifts/src/workoutprogrammes/fixtures'.
Traceback (most recent call last):
File "/Users/Robert/Desktop/Projects/Powerlifts/venv/lib/python3.8/site-packages/django/core/serializers/json.py", line 70, in Deserializer
yield from PythonDeserializer(objects, **options)
File "/Users/Robert/Desktop/Projects/Powerlifts/venv/lib/python3.8/site-packages/django/core/serializers/python.py", line 93, in Deserializer
Model = _get_model(d["model"])
KeyError: 'model'
The above exception was the direct cause of the following exception:... text continues on...
for obj in objects:
File "/Users/Robert/Desktop/Projects/Powerlifts/venv/lib/python3.8/site-packages/django/core/serializers/json.py", line 74, in Deserializer
raise DeserializationError() from exc
django.core.serializers.base.DeserializationError: Problem installing fixture '/Users/Robert/Desktop/Projects/Powerlifts/src/workoutprogrammes/fixtures/datafile_2.json':
auto_2022_migration.py file:
from django.db import migrations
from django.core.management import call_command
def db_migration(apps, schema_editor):
call_command('loaddata', '/filename.json', verbosity=3)
class Migration(migrations.Migration):
dependencies = [('workoutprogrammes', '0004_extable_delete_ex_table'),]
operations = [migrations.RunPython(db_migration),]
JSON file extract (start... end)
NB: all my PKs are UUIDs generated from postgresql
[{"pk":"af82d5f4-2814-4d52-b2b1-6add6cf18d3c","model":"workoutprogrammes.ex_table","fields":{"exercise_name":"Cable Alternating Front Raise","utility":"Auxiliary","mechanics":"Isolated","force":"Push","levator_scapulae":"Stabilisers",..."obliques":"Stabilisers","psoas_major":""}}]
I'm not sure what the original error was. I suspect it had to do with either editing the table/schema using psql separately from Django, OR using the somewhat outdated uuid-ossp package instead of pgcrypto package (native to Djagno via CryptoExtension()). I was never able to confirm. I did however manage to get it working by:
Deleting the table (schema, data) using psql and the app using Django. Creating a new app in Django.
Load the prev model data into this new_app/models.py. Make new migration file, first updating the CryptoExtension() operation before committing the migration.
Create Fixtures directory and relevant file (per Django specs, validated JSON online.
Validate my UUIDs online
Load the fixture data into the app. python3 manage.py loaddata /path/to/data/file_name.json
I have created a flask migrate script ,however, when I running the upgrade function, I get the following error:
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
INFO [alembic.runtime.migration] Running upgrade -> 6378428b838a, empty message
Traceback (most recent call last):
File "migrate.py", line 22, in <module>
manager.run()
File "/Users/slatifi/git/StaffTrainingLog/venv/lib/python3.7/site-packages/flask_script/__init__.py", line 417, in run
result = self.handle(argv[0], argv[1:])
File "/Users/slatifi/git/StaffTrainingLog/venv/lib/python3.7/site-packages/flask_script/__init__.py", line 386, in handle
res = handle(*args, **config)
File "/Users/slatifi/git/StaffTrainingLog/venv/lib/python3.7/site-packages/flask_script/commands.py", line 216, in __call__
return self.run(*args, **kwargs)
File "/Users/slatifi/git/StaffTrainingLog/venv/lib/python3.7/site-packages/flask_migrate/__init__.py", line 95, in wrapped
f(*args, **kwargs)
File "/Users/slatifi/git/StaffTrainingLog/venv/lib/python3.7/site-packages/flask_migrate/__init__.py", line 280, in upgrade
command.upgrade(config, revision, sql=sql, tag=tag)
File "/Users/slatifi/git/StaffTrainingLog/venv/lib/python3.7/site-packages/alembic/command.py", line 298, in upgrade
script.run_env()
File "/Users/slatifi/git/StaffTrainingLog/venv/lib/python3.7/site-packages/alembic/script/base.py", line 489, in run_env
util.load_python_file(self.dir, "env.py")
File "/Users/slatifi/git/StaffTrainingLog/venv/lib/python3.7/site-packages/alembic/util/pyfiles.py", line 98, in load_python_file
module = load_module_py(module_id, path)
File "/Users/slatifi/git/StaffTrainingLog/venv/lib/python3.7/site-packages/alembic/util/compat.py", line 173, in load_module_py
spec.loader.exec_module(module)
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "migrations/env.py", line 96, in <module>
run_migrations_online()
File "migrations/env.py", line 90, in run_migrations_online
context.run_migrations()
File "<string>", line 8, in run_migrations
File "/Users/slatifi/git/StaffTrainingLog/venv/lib/python3.7/site-packages/alembic/runtime/environment.py", line 846, in run_migrations
self.get_context().run_migrations(**kw)
File "/Users/slatifi/git/StaffTrainingLog/venv/lib/python3.7/site-packages/alembic/runtime/migration.py", line 518, in run_migrations
step.migration_fn(**kw)
File "/Users/slatifi/git/StaffTrainingLog/migrations/versions/6378428b838a_.py", line 23, in upgrade
batch_op.create_foreign_key(None, 'organisation', ['organisation'], ['id'])
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/contextlib.py", line 119, in __exit__
next(self.gen)
File "/Users/slatifi/git/StaffTrainingLog/venv/lib/python3.7/site-packages/alembic/operations/base.py", line 325, in batch_alter_table
impl.flush()
File "/Users/slatifi/git/StaffTrainingLog/venv/lib/python3.7/site-packages/alembic/operations/batch.py", line 106, in flush
fn(*arg, **kw)
File "/Users/slatifi/git/StaffTrainingLog/venv/lib/python3.7/site-packages/alembic/operations/batch.py", line 390, in add_constraint
raise ValueError("Constraint must have a name")
ValueError: Constraint must have a name
I have seen other people with the same error and they simply added render_as_batch to the env.py file. I did this but I still get the same error. Any thoughts?
Note: This is the modification I made in the env.py file:
with connectable.connect() as connection:
context.configure(
connection=connection,
target_metadata=target_metadata,
process_revision_directives=process_revision_directives,
**current_app.extensions['migrate'].configure_args,
render_as_batch=True
)
This is the upgrade script created by the migration
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision = '2838e3e96536'
down_revision = None
branch_labels = None
depends_on = None
def upgrade():
# ### commands auto generated by Alembic - please adjust! ###
with op.batch_alter_table('user', schema=None) as batch_op:
batch_op.add_column(sa.Column('organisation', sa.String(length=5), nullable=False))
batch_op.create_foreign_key(None, 'organisation', ['organisation'], ['id'])
# ### end Alembic commands ###
def downgrade():
# ### commands auto generated by Alembic - please adjust! ###
with op.batch_alter_table('user', schema=None) as batch_op:
batch_op.drop_constraint(None, type_='foreignkey')
batch_op.drop_column('organisation')
# ### en
d Alembic commands ###
This is normal because SQLite3 doesn't support ALTER tables.
You can pass the render_as_batch=True during the instantiation of Flask-Migrate like this :
migrate = Migrate(app,db,render_as_batch=True)
Don't need to modify the env file of Flask-Migrate.
In addition, to totally avoid the problem for your futures migrations, you can create a constraint naming templates for all types of constraints to the SQLAlchemy metadata, and then I think you will get consistent names.
See how to do this in the Flask-SQLAlchemy documentation. I'm copying the code example from the docs below for your convenience:
from sqlalchemy import MetaData
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
convention = {
"ix": 'ix_%(column_0_label)s',
"uq": "uq_%(table_name)s_%(column_0_name)s",
"ck": "ck_%(table_name)s_%(constraint_name)s",
"fk": "fk_%(table_name)s_%(column_0_name)s_%(referred_table_name)s",
"pk": "pk_%(table_name)s"
}
metadata = MetaData(naming_convention=convention)
db = SQLAlchemy(app, metadata=metadata)
The other answers show how to configure Flask to use named constraints in the future, but that doesn't solve the problem of dropping existing unnamed constraints in Alembic migrations. To handle any existing unnamed constraints, you need to also define the naming convention in the migration script, as explained in the Alembic docs.
For example, suppose you're trying to do a migration that adds ON DELETE CASCADE to an existing foreign key in your test table. If you've followed one of the other answers (e.g. https://stackoverflow.com/a/62651160/470844) and added the naming_convention to the Flask init script, then flask db upgrade will generate something like this:
def upgrade():
with op.batch_alter_table('test', schema=None) as batch_op:
batch_op.drop_constraint(None, type_='foreignkey')
batch_op.create_foreign_key(batch_op.f('fk_test_user_id_user'), 'user', ['user_id'], ['id'], ondelete='CASCADE')
Note that while the create_foreign_key call uses a constraint name (i.e. fk_test_user_id_user), the drop_constraint call still uses None as the constraint name, which will cause ValueError: Constraint must have a name error in this question's title.
To fix that, you need to edit the migration to use the naming_convention, and replace the None with the generated constraint name. For example, you'd change the above upgrade to:
naming_convention = {
"ix": 'ix_%(column_0_label)s',
"uq": "uq_%(table_name)s_%(column_0_name)s",
"ck": "ck_%(table_name)s_%(column_0_name)s",
"fk": "fk_%(table_name)s_%(column_0_name)s_%(referred_table_name)s",
"pk": "pk_%(table_name)s"
}
def upgrade():
with op.batch_alter_table('test', schema=None, naming_convention=naming_convention) as batch_op:
batch_op.drop_constraint('fk_test_user_id_user', type_='foreignkey')
batch_op.create_foreign_key(batch_op.f('fk_test_user_id_user'), 'user', ['user_id'], ['id'], ondelete='CASCADE')
(In this example, you'd also need to disable SQLite foreign key support while running the migration script, e.g. using the technique here, to avoid the batch migration itself triggering a cacading delete. The problem is documented here.)
Installing Flask-Migrate helps to avoid code references to declarative_base or modifying the env.py file.
For apps with app factory, split the metadata assignment and the db initiation between the extensions.py and __init__.py files.
extensions.py:
from sqlalchemy import MetaData
from flask_sqlalchemy import SQLAlchemy
metadata = MetaData(
naming_convention={
"ix": 'ix_%(column_0_label)s',
"uq": "uq_%(table_name)s_%(column_0_name)s",
"ck": "ck_%(table_name)s_%(constraint_name)s",
"fk": "fk_%(table_name)s_%(column_0_name)s_%(referred_table_name)s",
"pk": "pk_%(table_name)s"
}
)
db=SQLAlchemy(metadata=metadata)
__init__.py:
from flask import Flask, Blueprint
from config import ...
from flask_migrate import Migrate
from extensions import db
...
def create_app():
app = Flask(__name__,
template_folder='../app/templates')
app.config.from_object(config[...])
config[...].init_app(app)
db.init_app(app)
with app.app_context():
db.create_all()
migrate = Migrate(app, db)
...
from .templates.main import main
app.register_blueprint(main)
return app
My model (before) contains client-side defaults:
created_ts = db.Column(db.DateTime(timezone=True), default=dt.datetime.now)
My model (after) now contains server-side defaults:
created_ts = db.Column(db.DateTime(timezone=True), server_default=text('NOW()'))
However, I now start seeing the error:
InvalidRequestError: This session is in 'committed' state; no further SQL can be emitted within this transaction.
In my models_committed hook:
#models_committed.connect_via(app)
def handle(sender, changes):
for model, operation in changes:
model.to_dict() # error here
I stole to_dict from flask_sandboy:
def to_dict(self):
"""Return dict representation of class by iterating over database
columns."""
value = {}
for column in self.__table__.columns:
attribute = getattr(self, column.name) # error here
if isinstance(attribute, datetime.datetime):
attribute = str(attribute)
value[column.name] = attribute
return value
So, getattr(self, column.name) seems to trigger the server-side default somehow (presumably, since that's the change I introduced).
From this line in my own code, I provide the rest of the stack trace:
File "/code/models/session.py", line 20, in to_dict
attribute = getattr(self, column.name)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 239, in __get__
return self.impl.get(instance_state(instance), dict_)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 589, in get
value = callable_(state, passive)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/state.py", line 433, in __call__
self.manager.deferred_scalar_loader(self, toload)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 613, in load_scalar_attributes
only_load_props=attribute_names)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 235, in load_on_ident
return q.one()
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2398, in one
ret = list(self)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2441, in __iter__
return self._execute_and_instances(context)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2454, in _execute_and_instances
close_with_result=True)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2445, in _connection_from_session
**kw)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 880, in connection
execution_options=execution_options)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 885, in _connection_for_bind
engine, execution_options)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 305, in _connection_for_bind
self._assert_active()
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 196, in _assert_active
"This session is in 'committed' state; no further "
InvalidRequestError: This session is in 'committed' state; no further SQL can be emitted within this transaction.
How do I get around this problem?
pip freeze:
aniso8601==0.92
blinker==1.3
boto==2.36.0
Flask==0.10.1
Flask-Cors==1.10.3
Flask-HTTPAuth==2.4.0
Flask-SQLAlchemy==2.0
gunicorn==19.3.0
itsdangerous==0.24
psycopg2==2.6
pytz==2014.10
six==1.9.0
SQLAlchemy==0.9.9
Werkzeug==0.10.1
TL;DR solution #4 below works
The created_ts field is not known in that request thread, nor in the subsequent signal handler, since the value is chosen server-side by PostgresQL.
getattr(self, column.name) then tries to hydratepopulate that value by going back to the DB server, but unfortunately is now outside of a transaction.
A few options here:
Start a new session in the signal handler so as to retrieve the value from the DB (this is an untested suggestion)
Use the built-in _dict to get the model's state, but it won't have the created_ts field, because it is not known at this point:
Code
modeldict = dict(model.__dict__)
modeldict.pop('_sa_instance_state', None)
Stick with default=dt.datetime.now, but non-Python apps which write to the database need to provide created_ts.
So, the trade-off is whether the consumer of the model (as alluded to with the pika comment) needs created_ts or not.
(tested to work) Thanks to a combination of advice from agronholm on the #sqlalchemy IRC room and kbussel on this ticket, I tried to serialize the data while the session is still open, and when I'm sure the data is committed, I re-use the serialized data and by-pass going back to the DB:
from sqlalchemy.event import listens_for
from flask.ext.sqlalchemy import SignallingSession
#listens_for(SignallingSession, 'after_flush')
def after_flush_handler(session, tx):
try:
d = session._model_changes
except AttributeError:
return
if d:
changes = []
for model, operation in list(d.values()):
changes.append((model.to_dict(), operation))
session.info['my_changes'] = changes
d.clear()
#listens_for(SignallingSession, 'after_commit')
def after_commit_handler(session):
if 'my_changes' in session.info:
changes = session.info['my_changes']
for model, operation in changes:
# use model here, with all data populated
I decided to use factory_boy in my simple django application for test purposes.
But I had a problem with simple example. Here is the code of my simple test.
from django.utils import unittest
from ..models import Server, ServerAddress, L2TPServer, serialize_open_vpn_server_json
from factory import django as django_factory
class SshOpenVpnServerFactory(django_factory.DjangoModelFactory):
class Meta:
model = L2TPServer
django_get_or_create = ('name', 'address')
name = 'Hello'
address = 'Nono'
class ServersTestCase(unittest.TestCase):
def test_serialize_server_info(self):
print Server.objects.all()
SshOpenVpnServerFactory.build()
When test-runner executes this test, I get an error:
Traceback (most recent call last):
File "/Users/green/Development/Wasel/experimental/wasel_services/packages/waselcore/backend/tests/test_models.py", line 20, in test_serialize_server_info
SshOpenVpnServerFactory.build()
File "/Users/green/Development/Wasel/experimental/env/wasel_sevices/lib/python2.7/site-packages/factory/base.py", line 504, in build
attrs = cls.attributes(create=False, extra=kwargs)
File "/Users/green/Development/Wasel/experimental/env/wasel_sevices/lib/python2.7/site-packages/factory/base.py", line 365, in attributes
force_sequence=force_sequence,
File "/Users/green/Development/Wasel/experimental/env/wasel_sevices/lib/python2.7/site-packages/factory/containers.py", line 265, in build
sequence = self.factory._generate_next_sequence()
File "/Users/green/Development/Wasel/experimental/env/wasel_sevices/lib/python2.7/site-packages/factory/base.py", line 338, in _generate_next_sequence
cls._setup_counter()
File "/Users/green/Development/Wasel/experimental/env/wasel_sevices/lib/python2.7/site-packages/factory/base.py", line 318, in _setup_counter
first_seq = cls._setup_next_sequence()
File "/Users/green/Development/Wasel/experimental/env/wasel_sevices/lib/python2.7/site-packages/factory/django.py", line 83, in _setup_next_sequence
manager = cls._get_manager(model)
File "/Users/green/Development/Wasel/experimental/env/wasel_sevices/lib/python2.7/site-packages/factory/django.py", line 76, in _get_manager
return target_class.objects
AttributeError: 'NoneType' object has no attribute 'objects'
Where am I wrong? Does factory_boy support django 1.6.5?
The proposed syntax using the Meta class will only be released as part of the (unreleased) Factory Boy 2.4. Use SshOpenVpnServerFactory.FACTORY_FOR instead. Related issue: https://github.com/rbarrois/factory_boy/issues/143
So I was developing an app in Django and needed a function from the 1.4 version so I decided to update.
But then a weird error appeared when I wanted to do syncdb
I am using the new manage.py and as You can see it makes some of the tables but then fails :
./manage.py syncdb
Creating tables ...
Creating table auth_permission
Creating table auth_group_permissions
Creating table auth_group
Creating table auth_user_user_permissions
Creating table auth_user_groups
Creating table auth_user
Creating table django_content_type
Creating table django_session
Creating table django_site
Traceback (most recent call last):
File "./manage.py", line 9, in <module>
execute_from_command_line(sys.argv)
File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/core/management/__init__.py", line 443, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/core/management/__init__.py", line 382, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/core/management/base.py", line 196, in run_from_argv
self.execute(*args, **options.__dict__)
File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/core/management/base.py", line 232, in execute
output = self.handle(*args, **options)
File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/core/management/base.py", line 371, in handle
return self.handle_noargs(**options)
File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/core/management/commands/syncdb.py", line 91, in handle_noargs
sql, references = connection.creation.sql_create_model(model, self.style, seen_models)
File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/db/backends/creation.py", line 44, in sql_create_model
col_type = f.db_type(connection=self.connection)
TypeError: db_type() got an unexpected keyword argument 'connection'
I had the same issue, the definition for my custom field was missing the connection parameter.
from django.db import models
class BigIntegerField(models.IntegerField):
def db_type(self, connection):
return "bigint"
Although already old, answered and accepted question but I am adding my understanding I have added it because I am not using customized type and it is a Django Evolution error (but not syncdb)evolve --hint --execute. I think it may be helpful for someone in future. .
I am average in Python and new to Django. I also encounter same issue when I added some new features to my existing project. To add new feature I had to add some new fields of models.CharField() type,as follows.
included_domains = models.CharField(
"set of comma(,) seprated list of domains in target emails",
default="",
max_length=it_len.EMAIL_LEN*5)
excluded_domains = models.CharField(
"set of comma(,) seprated list of domains NOT in target emails",
default="",
max_length=it_len.EMAIL_LEN*5)
The Django version I am using is 1.3.1:
$ python -c "import django; print django.get_version()"
1.3.1 <--------# version
$python manage.py syncdb
Project signature has changed - an evolution is required
Django Evolution: Django Evolution is an extension to Django that allows you to track changes in your models over time, and to update the database to reflect those changes.
$ python manage.py evolve --hint
#----- Evolution for messagingframework
from django_evolution.mutations import AddField
from django.db import models
MUTATIONS = [
AddField('MessageConfiguration', 'excluded_domains', models.CharField, initial=u'', max_length=300),
AddField('MessageConfiguration', 'included_domains', models.CharField, initial=u'', max_length=300)
]
#----------------------
Trial evolution successful.
Run './manage.py evolve --hint --execute' to apply evolution.
The trial was susses and when I tried to apply changes in DB
$ python manage.py evolve --hint --execute
Traceback (most recent call last):
File "manage.py", line 25, in <module>
execute_manager(settings)
File "/var/www/sites/www.taxspanner.com/django/core/management/__init__.py", line 362, in execute_manager
utility.execute()
File "/var/www/sites/www.taxspanner.com/django/core/management/__init__.py", line 303, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/var/www/sites/www.taxspanner.com/django/core/management/base.py", line 195, in run_from_argv
self.execute(*args, **options.__dict__)
File "/var/www/sites/www.taxspanner.com/django/core/management/base.py", line 222, in execute
output = self.handle(*args, **options)
File "/usr/local/lib/python2.7/dist-packages/django_evolution-0.6.9.dev_r225-py2.7.egg/django_evolution/management/commands/evolve.py", line 60, in handle
self.evolve(*app_labels, **options)
File "/usr/local/lib/python2.7/dist-packages/django_evolution-0.6.9.dev_r225-py2.7.egg/django_evolution/management/commands/evolve.py", line 140, in evolve
database))
File "/usr/local/lib/python2.7/dist-packages/django_evolution-0.6.9.dev_r225-py2.7.egg/django_evolution/mutations.py", line 426, in mutate
return self.add_column(app_label, proj_sig, database)
File "/usr/local/lib/python2.7/dist-packages/django_evolution-0.6.9.dev_r225-py2.7.egg/django_evolution/mutations.py", line 438, in add_column
sql_statements = evolver.add_column(model, field, self.initial)
File "/usr/local/lib/python2.7/dist-packages/django_evolution-0.6.9.dev_r225-py2.7.egg/django_evolution/db/common.py", line 142, in add_column
f.db_type(connection=self.connection), # <=== here f is field class object
TypeError: db_type() got an unexpected keyword argument 'connection'
To understand this exception I check that this exception is something similar to:
>>> def f(a):
... print a
...
>>> f('b', b='a')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: f() got an unexpected keyword argument 'b'
>>>
So the function signature has been changed.
Because I have not added any new customized or enum fields but only two similar fields that was already in model and char type field is supported by most of database (I am ussing PostgreSQL) even I was getting this error!
Then I read from #: Russell Keith-Magee-4 Reply.
What you've hit here is the end of the deprecation cycle for code that
doesn't support multiple databases.
In Django 1.2, we introduced multiple database support; in order to
support this, the prototype for get_db_preb_lookup() and
get_db_prep_value() was changed.
For backwards compatibility, we added a shim that would transparently
'fix' these methods if they hadn't already been fixed by the
developer.
In Django 1.2, the usage of these shims raised a
PendingDeprecationWarning. In Django 1.3, they raised a
DeprecationWarning.
Under Django 1.4, the shim code was been removed -- so any code that
wasn't updated will now raise errors like the one you describe.
But I am not getting any DeprecationWarning warning assuming because of newer version of Django Evolution.
But from above quote I could understand that to support multiple databases function signature is added and an extra argument connection is needed. I also check the db_type() signature in my installation of Django as follows:
/django$ grep --exclude-dir=".svn" -n 'def db_type(' * -R
contrib/localflavor/us/models.py:8: def db_type(self):
contrib/localflavor/us/models.py:24: def db_type(self):
:
:
Ialso refer of Django documentation
Field.db_type(self, connection):
Returns the database column data type for the Field, taking into account the connection
object, and the settings associated with it.
And Then I could understand that to resolve this issue I have to inherited models.filed class and overwrite def db_type() function. And because I am using PostgreSQL in which to create 300 chars type field I need to return 'char(300)'. In my models.py I added:
class CharMaxlengthN(models.Field):
def db_type(self, connection):
return 'char(%d)' % self.max_length # because I am using postgresql
If you encounter similar problem please check your underline DB's manual that which type of column you need to create and return a string.
And changed the definition of new fields (that I need to add) read comments:
included_domains = CharMaxlengthN( # <--Notice change
"set of comma(,) seprated list of domains in target emails",
default="",
max_length=it_len.EMAIL_LEN*5)
excluded_domains = CharMaxlengthN( # <-- Notice change
"set of comma(,) seprated list of domains NOT in target emails",
default="",
max_length=it_len.EMAIL_LEN*5)
Then I executed same command that was failing previously:
t$ python manage.py evolve --hint --execute
You have requested a database evolution. This will alter tables
and data currently in the None database, and may result in
IRREVERSABLE DATA LOSS. Evolutions should be *thoroughly* reviewed
prior to execution.
Are you sure you want to execute the evolutions?
Type 'yes' to continue, or 'no' to cancel: yes
Evolution successful.
I also check my DB and tested my new added features It is now working perfectly, and no DB problem.
If you wants to create ENUM field read Specifying a mySQL ENUM in a Django model.
Edit: I realized instead of sub classing models.Field I should have inherit more specific subclass that is models.CharField.
Similarly I need to create Decimal DB fields so I added following class in model:
class DecimalField(models.DecimalField):
def db_type(self, connection):
d = {
'max_digits': self.max_digits,
'decimal_places': self.decimal_places,
}
return 'numeric(%(max_digits)s, %(decimal_places)s)' % d