I'm exploring a new DDD project using SQLAlchemy and Alembic for the first time. I'd like to use imperative mapping to isolate my domain objects.
All the doc I can find about auto generating migrations with Alembic is using declarative mapping. Is it because I have to manually write all migrations if I want to use imperative mapping ?
I had to import the metada of the Table I manually defined
I came to this page because autogenerating migrations no longer worked with the upgrade to SQLAlchemy 1.4, as the metadata were no longer recognized and the automatically generated migration deleted every table (DROP table in upgrade, CREATE TABLE in downgrade).
I have first tried to import the tables metadata like this :
target_metadata = [orm.table_company.metadata, orm.table_user.metadata, orm.table_3.metadata, orm.table_4.metadata]
It resulted in the following error code:
alembic/autogenerate/api.py", line 462, in table_key_to_table
ValueError: Duplicate table keys across multiple MetaData objects: "tb_company", "tb_user", "tb_3", "tb_4"
I have found that rather than importing one metadata object per table, you can access it in a single pass with target_metadata = orm.mapper_registry.metadata :
SQL Alchemy 1.4
adapters/orm.py
from myapp import domain
from sqlalchemy.orm import registry
from sqlalchemy.schema import MetaData
metadata = MetaData()
mapper_registry = registry(metadata=metadata)
# define your tables here
table_user = Table(
"tb_user",
mapper_registry.metadata,
Column("id", Integer, primary_key=True, autoincrement=True),
Column(
"pk_user",
UUID(as_uuid=True),
primary_key=True,
server_default=text("uuid_generate_v4()"),
default=uuid.uuid4,
unique=True,
nullable=False,
),
Column(
"fk_company",
UUID(as_uuid=True),
ForeignKey("tb_company.pk_company"),
),
Column("first_name", String(255)),
Column("last_name", String(255)),
)
# map your domain objects to the tables
def start_mappers():
mapper_registry.map_imperatively(
domain.model.User,
table_user,
properties={
"company": relationship(
domain.Company,
backref="users"
)
},
)
alembic/env.py
from myapp.adapters import orm
# (...)
target_metadata = orm.mapper_registry.metadata
SQL Alchemy 1.3
Using classical / imperative mapping, Alembic could generate migrations from SQL Alchemy 1.3 with the following syntax:
adapters/orm.py
from myapp import domain
# (...)
from sqlalchemy import Table, Column
from sqlalchemy.orm import mapper, relationship
from sqlalchemy.schema import MetaData
metadata = MetaData()
# define your tables here
table_user = Table(
"tb_user",
metadata,
Column("id", Integer, primary_key=True, autoincrement=True),
# (...)
)
# map your domain objects to the tables
def start_mappers():
user_mapper = mapper(
domain.User,
tb_user,
properties={
"company": relationship(
domain.Company,
backref="users"
),
},
)
alembic/env.py
from myapp.adapters import orm
# (...)
target_metadata = orm.metadata
Related
I am trying to use Django (django 1.11.4) to read data from a SQL-Server view (sql server 2012 - I use sql_server.pyodbc [aka django-pyodbc] for this), and nothing seems to work.
Here's my model:
class NumUsersAddedPerWeek(models.Model):
id = models.BigIntegerField(primary_key=True)
year = models.IntegerField('Year')
week = models.IntegerField('Week')
num_added = models.IntegerField('Number of Users Added')
if not settings.RUNNING_UNITTESTS:
class Meta:
managed = False
db_table = 'num_users_added_per_week'
and here's how the database view is created:
create view num_users_added_per_week
as
select row_number() over(order by datepart(year, created_at), datepart(week, created_at)) as 'id',
datepart(year, created_at) as 'year', datepart(week, created_at) as 'week', count(*) as 'num_added'
from [<database name>].[dbo].[<table name>]
where status = 'active' and created_at is not null
group by datepart(year, created_at), datepart(week, created_at)
The view works just fine by itself (e.g., running 'select * from num_users_added_per_week' runs just fine (and very quickly)...
I used the following django command (i.e., 'action') to try 3 different ways of attempting to pull data via the model, and none of them worked (although, judging from other posts, these approaches seemed to work with previous versions of django) :(:
from django.core.management.base import BaseCommand, CommandError
from <project name>.models import NumUsersAddedPerWeek
from django.db import connection
class Command(BaseCommand):
def handle(self, *args, **options):
# attempt # 1 ...
num_users_info = NumUsersAddedPerWeek.objects.all()
info = num_users_info.first()
for info in num_users_info:
print(info)
# attempt # 2 ...
cursor = connection.cursor()
cursor.execute('select * from num_users_added_per_week')
result = cursor.fetchall()
# attempt # 3 ...
num_users_info = NumUsersAddedPerWeek.objects.raw('select * from num_users_added_per_week')
for info in num_users_info:
print(info)
Each of the 3 different approaches gives me the same error: "('42S02', "[42S02] [Microsoft][ODBC SQL Server Driver][SQL Server]Invalid object name 'num_users_added_per_week'. (208) (SQLExecDirectW)")"
Please note: my migrations are running just fine - adding class Meta: managed = False is crucial with latest versions of Django in situations where you do not want migrations to create / update / delete your sql table structure...
I figured it out - I have a custom Database Router (in settings.DATABASE_ROUTERS) that I had not properly added this to (I am doing this because the project has multiple databases - see Multi-DB to see why and how to do this). (So boneheaded bug on my part)
But here's what I found out: It turns out all three of the methods I used should work, if you have 1 database in your project. If you have multiple databases then you can query the database through your model object (e.g., <Model Name>.objects.all()) or through raw sql, but you have to specify the raw sql via your model (e.g., <Model Name>.objects.raw(<select * from <view name>)) - otherwise your Database Router will not know which database to use.
I added a unique attribute uid for MyModel model:
class MyModel(db.Model):
...
uid = db.Column(db.String(50), nullable=False)
...
__table_args__ = (UniqueConstraint('uid', name='unique_uid'),)
I have a migration:
def upgrade():
op.add_column('mymodel', sa.Column('uid', sa.String(length=50), nullable=True))
mymodel = table('mymodel', column('uid'))
op.execute(mymodel.update().values(uid=generate_uid()))
op.create_unique_constraint('unique_uid', 'mymodel', ['uid'])
op.alter_column(
table_name='mymodel',
column_name='uid',
nullable=False
)
On run db upgrade i've got an error:
...
psycopg2.IntegrityError: could not create unique index "unique_uid"
DETAIL: Key (uid)=(c92U6txA2) is duplicated.
How to set unique value for each row on op.execute(mymodel.update().values(uid=generate_uid()))?
$ pip freeze
alembic==0.8.6
Flask==0.10.1
Flask-Fixtures==0.3.3
Flask-Login==0.3.2
Flask-Migrate==1.8.0
Flask-Script==2.0.5
Flask-SQLAlchemy==2.1
itsdangerous==0.24
Jinja2==2.8
Mako==1.0.4
MarkupSafe==0.23
psycopg2==2.6.1
python-editor==1.0
requests==2.10.0
SQLAlchemy==1.0.13
Werkzeug==0.11.9
The possible solution:
from sqlalchemy.orm import Session
from alembic import op
import sqlalchemy as sa
def upgrade():
conn = op.get_bind()
session = Session(bind=conn)
op.add_column('mymodel', sa.Column('uid', sa.String(length=50), nullable=True))
for item in session.query(MyModel).filter_by(uid=None):
item.uid = generate_uid()
session.commit()
op.create_unique_constraint('unique_uid', 'mymodel', ['uid'])
op.alter_column(
table_name='mymodel',
column_name='uid',
nullable=False
)
The migration script that you wrote puts the same uid on all the rows, the generate_uid() function is called once, and its result is then added into all the rows. So then when the index is created you get a duplicated key error.
Depending on what your uids are and the database you maybe able to write a single SQL statement that creates unique ids for all your rows, but the safe bet would be to do a loop and update each row separately.
I have a simple Flask-SQLAlchemy model (with event listener to create trigger):
from flask_sqlalchemy import SQLAlchemy
db = SQLAlchemy()
class Confirm(db.Model):
created = db.Column(db.DateTime, default=db.func.current_timestamp(), nullable=False)
modified = db.Column(db.DateTime, default=db.func.current_timestamp(), onupdate=db.func.current_timestamp(), nullable=False)
id = db.Column(db.String(36), primary_key=True)
class ConfirmOld(db.Model):
orig_created = db.Column(db.DateTime)
orig_modified = db.Column(db.DateTime)
orig_id = db.Column(db.String(36))
confirm_delete = DDL('''\
CREATE TRIGGER confirm_delete
BEFORE DELETE
ON confirm FOR EACH ROW
BEGIN
INSERT INTO confirm_old ( orig_created, orig_modified, orig_id )
VALUES ( OLD.created, OLD.modified, OLD.id );
END;
''')
event.listen(Confirm.__table__, 'after_create', confirm_delete)
When I run Alembic migrate and upgrade, the TRIGGER is not created (in MySQL). However, it is created and works properly when I use db.create_all().
Is it possible to get Alembic / Flask-Migrate to create and manage my triggers (i.e., custom DDL that is run on after_create events)?
I have faced the same issue tried a solution with Replacable object but didn't work:
I manage to make it work by editing the migration script and execute the trigger creation query.
Here are the step:
Run flask db migrate -m 'adding custom trigger on table x it will generate a migration script for you under version sub-folder of migration folder.
check the folder created under version and edit it like this :
create your trigger query like this :
in the file :
trigger = '''
CREATE TRIGGER confirm_delete
BEFORE DELETE
ON confirm FOR EACH ROW
BEGIN
INSERT INTO confirm_old ( orig_created, orig_modified, orig_id )
VALUES ( OLD.created, OLD.modified, OLD.id );
END;
'''
in the upgrade method :
add this line :
def upgrade():
# ### commands auto generated by Alembic - please adjust! ###
# ### end Alembic commands ###
### add your queries here execute
op.execute(trigger)
If you run flask db upgrade it will execute the query and update the database
to downgrade the database add this in the downgrade method:
def downgrade():
# ### commands auto generated by Alembic - please adjust! ###
# ### end Alembic commands ###
op.execute('drop trigger if exists confirm_delete on confirm cascade;')
If you check your database change will be applied .
PS : The more elegant solution should be what is suggest here
with Replaceable object , tried it but It doesn't work may be my alembic is not update .
Here is how the solution should looks like:
create a ReplaceableObjects class :
class ReplaceableObject(object):
def __init__(self, name, sqltext):
self.name = name
self.sqltext = sqltext
instantiate it with your query statement.
delete_trigger = ReplaceableObject('delete_trigger', trigger)
Update your upgrade and downgrade function like this :
def upgrade():
op.create_sp(delete_trigger)
def downgrade():
op.drop_sp(delete_trigger)
Hope it will helps others...
in Flask the listen is ignored.
Fixed this by using Table instead.
def after_create_table_handler(table: Table, conn: Connection, **kwargs):
pass
event.listen(Table, 'after_create', after_create_table_handler)
I'm working on a project with Flask, SQLAlchemy, Alembic and their wrappers for Flask (Flask-SQLAlchemy and Flask-Migrate). I have four migrations:
1c5f54d4aa34 -> 4250dfa822a4 (head), Feed: Countries
312c1d408043 -> 1c5f54d4aa34, Feed: Continents
41984a51dbb2 -> 312c1d408043, Basic Structure
<base> -> 41984a51dbb2, Init Alembic
When I start a new and clean database and try to run the migrations I get an error:
vagrant#precise32:/vagrant$ python manage.py db upgrade
...
sqlalchemy.exc.ProgrammingError: (ProgrammingError) relation "continent" does not exist
...
If I ask Flask-Migrate to run all migrations but the last, it works. If after that I run the upgrade command again, it works – that is, it fully upgrades my database without a single change in code:
vagrant#precise32:/vagrant$ python manage.py db upgrade 312c1d408043
INFO [alembic.migration] Context impl PostgresqlImpl.
INFO [alembic.migration] Will assume transactional DDL.
INFO [alembic.migration] Running upgrade -> 41984a51dbb2, Init Alembic
INFO [alembic.migration] Running upgrade 41984a51dbb2 -> 312c1d408043, Basic Structure
vagrant#precise32:/vagrant$ python manage.py db upgrade
INFO [alembic.migration] Context impl PostgresqlImpl.
INFO [alembic.migration] Will assume transactional DDL.
INFO [alembic.migration] Running upgrade 312c1d408043 -> 1c5f54d4aa34, Feed: Continents
INFO [alembic.migration] Running upgrade 1c5f54d4aa34 -> 4250dfa822a4, Feed: Countries
TL;DR
The last migration (Feed: Countries) run queries on the table fed by the previous one (Feed: Continents). If I have the continents table create and fed, the scripts should work. But it doesn't.
Why do I have to stop the migration process between then to re-start it in another command? I really don't get this. Is it some command Alembic executes after a serie of migrations? Any ideas?
Just in case
My models are defined as follows:
class Country(db.Model):
__tablename__ = 'country'
id = db.Column(db.Integer, primary_key=True)
alpha2 = db.Column(db.String(2), index=True, unique=True)
title = db.Column(db.String(140))
continent_id = db.Column(db.Integer, db.ForeignKey('continent.id'))
continent = db.relationship('Continent', backref='countries')
def __repr__(self):
return '<Country #{}: {}>'.format(self.id, self.title)
class Continent(db.Model):
__tablename__ = 'continent'
id = db.Column(db.Integer, primary_key=True)
alpha2 = db.Column(db.String(2), index=True, unique=True)
title = db.Column(db.String(140))
def __repr__(self):
return '<Continent #{}: {}>'.format(self.id, self.title)
Many thanks,
UPDATE 1: The upgrade method of the last two migrations
As #Miguel asked in a comment, here there are the upgrade methods of the last two migrations:
Feed: Continents
def upgrade():
csv_path = app.config['BASEDIR'].child('migrations', 'csv', 'en')
csv_file = csv_path.child('continents.csv')
with open(csv_file) as file_handler:
csv = list(reader(file_handler))
csv.pop(0)
data = [{'alpha2': c[0].lower(), 'title': c[1]} for c in csv]
op.bulk_insert(Continent.__table__, data)
Feed: Countries (which depends on the table fed on the last migration)
def upgrade():
# load countries iso3166.csv and build a dictionary
csv_path = app.config['BASEDIR'].child('migrations', 'csv', 'en')
csv_file = csv_path.child('iso3166.csv')
countries = dict()
with open(csv_file) as file_handler:
csv = list(reader(file_handler))
for c in csv:
countries[c[0]] = c[1]
# load countries-continents from country_continent.csv
csv_file = csv_path.child('country_continent.csv')
with open(csv_file) as file_handler:
csv = list(reader(file_handler))
country_continent = [{'country': c[0], 'continent': c[1]} for c in csv]
# loop
data = list()
for item in country_continent:
# get continent id
continent_guess = item['continent'].lower()
continent = Continent.query.filter_by(alpha2=continent_guess).first()
# include country
if continent is not None:
country_name = countries.get(item['country'], False)
if country_name:
data.append({'alpha2': item['country'].lower(),
'title': country_name,
'continent_id': continent.id})
The CSV I'm using are basically following this patterns:
continents.csv
...
AS, "Asia"
EU, "Europe"
NA, "North America"
...
iso3166.csv
...
CL,"Chile"
CM,"Cameroon"
CN,"China"
...
_country_continent.csv_
...
US,NA
UY,SA
UZ,AS
...
So Feed: Continents feeds the continent table, and Feed: Countries feeds the country table. But it has to query the continents table in order to make the proper link between the country and the continent.
UPDATE 2: Some one from Reddit already offered an explanation and a workaround
I asked the same question on Reddit, and themathemagician said:
I've run into this before, and the issue is that the migrations don't
execute individually, but instead alembic batches all of them (or all
of them that need to be run) and then executes the SQL. This means
that by the time the last migration is trying to run, the tables don't
actually exist yet so you can't actually make queries. Doing
from alembic import op
def upgrade():
#migration stuff
op.execute('COMMIT')
#run queries
This isn't the most elegant solution (and that was for Postgres, the
command may be different for other dbs), but it worked for me. Also,
this isn't actually an issue with Flask-Migrate as much as an issue
with alembic, so if you want to Google for more info, search for
alembic. Flask-Migrate is just a wrapper around alembic that works
with Flask-Script easily.
As indicated by #themathemagician on reddit, Alembic by default runs all the migrations in a single transaction, so depending on the database engine and what you do in your migration scripts, some operations that depend on things added in a previous migration may fail.
I haven't tried this myself, but Alembic 0.6.5 introduced a transaction_per_migration option, which might address this. This is an option to the configure() call in env.py. If you are using the default config files as Flask-Migrate creates them, then this is where you fix this in migrations/env.py:
def run_migrations_online():
"""Run migrations in 'online' mode.
# ...
context.configure(
connection=connection,
target_metadata=target_metadata,
transaction_per_migration=True # <-- add this
)
# ...
Also note that if you plan to also run offline migrations you need to fix the configure() call in the run_migrations_offline() in the same way.
Give this a try and let me know if it addresses the problem.
I am trying to forward migrate a model with existing data. The model has a new field with constraints unique=True and null=False.
When I do
./manage.py schemamigration myapp --auto
South lets me specify a default value for the new field by asking:
Specify a one-off value to use for existing columns now
Usually I set this to None but since this field needs to be unique I was wondering if it is possible to pass South a unique value via:
>>> import uuid; uuid.uuid1().hex[0:35]
This gives me an error message
! Invalid input: invalid syntax
Any ideas if it is possible to pass South random unique default values when migrating via the commandline?
Thanks.
Unfortunately only the datetime module is available for use as a one-off value in a schemamigration.
However, you can achieve the same effect by splitting this up into three migrations:
add new field to the model without constraints (with null=True, unique=False)
use a datamigration to add the UUID to the new field
add the constraint on the new field (with null=False, unique=True)
Tutorial on data migrations: http://south.readthedocs.org/en/0.7.6/tutorial/part3.html#data-migrations
In django 1.7+ you can do the following. It first adds the field with no indexing and no unique. It then assigns the unique values (I based them on the name and used slugify method which you need to create) and finally alters the field again to add index and unique attributes.
from django.db import migrations
import re
import django.contrib.postgres.fields
from common.utils import slugify
import django.core.validators
def set_slugs(apps, schema_editor):
categories = apps.get_model("myapp", "Category").objects.all()
for category in categories:
category.slug = slugify(category.name)
category.save()
class Migration(migrations.Migration):
dependencies = [
('myapp', '0034_auto_20150906_1936'),
]
operations = [
migrations.AddField(
model_name='category',
name='slug',
field=models.CharField(max_length=30, validators=[django.core.validators.MinLengthValidator(2), django.core.validators.RegexValidator(re.compile('^[0-9a-z-]+$'), 'Enter a valid slug.', 'invalid')], help_text='Required. 2 to 30 characters and can only contain a-z, 0-9, and the dash (-)', unique=False, db_index=False, null=True),
preserve_default=False,
),
migrations.RunPython(set_slugs),
migrations.AlterField(
model_name='category',
name='slug',
field=models.CharField(help_text='Required. 2 to 30 characters and can only contain a-z, 0-9, and the dash (-)', unique=True, max_length=30, db_index=True, validators=[django.core.validators.MinLengthValidator(2), django.core.validators.RegexValidator(re.compile('^[0-9a-z-]+$'), 'Enter a valid slug.', 'invalid')]),
),
]
Here is the Django's official how-to on migrating unique fields.
Migrations that add unique fields
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Applying a "plain" migration that adds a unique non-nullable field to a table
with existing rows will raise an error because the value used to populate
existing rows is generated only once, thus breaking the unique constraint.
Therefore, the following steps should be taken. In this example, we'll add a
non-nullable :class:`~django.db.models.UUIDField` with a default value. Modify
the respective field according to your needs.
* Add the field on your model with ``default=...`` and ``unique=True``
arguments. In the example, we use ``uuid.uuid4`` for the default.
* Run the :djadmin:`makemigrations` command.
* Edit the created migration file.
The generated migration class should look similar to this::
class Migration(migrations.Migration):
dependencies = [
('myapp', '0003_auto_20150129_1705'),
]
operations = [
migrations.AddField(
model_name='mymodel',
name='uuid',
field=models.UUIDField(max_length=32, unique=True, default=uuid.uuid4),
),
]
You will need to make three changes:
* Add a second :class:`~django.db.migrations.operations.AddField` operation
copied from the generated one and change it to
:class:`~django.db.migrations.operations.AlterField`.
* On the first operation (``AddField``), change ``unique=True`` to
``null=True`` -- this will create the intermediary null field.
* Between the two operations, add a
:class:`~django.db.migrations.operations.RunPython` or
:class:`~django.db.migrations.operations.RunSQL` operation to generate a
unique value (UUID in the example) for each existing row.
The resulting migration should look similar to this::
# -*- coding: utf-8 -*-
from __future__ import unicode_literals
from django.db import migrations, models
import uuid
def gen_uuid(apps, schema_editor):
MyModel = apps.get_model('myapp', 'MyModel')
for row in MyModel.objects.all():
row.uuid = uuid.uuid4()
row.save()
class Migration(migrations.Migration):
dependencies = [
('myapp', '0003_auto_20150129_1705'),
]
operations = [
migrations.AddField(
model_name='mymodel',
name='uuid',
field=models.UUIDField(default=uuid.uuid4, null=True),
),
# omit reverse_code=... if you don't want the migration to be reversible.
migrations.RunPython(gen_uuid, reverse_code=migrations.RunPython.noop),
migrations.AlterField(
model_name='mymodel',
name='uuid',
field=models.UUIDField(default=uuid.uuid4, unique=True),
),
]
* Now you can apply the migration as usual with the :djadmin:`migrate` command.
Note there is a race condition if you allow objects to be created while this
migration is running. Objects created after the ``AddField`` and before
``RunPython`` will have their original ``uuid``’s overwritten.
You can manually edit your migration file:
I needed to add random character to some field so I have imported random and randint
import random
import string
and changed the value of default to
default=random.choice(string.lowercase)
It worked.
There is way to do unique value for each row with South.
Define slug in models.py as:
class Foo(models.Model):
slug = models.SlugField(unique=True, default='')
....
Create new migration
run python manage.py schemamigration --auto foo
Open new migration file, and edit it:
# Change add_column to this:
db.add_column(u'account_funnel', 'slug',
self.gf('django.db.models.foo.Foo')(default='',
unique=False,
max_length=50),
keep_default=False)
# right above this add such python code:
foos = orm['foo.Foo'].objects.all()
for foo in foos:
foo.slug = slugify(funnel.name)
foo.save()
# Modify slug as unique field
db.create_unique(u'foo_foo', ['slug'])
ps mark this migration as no_dry_run = True
pss do not forget to import slugify function from django.template.defaultfilters import slugify