I have very specific problem with export Firebird database to another system. Full export takes very long time. My plan is a create copy of database and start with export to new system. Users can continue to write and change data to an uncopied database.
My problem is how to solve missing or updated data in copied database. Is possible to compare those databases and get third database only with new or updated data? After compare missing or updated data, import them to new system. Is there any tool, or command to achieve this?
Related
I migrated a database that was manipulated via SQL to use Django's migration module: https://docs.djangoproject.com/en/3.0/topics/migrations/.
Initially I thought of using migrations only for changes in the model such as changes in columns or new tables, performing deletes, inserts and updates through SQL, as they are constant and writing a migration for each case would be a little impractical.
Could using the migration module without using data migrations cause me some kind of problem or inconsistency in the future?
You can think about data migration if you made a change that required also a manual fix on the data.
Maybe you decided to normalize something in the database, e.g. to split a name column to the first name and the last name. If you have only one instance of the application with one database and you are the only developer then you will also not write a data migration, but if you want to change it on a live production site with 24 x 7 hours traffic or you cooperate with other developers then you probably prepare a data migration for their databases or you will thoroughly test the migration on a copy of live data that the update will work on the production site correctly without issues and with minimal shutdown. If you don't write a data migration and you had no problem immediately then it is OK and will be not worse then a possible ill-conceived data migration.
I have lots of models in my project like Advertisements, UserDetails etc. Currently I have to delete the entire database every year so as to not create any conflicts between this year data and previous year data.
I want to implement a feature that can allow me to switch between different years. What can be the best way to implement this?
I think you could switch schemas in PostgreSQL. It's not completely straightforward. There are several ways to do that you can look into. The way I did it was to use a default search path for the Django database user account (e.g. user2018, user2019, etc) that only included the schema I wanted to use. I can't check the exact settings right now because my office network is down. You can also do it in settings.py or in each individual model using db_table according to what I've read, although both those solutions seem more convoluted that using the search path.
You would have to shutdown, change the database username in settings.py (or change the search path in PostgreSQL, change the schema over to a new one, and then run migrate to create the tables again. If you have reference data in any of the tables then schema-to-schema copies are easy to do.
Try searching for change django database schema postgresql to see what options there are for specifying the schema.
I am recreating a web app in Django that was running in a server but it was terminated, fortunately, I did a backup of all the code. My problem comes with the database because but I do not know how to transfer all the data from the old db.sqlite3 Django database web app into the new one.
I found a similar question as mine Django: transfer data from one database to another but the user wanted to transfer data from specific columns because their models.pyfrom the old and new databases were slightly different. In my case, my models.py from the old and new databases are the same.
Alternatives
I am using the DB Browser for SQLite to explore the content of the old database and I could add manually each row into the Django administration but this will take me too much time.
I could copy the old db.sqlite and replace it in the new web app because the models.py file remains the same but this solution is not appropriate IMO, this solution is rude and I think it goes against the good practices of Software.
How should I proceed for transferring data from the old database to the new one?
This seems like a one time copy of one db to another. I don't see how this goes against good software practice unless you have to be copying this db frequently. I've done it before when migrating servers and it doesn't cause any issues assuming the two instances of the application are the same build.
I was able to do some minor tricks in order to solve my problem because there is not a straightforward functionality that allows you to transfer all your data from two sqlite databases in Django. Here is the trick I did:
Download the sqlite db browser to explore and export the contents of your old database in a .csv file. Open you database with sqlite db browser and hit on the tables option and you will see all your tables, then do a right click on any of those and hit the export as a csv file option to generate the csv file (name_of_your_csv_file.csv). The other alternative is to use the sqlite3.exe to open your database in cmd or powershell and then doing the export with:
.tables #this lets you explore your available tables
headers on
mode csv
output name_of_your_csv_file.csv
2.There are two choices up to this point: You can either insert all the records at once to your new database or you can drop your existing tables from the new database and then recreate them and import the .csv file. I went for the drop option because there were more than 100 records to migrate.
# sqlite3
# check the structure of your table so you can re-create it
.schema <table_name>
#the result might be something like CREATE TABLE IF NOT EXISTS "web_app_navigator_table" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "ticket" varchar(120) NOT NULL);
#drop the table
drop table web_app_navigator_table
#re-create the table
create table web_app_navigator_table(id integer not null primary key autoincrement, ticket varchar(120) not null);
#do the import of the csv file
.import C:/Users/a/PycharmProjects/apps/navigator/name_of_your_csv_file.csv table_name_goes_here
You might see an error such as csv:1: INSERT failed datatype mismatch but this indicates that the first row of your csv file was not inserted because it contains the headers of the exported data from your old database.
What options do I have to migrate data to hosted with Heroku Postgres Database?
I have Django app and my client is migrating is moving from his existing software will most likely produce data in excel format. I will figure out the data model and data conversion I just need to know what tools to use to do the actual update of the DB.
Your Question is probably too broad for this site. But briefly, for Postgres in general…
INSERT
The usual way to put data into a SQL database is the INSERT command.
COPY FROM
To add data in bulk rather than one record at a time, call COPY FROM. You specify a file to be imported.
I have two Django sites: one for development and one for production. Every once in a while, the data from the development database needs to be transferred to the production database or the other way around. I use postgresql.
This works fine: I empty the tables from the database I want to copy to, I generate sql from the applicable tables, and insert the data in the emptied tables. So far, so good.
But when I enter data into the database via the admin interface, Django raises IntegrityErrors, because appname_modelname_pkey already exists.
I think this is because the admin interface wants to add data with id 1, but that's already an imported record. Django isn't aware that id '1' is already taken.
How do I fix this problem? I want Django to increment the id (like SQL auto_increment would do), no matter what data is already stored.
Any help is appreciated!
If you're using postgres on both sides, then the sequence associated with the primary keys are going to be different.
For example, suppose you're moving production data to development. Also, suppose the sequence value in production is 20 and the sequence in development is 10. Then the first new item you add in development will have an id of 11. That id (probably) already exists in the production data, so you get an integrity error.
When you restore tables from a dump, you can reset the sequences by dropping and recreating the existing tables before you restore.
(Or, you can probably use the ALTER SEQUENCE command to sync up the sequences. However, I'm not familiar enough with postgres to say whether that's the right way to do it.)