I migrated data from SQL Server database to Aurora Postgres, using AWS DMS. Everything works and data is migrated correctly, but then validation fails. There are two types of validation errors:
GUIDS in the source database are all uppercase and in the target: lowercase.
{'record_id': 'DA7D98E2-06EA-4C3E-A148-3215E1C23384'}
{'record_id': 'da7d98e2-06ea-4c3e-a148-3215e1c23384'}
For some reason, validation fails between timestamp(4) column in Postgres and datetime2(4) column of SQLServer. It seems like the time in Postgres has two extra 0's at the end, but when selecting data from the table normally, the data is exactly the same.
{'created_datetime_utc': '2018-08-24 19:58:28.4900'}
{'created_datetime_utc': '2018-08-24 19:58:28.490000'}
Any ideas how to fix this? I tried to create transformation rules for columns, but they do not work.
Thank you.
Thanks to this article https://www.sentiatechblog.com/aws-database-migration-service-dms-tips-and-tricks, these new mapping rules fixed all validation issues. These rules cannot be added using AWS Console, only in the script.
Related
I am using DMS to move data (5.200.000 records) between an RDS SQL Server Express database and an RDS MySQL database. The number of records transferred is perfect : every record count per table matches exactly.
However when running the data validation step with the AWS tool, I get a lot of mismatched records that I believe is an issue in the DMS validation process at the driver level (charset configuration probably or character encoding).
Example:
RECORD_DIFF [{'Title': 'سلام ژوند څنګه دی (1) 😀🥳'}, {'Title': 'سلام ژوند څنګه دی (1) ??'},]
If I do the select query directly in the MySQL database with MySQL Workbench I get exactly the same emojis as I the DMS validation tool gets from the SQL Server.
Besides this I am also getting a lot of RECORD_DIFF [{'Content': '<VALUE_PRINT_NOT_SUPPORTED>'}, {'Content': '<VALUE_PRINT_NOT_SUPPORTED>'},] which I am not sure what they mean querying both database I see the values perfectly for those rows.
Should I consider using another validation tool or there is a known way to address the problems above?
Any help will be greatly appreciated.
It's as simple as that. Athena used to load databases and tables that I crawled using Glue. The data is present in S3 and Athena used to work before. But all of a sudden the loading icon goes round and round but it doesn't load the list of databases and tables.
I'm in the right region. It works when I send the queries through Python/SageMaker i.e. I use awswrangler and the data output from that is fine. But it's not possible to query within Athena itself even though I used to do it before.
Totally stumped on what the problem could be as I have no clues.
This has been solved. I am not sure what the fix was (This has been an issue since atleast 3 months and I have tried solving it before with similar methods).
But I did two things before it 'fixed itself':
Tried changing the Athena output location through the workgroup settings.
Tried changing the same(I'm not sure if both settings point to the same property) through the settings icon on the right top of the page.
And suddenly the list of databases and tables shows up in the Query Editor page.
We have noticed that if a table is empty in SQL Server, the empty table does not come via DMS. Only after inserting a record it starts to show up.
Just checking, is there a way to get the schema only from DMS?
Thanks
You can use Schema conversion tool for moving DB objects and Schema. Its a free tool by AWS and can be installed on On-Prem server or on EC2. It gives a good report before you can actually migrate the DB schema and other DB objects. It shows how many Tables, SP's Funcs etc can be directly migrated and shows possible solutions too.
I am trying to write a migration which grants readonly permissions to each schema in my multi-tenant postgres DB.
The migrations run once per schema, so what I would like to do would be capture the name of the schema for which it is running, and then use that schema_name in my SQL statement to grant permissions for that schema.
In django, I can create a migration operation called 'RunPython', and from within that python code I can determine for which schema the migrations are currently running (schema_editor.connection.connection_name).
What I want to do is pass that information to the next migration operation, namely "RunSQL", so that the SQL I run can be:
"GRANT SELECT ON ALL TABLES IN SCHEMA {schema_name_from_python_code} TO readaccess;"
If anyone can shed any light on this issue it would be greatly appreciated. Cheers!
I was able to figure this out by getting rid of the migrations.runSQL. I just have migrations.RunPython. From within that python forward_func I am able to access the DB and write sql there (with the necessary string interpolation)
:)
So I backed up my geodjango postgis database using pg_dump before performing some calculations which I've managed to mess up. I've created a new database using
createdb -T template0 -O proj proj_backup
psql proj_backup < proj_backup.pg
This seemed to work fine (though there were a few errors during the import), and connecting to the database using psql all my tables are there and appear to have the correct numbers of rows etc.
However, changing my settings.py to connect to my newly imported backup db (proj_backup in my example), gives me the following errors:
DatabaseError: relation "appname_model" does not exist
Any ideas? I'm guessing I did the dump wrong, or that I haven't maintained the ForeignKeys somehow. Thanks very much.
Update
So I figured out my first mistake: I had two similarly named backup databases and connected to the wrong one. Connecting to the correct one seems to have fixed everything. However, it's still quite strange that it didn't recognize the tables in the other backup database, which definitely did exist. Running syncdb on the incorrect database ends up duplicating those tables (if I remember correctly, there were duplicate table names when I listed them all from within psql). Unfortunately, the way I discovered my mistake was by dropping the bad table to recreate it, and so in order to reproduce this error I'll probably have to use time machine. Still very strange, I'll give that a shot when I can get physical access to my work machine.
So is your appname_model table there or is it a view? Was it in public or another named schema?
If the table is there, then chances are you have it in a schema that is not in your database search path. Check the search_path of your old database. It might have included something other than the default, or your default search schema is set in postgresql.conf and is non-standard.