aws data migration service not picking any source table - amazon-web-services

I created aws dms (data migration service) task but task is not picking any source schema table.
Both source and target end points are successful. I first tried Sybase to Oracle migration then to trouble-shoot Oracle to Oracle migration task.
In both cases it didn't pick any table.
I have tried by giving specific table name and also % in but it is not picking any table in both cases.
Both source and target end points use admin account and also source table has select to public grant.
Kindly suggest. I have attached jsons and images.
Thanks in advance.
Attached below is json for table mapping and complete task:
{
"rules": [
{
"rule-type": "selection",
"rule-id": "1",
"rule-name": "1",
"object-locator": {
"schema-name": "SRC_ORCL",
"table-name": "%"
},
"rule-action": "include"
}
]
}
Task Json
{
"TargetMetadata": {
"TargetSchema": "dbadmin",
"SupportLobs": true,
"FullLobMode": false,
"LobChunkSize": 0,
"LimitedSizeLobMode": true,
"LobMaxSize": 32,
"InlineLobMaxSize": 0,
"LoadMaxFileSize": 0,
"ParallelLoadThreads": 0,
"ParallelLoadBufferSize": 0,
"BatchApplyEnabled": false,
"TaskRecoveryTableEnabled": false,
"ParallelLoadQueuesPerThread": 0,
"ParallelApplyThreads": 0,
"ParallelApplyBufferSize": 0,
"ParallelApplyQueuesPerThread": 0
},
"FullLoadSettings": {
"TargetTablePrepMode": "DROP_AND_CREATE",
"CreatePkAfterFullLoad": false,
"StopTaskCachedChangesApplied": false,
"StopTaskCachedChangesNotApplied": false,
"MaxFullLoadSubTasks": 8,
"TransactionConsistencyTimeout": 600,
"CommitRate": 10000
},
"Logging": {
"EnableLogging": false,
"LogComponents": [
{
"Id": "TRANSFORMATION",
"Severity": "LOGGER_SEVERITY_DEFAULT"
},
{
"Id": "SOURCE_UNLOAD",
"Severity": "LOGGER_SEVERITY_DEFAULT"
},
{
"Id": "IO",
"Severity": "LOGGER_SEVERITY_DEFAULT"
},
{
"Id": "TARGET_LOAD",
"Severity": "LOGGER_SEVERITY_DEFAULT"
},
{
"Id": "PERFORMANCE",
"Severity": "LOGGER_SEVERITY_DEFAULT"
},
{
"Id": "SOURCE_CAPTURE",
"Severity": "LOGGER_SEVERITY_DEFAULT"
},
{
"Id": "SORTER",
"Severity": "LOGGER_SEVERITY_DEFAULT"
},
{
"Id": "REST_SERVER",
"Severity": "LOGGER_SEVERITY_DEFAULT"
},
{
"Id": "VALIDATOR_EXT",
"Severity": "LOGGER_SEVERITY_DEFAULT"
},
{
"Id": "TARGET_APPLY",
"Severity": "LOGGER_SEVERITY_DEFAULT"
},
{
"Id": "TASK_MANAGER",
"Severity": "LOGGER_SEVERITY_DEFAULT"
},
{
"Id": "TABLES_MANAGER",
"Severity": "LOGGER_SEVERITY_DEFAULT"
},
{
"Id": "METADATA_MANAGER",
"Severity": "LOGGER_SEVERITY_DEFAULT"
},
{
"Id": "FILE_FACTORY",
"Severity": "LOGGER_SEVERITY_DEFAULT"
},
{
"Id": "COMMON",
"Severity": "LOGGER_SEVERITY_DEFAULT"
},
{
"Id": "ADDONS",
"Severity": "LOGGER_SEVERITY_DEFAULT"
},
{
"Id": "DATA_STRUCTURE",
"Severity": "LOGGER_SEVERITY_DEFAULT"
},
{
"Id": "COMMUNICATION",
"Severity": "LOGGER_SEVERITY_DEFAULT"
},
{
"Id": "FILE_TRANSFER",
"Severity": "LOGGER_SEVERITY_DEFAULT"
}
],
"CloudWatchLogGroup": null,
"CloudWatchLogStream": null
},
"ControlTablesSettings": {
"historyTimeslotInMinutes": 5,
"ControlSchema": "TGT_ORCL",
"HistoryTimeslotInMinutes": 5,
"HistoryTableEnabled": true,
"SuspendedTablesTableEnabled": true,
"StatusTableEnabled": true
},
"StreamBufferSettings": {
"StreamBufferCount": 3,
"StreamBufferSizeInMB": 8,
"CtrlStreamBufferSizeInMB": 5
},
"ChangeProcessingDdlHandlingPolicy": {
"HandleSourceTableDropped": true,
"HandleSourceTableTruncated": true,
"HandleSourceTableAltered": true
},
"ErrorBehavior": {
"DataErrorPolicy": "LOG_ERROR",
"DataTruncationErrorPolicy": "LOG_ERROR",
"DataErrorEscalationPolicy": "SUSPEND_TABLE",
"DataErrorEscalationCount": 0,
"TableErrorPolicy": "SUSPEND_TABLE",
"TableErrorEscalationPolicy": "STOP_TASK",
"TableErrorEscalationCount": 0,
"RecoverableErrorCount": -1,
"RecoverableErrorInterval": 5,
"RecoverableErrorThrottling": true,
"RecoverableErrorThrottlingMax": 1800,
"ApplyErrorDeletePolicy": "IGNORE_RECORD",
"ApplyErrorInsertPolicy": "LOG_ERROR",
"ApplyErrorUpdatePolicy": "LOG_ERROR",
"ApplyErrorEscalationPolicy": "LOG_ERROR",
"ApplyErrorEscalationCount": 0,
"ApplyErrorFailOnTruncationDdl": false,
"FullLoadIgnoreConflicts": true,
"FailOnTransactionConsistencyBreached": false,
"FailOnNoTablesCaptured": false
},
"ChangeProcessingTuning": {
"BatchApplyPreserveTransaction": true,
"BatchApplyTimeoutMin": 1,
"BatchApplyTimeoutMax": 30,
"BatchApplyMemoryLimit": 500,
"BatchSplitSize": 0,
"MinTransactionSize": 1000,
"CommitTimeout": 1,
"MemoryLimitTotal": 1024,
"MemoryKeepTime": 60,
"StatementCacheSize": 50
},
"PostProcessingRules": null,
"CharacterSetSettings": null,
"LoopbackPreventionSettings": null,
"BeforeImageSettings": null
}

It worked, After I made below changes
i) Sybase source end point to use database as pubs2.
ii) In task specified % in both Schema and table name.

Related

AWS DMS Task - Reading from source is paused. Total storage used by swap files exceeded the limit

I am using AWS DMS to run a migration task full load + cdc. I am migrating a like for like rds mysql database to another rds mysql database.
Its been at 97% for a while now and i can see in the cloudwatch logs a few times the following message.
2023-02-01T19:52:57 [SORTER ]I: Reading from source is paused. Total storage used by swap files exceeded the limit 1048576000 bytes (sorter_transaction.c:110)
This suggests to me either the source, target or replication instance are using 1gb of swap storage. However when i check cloudwatch that does not seem to be the case.
Whats happening here
replica instance
target instance
source instance
task settings
{
"Logging": {
"EnableLogging": true,
"EnableLogContext": false,
"LogComponents": [
{
"Severity": "LOGGER_SEVERITY_DEFAULT",
"Id": "TRANSFORMATION"
},
{
"Severity": "LOGGER_SEVERITY_DEFAULT",
"Id": "SOURCE_UNLOAD"
},
{
"Severity": "LOGGER_SEVERITY_DEFAULT",
"Id": "IO"
},
{
"Severity": "LOGGER_SEVERITY_DEFAULT",
"Id": "TARGET_LOAD"
},
{
"Severity": "LOGGER_SEVERITY_DEFAULT",
"Id": "PERFORMANCE"
},
{
"Severity": "LOGGER_SEVERITY_DEFAULT",
"Id": "SOURCE_CAPTURE"
},
{
"Severity": "LOGGER_SEVERITY_DEFAULT",
"Id": "SORTER"
},
{
"Severity": "LOGGER_SEVERITY_DEFAULT",
"Id": "REST_SERVER"
},
{
"Severity": "LOGGER_SEVERITY_DEFAULT",
"Id": "VALIDATOR_EXT"
},
{
"Severity": "LOGGER_SEVERITY_DEFAULT",
"Id": "TARGET_APPLY"
},
{
"Severity": "LOGGER_SEVERITY_DEFAULT",
"Id": "TASK_MANAGER"
},
{
"Severity": "LOGGER_SEVERITY_DEFAULT",
"Id": "TABLES_MANAGER"
},
{
"Severity": "LOGGER_SEVERITY_DEFAULT",
"Id": "METADATA_MANAGER"
},
{
"Severity": "LOGGER_SEVERITY_DEFAULT",
"Id": "FILE_FACTORY"
},
{
"Severity": "LOGGER_SEVERITY_DEFAULT",
"Id": "COMMON"
},
{
"Severity": "LOGGER_SEVERITY_DEFAULT",
"Id": "ADDONS"
},
{
"Severity": "LOGGER_SEVERITY_DEFAULT",
"Id": "DATA_STRUCTURE"
},
{
"Severity": "LOGGER_SEVERITY_DEFAULT",
"Id": "COMMUNICATION"
},
{
"Severity": "LOGGER_SEVERITY_DEFAULT",
"Id": "FILE_TRANSFER"
}
],
"CloudWatchLogGroup": "dms-tasks-geeiq-prod-master-replication-instance",
"CloudWatchLogStream": "dms-task-PLBNPFYKIAWHZDEPAPTDASAD4P6GCIPMRW3ZRXA"
},
"StreamBufferSettings": {
"StreamBufferCount": 3,
"CtrlStreamBufferSizeInMB": 5,
"StreamBufferSizeInMB": 8
},
"ErrorBehavior": {
"FailOnNoTablesCaptured": true,
"ApplyErrorUpdatePolicy": "LOG_ERROR",
"FailOnTransactionConsistencyBreached": false,
"RecoverableErrorThrottlingMax": 1800,
"DataErrorEscalationPolicy": "SUSPEND_TABLE",
"ApplyErrorEscalationCount": 0,
"RecoverableErrorStopRetryAfterThrottlingMax": true,
"RecoverableErrorThrottling": true,
"ApplyErrorFailOnTruncationDdl": false,
"DataTruncationErrorPolicy": "LOG_ERROR",
"ApplyErrorInsertPolicy": "LOG_ERROR",
"EventErrorPolicy": "IGNORE",
"ApplyErrorEscalationPolicy": "LOG_ERROR",
"RecoverableErrorCount": -1,
"DataErrorEscalationCount": 0,
"TableErrorEscalationPolicy": "STOP_TASK",
"RecoverableErrorInterval": 5,
"ApplyErrorDeletePolicy": "IGNORE_RECORD",
"TableErrorEscalationCount": 0,
"FullLoadIgnoreConflicts": true,
"DataErrorPolicy": "LOG_ERROR",
"TableErrorPolicy": "SUSPEND_TABLE"
},
"TTSettings": {
"TTS3Settings": null,
"TTRecordSettings": null,
"EnableTT": false
},
"FullLoadSettings": {
"CommitRate": 10000,
"StopTaskCachedChangesApplied": false,
"StopTaskCachedChangesNotApplied": false,
"MaxFullLoadSubTasks": 8,
"TransactionConsistencyTimeout": 600,
"CreatePkAfterFullLoad": false,
"TargetTablePrepMode": "DROP_AND_CREATE"
},
"TargetMetadata": {
"ParallelApplyBufferSize": 0,
"ParallelApplyQueuesPerThread": 0,
"ParallelApplyThreads": 0,
"TargetSchema": "",
"InlineLobMaxSize": 0,
"ParallelLoadQueuesPerThread": 0,
"SupportLobs": true,
"LobChunkSize": 64,
"TaskRecoveryTableEnabled": false,
"ParallelLoadThreads": 0,
"LobMaxSize": 0,
"BatchApplyEnabled": false,
"FullLobMode": true,
"LimitedSizeLobMode": false,
"LoadMaxFileSize": 0,
"ParallelLoadBufferSize": 0
},
"BeforeImageSettings": null,
"ControlTablesSettings": {
"historyTimeslotInMinutes": 5,
"HistoryTimeslotInMinutes": 5,
"StatusTableEnabled": false,
"SuspendedTablesTableEnabled": false,
"HistoryTableEnabled": false,
"ControlSchema": "",
"FullLoadExceptionTableEnabled": false
},
"LoopbackPreventionSettings": null,
"CharacterSetSettings": null,
"FailTaskWhenCleanTaskResourceFailed": false,
"ChangeProcessingTuning": {
"StatementCacheSize": 50,
"CommitTimeout": 1,
"BatchApplyPreserveTransaction": true,
"BatchApplyTimeoutMin": 1,
"BatchSplitSize": 0,
"BatchApplyTimeoutMax": 30,
"MinTransactionSize": 1000,
"MemoryKeepTime": 60,
"BatchApplyMemoryLimit": 500,
"MemoryLimitTotal": 1024
},
"ChangeProcessingDdlHandlingPolicy": {
"HandleSourceTableDropped": true,
"HandleSourceTableTruncated": true,
"HandleSourceTableAltered": true
},
"PostProcessingRules": null
}
This error message "Reading from source is paused. Total storage used by swap files exceeded the limit ..." is related to contention on target database to apply changes, so, DMS pause source reading operation to avoid more accumulation of database and have storage full situation. Missing primary keys or indexes could case a full scan operation during UPDATE or DELETE operation causing a performance issues.
You could analyze your target apply latency, review link below to analyze your issue.
How can I troubleshoot high target latency on an AWS DMS task?
What are SWAP files and why are they consuming space on my AWS DMS instance?

AWS DMS Task doesn't apply inserts, updates, and deletes on Elasticache Redis cluster target

I have a full load and CDC DMS task configured to replicate data in Aurora MySQL cluster to Elasticache Redis cluster. For some reason I see that change come to task but are not applied to Redis cluster.
I tried to apply different task settings. Upgraded Redis engine version to 7.0.4.
Here's a current task settings:
{
"Logging": {
"EnableLogging": true,
"LogComponents": [
{
"Severity": "LOGGER_SEVERITY_DEFAULT",
"Id": "TRANSFORMATION"
},
{
"Severity": "LOGGER_SEVERITY_DEFAULT",
"Id": "SOURCE_UNLOAD"
},
{
"Severity": "LOGGER_SEVERITY_DEFAULT",
"Id": "IO"
},
{
"Severity": "LOGGER_SEVERITY_DEFAULT",
"Id": "TARGET_LOAD"
},
{
"Severity": "LOGGER_SEVERITY_DEFAULT",
"Id": "PERFORMANCE"
},
{
"Severity": "LOGGER_SEVERITY_DEFAULT",
"Id": "SOURCE_CAPTURE"
},
{
"Severity": "LOGGER_SEVERITY_DEFAULT",
"Id": "SORTER"
},
{
"Severity": "LOGGER_SEVERITY_DEFAULT",
"Id": "REST_SERVER"
},
{
"Severity": "LOGGER_SEVERITY_DEFAULT",
"Id": "VALIDATOR_EXT"
},
{
"Severity": "LOGGER_SEVERITY_DEFAULT",
"Id": "TARGET_APPLY"
},
{
"Severity": "LOGGER_SEVERITY_DEFAULT",
"Id": "TASK_MANAGER"
},
{
"Severity": "LOGGER_SEVERITY_DEFAULT",
"Id": "TABLES_MANAGER"
},
{
"Severity": "LOGGER_SEVERITY_DEFAULT",
"Id": "METADATA_MANAGER"
},
{
"Severity": "LOGGER_SEVERITY_DEFAULT",
"Id": "FILE_FACTORY"
},
{
"Severity": "LOGGER_SEVERITY_DEFAULT",
"Id": "COMMON"
},
{
"Severity": "LOGGER_SEVERITY_DEFAULT",
"Id": "ADDONS"
},
{
"Severity": "LOGGER_SEVERITY_DEFAULT",
"Id": "DATA_STRUCTURE"
},
{
"Severity": "LOGGER_SEVERITY_DEFAULT",
"Id": "COMMUNICATION"
},
{
"Severity": "LOGGER_SEVERITY_DEFAULT",
"Id": "FILE_TRANSFER"
}
],
"CloudWatchLogGroup": "dms-tasks-dsp-engine-prod-replication-instance",
"CloudWatchLogStream": "dms-task-VZ6ESTKIA3V6CAZAQRGK2Q76EAV4OASWIKR5JJY"
},
"StreamBufferSettings": {
"StreamBufferCount": 3,
"CtrlStreamBufferSizeInMB": 5,
"StreamBufferSizeInMB": 8
},
"ErrorBehavior": {
"FailOnNoTablesCaptured": true,
"ApplyErrorUpdatePolicy": "LOG_ERROR",
"FailOnTransactionConsistencyBreached": false,
"RecoverableErrorThrottlingMax": 1800,
"DataErrorEscalationPolicy": "SUSPEND_TABLE",
"ApplyErrorEscalationCount": 0,
"RecoverableErrorStopRetryAfterThrottlingMax": true,
"RecoverableErrorThrottling": true,
"ApplyErrorFailOnTruncationDdl": false,
"DataTruncationErrorPolicy": "LOG_ERROR",
"ApplyErrorInsertPolicy": "LOG_ERROR",
"EventErrorPolicy": "IGNORE",
"ApplyErrorEscalationPolicy": "LOG_ERROR",
"RecoverableErrorCount": 0,
"DataErrorEscalationCount": 50,
"TableErrorEscalationPolicy": "STOP_TASK",
"RecoverableErrorInterval": 5,
"ApplyErrorDeletePolicy": "IGNORE_RECORD",
"TableErrorEscalationCount": 50,
"FullLoadIgnoreConflicts": true,
"DataErrorPolicy": "LOG_ERROR",
"TableErrorPolicy": "SUSPEND_TABLE"
},
"TTSettings": {
"TTS3Settings": null,
"TTRecordSettings": null,
"EnableTT": false
},
"FullLoadSettings": {
"CommitRate": 10000,
"StopTaskCachedChangesApplied": false,
"StopTaskCachedChangesNotApplied": false,
"MaxFullLoadSubTasks": 8,
"TransactionConsistencyTimeout": 600,
"CreatePkAfterFullLoad": false,
"TargetTablePrepMode": "DROP_AND_CREATE"
},
"TargetMetadata": {
"ParallelApplyBufferSize": 100,
"ParallelApplyQueuesPerThread": 1,
"ParallelApplyThreads": 0,
"TargetSchema": "",
"InlineLobMaxSize": 0,
"ParallelLoadQueuesPerThread": 1,
"SupportLobs": true,
"LobChunkSize": 0,
"TaskRecoveryTableEnabled": false,
"ParallelLoadThreads": 0,
"LobMaxSize": 32,
"BatchApplyEnabled": false,
"FullLobMode": false,
"LimitedSizeLobMode": true,
"LoadMaxFileSize": 0,
"ParallelLoadBufferSize": 0
},
"BeforeImageSettings": {
"EnableBeforeImage": false,
"ColumnFilter": "pk-only",
"FieldName": ""
},
"ControlTablesSettings": {
"historyTimeslotInMinutes": 5,
"HistoryTimeslotInMinutes": 5,
"StatusTableEnabled": true,
"SuspendedTablesTableEnabled": false,
"HistoryTableEnabled": true,
"ControlSchema": "",
"FullLoadExceptionTableEnabled": false
},
"LoopbackPreventionSettings": null,
"CharacterSetSettings": null,
"FailTaskWhenCleanTaskResourceFailed": false,
"ChangeProcessingTuning": {
"StatementCacheSize": 50,
"CommitTimeout": 1,
"BatchApplyPreserveTransaction": true,
"BatchApplyTimeoutMin": 1,
"BatchSplitSize": 0,
"BatchApplyTimeoutMax": 30,
"MinTransactionSize": 1000,
"MemoryKeepTime": 60,
"BatchApplyMemoryLimit": 500,
"MemoryLimitTotal": 1024
},
"ChangeProcessingDdlHandlingPolicy": {
"HandleSourceTableDropped": true,
"HandleSourceTableTruncated": true,
"HandleSourceTableAltered": true
},
"PostProcessingRules": null
}
Redis engine version is 7.0.4
Is there any option that's actually controls whether changes should be applied?
It turned out that in my case it was because of source filters. They don't work the way I expected. Not only records are filtered in full load stage, but updates in cdc stage are filtered too. That's why there's Updates and Applied updates columns.

AWS DMS - InvalidParameterValueException) when calling the CreateReplicationTask operation: Replication Task Settings document error: Invalid json

I am trying to create a DMS replication task using the AWS DMS cli. I am trying to pass the task settings using a json file like this:
aws dms create-replication-task --replication-task-identifier dms-cli-test-replication-task-1 --source-endpoint-arn arn --target-endpoint-arn arn --replication-instance-arn arn --migration-type full-load-and-cdc --table-mappings ./table_mappings.json --replication-task-settings ./task_settings.json --region us-east-1
when I run this command, the below error is being thrown:
An error occurred (InvalidParameterValueException) when calling the CreateReplicationTask operation: Replication Task Settings document error: Invalid json
Below is the content of my task_settings.json file:
{
"TargetMetadata": {
"TargetSchema": "",
"SupportLobs": true,
"FullLobMode": true,
"LobChunkSize": 64,
"LimitedSizeLobMode": false,
"LobMaxSize": 0,
"InlineLobMaxSize": 0,
"LoadMaxFileSize": 0,
"ParallelLoadThreads": 0,
"ParallelLoadBufferSize": 0,
"BatchApplyEnabled": false,
"TaskRecoveryTableEnabled": false
},
"FullLoadSettings": {
"TargetTablePrepMode": "DO_NOTHING",
"CreatePkAfterFullLoad": false,
"StopTaskCachedChangesApplied": false,
"StopTaskCachedChangesNotApplied": false,
"MaxFullLoadSubTasks": 8,
"TransactionConsistencyTimeout": 600,
"CommitRate": 10000
},
"Logging": {
"EnableLogging": true,
"LogComponents": [
{
"Id": "SOURCE_UNLOAD",
"Severity": "LOGGER_SEVERITY_DEFAULT"
},
{
"Id": "SOURCE_CAPTURE",
"Severity": "LOGGER_SEVERITY_DEFAULT"
},
{
"Id": "TARGET_LOAD",
"Severity": "LOGGER_SEVERITY_DEFAULT"
},
{
"Id": "TARGET_APPLY",
"Severity": "LOGGER_SEVERITY_DEFAULT"
},
{
"Id": "TASK_MANAGER",
"Severity": "LOGGER_SEVERITY_DEFAULT"
}
]
},
"ControlTablesSettings": {
"historyTimeslotInMinutes": 5,
"ControlSchema": "",
"HistoryTimeslotInMinutes": 5,
"HistoryTableEnabled": true,
"SuspendedTablesTableEnabled": true,
"StatusTableEnabled": true
},
"StreamBufferSettings": {
"StreamBufferCount": 3,
"StreamBufferSizeInMB": 8,
"CtrlStreamBufferSizeInMB": 5
},
"ChangeProcessingDdlHandlingPolicy": {
"HandleSourceTableDropped": true,
"HandleSourceTableTruncated": true,
"HandleSourceTableAltered": true
},
"ErrorBehavior": {
"DataErrorPolicy": "LOG_ERROR",
"DataTruncationErrorPolicy": "LOG_ERROR",
"DataErrorEscalationPolicy": "SUSPEND_TABLE",
"DataErrorEscalationCount": 0,
"TableErrorPolicy": "SUSPEND_TABLE",
"TableErrorEscalationPolicy": "STOP_TASK",
"TableErrorEscalationCount": 0,
"RecoverableErrorCount": -1,
"RecoverableErrorInterval": 5,
"RecoverableErrorThrottling": true,
"RecoverableErrorThrottlingMax": 1800,
"ApplyErrorDeletePolicy": "IGNORE_RECORD",
"ApplyErrorInsertPolicy": "LOG_ERROR",
"ApplyErrorUpdatePolicy": "LOG_ERROR",
"ApplyErrorEscalationPolicy": "LOG_ERROR",
"ApplyErrorEscalationCount": 0,
"ApplyErrorFailOnTruncationDdl": false,
"FullLoadIgnoreConflicts": true,
"FailOnTransactionConsistencyBreached": false,
"FailOnNoTablesCaptured": false
},
"ChangeProcessingTuning": {
"BatchApplyPreserveTransaction": true,
"BatchApplyTimeoutMin": 1,
"BatchApplyTimeoutMax": 30,
"BatchApplyMemoryLimit": 500,
"BatchSplitSize": 0,
"MinTransactionSize": 1000,
"CommitTimeout": 1,
"MemoryLimitTotal": 1024,
"MemoryKeepTime": 60,
"StatementCacheSize": 50
},
"ValidationSettings": {
"EnableValidation": true,
"ValidationMode": "ROW_LEVEL",
"ThreadCount": 5,
"PartitionSize": 10000,
"FailureMaxCount": 10000,
"RecordFailureDelayInMinutes": 5,
"RecordSuspendDelayInMinutes": 30,
"MaxKeyColumnSize": 8096,
"TableFailureMaxCount": 1000,
"ValidationOnly": false,
"HandleCollationDiff": false,
"RecordFailureDelayLimitInMinutes": 0
},
"PostProcessingRules": null,
"CharacterSetSettings": null
}
I don't see any issues with the formatting of my json. I don't understand why it says invalid json. any advice is appreciated. Thank you.
This is not actually you pass setting document or any document to aws-cli. It usually precedes the path with "file://" while passing JSON file to any parameter of cli.
Try using below command:
aws dms create-replication-task --replication-task-identifier dms-cli-test-replication-task-1 --source-endpoint-arn arn --target-endpoint-arn arn --replication-instance-arn arn --migration-type full-load-and-cdc --table-mappings file://table_mappings.json --replication-task-settings file://task_settings.json --region us-east-1
However, I can't see this particular parameter takes json document as a setting file. But I think you should try. Check below snippet from the doc.
--replication-task-identifier (string)
The replication task identifier.
Constraints:
Must contain from 1 to 255 alphanumeric characters or hyphens.
First character must be a letter.
Cannot end with a hyphen or contain two consecutive hyphens.

AWS DMS not giving 100% migaration

HI all we are migrating out database from on premises to Amazon aurora.our database size is around 136GB moreover few tables have over millions of records each. Howover after full load complete out of millions rows approx 200,000 to 300,000 rows gets migrated.WE dont know where we are falling since we are new to DMS.Can anyone know how can we migrate exact count of rows.
migration type :full load
Here are our AWS DMS task settings
{
"TargetMetadata": {
"TargetSchema": "",
"SupportLobs": true,
"FullLobMode": true,
"LobChunkSize": 64,
"LimitedSizeLobMode": false,
"LobMaxSize": 0,
"LoadMaxFileSize": 0,
"ParallelLoadThreads": 0,
"BatchApplyEnabled": false
},
"FullLoadSettings": {
"FullLoadEnabled": true,
"ApplyChangesEnabled": false,
"TargetTablePrepMode": "TRUNCATE_BEFORE_LOAD",
"CreatePkAfterFullLoad": false,
"StopTaskCachedChangesApplied": false,
"StopTaskCachedChangesNotApplied": false,
"ResumeEnabled": false,
"ResumeMinTableSize": 100000,
"ResumeOnlyClusteredPKTables": true,
"MaxFullLoadSubTasks": 15,
"TransactionConsistencyTimeout": 600,
"CommitRate": 10000
},
"Logging": {
"EnableLogging": true,
"LogComponents": [
{
"Id": "SOURCE_UNLOAD",
"Severity": "LOGGER_SEVERITY_DEFAULT"
},
{
"Id": "SOURCE_CAPTURE",
"Severity": "LOGGER_SEVERITY_DEFAULT"
},
{
"Id": "TARGET_LOAD",
"Severity": "LOGGER_SEVERITY_DEFAULT"
},
{
"Id": "TARGET_APPLY",
"Severity": "LOGGER_SEVERITY_DEFAULT"
},
{
"Id": "TASK_MANAGER",
"Severity": "LOGGER_SEVERITY_DEFAULT"
}
],
"CloudWatchLogGroup": "dms-tasks-krishna-smartdata",
"CloudWatchLogStream": "dms-task-UERQWLR6AYHYIEKMR3HN2VL7T4"
},
"ControlTablesSettings": {
"historyTimeslotInMinutes": 5,
"ControlSchema": "",
"HistoryTimeslotInMinutes": 5,
"HistoryTableEnabled": true,
"SuspendedTablesTableEnabled": true,
"StatusTableEnabled": true
},
"StreamBufferSettings": {
"StreamBufferCount": 3,
"StreamBufferSizeInMB": 8,
"CtrlStreamBufferSizeInMB": 5
},
"ChangeProcessingDdlHandlingPolicy": {
"HandleSourceTableDropped": true,
"HandleSourceTableTruncated": true,
"HandleSourceTableAltered": true
},
"ErrorBehavior": {
"DataErrorPolicy": "LOG_ERROR",
"DataTruncationErrorPolicy": "LOG_ERROR",
"DataErrorEscalationPolicy": "SUSPEND_TABLE",
"DataErrorEscalationCount": 0,
"TableErrorPolicy": "SUSPEND_TABLE",
"TableErrorEscalationPolicy": "STOP_TASK",
"TableErrorEscalationCount": 0,
"RecoverableErrorCount": -1,
"RecoverableErrorInterval": 5,
"RecoverableErrorThrottling": true,
"RecoverableErrorThrottlingMax": 1800,
"ApplyErrorDeletePolicy": "IGNORE_RECORD",
"ApplyErrorInsertPolicy": "LOG_ERROR",
"ApplyErrorUpdatePolicy": "LOG_ERROR",
"ApplyErrorEscalationPolicy": "LOG_ERROR",
"ApplyErrorEscalationCount": 0,
"FullLoadIgnoreConflicts": true
},
"ChangeProcessingTuning": {
"BatchApplyPreserveTransaction": true,
"BatchApplyTimeoutMin": 1,
"BatchApplyTimeoutMax": 30,
"BatchApplyMemoryLimit": 500,
"BatchSplitSize": 0,
"MinTransactionSize": 1000,
"CommitTimeout": 1,
"MemoryLimitTotal": 1024,
"MemoryKeepTime": 60,
"StatementCacheSize": 50
}
}
Mapping Method:
{
"rules": [
{
"rule-type": "selection",
"rule-id": "1",
"rule-name": "1",
"object-locator": {
"schema-name": "dbo",
"table-name": "%"
},
"rule-action": "include"
},
{
"rule-type": "transformation",
"rule-id": "2",
"rule-name": "2",
"rule-target": "schema",
"object-locator": {
"schema-name": "dbo"
},
"rule-action": "rename",
"value": "smartdata_int"
}
]
}
You should have the option of setting up CloudWatch logs for each DMS task. Have you inspected the logs for this task? Do you have varchar/text columns > 32KB? These will be truncated when migrating data into a target like redshift, so be aware that this will count towards your error count.
First thing to do is to increase log level :
"Logging": {
"EnableLogging": true,
"LogComponents": [{
"Id": "SOURCE_UNLOAD",
"Severity": "LOGGER_SEVERITY_DETAILED_DEBUG"
},{
"Id": "SOURCE_CAPTURE",
"Severity": "LOGGER_SEVERITY_DETAILED_DEBUG"
},{
"Id": "TARGET_LOAD",
"Severity": "LOGGER_SEVERITY_DETAILED_DEBUG"
},{
"Id": "TARGET_APPLY",
"Severity": "LOGGER_SEVERITY_DETAILED_DEBUG"
},{
"Id": "TASK_MANAGER",
"Severity": "LOGGER_SEVERITY_DETAILED_DEBUG"
}]
},
Then you will be able to get details about errors occuring.
Turn on validation:
https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Validating.html
This will slow the migration down so you cloud also look at splitting this out into multiple tasks and running them on multiple replication instances, expand rule 1 out into multiple rules, rather than '%' add a condition that meets a subset of the tables.
You might also try a different replication engine, 3.1.1 has just been released, at the time of writing there are no release notes for 3.1.1.
https://docs.aws.amazon.com/dms/latest/userguide/CHAP_ReleaseNotes.html

how to apply conditions on JSON response in Jmeter?

Response is as follows,
I have to extract all 'deviceResponseStatus.id' where status="PENDING"
how to implement it using JSON Path or REGEX in Jmeter?
{
"apiResponseStatus": "SUCCESS",
"deviceResponseList": [
{
"apiResponseStatus": "SUCCESS",
"id": 23,
"ownership": "CORPORATE",
"deviceName": "Demimbu",
"deviceType": "MOBILE",
"osType": "ANDROID",
"inTotalStorage": 0,
"inAvailableStorage": 0,
"exTotalStorage": 0,
"exAvailableStorage": 0,
"osVersion": "4.2",
"buildVersion": "1.01.08",
"status": "PENDING",
"isDeleted": false,
"policyIdNames": {
"id": 3,
"name": "Test Policy 3"
},
"deviceGroupIdNames": {
"id": 1,
"name": "Default Group"
},
"userIdName": {
"id": 1,
"name": "Randy Lewis"
},
"lastConnected": 1423810405000,
"createdByIdName": {
"id": 1,
"name": "Randy Lewis"
},
"locked": false,
"traceOn": false,
"userEmail": "mdm.user1#gmail.com",
"enrollmentDate": 1423790255000,
"simNumbers": [
"1236547896"
],
"imeiCodes": [],
"mobileNumbers": [
"4558585858858"
],
"createDate": 1421927250000,
"lastModified": 1423810406000,
"rooted": false,
"roaming": false,
"compliant": false
},
{
"apiResponseStatus": "SUCCESS",
"id": 24,
"ownership": "CORPORATE",
"deviceName": "Photobean",
"deviceType": "MOBILE",
"osType": "ANDROID",
"inTotalStorage": 20941258752,
"inAvailableStorage": 8024924160,
"exTotalStorage": 0,
"exAvailableStorage": 0,
"bluetoothMacAddress": "8C:3A:E3:BE:C2:04",
"wirelessMacAddress": "8c:3a:e3:51:bb:e3",
"model": "LG-D686",
"osVersion": "4.4.2",
"buildVersion": "1.01.09",
"status": "PENDING",
"isDeleted": false,
"policyIdNames": {
"id": 3,
"name": "Test Policy 3"
},
"deviceGroupIdNames": {
"id": 1,
"name": "Default Group"
},
"userIdName": {
"id": 1,
"name": "Randy Lewis"
},
"make": "LGE",
"lastConnected": 1423648831000,
"createdByIdName": {
"id": 1,
"name": "Randy Lewis"
},
"locked": false,
"traceOn": false,
"userEmail": "mdm.user1#gmail.com",
"enrollmentDate": 1423628675000,
"simNumbers": [
"404909008623970"
],
"imeiCodes": [
"359004053152889",
"359004053152871"
],
"mobileNumbers": [
"638847339387"
],
"createDate": 1422435029000,
"lastModified": 1423648831000,
"rooted": true,
"roaming": false,
"compliant": false
},
{
"apiResponseStatus": "SUCCESS",
"id": 25,
"ownership": "CORPORATE",
"deviceName": "Livetube",
"deviceType": "MOBILE",
"osType": "ANDROID",
"inTotalStorage": 0,
"inAvailableStorage": 0,
"exTotalStorage": 0,
"exAvailableStorage": 0,
"osVersion": "4.2",
"buildVersion": "1.01.08",
"status": "APPROVAL_PENDING",
"isDeleted": false,
"policyIdNames": {
"id": 3,
"name": "Test Policy 3"
},
"deviceGroupIdNames": {
"id": 1,
"name": "Default Group"
},
"userIdName": {
"id": 1,
"name": "Randy Lewis"
},
"lastConnected": 1423810522000,
"createdByIdName": {
"id": 1,
"name": "Randy Lewis"
},
"locked": false,
"traceOn": false,
"userEmail": "mdm.user1#gmail.com",
"enrollmentDate": 1423790371000,
"simNumbers": [
"1236547896"
],
"imeiCodes": [],
"mobileNumbers": [
"4558585858858"
],
"createDate": 1422435030000,
"lastModified": 1423810522000,
"rooted": false,
"roaming": false,
"compliant": false
},
{
"apiResponseStatus": "SUCCESS",
"id": 26,
"ownership": "CORPORATE",
"deviceName": "Riffwire",
"deviceType": "MOBILE",
"osType": "IOS",
"inTotalStorage": 0,
"inAvailableStorage": 0,
"exTotalStorage": 0,
"exAvailableStorage": 0,
"status": "PENDING",
"isDeleted": false,
"policyIdNames": {
"id": 3,
"name": "Test Policy 3"
},
"deviceGroupIdNames": {
"id": 1,
"name": "Default Group"
},
"userIdName": {
"id": 1,
"name": "Randy Lewis"
},
"createdByIdName": {
"id": 1,
"name": "Randy Lewis"
},
"locked": false,
"traceOn": false,
"userEmail": "mdm.user1#gmail.com",
"simNumbers": [],
"imeiCodes": [],
"mobileNumbers": [],
"createDate": 1422435030000,
"lastModified": 1423216312000,
"rooted": false,
"roaming": false,
"compliant": false
},
Also, Please suggest any good option if any.
You question is JMeter specific, but i believe it's actually regular Json Path expression. For example, in SoapUI it could be done with
$.deviceResponseList[?(#.status=='PENDING')].id
Probably this could help in some way
JMeter provides JSON Path Extractor via JMeter Plugins Extras With Libs Set
I believe that this is the best way to accomplish your goal.
See Using the XPath Extractor in JMeter guide (scroll down to "Parsing JSON" chapter) for the plugin installation instructions and some sample JSON Path queries.