Boto3 DMS 'modify_replication_task' error on replication task settings JSON - amazon-web-services

Question

I'm using boto3 to create DMS replication tasks. I'm using the following replication_task_settings.json to create the replication tasks:
{
"TargetMetadata": {
"TargetSchema": "",
"SupportLobs": true,
"FullLobMode": false,
"LobChunkSize": 0,
"LimitedSizeLobMode": true,
"LobMaxSize": 256,
"InlineLobMaxSize": 0,
"LoadMaxFileSize": 0,
"ParallelLoadThreads": 0,
"ParallelLoadBufferSize": 0,
"BatchApplyEnabled": false,
"TaskRecoveryTableEnabled": false
},
"FullLoadSettings": {
"TargetTablePrepMode": "TRUNCATE_BEFORE_LOAD",
"CreatePkAfterFullLoad": false,
"StopTaskCachedChangesApplied": false,
"StopTaskCachedChangesNotApplied": false,
"MaxFullLoadSubTasks": 8,
"TransactionConsistencyTimeout": 1000,
"CommitRate": 10000
},
"Logging": {
"EnableLogging": true,
"LogComponents": [
{
"Id": "SOURCE_UNLOAD",
"Severity": "LOGGER_SEVERITY_DEFAULT"
},
{
"Id": "TARGET_LOAD",
"Severity": "LOGGER_SEVERITY_DEFAULT"
},
{
"Id": "SOURCE_CAPTURE",
"Severity": "LOGGER_SEVERITY_DEFAULT"
},
{
"Id": "TARGET_APPLY",
"Severity": "LOGGER_SEVERITY_DEFAULT"
},
{
"Id": "TASK_MANAGER",
"Severity": "LOGGER_SEVERITY_DEFAULT"
}
],
},
"ControlTablesSettings": {
"ControlSchema": "control",
"HistoryTimeslotInMinutes": 5,
"HistoryTableEnabled": true,
"SuspendedTablesTableEnabled": true,
"StatusTableEnabled": true
},
"StreamBufferSettings": {
"StreamBufferCount": 3,
"StreamBufferSizeInMB": 8,
"CtrlStreamBufferSizeInMB": 5
},
"ChangeProcessingDdlHandlingPolicy": {
"HandleSourceTableDropped": false,
"HandleSourceTableTruncated": true,
"HandleSourceTableAltered": false
},
"ErrorBehavior": {
"DataErrorPolicy": "LOG_ERROR",
"DataTruncationErrorPolicy": "LOG_ERROR",
"DataErrorEscalationPolicy": "SUSPEND_TABLE",
"DataErrorEscalationCount": 0,
"TableErrorPolicy": "SUSPEND_TABLE",
"TableErrorEscalationPolicy": "STOP_TASK",
"TableErrorEscalationCount": 0,
"RecoverableErrorCount": -1,
"RecoverableErrorInterval": 5,
"RecoverableErrorThrottling": true,
"RecoverableErrorThrottlingMax": 1800,
"ApplyErrorDeletePolicy": "IGNORE_RECORD",
"ApplyErrorInsertPolicy": "LOG_ERROR",
"ApplyErrorUpdatePolicy": "LOG_ERROR",
"ApplyErrorEscalationPolicy": "LOG_ERROR",
"ApplyErrorEscalationCount": 0,
"ApplyErrorFailOnTruncationDdl": false,
"FullLoadIgnoreConflicts": true,
"FailOnTransactionConsistencyBreached": false,
"FailOnNoTablesCaptured": false
},
"ChangeProcessingTuning": {
"BatchApplyPreserveTransaction": true,
"BatchApplyTimeoutMin": 1,
"BatchApplyTimeoutMax": 30,
"BatchApplyMemoryLimit": 500,
"BatchSplitSize": 0,
"MinTransactionSize": 1000,
"CommitTimeout": 1,
"MemoryLimitTotal": 1024,
"MemoryKeepTime": 60,
"StatementCacheSize": 50
},
"ValidationSettings": {
"EnableValidation": true,
"ValidationMode": "ROW_LEVEL",
"ThreadCount": 5,
"PartitionSize": 10000,
"FailureMaxCount": 10000,
"RecordFailureDelayInMinutes": 5,
"RecordSuspendDelayInMinutes": 30,
"MaxKeyColumnSize": 8096,
"TableFailureMaxCount": 1000,
"ValidationOnly": false,
"HandleCollationDiff": false,
"RecordFailureDelayLimitInMinutes": 0
}
}
The JSON above works fine when calling dms_client.create_replication_task. However, it doesn't work when modifying the replication tasks.
When calling dms_client.modify_replication_task with the replication_task_settings.json mentioned above I get the following error:
botocore.exceptions.ClientError: An error occurred (InvalidParameterValueException) when calling the ModifyReplicationTask operation: Invalid task settings JSON
I'm not sure why this is happening and any help would be greatly appreciated!
I tried removing some settings that are already defaulted. I tried looking for malformed JSON but nothing obvious.
I would expect the replication_task_settings,json to work for both creating and modifying DMS replication tasks.

Answer 1

I had similar issue while trying to modify the replication task using CLI :
aws --profile non-prod dms modify-replication-task --replication-task-arn arn:aws:dms:ap-southeast-3:567384657322:task:ABC --replication-task-settings file:/json_task --region ap-southeast-3
This is the error :
An error occurred (InvalidParameterValueException) when calling the
ModifyReplicationTask operation: Invalid task settings JSON
Below command worked :
aws --profile non-prod dms modify-replication-task --replication-task-arn arn:aws:dms:ap-southeast-3:567384657322:task:ABC --replication-task-settings file://json_task --region ap-southeast-3
Change is to use two "//" before JSON file name to modify the task.

Answer 2

The boto3 document for modify_replication_task() for TableMappings suggests -- When using the CLI or boto3, provide the path of the JSON file that contains the table mappings. Precede the path with file:// . For example, --table-mappings file://mappingfile.json . When working with the DMS API, provide the JSON as the parameter value.
Yes, you are absolutely correct. Providing JSON file as a parameter works in CLI but not with boto3. In boto3, it throws the error 'Invalid TableMappings / Invalid JSON error'.
Try the below to make it work:
import boto3
dms_client = boto3.client('dms')
table_mapping = {
"rules":[
{
"rule-type":"selection",
"rule-id":str(idx),
"rule-name":str(idx),
"object-locator":{
"schema-name":schema_name,
"table-name":table_name
},
"rule-action":"include"
}
]
}
dms_client.modify_replication_task(
ReplicationTaskArn=repl_task_arn,
TableMappings=json.dumps(table_mapping)
)

Answer 3

I faced a similar issue yesterday.
My json is in S3 and the final solution was simply applying indent=4
obj = s3.Object(bucket, key)
new_json = obj.get()['Body'].read().decode('utf-8')
new_json = json.loads(new_json)
client.modify_replication_task(
ReplicationTaskArn=taskarn,
TableMappings=json.dumps(new_json, indent=4)
)