What will happen if one column mentioned as primary key in DB and other column as primary key in mapping structure in informatica Powercenter or IICS.
It would not create an issue, as long as it is for reading the data.
But, if we are going to write the data in the target(using different keys) then Informatica would perform the record operation(I/U/D) on the basis of the key defined in Informatica.
Related
I'm trying to set up a Postgresql migration using the DMS to s3 as target. But after running I noticided that some tables were missing some columns.
After checking the logs I noticed this message:
Column 'column_name' was removed from table definition 'schema.table': the column data type is LOB and the table has no primary key or unique index
In the settings of the task migration I tried to increase the lob limit in the option
Maximum LOB size to 2000000
But still getting the same result.
Does anyone know a workaround for this problem?
I guess, the problem is you do not have the primary key in your table.
From AWS documentation:
Currently, a table must have a primary key for AWS DMS to capture LOB
changes. If a table that contains LOBs doesn't have a primary key,
there are several actions you can take to capture LOB changes:
Add a primary key to the table. This can be as simple as adding an ID
column and populating it with a sequence using a trigger.
Create a materialized view of the table that includes a
system-generated ID as the primary key and migrate the materialized
view rather than the table.
Create a logical standby, add a primary key to the table, and migrate
from the logical standby.
Learn more
It is also important to have the primary key of a simple type, not LOB:
In FULL LOB or LIMITED LOB mode, AWS DMS doesn't support replication of primary keys that are LOB data types.
Learn more
I am working on a backfill issue where I need to fetch all the unique values for an attribute in a dynamo db table and call a service to add these to the storage of that service. I am thinking of creating a temporary dynamo db table. I can read the original table in a lambda function and write only the unique values in the temp table. Is there any other approach possible?
The dynamo DB table has approximately 1,400,000 rows.
1,400,000 records is not that many. Probably you can just read the table.
You can improve the read by making your attribute a global secondary key. It need not be unique. Then you can read only the attribute in question or check uniqueness.
If the records in your table are constantly updated, you can listen to the DynamoDB update stream and just update your temporary table with the new values.
Using the single table pattern https://www.youtube.com/watch?v=EOQqi6Yun7g - your "temporary" table can be just a different primary key prefix.
If you have to scan the table and the process is too long, you can split it to multiple lambda calls by passing around the LastEvaluatedKey value (e.g. with a step machine).
You can scan the whole table, using projection expression fetch only the relevant columns and extract unique values.
One more approach can be, you can take a backup of DynamoDB table to S3 and then process the S3 file to extract unique column values.
everyone,
I've a problem in Informatica PowerCenter.
In my mapping I have 5 objects:
1x Source Table
1x Source Qualifier
1x Expression Transformation
1x Update Strategy
1x Target Table
The source and target table have no primary key, how come Informatica PowerCenter expects a key?
I have tried changing the "Treat source rows as" property of my workflow session from "Insert" to "Data driven" and it is working.
You have an update strategy in your mapping. Which expects you must have some key defined on target. Infa fires query like
UPDATE tgt SET col =? WHERE KEY = ?
Question mark 1 is updated column and question mark 2 is the key.
You can set unique keys as primary key.
If you don't have a primary or unique keys in target, pls define all columns as keys except the updatable column.
Or, you can use target overwrite to write sql to update target, but here too, you have to set similar query like above.
Data driven should be set.
In Informatica, the ports marked as keys in Target Transformation indicate what should be used to build the Update statement in DB. It has nothing physically to do with real Primary Key defined in the database itself. Usually you use same columns as keys in Informatica and in DB - but this is not necessary. DB is unaware of what is set in Informatica and vice versa.
It's even perfectly valid to have same database table defined multiple times in Informatica and have different mappings that will update the data using different columns as keys.
Note however that if you use Update Strategy you have to define which columns to use as keys.
The dynamo db will not allow data to be inserted into table unless the value contains the primary key set during table creation.
Dynamodb table:
id (primary key)
device_id
temperature_value
I am sending data from IoT core rule engine into the Dynamodb (Split message into multiple columns of a DynamoDB table (DynamoDBv2)). However, data does not arrive at the dynamo db table if the msg is missing the id attribute.
Is there any way to set primary key to be auto incrementing every time a new data point arrives?
DynamoDB does not support auto incrementing functionality for keys as it might have in a relational database.
Instead this will need to be generated by you at the time of inserting the record into DynamoDB.
There are a few options to generate:
Use a primary key combined of partition key (referencing your sensor id) and a sort key (something such as an event time, or a randomly generated string).
Generate a random string instead and insert this.
Use a seperate data store such as relational or Redis, where you autoincrement a value and use this. This is really not ideal.
Use a seperate DynamoDB table to include this value ensuring you use a transactional write to lock the row and increment, and strongly consistent read to get the latest value. Again this is not ideal
I have a Dynamo DB table which has a primary key as well as a secondary index (routeId).
I need to retrieve records containing the secondary index. However, I need to get results for multiple routeId values in a single run. Any way to do it?