Is it possible to use MS Sync Framework to synchronize different tables (with completely different structures) in different databases, assuming one of the tables has a primary key and the other a unique index whose values coincide with the first table's PK (but cannot be promoted to PK)? The column names can be different too.
that should be possible, have a look at these series of posts:
Part 1 – Upload Synchronization where the Client and Server Primary Keys are different
Part 2 – Bidirectional Synchronization where the Client and Server Primary Keys are different
Part 3 – Synchronizing tables where Client and Server Primary Keys are different
Related
From architectural perspective: Could you please help me understand why NoSQL DynamoDB is so hype.
DynamoDB supports some of the world’s largest scale applications by
providing consistent, single-digit millisecond response times at any
scale.
I'm trying to critic, in order to understand WHY part of the question.
We always have to specify partition Key and key attribute while retrieving to get millisecond of performance
If I design RDBMS:
where primary key or alternate key (INDEXED) always needs to be specified by in the query
I can use partition key to find out in which database my data is stored.
Never do JOINs
Isn't it same as NoSQL kind of architecture without any marketing buzz around it?
We're shifting to DynamoDB anyways but this is my innocent curiosity, there must be a strong reason which RDMBS can't do. Let's skip backup and maintenance advantages etc.
You are conflating two different things.
The definition of NoSQL
There isn't one, at least not one that can apply in all cases.
In most uses, NoSQL databases don't force your data into the fixed-schema "rows and columns" of a relational database. Although modern relational databases, such as Postgres, support data types such as JSONB that would have E. F. Codd spinning in his grave.
DynamoDB is a document database: it is optimized for retrieving and updating single documents based on a unique key, and it does not restrict the fields that those documents contain (other than requiring the ones used for a key).
Distributed Databases
A distributed database stores data on multiple nodes, and is able to perform parallel queries on those nodes and combine the results.
There are distributed SQL database: Redshift and BigQuery are optimized for queries against large datasets that may include joins, while MySQL (and no doubt others) which can run multiple engines and distribute the queries between them. It is possible for SQL databases to perform joins, including joins that cross nodes, but such joins generally perform poorly.
DynamoDB distributes items on shards based on their partition key. This makes it very fast for retrieving single items, because the query can be directed to a single shard. It is much less performant when scanning for items that reside on multiple shards.
As you note in your question, you can implement a sharded document DB on top of a relational database (either using native JSON columns or storing everything in a CLOB that is parsed for each access). But enough other people have done this (including DynamoDB) that it doesn't make sense (to me, at least) to re-implement.
Let’s assume there are mainly 3 tables for the current database.
Pkey = partition key
Admin
-id(Pkey), username, email, createdAt,UpdatedAt
Banner
-id(Pkey), isActive, createdAt, caption
News
-id(Pkey), createdAt, isActive, title, message
None of the above tables have relation with other tables, and more tables will be required in the future(I think most of it also don’t have the relation with other tables).
According to the aws document
You should maintain as few tables as possible in a DynamoDB application.
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-general-nosql-design.html
So I was considering the need to combine these 3 tables into a single table.
Should I start to use a single table from now on, or keep using multiple tables for the database?
If using a single table, how should I design the table schema?
DynamoDB is a NoSQL database, hence you design your schema specifically to make the most common and important queries as fast and as inexpensive as possible. Your data structures are tailored to the specific requirements of your business use cases.
When designing a data model for your DynamoDB Table, you should start from the access patterns of your data that would in turn inform the relation (or lack thereof) among them.
Two interesting resources that would help you get started are From SQL to NoSQL and NoSQL Design for DynamoDB, both part of the AWS Developer Documentation of DynamoDB.
In your specific example, based on the questions you're trying to answer (i.e. use case & access patterns), you could either work with only the Partition Key or more likely, benefit from the usage of composite Sort Keys / Sort Key overloading as described in Best Practices for Using Sort Keys to Organize Data.
Update, add example table design to get you started:
Currently I'm working with a client on an IOT project involving sensors. Currently all their data is being put into one table. This data is coming from multiple sensor nodes. They want one table for every sensor node. I want to know if through AWS Dynamo Db it is possible to split the data into multiple separate tables using the hash key from an existing table. I have looked into GSI's and LSI's but this still isn't exactly what my client wants. Also would having multiple table even be more effective than using and LSI or GSI ? I am new to nosql and dynamo db so all the help is very appreciated.
DynamoDB does not support splitting data into multiple tables - in the sense that DynamoDB operations themselves, including the atomic conditional checks, can't be performed across table boundaries. But that doesn't mean that splitting data across tables is incompatible with DynamoDB - just that you have to add the logic in your application.
You can definitely do so as long as the data from the different sensors is isolated enough. A more common scenario would be to split data into multiple tables across time boundaries in order to discard/archive old data, since DynamoDB already makes it possible and convenient to handle partitioning your data with hash keys and global secondary indexes.
In the end I would say that there is no need and it doesn't make sense to split data into multiple tables on the hash key - but it can be done. However, a more useful case is to split data into multiple tables on some other attribute of the data that is not part of the hash, or range key (such as the time-series data example).
I am trying to synchronize 2 data stores using 2 custom providers. Each provider handles its own data store and each provider communicates using APIs which means I can't access the databases using built-in providers. The problem is replica A uses long as primary keys, but the Replica B uses GUIDs instead.
When I synchronize I get an exception saying:
System.InvalidOperationException: The IsVariable setting of the specified ID instance is not consistent with the requested data type
at Microsoft.Synchronization.SyncId.GetGuidId()
I understand that the sync agent is trying to find the ItemMetdata using the long key but actually the ItemMetata exists but it was stored in GUIDs.
How can my application handle two replicas with 2 different primary keys?
Please note that I am not targeting SQL Databases.
I am using Hibernate L2 cache with Coherence for caching in two different web services.
Scenario
First web service has an entity class Employee with 5 fields
Second web service has the same entity class Employee but with 3 fields.
Both are pointing to same table/schema and the package hierarchy is also same.
Now when fresh request for employeeId=1 comes to second web service, it fetches from the value from the database and caches the 3 columns; keeps the other 2 as null.
Now when a request for employeeId=1 comes from the first web service, it directly fetches from cache by providing 3 columns and returns the other 2 as null, even though in the database the 2 columns have non-null values.
Is there a way by which I can force it get these column from database?
Approaches already tried
If I keep the columns in both the web services as same the problem goes away but this is not a acceptable solution in my scenario.
I tried added different serialVersion but it doesn't work.
Keeping the fully qualified name different works, but this is force us add overhead to performing manual eviction
You should be able to use the Evolvable interface for this, which will allow you to insert an object into the grid that is both forward and backward compatible. You just need to ensure that Second Webservice sets a lower version than First.