How to export an existing dynamo table schema to json? - amazon-web-services

Question

I'd like to replicate some dynamodb tables, schema only, into my local environment for testing purposes. First I've tried:
aws dynamodb describe-table --table-name Foo > FooTable.json
But it's obvious that the output schema is not compliant to the input schema from the create-table command:
aws dynamodb create-table --cli-input-json file://FooTable.json --endpoint=http://localhost:8000
What I'm trying to avoid is to generate dozens of skeletons with aws dynamodb create-table --generate-cli-skeleton and fill them manually :/
Is there a way to get the table schema in a format that is "useful" for recreation? I find it unbelievable that there are no straightforward way of doing it through the web graphic interface or the standard aws command line - after hearing how "good" was their service.

Answer 1

I just managed to do a complete dump and "restore" using bchew/dynamodump:
git clone git#github.com:bchew/dynamodump.git
Notice the --schemaOnly option in the documentation https://github.com/bchew/dynamodump. Command was:
./dynamodump.py -m backup --schemaOnly --region foo-region --host localhost --srcTable '*' --port 8000 --accessKey fooKey --secretKey barKey
Then you can use the -m restore mode to put the data or schema back into a local dynamodb or wherever desired :)
With that said, I still find it unbelievable how bad is the amazon dynamodb tool-chain. Come on guys.

Answer 2

This takes aws dynamodb describe-table output, and transforms it into the input-format of aws dynamodb create-table --cli-input-json:
AWS_PROFILE=xyz aws dynamodb describe-table --table-name MyTable > mytable_full.json
# Pull out just what is minimally needed for table-creation:
#
# TableName
# KeySchema
# AttributeDefinitions (must only contain attributes used in keys)
# Global/Local Secondary Indexes
# Defaults BillingMode to PAY_PER_REQUEST
# (any provisioning can be set up manually based on load)
jq <mytable_full.json '.Table | {TableName, KeySchema, AttributeDefinitions} + (try {LocalSecondaryIndexes: [ .LocalSecondaryIndexes[] | {IndexName, KeySchema, Projection} ]} // {}) + (try {GlobalSecondaryIndexes: [ .GlobalSecondaryIndexes[] | {IndexName, KeySchema, Projection} ]} // {}) + {BillingMode: "PAY_PER_REQUEST"}' ​>mytable.json
AWS_PROFILE=xyz aws dynamodb create-table --cli-input-json file://mytable.json
You can also paste the json into python (the python dict syntax closely matches json) eg
import boto3
dynamodb = boto3.resource("dynamodb")
tabledef = {
"TableName": "MyTable",
"KeySchema": [
...
}
table = dynamodb.create_table(**tabledef)
print("Table status: ", table.table_status)
References:
https://docs.aws.amazon.com/cli/latest/reference/dynamodb/describe-table.html
https://docs.aws.amazon.com/cli/latest/reference/dynamodb/create-table.html
https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_CreateTable.html
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/dynamodb.html#creating-a-new-table

Answer 3

Here is a version using C#, AWS CLI and Newtonsoft JSON on Windows. Start by running this command: -
aws dynamodb describe-table --table-name TheTable --profile SourceAWSCredsProfile > TheTable.json
Pick up the file, deserialize and serialize to the --cli-input-json friendly class: -
TableContainer tableContainer;
string sourceFile = "TheTable.json";
string destFile = "TheTable.cli-input.json";
using (StreamReader file = File.OpenText(sourceFile))
{
JsonSerializer serializer = new JsonSerializer();
tableContainer = (TableContainer)serializer.Deserialize(file, typeof(TableContainer));
}
File.WriteAllText(destFile, JsonConvert.SerializeObject(tableContainer.Table, Formatting.Indented, new JsonSerializerSettings
{
NullValueHandling = NullValueHandling.Ignore
}
));
Now run this command to import the table definition: -
aws dynamodb create-table --cli-input-json file://TheTable.cli-input.json --profile DestinationAWSCredsProfile
The TableContainer class definition is below. The absence of certain properties cleans out everything that the --cli-input-json parameter doesn't need. You can recreate this class anytime by running: -
aws dynamodb create-table --generate-cli-skeleton
Then copy and paste the output into a new class file, using the very handy Paste Special... Paste JSON as Classes feature in Visual Studio.
public class TableContainer
{
public DynamoTableCLIJSON Table { get; set; }
}
public class DynamoTableCLIJSON
{
public Attributedefinition[] AttributeDefinitions { get; set; }
public string TableName { get; set; }
public Keyschema[] KeySchema { get; set; }
public Localsecondaryindex[] LocalSecondaryIndexes { get; set; }
public Globalsecondaryindex[] GlobalSecondaryIndexes { get; set; }
public string BillingMode { get; set; }
public Provisionedthroughput ProvisionedThroughput { get; set; }
public Streamspecification StreamSpecification { get; set; }
public Ssespecification SSESpecification { get; set; }
public Tag[] Tags { get; set; }
}
public class Provisionedthroughput
{
public int ReadCapacityUnits { get; set; }
public int WriteCapacityUnits { get; set; }
}
public class Streamspecification
{
public bool StreamEnabled { get; set; }
public string StreamViewType { get; set; }
}
public class Ssespecification
{
public bool Enabled { get; set; }
public string SSEType { get; set; }
public string KMSMasterKeyId { get; set; }
}
public class Attributedefinition
{
public string AttributeName { get; set; }
public string AttributeType { get; set; }
}
public class Keyschema
{
public string AttributeName { get; set; }
public string KeyType { get; set; }
}
public class Localsecondaryindex
{
public string IndexName { get; set; }
public Keyschema1[] KeySchema { get; set; }
public Projection Projection { get; set; }
}
public class Projection
{
public string ProjectionType { get; set; }
public string[] NonKeyAttributes { get; set; }
}
public class Keyschema1
{
public string AttributeName { get; set; }
public string KeyType { get; set; }
}
public class Globalsecondaryindex
{
public string IndexName { get; set; }
public Keyschema2[] KeySchema { get; set; }
public Projection1 Projection { get; set; }
public Provisionedthroughput1 ProvisionedThroughput { get; set; }
}
public class Projection1
{
public string ProjectionType { get; set; }
public string[] NonKeyAttributes { get; set; }
}
public class Provisionedthroughput1
{
public int ReadCapacityUnits { get; set; }
public int WriteCapacityUnits { get; set; }
}
public class Keyschema2
{
public string AttributeName { get; set; }
public string KeyType { get; set; }
}
public class Tag
{
public string Key { get; set; }
public string Value { get; set; }
}