Redis Multi-AZ feature in CloudFormation - amazon-web-services

I'm designing a template which includes a Redis service and I would like to enable Multi-AZ feature in Redis such that upon the primary cluster failure, read replica can be promoted to primary. I looked in the CloudFormation documentation but I couldn't find this feature i.e. Multi-AZ. It is available for RDS service but not for Redis. Can I know how I can include the feature for redis such AWS take cares of the automatic failover ?
Source:
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-elasticache-cache-cluster.html
The list of properties that are available for the elastic cache are listed below.
"AutoMinorVersionUpgrade" : Boolean,
"AZMode" : String,
"CacheNodeType" : String,
"CacheParameterGroupName" : String,
"CacheSecurityGroupNames" : [ String, ... ],
"CacheSubnetGroupName" : String,
"ClusterName" : String,
"Engine" : String,
"EngineVersion" : String,
"NotificationTopicArn" : String,
"Port" : Integer,
"PreferredAvailabilityZone" : String,
"PreferredAvailabilityZones" : [String, ... ],
"PreferredMaintenanceWindow" : String,
"SnapshotArns" : [String, ... ],
"SnapshotName" : String,
"SnapshotRetentionLimit" : Integer,
"SnapshotWindow" : String,
"Tags" : [Resource Tag, ...],
"VpcSecurityGroupIds" : [String, ...]

This are the two ways you can set Redis to use Multi Az programatically.
Using CLI
aws elasticache modify-replication-group \
--replication-group-id myReplGroup \
--automatic-failover-enabled
Using Elasticache API
https://elasticache.us-west-2.amazonaws.com/
?Action=ModifyReplicationGroup
&AutoFailover=true
&ReplicationGroupId=myReplGroup
&Version=2015-02-02
&SignatureVersion=4
&SignatureMethod=HmacSHA256
&Timestamp=20140401T192317Z
&X-Amz-Credential=<credential>
This are some of the notes that you should read while selecting Multi Az for redis.
http://docs.aws.amazon.com/AmazonElastiCache/latest/UserGuide/AutoFailover.html#AutoFailover.Notes
For Cloud Formation below are the Properties:
{
"Type" : "AWS::ElastiCache::ReplicationGroup",
"Properties" : {
"AutomaticFailoverEnabled" : Boolean,
"AutoMinorVersionUpgrade" : Boolean,
"CacheNodeType" : String,
"CacheParameterGroupName" : String,
"CacheSecurityGroupNames" : [ String, ... ],
"CacheSubnetGroupName" : String,
"Engine" : String,
"EngineVersion" : String,
"NotificationTopicArn" : String,
"NumCacheClusters" : Integer,
"Port" : Integer,
"PreferredCacheClusterAZs" : [ String, ... ],
"PreferredMaintenanceWindow" : String,
"ReplicationGroupDescription" : String,
"SecurityGroupIds" : [ String, ... ],
"SnapshotArns" : [ String, ... ],
"SnapshotRetentionLimit" : Integer,
"SnapshotWindow" : String
}
}
You have to tweak this property for Multi Az
AutomaticFailoverEnabled
Indicates whether Multi-AZ is enabled. When Multi-AZ is enabled, a read-only replica is automatically promoted to a read-write primary cluster if the existing primary cluster fails. If you specify true, you must specify a value greater than 1 for the NumCacheNodes property. By default, AWS CloudFormation sets the value to true.
For more information about Multi-AZ, see Multi-AZ with Redis Replication Groups in the Amazon ElastiCache User Guide.
Note
You cannot enable automatic failover for Redis versions earlier than 2.8.6 or for T1 and T2 cache node types.
Required: No
Type: Boolean
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-elasticache-replicationgroup.html

Related

How can I let Kibana read custom index for metric?

I deployed Elasticsearch and Kibana 7.10.1. And I am streaming cloudwatch metrics data (raw json) to Elasticsearch.
The metric raw data format looks like:
{
"metric_stream_name" : "metric-stream-elk",
"account_id" : "264100014405",
"region" : "ap-southeast-2",
"namespace" : "AWS/DynamoDB",
"metric_name" : "ReturnedRecordsCount",
"dimensions" : {
"Operation" : "GetRecords",
"StreamLabel" : "2021-06-18T01:12:31.851",
"TableName" : "dev-dms-iac-events"
},
"timestamp" : 1624924620000,
"value" : {
"count" : 121,
"sum" : 0,
"max" : 0,
"min" : 0
},
"unit" : "Count"
}
I can see that these raw data are saved in Elasitcsearch with a custom index name aws-metrics-YYYY-MM-DD. Now how can I let Kibana read metrics from this index?
I don't want to use metricbeat because it queries metrics from AWS. My event flow is streaming AWS metrics to Elasticsearch. How can I achieve that?

Application deployed to Docker Swarm is not connecting with MongoDB Replica Set

I have deployed my application using Docker Swarm with 3 machines.
MongoDB Replica Set is configured manually and its working as a service on Ubuntu machine.
I am trying to connect to my Backend application to MongoDB Replica Set but I am getting context deadline exceeded error. I am using Private-ip to connect since machines are in same AWS VPC. Port 27017 is open in the security group and can be used by VPC network IP.
/etc/hosts is correctly configured on every machine.
I am using Docker-Compose file to deploy the stack.
Replica Set is working fine. I have checked it with manually inserting few documents.
The picture will help readers to understand the context better.
Abbreviations
BE = Backend
FE = Frontend
Mac 1 = Machine 1
AZ-1 = Availability Zone 1
VPC = Virtual Private Cloud
My Guess:
Is it because Replica-Set in not in the Swarm Network and that's why its unable to connect ???
I am trying to fix this issue for quite sometime now and have not been successful yet. Help is required now.
I found the solution.
The problem was related to the names of the MongoDB replica instances.
The host name for first member is "host" : "10.0.0.223:27017"
The host name for second member is "host" : "node2:27017"
The host name for third member is "host" : "node3:27017"
Due to this inconsistency, the backend application was not able to connect to the replica set.
{
"_id" : "replica1",
"version" : 5,
"term" : 5,
"protocolVersion" : NumberLong(1),
"writeConcernMajorityJournalDefault" : true,
"members" : [
{
"_id" : 0,
"host" : "10.0.0.223:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {
},
"slaveDelay" : NumberLong(0),
"votes" : 1
},
{
"_id" : 1,
"host" : "node2:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {
},
"slaveDelay" : NumberLong(0),
"votes" : 1
},
{
"_id" : 2,
"host" : "node3:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {
},
"slaveDelay" : NumberLong(0),
"votes" : 1
}
],
Solution
To solve this I reconfigured the replica set nodes with private IPs. Node that I have used Private IP for replica set configuration.
First ssh to primary machine. Login to mongo primary node shell and execute the following commands to change the host entry for 2nd mongo node in JSON.
> cfg = rs.conf()
> cfg.members[2].host = "10.0.5.242:27017" (Private IP of second mongodb instance)
> rs.reconfig(cfg)
Did the same for 3rd mongo entry in JSON.
Here is the link to change the hostname in Replica Set.
https://docs.mongodb.com/manual/tutorial/change-hostnames-in-a-replica-set/
Another Solution:
If your MongoDB deployment is all fine and everything is set up properly, but still you are unable to connect to the DB, try to Change the Primary DB Instance.
ssh to primary machine.
Login to mongo primary node shell and execute this commands
replica:PRIMARY>rs.stepDown(120). This will make your other instance Primary and your current instance Secondary.
I experienced this problem and after continuous hit and try for 3 days, this solution worked for me.
Hope this would help readers to solve similar problem.

How to read ssm parameters in a shell script in aws data pipeline?

I'm setting up a data pipeline in aws. and plan to use a "getting started using ShellCommandActivity" template to run a shell script. how can i pass credentials stored in ssm parameter as a parameter to this script.
I haven't verified that, but ShellCommandActivity is similar to ShellScriptConfig from what I can tell. Based on the examples provided for these commands, I would think that you could pass the ssm param name as follows:
{
"id" : "CreateDirectory",
"type" : "ShellCommandActivity",
"command" : "your-script.sh <name-of-your-parameter>"
}
or
{
"id" : "CreateDirectory",
"type" : "ShellCommandActivity",
"scriptUri" : "s3://my-bucket/example.sh",
"scriptArgument" : ["<name-of-your-parameter>"]
}
and in the example.sh you would use $1 to refer to the value of the argument passed.

Capture start and end of lambda functions

Is it possible to capture the startTime and endTime of execution of lambda functions along with parameters that were passed to it ?
I couldn't find any state-change event configurations that could be configured to send events when lambda function starts/terminates?
A crappy alternative is to record parameters & start time in database when the lambda is being invoked and have the lambda update the endgame as final step before it's completion. This appears prone to failures scenarios like function erroring out before updating DB.
Are there other alternatives to capture this information
aws x-ray may be a good solution here. It is easy to integrate and use. You may enable it aws console.
Go to your lambda function/ configuration tab
Scroll down & in AWS X-Ray box choose active tracing.
Without any configuration in the code, it is going to record start_time and end_time of the function with additional meta data. You may integrate it as a library to your lambda function and send additional subsegments such as request parameters. Please check here for documentation
Here is a sample payload;
{
"trace_id" : "1-5759e988-bd862e3fe1be46a994272793",
"id" : "defdfd9912dc5a56",
"start_time" : 1461096053.37518,
"end_time" : 1461096053.4042,
"name" : "www.example.com",
"http" : {
"request" : {
"url" : "https://www.example.com/health",
"method" : "GET",
"user_agent" : "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/601.7.7",
"client_ip" : "11.0.3.111"
},
"response" : {
"status" : 200,
"content_length" : 86
}
},
"subsegments" : [
{
"id" : "53995c3f42cd8ad8",
"name" : "api.example.com",
"start_time" : 1461096053.37769,
"end_time" : 1461096053.40379,
"namespace" : "remote",
"http" : {
"request" : {
"url" : "https://api.example.com/health",
"method" : "POST",
"traced" : true
},
"response" : {
"status" : 200,
"content_length" : 861
}
}
}
]
}

AWS Datapipeline : moving data from RDS database(postgres) to Redshift using pipeline

Basically i am trying to transfer data from postgres to redshift using aws datapipeline and the process i am following
Write a pipeline(CopyActivity) that moves data from postgres to s3
Write a pipeline(RedShiftCopyActivity) that moves data from s3 to redshift
So in my case both are working perfectly with the pipelines i wrote, but the problem is the data was duplicating in the Redshift database
For example below is the data from postgres database in a table called company
After the successful run of s3 to redshift(RedShiftCopyActivity) pipeline the data was copied but it was duplicated as below
below is the some of the definition part from RedShiftCopyActivity(S3 to Redshift) pipeline
pipeline_definition = [{
"id":"redshift_database_instance_output",
"name":"redshift_database_instance_output",
"fields":[
{
"key" : "database",
"refValue" : "RedshiftDatabaseId_S34X5",
},
{
"key" : "primaryKeys",
"stringValue" : "id",
},
{
"key" : "type",
"stringValue" : "RedshiftDataNode",
},
{
"key" : "tableName",
"stringValue" : "company",
},
{
"key" : "schedule",
"refValue" : "DefaultScheduleTime",
},
{
"key" : "schemaName",
"stringValue" : RedShiftSchemaName,
},
]
},
{
"id":"CopyS3ToRedshift",
"name":"CopyS3ToRedshift",
"fields":[
{
"key" : "output",
"refValue" : "redshift_database_instance_output",
},
{
"key" : "input",
"refValue" : "s3_input_data",
},
{
"key" : "runsOn",
"refValue" : "ResourceId_z9RNH",
},
{
"key" : "type",
"stringValue" : "RedshiftCopyActivity",
},
{
"key" : "insertMode",
"stringValue" : "KEEP_EXISTING",
},
{
"key" : "schedule",
"refValue" : "DefaultScheduleTime",
},
]
},]
So according to the docs of RedShitCopyActivity we need to use insertMode to describe how the data should behave(inserted/updated/deleted) when copying to database table as below
insertMode : Determines what AWS Data Pipeline does with pre-existing data in the target table that overlaps with rows in the data to be loaded. Valid values are KEEP_EXISTING, OVERWRITE_EXISTING, TRUNCATE and APPEND. KEEP_EXISTING adds new rows to the table, while leaving any existing rows unmodified. KEEP_EXISTING and OVERWRITE_EXISTING use the primary key, sort, and distribution keys to identify which incoming rows to match with existing rows, according to the information provided in Updating and inserting new data in the Amazon Redshift Database Developer Guide. TRUNCATE deletes all the data in the destination table before writing the new data. APPEND will add all records to the end of the Redshift table. APPEND does not require a primary, distribution key, or sort key so items that may be potential duplicates may be appended.
So what my requirements are
When copying from postgres (infact data is in s3 now) to Redshift database if it found already existing rows then just update it
If it founds new records from s3 then create new records in Redshift
But for me even though i have used KEEP_EXISTING or OVERWRITE_EXISTING, the data was just repeating over and over again as shown in the above redshift database picture
So finally how to achieve my requirements ? are there still any tweaks or settings to add to my configuration ?
Edit
Table(company) definition from redshift
If you want to avoid duplication , you must define Primary key in redshift and also set myInsertMode as "OVERWRITE_EXISTING" .