I have successfully created an ASG with rolling update which seem to work. I have however, a rather unique use case. I would like to have a update strategy where I run both in parallell (EC2_old and EC2_new). Meaning, I want to make sure the new one is up and running during a test session of 15-30 min. During these 15-30 min I also want the deployment process to continue and not get stuck in a waiting mode for this transition to become complete. In a way I'm looking for a blue/green deployment strategy and I don't know if it is even possible.
I did some reading and came across WillReplace update policy. This could do the trick but the cfn options seem rather limited. Has anyone implemented an update strategy of this complexity?
Current policy looks like this:
updatePolicy = {
autoScalingRollingUpdate: {
maxBatchSize: 1,
minInstancesInService: 1,
pauseTime: "PT1H",
waitOnResourceSignals: true,
suspendProcesses: [
"HealthCheck",
"ReplaceUnhealthy",
"AZRebalance",
"ScheduledActions",
"AlarmNotification"
]
}
};
willReplace won't be a blue/green strategy. It does create a new ASG, but it will swap the target group to the new ASG instances as soon as they are all healthy. If you google aws blue green deployment you should find a quick start that goes over how to set up what you are looking for.
Related
I have been trying to come up with a way to track any and all instance type changed that happen in our companies account. (ex: t2.micro to t2.nano)
I settled on creating a custom config rule that would alert us if the instance changed with a uncompliant warning, but I think this might be over complicating it and am suspecting that I should be using CloudWatch alarms or EventBridge.
I have used the following setup (from the CLI):
rdk create ec2_check_instance_type --runtime python3.7 --resource-types AWS::ED2::Instance --input-parameters '{"modify-instance-type":"*"}'
modify-instance-type seemed to be the only thing I could find which related to what I was looking for the lambda function to track and I used the wildcard to signify any changes.
I then added the following to the lambda function:
if configuration_item['resourceType'] != 'AWS::EC2::Instance':
return 'NOT_APPLICABLE'
if configuration_item['configuration']['instanceType'] == valid_rule_parameters['ModifyInstanceAttribute']:
return 'NON_COMPLIANT'
is there a different input-parameter that I should be using for this instead of "modify-instance-type"? so far this has returned nothing. I don't think it is evaluating properly.
or does anyone have a service that might be a better way to track configuration changes like this within aws that I'm just not thinking of?
We are using AWS DMS for on-going replication of specific tables from one Oracle RDS database instance to another Oracle RDS database (both 11g).
Intermittently, the replication seems to fall behind or get out of sync. There are no errors in the log and everything is reported as successful, but data is missing.
We can kick off a full refresh and the data will show up, but this isn't a viable option on a regular basis. This is a production system and a full refresh takes upwards of 14 hours
We would like to monitor whether the destination database is [at least mostly] up to date. Meaning, no more than 2-3 hours behind.
I've found that you can get the current SCN from the source database using "SELECT current_scn FROM V$DATABASE" and from the target in the "awsdms_txn_state" table.
However, that table doesn't exist and I don't see any option to enable TaskRecoveryTableEnabled when creating or modifying a task.
Is there an existing feature that will automatically monitor these values? Can it be done through Lambda?
If DMS is reporting success, then we have no way of knowing that our data is hours or days behind until someone calls us complaining.
I do see an option in the DMS task to "Enable validation", but intuition tells me that's going to create a significant amount of unwanted overhead.
Thanks in advance.
There are a few questions here:
Task Monitoring of CDC Latency
How to set TaskRecoveryTableEnabled
For the first, task Monitoring provides a number of CloudWatch metrics (see all CDC* metrics).
It is possible to see on these metrics when the target is out of sync with the source, and where in the replication instance's process these changes are. The detailed blog from AWS explaining these Task Monitoring metrics is worth reading.
One option is to put a CloudWatch Alarm on the CDCLatencySource.
Alternatively you can create your own Lambda on a CloudWatch schedule to run your SCN queries on source and target and output a custom CloudWatch Metric using PutMetricData. You can create a CloudWatch Alarm on this metric if they are out of sync.
For the second question, to set the TaskRecoveryTableEnabled via the console tick the option "Create recovery table on target DB"
After ticking this you can confirm that the TaskRecoveryTableEnabled is set to Yes by looking at the Overview tab of the task. At the bottom there is the Task Settings json which will have something like:
"TargetMetadata": {
"TargetSchema": "",
"SupportLobs": true,
"FullLobMode": false,
"LobChunkSize": 0,
"LimitedSizeLobMode": true,
"LobMaxSize": 32,
"InlineLobMaxSize": 0,
"LoadMaxFileSize": 0,
"ParallelLoadThreads": 0,
"ParallelLoadBufferSize": 0,
"BatchApplyEnabled": false,
"TaskRecoveryTableEnabled": true
}
We are using AWS Elasticsearch for logs. The logs are streamed via Logstash continuously. What is the best way to periodically remove the old indexes?
I have searched and various approaches recommended are:
Use lambda to delete old indexes - https://medium.com/#egonbraun/periodically-cleaning-elasticsearch-indexes-using-aws-lambda-f8df0ebf4d9f
Use scheduled docker containers - http://www.tothenew.com/blog/running-curator-in-docker-container-to-remove-old-elasticsearch-indexes/
These approaches seem like an overkill for such a basic requirement as "delete indexes older than 15 days"
What is the best way to achieve that? Does AWS provide any setting that I can tweak?
Elasticsearch 6.6 brings a new technology called Index Lifecycle Manager See here. Each index is assigned a lifecycle policy, which governs how the index transitions through specific stages until they are deleted.
For example, if you are indexing metrics data from a fleet of ATMs into Elasticsearch, you might define a policy that says:
When the index reaches 50GB, roll over to a new index.
Move the old index into the warm stage, mark it read only, and shrink it down to a single shard.
After 7 days, move the index into the cold stage and move it to less expensive hardware.
Delete the index once the required 30 day retention period is reached.
The technology is in beta stage yet, however is probably the way to go from now on.
Running curator is pretty light and easy.
Here you can find a Dockerfile, config and action-file.
https://github.com/zakkg3/curator
Also, Curator can help you if you need to (among others):
Add or remove indices (or both!) from an alias
Change shard routing allocation
Delete snapshots
Open closed indices
forceMerge indices
reindex indices, including from remote clusters
Change the number of replicas per shard for indices
rollover indices
Take a snapshot (backup) of indices
Restore snapshots
https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html
Here is a typical action file for delete indices older than 15 days:
actions:
1:
action: delete_indices
description: >-
Delete indices older than 15 days (based on index name), for logstash-
prefixed indices. Ignore the error if the filter does not result in an
actionable list of indices (ignore_empty_list) and exit cleanly.
options:
ignore_empty_list: True
disable_action: True
filters:
- filtertype: pattern
kind: prefix
value: logstash-
- filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 15
I followed the elasticsearch-curator documentation to install the package:
https://www.elastic.co/guide/en/elasticsearch/client/curator/current/pip.html
Then I used the AWS base example of how to automate the indexes cleanup using the signed based authentication provided by requests_aws4auth package:
https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/curator.html
It worked like a charm.
You can decide to run this inside a lambda, docker or include it in your own DevOps cli.
I want to use AWS elasticsearch to store the log of my application. Since there a huge amount of data to input to AWS elasticsearch ( ~30GB daily), so i would only keep 3 days of data. Are there any way to schedule data removal from AWS elasticsearch or do a log rotation? What happen if the AWS elasticsearch storage is full?
Thanks for the help
A possible way is to specify the index parameter in elasticsearchoutput to something like logstash-%{appname}-%{date_format}". Hence you can then use curator plugin in order to delete the old indices by number of days or so.
This SO pretty much explains the same. Hope it helps!
I assume you are using the AWS Amazon Elasticsearch Service?
The storage type is an EBS volume with a fixed size of disk space. If you want to keep only the last three days, I assume you have 3 indices then, like that
my-index-2017.01.30
my-index-2017.01.31
my-index-2017.02.01
Basically you can write some simple script which deletes indices older than 3 days. With the REST API it just is in Sense DELETE my-index-2017.01.30.
I recommend to use Elasticsearch Curator for the job. See https://www.elastic.co/guide/en/elasticsearch/client/curator/current/delete_indices.html
I'm not sure if the Service interface itself has an option for that. But Elasticsearch Curator should do the job for you.
Update for 2020:
AWS ES has now support for Index state management which lets you define custom management policies to automate routine tasks and apply them to indices and index patterns. You no longer need to set up and manage external processes to run your index operations.
For example, you can define a policy that moves your index into a read_only state after 30 days and then ultimately deletes it after 90 days.
Index State Management - https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/ism.html
I'm accessing EC2 with the aws-sdk for Ruby. I have an array of instances from describe_instances().
This provides me with the state of the instances and even a state transition reason. But how can I get a time for the state transition?
Edit
So I have:
client=Aws::EC2::Client()
resp =client.describe_instances({ filters })
and I would need
resp.reservations[0].instances[0].state_transition_time #=> Time
similar to
resp.reservations[0].instances[0].state_transition_reason #=> String
This information is not available via the Amazon EC2 API at this time. The aws-sdk gem returns all of the information available from the DescribeInstances operation as documented here: http://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeInstances.html
The State Transition Reason is not always populated with a date and time and may not even be populated at all per the documentation. I have not found any hints in the documentation that specify the conditions in which you DO get a date/time, but in my experience, the date/time are present in the State Transition Reason for between 30 and 90 days. After that, the reason seems to persist, but the date is dropped from the string.
All of the documentation that I can find is listed here:
Attribute Definition
EC2 API - Ruby