AWS Custom Resource ROLLBACK_FAILED because of a catch22-like situation - amazon-web-services

Question

I have a Custom Resource which gets stuck in a catch22-like situation whenever a ROLLBACK occurs.
The code below is a simplified example of what my code is doing. In case of a create request, it creates a table, for delete it deletes, and in case of an update it compares the old properties with the new, and returns an error when one of the columns has a new value (column updates are not supported yet).
A problematic ROLLBACK_FAILED occurs when
[SOLVED] whenever a create request type fails (due to sql syntax error for example). In this case it will trigger a delete request for the ROLLBACK phase, but this request will fail because the table does not yet exist.
whenever a update request type fails due to a updated column value. In this case it will trigger a new update request for the ROLLBACK phase in which the event.ResourceProperties and event.OldResourceProperties are switched, which will still cause an error.
package main
import (
"context"
"encoding/json"
"fmt"
"strings"
"github.com/aws/aws-lambda-go/cfn"
"github.com/aws/aws-lambda-go/lambda"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/service/rdsdataservice"
"github.com/google/uuid"
)
func main() {
lambda.Start(cfn.LambdaWrap(handler))
}
type properties struct {
TableName string
Columns []struct {
Name string
Value string
}
}
func handler(ctx context.Context, event cfn.Event) (physicalResourceID string, data map[string]interface{}, err error) {
prid := event.PhysicalResourceID
if event.RequestType == cfn.RequestCreate {
prid = strings.ReplaceAll(uuid.New().String(), "-", "")
}
var props properties
b, _ := json.Marshal(event.ResourceProperties)
json.Unmarshal(b, &props)
rds := rdsdataservice.New(nil)
if event.RequestType == cfn.RequestCreate {
rds.ExecuteStatement(&rdsdataservice.ExecuteStatementInput{
Sql: aws.String(fmt.Sprintf("CREATE TABLE %s", props.TableName)),
})
}
if event.RequestType == cfn.RequestDelete {
rds.ExecuteStatement(&rdsdataservice.ExecuteStatementInput{
Sql: aws.String(fmt.Sprintf("CREATE TABLE %s", props.TableName)),
})
}
if event.RequestType == cfn.RequestUpdate {
var oldProps properties
b, _ := json.Marshal(event.OldResourceProperties)
json.Unmarshal(b, &oldProps)
columns := map[string]string{}
for _, column := range props.Columns {
columns[column.Name] = column.Value
}
for _, column := range oldProps.Columns {
if val, ok := columns[column.Name]; ok {
if val != column.Value {
return "", nil, fmt.Errorf("cannot change column type")
}
}
}
// Do some extra stuff here for adding/removing columns
}
return prid, nil, nil
}
I have thought of 2 possible solutions. One of them I could implement, with potential issues. But it seems to me there should be a better way, as I can't be the only one with this problem. Or I am doing something very wrong..
disable rollback for this specific resource only, in some cases (sometimes I still want a rollback)
have access to the last status, so that I can check what to do; in case of delete with last status CREATE_FAILED, don't do anything. In case of an update with last status UPDATE_FAILED, don't do anything.
The second option I could implement by using the code below. But as the number of events grow, this could become very problematic.
events, err := cloud.DescribeStackEvents(&cloudformation.DescribeStackEventsInput{
StackName: &event.StackID,
})

Answer 1

For anyone who runs into the same problem; I have solved this issue by looking at the stack events, and find the last resource status that is being updated. As the current status is always in the list, I ignore IN_PROGRESS values. If the first value after status IN_PROGRESS is FAILED, it means this resource could not be updated, and thus a different ROLLBACK strategie can be applied.
The corresponding function, in GO
func isFailedEvent(sess *session.Session, event cfn.Event) (bool, error) {
cloud := cloudformation.New(sess)
var isFailedResource bool
if err := cloud.DescribeStackEventsPages(&cloudformation.DescribeStackEventsInput{
StackName: aws.String(event.StackID),
}, func(out *cloudformation.DescribeStackEventsOutput, lastPage bool) bool {
for _, e := range out.StackEvents {
if *e.LogicalResourceId == event.LogicalResourceID {
if strings.HasSuffix(*e.ResourceStatus, "IN_PROGRESS") {
continue
}
if strings.HasSuffix(*e.ResourceStatus, "FAILED") && !strings.Contains(*e.ResourceStatusReason, "cancelled") {
isFailedResource = true
}
return false
}
}
return true
}); err != nil {
return false, fmt.Errorf("describe stack events: %s", err)
}
return isFailedResource, nil
}