Is there a way to create a resource if it doesn't exist and use an existing resource if it does?
resources:
- name: "my-topic"
type: gcp-types/pubsub-v1:projects.topics
properties:
topic: "this-exists-already"
- name: "my-other-resource"
type: 'gcp-types/cloudfunctions-v1:projects.locations.functions'
properties:
functionName: "function"
environmentVariables:
# get a ref to new or already existing topic
my-toptic: "$(ref.my-topic.name)"
Per #kolban's link I think I want to use abandon here. Can I selectively "abandon" a specific resource so I can, for example, attach an accessControl policy to an existing bucket but then NOT delete that bucket if the deployment is deleted?
ABANDON - This removes any references to the resource from the
deployment but does not delete the underlying resource. For example,
abandoning an instance means that it is removed from a deployment but
the instance still exists for you to use.
Edit
Maybe I should use an "action" to assign an acl instead of a resource? Is this the right way and are there examples of this? So DM would essentially just execute an api call to apply an acl out-of-band. That would mean it would leave the acl behind if the deployment is deleted but I'm okay with that.
It looks like I want to do something like this but instead of applying an acl to a specific file I want to set it on the bucket (with an action) https://github.com/GoogleCloudPlatform/deploymentmanager-samples/blob/master/community/storage-bucket-acl/storagebucket-acl.jinja#L29.
If we read this section of the Deployment Manager documentation:
https://cloud.google.com/deployment-manager/docs/deployments/updating-deployments#policies_for_adding_resources
We read about the concept of "create or acquire". The way I read this is that if a resource your configuration says should be created then the default appears to be that if it already exists, it will not cause an error and will be "acquired" for this deployment which I take to mean that it will be as though it had been created.
Related
I created and deployed a S3 resource (bucket) using cloudformation.
After that i deployed a version without that resource.
Then i deployed a version with the resource.
Since the bucket exists, it gives me an error that it cannot deploy.
This has happened to me before, in past times I deleted the resource and deployed again.
I'm looking for a way to use the resource for future deployments. It is the exact same resource, this is the yaml :
Bucket:
Type: AWS::S3::Bucket
Properties:
BucketName: !Sub "myBucketName"
Is there anything I can add to the resource, a policy, a unique ID, anything so that i could use the existing resource?
Thanks!
To "use the existing resource" in a CFN, you have to import it. Also its a bad practice to keep modify resources created by CFN outside of CFN. This leads to drift and number of issues, one of which you are experiencing.
I'm trying to use CloudFormation AddOn template in the following scenario:
Service 1
creates an SNS Topic and a Managed Policy that has all the necessary permissions to publish to it. The SNS Topic will collect "Activity" records and then fan them out to multiple subscribers.
A common code library abstracts away the usage of SNS - any applications that need to post activity messages do so without any knowledge that SNS is being used underneath the covers.
Service N needs to publish activity messages using the common code library and needs whatever permissions are necessary.
So service 1 writes the Managed Policy ARN out as an exported output to the AddOn stack like so:
Outputs:
activityPublishPolicy:
Description: "Activity Publish Policy ARN"
Value: !Ref activitySnsTopicPublishPolicy
Export:
Name: !Sub ${App}-${Env}-activity-publish-policy
Then in service N, I was hoping to import the ARN of the publishing policy and get it attached to the task role:
Outputs:
activityPublishAccessPolicy:
Description: "The IAM::ManagedPolicy to attach to the task role."
Value: !ImportValue
'Fn::Sub': '${App}-${Env}-activity-publish-policy'
The ARN is imported just fine and written out to the Cloud Formation stack of Service N; however, the Task Role does not get the Managed Policy attached to it.
I did a quick test to see if adding the policy directly to the AddOn stack would attach and that does indeed work.
Outputs:
activityPublishAccessPolicy:
Description: "The IAM::ManagedPolicy to attach to the task role."
Value: !Ref activityPolicy
This leads me to believe that Copilot only attaches ManagedPolicies to the Task Role that are created in its own AddOn Stack, but that's just a guess.
I'd prefer not to write a new policy in every service to do this, and I'd prefer not to open up the topic policy our whole VPC if possible.
Is there a better way of doing this?
Thanks!
This is because Copilot scans the Addons template to determine the type of the resource you're outputting. There are several "magic" outputs for addons. They are:
Security Groups
Managed Policies
Secrets
To detect these outputs, we scan the template looking for the logical ID of the referenced resource. This means that we don't currently have a way of deriving the resource type of the results of Fn::ImportValue calls, since they don't refer to a logical ID defined in that addons template!
I'm sorry this is causing you problems--it seems like you may need to add the managed policy to the addons stack of each service you want to grant this access to. This is something we might be able to do something about, though, and would love if if you could cut us a Github issue so we can prioritize and gather feedback on a proposal.
while configuring resource configuration , is there any way I can use so that serverless wont create throw any error if resource is already present.
eg. don't throw this error if following resource is already present.
Error : An error occurred: PaymentQueue - dev_payment_cron_queue already exists in stack
resources:
Resources:
PaymentQueue:
Type: "AWS::SQS::Queue"
Properties:
QueueName: ${self:provider.stage}_payment_cron_queue
VisibilityTimeout: 40
Sadly, there is no such way with pure CloudFormation (CFN), as this is not how CFN (or Terraform as a matter of fact) was designed to work. From CFN perspective, a given resource exists and is managed by CFN, or it does not exist at all. There is no middle ground.
If your resource already exist, you have to import it to CFN so that it gets managed by CFN. Alternatively, you have to create custom resource in the form of a lambda function. The function would perform any action you want based on the existing resources, including checking if it exists or not.
Im creating API gateway stage using cloudformation.
ApiDeployment:
Type: AWS::ApiGateway::Deployment
Properties:
RestApiId: !Ref ExampleRestApi
StageName: dev
Here is the problem, Whenever I create a new API, I just need to deploy the stage using AWS console. is there any way that I can automate the deploy process so that no further console action is required.
When you define a Deployment resource like this, CloudFormation will create the deployment only on the first run. On the second run it will observe that the resource already exists and the CloudFormation definition did not change, so it won't create another deployment. To work around that, you can add something like a UUID/timestamp placeholder to the resource ID and replace it everytime before doing the CloudFormation update:
ApiDeployment#TIMESTAMP#:
Type: AWS::ApiGateway::Deployment
Properties:
RestApiId: !Ref ExampleRestApi
StageName: dev
This way you are still able to see your deployment history in the API Gateway console.
If you don't want to manipulate your template like this, you can also add a Lambda-backed Custom Resource to your CloudFormation stack. Using an AWS SDK, you can have the Lambda function automatically creating new deployments for you when the API was updated.
I've found berenbums response to be mostly correct, but there are a few things I don't like.
The proposed method of creating a resource like ApiDeployment#TIMESTAMP# doesn't keep the deployment history. This makes sense, since the old ApiDeployment#TIMESTAMP# element is being deleted and a new one is being created every time.
Using ApiDeployment#TIMESTAMP# creates a new deployment every time the template is deployed, which might be undesirable if the template is being deployed to create/update other resources.
Also, using ApiDeployment#TIMESTAMP# didn't work well when adding the StageDescription property. A potential solution is to add a static APIGwDeployment resource for the initial deployment (with StageDescription) and ApiDeployment#TIMESTAMP# for the updates.
The fundamental issue though, is that creating a new api gw deployment is not well suited for cloudformation (beyond the initial deployment). I think after the initial deployment, it's better to do an AWS API invocation to update the deployment (see https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-deployments.html).
In my particular case I created a small Ansible module to invoke aws apigateway create-deployment which updates an existing stage in one operation.
Is there a way to access auto-generated URLs for deployed resources before the deployment is finished? (like db host, lambda function URL, etc.)
I can access them after the deployment is finished, but sometimes I need to access them while building my stack. (E.g. use them in other resources).
What is a good solution to handle this use-case? I was thinking about outputting them into the SSM parameter store from CloudFormation template, but I'm not sure if this is even possible.
Thanks for any suggestion or guidance!
If "use them in other resources" means another serverless service or another CloudFormation stack, then use CloudFormation Outputs to export the values you are interested in. Then use CloudFormation ImportValue function to reference that value in another stack.
See https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/outputs-section-structure.html and https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-importvalue.html
Within Serverless Framework, you can access a CloudFormation Output value using https://serverless.com/framework/docs/providers/aws/guide/variables/#reference-cloudformation-outputs
If you want to use the autogenerated value within the same stack, then just use CloudFormation GetAtt function. See https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-getatt.html.
For example, I have a CloudFormation stack that outputs the URL for an ElasticSearch cluster.
Resources:
Search:
Type: AWS::Elasticsearch::Domain
Properties: <redacted>
Outputs:
SearchUrl:
Value: !GetAtt Search.DomainEndpoint
Export:
Name: myapp:search-url
Assuming that the CloudFormation stack name is "mystack", then in my Serverless service, I can reference the SearchUrl by:
custom:
searchUrl: ${cf:mystack.SearchUrl}
To add to bwinant's answer, ${cf:<stack name>.<output name>} does not work if you want to reference a variable in another stack which is located in another region. There is a plugin to achieve this called serverless-plugin-cloudformation-cross-region-variables. You can use it like so
plugins:
- serverless-plugin-cloudformation-cross-region-variables
custom:
myVariable: ${cfcr:ca-central-1:my-other-stack:MyVariable}