I have an AWS Code Pipeline whose build step uses the AWS CLI, so of course it needs AWS credentials to work.
I don't see how to do this ... the 'source stage' checks out from github and therefore has the source artifact as its output artifact.
I wound up just committing the credentials into the branch I'm building from (currently unencrypted, though I can of course encrypt them if this workaround has to go into production) ... and this works.
Isn't there some way to do this "correctly" without getting the credentials from that branch?
In the build action, if you are using CodeBuild project, You need to attach custom policies to CodeBuild service role. So, when you use AWS CLI goes through the credentials providers in order and will fetch temporary credentials for that role.
This way you don't have to commit sensitive environment configuration in your repo.
Related
When deploying to AWS from gitlab-ci.yml file, you usually use aws-cli commands as scripts. At my current workplace, before I can use the aws-cli normally, I have to login via aws-azure-cli, authenticate via 2FA, then my workstation is given a secret key than expires after 8 hours.
Gitlab has CI/CD variables where I would usually put the AWS_ACCESS_KEY and AWS_SECRET_KEY, but I can't create IAM role to get these. So I can't use aws-cli commands in the script, which means I can't deploy.
Is there anyway to authenticate Gitlab other than this? I can reach out to our cloud services team, but that will take a week.
You can configure OpenID to retrieve temporary credentials from AWS without needing to store secrets.
In my view its actually a best practice too, to use OopenID roles instead of storing actual credentials.
Add the identity provider fir gitlab in aws
Configure the role and trust
Retrieve a temporary credential
follow this https://docs.gitlab.com/ee/ci/cloud_services/aws/ or a more detailed version https://oblcc.com/blog/configure-openid-connect-for-gitlab-and-aws/
I need to create a CloudFormation SAM template that creates multiple resources. I also need to integrate GitHub Actions so code gets auto deployed on push. The problem is this requires that I give GitHub a lot of IAM permissions to do the necessary work. Note that I'm using an Open ID identity provider for GitHub as opposed to api keys.
How do I find out which IAM permissions are needed at minimum so I don't give GitHub more than what is necessary? Is there an AWS tool that can parse the deployment template and tell me which permissions are needed for both deployments and rollbacks?
No, there's no such tool. You can however look at the Cloudtrail logs to see which actions are being invoked by CloudFormation, and based on that you could put together the list of required policies.
Alternatively, you could replicate what the CloudFormation template is doing by using the AWS CLI. That way, you could use iamlive to generate the IAM policies required for those CLI commands.
Right now we are using two AWS accounts with Terraform (INT and PROD). Currently we have the same repositories in both of the accounts, which is obviously not best practice, so I'm trying to set up cross-account access. The goal is to have the repos only on the INT account, and make the CodeBuild project or CodePipeline instance on the PROD account to use those repos.
I have already created the necessary policies and roles on both accounts so lack of permission should not be a problem. However, what I'm struggling with is using the INT repos as source for the PROD CodeBuild project for example.
The project's source block is configured in the following way:
source {
type = "CODECOMMIT"
location = "https://git-codecommit.region.amazonaws.com/v1/repos/example-repo"
buildspec = data.local_file.buildspec_local.content
}
I can't seem to understand how should I proceed here. From my understanding, I should somehow reference the repo in the INT account, but from what I know there's only one way to do that, which is using multiple aws provider blocks with the accounts' access key id and secret key.
This solution is not possible for us, since we have to use the company's SSO tool to generate credentials which are valid for 12 hours to use the AWS accounts.
I am aware that there are loads of questions regarding cross-account access in AWS, but unfortunately I haven't found one that helps me.
Welcome to the world of AWS Terraform! Not sure how far you have gotten on this, but there are two options for you. Option 1 leverages AWS Cross Account Pipeline . Now I am currently in the process of building this out with terraform, but this would be considered the best practice for this type of development.
If you are just looking to pull in the repos from another account, then you can leverage Codebuild to ssh and clone the repository into your build job. to do this, you will need to create an ssh user in the repo account, which we will label A. You will need the ssh users' User Key and the Public Key. You will create two variables in the parameter store called variable_ssh_key and variable_codecommit_user.
Leveraging the buildspec provided here, you can leverage git clone to pull down any repository you need. The only thing you will need is in Account B to create one repo that stores the buildspec and calls the other repos. See buildspec as a reference.
With AWS CodeBuild you define GitHub as the source. If your repo is private, you can pass a personal token. I've built a CodeBuild process and confirmed this worked.
What I'd like to do is not use a personal token, instead use a GitHub Deploy Key to authenticate to the private repo. I've setup the Deploy Key in GitHub. How do I configure this either in the AWS CodeBuild Console or in the YAML script?
CodeBuild doesn't natively support deploy keys. It is on our product backlog and is a feature that we will likely support in a future release.
In order to use your existing deploy key in CodeBuild, please follow the instruction that Adrian has highlighted in https://adrianhesketh.com/2018/05/02/go-private-repositories-and-aws-codebuild/. You will need to setup the key in parameter-store and use that in your buildspec.
You can use the source type as "no_source", since you would be doing the source cloning with the deploy key in this case.
For those struggling using your SSH Keys in CodeBuild, you can use AWS' System Manager Service.
Simply put, can store your key in the Parameter Store and use that in your buildspec.yml
You can refer to this tutorial for a detailed implementation: https://medium.com/#cristiano.ventura/working-with-git-submodules-in-codepipeline-83e843e5d0a
It works for my case. Bitbucket webhook integration with private repo deps in the package.json
Best of luck y'all!
Ok so I have the exact build script to build my nodejs app.
The idea is to setup a CI/CD pipeline that will build our app across different aws accounts such as DEV, STAGING, and PROD.
I have a repo in Account A (Dev) that hosts a nodejs app
I want to be able to create a pipeline in Account B that checks out code from repo in account A
And finally when the pipeline is complete, it should deploy the built/compiled nodejs app to Account C (QA) and D (Prod)
My issue is not on how to build a nodejs app but rather how to allow codepipeline, in account B, to checkout the repo in account A and finally deploy the built app to staging and prod.
My ideas from reading around:
Create IAM roles for the pipeline in Account A which allows to
checkout codecommit repos
Have the pipeline in account B assume the role from account A somehow
It's still not clear to me how to go about doing this; I'm just getting into aws
I will update this post if I come across a solution but maybe someone has a tutorial or could point me to a documentation or list the steps or an example here
From my understanding it is not possible to build a cross account pipeline using codepipeline. What you need to do is build your pipeline in your central account first.
The central account pipeline would include the appropriate number of stages which result in a release candidate for the AWS accounts dev, staging and prod.
As you get to the different account stages, you should push your artifacts to S3 buckets in the related accounts. And these S3 buckets should be the sources of codepipelines the those accounts.
This way you create "deployment" pipelines in each account which start in S3 and end in whatever environment you are thinking of. The S3 buckets of these accounts can be created to have bucket policies to only receive files from your central account.
This is of course, not ideal but its how I solved this issue before hand. Build in one account and deliver to deployment pipelines in other accounts. If someone knows a better solution, I would love to hear it.
Good luck!
AWS provide a guide with source that does something close to what you are trying to do.
It should get you close enough and covers the permissions needed for the account to assume a role to checkout your repo in another account.