I have a repository in a [company hosted] GitHub enterprise site that I need to integrate within a CI/CD pipeline in AWS using the CodePipeline service. The CodePipeline service then will connect with a Jenkins Service running in an EC2 instance. I am following this AWS documentation to do this.
Note that this is an enterprise GitHub repo and not hosted on GitHub.com.
I am unable to connect to the GitHub Enterprise site in the "Create a pipeline" wizard. Selecting GitHub as a source only lets me connect to GitHub.com, and not to my organization's enterprise site.
How do I connect to the enterprise GitHub site so that I can add the repo as a source for my codepipeline?
UPDATE: An integration was finally released for github enterprise to work with AWS Codepipeline. Please see this answer by Pedreiro
As Sid Malani said in the comments, there's no straightforward or direct way to use a privately hosted GitHub Enterprise Repository in AWS CodePipeline.
One way would be to use CodeBuild, which allows to add custom github repositories which can then be used in CodePipeline. This, however, should only be considered as a stop-gap solution.
Thank you Sid Malani for the alternative strategy
I doubt AWS will ship this feature anytime soon (hopefully I eat my words) since the workaround is well-documented and works fine. The gist of it can be found here, which I'll summarize as follows:
Create a CodeBuild project (which can take GitHub Enterprise as a source) which essentially just listens to a push event (webhook) of your development or master branch and outputs a .zip file of the source-code into an S3 bucket.
Use the S3 bucket as the source of your CodePipeline.
The official integration was finally released.
https://aws.amazon.com/about-aws/whats-new/2020/09/aws-codepipeline-supports-github-enterprise-server
Here is the documentation:
https://docs.aws.amazon.com/codepipeline/latest/userguide/connections-ghes.html
There is a way to integrate 3rd party Git repos with AWS CodePipeline. AWS suggested the following process. It used S3 buckets and API Gateways and Lambdas to connected Github enterprise repos to CodePipeline. It's not ideal but it can work.
Unofficial: I've heard CodePipeline support for Github Enterprise will be out second half of 2019.
Related
I have my source code in code commit and my new client is with GCP. They wanted to connect code-commit from google cloud-build, is there any option for that ?
Given the fact that GCP and AWS are competitor cloud providers I would say that you will not find a way to trigger Google Cloud Build from AWS CodeCommit, which is what I believe you mean with "integrate" both products.
What I would do in your scenario is replicate you CodeCommit repository in it's equivalent in GCP, which is Google Cloud Source Repositories. You can find a tutorial for how to setup
Build Triggers from Cloud Source Repositories in this documentation. Another option is pushing a container ready to be deployed into Cloud Registry and deploying that instead, you can follow these steps for that.
I am looking to integrate enterprise bitbucket server with aws ci/cd pipeline features.
I have tried creating a project within aws codebuild but do not see any option for bitbucket enterprise .
If this is not possible then what is the long route using api gateway / webhooks etc ?
AWS Codebuild only supports the Bitbucket cloud. To integrate with Bitbucket self hosted solution, you will need to create a API gateway + Lambda. And then add this gateway address as a webhook in the bitbucket repo. The Lambda will then be responsible to process the incoming events from Bitbucket server. There could be 2 routes from here.
One way could be to download the zip for the particular commit and upload it on a S3 bucket. Add S3 as a source trigger for the build project. You lose the ability to run any git specific commands in such a case though as it's just a zip file containing the specific version of files.
Second option could be to pass on the relevant info to codebuild by directly invoking it from Lambda. Passing off details like commit_id, event (pr or push), branch etc as environment variables. Based on this info, run a git clone in codebuild before running other build steps. This way you would have access to git specific commands.
Here is an example workflow from AWS (it is for codepipeline, but you can modify it suitably for codebuild)
I am trying to deploy(continuous deployment) my bitbucket repository using AWS CodeDeploy and following the blog by aws https://aws.amazon.com/blogs/apn/announcing-atlassian-bitbucket-support-for-aws-codedeploy/
While adding AWS CodeDeploy plugin to BitBucket, it shows message in Bitbucket marketplace
This app is hidden from public view because it no longer meets the requirements for Atlassian Marketplace.
Is there any other plugin which has replaced it or bitbucket doesn't supports AWS CodeDeploy anymore?
That is a very old document and probably the plugin is not maintained anymore. This article on Atlassian site seems more relevant and up-to-date but it uses Atlassian 'Pipeline' product:
Deploy to AWS with CodeDeploy: https://confluence.atlassian.com/bitbucket/deploy-to-aws-with-codedeploy-976773337.html
Second option is using AWS CodePipeline to fetch BitBucket code and Deploy using CodeDeploy deploy action. This tutorial will be helpful (replace CodeCommit with BitBucket):
Tutorial: Create a Simple Pipeline (CodeCommit Repository) - https://docs.aws.amazon.com/codepipeline/latest/userguide/tutorials-simple-codecommit.html
With AWS CodeBuild you define GitHub as the source. If your repo is private, you can pass a personal token. I've built a CodeBuild process and confirmed this worked.
What I'd like to do is not use a personal token, instead use a GitHub Deploy Key to authenticate to the private repo. I've setup the Deploy Key in GitHub. How do I configure this either in the AWS CodeBuild Console or in the YAML script?
CodeBuild doesn't natively support deploy keys. It is on our product backlog and is a feature that we will likely support in a future release.
In order to use your existing deploy key in CodeBuild, please follow the instruction that Adrian has highlighted in https://adrianhesketh.com/2018/05/02/go-private-repositories-and-aws-codebuild/. You will need to setup the key in parameter-store and use that in your buildspec.
You can use the source type as "no_source", since you would be doing the source cloning with the deploy key in this case.
For those struggling using your SSH Keys in CodeBuild, you can use AWS' System Manager Service.
Simply put, can store your key in the Parameter Store and use that in your buildspec.yml
You can refer to this tutorial for a detailed implementation: https://medium.com/#cristiano.ventura/working-with-git-submodules-in-codepipeline-83e843e5d0a
It works for my case. Bitbucket webhook integration with private repo deps in the package.json
Best of luck y'all!
What I'm trying to do is to create a following CI flow with standard AWS tools: run a build of a commit when a Pull Request in Github is created or updated. Or run a build of any branch on my command. Very similar to what Codeship, Travis and many other CI services offer.
Is it possible with CodeBuild + CodePipeline? I noticed that I have to specify exact branch in CodePipeline and, unfortunately, could not find how to integrate Github Pull requests into it. Maybe I overlooked it?
CodeBuild now directly supports building GitHub pull requests (without Lambda intermediate step), if you're looking to simply run a build as part of the PR. For running more steps with CodePipeline as part of a PR, you'll still need to set up some scaffolding as the other answers suggest.
https://aws.amazon.com/about-aws/whats-new/2017/09/aws-codebuild-now-supports-building-github-pull-requests/
CodePipeline does support basic, fully-managed integrations with both GitHub and CodeBuild, as listed in Product and Service Integrations with AWS CodePipeline. With these integrations, it is possible to use CodeBuild with CodePipeline to run a build of a commit when a commit is pushed to a branch on GitHub. See Use AWS CodePipeline with AWS CodeBuild to Run Builds for details on integrating CodeBuild with CodePipeline as a Build action provider, and see the Four-Stage Pipeline Tutorial for details on integrating Github with CodePipeline as a Source action provider.
Currently, the Pull Request feature in Github is not supported in the official CodePipeline integration, you did not overlook anything. For an interesting AWS-ecosystem open source project (not yet v1.0) that does support GitHub Pull Request integration (though not yet CodePipeline), you might want to check out LambCI.
It looks like this can be done somewhat manually by using Lambda and S3 - https://aws.amazon.com/blogs/devops/integrating-git-with-aws-codepipeline/
Webhooks notify a remote service by issuing an HTTP POST when a commit is pushed to the repository. AWS Lambda receives the HTTP POST through Amazon API Gateway, and then downloads a copy of the repository. It places a zipped copy of the repository into a versioned S3 bucket. AWS CodePipeline can then use the zip file in S3 as a source; the pipeline will be triggered whenever the Git repository is updated.
You could try https://www.deploytoproduction.com for Github Pull Request build status integration with AWS CodeBuild. It is free for a single Github repository with a subscription plan available for multiple repositories.
The service doesn't currently integrate with CodePipeline but that is coming soon.
If you wanted to build something yourself, you could make a new integration on GitHub that uses the webhook functionality to trigger a lambda function which in turn triggers your CodeBuild jobs or pushes an artifact to S3 to start a CodePipeline.
Full disclosure I am the author of this service