caching go modules in codebuild without custom docker image - amazon-web-services

TL;DR
How can I cache my modules in codebuild using AWS provided image (Go 1.12)?
Background
I'm trying to cache go modules in codebuild using the go image (1.12) from AWS.
Trying to cache /go/pkg/mod
After digging deeper, I found that there is no /go/pkg folder in that image. Hence, when I tried to cache /go/pkg it would throw an error.
Error mounting /go/pkg/mod: symlink /codebuild/local-cache/custom//go/pkg/mod /go/pkg/mod: no such file or directory
Even after I run go mod download (which will create the /go/pkg/mod, it won't cache the folder because codebuild cannot mounted it earlier).
This is my codebuild.yml
version: 0.2
phases:
install:
runtime-versions:
golang: 1.12
nodejs: 10
commands:
- npm install
build:
commands:
- go build -ldflags="-s -w" -o api/bin/main api/main.go
cache:
paths:
- /go/src/**/*
- /go/pkg/mod/**/*
Trying to cache ./vendor
I also tried caching ./vendor folder which doesn't throw any errors in codebuild. However, I don't think it's caching anything because the build time doesn't decrease. It also says it ignores the symlink.
warning: ignoring symlink /codebuild/output/src074479210/src/github.com/kkesley/myrepo/vendor
go: finding github.com/aws/aws-lambda-go v1.11.1
go: finding github.com/stretchr/testify v1.2.1
go: finding github.com/pmezard/go-difflib v1.0.0
go: finding github.com/davecgh/go-spew v1.1.0
go: finding gopkg.in/urfave/cli.v1 v1.20.0
go: downloading github.com/aws/aws-lambda-go v1.11.1
go: extracting github.com/aws/aws-lambda-go v1.11.1
This is my codebuild.yml for this version:
version: 0.2
phases:
install:
runtime-versions:
golang: 1.12
nodejs: 10
commands:
- npm install
- go mod vendor
build:
commands:
- go build -mod vendor -ldflags="-s -w" -o api/bin/main api/main.go
cache:
paths:
- /go/src/**/*
- vendor/**/*
Question
How do you cache go modules in code build without using custom docker image? Is it possible?

To get it working with the default CodeBuild Ubuntu build image (I'm using v4)
Make sure that local caching is enabled on the CodeBuild project. Go to to edit then artifacts then make sure Custom Cache is ticked
Set the path to cache as /go/pkg/**.*
cache:
paths:
- '/go/pkg/**/*'
Make sure your build script includes a step before building for go mod download. Before I did this caching didn't seem to work so this seemed a key step.
Here is my full buildspec.yml for reference
version: 0.2
phases:
install:
runtime-versions:
golang: latest
commands:
- "git config --global credential.helper '!aws codecommit credential-helper $#'"
- "git config --global credential.UseHttpPath true"
build:
commands:
- 'go mod edit -dropreplace git-codecommit.ap-southeast-2.amazonaws.com/v1/repos/xyz'
- 'go mod download'
# Use latest from develop for the build (test env only)
- 'go get git-codecommit.ap-southeast-2.amazonaws.com/v1/repos/xyz#develop'
- 'rm -rf "dist"'
- 'cp -r "eb-template" "dist"'
- 'env GOOS=linux GOARCH=amd64 go build -o "dist/bin/server"'
- 'go mod edit -replace git-codecommit.ap-southeast-2.amazonaws.com/v1/repos/xyz=../xyz'
- 'echo -n ${CODEBUILD_RESOLVED_SOURCE_VERSION} > dist/commithash'
artifacts:
base-directory: dist
files:
- '**/*'
cache:
paths:
- '/go/pkg/**/*'

Related

Beanstalk's nginx not picking the .conf file on my application source bundle

I have a Spring Boot application built on Beanstalk (Amazon Linux 2), I need to increase the client_max_body_size because some form data I'm posting contains images and I'm getting the 413: Request Too Large Nginx error.
I followed AWS's documentation on how to change this property.
My project structure looks like this now:
And the content of the file is:
client_max_body_size 50M;
After deploying I keep getting the same error (with images > 1MB total).
No file has been created in conf.d:
Is this because how my buildSpec packages my application?
version: 0.2
phases:
install:
runtime-versions:
java: corretto17
pre_build:
commands:
- echo Nothing to do in the pre_build phase...
build:
commands:
- echo Build started on `date`
- mvn package -Dmaven.test.skip
post_build:
commands:
- echo Build completed on `date`
artifacts:
files:
- .platform/nginx/conf.d/proxy.conf
- target/myApp-0.0.1-SNAPSHOT.jar
- appspec.yml
discard-paths: yes
I also tried adding the configuration file into the artifacts.files section of my buildspec.yml.
I also tried to create the file its content from the files section on the buildspec.
I feel like I tried everything, is there anything I may be missing?
For now, my workaround:
I edited mannually the file
cd /etc/nginx/
sudo nano nginx.conf
and restarted. That worked, but I want to avoid this manual configuration so it's configured from the application source, as a good practice.
The problem was on my buildspec.
discard-paths: yes
was putting all the files on the root path of the bundle. I needed this so I the jar was on the root, but it was putting the proxy.conf in the root as well.
Setting that property to no (or removing it) made it work, but I needed a way to change the jar from /target/ to the root so I did it with a post-build command:
version: 0.2
phases:
install:
runtime-versions:
java: corretto17
pre_build:
commands:
- echo Nothing to do in the pre_build phase...
build:
commands:
- echo Build started on `date`
- mvn package -Dmaven.test.skip
post_build:
commands:
- mv target/myApp-0.0.1-SNAPSHOT.jar myApp-0.0.1-SNAPSHOT.jar <------ HERE
- echo Build completed on `date`
artifacts:
files:
- .platform/nginx/conf.d/proxy.conf
- myApp-0.0.1-SNAPSHOT.jar
- appspec.yml

AWS CodeDeploy, Files getting removed from other folder

I have a code pipeline setup for React application. Whenever a new build is updated(In client folder) , the files(directories remain) are getting removed the server folder.
Tried seeing the event logs in code deploy, not able to get much info
could someone please help me with this?
appSpec.yml
version: 0.0
os: linux
files:
- source: /
destination: /home/ubuntu/client
hooks:
AfterInstall:
- location: scripts/after.sh
timeout: 300
runas: root
buildSpec.yml
version: 0.2
env:
variables:
NODE_ENV: "development"
phases:
install:
runtime-versions:
nodejs: 14
pre_build:
commands:
- npm install
build:
commands:
- npm run build-qa
artifacts:
type: zip
paths:
- './build/**/*'
- './appspec.yml'
- './scripts/**/*'
after.sh
#!/bin/bash
cd /home/ubuntu/client
echo '---Process started---'
echo '---Taking backup---'
sudo rm -r backup_*
sudo mv /home/ubuntu/client/deploy /home/ubuntu/client/backup_$(date +"%d-%m-%Y")
echo '---backup complete---'
echo '---renaming build---'
sudo mv /home/ubuntu/client/build /home/ubuntu/client/deploy
echo '---renaming complete---'
Edit: I was running two CI/CD pipelines (One for Frontend and other was backend) both running for the same EC2 instance, I was using the same deployment group and application for both the CI/CD which was causing the issue, later I created separate deployment application and group for each CI/CD which resolved . Thanks #Shivkumar Mallesappa for the lead

How I can deploy a subfolder into my bucket's root path?

For my react App I use the following buildspec.yml to deploy a reast Single Page Application:
version: 0.2
phases:
pre_build:
commands:
- npm install
build:
commands:
- npm run build
cache:
paths:
- /root/.npm/**/*
artifacts:
files:
- ./build/**
I also created an s3 bucket as static website and I deploy it directly via codeploy. But the folder that is deployed is into the ./build istead of the bucket's root. Is there a way to translate the paths of the codebuild's output artifacts into the deployed paths?
The codebuild path is used as a step in a codepipeline.
In order to do that you must specify a base-directory then tell the codebuild to set everything as artifact. This would be your final buildspec.yml:
version: 0.2
phases:
pre_build:
commands:
- npm install
build:
commands:
- npm run build
cache:
paths:
- /root/.npm/**/*
artifacts:
files:
- '**/*'
base-directory: './build'
And you also can safely delete the rogue ./build folder into your s3 bucket.

CodeBuild + ReactNative + Expo Web - This build image requires selecting at least one runtime version

Trying to use CodeBuild for the first time, pulling data from CodeCommit. But I'm having issues with my buildspec. This is the code I have on it so far:
version: 0.2
phases:
INSTALL:
runtime-versions:
nodejs: 10
commands:
- npm install
PRE_BUILD:
commands:
- npm install --quiet --global expo-cli
- >
if [ -f yarn.lock ]; then
yarn
elif [ -f package-lock.json ] || [ -f npm-shrinkwrap.json ]; then
npm ci
else
npm install
fi
BUILD:
commands:
- expo build:web
artifacts:
baseDirectory: web-build
files:
- '**/*'
name:
myname-$(date +%Y-%m-%d)
cache:
paths:
- node_modules/**/*
- $(npm root --global)/**/*
I have already added the runtime for nodejs 10, it had stopped to trigger this error, but now it kicked again. Does anyone know how to properly tweak it for React-Native web projects?
I believe the phase names are case sensitive, so change them to install, pre_build and build.

AWS CodeBuild golang build fails

When I run the following command: - go build -o app I get the following error (for multiple dependencies) : main.go:21:2: cannot find package "github.com/gorilla/mux" in any of:
/usr/local/go/src/github.com/gorilla/mux (from $GOROOT)
/go/src/github.com/gorilla/mux (from $GOPATH)
/codebuild/output/src324986171/src/github.com/gorilla/mux
Meaning the codebuild fails. Any idea how I can fix this, or in general where the problem is ?
Thanks for your help.
EDIT:
After adding go get ./... to the build I get the following error for all my local packages:# cd .; git clone https://github.com/aristotle/dbhelper /go/src/github.com/aristotle/dbhelper
Cloning into '/go/src/github.com/aristotle/dbhelper'...
My buildspec.yml looks like this:
version: 0.2
phases:
install:
commands:
- echo CODEBUILD_SRC_DIR - $CODEBUILD_SRC_DIR
- echo GOPATH - $GOPATH
- echo GOROOT - $GOROOT
build:
commands:
- echo Build started on `date`
- echo Getting packages
- go get ./...
- echo Compiling the Go code...
- go build -o app main.go
post_build:
commands:
- echo Build completed on `date`
artifacts:
files:
- app
According to this article you need to add it to the install section of your buildspec.yml file.
install:
commands:
- go get github.com/gorilla/mux
maybe it does also work to include go get ./... this will resolve all dependencies... but if you do not have too many it is a good practise to list them explicitly.
this is the source article: https://www.contributing.md/2017/06/30/golang-with-aws-codebuild/