AWS Codepipeline Spring boot War file - amazon-web-services

I have a Spring Boot Application with gradle as the dependency manager. I have been creating the war file using Export As war file. And have deployed the same in AWS ElasticBeanstalk. The platform I use is Tomcat 8.5 with Java 8 running on 64bit Amazon Linux/3.4.5
Now, I want to use CI/CD of AWS and want to create a Code Pipeline to deploy the war file to EBS.
While creating the Code-Pipeline, it asks me to build the Code Build Project.
Following is my build command :
version: 0.2
#env:
#variables:
# key: "value"
# key: "value"
#parameter-store:
# key: "value"
# key: "value"
#secrets-manager:
# key: secret-id:json-key:version-stage:version-id
# key: secret-id:json-key:version-stage:version-id
#exported-variables:
# - variable
# - variable
#git-credential-helper: yes
#batch:
#fast-fail: true
#build-list:
#build-matrix:
#build-graph:
phases:
#install:
#If you use the Ubuntu standard image 2.0 or later, you must specify runtime-versions.
#If you specify runtime-versions and use an image other than Ubuntu standard image 2.0, the build fails.
#runtime-versions:
# name: version
# name: version
#commands:
# - command
# - command
#pre_build:
#commands:
# - command
# - command
build:
commands:
- gradle build
# - command
#post_build:
#commands:
# - command
# - command
#reports:
#report-name-or-arn:
#files:
# - location
# - location
#base-directory: location
#discard-paths: yes
#file-format: JunitXml | CucumberJson
#artifacts:
#files:
# - location
# - location
#name: $(date +%Y-%m-%d)
#discard-paths: yes
#base-directory: location
#cache:
#paths:
# - paths
What changes do I make here, so that the war file is created and deployed to EBS. Since, I am pretty new to the Build Process and CI/CD, I am a bit confused.
My build.gradle file is :
plugins {
id 'org.springframework.boot' version '2.2.7.RELEASE'
id 'io.spring.dependency-management' version '1.0.9.RELEASE'
id 'java'
id 'war'
}
group = 'com.example'
version = '0.0.1-SNAPSHOT'
sourceCompatibility = '1.8'
configurations {
compileOnly {
extendsFrom annotationProcessor
}
developmentOnly
runtimeClasspath {
extendsFrom developmentOnly
}
}
repositories {
mavenCentral()
}
dependencies {
implementation 'org.springframework.boot:spring-boot-starter-web'
annotationProcessor 'org.projectlombok:lombok'
providedRuntime 'org.springframework.boot:spring-boot-starter-tomcat'
//compile group: 'org.hibernate', name: 'hibernate-gradle-plugin', version: '5.4.15.Final'
compile group: 'org.springframework.boot', name: 'spring-boot-starter-data-jpa', version: '2.2.7.RELEASE'
compile group: 'mysql', name: 'mysql-connector-java', version: '8.0.20'
compile group: 'com.fasterxml.jackson.core', name: 'jackson-databind', version: '2.11.0'
compile group: 'io.jsonwebtoken', name: 'jjwt', version: '0.9.1'
compile group: 'org.springframework.boot', name: 'spring-boot-starter-mail', version: '2.3.0.RELEASE'
compile group: 'org.springframework.boot', name: 'spring-boot-starter-security', version: '2.3.0.RELEASE'
compile group: 'com.amazonaws', name: 'aws-java-sdk', version: '1.11.784'
providedCompile group: 'org.projectlombok', name: 'lombok', version: '0.11.0'
// https://mvnrepository.com/artifact/org.projectlombok/lombok
compileOnly group: 'org.projectlombok', name: 'lombok', version: '1.18.18'
compile group: 'com.google.code.gson', name: 'gson', version: '2.8.6'
compile group: 'org.apache.poi', name: 'poi-ooxml', version: '4.1.2'
compile group: 'org.apache.poi', name: 'poi', version: '4.1.2'
compile group: 'org.codehaus.mojo', name: 'exec-maven-plugin', version: '3.0.0'
compile group: 'com.googlecode.json-simple', name: 'json-simple', version: '1.1.1'
compile group: 'com.google.zxing', name: 'core', version: '3.4.1'
implementation group: 'org.apache.commons', name: 'commons-lang3', version: '3.12.0'
implementation group: 'com.google.common', name: 'google-collect', version: '0.5'
developmentOnly("org.springframework.boot:spring-boot-devtools")
//testImplementation 'org.springframework.security:spring-security-test'
}
test {
useJUnitPlatform()
}

I came across the same problem a few days ago and I hardly found any good resources for gradle. You can easily deploy your application directly to Elastic Beanstalk by uploading a WAR file generated during the gradle build.
When you upload your WAR file to Beanstalk, it gets extracted/exploded and Beanstalk does this for you internally. But when you use the pipeline, you need to upload the exploded files directly instead of the WAR. It seems that the AWS Codepipeline is still not WAR friendly.
First off, to generate an exploded WAR file, you need to add the following to your build.gradle:
task explodedWar(type: Sync) {
into "${buildDir}/exploded"
with war }
To generate the exploded folder, you need to run the following task through your terminal using the command:
./gradlew explodedWar
You will find a folder named exploded in the build folder where you will be able to see the contents of the exploded WAR.
Here is the following builspec.yml file you can use to build the code and deploy it using the pipeline. The environment used by me is Tomcat 8.5 with Corretto 8 running on 64bit Amazon Linux 2:
version: 0.1
phases:
install:
run-as: root
commands:
- echo Installing gradle
- wget https://services.gradle.org/distributions/gradle-7.1.1-bin.zip
- unzip -d /opt/gradle gradle-7.1.1-bin.zip
pre_build:
commands:
- echo Entering pre build
build:
commands:
- /opt/gradle/gradle-7.1.1/bin/gradle clean
- /opt/gradle/gradle-7.1.1/bin/gradle explodedWar
post_build:
commands:
- echo Entering post build
artifacts:
files:
- '**/*'
base-directory: build/exploded/
discard-paths: no
You can check the AWS buildspec.yml documentation for more syntax and features depending on your use case.

Related

whitelist AWS RDS on CircleCI

I have a circleCI configuration to run my tests before merge to the master, I start my server to do my tests and the I should connect to my RDS database and its protected with security groups I tried to whitelist circleci ip to allow this happen but with no luck
version: 2.1
orbs:
aws-white-list-circleci-ip: configure/aws-white-list-circleci-ip#1.0.0
aws-cli: circleci/aws-cli#0.1.13
jobs:
aws_setup:
docker:
- image: cimg/python:3.11.0
steps:
- aws-cli/install
- aws-white-list-circleci-ip/add
build:
docker:
- image: cimg/node:18.4
steps:
- checkout
- run: node --version
- restore_cache:
name: Restore Npm Package Cache
keys:
# Find a cache corresponding to this specific package-lock.json checksum
# when this file is changed, this key will fail
- v1-npm-deps-{{ checksum "package-lock.json" }}
# Find the most recently generated cache used from any branch
- v1-npm-deps-
- run: npm install
- run:
name: start the server
command: npm start
background: true
- save_cache:
name: Save Npm Package Cache
key: v1-npm-deps-{{ checksum "package-lock.json" }}
paths:
- ./node_modules
- run:
name: run tests
command: npm run test
- aws-white-list-circleci-ip/remove
workflows:
build-workflow:
jobs:
- aws_setup:
context: aws_context
- build:
requires:
- aws_setup
context: aws_context
my context environment
AWS_ACCESS_KEY_ID
AWS_DEFAULT_REGION
AWS_SECRET_ACCESS_KEY
GROUPID
the error
the orbs I am using
https://circleci.com/developer/orbs/orb/configure/aws-white-list-circleci-ip
I figure it out
version: 2.1
orbs:
aws-cli: circleci/aws-cli#0.1.13
jobs:
build:
docker:
- image: cimg/python:3.11.0-node
steps:
- checkout
- run: node --version
- restore_cache:
name: Restore Npm Package Cache
keys:
# Find a cache corresponding to this specific package-lock.json checksum
# when this file is changed, this key will fail
- v1-npm-deps-{{ checksum "package-lock.json" }}
# Find the most recently generated cache used from any branch
- v1-npm-deps-
- run: npm install
- aws-cli/install
- run:
command: |
public_ip_address=$(wget -qO- http://checkip.amazonaws.com)
echo "this computers public ip address is $public_ip_address"
aws ec2 authorize-security-group-ingress --region $AWS_DEFAULT_REGION --group-id $GROUPID --ip-permissions "[{\"IpProtocol\": \"tcp\", \"FromPort\": 22, \"ToPort\": 7000, \"IpRanges\": [{\"CidrIp\": \"${public_ip_address}/32\",\"Description\":\"CircleCi\"}]}]"
- save_cache:
name: Save Npm Package Cache
key: v1-npm-deps-{{ checksum "package-lock.json" }}
paths:
- ./node_modules
- run:
name: run tests
command: npm run test
# Invoke jobs via workflows
# See: https://circleci.com/docs/2.0/configuration-reference/#workflows
workflows:
build-workflow:
jobs:
- build:
context: aws_context

Django/React - Azure App Service can't find static files

I've deployed my Django React app previously through a dedicated server and now I am trying to achieve the same with Azure Web App function so I can use CI/CD easier. I've configured my project as below but only my django appears to deploy as I get a '404 main.js and index.css not found'.
This makes me think there is an issue with my static file configuration but I'm unsure.
.yml file:
name: Build and deploy Python app to Azure Web App - test123
on:
push:
branches:
- main
workflow_dispatch:
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- name: npm install, build, and test
run: |
npm install
npm run build --if-present
working-directory: ./frontend
- name: Set up Python version
uses: actions/setup-python#v1
with:
python-version: '3.8'
- name: Create and start virtual environment
run: |
python -m venv venv
source venv/bin/activate
- name: Install dependencies
run: |
pip install -r requirements.txt
python manage.py collectstatic --noinput
- name: Zip artifact for deployment
run: zip pythonrelease.zip ./* -r
- name: Upload artifact for deployment job
uses: actions/upload-artifact#v2
with:
name: python-app
path: pythonrelease.zip
# Optional: Add step to run tests here (PyTest, Django test suites, etc.)
deploy:
runs-on: ubuntu-latest
needs: build
environment:
name: 'Production'
url: ${{ steps.deploy-to-webapp.outputs.webapp-url }}
steps:
- name: Download artifact from build job
uses: actions/download-artifact#v2
with:
name: python-app
path: .
- name: unzip artifact for deployment
run: unzip pythonrelease.zip
- name: 'Deploy to Azure Web App'
uses: azure/webapps-deploy#v2
id: deploy-to-webapp
with:
app-name: 'test123'
slot-name: 'Production'
publish-profile: ${{ secrets.secret}}
settings.py
STATIC_URL = '/static/'
STATIC_ROOT = 'staticfiles'
STATICFILES_DIRS = (
os.path.join(BASE_DIR, "static"),
)
Repo Structure:
Any advice would be greatly appreciated.
Cheers
To host static files in your web app, add the whitenoise package to requirements.txt and the configuration for it to settings.py. as mentioned here : Django Tips
requirements.txt | whitenoise==4.1.2

New line stripped from first line of Codebuild build spec when deploying with Serverless Framework

I am deploying a build spec for AWS codebuild using Serverless Framework. When I deploy, the new line after the first line is absent in the build spec. This resource previously deployed without a problem and I cannot see anything I have done to break it. Is this a problem on my end or a bug with Serverless/CloudFormation?
Below is the CloudFormation template and the resulting build spec copied from the AWS console.
Resources:
CodeBuild:
Type: 'AWS::CodeBuild::Project'
Properties:
Name: sls-retrobase-frontend-CodeBuild-${opt:stage}
ServiceRole: !GetAtt CodeBuildRole.Arn
Artifacts:
Type: CODEPIPELINE
Name: sls-retrobase
Environment:
Type: LINUX_CONTAINER
ComputeType: BUILD_GENERAL1_SMALL
Image: "aws/codebuild/amazonlinux2-x86_64-standard:3.0"
Source:
Type: CODEPIPELINE
BuildSpec: !Sub
- >
version: 0.2
phases:
pre_build:
commands:
- echo List directory files...
- ls
- echo Installing source NPM dependencies...
- npm install
build:
commands:
- echo List active directory...
- ls
- echo Inserting Api Url into config.json from environment
- node makeConfig.js ${apiUrl} ${auth0Audience} ${auth0Domain} ${auth0ClientId}
- echo Build started on `date`
- npm run build
post_build:
commands:
- echo List build directory...
- ls ./build
- aws s3 cp --recursive --acl public-read ./build s3://${Website}
artifacts:
files:
- '**/*'
- apiUrl: !Join ['', [ "https://", !Ref QueryRestApi, ".execute-api.us-east-1.amazonaws.com/${opt:stage}/sls-retrobase-${opt:stage}"]]
websiteUrl: !GetAtt Website.WebsiteURL
auth0Audience: '***'
auth0Domain: '***'
auth0ClientId: '***'
build spec:
version: 0.2 phases:
pre_build:
commands:
- echo List directory files...
- ls
- echo Installing source NPM dependencies...
- npm install
build:
commands:
- echo List active directory...
- ls
- echo Inserting Api Url into config.json from environment
- node makeConfig.js https://h0suk54yw0.execute-api.us-east-1.amazonaws.com/test/sls-retrobase-test https://sls-retrobase https://dev-y33gimcf.eu.auth0.com/ Xn6SDc43vE8P0sQHkVLtiBSBVFT5rJMU
- echo Build started on `date`
- npm run build
post_build:
commands:
- echo List build directory...
- ls ./build
- aws s3 cp --recursive --acl public-read ./build s3://serverless-retrobase-resources-test-website-7cvhqlgbkfj7
artifacts:
files:
- '**/*'
This is probably because of your use of >. Please change it to |:
BuildSpec: !Sub
- |
version: 0.2
phases:
Alternatively, fix your spaces when using >. > and your code are not aligned.

Google cloud - cloubduild and app variabile substitution in trigger

I'm using cloudbuild to deploy new version of my app when a new commit appears in github.
Everything is working good.
Now I'm trying to setup a variable substitution in the trigger configuration, because I want to put my version number in the trigger once, so that I can find the deployed correct version without modifying cloudbuild configuration file.
Variabile substitution works great in my cloudbuild file, for example:
(cloudbuild.yaml)
# TEST: PRINT VARIABLE IN LOG
- name: 'gcr.io/cloud-builders/gcloud'
entrypoint: 'bash'
args: ['-c', 'echo', '${_VERSION}']
# DEPLOY APP
- name: "gcr.io/cloud-builders/gcloud"
args: ["app", "deploy", "-v", "${_VERSION}", "app.yaml"]
dir: 'frontend'
timeout: "20m"
${_VERSION} is correctly replaced with the string I put into my trigger.
Now I want to obtain the same result in app.yaml file, substituting an env variabile, something like:
(app.yaml)
runtime: nodejs
env: flex
service: backend
env_variables:
VERSION: "${_VERSION}"
TEST_ENV: "read from google"
When I read TEST_ENV from my app, it works, but _VERSION is not replaced.
Any suggestion?
When you perform this step
# DEPLOY APP
- name: "gcr.io/cloud-builders/gcloud"
args: ["app", "deploy", "-v", "${_VERSION}", "app.yaml"]
dir: 'frontend'
timeout: "20m"
The app.yaml is provided as-is to the gcloud command, and it's not evaluated. You have to update it manually. Something like this
# REPLACE: PUT THE CORRECT VALUE IN APP.YAML FILE
- name: 'gcr.io/cloud-builders/gcloud'
entrypoint: 'bash'
args: ['-c', 'sed', "-i", "sed -i "s/\$${_VERSION}/${_VERSION}/g", 'app.yaml']
Of course if you let the
env_variables:
VERSION: "${_VERSION}"
as-is in your app.yaml file. You can change this replacement string
I want to add this solution in case someone has problems with the one proposed by giullade (in my case, cloudbuild gave me an error in executing the sed command).
I also changed my replacement string to one more readable and to avoid escaping the $ sign.
# Step 0: REPLACE variables in app.yaml file
- name: 'gcr.io/cloud-builders/gcloud'
entrypoint: 'bash'
dir: 'backend'
args:
- '-c'
- |
sed -i "s/__VERSION/${_VERSION}/g" app-staging.yaml
and in my app.yaml:
env_variables:
VERSION_ENV: "__VERSION"

Gradle Build Failed Error: package javax.servlet does not exist

I am pretty new to gradle and build systems,
I am trying to build project with gradle, but it cannot find packages of Tomcat server that I use in several classes of my project.
My build config:
apply plugin: 'java'
apply plugin: 'war'
repositories {
flatDir { dirs "WebContent/WEB-INF/lib" }
mavenCentral()
}
dependencies {
compile group: 'com.orientechnologies', name: 'orient-commons', version: '1.3.0'
compile group: 'com.orientechnologies', name: 'orientdb-client', version: '1.3.0'
compile group: 'com.orientechnologies', name: 'orientdb-core', version: '1.3.0'
compile group: 'com.orientechnologies', name: 'orientdb-graphdb', version: '1.3.0'
compile group: 'com.orientechnologies', name: 'orientdb-enterprise', version: '1.3.0'
compile group: 'com.tinkerpop.blueprints', name: 'blueprints-core', version: '2.3.0'
compile group: 'com.tinkerpop.blueprints', name: 'blueprints-orient-graph', version: '2.3.0'
compile group: 'com.tinkerpop', name: 'pipes', version: '2.3.0'
compile group: 'com.tinkerpop.gremlin', name: 'gremlin-java', version: '2.3.0'
compile group: 'com.tinkerpop.gremlin', name: 'gremlin-groovy', version: '2.3.0'
testCompile group: 'junit', name: 'junit', version: '4.+'
testCompile group: 'org.mockito', name: 'mockito-all', version: '1.9.5'
}
sourceSets {
main {
java {
srcDir 'src'
}
}
test {
java {
srcDir 'test'
}
}
}
task wrapper(type: Wrapper) {
gradleVersion = '1.3'
}
war {
from 'WebContent'
}
Errors occur, when I launch Gradle Task - Build:
OrientDBFilter.java:6: error: package javax.servlet does not exist
import javax.servlet.FilterChain;
OrientDBFilter.java:5: error: package javax.servlet does not exist
import javax.servlet.Filter;
....
Usually you would use providedCompile. Something like:
providedCompile group: 'javax.servlet', name: 'javax.servlet-api', version: '3.0.1'
Then your app will compile, but gradle won't include the servlet api in the final war file.
I had the same problem and the solution with providedCompile did not work, but simply
compile group: 'javax.servlet', name: 'javax.servlet-api', version: '3.0.1' did work.
I just wanted to share this if somebody has the same problem.
I had same problem and this line worked
compileOnly group: 'javax.servlet', name: 'javax.servlet-api', version: '4.0.1'