I have a repository on GitLab with a directory structure similar to this:
folder-a\
-python-a.py\
folder-b\
-python-b.py
I am trying to set up a CI/CD pipeline on gitlab that will detect changes made to the python code, and deploy them to a production server. What I have currently is the user have to trigger the pipeline manually, and input in the folder name as a variable, which will then cause the pipeline to "cd" into the folder and deploy the code inside the folder.
Is there any configuration or settings that can be added to the pipeline so whenever a Merge Request is merged to the main branch, the pipeline triggers and detects which code was changed, and then deploy the respective code without having the user to manually trigger it and inputting the folder name as a variable?
You might be able to use only:changes / except:changes to do that.
You can have two jobs. One job that goes to folder-a if something under folder-a/* has changed and the other job goes to folder-b if something under folder-b/* has changed.
Related
I am trying to work my way to have a ci/cd for the Api part of the application.
I have 3 steps:
1: Source (git hub version2)
2: Build (currently has no commands)
3: Deploy(provider is code deploy(application))
Here is the screenshot of the events in code deploy.
.
While creating the Deployment Group. I chose the option of downloading the code deploy provider from the option(though it was necessary).
While setting up the code pipeline chose
Felt that was appropriate.
This code pipeline has put an object into the S3 bucket for this pipeline.
Code deploy is acting on that source artifact.
Note:
We have nothing on this Ec2 image it's just a place where we have our API.
Currently, Ec2 is empty.
What would be the proper way to implement this? How can I overcome the issues I am facing.
Without appspec.yml your deployment will fail. From docs:
An AppSpec file must be a YAML-formatted file named appspec.yml and it must be placed in the root of the directory structure of an application's source code. Otherwise, deployments fail.
I am just starting to use AWS Amplify but can't figure out how you are supposed to commit the project to a source code repository so that others can work on the same project.
I created an react Serverless project 'web_app' and have created a few APIs and a simple front end application and now want to commit this to CodeCommit so it can be accessed by others.
Things get a bit confusing now because for the CI/CD it seems once should create a repository for the front end application - usually the source files are in the 'web_app/src' folder.
But Amplify seems to have already created a git repository at the 'web_app' folder level so am I supposed to create a CodeCommit repository and push the 'web_app' local repo to the remote repository and then separately create another repository for the front end in order to be able to use the CI/CD functions in AWS?
For some reason if I do try and push anything to AWS CodeCommit I always get an error 403.
OK - I'll answer this myself.
You just commit the entire project to a repo in CodeCommit. The project folder contains both the backend and the frontend code. The frontend code is usually in the /src folder and the backend code (CloudFormation files) is usually in the amplify folder.
Once you have the CodeCommit repo setup you can use the Amplify Console or the amplify-cli to create a new backend or frontend environment. Amplify is smart enough to know where to find the backend and frontend code.
Bear in mind that the backend amplify-cli code creates a bunch of files that are placed in the frontend folder (/src), including the graphql mutations and queries that will be used in the frontend code.
If you have set up CI/CD then any 'git push' will result in a new build for the environment you are in. You can modify the build script to include or exclude rebuilding the backend - I think by default it will rebuild the backend if there are changed.
You can also manually rebuild the backend by using the amplify-cli 'amplify push' command.
Take care because things can get out of sync and it seems old files can be left lying around that cause problems. Fortunately it doesn't take long to delete and rebuild and entire environment. Of course you may have to backup and reload your data first. Having some scripts to automatically load any seed data for development or testing is useful.
There is a lot of documentation out there but a lot of it seems to be quite confusing.
I am using AWS codepipeline as my CI/CD tool. I have a code pipeline template yml file on my git and I wonder how I can link the file to AWS codepipeline. What I want to do is to let codepipeline to create/update the pipeline based on my pipeline yml file in github.
I have searched and tried on AWS console. All I can do is to manually create a pipeline via console and upload the template file. It works but it is not pipeline as code. If I want to change the stages in the pipeline, I will have to manually update the pipeline on AWS console or via cloudformation command.
Let me give an example, if I need to add a new stage in my pipeline. What I'd like to do is to update the yml file in github repo and commit it, then AWS codepipeline reads this yml file to update itself. I don't want to manually update the stage via AWS console.
Is there a way for me to sync the codepipeline to my pipeline yml file under source control?
I have seen lot of people wondering about this setup where everything is managed via code and I personally use this too with CodePipeline. I can see many people have replied but let me put it here with detials so that it can be help to anyone who wants to do this.
There are two ways to achieve this and let me try to explain both option here:
Option:1
Create two Seperate Pipeline:
"Pipeline -1" (Responsible for config change like adding extra stages to main pipeline "Pipeline -2", with two stage source and deploy (CloudFormation)
source_Config (gitrepo_config) --> deploy_Config_Cfn
"Pipeline -2" (Actual deployment Pipeline with stages like source, buid, deploy stage which will be created by using resource.yaml)
source_Resource (gitrepo_resource) --> build_Resource --> Deploy_Resource
Based on above config upload the template you use to create the main pipeline "resource.yaml" to repo "gitrepo_config".
Upload all the Code in repo "gitrepo_resource" based on the deployment provide you are using for "Deploy_Resource"
Once above setup is done when you want to put extra stages in pipeline you can make changes in file "resource.yaml" in git repo and "Pipeline -1" will do the rest.
Option:2 (Little Complex But let me see if I can explain)
I was using option 1 until I came up with this option.
This second way is like 100% code because even in above option I have to create the "Pipeline -1" either manually or via CFN for first time and later for update also I need to go to console.
To overcome this we can include both Pipeline in same CloudFormation template "resource.yaml" and only one time we have to execute that CloudFormation stack and later everything else is automatic.
I hope this will be helpful to everyone.
Note: Also we have to keep in mind in both option if during any config change if pipeline execution is in progress for resource pipeline "Pipeline -2 " then it might be marked as failed so to overcome this issue you can always set additional trigger which will trigger the "Pipeline -2" based on success state of "Pipeline -1" in addition to the source code trigger.
The Problem
When using, cloudbuild.yaml files, specifically named for their build environment, such as cloudbuild-dev.yaml and cloudbuild-prod.yaml, and configured/targeted in the Trigger settings they aren't found/recognized when GCB reacts to a GitHub event (push etc).
However, it's working just fine when manually running the Trigger from GCB console.
When using an ordinarily named cloudbuild.yaml, in the root of the project, Cloud Build correctly runs the expected steps.
The Workaround
In short, there isn't an easy one (imo). But to get it run you need to use just a single _cloudbuild.yaml).
However, to effectively re-use that for both dev and prod environments one is blocked by this issue
When I trigger via Jenkins (code deploy plugin), I get the following error -
No such file or directory - /opt/codedeploy-agent/deployment-root/edbe4bd2-3999-4820-b782-42d8aceb18e6/d-8C01LCBMG/deployment-archive/appspec.yml
However, if I trigger deployment into the same deployment group via code deploy directly, and specify the same zip in S3 (obtained via Jenkins trigger), this step passes.
What does this mean, and how do I find a workaround to this? I am currently working on integrating a few things and so, will need to deploy via code deploy and via Jenkins simultaneously. I will run the code deploy triggered deployment when I will need to ensure that the smaller unit is functioning well.
Update
Just mentioning another point, in case it applies. I was previously using a different codedeploy "application" and "deployment group" on the same ec2 instances, and deplying using jenkins and code deploy directly as well. In order to fix some issue (not allowing to overwrite existing files due to failed deployments, allegedly), I had deleted everything inside the /opt/codedeploy-agent/deployment-root/<directory containing deployments> directory, trying to follow what was mentioned in this answer. However, note that I deleted only items inside that directory. Thereafter, I started getting this error appspec.yml not found in deployment archive. So, then I created a new application and deployment group and since then, I am working on it.
So, another point to consider is whether I should do some further cleanup, if the jenkins triggered deployment is somehow still affected by those deletions (even though it is referring to the new application and deployment group).
As part of its process, CodeDeploy needs to reference previous deployments for Redeployments and Deployment Rollbacks operations. These references are maintained outside of the deployment archive folders. If you delete these archives manually as you indicate, then a CodeDeploy install can get fatally corrupted: the references left to previous deployments are no longer correct or consistent, and deploys will fail.
The best thing at this point is to remove the old installation completely, and re-install. This will allow the code deploy agent to work correctly again.
I have learned the hard way not to remove/modify any of the CodeDeploy install folders or files manually. Even if you change apps or deployment groups, CodeDeploy will figure it out itself, without the need for any manual cleanup.
In order to do a deployment, the bundle needs to contain a appspec.yml file, and the file needs to be put at the top directory. Seems the error message is due to the host agent can't find the appspec.yml file.