I have a project with multiple modules say "Application A" and "Application B" modules (these are separate module with its own pom file but are not related to each other).
In the dev cycle, each of these modules have its own feature branch. Say,
Application A --- Master
\
- Feature 1
Application B --- Master
\
- Feature 1
Say Application A is independent and has its own release cycle/version.
Application B uses Application A as a jar. And is defined in its pom dependency.
Now, both teams are working on a feature branch say "Feature 1". What is the best way to setup Jenkins build such that, Build job for Application B is able to use the latest jar from "Feature 1" branch of Application A.
Given Feature 1 is not allowed to deploy its artifacts to maven repository.
Somehow I want the jar from Application A's Feature 1 branch to be supplied as the correct dependency for Application B?
Why wouldn't you allow feature1 to deploy to the maven repository?
If it's about not mixing unstable and stable code in the repo, how about setting up a second maven repository only for unstable code?
Should you not just fix that with a new version and creating snapshots? For your new feature you create a snapshot for the upcoming specific version. (e.g. 01.01.00.001-SNAPSHOT). The master still has a different version (e.g. 01.00.00.055) and will not be bothered with this new version snapshot. Though the feature branch on your other app can use version 01.01.00.001-SNAPSHOT as a dependency.
Related
I just decided to jump into using docker to test out building a microservice application using AWS fargate.
My question really relates to hearing about many development teams using Docker to avoid people saying the phrase "works on my machine" when committing code. Although I see the solution to that problem being solved, I still do not see how Docker images actually can be used in development environment.
The workflow for anything above production baffles me. Example of my thinking is...
team of 10 devs all use docker, each pull the image from the repo to there container, with the source code, if they all have a individual version of the image, that means any edits they make to that image is their own and when they push back to the repo where none of the edits can be merged (along with that to edit a image source code is not easily done as well).
I am thinking of it in the say way as git -GitHub, where code is pushed to a branch and then merged to master to create a finished product.
I guess if you pull the code from the GitHub master and create the Docker image is the way for it to be used, but again that points back to my original assumption of Docker being used for Production environments over development.
Is docker being used in development, more so the dev can just test the feature on the container that ever other dev on the team is using so all the environments match across the team?
I just really do not understand the workflow of development environments with docker.
I'd highlight three cases where I've found Docker particularly useful, prior to a production deploy:
Docker is really useful for installing local dependencies. If your application needs a database, docker run postgresql with appropriate options. Need a clean start? Delete the container. Running two microservices that need separate databases? Start two containers. The second microservice is maintained by another team? Run it in a container too.
Docker is useful for capturing the build environment in the CI system. Jenkins, for example, can run build steps inside a container, bind-mounting the current work tree in, so it's useful to build an image that just contains build-time dependencies (which can be updated independently of the CI system itself).
If you're running Docker in production, you can test the exact thing you're about to run. You're guaranteed the install environment will be the same in the QA and prod environments, because it's encapsulated inside the same Docker image. A developer can debug problems against the production-installed code without actually being in production.
In the basic scenario you describe, an important detail to note is that you never "edit an image"; you always docker build a new image from its Dockerfile and other source code. In compiled languages (C++, Go, Java, Rust, Haskell) the source code won't be in the image. Even if you're "using Docker in development" the actual source code will be in some other system (frequently Git), and typically you will have a CI system that builds "official" images from that source code.
Where I see Docker proposed for day-to-day development, it's either because the language ecosystem in use makes it hard to have multiple versions concurrently installed, or to avoid installing software on the host system. You need specific tooling support to "develop inside a container", and if developers choose their own IDE, this support is not universal. Conversely, in between OS package managers (APT, Homebrew) and interpreter version managers (rbenv, nvm) it's usually straightforward to install a couple of things on the host. If your application isn't that sensitive to, say, the specific version of Node, it's probably easier to use whichever version is already installed on your host than to try to insert Docker into the process.
currently our process works, but it takes too much time due that the fronend Ember app needs to be build into every single environment we have ( 5 environments ). because we never know which environment will be available when we release it.
we intend to add even more environments because every developer should have his own working development environment. (because of the backend)
how we do it, is that we create a frontend build and a backend build which creates artifacts.
now the frontent build takes around 2 minutes for every environment.
ember build --env=test and ember build --env=acceptance and ember build --env=development ... and more
when the artifacts are created we then create the release picking the correct ones depending on which environment we release (this done via release pipeline).
my question is can we make a frontend ember build somehow not depending on the environment?
i would like to note that we are using azure service fabric.
I don't think there is anyway around multiple Ember builds because each one will be different (i.e. production vs. development).
You can batch together each build inside one CI build/build task and produce artifact(s) to be used in your release pipeline.
Run the following command once for each environment you have (assuming you are using Ember-CLI) sequentially in one build task.
ember build --environment={{YOUR-ENV-HERE}} --output-path="dist/{{YOUR-ENV-HERE}}/"
You can then either upload the entire dist/ folder as an artifact and scope each environment in your release pipeline to the corresponding artifact subdirectory, or you can upload each folder inside /dist as an individual artifact and scope each environment in your release pipeline to its corresponding artifact.
only the configuration it changes. basically the api endpoints
We're using the hosted build agent on VSTS to build and release our ASP.NET Core code to Azure App service.
My question is: can we run WebPack to handle front-end tasks on this hosted build on VSTS or do we have to do it manually before checking the code into our repository?
Update:
I'm utilizing the new ASP.NET Core Build (Preview) template that's available on VSTS -- see below:
Here are the steps -- out of the box:
For VSTS we're working on an extension, currently it's in beta phase, you can ask for a share.
Check the VSTS marketplace.
Check this github repo.
Webpack is definitively not a first class citizen for VS2015 and VSTS. Streamlining webpack for CI/CD has been a real headache in my case, especially as webpack was introduced hastily to solve dreadful performance issues with a large monolithic SPA (ASP.NET 4.6, Kendo, 15,000 files, 2000 folders). To cut short, after trying many scenarios to make sure that freshly rebuilt bundles would end up in IIS and Azure webapp, I did a 2-pass build. The sequence of VSTS tasks is as follows: npm install global, npm install local, npm webpack install local, npm webpack install global, build pass 1, webpack, build pass 2, etc... This works with hosted and private agents, providing you supply the proper path for webpack as webpack is installed in a different location in host and in private (did not find a way to chose the webpack install location for consistency). I scorch everything before starting the build. Also need to do these in VS2015 solution : (1) unload "built" folder, and (2) Add Content Include="Built\StarStar" in project file. The "built" folder contains the bundles and should appear greyed, otherwise more bad surprises and instabilities to deal with...
Build-Pass #2 task in VSTS BUILD allows to collect the fresh bundles generated by Build-Pass #1 and includes them automatically in the package to be published.
Without a second build-pass, collecting the bundles and merging them in the zip package is a nightmare, especially when you have 15,000 files to unzip then rezip (300 ms per file!!). Did not find file-merging capability that I could readily use in VSTS.
I have my hears to the ground listening for someone coming up with a more efficient CI/CD scheme for webpack. In the meanwhile, my 2-pass-build workaround is working flawlessly, but slow indeed.
I anticipate that the advances with ASP.NET core, Angular 2 and webpack will look into solving this elegantly.
I have configured TeamCity with Git to get my ASP.NET MVC project.
My solution contains the web app and the corresponding unit tests:
MY_SOLUTION.sln:
- WebAppProject
- SomeCoreLibrary
- SomeCoreLibraryTests
- OtherProjects...
The steps that I have configured in TeamCity are the following:
Get external packages using NuGet
Build the solution and deploy it
Run Unit Tests
Run Automated Tests (using Selenium)
I want to run the unit tests after building but before deployment and stop deployment if the unit tests failed. Currently the deployment is done after the build using the following Command Line Parameters:
/p:VisualStudioVersion=11.0
/p:DeployOnBuild=true I want this to be done only after SomeCoreLibraryTests.dll unit tests have passed
/p:PublishProfile=MyWebDeploy
/P:AllowUntrustedCertificate=True
/P:UserName=username_here
/P:Password=password_here
Thanks,
Ionut
What I've done in similar cases is to use RoboCopy to just mirror the new website into the deployment path. Doesn't that work for you?
P.S.: if you do get this working, I'd suggest doing a performance improvement change in TeamCity (which would allow you to run the unit tests in parallel to the automated tests):
I assume you are employing a single build configuration for all those steps. If that is the case, what I would recommend instead is using Dependent Build configurations to separate the different concerns. You can see an example here in an open source project of mine:
http://teamcity.codebetter.com/viewLog.html?buildId=112432&buildTypeId=bt1075&tab=dependencies
Log in as Guest and expand the Testeroids :: Publish to NuGet tree node to visualize the build flow.
To achieve this, basically you pass around the result of your build step in the artifacts (e.g. you pass the resulting binaries from Compile into Unit Test). You gain several things by using dependent builds: several independent build steps can run in parallel on different agents, plus if one of your build steps fails because of external factors (e.g. let's say Publish failed because the network went down), you can trigger again the build and it will only rebuild the failed steps.
I am not familiar with the tools that you use. However, I would, in general, use a few build configurations for a project:
build configuration, triggered on change, containing these steps: get the latest source code and packages, build/compile and unit test. Then create an artifact for deployment task.
build configuration to deploy to a development server, triggered by successful completion of and using artifact (via dependency) from (1).
build configuration for long running (eg integration/functional) testing that is scheduled to run less frequently.
An advantage of (2) is that you can, if necessary, re-deploy a build/artifact without having to rebuild the artifact first. Also, if you have multiple agents, (2) and (3) can run independently of each other.
Furthermore, you can also tag build in (2) that have passed development checks and then use its artifact in another build configuration to deploy it to test server, etc.
Question: What is the best solution for executing a 'mvn deploy' such that the deploy part is only run after all unit tests succeed and no processing steps are duplicated?
I was hoping the simple answer was: Execute maven command 'x' (or use a flag) such that the deploy can be run without invoking the prior goals in the default lifecycle.
Sadly this does not appear to have a simple answer. I have included the details on the path I have followed so far below.
We have the following three requirements:
Execute the maven deploy goal to deploy all multi-module artifacts to a remote repository.
Only deploy if ALL unit tests across all projects pass.
Do not repeat any processing.
We started with simply "mvn clean deploy", however we noticed a couple issues:
the build would stop before completing all unit tests :: so we added the --fail-at-end flag
The deploy goal would execute against any modules that were successful.
This results in a "corrupted" state where the remote repository may only has a partial deployment (if there were modules with failures later in the build).
We looked at 3 different solutions:
Staging the artifacts prior to deploying :: this was determined to be too heavy for a fully automated process.
Use a profile to override the default lifecycle such that 'mvn deploy -Pci-deploy' would run without invoking any prior goals :: this worked and was fast, but is obviously an unconventional approach.
Simply running 'mvn clean package' and then only iff successful execute 'mvn deploy' :: this appears to work and seems to only take a minor hit when the goals are invoked (though some of them are smart enough not to reprocess an unchanged workspace)
I pose this question to the community with the background details I have provided to determine if there is a better approach or a strong opinion regarding (potentially) making one of the following requests:
A new deploy goal that can run separate and apart from all other lifecycle goals with the expectation that: all prior steps have already been run and that it will execute the deploy identically to "mvn deploy"
a flag in the deploy goal which would effectively disable the previous goals.
a little more out of the box and definitely against the current convention:
a flag that would tell maven to run the [unit] test goal for all modules prior to proceeding.
Notes:
We are using Jenkins, but for the purposes of this question the CI environment is not the complication.
I tried the 'mvn deploy:deploy' goal, but it had a number of unclear errors.
I have not considered integration tests as part of the requirements.
Update 8/20/2013
I tested the deferred deploy plugin and determined that the tool worked as expected, but took way to long.
For our code base:
mvn clean deploy: for all goals executed in 2:44
mvn clean install 'deferred-deploy-plugin': for all goals executed in 15 min
mvn clean package; mvn deploy -Pci-deploy a custom build profile that disables the earlier goals executed:
for all goals (including deploy): 4:30
deploy only: 1:45
mvn clean package; mvn deploy -Dmaven.test.skip=true on the same workspace executed:
for all goals (including deploy): 4:40
deploy only: 1:54
The clean package followed by deploy skipping the tests runs faster than the deferred deploy and accomplished our desire to delay the deploy until after the tests succeed.
There appears to be a minor time hit for when the deploy lifecycle executes and exits each of the preceding goals (process, compile, test, package, etc). However the only alternative is to hack a non-standard execution, which only saves 10 seconds.
There's a new answer now. Since version 2.8 of the maven deploy plugin there's a way to do this "natively". See the jira issue for details.
Basically you need to force at least v2.8 of the plugin
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-deploy-plugin</artifactId>
<version>2.8</version>
</plugin>
and use the new parameter deployAtEnd. more info here. This setting usually goes along with installAtEnd of the maven-install-plugin
As an alternative, I also found this
http://code.google.com/p/maven-deferred-deploy-plugin/
A maven plugin that iterates through all projects in a reactor and
executes a deploy on each project individually. Can be used to produce
a near-atomic build for a reactor by deferring artifact deployment
until the install phase has completed.
Sounds alot like what you were asking for. I still think my other answer is easier to implement since you use jenkins, just check a checkbox
Two things.
Disabling all the previous phases i don't see it as an option. It is a basic feature of maven, you would be altering the standard lifecycle so i highly doubt anyone would implement something in a plugin to allow this
Since you said you use Jenkins, there is a setting in jenkins specifically for the case of deploying at the end to guarantee that the repo is not in a corrupt/intermediate state
In "Post-build actions"
Deploy artifacts to a Maven repository. In comparison with the
standard mvn deploy, this feature allows you to deploy artifacts after
the entire build is confirmed to be successful.
This prevents a typical problem in Maven, where some modules are deployed before a critical failure is discovered later down the road,
rendering the repository state inconsistent.
Note that regardless of this configuration, you can always manually come back to Jenkins and deploy any of the past artifacts to
any repository of your choice, after the fact.
To use this feature you shouldn't deactivate the automatic artifact archiving.
I have never used this so i can't confirm whether it works, I just know it's there for this particular use-case