I have multiple different, independent AWS CodeBuild projects set up. All of them are using the aws/codebuild/amazonlinux2-x86_64-standard:3.0 image.
What I expect is for them to run concurrently, whenever a BitBucket webhook triggers them on PUSH. However, what happens is that one build from one build definition runs, whereas the others, belonging to different build definition projects are in QUEUED status, waiting for the first build to complete. Only then do they get the build environment provisioned.
I have also tried the ubuntu standard 5.0 runtime.
Is there a way to instruct CodeBuild to run different build projects in parallel, not waiting for each other to finish?
Related
Basically I would like to have a simple manual deploy step that's not directly linked to a build. For use cases, when using containers, I wouldn't like to perform a build separately per environment (eg: once my build puts an image tag in ECR, I would like to deploy that to any number of environments).
Now, I know in CodePipeline I can have a number of actions and I can precede them with manual approval.
The problem with that is that should I not want to perform the last manually approved deploy, subsequent executions will pile on - the pipeline execution doesn't complete and what comes next will just have to wait. I can set a timeout, for sure, but there are moments when 20 builds come in fast and I don't know which one of them I may want to deploy to which environment (they generally all go to some QA/staging, but some need to manually deployed to a particular dev-related environment or even to production).
Manually updating task definitions all around in ECS is tedious.
I have a solution where I can manually patch a task definition using awscli and yq but is there a way to have a simple pipeline with one step that takes a manual input (aka image tag) and either uses an ECS deploy step (the only place where you can provide a clean straight patch json to patch the task definition) or uses my yq script to deploy?
I'm trying to automate my builds using Google Cloud Build. For various reasons the primary use flow will be to manually kick off builds rather than using a Github trigger.
Right now it looks like when you run
gcloud builds submit .
it kicks off the build process but the gcloud command must stay running for the build to continue. If I Ctrl-C it then it stops the entire build process.
I could run it in the background but presumably if I get disconnected from the network or my laptop goes to sleep that will also interrupt the build process(?)
Clearly the build can run in the background since that's what a Github trigger does; is there any way I can do the same thing from the command line?
If you add --async to your gcloud builds ... command, the job will be run asynchronously, as a long-running operation.
You may query the state of this operation in order to determine the state of the build.
https://cloud.google.com/sdk/gcloud/reference/builds/submit
Alternatively, you may use something like Linux screen to keep any job running after you detach.
The Problem
When using, cloudbuild.yaml files, specifically named for their build environment, such as cloudbuild-dev.yaml and cloudbuild-prod.yaml, and configured/targeted in the Trigger settings they aren't found/recognized when GCB reacts to a GitHub event (push etc).
However, it's working just fine when manually running the Trigger from GCB console.
When using an ordinarily named cloudbuild.yaml, in the root of the project, Cloud Build correctly runs the expected steps.
The Workaround
In short, there isn't an easy one (imo). But to get it run you need to use just a single _cloudbuild.yaml).
However, to effectively re-use that for both dev and prod environments one is blocked by this issue
Is there are way to make a CodeBuild project execute build tasks one at a time (max concurrency = 1)?
I know one of the selling points of CodeBuild is that you can run builds concurrently and I like that feature.
However, for this one specific project, I NEED to make sure only one CodeBuild build task for this project executes at a time. If there is an incoming "startBuild" request while a previous request is still running, I want it to be queued and wait until the previous build task if finished.
As additional info on the project, this project runs integration tests across our various APIs (serverless APIs and legacy APIs on EC2) and running those tests concurrently may cause the tests to fail due to their setup and teardown procedures.
I am from the AWS CodeBuild team. Thanks for your feedback. At this point the feature you requested isn't supported. We'll pass along the feedback to our product management team so they may consider adding it to our future roadmap.
However, you maybe able to implement something at your end by using CodeBuild's Build notifications feature. At a high level, you could listen to the CloudWatch Events sent by CodeBuild to find when a build completes, and at that time "release" a new build from a queue that you maintain at your end.
Update (2021-02-17)
AWS CodeBuild just added the ability to define a maximum number of concurrent builds per project.
Reference: https://docs.aws.amazon.com/codebuild/latest/userguide/create-project-console.html#create-project-console-project-config
We are building several project with TeamCity. In addition to an agent on the main server which runs on Linux, we also have three additional agents that run on separate boxes. One on Linux, one on Mac and one on Windows.
If all agents are idle, the first available agent, as they are listed, is chosen for the build. This means that if the load on TeamCity is small, the same agent is always used. We have had situations where a project had been built successfully by the same Linux agent for more than 50 builds, then when it finally had a run on the Windows agent, a test failed, due to code that was committed fairly early in the run of the 50 builds.
Since many of our tests may be affected by the environment, we are looking for ways to spread the builds on the agents, automatically. Is there any way of setting up a round robin agent selection policy? Or any other way to spread the builds on the agents?
You can have a schedule trigger that runs a build on all agents.
Or configure a build for each platform (linux,windows, mac, ...) that will run on a specific agent, selected with some specific requirements.
The answer to my question, at least for TeamCity 8 and earlier, is NO.
See JetBrains own TeamCity Developer forum: https://devnet.jetbrains.com/message/5533629