A Continuous Integration (CI) server coupled with a comprehensive Build Process is a core foundation of agile development. Unfortunately the Build Process does not ensure that the repository is free of defects instead it can only inform the team whether the source can be trusted or not. The problem is that the analysis is triggered after code has been committed into the repository by which time the damage has already been done. At the commencement of a project while the code base is small and the number of automated tests is low it is trivial for the team members to run the build locally to ensure that each commit won’t break the build. However as the project grows and the execution of the build and tests gets longer and longer it is inevitable that the team starts skipping running a local build and instead uses the CI server to validate code changes. This poor development practice can lead to having the build broken for extremely long periods.
1. Accept trunk can be intermittently broken and tag source control each time the build is runs successfully
a. Easy for developers to grab the latest working copy
b. Easy to deploy the latest working copy into test environments
c. Build often will sit at RED for long periods until team does a “clean up” to get it GREEN for a deployment
2. Each developer has their own private repository and only after their private branch is validated for correctness by the Build Process are their changes merged into trunk
a. Only build process can check in to TRUNK
b. Build process only commits code changes if the private build is GREEN
c. Concept is built into modern CI tools such as TeamCity and Team Foundation Server
d. Developers that follow poor development practices are regularly blocked from getting their code into TRUNK
Classic CI Process
In a classic CI process the instant that a developer commits to a repository its status (Green or Red) is indeterminate and you need to wait for the build process to complete to determine whether the branch is a candidate to release to a test environment.
As projects grow in size the build time starts to increase and can blow out from minutes to hours if complex Acceptance Tests are developed.
To mitigate slow build times:
· CI Tools now are agent based allowing multiple parallel builds for the same project to occur
o With a large enough build pool ever single commit results in a parallel build and thus it possible to easily identify the code that “broke the build”
· Automated Tests are grouped and build project and split across categories such as fast component tests, sluggish integration tests and slow acceptance tests
o Team will accept Acceptance test project being RED for a period but will stop work immediately a build breaks at the Compile or Component Test projects
· Team stops checking into source control (code freeze) near the release so that changes to the repository and minimised and that a build remains completely green
Personal Build Process
With the advent of Distributed Revision Control Systems such as GIT and Mercurial and their superior merging capability many teams use the feature branch concept. In this development model code is branched enhanced and changes and pushed back into trunk extremely quickly. The ultimate destination of this strategy is the personal build concept. Each developer has their own branch and on check in the build process runs the entire build against their personal branch.
If the build process is successful that committed code is then automatically merged into trunk repository.
· significantly increases the quality of the code that resides in trunk
· Trunk can be treated as a Release Candidate branch
· Developers that have poor coding standards find it difficult to get their changes into Trunk
Why is Trunk Red?
Although Gated builds appear to be a nirvana at increasing the quality of trunk code the process can fall down when developers are committing in parallel. Two developers can write code that passes on their personal builds but when the changes are merged the build can fail and just like in the more common CI processes there is a period where the quality of trunk goes into an indeterminate state.
The solution to this problem is adding a layer between trunk and the personal builds through an Integration branch. After each personal branch succeeds the code is pushed to an Integration branch where the build is run again and its only after that branch succeeds does the code get pushed to TRUNK. Hence the Integration branch can go red but the trunk always remains clean.
The Gated build concept relies heavily on the ease of merging code between branches. It needs to be trivial to copy the code from Trunk into the Personal build and then be able to push those changes back again. GIT and Mercurial excel at merging as they have been designed for development teams with thousands of developers all working concurrently on multiple parallel branches.
The personal build concept is available from within Team Foundation Server and TeamCity.
Once a team invests in Gated builds the team have the freedom to use the build to penalise unwanted behaviour.
1. Personal builds that are RED for long periods can be automatically identified and flagged
a. Some teams will automatically delete a personal build that is broken for more than three days
b. Get the latest from trunk and start again
2. Integration build that is red can be flushed and re-branched off trunk
a. Allows team members that have better practices or that are closer to trunk to “try again” by kicking off their personal build and thus if it and integration succeeds their work can get promoted to trunk
b. Developers with poorer practices then get lumbered without the ability to get their code out
3. Certain Gates can be set to only run on the Personal Branch
a. If the Code Coverage drops fail build
b. If Cyclomatic Complexity increases fail the build
c. Reduces the build time for the Integration build
Negatives of the Approach
· Poor developers can have their local builds red for long periods which negates the benefits of continuous integration as code doesn’t get into the wild as quickly as it would in the SDLC without the Personal Build concept.
· Only required if trunk goes Red for long periods and on a team that has exceptional practices such as doing a full local build before check in it may not be necessary.
· Higher infrastructure costs as more build agents and branches are necessary.