Gitflow branching with Atlassian JIRA, Bitbucket and Bamboo with Octopus Deploy.

Gitflow branching with Atlassian JIRA, Bitbucket and Bamboo with Octopus Deploy doing branching UAT deployments.

Snappy title I know but it's basically an amalgamation of some of the more prevalent search terms I tried in getting to the final product.

Our Setup

If you take a look at the current Octopus Deploy set up and have a look at their branching strategies: Environment Branches. This is what we were doing, I walked into this, I'm not going to comment on it, it does work for us.

However, we had some old versions of the Atlassian products. (JIRA was on version 7 but Bitbucket was on version 4.1 and Bamboo was on version 5.10, to top that off Octopus Deploy was on version 2.6) So we needed to upgrade, the easiest and safest thing to do was start with a new server and reinstall everything. Upgrades where possible, but essentially new installations.

With this new software, I thought I'd take a look at the way we manage and release software. First off I looked at the different git branching techniques and GitFlow seemed to be the one best suited to our environment. This in turn totally changed the way we deployed our projects.

What we used to do was: Have a master branch and 3 permanent release branches. Sometimes if it was a big project we would branch off one of these release branches and the merge changes back into the release branch, then deploy the branch to a UAT environment for the business to test. This meant a large project could foul up a UAT and then we'd only have two left, another project and then we're down to one environment to do any hotfixes and no more UAT. At various points, other branches would be cross merged in order to update the other branches with the new code. Finally, one of these branches would be deployed to UAT then promoted to Live. This isn't exactly difficult to manage but it's not easy either, we have a whiteboard dedicated to where stuff is. One final thing to mention is that we would then update master from the release branch when the release had gone out and everyone was happy with it.

GitFlow works differently, first of all, I couldn't see any reference to UAT deployments, it seemed like once you were happy with it, merge it into master then push the project live. This certainly seems to be backed up by the way Octopus Deploy works. We still can't work like that, master has to reflect the live environment once we're happy, so the website can be quickly rolled back. I realise there are two other pieces of software that allow you to roll back, but it's the way we work here, after rollback though Octopus or Bamboo master would now be out of sync. 

So what we've done is set up Bamboo and Octopus Deploy to release from the feature branches out of the develop branch, that way we can test our website right out of the branch, not have to worry about blocking a release environment and keep our code as close to live as possible.

How we set that up

So first off we installed JIRA:

• Backup the setting in the admin console in the old JIRA admin console
• Install the new JIRA
• We use an SQL DB, so that needs setting up.
• Launch JIRA
• Restore the settings from the admin console in the new JIRA application.
• Despite what the instructions say the attachments do not get back upped and restored, copy those over, they are in the data directory in the home directory.
• The rest is just licensing.

Next, we installed Bitbucket:

• Clone the Home folder from the old BitBucket (or Stash) server
• Clone the old database, we backed up our SQL DB and restored to a new location
• In the new cloned Home folder:
• Edit the bitbucket.properties (Home folder\shared) add the line: "server.port=<port number>" replace the port number with the port for website access.
• Set the SQL to the new SQL DB
• Install Bitbucket, use the upgrade path and point it to the cloned Home folder.
• Set up BitBucket and ElasticSearch as services 
• The Config-template folder in the Atlassian Application folder needed renaming to just “config” to install ElasticSearch.
• To align the services with the start menu icons use:
• service install AtlassianBitbucket
• service install AtlassianBitbucketElasticsearch

Whenever you need to update Bitbucket, (if you are running two servers side by side) you can simply stop the service copy the Home FOlder\Data\repositories folder over, then restart the service.

Next, we installed Bamboo:

• In the old Bamboo goto the admin section and Export the settings.
• Create the new database
• Install the Java JDK, we installed JDK 1.8, JRE 7 seems to have come from somewhere.
• Install Bamboo
• Custom Install 
• Use the external MSSQL option
• Import the backup from step 1
• We had to set up an admin user at this point.
• Install NuGet

Finally Octopus Deploy:

This was a fresh install, there's no point upgrading from such an old version and there's nothing in Octopus Deploy that can't be easily reconfigured. Plus so much has changed. One thing I will mention; in our case we've had to test on a separate server, the live environment has it's own tentacle so I can't install the up to date one alongside and v2 and v2018 do not use the same tentacle.

Configuration

There's a lot to be configured in JIRA, BitBucket and Bamboo, and there are some good guides out there for that, most of it is self-explanatory, and I'll not go into that here. Needless to say, we configured the application links, security options, look and feel, mail, user access and I think all three applications have a self-test/diagnostic page to clean up any errors from the upgrades.

Project setup:

We had to slightly restructure our branches to meet with the GitFlow method, other than that only a few things changes.

JIRA was left alone, it's managing the links between projects and problems but we don't use it beyond tagging a branch with an issue so not much has changed.

BitBucket has the repositories as we changed them to the GitFlow method, nothing new here.

Bamboo changed a bit. Instead of having 4 different plans for 4 different permanent branches there is now one plan that uses the branching feature in Bamboo to build the other branches that we make. A very useful feature, and it suits us well. Basically, if you branch off a branch with a plan, that plan can run separately for the new branch.

Octopus Deploy should receive an award for the first run and smoothness of their application, it'll walk you through the procedure and guide right down the way it wants to do things. Which actually works for a wide range of setups. I think it's important to point out at this point, something Octopus Deploy wrote on their own website:

"Octopus doesn't care about branches. It cares about NuGet packages."

Application setup:

So this is the result of 2 weeks of trial and testing, researching, more testing, stress, late nights and arguments on the internet.

Again JIRA requires no extra configuration for this GitFlow setup.

BitBucket is already assuming you're going to use a GitFlow style setup anyway, but we have a few setting changes.

Branching Model:

and:

But this is very much down to the individual setup of each project.

Bamboo is where all the work happens:
I'm not sure where to begin with this or which direction to go in, I thought about trying to reproduce my steps, walk you through all the headaches and reasons, however that would take a long time and create a post with a lot of useless information. So I'm going to do this out of order, and I'll try and explain what I did and why when and it should all make sense at the end.

First off install the GitVersion addon:

GitVersion generates variables for build numbers in SemVer for use in Bamboo. This helps to name the branch plans and deployments for tracking through Octopus.

Yes, it's incompatible, but it works just fine, bar one insignificant issue I'll try to remember to mention later. Hopefully, Carolyn Van Slyck will get round to updating it soon.

Part of this install requires Chocolatey and the portable version of GitVersion you want version 4 which is currently in Beta, the reason for this is the stable build seems to randomly take minutes to run GitVersion, other times its seconds. This messes with Bamboo and causes it to fail the task.

From an admin console:

> @"%SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe" -NoProfile -InputFormat None -ExecutionPolicy Bypass -Command "iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))" && SET "PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin"

> choco install gitversion.portable --pre

If you want to install the current stable build knock off the "--pre" which works just fine on my development PC.

Install the Octopus Deploy Bamboo Plugin.

Expiry is the next thing to set up because we are going to get a lot of files from all the branches:

 90 days but retaining the latest 5, seemed good to me.

Setup a couple of Global Variables:

Name	Value
OctoPackPath	${bamboo.MSBuildPath}\_PublishedWebsites\<websitename>
MSBuildPath	${bamboo.build.working.directory}\Publish

Download the Octo.exe command line app, then set up an executable I called ours "Octo".

Under the Server Capabilities page:

Heres a biggy: You going to need the Visual Studio IDE for testing, otherwise you can get away with just MSBuild. Once installed, add an executable for the MSBuild.exe and the Visual Studio IDE, this last one is just the IDE folder:

C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE 

for example.

If you click Detect server capabilities it should find these for you. You should have GitVersion, a few MSBuild, Octo, and Visual Studio 2017

So let's go ahead and set up a build project. I'm going to cover the important stuff, you'll have to fill in some other details like name etc, but that's normal. I'm going to come to Stages last as it's the longest.

• Set up the repository, assuming you've set up the links between apps you should have the Bitbucket server and the repositories in it, select the Develop branch.
• Under Branches select

• Triggers should be the same as the parent plan, set up a delete plan branch if you want to save space.

Under Stages set up a new job:

Add a script task:

Script:

For /f "tokens=1-4 delims=/ " %%a in ('date /t') do (echo %%c-%%b-%%a %time% > buildTimeStamp.txt)

Add a source code checkout from your repository

Add a GitVersion step:

Add a script task like above but with the following script:

setlocal ENABLEDELAYEDEXPANSION
IF EXIST deployBuildTimeStamp.txt (
   set /p build=<deployBuildTimeStamp.txt
   git log --since "!build!">git.log
) ELSE (
   git log origin/Develop...HEAD > git.log
)

this creates a git log file based on the parent branch or the last time this branch was deployed. This took some time; if you know a better way I'd be happy to learn.

Add another script step

"C:\Program Files (x86)\NuGet\nuget.exe" restore "${bamboo.build.working.directory}\<solutionfile>"

this is to restore the packages in Nuget if you use them, you most likely do.

The next few steps are for Unit testing:

Add an MSBuild step

• Give it your project file name, don't need the path
• give it the options: "/target:"<path to test project>" /m"

Now your <path to test project> can be hard to figure out, it's not the file system, it's how it appears in the folder structure in the solution explorer. There is a C# script you can run in LinqPad to give you a list of possible targets. I recommend you run the :rebuild version. (You'll see what I mean if you run that script) Don't forget dots in the path must be replaced with an underscore

Add another script step

del "tr${bamboo.planName}_${bamboo.buildNumber}.trx"

This removes any previous test plan results which would otherwise crash the build plan at test stage.

Add an MSTest Runner step

• Use the Visual Studio 2017 executable
• Use the dll in the file system to locate the dll file from the unit test project for the Container: e.g. "Tests\bin\Debug\Tests.dll"
• Set the Result Filename to: "tr${bamboo.planName}_${bamboo.buildNumber}.trx"

Add an MSBuild step

• Set the Executable to the latest MSBuild
• Set the project file, again it's just the file name
• set the options to be: "/target:<main project path> /p:OutputPath=${bamboo.MSBuildPath} /m /p:Configuration=Release"

again this <main project path> is the internal target so use the C# script above to find that, if it's a large project you may want to forgo the rebuild.

We're done with the steps now. Switch to the Artifacts tab.

Setup two Artifacts:

1. Git Log - Copy pattern: git.log
2. Website Files - Location: Publish, Copy pattern: _PublishedWebsites/<Websitename>/**

Replace the <Websitename> with the actual name of the website, it's usually the project name, but you may want to run the plan once to find out.

You can, at this point, use the "+ Create plan branch" if you already have any branches from the develop branch.

Let's go ahead and now create a deployment plan. A lot of options are already pre-filled by Bamboo, so I'm going to cover what we changed.

Under Release Versioning set the version to: "${bamboo.GitVersion.SemVer}.${bamboo.buildNumber}"

Apply the same naming scheme to releases created from branches

but don't use the "last number in version". Generate a preview won't work, these variables are build related.

Add an Environment, this can be used to identify different deploy plans to automate multiple Octopus Deploy deployments.

Save that then go back and add or edit tasks

• It should have the Clean working directory task and the Artifact Download already in there
• Add a script task to update the deployment release date:
• copy ${bamboo.agentWorkingDirectory}\\${bamboo.planKey}-JOB1\\buildTimeStamp.txt ${bamboo.agentWorkingDirectory}\\${bamboo.planKey}-JOB1\\deployBuildTimeStamp.txt /Y
• JOB1 comes from the default job in the build stage if this doesn't come from the first job you'll need to change that
• The next few steps were drawn from the excellent Octopus Deploy documentation
• Add an Octopus Deploy: Pack Package step
• Octo should be you CLI
• Package ID needs to be Semver compliant, no spaces. Name.Web something like that
• Version Number: ${bamboo.GitVersion.SemVer}
• You won't be able to find that variable in the metadata from the build plan, that's what doesn't work in this version GitVersion on Bamboo, yes I remembered.
• Package base folder: _PublishedWebsites\<Websitename>
• Overwrite existing package
• Add an Octopus Deploy: Push Packages step
• Octo is you CLI
• URL and API key you'll have to get from your Octopus interface, see the documentation above.
• Package paths: "**/*${bamboo.GitVersion.SemVer}.zip"
• Force overwrite existing packages
• Add an Octopus Deploy: Create Release step
• Octo is your CLI
• Project comes from a project in Octopus Deploy UI, set one up if you need one
• Same URL and API key from before
• Release Number: "${bamboo.deploy.release}"
• Ignore existing releases
• Additional command line arguments: "--releasenotesfile=git.log  --packageVersion=${bamboo.deploy.release}"

I'm thinking that's it for now. If you create a branch from the Develop branch now, Bamboo should pick that up, and build it, then you can release it and it should then appear as a release in Octopus Deploy. This took me an afternoon to write up but over a fortnight to figure out, hopefully, it'll save you some hassle.