So, I decided to actually setup the continuous integration pipeline outlined in the article: A Pragmatic Workflow – http://www.praqma.com/stories/a-pragmatic-workflow/ you might actually want to read that first, because then you’ll have an idea of what’s going on.

This article has pretty much the exact steps I went through to set the workflow up.

Do bear in mind, like a professional tv-chef, I already had Git and Git Bash installed and a GitHub account I could push and pull from. I don’t think this ruins the illusion though, great guides exist on how to do either.

Also, there’s a bit of Q&A during the article, discussing some cool internals. You can skip these if you just need the setup-part.

What we want to set up, (from the blogpost)

Git as the Version Control System (VCS)
GitHub as the Distributed Version Control System (DVCS), (alternatives include bitbucket, etc.)
Waffle.io as the issue tracker (alternatives include Jira, etc.)
Docker as the application manager, because pulling a container and running it, is like installing a piece of software that actually works out of the box.
Jenkins as the Continuous Integration platform (alternatives include TravisCI or CircleCI).
- Jenkins plugins, Git, GitHub and Pretested Integration
GHI as the bridge between GitHub Issues online and our local Git instance.

Why all of these?! Because they work, they’re pretty much free and it was the ones they selected in the article. This isn’t an argument about technologies, it’s a step-by-step guide on how to set up a successful pipeline.

Is that all? No.

DigitalOcean as the cloud computing provider (alternatives include, Amazon AWS, Kubernetes, Google Cloud Platform).

They didn’t suggest one in the article, but obviously we need a server to have Jenkins running on (maybe even one for deployment?). I’m just using DigitalOcean as I had standing credits and I already knew how to use it, so for a proof of concept it was excellent.

Building the pipeline

You can of course do the points below in any order you like, you can actually do anything completely as you like.. but since you’ve of course, already read their blogpost, my order doesn’t have to make sense from an understanding point of view, and I can present them in a more dependency-oriented order.

Install Git and Git Bash locally

Download and install from here: https://git-scm.com/downloads

Create an account on GitHub

That’s here: https://github.com

Create a repository on GitHub

The repo can be created from here, https://github.com/new

Install GitHub Issues (GHI) on your local system

On the host. In a browser,

Go to http://rubyinstaller.org/downloads/ and install some suitable version of Ruby.

On the host, in git bash

$ gem install ghi # to install GitHub Issues shell interfaceIf this fails with something like,”ERROR: Error installing ghi:
The ‘yajl-ruby’ native gem requires installed build tools. Please update your PATH to include build tools or download the DevKit
from ‘http://rubyinstaller.org/downloads’ and follow the instructions
at ‘http://github.com/oneclick/rubyinstaller/wiki/Development-Kit’
1. “Go to the rubyinstaller site (again), and
2. install the DEVELOPMENT KIT corresponding to your ruby installation.
  “Quick start: Download it, run it to extract it somewhere (permanent). Then cd to it, run ruby dk.rb init and ruby dk.rb install to bind it to ruby installations in your path.” – https://github.com/oneclick/rubyinstaller/wiki/Development-Kit
  See: http://stackoverflow.com/questions/18908708/installing-ruby-gem-in-windows
$ ghi config --auth github_username # login to GitHub

Locate the folder where you want the project locally

Open Git Bash and git init a git repo in your favourite project-folder

Setup the labels we will use with our repository

It’s easiest to do this from the command line inside the folder of a repository linked to a GitHub repository.

Go to the blogpost and copy the snippet to delete default labels, fire this in Git Bash.
1. Check with
  $ ghi label --list
  That all the default labels have in fact been deleted, otherwise do
  $ ghi label -D "missing label"
Go to the blogpost and copy the block with their custom labels and fire them in Git Bash.

Add a Waffle.IO board to your repo

Locate the URL of your repository, something like
https://github.com/user/repo
https://github.com/figaw/probable-octo-robot
Go to the url
https://waffle.io/user/repo
https://waffle.io/figaw/probable-octo-robot
Click the “Turn into waffle.io project. Let’s go!” button.

Add the “Up Next” column and set up the “system – .. ” GHI-labels on Waffle.IO

Go to your Waffle.IO board, click the cog and select Customize Board Columns.
Click Add Column
1. Call it “Up Next”
2. Set “Feeding the work in progress with prioritized work” as the description
Set the “System – ..” GHI-labels to their corresponding columns
1. The “Ready” column gets “Status – workable”
2. The “Up Next” column gets “Status – up next”
3. The “In Progress” column gets “Status – in progress”
Done.

Q. Wait, why did we do this?
A. After adding the alias’ in the next step. When we use them, they add/remove the system-labels from the issues, and Waffle.io will automatically move issues around on the board.

Q. Okay, so I’ve looked at the alias’, what are the Ready, Up Next and Done columns for? Nothing’s moved there by the alias’?
A. The Ready and Up Next columns, are for planning. We move stuff there manually to show what’s about to happen; agile-planning. The Done column is used* by the Pretested Integration plugin to move an issue to the done column.

*This is not actually true, but when the Pretested Integration plugin merges a branch into the master, it closes the commit with a message similar to “close #123 issue-name”, and Waffle.io picks this up from GitHub, and closes the issue with id “123”, i.e. moves it to the Done column.

Add the Git alias’ from the blogpost

Go to the blogpost and copy the alias’ and add them to your .gitconfig file.
NB: You might not want to add it to the –global file but only the project .gitconfig file. The Alias’ will work, but the labels will obviously only be thrown around in a GitHub project that actually knows these labels.

Online-automation

Okay, so now we have the pipeline set up locally, and we have a project on GitHub, and we can start working on it with the pragmatic workflow. Now we set up the “online”-automation parts.

Jenkins on DigitalOcean

I’m doing this on a Windows host. I have Git (bash) installed, so I can use that as a terminal.

On DigitalOcean.com

Create a droplet, and
attach ssh-key # generate one if you don’t have it.

SSH into the droplet

On host, in Git Bash.

$ eval `ssh-agent`
$ ssh-add <path_to id_rsa>
input passphrase
SSH to droplet ip in favourite ssh-client (Git Bash), with
ssh root@ip-to-droplet
– boom, we’re in.

Also, seriously, check out this article on how to autostart the ssh-agent in git bash, it’d probably save me a small hour of keystrokes if I’d known about it a year ago. Working with SSH key passphrases @GitHub and perhaps this one, How I Distributed my .bashrc on Windows.

On Droplet, install/run Jenkins

On the droplet (… you JUST ssh’ed into..), run commands

$ apt-get docker.io # install docker
$ mkdir /var/jenkins # make a directory for saving jenkins data
$ chown 1000 /var/jenkins
# (Depending on where you put the folder, you might need to..)
# give the jenkins-user (uid 1000) rights to the folder.
$ docker run -d --name myjenkins -p 8080:8080 -p 50000:50000 -v /var/jenkins:/var/jenkins_home jenkins:alpine
– Run jenkins container
– launch it detached,
– name it myjenkins, # for reference
– forward hostports 8080 and 50000 to containerports,
– mount a volume folder for persistent storage, and
– point to the jenkins:alpine tag # alpine is more lightweight..
see: https://hub.docker.com/_/jenkins/
$ docker logs myjenkins # see the output of the container
because we need to know the admin password in order to configure Jenkins in a browser. (If you can’t see a password, you might’ve been too quick with the command, try the --follow flag of logs to see the output streamed to your console,
$ docker logs --follow myjenkins # stream output to terminal [ctrl + c] # stop the streaming.

Configure Jenkins in a browser

On the host. In a browser,

go to droplet-ip:8080 # the port we chose before
input the admin password from previous section,
click Install with suggested plugins,
create a user,
click Continue as admin and Start using Jenkins,

Install the plugins

Go to Manage Jenkins -> Manage Plugins
Install plugins:
1. Pretested Integration Plugin
  Git branch, */ready/**, auto-integrate-magic
2. Job-DSL plugin
  Configure-as-code for Jenkins jobs.
  I will not go into details about this, in this blogpost, but you can study it here http://www.praqma.com/stories/#Job DSL
3. GIT plugin # .. Is installed by default. Move on.
4. Locale Plugin # Optional! – allows you to change the language in the UI: it defaults to the browser-language, mine was Danish (.. right?)
  1. After install there’s a language setting under “Configure System” (which will obviously be called something else if it’s not in english..)
  2. In the field, write en or en_US for english.
  3. Check the field “Ignore browser preference and force this language to all users” and save the changes.
5. Restart Jenkins after install. # for good luck..
6. “DON’T PANIC” – The Hitchikers’ Guide to the Galaxy.
  It might take a while; the plugins are a bit heavy on the side.

Configure the pretested integration plugin

There’s actually not much to do here, but because it’ll pretend to do the automated merging for you, you need to give it a git-username and git-email, or it’ll encounter errors when it tries to merge multiple commits into a master branch.

Go to Manage Jenkins
Click Configure System
Scroll down to Git plugin and set the user.name and user.email
1. user.name = Jenkins, and
2. user.email = jenkins@localhost will work fine. You just need “something” there.
3. Save the changes.

Set up A job

You’re probably on the dashboard now after clicking save in the last step, and you probably don’t have any jobs, so this is pretty easy.

click Create new jobs,
give it a name “Awesome Job”,
select Freestyle Project,
click OK.

Under Source Code Management,

Select Git
Paste the url to your repo in Repository URL
– If it’s a private repository, you’ll see an error, that’s fine.
Add your credentials (it doesn’t matter if it’s a public repository that you can easily pull from without credentials.. You need the credentials so the Pretested Integration plugin can make pushes/merges
– If you’ve added them successfully, (and it’s a private repo..) you’ll see the error from before disappear.
1. Depending on your level of NSA, you might want something other than user/password authentication, i.e. RSA. For me, this is more of a proof of concept, so that level of security is fine.
In the Branch Specifier field, you want to put
*/ready/** which means Jenkins will pay attention to branches like origin/ready/super-cool-issue-name#123.

Here’s a picture of it,

Under Build Triggers

Check, Build when a change is pushed to GitHub. Because you’ve marked */ready/** as your branch specifier, Jenkins will look for changes/new branches, matching this pattern.

Under Build Environment

Check Use pretested integration
Set master as your Integration branch (… if this is the branch you want successful pushes to a ready-branch to be merged into.)
Origin is the default Repository name, if you don’t know what you should change it to, you “probably” don’t need to change it.
Use Squashed commit’s, unless you really want an explicit log with –no-ff commits all over. You don’t. You want to use Squashed commits.

Under Build

You probably want to add some form of build step. I have no idea what you’re building, but if you have a bunch of code, and it’s tested by running a script called ./test.sh, you probably want to add an Execute Shell step, and invoke this command.

If the script succeeds (exit with exit code 0, zero), the post-build actions, are executed. If not, it’s just stated in the console-logs for the job what went wrong, and you can go it over to debug it. I wont go into a lot of detail about this, but you probably want something like, “when a build fails, send an email to the person that commited the ready-branch, with the output of the console-log”.

Under Post-build Actions

Click Add post-build action
Select Pretested Integration Publisher

Now, when a build succeeds, the Pretested Integration plugin will merge your */ready/** branch, into the master and delete it.

Finally Save your job.

“Are we there, yet??” – Donkey, Shrek (2001)

No. We need one tiny step more.

Set the hook for posting repository changes to Jenkins

Jenkins is set to run our Awesome Job whenever a change is made to our repository, which means we need to alert Jenkins about this,

go to your GitHub repository,
go to Settings,
go to Webhooks & Services.

or (since we’re so happy about rewriting the repo-url’s)

Go to https://github.com/user/repo/settings/hooks
e.g. https://github.com/figaw/probable-octo-robot/settings/hooks

Click Add service

Either add the Jenkins (GitHub plugin) or the Jenkins (Git plugin) service, depending on what’s installed on your Jenkins server.

It seems that the GitHub plugin is installed by default as a recommended plugin, so I just use this, but by all means, BOTH ARE DOCUMENTED ON GITHUB WHEN YOU’RE TRYING TO SET IT UP. I,

Added the Jenkins (GitHub plugin)
Pasted the url:port to my Jenkins server and the webhook, e.g.
http://ip:port/github-webhook/
(if you’ve followed the guide, your port is probably 8080.)
NB: the “http://” part is important!
Clicked Update Service.

Now, the next time you make changes to this repository, Jenkins will check whether it was on a */ready/** branch. If that is true, Jenkins fires the job, and if it succeeds, the Pretested Integration plugin merges the branch into master and deletes it afterwards.

So what now?

Well, you still don’t have any continuous deployment, because you might want to push certain releases, when you’ve gathered a few ready-branches on the master. But there’s nothing stopping you from setting up,

A Jenkins job that
1. looks for changes to the master branch
2. Runs a build step that
  1. SSH’s into a remote server,
  2. Pull’s from the master branch
  3. Restart‘s your service with the updated code.
3. Performs a post-build action with sending you an email, does a blogpost, fires a twitter message saying “Version x.y is out!” etc.

This was quite a mouthful..

Well, yes. But, now your ENTIRE DEVELOPMENT CYCLE has become:

$ ghi # list issues on GitHub
$ git work-on 123 # start working on issue 123
$ git wrapup # end work on issue 123
$ git deliver # push work done
Get coffee, and wait for a message about the build status
1. If it suceeded
  1. $ git purge-all-delivered
    # removes delivered/** branches and checks out to master branch
2. If it failed
  1. With the alias I give in the Q&A below,
    Reproduced here for completeness. Git Fix alias to rename a delivered/ branch,
    fix = "!f() { NEWNAME=`git rev-parse --abbrev-ref HEAD | sed 's/^delivered\\///g'`; git branch -m $NEWNAME; }; f"
    1. $ git fix
    2. Make changes
    3. $ git wrapup
    4. $ git deliver
  2. Without the alias
    1. Make some changes
    2. $ git wrapup
    3. $ git deliver
    4. Remove the broken ready/branch-name on the origin

Done

Final thoughts. It is a small tonne of setup, but once it’s done, continuous integration/delivery becomes a breeze. – even as a team, because you literally only need to install Git, GHI, the alias’ and pull the project, in order to start working.

Extra credit: put all of the above in a Dockerfile along with your favourite IDE, and now you only need Docker!

As always, if you find typos, have thoughts or want to discuss something, drop a comment below!

F.A.Q.

Q. Don’t I have to pull the changes from origin master before it checks out a new issue-branch and I start working on it?
A. Nope. The work-on alias fetch’es the origin before it checks out to the newly created branch

Q. The second time I write $ git deliver on a branch, it’s name suddently becomes delivered/delivered/branch-name and it’s pushed to ready/delivered/branch-name?
A. Yes, when you begin work on a branch that’s been delivered, you want to change the name back, let’s call it wanting to “fix” a broken deliver. I’ve added the alias git fix:
fix = "!f() { NEWNAME=`git rev-parse --abbrev-ref HEAD | sed 's/^delivered\\///g'`; git branch -m $NEWNAME; }; f"(don’t get fooled by wordpress btw, it’s a single line!)
what it does, is
1. Get the name of your current branch without the “delivered/” part,
2. rename your current branch to what’s left.
Which means “delivered/issue123” and “issue123” both becomes just “issue123”. So the second time you deliver, changes are still pushed to “ready/issue123” and the branch is renamed to “delivered/issue123”.

Q. Why have the delivered/ renaming in the first place?
A. Because, then you can see what’s on the master, and what’s purely local.

Q. Couldn’t I just pull and push to the ready/branch-name on the master?
A. Sure, you can do whatever you want.. but you want to use these alias’ right?.. wrapup, deliver, work-on, and not “git pull origin ready/…”, “git add ..”, “git commit ..” and finally “git push origin ready/…”.

From Zero to A Pragmatic Workflow or How to setup a Continuous Integration Pipeline with Git, Github and -Issues, an automated Waffle.io board and Jenkins CI in Docker on a DigitalOcean Droplet.