Continuous Everything: Fast Feedback Driven Development

Fast feedback is a cornerstone of agile software development. When developing the LHOTSE project at Otto, we tried to be as agile as possible and many of our means and methods revolve around fast feedback. Here is a list of my favourite things we do to foster fast feedback. It does not at all cover everything we do in our daily work, let alone everything one possibly could do.

All methods have one thing in common: They try to let the development team know as early as possible when things are going into the wrong direction. The key hypothesis is: The sooner you recognize a mistake, the easier it is to fix it. If you introduce a bug in the software, it is easiest to fix it right away, when you still know what you where doing and when you can associate the bug with the small change you just did. When you learn about a bug later, you first have to identify the change that introduced it, then try to remember the intentions of that change. When adjusting the change, you have to be careful not to break anything that was built on top of it later.

Pair Programming: Instant human feedback

The possibly fastest way to get feedback is to not work alone. With a pairing partner at your side, everything you do is put into question immediately. Ideally, the pair discusses every line of code before it is written. So you get feedback even before writing a unit test.

It doesn’t really matter whether your pairing partner is a professional hacker or rather a newbie in the craft. The pairing partners questions will make you reflect on your strategy, hence you will feedback yourself. A beginner’s question might be much more helpful and eye opening than that of some experienced programmer already familiar with the project.

Of course it is perfectly reasonable to assume that two developers might walk in the same wrong direction together once in a while. This is why we try to promote pair rotation. One developer stays with a task or story while getting different feedback from different pairing partners.

You might enjoy Robert’s story about pair programming (in german).

Unit Tests: Simple fast feedback machines

„Test first. Test first. Test first.“ It’s a mantra. Repeat it, repeat it and then repeat it again. Once you got used to test driven development, you can’t imagine it has ever been different. For me it was different before I joined Otto. There are lots of developers out there for whom it still is different. In this very year of 2013 they are hacking their stuff cowboy style. They could know better.

A unit test is a feedback machine. You construct it before you build the real thing. Thus, you get feedback whether your production code is doing the right thing as soon as it is finished. After initial development the unit test stays a feedback machine. After every future code change it gives you timely feedback when you change prior behaviour of your software.

This gives you the possibility to find out if this is a wanted change or a malfunction. Because of the quick feedback you will know before you give the malfunctioning software to anybody else.

Like everything else in this list a dogmatic test first approach is debatable, too. As always you have to choose whatever means and methods suit you and your challenges best.

Unit tests have to execute really, really fast. Read on to learn why.

Behaviour Driven Development: The unit test of the product owner

What the Unit Test is to the developer, the BDD-test is to the product owner. A BDD style testing approach lets the product owner, assisted by the QA-manager, give feedback to the developers before development even started.


Artificial aardvarks are the contemporary mad scientist’s favourite doomsday device

This is done by a little trick: The product owner defines the desired change of behaviour of the software in the form of specifications that follow a simple predefined form: „Given … When … Then …“. The developers are able to implement automated tests for these formalized specifications. The simple examples shows how the method is helpful in the construction of artificial aardvarks:

Given an aardvark
When it eats 5 ants
Then it should be happy.

Those tests are implemented first and run by the developers during implementation. When the desired behaviour of the software is implemented, the developer recieves instant feedback in form of the BDD test which is not failing anymore. The feedback is two-way here. It is also the product owner who  gets feedback on whether or not the specification is precise, consistent and complete .

Just like every automated test, BDD tests become regression tests once a story is completed. They give instant feedback if the software is changed in a way that breaks former business requirements.

If you want to learn more about BDD, you can start out with this excellent article by Dan North that coined the term in 2006.

Git: the miracle of pull --rebase

There was a big fuss when John Lennon suggested the Beatles might be more popular than Jesus Christ. Here is my (slightly less) heretic statement of the day: Git is bigger than Linux. Git is the best thing Linus Torvalds ever made. And just like Linux he made it free software. Thanks Linus! (And while I’m at it: Thank you John!)

comparison of rebase, merge and subway

excerpts of
A) a clean history due to rebasing
B) a messy history due to merging
C) Hamburg’s public transport plan

Git makes branching a breeze, especially when compared to tools of the foretime like cvs, svn and stone axes. But however well branches are technically supported I still think they are evil. This is at least true for our situation: The developers of each team work closely together, sitting together in the same office space, each team working on one independent code repository.

In my team of at the moment 10 co-located developers we work together on one single branch. The pairs pull --rebase all the time and try to push their changes as early and often as possible. This gives fast feedback on diverging codebases and admits early detection of conflicting changes.

Those moments, when you unexpectedly realize that you have to merge a huge diff (and you just wanted to go home in a few minutes), are a pain of the past. Instead of merging the work of days in hours we merge the work of minutes in seconds. A nice side effect is, that you get a nifty linear commit history instead of something rather resembling Hamburg’s public transport plan. See the Image to get an impression.

Feature branches are commonly used to prevent halve baked code from being executed in the production environment. We use feature toggles instead to ensure that new code is only used in the production environment after being properly signed off.

PS: I am aware you can first branch and later rebase the branches on the master branch. That leaves you with the problem of late feedback on diverging code.

Continuous Integration: Bring them together

Continuous Integration is about bringing the software of collaborating teams and production-like data together in a production-like runtime environment. And it’s about doing it early.

In our setup every commit leads to a build in jenkins – our continuous integration server. All the unit and BDD tests are executed. If they pass, the software is deployed to our integration environment, where the CDC tests are executed. If no failure is detected, the software is ready to be deployed to the testing system where manual QA-steps are performed. The benefit here is obvious: The collaborating teams learn early whether their respective software versions become incompatible with each other.

Consider what was said before: We write a unit test for every bit we do. We write BDD tests for every changed behaviour of our software. We commit and push early and often. We build the software and run the suite of all tests for every push. Now it should be clear why having fast and efficient tests is so important. Consequently, we invest a significant share of our time in making the tests we have faster.

CDC Tests: Unit tests for collaborating systems

Imagine you collaborate with another development team and the software of your team offers an interface to theirs. Then your team can break functionality of the other software by making an incompatible change to that interface. The change can happen either by accident or on purpose. Your software goes live, theirs breaks. You’ll either have to hotfix one of the systems or roll back your changes.

In this common situation CDC tests become handy. CDC means „Consumer Driven Contract“. The consumer of an interface or service creates some automated tests to make sure the interface works as desired. This test is then executed not in the consumers, but in the providers build pipeline. Thus, the provider gets early feedback and is prevented from going live with a change that would otherwise break the consumers system.

In our setup the CDC tests are executed after the deployment to the integration system. We check out the code of the main consumer of our interfaces, the shopoffice system. We compile the code and execute the suite of CDC tests the shopoffice team wrote. If we break any expectations of the shopoffice, we get the feedback first and are able to fix it before the other team is bothered.

We try to avoid such accidents by communicating interface changes. We use json-home documents and mediatypes to document them properly. We try and design consumers of interfaces to be robust against changes. We do all that, but to err is human. And to err is not a problem. As long as the production system is not broken.

Continuous Delivery: The journey and the reward

If you have applied the things I mentioned in the previous sections, then you might be ready for the next step. Your confidence in the software you are developing is now so high that you think: „Why not push every single commit through to the production system?“.

That’s a question we actually ask ourselves. At the moment it is not yet really possible for us to go live with a frequency of maybe up to 50 commits a day, because we need some human testing and we have to do a minimum of coordination with the other teams. It still is an ideal we strive for. At the moment we are able to go live only a few times a day. I think that is already quite good, but we are working hard on improving.

Continuous Delivery is both: the goal and a way of reaching it. It is part of the journey and it is the reward.

Continuous Delivery is part of the journey because it obviously allows for an as-early-as-possible feedback from the real users of your software. How do your changes apply to real world data and to real world usage patterns? Deploying smaller changes means not only to get the feedback faster, but it is also more specific. Finding out exactly what change broke some functionality is much easier if you only deployed a handful of changes as opposed to deploying the work of a few week’s sprint in one big bang.

Continuous Delivery is the reward because it has several benefits:

  • Deployment becomes a non event. Releasing small changes becomes cheap and fast. The product owner’s wishes can hit the market nearly in realtime.
  • When you are quick in pushing forward changes to the production system, you are also quick pushing backward changes i.e. bugfixes. If we ever need to, we can push a bugfix to production in a matter off a few minutes. That is only because we constantly practice our deployment skills.

Read Guido’s (german) article on blue/green deployment to learn how we implemented zero downtime deployments, a requisite ingredient to Continuous Delivery.

That’s it

Last time I told you about something I strongly dislike: sprint commitment. An ugly concept – disruptive to any continuity. This time I told you about something I really love: Agile software development. Development with a steady flow of high quality output. It is all about fast feedback. It is about Continuous Everything. Now I’d like to hear your opinions again. What are the gems of your agile development process? What makes you really slick and quick. And what burden still hinders you and you’d like to get rid of it?

Christian is a software developer at Otto.

Tagged with: , , , , ,
Veröffentlicht in Development, Testing
6 comments on “Continuous Everything: Fast Feedback Driven Development
  1. hpkorn sagt:

    … und wo ist hier – technisch gesprochen – die Dämpfung?
    Denn feedback ohne Dämpfung führt bekanntlich zu einem immer stärken Aufschaukeln so lange, bis das System in Brüche geht („positive Rückkopplung“)….

    • Christian Stamm sagt:

      Hallo hp,

      das beschriebene Feedback ist zwar schnell, aber nicht notwendig positiv. Als Dämpfung könntest du also betrachten, dass das Team in der Entwicklungsarbeit innehält und zunächst einen Fehler beseitigt, über den es (negatives) Feedback erhalten hat. Ein Aufschaukeln kann ich mir da nicht vorstellen.

      • hpkorn sagt:

        Hallo Christian, diese von dir beschrieben „Dämpfung“ erscheint mir essentiell, wird aber allzu selten erwähnt. Statt dessen gibt es oft ein „von Sprint zu Sprint Gehetze“ mit einem immer grösseren Berg vor sich hergeschobener Berges „Schulden“.
        Also: Das bewusste Innehalten und selbstkritische Analysieren der Ergebnisse mit dem Anspruch, des Craftmanship wirklich Clean Code zu produzieren ist fundamental.

  2. johannes mainusch sagt:

    Hey Christian,
    ich fände es cool, hier einen screenshot von eurem Jenkins-Monitor mit der Build-Pipeline zu sehen (gern auch mal mit einem roten System). Und n kurzes Statement, was es für die anderen Teams bedeutet, wenn das Integrationssystem rot ist…

    • Christian Stamm sagt:

      Hi Johannes,

      über unsere Build-Pipeline haben wir in unserem Artikel für das OBJEKTspektrum einiges gesschrieben. Diesen gibt es mittlerweile kostenlos hier:

      In dem Artikel haben wir die Pipeline etwas vereinfacht und genau die Integrationsumgebung nicht dargestellt. Tatsächlich befindet sich diese vor der Testumgebung. Auf letzterer findet die eigentliche Arbeit (z.B. Abnahmetests) statt. Deshalb sind die Auswirkungen, wenn das System eines Teams in der Integrationsumgebung defekt (rot) ist relativ gering. Erst wenn wir die Testumgebung kaputt machen, halten wir andere wirklich von der Arbeit ab. Genau davor soll uns das vorherige Deployment in die Integrationsumgebung schützen.

      Nichtdestotrotz hat die Reparatur eines defekten Systems immer Priorität vor allen anderen Aufgaben. Siehe dazu auch den verlinkten Artikel.

Kommentar verfassen

Trage deine Daten unten ein oder klicke ein Icon um dich einzuloggen:

Du kommentierst mit Deinem Abmelden / Ändern )


Du kommentierst mit Deinem Twitter-Konto. Abmelden / Ändern )


Du kommentierst mit Deinem Facebook-Konto. Abmelden / Ändern )

Google+ Foto

Du kommentierst mit Deinem Google+-Konto. Abmelden / Ändern )

Verbinde mit %s

%d Bloggern gefällt das: