Microservices: CI with LambdaCD – Microservices and continuous integration with LambdaCD (2/3)

Abstract

In the last two months, we started our journey towards a new microservices architecture. Among other things, we found that our existing CD tools were not ready to scale with new requirements. So we tried a new approach, defining our pipelines in code using LambdaCD. In combination with a Mesos cluster we can deploy new applications after a few minutes to see how they fit into our architecture by running tests against existing services.

Part 1: The underlying infrastructure
Part 2: Microservices and continuous integration
Part 3: Current architecture and vision for the future

In this part of my article I want to explain how we define microservices and why we think they are the best choice for our applications. Furthermore I will give you a brief introduction outlining which problems we have with common CI tools and how we want to solve them with LambdaCD.

Microservices

A few years ago we split our monolithic application vertically in to smaller pieces. But over time these verticals grew to monoliths and we decided to split them again. The microservice architecture style is very popular at the moment and promises to solve many problems people have with monoliths.
If you start a new project, a monolithic approach seems to be a very natural way to do that because you only have a few requirements and they fit perfectly in to one application. But keep in mind that your application will continue to grow and you need a way to modularize the different parts to maintain a good overview. Monoliths tend to couple these parts very tightly because a programming language gives you the ability to do that, for example by disregading the existence of interfaces. But in a tightly coupled system it is hard to make assumptions about what will happen with the rest of the application if you change one part of it. Furthermore, if you want to test one part of your application you have to mock all the other ones to get your test running. So many times developers say the application is untestable because they don’t know how all parts interact with each other.
As your application grows, it becomes harder and harder for the team to fully understand the entire structure and you need more time to on-board new team-members. This process slows down development and you lose time to implement new features. Moreover, collaboration on a monolith doesn’t scale. Certainly, you can add developers to your team but the development will not become faster, because you have to coordinate all activities to make sure developers don’t get in each other’s way.
The microservice architecture helps us to get rid of the problems described above. If you split you application in smaller pieces you can create a system of loosely coupled components. You can change the implementation of one part without being afraid of problems in another part.
It is important to make sure that every microservice has only one responsibility so that it remains small and focused. This lets developers understand a service within a short time. So if you plan to add a new microservice you can scale your team by splitting it in to small groups and let each of them work on different services to speed up your development and make your changes independently.

There are more advantages you can read about on our blog: Scaling with Microservices and Vertical Decomposition and Gibt es etwas zwischen Microservices und monolithischen Architekturen? [german].

Problems with common CI tools

In recent years we used the common CI tool Jenkins to deploy our applications. It is great if you just have a few simple pipelines to support but over time our pipelines became more and more complex. Furthermore if you decide to introduce microservices, the number of pipelines grows very fast why automation is more important as it is in a monolithic world.
A clever template mechanism would be helpful to create new pipelines but you can only copy old pipelines to avoid starting from scratch. The problem is that all copies tend to diverge from the original pipeline. So we tried a second approach by creating disabled pipelines which are triggered and parameterized by activated pipelines which is very similar to the functionality of the Job Generator plugin. So you can reuse common steps but it is a tradeoff between automation and clarity. If want to figure out what a pipeline does you have to inspect different disabled pipelines. And if you also want to know which parameters are used and how, you have to find the corresponding scripts which are distributed over a few repositories.
To automate the creation of new pipeline you can also use a plugin like JobDSL. But if you use such a DSL you are always restricted by its completeness and you can’t add additional commands for things the author did not implement. Best of all would be if you could extend your CI tool by creating steps and pipelines in the same language used by the tool itself. So you have no restrictions and you can modify your tool according to your individual wants and needs.
Moreover we discover a requirement which is neglected by all common CI tools: Build and deployment are two of the most important tasks to guarantee a trouble-free continuous delivery. That’s why we want to run automated tests for our pipelines. If you use a DSL plugin you can try to parse your scripts to check that you have no typos or unreasonable parameters for your steps but this is not the same as running the step and comparing its output with your expectations.
Because of these problems our current CI solution does not fit in to the new world of microservices. So we started looking for an alternative which is more flexible and gives us a better control of how we create and organize pipelines.

LambdaCD

LambdaCD is an open-source project initiated by Florian Sellmayr. It is written in clojure and promises to let you define pipelines in code which empowers you to customize this CI tool completely. By using a programming language to define entire pipelines you are not restricted by the completeness of any DSL and you can add all the functionality you need in your steps.
It has many more advantages such as you can use your favorite IDE and SCM for writing and organizing your pipelines. Furthermore you can test your code with common tools and write your own library to reuse tasks in different pipelines. We do this to manage our deployment to Marathon at a central place.
To automate the creation of new pipelines we created a Leiningen template which contains a typical configuration. After specifying a project name and a Git repository we have a production-ready pipeline which can run immediately in our new Mesos cluster.
Because every pipeline is just a clojure project you can use LambdaCD to build and deploy it. You have read correctly: You can use a LambdaCD pipeline to build and deploy our LambdaCD pipelines! For this task every pipeline we create also has a second pipeline called the meta-pipeline. If you change the definition of a pipeline it is triggered by your commit and deploy itself.
LambdaCD is a highly configurable CI tool which solve the most our problems. Surely, it is a challenge to switch to a new CI tool but it fits better as common CI tools in our dynamic infrastrature.

LambdaCD Tutorial

To give you a better feeling what LambdaCD does let us set up a pipeline for a simple project.

  1. Requirements: Linux, Git, JDK, Leinigen
  2. Create a simple project: In this tutorial we use the compojure template which provides a configured ring server with a hello world webpage.
    lein new compojure simple-project

    Change your current directory to simple-project and start the application to see a hello world page in your browser (localhost:3000).

    cd simple-project/
    lein ring server-headless

    If you want to understand how compojure works look at the files project.clj, src/simple_project/handler.clj and test/simple_project/handler_test.clj

  3. Create you first LambdaCD pipeline: To create a pipeline you can although use a Leiningen template.
    lein new lambdacd mypipeline

    Open the project configuration (project.clj) and update the LambdaCD dependency to the current version if it is not already used by the template.

    vi mypipeline/project.clj
        :dependencies [[lambdacd "0.2.0"] -> :dependencies [[lambdacd "0.2.3"]

    Start your pipeline to become familiar with the web UI (localhost:8080)

    cd mypipeline
    lein run

    lambdacd You can see an example pipeline starting with a manual trigger. Click on the arrow sign to start the first step. It stops at the second manual trigger. lambdacd-1If you want to get more information about any step just click on it. Activate the second manual trigger to continue.lambdacd-2

  4. Modify the pipeline: Now we want to use the pipeline to build, test and deploy our simple project. Open the file src/mypipeline/pipeline.clj tomodify its structure.
    vi src/mypipeline/pipeline.clj
    [...]
    (def pipeline-def
      `(lambdacd.steps.manualtrigger/wait-for-manual-trigger
        some-step-that-does-nothing
        (in-parallel
          some-step-that-echos-foo
          some-step-that-echos-bar)
        lambdacd.steps.manualtrigger/wait-for-manual-trigger
        some-failing-step))
    [...]

    The pipeline definition has the same steps which you already know from the web UI. There a two kinds of steps: predefined (manual-trigger; in-parallel) and user steps. You can find the user steps in a separate namespace src/mypipeline/steps.clj. Before we have a closer look at the definition of steps let us write our own pipeline definition.

    (def pipeline-def
      `((either
          lambdacd.steps.manualtrigger/wait-for-manual-trigger
          wait-for-repo)
        (with-repo
          build-jar
          run-tests
          deploy-jar)))

    Our pipeline should run if we activate the manual trigger or commit some changes to the simple project. Then we clone the Git repository, build the project, run some tests and simulate the deployment. If you try to run the pipeline you will get many errors because we have to define the steps first.

    (def repo "~/simple-project")
    (def branch "master")
    
    (defn wait-for-repo [args ctx]
      (git/wait-with-details ctx repo branch))
    
    (defn ^{:display-type :container} with-repo [& steps]
      (git/with-git repo steps))
    
    (defn build-jar [{cwd :cwd :as args} ctx]
      (shell/bash ctx cwd "lein uberjar"))
    
    (defn run-tests [{cwd :cwd :as args} ctx]
      (shell/bash ctx cwd "lein test"))
    
    (defn deploy-jar [{cwd :cwd :as args} ctx]
      (shell/bash ctx cwd (str "cp " cwd "/target/*-standalone.jar ~/")))

    In the first two lines we only define symbols to store the location of the simple project Git repository. The defined steps are very simple and use the Git or bash module shipped with LambdaCD. Every step has two parameters: args (arguments) and ctx (context). „The first parameter args contains a map of key-value pairs that were returned by the previous build-step and the second parameter context contains lower-level details you’ll need if you are planning to implement your own control-flow operations and other lower level functionality“ (LambdaCD documentation). That’s all to define your pipeline and the corresponding steps. But at the moment you can’t run it because the simple project isn’t a Git repository. So let’s fix this.

    cd simple-project
    git init
    git add -A
    git commit -m "first commit"

    Now it’s time to start the pipeline (lein run). Activate the manual trigger for a first test.lambdacd-3You should see the file simple-project-0.1.0-SNAPSHOT-standalone.jar in your home directory. To prove the functionality of the pipeline let us change a test.

    cd simple-project
    vi test/simple_project/handler_test.clj
        (is (= (:body response) "Hello World")))) -> (is (= (:body response) "NOT Hello World"))))
    git add test/simple_project/handler_test.clj
    git commit -m "failing test"

    If your pipeline is still running, you don’t have to activate the manual trigger because we add a Git trigger as first step.lambdacd-4

  5. Explore LambdaCD: We created our first LambdaCD pipeline and now it’s up to you. Explore the remaining code and try to build your own project with LambdaCD. You can write tests for your steps or you can create a library for common tasks. You have so many possibilities because your pipeline is just code. Check out the LambdaCD documentation and Florian Sellmayr’s blog if you want to learn more about it.

Conclusion

The introduction of a microservice architecture changed our requirements on our continuous delivery processes. We are confronted with a large number of services and pipelines and we have to automate as much as possible. Common CI tools reach their limits and we have to find a future-proof alternative. LambdaCD has a completely different approach to build pipelines whereby it meets most of our requirements. In the next part I will tell you how we use LambdaCD in production, which issues we still have and what our plans are to get rid of them.

Hi, ich bin Simon Monecke, 25 jahre alt, angehender Software-Entwickler aus Hamburg. Als Student der FH Wedel absolviere ich ein sechsmonatiges Praktikum bei Otto, um dort meine Masterarbeit zu schreiben. Ich beschäftige mich dabei hauptsächlich mit den Themen "Mircoservices" und "Continuous Integration".

Tagged with: , , , ,
Veröffentlicht in Architektur, Development, Grundlagen
2 comments on “Microservices: CI with LambdaCD – Microservices and continuous integration with LambdaCD (2/3)
  1. Hauke Juhls sagt:

    Hi Simon, thanks for sharing this great archticle – the insight & your experiences are very valuable. Can you tell me for how long you’ve been using LambdaCD. Is it something that is in use at Otto for a long time (and thus has proven to be up to the job) or is it rather something Otto is evaluating to ease the pain you’re having with Jenkins?
    Regards, Hauke

    • Simon Monecke sagt:

      Hi Hauke,

      now we use LambdaCD for over one year. We started by migrating some small Jenkins pipelines to get a feeling how LCD works. After a few months we decided to use it for all projects in our team. Other teams, which also have problems with Jenkins, joined and now it is used in many projects. We created a template (only for internal projects, sorry) to deploy new pipelines within minutes only by answering a few questions (repo url, project name, marathon url, …). Additionally we wrote libraries with common LambdaCD steps to use these in all of our projects. You can find some of them on GitHub (Otto – GitHub
      We are very happy with our decision, because for us as software engineers we can fix pipeline problems with LambdaCD by ourselves without waiting for any Jenkins specialist 😉

      Regards,
      Simon

Kommentar verfassen

Trage deine Daten unten ein oder klicke ein Icon um dich einzuloggen:

WordPress.com-Logo

Du kommentierst mit Deinem WordPress.com-Konto. Abmelden / Ändern )

Twitter-Bild

Du kommentierst mit Deinem Twitter-Konto. Abmelden / Ändern )

Facebook-Foto

Du kommentierst mit Deinem Facebook-Konto. Abmelden / Ändern )

Google+ Foto

Du kommentierst mit Deinem Google+-Konto. Abmelden / Ändern )

Verbinde mit %s

%d Bloggern gefällt das: