July 26, 2020

Why do we need monorepositories in the first place?

This article does a great job in explaining what a monorepo is and why you may need it: What is a monorepo?

To summarize it you have a couple of advantages:

Single source of truth
Pull-Requests accross services are possible and reviewable in one pull request
Simplified code sharing accross services
Common build logic for all services

There are also some downsides for using a monorepository for example vsc scalability, cross team ownership or the temptation to introduce tight coupling between services.

Why did I choose to organize my code in a monorepository?

The main arguments for me where that I can consolidate my build logic for gradle in one gradle project and don’t have to share the code through plugins.

Secondary thaughts are, that we could consolidate the documentation in one repository and that several repositories in bitbucket are hard to keep track of.

What is required to get my monorepository up and running?

The requirements are comparable minimal:

Git (I used 2.19.1)
Gradle (I used 6.3)
Kubernetes (I used 1.16.8-eks-e16311) if you want to deploy your services
Helm (I used 3.2.1) for templating in kubernetes
Multiple services owned by the same team (monorepositories are hard when the cross team boundaries)

Initial Setup

We will start with a bare Git repository with a Gradle project using Kotlin DSL (to get coding support in IntelliJ IDEA).

~\..\gradle_monorepo > git init
Initialized empty Git repository in C:/Users/lmeisege/repositories/gradle_monorepo/.git/
~\..\gradle_monorepo git: master > gradle init
Starting a Gradle Daemon, 2 busy and 2 incompatible Daemons could not be reused, use --status for details

  1: basic
  3: library
  4: Gradle plugin
Enter selection (default: basic) [1..4] 1

Select build script DSL:
  1: Groovy
  2: Kotlin
Enter selection (default: Groovy) [1..2] 2

Project name (default: gradle_monorepo):

> Task :init
Get more help with your project: https://guides.gradle.org/creating-new-gradle-builds

BUILD SUCCESSFUL in 29s
2 actionable tasks: 2 executed

~\..\gradle_monorepo git: master > git add -A
~\..\gradle_monorepo git: master > git commit -m "initial commit"
[master (root-commit) 98788f2] initial commit
 8 files changed, 318 insertions(+)
 create mode 100644 .gitattributes
 create mode 100644 .gitignore
 create mode 100644 build.gradle.kts
 create mode 100644 gradle/wrapper/gradle-wrapper.jar
 create mode 100644 gradle/wrapper/gradle-wrapper.properties
 create mode 100644 gradlew
 create mode 100644 gradlew.bat
 create mode 100644 settings.gradle.kts

This leaves us with an initialized gradle repository.

Add services

For simplification reasons we will only deploy 3 docker images into an existing kubernetes cluster.

We will create 3 simple blogging sites:

~\..\gradle_monorepo git: master > mkdir kotlin_blog
~\..\gradle_monorepo git: master > mkdir java_blog
~\..\gradle_monorepo git: master > mkdir devops_blog

Each of them need their own build.gradle.kts file:

repositories {
  jcenter()
}

This file needs to be present in this locations:

kotlin_blog/build.gradle.kts
java_blog/build.gradle.kts
devops_blog/build.gradle.kts

Now we include the three modules in the settings.gradle.kts :

rootProject.name = "gradle_monorepo"
include("kotlin_blog")
include("java_blog")
include("devops_blog")

We can now run ./gradlew projects and will see our 3 subprojects:

~\..\gradle_monorepo git: master > ./gradlew projects

> Task :projects

------------------------------------------------------------
Root project
------------------------------------------------------------

Root project 'gradle_monorepo'
+--- Project ':devops_blog'
+--- Project ':java_blog'
\--- Project ':kotlin_blog'

To see a list of the tasks of a project, run gradlew <project-path>:tasks
For example, try running gradlew :devops_blog:tasks

BUILD SUCCESSFUL in 3s
1 actionable task: 1 executed

Now we commit this stage and after that we can begin filling this services:

~\..\gradle_monorepo git: master > git add -A
~\..\gradle_monorepo git: master > git commit -m "added subprojects"

Dockerize the blogs

We will start with a simple Dockerfile for the kotlin_blog subproject:

kotlin_blog/src/main/docker/Dockerfile

FROM nginx

COPY index.html /usr/share/nginx/html

We take the nginx image and copy our sample index.html into this image to /usr/share/nginx/html which is the default directory for service html files.

kotlin_blog/src/main/docker/index.html

<!DOCTYPE html>
<html>
<body>

<h1>Kotlin blog</h1>

<p>This is the most sophisticated kotlin blog!</p>

</body>
</html>

We can now build and run the image to display our kotlin_blog:

~\..\gradle_monorepo\kotlin_blog\src\main\docker git: master > docker build .
Sending build context to Docker daemon  3.072kB
Step 1/2 : FROM nginx
 ---> 2622e6cca7eb
Step 2/2 : COPY index.html /usr/share/nginx/html
 ---> 85660c94fe0c
Successfully built 85660c94fe0c
~\..\gradle_monorepo\kotlin_blog\src\main\docker git: master > docker run -p 8080:80 85660c94fe0c

While the last command runs we can visit the blog through http://localhost:8080 and we should see our html page rendered.

Now we will use gradle to build this docker image for us:
kotlin_blog/build.gradle.kts

repositories {
  jcenter()
}
tasks {
  register<Exec>("dockerBuild") {
        workingDir("src/main/docker")
        commandLine("docker", "build", ".")
  }
}

Now we can build our kotlin_blog docker image with the following gradle task:

~\..\gradle_monorepo git: master > ./gradlew kotlin_blog:dockerBuild
> Task :kotlin_blog:dockerBuild
Sending build context to Docker daemon  3.072kB
Step 1/2 : FROM nginx
 ---> 2622e6cca7eb
Step 2/2 : COPY index.html /usr/share/nginx/html
 ---> Using cache
 ---> 85660c94fe0c
Successfully built 85660c94fe0c

BUILD SUCCESSFUL in 3s
1 actionable task: 1 executed

Use a gradle plugin to build the docker image

To not duplicate this logic for each of our blogs we will now move it into the buildSrc directory. This directory is scanned from gradle per default before any project configuration is done.

So we can define methods, tasks and plugins there and use them in the rest of our project.

The buildSrc project is like any other subproject in gradle and needs a build.gradle.kts file.

buildSrc/build.gradle.kts

plugins {
    kotlin("jvm") version "1.3.72"
}

repositories {
  jcenter()
}

We will create a small build plugin for our task.

First thing we need to do now is to create our plugin file:

buildSrc/src/main/kotlin/DockerBuildPlugin.kt

import org.gradle.api.Plugin
import org.gradle.api.Project

class DockerBuildPlugin : Plugin<Project> {

    override fun apply(project: Project) {

        project.tasks.register("dockerBuild"){
            it.doLast {
                project.exec {
                    it.workingDir("src/main/docker")
                    it.commandLine("docker", "build", ".")
                }
            }
        }
    }
}

As you can see we had to make some small changes to our code and now use project.exec instead of the Exec-Task.

To let gradle know of our newly created plugin, we have to create an addition file in the buildSrc/src/main/resources/META-INF/gradle-plugins folder:

buildSrc/src/main/resources/META-INF/gradle-plugins/DockerPlugin.properties

implementation-class=DockerBuildPlugin

If we had a fully qualified package we had to create a file with the name de.test.DockerPlugin.properties
and had to set the fully qualified name for the implementation-class: de.test.DockerBuildPlugin

Now gradle will automatically discover our plugin and we can use it in our blogs build.gradle.kts files:

plugins {
    id("DockerPlugin")
}
repositories {
    jcenter()
}

We removed the dockerBuild task because this task is not automatically configured through our gradle plugin.

If we now execute the dockerBuild task in the root project we will see that it automatically launches all docker builds for all blogs:

~\..\gradle_monorepo\ git: master > .\gradlew dockerBuild

> Task :devops_blog:dockerBuild
Sending build context to Docker daemon  3.072kB
Step 1/2 : FROM nginx
 ---> 8cf1bfb43ff5
Step 2/2 : COPY index.html /usr/share/nginx/html
 ---> 8fe6bdacbe37
Successfully built 8fe6bdacbe37

> Task :java_blog:dockerBuild
Sending build context to Docker daemon  3.072kB
Step 1/2 : FROM nginx
 ---> 8cf1bfb43ff5
Step 2/2 : COPY index.html /usr/share/nginx/html
 ---> 5c500558e345
Successfully built 5c500558e345

> Task :kotlin_blog:dockerBuild
Sending build context to Docker daemon  3.072kB
Step 1/2 : FROM nginx
 ---> 8cf1bfb43ff5
Step 2/2 : COPY index.html /usr/share/nginx/html
 ---> Using cache
 ---> caf48f63df58
Successfully built caf48f63df58

BUILD SUCCESSFUL in 10s
3 actionable tasks: 3 executed

Tagging the docker images

The next problem we encounter is that we cannot really differentiate between the images of our blogs because the only get assigned an arbitrary image id during the build phase.
However we can design our plugin in a way that enables us to tag our images according to the project_name.

buildSrc/src/main/kotlin/DockerBuildPlugin.kt

import org.gradle.api.Plugin
import org.gradle.api.Project

class DockerBuildPlugin : Plugin<Project> {

    override fun apply(project: Project) {

        project.tasks.register("dockerBuild"){
            it.doLast {
                project.exec {
                    it.workingDir("src/main/docker")
                    it.commandLine("docker", "build", ".", "-t", project.name)
                }
            }
        }
    }
}

This change will tag the docker images with the subprojects name the task is execute from. (e.g. java_blog, kotlin_blog, devops_blog).

If we now build our project you will see that the tags are assigned correctly:

~\..\gradle_monorepo\ git: master > .\gradlew dockerBuild

> Task :devops_blog:dockerBuild
Sending build context to Docker daemon  3.072kB
Step 1/2 : FROM nginx
 ---> 8cf1bfb43ff5
Step 2/2 : COPY index.html /usr/share/nginx/html
 ---> Using cache
 ---> 8fe6bdacbe37
Successfully built 8fe6bdacbe37
Successfully tagged devops_blog:latest

> Task :java_blog:dockerBuild
Sending build context to Docker daemon  3.072kB
Step 1/2 : FROM nginx
 ---> 8cf1bfb43ff5
Step 2/2 : COPY index.html /usr/share/nginx/html
 ---> Using cache
 ---> 5c500558e345
Successfully built 5c500558e345
Successfully tagged java_blog:latest

> Task :kotlin_blog:dockerBuild
Sending build context to Docker daemon  3.072kB
Step 1/2 : FROM nginx
 ---> 8cf1bfb43ff5
Step 2/2 : COPY index.html /usr/share/nginx/html
 ---> Using cache
 ---> caf48f63df58
Successfully built caf48f63df58
Successfully tagged kotlin_blog:latest

BUILD SUCCESSFUL in 9s
3 actionable tasks: 3 executed

Now we can use docker run again to test our image:

~\..\gradle_monorepo\ git: master > docker run -p 8080:80 java_blog

If we access the page in the browser through http://localhost:8080 we should see our java_blog welcome message.

Wrapping up

We used grade to build docker images
We then added multiple docker images we all wanted to build with a similar configuration
We build a gradle plugin to automate the docker build process

The source is available on Github:

gradle_monorepo

Why do we need monorepositories in the first place?

Why did I choose to organize my code in a monorepository?

What is required to get my monorepository up and running?

Initial Setup

Add services

Dockerize the blogs

Use a gradle plugin to build the docker image

Tagging the docker images

Wrapping up

Suche

Kategorien

Tags

Gradle as a Tool for monorepositories

Why do we need monorepositories in the first place?

Why did I choose to organize my code in a monorepository?

What is required to get my monorepository up and running?

Initial Setup

Add services

Dockerize the blogs

Use a gradle plugin to build the docker image

Tagging the docker images

Wrapping up

Suche

Kategorien

Tags