Bob’s Architecture

Bob Core

Bob follows the UNIX Philosophy of being small, robust and do one thing and do it well. Unlike most of the popular CI tooling, Bob tries to be a collection of small components which does one thing really well and allows you to compose them in the way you want and build your CI platform.

The core of the project lives in this repository. Its entirely written in Clojure, which allows Bob to be really small, scale easily and handle concurrency well. It uses XTDB as its temporal, document database.

ALL of the internals is exposed via a standard REST API.

Pipeline

A Pipeline is the one and only build unit for Bob. Bob is simple and therefore avoids the need of complex hierarchies (e.g. Jobs). A pipeline is completely self contained and can be linked to other pipelines either in up/downstream.

Extending Bob and Plugins

Extending should be external, i.e. one should be able to change Bob’s behavior by calling/using it differently without having to dig too deep into the core.

Extension of a CI/CD system is generally needed in the case of making the system interact with the world differently. Examples would be cloning a different kind of source control, reading a file from network, reacting to events etc. For this Bob defines a Resource, its way of abstracting out the need to know how to fetch something.

The other need for extension is to store/deploy the results of a build somewhere. For this Bob defines an Artifact, its way of abstracting out the need to know how to publish its build results.

Problems with Extending other CI tooling

Bob strongly rejects the idea of traditional plugins wherein the plugin is generally written in the same technologies as the core and is loaded into the same process as the core. Well known examples for this can be seen in Jenkins, GoCD, TeamCity and others. This style of extending the core functionality presents the following issues:

• The plugin author(s) must know the technologies used to develop the core.
• The core has now one more possible attack surface where an issue in the plugin can cause nasty things to happen in the core ranging from it being unstable to security issues.
• Deploying these plugins often means a restart of the system, downtimes, misconfiguration and more importantly this needs happen on the same machine as the core and opens up a possibility of creating snowflake servers.

The Execution Model

Like its depicted in the diagram above, Bob uses Podman internally as its execution engine. This provides an easily provisioned, isolated and disposable environment for build to take place in. The runners in Bob runners are guaranteed to be rootless hence are safe to be run in a cloud native environment. Each runner can execute multiple pipeline runs concurrently fully isolated from each other.

A pipeline is executed in the following way:

1. The image provided in the pipeline is pulled by the Podman (if already not present).
2. A container is created with the command specified in the first step.
3. If any environment variables are defined, they are added to the container.
4. If the step has defined a needs_resource key, the corresponding resource is fetched from the provider and copied over to the container.
5. The relevant working directory is set: the resource folder if a resource was mounted or the home of the container.
6. The container is started and Bob waits for its completion.
7. Bob attaches to the stderr and stdout of the container while its running and streams the log to the DB.
8. If the container exits with code as zero and if a produces_artifact key was defined in the step, Bob streams the artifact out from the path on the container to the Artifact Store. If the exit was anything other than zero, Bob marks the pipeline run as failed and stops executing the rest of the steps.
9. If the last step succeeded, Bob creates a diff of the current container which contains the effects of the last command via the commit feature. This becomes the next image in the series of execution of steps.
10. This recursively continues until there are no steps left. If all steps pass, Bob marks the pipeline run as passed.