Automating an Automation Server
Job DSL or scripted/declarative pipelines are becoming the standard when it comes to defining jobs in Jenkins. Now we need a similar solution to maintain Jenkins itself.
Jenkins can be installed through native system packages, Docker, or run standalone by any machine with a Java Runtime Environment (JRE) installed. But configuring Jenkins is a tedious process, no matter how you choose to run it, and the more plugins you use the more tedious it gets. Given that there are more than 1400 plugins available it’s easy to reach an unmanageable state.
Both Jenkins and its plugins require manual configuration, by default available only via UI. The Jenkins Configuration as Code plugin (JCasC) allows us to manage such configuration via human readable configuration files.
Configuration as code is not a new concept and Jenkins already provides a way to automate its deployment e.g. by introducing support for init scripts written in Groovy. The problem is that it requires you to know Groovy and understand the inner workings of Jenkins and its plugins. It also mixes scripts with configuration and doesn’t honor one of our important principles of separating the “what” from the “how.”
One specific example is JenkinsAsCodeReference developed at Praqma. I worked with a customer who wanted to introduce this approach and I loved it from the very beginning. I didn’t have much experience with Jenkins at the time, so having a way to preserve old configuration, without the need of saving non-readable xml or taking screenshot of existing setup for documentation, was a big relief. And it allowed me to experiment and implement without fear.
JenkinsAsCodeReference is now being used by a number of companies, but it has issues. It forces you to use Docker and it also handles deployment of Jenkins. Redeployment is time-consuming and the actual configuration is mixed with Groovy scripts. This started to get annoying so we decided to come up with a new, better solution that would provide the same features (and much more) but would be easier to maintain, faster, and could be used in various (not only Docker) scenarios.
By we I mean me and other Praqma developers, later joined by CloudBees engineers, and by better solution I mean a plugin.
While preparing the prototype of the plugin we found existing ones. CloudBees engineers and regular Jenkins users were trying to do exactly the same thing. We saw this as a great opportunity for collaboration, so we got in touch with CloudBees and found that our visions, goals and motivations were very much aligned.
And now we’re working together on making this the Configuration as Code solution for Jenkins.
The goal of the plugin is to do for Jenkins configuration what Job DSL once did for jobs. Move it to code! We also want to learn from the pains that were involved in making Job DSL. That is, for the plugin to succeed, it has to support as many plugins as possible out of the box, without needing to write glue code for each plugin. Let’s remind ourselves that there is (much) more than 1000 of them. Not ALL of them require some sort of global configuration, but the number that need to be supported by the Jenkins Configuration as Code plugin is still quite high.
It is a challenge and we already know we won’t be able to support all of the plugins. Some glue code and a number of Pull Requests to make the plugins support our requirements will be needed, but we believe it’s worth it.
Jenkins Configuration as Code does not help you deploy Jenkins - you will still need to do that your favorite way using containers, Kubernetes, native system package, Ansible or the like, but as soon as you’re about to start Jenkins, JCasC will be there to save you. There’ll be no more manual editing of the Jenkins global configuration under Manage Jenkins after deployment.
Once you have prepared your configuration file Jenkins can be configured automatically when it starts. You can restore your Jenkins easily - within seconds. You can also reuse the same file to configure multiple Jenkins instances, or you can spin up a local instance to test your changes before you make them part of the production environment.
If you decide to keep your configuration file in some VCS e.g. Git repository - which I believe you should - you can keep track of changes in the Jenkins global configuration. This makes it possible to quickly restore a previous configuration if the new one proves to be unsatisfying. It also makes it easy to compare and trace changes when something is broken.
Jenkins Configuration as Code is the last missing piece in the puzzle for you to completely manage your Jenkins instance as code. You can now have your infrastructure as code.
One of the many challenges we set ourselves was to mimic Jenkins UI in a configuration file as much as possible. We didn’t want to have to scroll through long documentation pages to find out how to configure the Mailer plugin or Artifactory Server details. And, of course, we didn’t want to implement separate solutions for each plugin either. So, if you have some Jenkins management experience, writing such a file will come naturally to you. But don’t worry if it doesn’t, we’re not the enemies of documentation, we still allow it to generate itself! Furthermore, it will be generated based on your actual Jenkins setup and you can access it directly on your own server.
The configuration files for the global Jenkins configuration should be human readable. We don’t want it to be language specific, it can’t require specific knowledge, and it should support comments. These are the reasons we chose YAML. It is easy to write and because of available schema it is also possible to get support within your IDE and verify your configuration without running Jenkins instance.
And, most importantly, once you introduce the change in your configuration file you’ll be able to reload it without restarting Jenkins. No more time consuming redeployment!
I mentioned the involvement of CloudBees engineers in the plugin, but there’s a little more to that story. We’ve presented the concept at a number of meet-ups and CloudBees made some noise about it during FOSDEM at the beginning of February. The reaction has been great - we’re are already getting external contributors. The Pull Requests and feature ideas are coming!
You can get involved too - the more PRs the better. Feel free to create issues with features or bugs. We don’t just want to solve our problems, we want to solve yours too!
https://github.com/jenkinsci/configuration-as-code-plugin is the place to go now!
Mastering Continuous Delivery with Jenkins
Use this training to jump start your Continuous Delivery implementation
Jenkins Configuration as Code is finally here. In this blog I’ll cover how to convert an existing Jenkins instance to JCasC and also how to start from scratch. Let’s get it up and running!
Getting started with Jenkins Configuration as Code
How to spin it up from scratch or migrate an existing instance
What do we need from a CI/CD system? How should we decide which one to use? In this blog we ask ourselves what a modern CI/CD system should look like and compare two commonly used build systems: Jenkins pipelines and Concourse CI.
Concourse vs. Jenkins
An in-depth comparison of two CI/CD servers
What testing steps should you include in your Continuous Delivery pipeline? Don’t just string together existing manual processes - use simple, collaborative tools to design something better!
Pipeline - The Game that Delivers!
A new card game to design Continuous Delivery pipelines
Praqma joins the RADON project
Unlocking the Benefits of Serverless FaaS for the European Software Industry
In the Accelerate book, researchers explain several metrics which they have shown will measure the performance of a DevOps organization, and crucially, drive performance of the organization as a whole. I will explain why this is important, using an analogy with your risk of a heart attack.
Winning with DevOps and Reducing Your Risk of a Heart Attack
Clinical Trials and Software Process
Your repositories are cluttered, you’re storing irrelevant builds, and your disks are full. Luckily, you’ve come to the right place for advice. Here’s how to set up automatic cleanup.
Artifactory Retention Policies
How to set up automatic Artifactory repository cleaning
In many security and safety critical environments there can be requirements for long-term reproducibility of artifacts that are created. In this talk from NDC Tech Town, Mike Long shows how to use Docker to create binary reproducible applications from a C++ application.
Binary Reproducible Builds
How to Create Binary Reproducible Builds using Native Build Tools and Docker
In distributed architectures networking is usually taken as a given. However, when it does not function as we expect it is often hard to understand the reason. This blog post outlines some general insights into networking with more specific techniques for understanding the networking infrastructure of a Kubernetes cluster.
Debugging Kubernetes Networking
How to Find a Needle in a Virtualized Network
Choosing the perfect CI/CD tool for your project can be tricky. In this post we compare two managed CI/CD services - CircleCI and Google Cloud Build. Sami Alajrami is on hand to score our exciting CI/CD matchup.
CircleCI vs Google Cloud Build
How to choose the CI/CD tool that’s right for you
Simply aligning your IT capability with your business goals isn’t enough to grow your company. To achieve real IT-enabled growth you’ll need to build on a foundation of technical excellence and thanks to DORA that’s something you can now actually measure.
Avoiding the Agile Alignment Trap
Why a new software process won’t fix your delivery effectiveness
Setting up an on-premise highly available Kubernetes cluster comes with many challenges. Rancher can help build a production grade cluster. Watch our Certified Kubernetes Administrator Henrik Hoegh set up a cluster and test the deployment.
How to install a Kubernetes Cluster with Rancher
A video guide to setup a highly available on-premise Kubernetes Cluster
Hear about upcoming events in Scandinavia, latest tech blogs, and training in the field of Continuous Delivery and DevOps