A setup for working with Windows build slaves
Installation and management of CI servers is a critical task for any IT team. Kubernetes and its package manager (Helm) provide an easy way to customize Jenkins installations. Let’s see how to do this and add Windows build slaves.
Container solutions (e.g. Docker) enable your developers to develop applications and ship them for testing or deployment without having to worry about environment setup or facing the “works-on-my-machine” phenomenon.
Kubernetes (aka k8s) provides an orchestration framework for containerized applications. This framework ensures the availability of the containerized application through health monitoring and the recovery of failed containers.
Containers and k8s speed up your dev-build-test-deploy cycle by abstracting most of the infrastructure management overhead. New tools build on top of k8s and containers to enable infrastructure, configurations, and even application deployment to be created and managed from code. Such tools include: Kops -a k8s cluster management tool- and Helm (a k8s package management tool). These tools allow you to automate your k8s cluster creation and management. Further, they allow you to easily deploy and manage your applications on the k8s cluster.
Helm enables you to manage applications deployment on your k8s cluster. It provides a bunch of stable packages -called charts- which you can deploy to your k8s cluster out of the box. Alternatively, Helm allows you to create your own chart and reuse it.
In this post, we will deploy Jenkins CI server on k8s cluster -on AWS- using Helm, and then connect Windows slaves to it. The following steps are needed to deploy Jenkins on k8s:
Note: While this post uses AWS as a cloud provider. The concepts and steps discussed here can be mapped to other cloud providers offering similar services.
Once you have the cluster created you need to install Kubectl and then install helm on the machine which you use to manage your cluster. Once installed, Helm needs to be initialized so it automatically installs its server. This is called Tiller on the k8s cluster you are using.
Now that you have a cluster, and Helm is installed and initialized, you can install the stable Jenkins chart on the k8s cluster using the following command:
helm install --name <<your-release-name>> stable/jenkins
That’s it! You now have a Jenkins master deployed on your cluster and exposed to the rest of the world using a load balancer (AWS ELB in that case). Helm output will tell you how to extract the URL of your Jenkins master and the admin password.
The figure below shows what has happened. Helm has created a deployment and a couple of services on your k8s cluster. The deployment defines a desired state for the pod containing the Jenkins master. This allows k8s to bring up a new pod running a Jenkins master inside a docker container if the current pod dies. Since pods are mortal, the services created by this step allow other pods / applications to reach the Jenkins master using a DNS (service) name, instead of the IP of the underlying pod. The Jenkins master port is exposed externally (to be accessed from anywhere on the internet) using the cloud provider’s load balancer (in this case, AWS ELB). So, no matter what happens to the pods inside the cluster, you will always be able to access your Jenkins master through the load balancer IP. The Jenkins agent service is, however, exposed with Cluster IP meaning it is only accessible from within the cluster.
You need not worry as the Jenkins Helm chart has a configurable parameters file which you can customize. For example, you can specify which plugins should be installed with your Jenkins and which docker images to use for both master and slaves.
You might be wondering - where are the slaves?
We used the default configuration provided with the Jenkins Helm chart. In that configuration Jenkins comes with the Jenkins Kubernetes plugin installed. This plugin allows the master to create (on demand) slaves as docker containers in pods. This means that whenever there is a job the master will create a slave, on the fly, inside your k8s cluster. The slave will perform the job and then will be discarded.
Now, you might be wondering if you can have Windows slaves, right?
In a previous post, we discussed the current state of the Windows support in k8s and concluded that at this point of time it is not mature enough for production-like environments when you use Kops to manage your k8s cluster. As a result, we recommended that Windows hosts are created outside of the cluster and are allowed to communicate with applications deployed on your k8s cluster. The following diagram illustrates the architecture we use:
We created a separate AWS VPC to host Windows VMs hosting permanent Jenkins slaves.
Helm has exposed the Jenkins agent service (responsible for connecting slaves) to the cluster only (using the Cluster IP service type ). Helm assumes that all slaves will be connecting from within the k8s cluster. However, the Windows slaves will be connecting from outside the cluster as we described above.
To make the Jenkins agent service accessible by Windows slaves connecting from the Windows VPC, there are two options:
Exposing the slaves agent using NodePort service type
Node port exposes the service on a specific port across all cluster nodes. This makes the service accessible from within the cluster using Cluster IPs, from within the k8s VPC using AWS private IPs, and externally (from outside the k8s VPC) using any public IP of any of the k8s nodes. The problem with that approach is that k8s nodes are mortal and when they die they are replaced with new nodes (and consequently, with new external IPs).
Or, exposing the slaves agent using LoadBalancer service type
A load balancer will always connect to the Jenkins agent service from the Windows VPC regardless of the internal IP of the service. Again, you will need to allow traffic from the Windows VPC only for security reasons.
Note: for the first option you need to allow traffic on the Jenkins Slavelistener port (default 50000) from your Windows VPC in your k8s nodes’ security group. By default this security group blocks connections from outside the k8s cluster VPC.
Note: If you have multiple services that need to be accessible from outside your k8s cluster then you might consider using an HTTP reverse proxy (e.g. Træfik) to avoid receiving hefty bills for AWS ELB.
Now you can have a Jenkins master deployed on your k8s cluster that is able to create Linux slaves on demand inside the cluster. Further, it can connect to permanent Windows slaves hosted on VMs outside the k8s cluster. In an upcoming post we will talk about provisioning Jenkins Windows build slaves using Packer and Terraform. Stay tuned!
Helm charts lifecycle management is a manual task. Helmsman allows you to automate your Helm charts lifecycle management using declarative configuration files.
How to automatically deploy Helm charts to a Kubernetes cluster
As of Kubernetes 1.5, Windows containers support was introduced in an alpha release. With the beta version just around the corner, we put the Windows support to the test. Is it mature enough for production environments?
A verdict on adding Windows minions to Kubernetes clusters in AWS
Do you have a tendency to use the backlog as an eternal placeholder? If so, you probably have a lot of clutter that’s creating a lot of frustrations for your end-users. In this post we’ll show you how to clean up your Jira issues and reduce the backlog with some basic JQL queries.
Tips to improve project management in the Atlassian suite
How to test Kubernetes artifacts like Helm charts and YAML manifests in your CI pipelines with a low-overhead, on-demand Kubernetes cluster deployed with KIND - Kubernetes in Docker.
Low overhead, on-demand Kubernetes clusters deployed on CI Workers Nodes with KIND
Had enough of sluggish polling? With instant Artifactory event triggers you can give responsiveness in Jenkins a real boost. Here’s an easy way to set it up.
A super easy configuration guide
With the arrival of microservices code is becoming disposable. Does this mean that we no longer need maintainable code? Is it the end of refactoring?
Still relevant or increasingly redundant?
In software development tight coupling is one of our biggest enemies. On the function level it makes our application hard to change and fragile. Unfortunately, tight coupling is like the entropy of software development, so we have always have to be working to reduce it.
How to safely introduce modular architecture to legacy software
I am an Atlassian certified trainer and over the years I have been spending much time with clients and their Jiras. In this blogpost, I have collected some small tips and tricks that will make your Jira usage better.
Jira Software is a powerful tool deployed in so many organizations, yet in day to day usage people are missing out on improvements, big and small.
In this post, I’ll take a closer look at the version of Jenkins X using Tekton, to give you an idea of how the general development, build, test, deploy flow looks like with Jenkins X. How does it feel to ship your code to production using a product coming from the Jenkins community that has very little Jenkins in it?
A crash course in Jenkins X and how to test it out on a local Kubernetes cluster
In this blog I will show you how to create snapshots of Persistent volumes in Kubernetes clusters and restore them again by only talking to the api server. This can be useful for either backups or when scaling stateful applications that need “startup data”.
Sneak peak at CSI Volume snapshotting Alpha feature
Hear about upcoming events in Scandinavia, latest tech blogs, and training in the field of Continuous Delivery and DevOps