A Recipe for Continuous Integration Using an OpenStack Private Cloud
A few weeks ago, my co-founder Bich ran a fantastic post on How We Run Platform9 on Platform9. Since then, we’ve been asked many times for our recipe on using Continuous Integration (CI/CD) with OpenStack. This post describes such a recipe for Platform9 customers or other OpenStack users, for use with Jenkins or Teamcity based continuous integration.
Building a Software-as-a-Service product means having the ability to ship early and ship often. Building an enterprise-grade management platform that customers can rely on to run their production workloads means we need to have very high confidence in the quality of our software before we push it to our customers. This is why our highly automated CI/CD process is critical: it ensures we can ship high quality software frequently.
The Benefits
- Efficient Utilization of resource – OpenStack allows us to set the desired level of over-subscription for CPU, Memory and Storage resources, and then to continuously monitor current allocation and utilization levels to tweak this accordingly, ultimately to get optimal utilization of resources.
- End-To-End Automation – The entire test lifecycle is 100% automated, with very little manual intervention. This means very high developer productivity.
- Better Visibility – Our DevOps folks have much better visibility into artifacts for current running builds and workloads. They are available for debugging and testing both via UI and API when needed.
The Ingredients (Requirements)
- An OpenStack deployment configured with sufficient compute, memory, storage infrastructure to run your CI/CD setup.
- Vanilla VM images with base Operating System installed for your specific use case
- Jenkins or TeamCity to drive your CI/CD workflow pipeline
- Your Version Control System (VCS) managing your source code. (Rest of this procedure assumes Git as the default VCS)
The size of your deployment would depend on scale of workload you wish to run.
Specs for one of our internal environment
| Total | Server config | Memory | Storage | Price |
|---|---|---|---|---|
| 100Ghz CPU 500GB Memory and 2.5TB Storage | 5 AMD Opteron G5 Model Servers - 2 Sockets 3GHz x 12 cores each (HP Proliant DL385) | 100 GB Memory each DDR3 SDRAM | 500 GB Storage | ~4K price |
Over-Allocation:
| Compute | Memory | Storage |
|---|---|---|
| We heavily over-allocate compute in this environment - 5x or 6x but can go up to 16x | Memory overallocation is allowed up to 1.5x but in practice, we try not to reach that level as it results in slow-downs of non-test workloads | Storage is COW and heavily utilized. In our setup, Storage tends to be the bottleneck |
Workload
| VMs run each day | VMs run at any given point | Common VM configs |
|---|---|---|
| ~1000 VM instances get started and torn down each day | ~200-300 VMs running at any given point | 1 vCPU - 1GB Mem - 10GB Storage, 1 vCPU - 4GB Mem - 8GB Storage |
The Recipe (Procedure)
Get you OpenStack environment up and running.
NOTE: Platform9 Managed OpenStack is NOT a requirement for this procedure. You can run your CI/CD setup with any OpenStack deployment. Skip this step and move to Step 2 if you already have a running OpenStack deployment.
The beauty of Platform9 is that it makes getting your OpenStack Private Cloud extremely simple, using commodity hardware, at low cost. For those interested, here’s a Step-By-Step Tutorial on getting started with Platform9.
The above table should act as a guideline for creating an appropriate hardware setup.
Configure Your Continuous Integration (CI) Server
At Platform9, we switched to TeamCity after having years of experience running Jenkins. While both are mostly on-par with supported features, one is free while the other is a paid product. Here are a few good comparisons between the two for those interested. Complete details of how our TeamCity builder is configured will come in a separate post. For now, lets assume that the reader has knowledge of configuring build chains and dependencies in his favorite CI server.
The Workflow
Step 1 – Developer pushes a changeset to Git
The moment a developer is ready to push a change to Git, a private branch gets created with a specific name pattern.
The TeamCity builder is watching for activity on branches with this pattern. Once noticed, it pulls the changes from the branch to start the build chain process.
Step 2 – Run Unit-Tests and create build artifacts.
The first step of the chain is to do basic validation of the change by running unit tests. TeamCity slave is spawned to run unit tests on the component that the software changeset touches, as well as each dependent components. Any failures in unit tests halt the job immediately and the developer is notified.
If all the unit tests pass, build scripts specific to Platform9 are run to create build artifacts corresponding to our deployment.
NOTE This part will be specific to your own environment.
Step 3 – Deploy a Virtual Machine ‘Pod’
At this point, we are ready to trigger our integration tests. The TeamCity builder talks to our internal deployment of Platform9 and creates a Virtual Machine ‘Pod’. A Pod is simply a collection of golden Virtual Machine images needed to test out a given change. We standardize on a single Pod configuration for Platform9, you might have multiple Pods created in your environment – each specific to a given use case.
Platform9’s Pod consists of:
- One or more Platform9 ‘controller’ VMs, deployed with Platform9 specific code
- One or more database VMs
- One or more hypervisor VMs to act as ‘hosts’ for our testing
The Teamcity builder communicates with Platform9 via REST APIs and deploys the Pod.
Step 4 – Configure the Virtual Machine ‘Pod’
Once the Pod is deployed, Teamcity builder communicates with Platform9 via REST APIs to configure the following:
– The Virtual Machines deployed within the Pod are Tagged with information about the specific build version being created, name of the user deploying the pod, etc. This data is useful for later tracking of the running Pods.
– Each VM in the Pod is customized with a default hostname, injected with ssh keys and a starter Ansible script to enable further customization via Ansible. Platform9 makes all this customization easy and possible.
– Once the Pod is up and running, Ansible scripts running in the Teamcity builder communicate with the Pod and customizes the VMs to have a specific identity. This is the step that deploys Platform9 specific packages in one or more of the VMs in the Pod.
Step 5 – Create multiple copies of the ‘Pod’- run integration tests
Once the Pod is configured and ready, Teamcity builder snapshots the Pod to create new set of golden images. It then creates multiple copies of the Pod. Each individual copy is utilized for running a specific set of integration tests.
At this point, the Teamcity builder powers-on multiple Pod copies in parallel. Each Pod is then utilized to simulate a different configuration and run corresponding integration tests in parallel.
Step 6 – Pass or fail
If all integration tests pass, the changeset is automatically pushed into main git branch. For any failures, the appropriate developer is notified. All the build configurations are saved so that the developer can log into the Platform9 UI and work on appropriate Pods to debug the issues.
Step 7 – Cleanup
Finally a scheduled job runs periodic cleanup of usused VMs based on the tags.
Hope you found this useful. Feel free to leave comments or reach out for specific questions.
- Introducing Arlon! Your Kubernetes ClusterManagement Assembly Line - September 20, 2022
- Watch My GeekWire Talk: Not Your Mother’s Cloud – Best Practices for Enterprise Hybrid Cloud - July 3, 2018
- Kubernetes Networking: Achieving High Performance with Calico - April 17, 2018