Handling OpenStack Cloud Images in Platform9
OpenStack Image Management
One of the most fundamental functions of an any cloud platform is the management of virtual machine images. In OpenStack, this functionality is mostly taken care of by the Glance collection of services. Glance services provide a repository for virtual machine images and metadata, and endpoints for delivering the OpenStack cloud images to both the nova compute service and the cinder volume service.
Glance is made up of two primary services, both of which need access to a relational database:
- The glance-api service provides the external REST API for the glance services. It exposes resources that allow CRUD operations on both image metadata and images themselves. In addition it also manages the image ‘store’, which provides bulk storage of image data. When an image is downloaded from (or uploaded to) glance, glance-api accepts the request and serves (or stores) the image data. Most metadata operations are sent to glance-registry, but in some cases, glance-api accesses the database directly.
- The glance-registry service acts as a metadata store for glance. Requests to glance-api that involve updating or retrieving metadata are (mostly) forwarded to glance-registry to update the glance database.
Glance image management at Platform9. This diagram shows virtual machine provisioning and snapshot image creation in a Platform9 cloud. In this configuration, multiple imagelibrary-authorized hosts manage images on a shared external filesystem.
Why is Platform9’s Image Management Use Case Special?
Platform9 makes a very strict distinction between on-premise, customer-owned services, and Platform9-managed, controller-based services. You can read about Platform9’s architecture in our earlier blog post on this topic. All customer data, including virtual machine images must remain on the customer’s hosts. Because of this, we want the image repository to be based on customer hosts, and any image copies to or from that repository to be limited to the customer’s network.
Given these requirements, we’re left with the choice of whether to host the Glance image service components on the controller, or install them as a role on one of the customer hosts. Initially, we did the former; for the current version of Platform9, we’re doing the latter.
Customer Image Library
Early on, we made the assumption that any service that needed to access Platform9-managed databases should live in the cloud. Because of this, we hosted both glance-api and glance-registry on the cloud-based platform9 controller. Since we still needed to limit customer images to on-premise hosts, we could not allow the controller-based glance-api service to actually store image data. To deal with this we introduced the Image Library on-host role.
The Image Library service provides two functions:
- It monitors a filesystem location on a customer host for the introduction or removal of virtual machine images. When a customer creates a file in this filesystem location, the Image Library tries to determine the properties of the file and generates metadata for it. This data is then sent to the controller-based glance-registry service, where it is saved to the database, and shows up as an ‘image’ in glance. Note that since the image data is not actually available to glance-api – it remains on the host, the standard glance REST API can only provide the image metadata, not the image file itself.
- It implements a simple web server that has access to the monitored directories. Given the metadata of a discovered image and a keystone authentication context, a client can contact this web server and download the image. It also has the ability to upload images, which is necessary for nova and cinder to save image snapshots back to our image repository.
In order to add an image to the glance service, a customer simply needs to copy the file into the monitored filesystem directory. Within a second or two, the file shows up as a glance image and is available for deployment.
When a customer requests that we deploy an image into a virtual machine using nova, nova-compute fetches the image metadata from the glance-api service (an https call to the controller), and uses that metadata to locate the Image Library service. From there, it can download the image from the Image Library web server to the compute host.
On the other hand, when a customer snapshots an image, nova-compute creates the disk snapshot and uploads it to the Image Library web server, where it is made available for download.
On-Host Glance
The Image Library approach has a number of disadvantages. Since glance-api image upload and download is not really available, we’re not making the full OpenStack REST API available to customers, and we can’t leverage the existing glance command line client. Existing scripts a customer might have already written cannot be used for our service. In addition, other features like multi-region, identity federation, and the VMware on-datastore image store, would have to be implemented by us as an extension to the image library.
To make the full glance API available, and to allow us to more fully leverage the functionality already in glance, we decided to begin managing the glance-api service as an on-host role. Since Platform9 version 1.3 provides us with the ability to tunnel any protocol through https to the controller, we were able to simply package glance-api as an on-host role, and configure it to talk to the database and glance-registry through a proxy.
To make it possible for clients to actually find the glance-api server after a customer has authorized the role, we leveraged the keystone service catalog. When a customer authorizes the glance-api role on one of his/her hosts, the address of that host is automatically registered as an endpoint in the keystone service catalog. Operations in the glance command line client first check the keystone service catalog for the glance-api endpoint. From here, metadata CRUD, image upload and download all work through the standard glance REST API.
In addition, we’ve modified nova and cinder to use the keystone service catalog endpoints for any glance operations. When for example, nova needs to download an image, it gets the glance-api endpoint from keystone. From there it can check the image’s metadata, and download the image. Note that we’ve kept the filesystem monitoring function in the Image Library to continue giving customers the option to simply copy images to the glance/Image Library host.
This explains how we are able to provide VM images for our Platform9 customers. In future posts, we’ll talk about how we enable the discovery and importing of VMware vSphere templates for use as glance images. We’ll also talk about how we will be enabling distributed glance services (one or more instances per datacenter) managed by the same OpenStack controllers. So stay tuned.
- Beyond Kubernetes Operations: Discover Platform9’s Always-On Assurance™ - November 29, 2023
- KubeCon 2023 Through Platform9’s Lens: Key Takeaways and Innovative Demos - November 14, 2023
- Getting to know Nate Conger: A candid conversation - June 12, 2023