Troubleshooting the Management Plane
There are two components on the Management Workstation
- The airctl program and its associated services on the management host (as described in the Architecture Overview section)
- The management DU (the VM running the Platform9 management host).
Airctl and Services
As described in the architecture section, the management workstation contains the following primary components:
Airctl
The airctl utility is the most important tool used to manage the Platform9 DU.
Config-file
Verify you are explicitly specifying the config file as part of the command line, or make sure the config-file is named airctl-config.yaml in the user's home directory. **
# /opt/pf9/airctl/airctl --config /opt/pf9/airctl/conf/airctl-config.yaml get-du-version
DU Version: v-5.3.0-1716385
If the config-file is not present, you may see an error similar to the following output.
/opt/pf9/airctl/airctl get-du-version
fatal error: required config value duFqdn is undefined
The config-file also contains the configuration information for each of the hosts. Some clients create a copy of config-file for each cluster and then add the nodes/hosts to the nodeHostnames list.
# Defaults are shown in comments
installDir: /opt/pf9/airctl
# logDir: <defaults to the home directory of the user>
duFqdn: airctl-1-1720441-948.pf9.localnet
nodeHostnames:
- 10.128.145.53
- 10.128.144.194
- 10.128.144.196
- 10.128.144.142
State-dir
The state directory is located within the user's /root/.airctl folder. It is created and is employed every time airctl runs. It also contains a backup of the configuration for the Platform9 software and hence must remain secure, so please ensure it is not deleted accidentally.
[root@test-pf9-du-host-airgap-c7-1720441-150 .airctl]# pwd
/root/.airctl
[root@test-pf9-du-host-airgap-c7-1720441-150 .airctl]# ls -al
total 48
drwxr-xr-x. 3 root root 80 Oct 15 14:55 .
dr-xr-x---. 6 root root 256 Oct 15 15:42 ..
-rw-r--r--. 1 root root 43008 Oct 15 14:38 cloud-init-2021-10-15T14:38:39Z.iso
drwxr-xr-x. 3 root root 80 Oct 15 14:56 mongo
-rw-------. 1 root root 297 Oct 15 14:38 state.yaml
Furthermore, it is designed to run as a normal user, but for certain commands it may require root privileges, specifically, configure-localhost.
Libvirt & Qemu
Platform9 uses a VM to run the Platform9 management plane (the DU). Please ensure that both libvirt and qemu are installed and configured to be accessible by the user running the airctl command. This can be verified using the following commands.
[root@test-pf9-du-host-airgap-c7-1720441-150 .airctl]# yum list installed | grep libvirt
libvirt.x86_64 4.5.0-36.el7_9.5 @updates
libvirt-bash-completion.x86_64 4.5.0-36.el7_9.5 @updates
libvirt-client.x86_64 4.5.0-36.el7_9.5 @updates
libvirt-daemon.x86_64 4.5.0-36.el7_9.5 @updates
libvirt-daemon-config-network.x86_64 4.5.0-36.el7_9.5 @updates
libvirt-daemon-config-nwfilter.x86_64 4.5.0-36.el7_9.5 @updates
libvirt-daemon-driver-interface.x86_64
libvirt-daemon-driver-lxc.x86_64 4.5.0-36.el7_9.5 @updates
libvirt-daemon-driver-network.x86_64 4.5.0-36.el7_9.5 @updates
libvirt-daemon-driver-nodedev.x86_64 4.5.0-36.el7_9.5 @updates
libvirt-daemon-driver-nwfilter.x86_64 4.5.0-36.el7_9.5 @updates
libvirt-daemon-driver-qemu.x86_64 4.5.0-36.el7_9.5 @updates
libvirt-daemon-driver-secret.x86_64 4.5.0-36.el7_9.5 @updates
libvirt-daemon-driver-storage.x86_64 4.5.0-36.el7_9.5 @updates
libvirt-daemon-driver-storage-core.x86_64
libvirt-daemon-driver-storage-disk.x86_64
libvirt-daemon-driver-storage-gluster.x86_64
libvirt-daemon-driver-storage-iscsi.x86_64
libvirt-daemon-driver-storage-logical.x86_64
libvirt-daemon-driver-storage-mpath.x86_64
libvirt-daemon-driver-storage-rbd.x86_64
libvirt-daemon-driver-storage-scsi.x86_64
libvirt-libs.x86_64 4.5.0-36.el7_9.5 @updates
[root@test-pf9-du-host-airgap-c7-1720441-150 .airctl]# yum list installed | grep qemu
ipxe-roms-qemu.noarch 20180825-3.git133f4c.el7 @base
libvirt-daemon-driver-qemu.x86_64 4.5.0-36.el7_9.5 @updates
qemu-guest-agent.x86_64 10:2.12.0-3.el7 @base/7.8.2003
qemu-img.x86_64 10:1.5.3-175.el7_9.4 @updates
qemu-kvm.x86_64 10:1.5.3-175.el7_9.4 @updates
qemu-kvm-common.x86_64 10:1.5.3-175.el7_9.4 @updates
Additionally, verify that libvirt is functioning normally and is accessible to the user.
Virsh Checks
The following commands list the three most important aspects of the VM.
- virsh list - The name of the VM is pf9-mplane
- virsh net-list - The network pf9-mplane uses is active
- virsh net-dhcp-leases pf9-mplane - The IP address 192.168.120.254 is assigned to the VM
[root@test-pf9-du-host-airgap-c7-1720441-150 ~]# virsh list
Id Name State
----------------------------------------------------
1 pf9-mplane running
[root@test-pf9-du-host-airgap-c7-1720441-150 ~]# virsh net-list
Name State Autostart Persistent
----------------------------------------------------------
default active yes yes
pf9-mplane active yes yes
[root@test-pf9-du-host-airgap-c7-1720441-150 ~]# virsh net-dhcp-leases pf9-mplane
Expiry Time MAC address Protocol IP address Hostname Client ID or DUID
---------------------------------------------------------------------------------------------------------
2021-10-15 20:41:30 24:4d:74:b9:fc:43 ipv4 192.168.120.254/24 pf9-mplane -
The current version of the Platform9 management plane requires kvm/qemu to run in the VM. If the underlying machine is a VM, the qemu VM created may be very slow and possibly unusable. There are methods to run a VM in a nested configuration on both OpenStack & VMware, but this may require specific configuration to expose the KVM device (in cases of OpenStack) and the Virtualization features in case of VMware to the underlying VM.
HAProxy, IPTables, IP Address & DNS
The external port 443 listens for traffic coming into the pf9-management plane. The filtered traffic is allowed based on the IPtables rules and HAProxy settings. The Platform9 management plane VM listens specifically for traffic on port 443 coming from HAProxy. The following diagram better illustrates this relationship.
- HAProxy runs as a container and listens on port 8443 on the docker host network.
- Pf9-management plane runs on pf9-mplane libvirt network and has port 443 and port 22 open
- HAproxy transfers all the traffic from the incoming 8443 port to port 443 in the pf9-mplane
- IPTables assists in getting traffic from the
externalIp
config in the config file to the HaProxy

Review the following command to ensure the setup is correct, and the traffic is being correctly directed from the external port 443 on “externalIp” to the HaProxy.