Common Issues

The purpose of this page is to provide solutions to common issues during Community Edition (CE) installation & operation.

Resolving failed prerequisites checks

Swap enabled

Swap must be disabled on the CE host for the installation to succeed successfully. To disable swap, follow these steps:

1.) Turn off swap by executing the following command:

sudo swapoff -a

2.) Prevent swap from being enabled on reboot by commenting out the line(s) in /etc/fstab . Open the /etc/fstab file with a text editor using the command sudo nano /etc/fstab . Use the arrow keys to move the cursor to the beginning of the line that configures swap, and add a # as the first character in the line to comment it out. Repeat as needed for any other lines designating swap space. Press CTRL + X to exit nano, and press Y to save the changes.

Ports should not be in use

The CE installer expects to be installed in a dedicated system with the following ports available: 443, 2379, 2380, 3306, 4194, 5395, 5672, 5673, 6264, 8023, 8158, 8285, 8558, 9080, 10250, 10255. Typically, this error will be encountered after a previous installation failure. To completely clean up an installation failure, use airctl delete-cluster and then retry the installation.

# Deletes the CE installation, including k3s

sudo airctl delete-cluster --config /opt/pf9/airctl/conf/airctl-config.yaml

# deletes logs & downloaded artifacts in the user's home directory

rm -r "$HOME/airctl-logs"

rm -r "$HOME/pcd-ce"

# runs the install script from the internet

curl -sfL https://go.pcd.run | bash

Insufficient CPU or RAM

The CE install script checks the number of logical CPUs available at system start. If the install fails this check, run the installation again after provisioning more logical CPUs to the system (if running as a virtual machine) or on physical hardware with more logical CPUs available.

The install script calculates the amount of usable memory using the MemTotal line in /proc/meminfo , and checks for an amount less than 28GB. Additional memory must be made available to the CE system.

Multiple default routes with the same metric

The CE install script checks for more than one default IPv4 route with the same metric. If more than one route with the same metric is found, this check will fail, as the system will not know which default gateway to use. To resolve this error, either change the metric on the additional default routes, or delete any additional default routes with the same metric if they aren't needed.

# list all IPv4 routes

ip -4 route show

# Change the metric on the additional default route(s)

sudo ip route change default via <gateway-ip> metric <new-metric>

# Delete the additional default route(s) if they are not needed

sudo ip route del default via <gateway-ip>

Unsupported OS or version

The CE install script checks the operating system name and version. If this check fails, please try installing using a supported operating system and version as described in the CE Pre-requisites.

Kernel panic or vm panic on oom

The CE install script checks to make sure kernel & oom (out of memory) panics are configured correctly. /proc/sys/kernel/panic should be set to 10 and /proc/sys/vm/panic_on_oom should be set to 0 .

Failed to check firewalld status or service not found

The CE install script checks systemctl to see if firewalld.service is inactive or not running, and will fail if the service is set to active. To resolve this, use the following commands.

# Stop the firewalld service

sudo systemctl stop firewalld

# Prevent it from starting on boot

sudo systemctl disable firewalld

IPv6 is disabled

The CE install script checks /proc/sys/net/ipv6/conf/all/disable_ipv6 to see if it is set to 0 . IPv6 should be enabled by default, but if it has been explicitly disabled, please enable IPv6 and reboot the CE system before trying the installation again.

Resolving installation failures

Calico CNI

CE installation can fail if it encounters issues with Calico CNI (container networking interface). Calico sets up networking within CE's Kubernetes pods and provides DNS resolution. Typically, these issues are either related to slowness when retrieving the container images from a remote image repository causing the install script to timeout while waiting, or are related to a previous installation not being fully cleaned up before an installation is attempted again. These issues can usually be resolved by fully cleaning up the installation attempt and trying again.

You may also consider setting the install's default timeout to an amount higher than 600 seconds. See the Extending default deployment timeout section of the Custom Installation page.

# Deletes the CE installation, including k3s

sudo airctl delete-cluster --config /opt/pf9/airctl/conf/airctl-config.yaml

# deletes logs & downloaded artifacts in the user's home directory

rm -r "$HOME/airctl-logs"

rm -r "$HOME/pcd-ce"

# runs the install script from the internet

curl -sfL https://go.pcd.run | bash

Failed to apply logrotation on node - exit status 127

The CE installation will fail if it is unable to create a cronjob to automatically rotate log files, and will return exit status 127 if cron is not installed on the system. This is a default package on the Ubuntu 22.04 AMD64 server cloud image distribution, but is not included with the Ubuntu 22.04 AMD64 minimal cloud image. Installing cron will resolve this issue, but using the minimal cloud image is not supported and doing so may result in additional install failures.

Curl responds with an error, or does not work

This is typically caused by network restrictions such as firewalls or lack of internet access. Using curl with verbose mode enabled (-v ) is useful for debugging.

# Example

curl -v https://go.pcd.run | bash

Installation fails due to curl timeouts

This issue typically occurs early in the installation process, and can happen when the CIDRs used for Kubernetes services & pods conflicts with external local area networks. The default for the services CIDR is 10.21.0.0/16, and the default for the pods CIDR is 10.20.0.0/16. See the Community Edition deployment networking section on the Custom Installation page for information on how to change these defaults using environment variables. After the environment variables are set, use the same terminal session to follow the steps in the section titled "Recovering a failed installation" on this page.

Failed to find running pod percona-db-pxc-db-pxc-0

This error can occur when installing Community Edition on CPUs older than Intel Nehalem or AMD Bulldozer generations, as percona requires the x86-64-v2 microarchitecture level. This can be confirmed by running the following command:

sudo /lib64/ld-linux-x86-64.so.2 --help

Look for Subdirectories of glibc-hwcaps directories in the output.The x86-64-v2 line should not say supported, searched .

Please try installing Community Edition again on CPUs newer than Intel Nehalem or AMD Bulldozer generations.

Required ports & external URLs

Required ports:

• TCP: 53, 443
• UDP: 53, 123

Should you need to allow access to external URLs in order to complete a Community Edition installation, here are all of the URLs that are currently accessed during install:

api2.amplitude.com

auth.docker.io

cdn.dl.k8s.io

cdn01.quay.io

check.percona.com

checkpoint-api.hashicorp.com

cr.fluentbit.io

dl.k8s.io

dockermirror.platform9.io

docs.tigera.io

ghcr.io

github.com

go.pcd.run

grafana.com

opencloud-dev-charts.s3.us-east-2.amazonaws.com

pcd-community.s3-accelerate.amazonaws.com

pkg-containers.githubusercontent.com

prod-registry-k8s-io-us-east-1.s3.dualstack.us-east-1.amazonaws.com

production.cloudflare.docker.com

pypi.org

quay.io

registry-1.docker.io

registry.k8s.io

release-assets.githubusercontent.com

storage.googleapis.com

us-east4-docker.pkg.dev

usage.projectcalico.org

Here is the list of required external URLs in order to onboard a hypervisor host to Community Edition.

esm.ubuntu.com

files.pythonhosted.org

github.com

motd.ubuntu.com

pcdctl.s3.us-west-2.amazonaws.com

pypi.org

www.python.org

Recovering a failed installation

Situation: A Community Edition installation has failed, the issue has been rectified, and you'd like to restart the installation.

Resolution: You must first uninstall/delete the CE installation (and k3s) using the following command, and then run the install script again.

# Fully delete CE installation & k3s

/opt/pf9/airctl/airctl delete-cluster --config /opt/pf9/airctl/conf/airctl-config.yaml

# Run install script again

curl -sfL https://go.pcd.run | bash

After these steps have completed successfully, you may complete the rest of the deployment process starting with the Local DNS Entries section in Private Cloud Director: Community Edition.

General troubleshooting

For CE hosts:

• Check the install logs at airctl-logs/airctl.log

• kubectl describe node

  • The allocated resources block should show requests CPU and memory near or at 100%, if resources are constrained.

• kubectl get pods -A Look for any pods that are not Running or Completed. Pods in CrashLoopBackOff or with high restart counts and a young age should have their logs investigated for more detail.

• kubectl logs <pod> -n <namespace> to view pod logs.

• The default Kubernetes namespaces are pcd for the infrastructure region and pcd-community for the workload region, unless DU_FQDN and/or the REGION_NAME environment variables were customized prior to installation as described in the Deployment URL & region name section of Custom Installation.