Support Bundles

Support bundles is an archive generated for sole purpose of submitting to the Platform9 support team. There are two types of support bundles

• One for the Management workstation and Management Plane
• Other for the host components

Management Workstation

For the Management workstation, using the airctl command provides a method to generate a support bundle. Below are the full options available when using airctl to create the bundle.

$ ./build/airctl gen-support-bundle --help

generate support bundle

​

Usage:

  airctl gen-support-bundle [flags]

​

Flags:

      --full         retrieve full DU syslogs [large]

  -h, --help         help for gen-support-bundle

      --out string   the output file name (default "/tmp/supportbundle.tgz")

      --skip-du      skip adding du logs

      --skip-state   skip adding state information

​

Global Flags:

      --config string   config file (default is $HOME/airctl-config.yaml)

      --json            json output for commands (configure-hosts only currently)

      --verbose         print verbose logs to the console

By default, the support bundle generates information that includes:

• ~/airctl-logs/: typically in the home directory of the designated user contains the logs for the airctl command line runs, it is useful for starting, upgrade, rollback, backup and other tasks on the management workstation. This directory and contents are useful for debugging the management plane's lifecycle. It should also be noted that the airctl logs are written to the path defined by logDir parameter in the airctl configuration file. The default location for this directory is in the airctl user's home directory. If the user is unable to locate the logs in the airctl user's home directory, then the logs will be present in the directory defined by the logDir parameter in the config file.
• ~/.airctl/ : typically in the home directory of the designated user, it contains the 'state' information of the DU that includes the mysql and other configuration backup. This may contain sensitive information and the inclusion can be skipped with "--skip-state" flag.
• <du>/var/log/pf9: These are the logs from inside the Deployment Unit (or DU) VM and can help debug problems when upgrading the DU or if DU start or debugging other issues users may encounter after the DU is started or upgraded. If you want to skip the DU log, you can use "--skip-du" flag.
• <du> syslogs: There is an option to include all the DUs syslog info which can be achieved by adding the _--full_ flag to the command line.

Cluster Host

There are multiple troubleshooting options available when supporting remote nodes. These include using a Support Bundle or utilizing the Platform9 Advanced Remote Support option (via SSH mode).

A support bundle is an archive generated on a host and contains useful logs from the cluster nodes. It is then sent over Platform9’s management plane to aid in diagnosing and troubleshooting host issues. For many Kubernetes issues, the end-user first step should be to create a support bundle for the affected node(s).

The bundle contains two main components:

• A pre-defined list of host processes detailed within log files.
• The CLI output gathered during the support bundle generation

The support bundle mechanism is pluggable, which renders the ability to add commands, additional logs, or other data to be easily captured during the archive process using the steps outlined below.

Method 1: Generating a Host Support Bundle

The support bundle can be easily generated from the DU VM and can be invoked as part of the resource manager API on the management plane. To accomplish this, we log in and identify the cluster node’s UUID with the following commands.

Step 1. Login to the DU VM from the deployment host.

#ssh [IP-of-DU-VM]

ssh 192.168.120.254

Step 2. Then, sudo up to the root user account.

sudo -i

Step 3. Next, source the rc file to execute the file changes.

source admin_admin.rc

Step 4. Finally, list the nodes connected with the DU

/opt/pf9/du-tools/du-ctl/du_ctl --format table host list

|                  id                  |          hostname         |       ip_addresses       | responding | role_status |    roles    |

+--------------------------------------+---------------------------+--------------------------+------------+-------------+-------------+

| 01167acf-8ba3-4bd8-9ddb-dcc87a2fc96d |   qnecc058c.pf9.net   |   flannel0:10.20.45.0    |    True    |      ok     |   pf9-kube  |

|                                      |                           |     cni0:10.20.45.1      |            |             |             |

|                                      |                           |  bond0.1120:10.47.1.64   |            |             |             |

|                                      |                           |     p1p1:10.44.4.176     |            |             |             |

|                                      |                           |                          |            |             |             |

| 01ab5358-4d89-43ee-9e38-fd17dce929e6 |   qnecc011b.pf9.net   |   flannel0:10.20.96.0    |    True    |      ok     |   pf9-kube  |

|                                      |                           |     cni0:10.20.96.1      |            |             |             |

|                                      |                           |  bond0.1120:10.47.0.127  |            |             |             |

|                                      |                           |     em1:10.44.3.156      |            |             |             |

|                                      |                           |                          |            |             |             |

Step 5. Now generate the support bundle for the desired host

/opt/pf9/du-tools/du-ctl/du_ctl host support bundle --host_id f71c8df2-8b03-4d4d-8123-e9e711762d61

Waiting for support bundle...

Waiting for support bundle...

Wrote new bundle: /opt/pf9/support/f71c8df2-8b03-4d4d-8123-e9e711762d61/ns6020087.ip-178-33-61.eu-2018-01-10-22-52-37.tgz

Method 2: Generating a Host Support Bundle

If a timeout or other issue occurs when using the first method, the following method can be used to create a support bundle. The following command can be run to pull a support bundle via a Sidekick tunnel.

/opt/pf9/du-tools/du-ctl/du_ctl sidekick support bundle --host_id <host_uuid>

Example:

[root@airctl-5.pf9.localnet ~(admin)]# /opt/pf9/du-tools/du-ctl/du_ctl sidekick support bundle --host_id 1f6fc846-ad6c-473f-b419-f6dff1fce78f

Support bundle requested for 1f6fc846-ad6c-473f-b419-f6dff1fce78f

​

[root@airctl-5.pf9.localnet tmp(admin)]# ls -lrt bundle*

-rw-r--r-- 1 pf9 users 42236390 Apr  9 01:49 bundle_1f6fc846-ad6c-473f-b419-f6dff1fce78f_1617932949.tgz

Advanced - Add other Log files to the Support Bundle

The support bundle generation process captures output from the following locations on the host:

• /etc/pf9/*
• /var/log/pf9/*

If the service does not specifically log to the locations noted above or have a persistent log on the local filesystem, we can evaluate the following options:

1. Create a link to the log files under the /var/log/pf9 directory
2. Copy existing log files from another location to the /var/log/pf9 directory
3. Run additional combined commands during log bundle generation to generate log files under /var/log/pf9 folder (see steps below)

Capturing Additional Output From a Remote Host in the Support Bundle

As a component of the support bundle generation procedure, ancillary scripts can be applied from a specific directory which follow a predictable naming convention. If we choose to run these scripts as a component of the support bundle generation, best practice dictates the following processes.

• Create file(s) containing the commands or scripts to execute during the bundle generation. The output of the commands should be written to stdout or stderr by the file.
• The file(s) can be in any programming language.
• Ensure that this file has the corresponding privileges of the pf9 user, as the file is executed in the pf9 user’s context.
• The file output should be directly under the /opt/pf9/support directory on the host and not under a subdirectory.
• The stdout, stderr file output is what is captured in the support bundle.
• Use a timeout to kill any commands that might hang. The framework has a grace period of 3 mins per script before the process is killed. The ideal duration of each script should run in less than a few seconds.

See the section below on how to obtain those privileges for the pf9 user.

Sample script

#!/bin/bash

​

# Show the status from the proc folder

svcname="pf9-hostagent"

pid=`cat /var/run/$svcname.pid`

​

digits_re="[0-9]+"

if [[ "$pid" =~ $digits_re ]] && [ -d "/proc/$pid" ] && [ -f /proc/$pid/status ]; then

    ( set -x && cat /proc/$pid/status )

else

    echo "Could not find proc status using PID: $pid" 1>&2

fi

​

set -x

​

service --status-all

ps aux | grep pf9

​

systemctl list-units --all --no-pager

systemd-cgls --no-pager

Command Output to Support Archive

Once a support bundle has been produced, the output of these commands can be read by extracting the archive to a temporary directory, and then explore the directory <extracted_archive_dir>/var/log/pf9/support/<command_file_name>.

Uploading Support Bundles and Logs

Data Compression

Once all the information has been gathered, it needs to be shared with Platform9 Support or your internal IT/DevOps department. The most efficient method is to utilize both the tar and gzip commands to compress the multiple data entries into a more manageable format. The syntax we should use is as follows. This will recursively compress all the data within the folder.

tar -czvf name-of-archive.tar.gz /path/to/directory-or-file

To break down the above example, we define the switches below.

-c: Creates the archive

-z: Compress that archive using gzip

-v: This switch display the progress in the terminal while the archive is being creating, This is also known as verbose mode. The -v switch is technically optional when using this command, but it can be helpful to visualize its progress.

-f: This allows the user to define specific filename for the archive

To compress multiple files and folders, simply add a list of files and folders you need to compress. For example, using the command below, we compress folder1, folder2, as well as datafile1.txt and datafile2.txt files.

tar -czvf archive.tar.gz /home/user/folder1 /usr/local/folder2 /home/user/Documents/datafile1.txt /home/user/Downloads/datafile2.txt

Upload Log Bundles

Once we have the data compressed into a single file, we can then upload that file using the following command.

curl -T <filename> https://s3-us-west-1.amazonaws.com/uploads.platform9.com/YourUserName/zd-tkt-number/filename

For example, below is the command we would use to upload bundle9c11bbb4-53b2-2379-92ab-66621300f915 1631111111.tgz, referencing Zendesk ticket 1352048, associated with the "YourUserName" account.

https://s3-us-west-1.amazonaws.com/uploads.platform9.com/YourUserName/1352048/bundle_9c11bbb4-53b2-2379-92ab-66621300f915_1631111111.tgz

FAQ

• Customers may also create internal templates for uploading logs and command outputs to Platform9 based on the commands provided above
• To understand more about the important log files that can be referred to troubleshoot issues with your cluster nodes, refer to the following link Important log Files in Platform9 Managed Kubernetes
• For General Documentation about Platform9 Managed Kubernetes, refer to Platform9 Managed Kubernetes Documentation.
• For faster resolution of issues with Platform9 Managed Kubernetes also refer to our extensive Knowledge Base,