Nodelet
A nodelet is a small software agent installed and run on each node as part of the Platform9 Managed Kubernetes (PMK) stack within a cluster. The nodelet agent provides multiple functions on both the Primary/Master and the nodes. This includes the installation and configuration of multiple Kubernetes services.
Syntax
[root@server ~]# nodeletd phases [command]
Nodelet Phases
Generate Certificates
Role:
Generates prerequisites checks needed to install various certificates.
Prepare Kube Configs
Role:
Customizes the kubeconfigs needed to start the Kubernetes cluster.
Docker Configure
Role:
Installs and configures docker and containerd.
Docker Start
Role:
Installs and verifies running docker containers.
Etcd Configure
Role:
Verifies, configures, and runs etcd on the primary host server's filesystem.
Etcd Run
Role:
Starts and confirms the etcd service is running on the container.
Network Configure
Role:
Ensure that the CIDR configuration for flannel is up-to-date (It does not target other network plugins like Calico, Canal and Weave).
CNI Configure
Role:
Configures the Container Network Interface (CNI).
Auth Webhook
Role:
Uses bouncer as a simple webhook endpoint server to validate/authenticates images created within the Kubernetes clusters (specifically, the admission controllers GenericAdmissionWebhook and the ValidatingAdmissionWebhook).
Misc Scripts
Role:
Responsible for composing the cloud provider config on the filesystems of all nodes.
Kubelet Configure/Start
Role:
Starts and manages the proper configurations on Kubelets.
Kube Proxy Start
Role:
Starts and configures the kube-proxy service.
Wait for K8s Services
Role:
Starts and pauses various K8s services to ensure availability.
Label and Taint Node
Role:
Designates "master" or "worker" nodes. Additionally, taints workloads not allowed on master
Dynamic Kubelet Configuration
Role:
Creates a ConfigMap in the kube-system namespace which defines the default kubelet type as either a "master" or "worker" node.
Uncordon Node
Role:
Marks nodes as schedulable using the kubectl uncordon node command
Deploy App Catalog
Role:
Configures and deploys the Monocular and Tiller services
Configure/Start Keepalived
Role:
Configures and starts the KeepAlive service
Deploy Luigi Operator
Role:
Activates the Luigi Networking Operator.
Deploy KubeVirt
Role:
Deploys KubeVirt operator in addition to its other custom resources.
Enable PF9 Sentry
Role:
Initiates and configures the pf9-sentry service within the platform9-system namespace
Enable PF9 Add-on Operator
Role:
Starts and configures the pf9-addon-operator service within the pf9-addons namespace
Drain All Pods (Stop Only)
Role:
If invoked, this task drains the node before implementing a stop function on other tasks. When the _pf9-kube_ service begins draining the node, it executes a priority stop function. This ensures the task is prioritized over the stop function of other tasks.
CLI
The following section specifies the nodeletd phase related commands used to interact with the k8s stack via the command line.
/opt/pf9/nodelet/nodeletd phases --helpCommands related to phases related to bring up of k8s stackUsage: nodeletd phases [command]Available Commands: list Lists the phases and their index numbers to use with rest of commands restart restarts pf9 kube stack. Takes optional --phase param to allow restarting from the specific phase start starts pf9 kube stack. Takes optional --phase param to allow starting from the specific phase status checks the status of Platform9 Kube on this host. Takes optional --phase param to check the status of a specific phase stop stops pf9 kube stack. Takes optional --phase param to allow stopping till the specific phaseFlags: -h, --help help for phasesUse "nodeletd phases [command] --help" for more information about a command.List Phases
The nodeletd service shows the phases flag list functions.
/opt/pf9/nodelet/nodeletd phases list INDEX NUMBER FILE NAME STATUS CHECK 1 020-gen_certs.sh Generate certs/Send signing request to CA true 2 030-prepare_kube_configs.sh Prepare configuration false 3 040-docker_configure.sh Configure Docker false 4 045-docker_start.sh Start DockerStop
These nodeletd phase options stop the pf9-kube stack.
/opt/pf9/nodelet/nodeletd phases stop/opt/pf9/nodelet/nodeletd phases stop --till-phase <index>Start
These nodelet phase options start the pf9-kube stack.
/opt/pf9/nodelet/nodeletd phases start/opt/pf9/nodelet/nodeletd phases start --from-phase <index>Restart
These options restart the pf9-kube stack.
/opt/pf9/nodelet/nodeletd phases restart/opt/pf9/nodelet/nodeletd phases restart -p <index>Status
This option provides information on the condition and state of the pf9-kube stack.
/opt/pf9/nodelet/nodeletd phases status --verboseINDEX NUMBER FILE NAME PHASE STATUS 1 020-gen_certs.sh Generate certs/Send signing request to CA running 2 030-prepare_kube_configs.sh Prepare configuration N/A 3 040-docker_configure.sh Configure Docker N/A 4 045-docker_start.sh Start Docker N/ANote: CLI output will contain info about the various phases that run before the following table is displayed. This info is also contained in the /var/log/pf9/kube/kube.log file.
Node Health
This curl command provides an exhaustive overview of a specific node. A sample output is shown below.
curl -H "X-Auth-Token: $TOKEN" -H "Content-Type: application/json" https://$DU/resmgr/v1/hosts | jq '.[] | select(.extensions.pf9_kube_status.data.pf9_cluster_id |contains("'$UUID'")) | .extensions.pf9_kube_status.data.pf9_kube_node_state'{ "pf9_kube_start_attempt": 0, // Number of start attempts till now "last_failed_status_check": "", //* "pf9_cluster_role": "master", "last_failed_task": "", // The task/phase script that failed on (pf9_kube_start_attempt-1) attempt "all_tasks": [ "Generate certs / Send signing request to CA", "Prepare configuration", "Configure Docker", "Start Docker", "Configure etcd", "Start etcd", "Network configuration", "Configure CNI plugin", "Configure and start auth web hook / pf9-bouncer", "Miscellaneous scripts and checks", "Configure and start kubelet", "Configure and start kube-proxy", "Wait for k8s services and network to be up", "Apply and validate node taints", "Apply dynamic kubelet configuration", "Uncordon node", "Validate k8s DNS", "Deploy dashboard", "Deploy app catalog", "Deploy metrics server", "Configure and start Keepalived", "Configure and start MetalLB", "Configure and start Autoscaler", "Configure and start pf9-sentry", "Drain all pods (stop only operation)" ], "pf9_kube_node_state": "ok", // ** "current_status_check": "", "completed_tasks": [ "Generate certs / Send signing request to CA", "Prepare configuration", "Configure Docker", "Start Docker", "Configure etcd", "Start etcd", "Network configuration", "Configure CNI plugin", "Configure and start auth web hook / pf9-bouncer", "Miscellaneous scripts and checks", "Configure and start kubelet", "Configure and start kube-proxy", "Wait for k8s services and network to be up", "Apply and validate node taints", "Apply dynamic kubelet configuration", "Uncordon node", "Validate k8s DNS", "Deploy dashboard", "Deploy app catalog", "Deploy metrics server", "Configure and start Keepalived", "Configure and start MetalLB", "Configure and start Autoscaler", "Configure and start pf9-sentry", "Drain all pods (stop only operation)" ], "pf9_kube_service_state": "true", "all_status_checks": [ "Generate certs / Send signing request to CA", "Start Docker", "Start etcd", "Network configuration", "Configure and start auth web hook / pf9-bouncer", "Miscellaneous scripts and checks", "Configure and start kubelet", "Configure and start kube-proxy", "Wait for k8s services and network to be up", "Configure and start Keepalived" ], "last_failed_status_time": 0, //* "pf9_cluster_id": "37ba60bb-1a36-4f78-8b83-528adea459bf", "current_task": "", "status_check_timestamp": 1594197441 //*}Note:
The last_failed_status_check field is cleared 10 minutes after the status check is successful.
The pf9_kube_service_state tries to simulate the node state as reported by the hostAgent. The values this field can report on are as follows:
| Status | Description |
|---|---|
| OK | Everything is fine. |
| Converging | Starting pf9-kube failed and this is the initial attempt to restart it. |
| Retrying | Starting pf9-kube failed and Nodelet has tries less than 10 times to start pf9-kube. |
| Failed | Starting pf9-kube failed and Nodelet has tried more than 10 times to start pf9-kube. |
Advanced / Experimental Features
The advanced and/or experimental features listed below are not advised for use in a production environment, and should only be used for testing purposes or as a "last-resort". Enterprise customers should consult Platform9 Support prior to running any of the commands listed below.
Single Phase Operation
Clients can operate on a single phase by specifying the --single argument to start/stop/restart operation.
/opt/pf9/nodelet/nodeletd phases stop -p <index number> --singleForce Stop
By default, Nodelet will abort a "stop" operation if one of the phases fails to “stop” cleanly.
This action can be overridden by adding the --force flag to the stop operation.
/opt/pf9/nodelet/nodeletd phases stop --force