Hardening the OS in Kubernetes Clusters is A Critical Security Requirement
A critical security requirement for our customers is to harden their servers to reduce the attack surface by removing unnecessary software and by configuring the remaining components so an attacker has fewer opportunities to compromise the server.
The Center For Internet Security
A non-profit organization called Center of Internet Security (CIS) was formed in 2000 with the mission to “identify, develop, validate, promote, and sustain best practice solutions for cyber defense”. CIS provides benchmarks and best practices for securing the configuration of systems for different vendors, developed through a unique consensus-based process compromised of cybersecurity professionals and subject matter experts around the world.
CIS Benchmarks are consensus-based, best-practice security configuration guides developed and accepted by the government, business, industry, and academia.
Platform9 Hardening Example
In this guided tutorial, I’m sharing specific steps on“how to harden Ubuntu 20.04” based on CIS Ubuntu 20.04 L1 L2 v1.0.0 benchmark, leveraging Ansible playbooks, so hardened servers can be used as Worker and Master nodes.
Instructions
The following Ansible playbook will harden an Ubuntu 20.04 server based on CIS Ubuntu benchmark.
The next procedure will use an Ubuntu 20.04 to trigger the ansible playbook.
1. Download the following file
2. Unzip the file
ubuntu@ansible00:~$ unzip Ubuntu2004-CIS-PF9.zipThe zip file contains a folder which represent an Ansible role the we will invoke from another file that is created in step 4, the before mentioned role contains tasks files for each of the 6 sections of the hardening phases for Ubuntu 20.04 CIS v1.0.0.
Note: The following table describes what actions controlled by the variables declared and evaluated to True in defaults/main.yaml are going to take place via the Ansible playbook.
- name "1.1.1.1 | Ensure mounting of cramfs filesystems is disabled"
- name "1.1.1.1 | Remove cramfs module"
- name "1.1.1.2 | Ensure mounting of freevxfs filesystems is disabled"
- name "1.1.1.2 | Remove freevxfs module"
- name "1.1.1.3 | Ensure mounting of jffs2 filesystems is disabled"
- name "1.1.1.3 | Remove jffs2 module"
- name "1.1.1.4 | Ensure mounting of hfs filesystems is disabled"
- name "1.1.1.4 | Remove hfs module"
- name "1.1.1.5 | Ensure mounting of hfsplus filesystems is disabled"
- name "1.1.1.5 | Remove hfsplus module"
- name "1.1.1.6 | Ensure mounting of udf filesystems is disabled"
- name "1.1.1.6 | Remove udf module"
- name "1.1.2 | Ensure separate partition exists for /tmp | enable and start/restart tmp.mount"
- name "1.1.2 | Ensure separate partition exists for /tmp | enable and start/restart tmp.mount"
- name "1.1.3 | Ensure nodev option set on /tmp partition\n
- name "1.1.7 | Ensure nodev option set on /dev/shm partition\n
- name "1.1.10 | Ensure separate partition exists for /var"
- name "1.1.11 | Ensure separate partition exists for /var/tmp"
- name "1.1.12 | Ensure nodev option set on /var/tmp partition\n
- name "1.1.15 | Ensure separate partition exists for /var/log"
- name "1.1.16 | Ensure separate partition exists for /var/log/audit"
- name "1.1.17 | Ensure separate partition exists for /home"
- name "1.1.18 | Ensure nodev option set on /home partition"
- name "1.1.19 | Ensure nodev option set on removable media partitions"
- name "1.1.20 | Ensure nosuid option set on removable media partitions"
- name "1.1.21 | Ensure noexec option set on removable media partitions"
- name "1.1.22 | Ensure sticky bit is set on all world-writable directories"
- name "1.1.23 | Disable Automounting"
- name "1.1.24 | Ensure USB storage is disabled"
- name "1.1.24 | Remove usb-storage module"
- name "1.2.1 | Ensure package manager repositories are configured"
- name "1.2.2 | Ensure GPG keys are configured"
- name "1.3.2 | Ensure sudo commands use pty"
- name "1.3.3 | Ensure sudo log file exists"
- name "1.4.1 | Ensure AIDE is installed (install nullmailer instead of postfix)"
- name "1.4.1 | Ensure AIDE is installed"
- name "1.4.1 | Stat AIDE DB"
- name "1.4.1 | Init AIDE | This may take a LONG time"
- name "1.4.2 | Ensure filesystem integrity is regularly checked"
- name "1.5.1 | Ensure bootloader password is set - generate password"
- name "1.5.1 | Ensure bootloader password is set - generate config"
- name "1.5.1 | Ensure bootloader password is set - disable password for system boot"
- name "1.5.2 | Ensure permissions on bootloader config are configured"
- name "1.5.3 | Ensure authentication required for single user mode"
- name "1.6.1 | Ensure XD/NX support is enabled"
- name "1.6.2 | Ensure address space layout randomization (ASLR) is enabled"
- name "1.6.3 | Ensure prelink is disabled"
- name "1.6.3 | Ensure prelink is disabled"
- name "1.6.4 | Ensure core dumps are restricted"
- name "1.6.4 | Ensure core dumps are restricted"
- name "1.7.1.1 | Ensure SELinux or AppArmor are installed"
- name "1.7.1.2 | Ensure AppArmor is enabled in bootloader configuration"
- name "1.7.1.3 | AppArmor - Ensure no unconfined daemons exist"
- name "1.7.1.4 | Ensure all AppArmor Profiles are enforcing"
- name "1.8.1.1 | Ensure message of the day is configured properly"
- name "1.8.1.2 | Ensure local login warning banner is configured properly"
- name "1.8.1.3 | Ensure remote login warning banner is configured properly"
- name "1.8.1.4 | Ensure permissions on /etc/motd are configured"
- name "1.8.1.5 | Ensure permissions on /etc/issue are configured"
- name "1.8.1.6 | Ensure permissions on /etc/issue.net are configured"
- name "1.9 | Ensure updates, patches, and additional security software are installed"
- name "1.10 | Ensure GDM login banner is configured"
- name "2.2.1.1 | Ensure time synchronization is in use"
- name 2.2.1.2 Ensure systemd-timesyncd is configured
- name "2.2.1.3 | Ensure chrony is configured"
- name "2.2.1.3 | Ensure chrony is configured"
- name "2.2.1.3 | Ensure chrony is configured | modify /etc/sysconfig/chronyd | 1"
- name "SCORED | 2.2.1.4 | PATCH | Ensure ntp is configured"
- name "2.2.2 | Ensure X Window System is not installed"
- name "2.2.3 | Ensure Avahi Server is not enabled"
- name "2.2.4 | Ensure CUPS is not enabled"
- name "2.2.5 | Ensure DHCP Server is not enabled"
- name "2.2.6 | Ensure LDAP server is not enabled"
- name "2.2.7 | Ensure NFS and RPC are not enabled"
- name "2.2.8 | Ensure DNS Server is not enabled"
- name "2.2.9 | Ensure FTP Server is not enabled"
- name "2.2.10 | Ensure HTTP server is not enabled"
- name "2.2.11 | Ensure IMAP and POP3 server is not enabled"
- name "2.2.12 | Ensure Samba is not enabled"
- name "2.2.13 | Ensure HTTP Proxy Server is not enabled"
- name "2.2.14 | Ensure SNMP Server is not enabled"
- name "2.2.15 | Ensure mail transfer agent is configured for local-only mode"
- name "2.2.16 | Ensure rsync service is not installed"
- name "2.2.17 | Ensure NIS Server is not enabled"
- name "2.3.1 | Ensure NIS Client is not installed"
- name "2.3.2 | Ensure rsh client is not installed"
- name "2.3.3 | Ensure talk client is not installed"
- name "2.3.4 | Ensure telnet client is not installed"
- name "2.3.5 | Ensure LDAP client is not installed"
- name "3.1.1 Disable IPv6"
- name 3.1.2 | Ensure wireless interfaces are disabled
- name "3.2.1 | Ensure packet redirect sending is disabled"
- name "3.2.2 | Ensure IP forwarding is disabled"
- name "3.3.1 | Ensure source routed packets are not accepted"
- name "3.3.2 | Ensure ICMP redirects are not accepted"
- name "3.3.2 | Ensure ICMP redirects are not accepted"
- name "3.3.3 | Ensure secure ICMP redirects are not accepted"
- name "3.3.4 | Ensure suspicious packets are logged"
- name "3.3.5 | Ensure broadcast ICMP requests are ignored"
- name "3.3.6 | Ensure bogus ICMP responses are ignored"
- name "3.3.7 | Ensure Reverse Path Filtering is enabled"
- name "3.3.8 | Ensure TCP SYN Cookies is enabled"
- name "3.3.9 | Ensure IPv6 router advertisements are not accepted"
- name "3.4.1 | Ensure DCCP is disabled"
- name "3.4.2 | Ensure SCTP is disabled"
- name "3.4.3 | Ensure RDS is disabled"
- name "3.4.4 | Ensure TIPC is disabled"
- name 3.5.1.3 Ensure ufw service is enabled
- name 3.5.1.4 Ensure loopback traffic is configured
- name 3.5.1.7 Ensure default deny firewall policy
- name "3.5.3.1.1 | Ensure iptables is installed"
- name "3.5.3.1.1 | Ensure iptables is installed and started"
- name "3.5.3.1.3 | Ensure iptables is installed"
- name "3.5.3.2.1 | Ensure default deny firewall policy"
- name "3.5.3.2.2 | Ensure loopback traffic is configured"
- name "3.5.3.2.3 | Ensure outbound and established connections are configured"
- name "3.5.3.2.4 | Ensure firewall rules exist for all open ports"
- name "4.1.1.1 | Ensure auditd is installed"
- name "4.1.1.2 | Ensure auditd service is enabled"
- name "4.1.1.3 | Ensure auditing for processes that start prior to auditd is enabled"
- name 4.1.1.4 Ensure audit_backlog_limit is sufficient
- name "4.1.2.1 | Ensure audit log storage size is configured"
- name "4.1.2.2 | Ensure audit logs are not automatically deleted"
- name "4.1.2.3 | Ensure system is disabled when audit logs are full"
- name "4.1.3 | Ensure events that modify date and time information are collected"
- name "4.1.4 | Ensure events that modify user/group information are collected"
- name "4.1.5 | Ensure events that modify the system's network environment are collected"
- name "4.1.6 | Ensure events that modify the system's Mandatory Access Controls are collected"
- name "4.1.7 | Ensure login and logout events are collected"
- name "4.1.8 | Ensure session initiation information is collected"
- name "4.1.9 | Ensure discretionary access control permission modification events are collected"
- name "4.1.10 | Ensure unsuccessful unauthorized file access attempts are collected"
- name "4.1.11 | Ensure use of privileged commands is collected"
- name "4.1.12 | Ensure successful file system mounts are collected"
- name "4.1.13 | Ensure file deletion events by users are collected"
- name "4.1.14 | Ensure changes to system administration scope (sudoers) is collected"
- name "4.1.15 | Ensure system administrator actions (sudolog) are collected"
- name "4.1.16 | Ensure kernel module loading and unloading is collected"
- name "4.1.17 | Ensure the audit configuration is immutable"
- name "4.2.1.1 | Ensure rsyslog or syslog-ng is installed"
- name "4.2.1.2 | Ensure rsyslog Service is enabled"
- name "4.2.1.3 | Ensure logging is configured"
- name "4.2.1.4 | Ensure rsyslog default file permissions configured"
- name "4.2.1.5 | Ensure rsyslog is configured to send logs to a remote log host"
- name "4.2.1.6 | Ensure remote rsyslog messages are only accepted on designated log hosts."
- name 4.2.2.2 | Ensure journald is configured to compress large log files
- name 4.2.2.3 | Ensure journald is configured to write logfiles to persistent disk
- name "4.2.3 | Ensure permissions on all logfiles are configured"
- name "4.3 | Ensure logrotate is configured"
- name "4.4 | Ensure logrotate assigns appropriate permissions"
- name "5.1.1 | Ensure cron daemon is enabled"
- name "5.1.2 | Ensure permissions on /etc/crontab are configured"
- name "5.1.3 | Ensure permissions on /etc/cron.hourly are configured"
- name "5.1.4 | Ensure permissions on /etc/cron.daily are configured"
- name "5.1.5 | Ensure permissions on /etc/cron.weekly are configured"
- name "5.1.6 | Ensure permissions on /etc/cron.monthly are configured"
- name "5.1.7 | Ensure permissions on /etc/cron.d are configured"
- name "5.1.8 | Ensure cron is restricted to authorized users"
- name "5.1.9 | Ensure at is restricted to authorized users"
- name "5.2.1 | Ensure permissions on /etc/ssh/sshd_config are configured"
- name "5.2.4 | Ensure SSH LogLevel is set to INFO"
- name "5.2.5 | Ensure SSH X11 forwarding is disabled"
- name "5.2.6 | Ensure SSH MaxAuthTries is set to 4 or less"
- name "5.2.7 | Ensure SSH IgnoreRhosts is enabled"
- name "5.2.8 | Ensure SSH HostbasedAuthentication is disabled"
- name "5.2.9 | Ensure SSH root login is disabled"
- name "5.2.10 | Ensure SSH PermitEmptyPasswords is disabled"
- name "5.2.11 | Ensure SSH PermitUserEnvironment is disabled"
- name 5.2.12 | Ensure only strong Ciphers are used
- name "5.2.13 | Ensure only approved MAC algorithms are used"
- name 5.2.14 Ensure only strong Key Exchange algorithms are used
- name "5.2.15 | Ensure SSH Idle Timeout Interval is configured"
- name "5.2.16 | Ensure SSH LoginGraceTime is set to one minute or less"
- name "5.2.17 | Ensure SSH access is limited"
- name "5.2.18 | Ensure SSH warning banner is configured"
- name "5.2.20 | Ensure SSH AllowTcpForwarding is disabled"
- name "5.2.21 | Ensure SSH MaxStartups is configured"
- name "5.3.1 | Ensure password creation requirements are configured"
- name "5.3.2 | Ensure lockout for failed password attempts is configured"
- name "5.3.3 | Ensure password reuse is limited"
- name "5.3.4 | Ensure password hashing algorithm is SHA-512"
- name "5.4.1.1 | Ensure password expiration is 365 days or less"
- name "5.4.1.2 | Ensure minimum days between password changes is 7 or more"
- name "5.4.1.3 | Ensure password expiration warning days is 7 or more"
- name "5.4.1.4 | Ensure inactive password lock is 30 days or less"
- name "5.4.2 | Ensure system accounts are secured"
- name "5.4.3 | Ensure default group for the root account is GID 0"
- name "5.4.4 | Ensure default user umask is 027 or more restrictive"
- name "5.4.5 | Ensure default user shell timeout is 900 seconds or less"
- name "5.5 | Ensure root login is restricted to system console"
- name "5.6 | Ensure access to the su command is restricted"
- name "6.1.1 | Audit system file permissions"
- name "6.1.2 | Ensure permissions on /etc/passwd are configured"
- name "6.1.3 | Ensure permissions on /etc/gshadow- are configured"
- name "6.1.4 | Ensure permissions on /etc/shadow are configured"
- name "6.1.5 | Ensure permissions on /etc/group are configured"
- name "6.1.6 | Ensure permissions on /etc/passwd- are configured"
- name "6.1.7 | Ensure permissions on /etc/shadow- are configured"
- name "6.1.8 | Ensure permissions on /etc/group- are configured"
- name "6.1.9 | Ensure permissions on /etc/gshadow are configured"
- name "6.1.10 | Ensure no world writable files exist"
- name "6.1.11 | Ensure no unowned files or directories exist"
- name "6.1.12 | Ensure no ungrouped files or directories exist"
- name "6.1.13 | Audit SUID executables"
- name "6.1.14 | Audit SGID executables"
- name "6.2.1 | Ensure password fields are not empty"
- name "6.2.2 | Ensure root is the only UID 0 account"
- name "6.2.3 | Ensure root PATH Integrity"
- name "6.2.3 | Ensure root PATH Integrity"
- name "6.2.3 | Ensure root PATH Integrity"
- name "6.2.4 | Ensure all users' home directories exist"
- name "6.2.5 | Ensure users' home directories permissions are 750 or more restrictive"
- name "6.2.6 | Ensure users own their home directories"
- name "6.2.7 | Ensure users' dot files are not group or world writable"
- name "6.2.8 | Ensure no users have .forward files"
- name "6.2.9 | Ensure no users have .netrc files"
- name "6.2.11 | Ensure no users have .rhosts files"
- name "6.2.12 | Ensure all groups in /etc/passwd exist in /etc/group"
- name "6.2.13 | Ensure no duplicate UIDs exist"
- name "6.2.14 | Ensure no duplicate GIDs exist"
- name "6.2.15 | Ensure no duplicate user names exist"
- name "6.2.16 | Ensure no duplicate group names exist"3. Update host file under Ubuntu2004-CIS-PF9/host with the the IP and user under the “servers” group, and assign the ansible_user=”user” accordingly.
4. Generate the following file at the same level of the Ubuntu2004-CIS-PF9 folder.
ubuntu@ansible00:~$ ls
Ubuntu2004-CIS-PF9 Ubuntu2004-CIS-PF9.zip pf9 playbook-local.ymlThe playbook-local.yml will call the Ubuntu2004-CIS-PF9 role, which triggers the 6 sections of the hardening process for Level 1 and Level 2 findings.
ubuntu@ansible00:~$ vi playbook-local.yml- name: Harden Server
hosts: servers
become: yes
roles:
- Ubuntu2004-CIS-PF95. Trigger the Ansible playbook to harden the Ubuntu node
ansible-playbook -i Ubuntu2004-CIS-PF9/host playbook-local.yml -K -kOnce the playbook finishes executing, the node will be hardened and ready to be onboarded and use it as a node to create a Cluster.
Next time the server is audited with any CSI auditing tool, it will mostly pass subject to any false positives and further investigation.
As part of the hardening process we should leverage a vendor that provides the least false positives during the auditing of an asset/server – in this occasion, we chose Nessus from the company Tenable, the leader in Vulnerability Management
The following report generated by Nessus shows 43 Failed, some of which are false positives and some others need to be addressed by the Infra teams. There are few failures that, if applied to harden the OS, would break the way kubernetes works overall. The report shows that we are in compliance with 384.
What are some Kubernetes CSI Benchmarking tools?
Though we chose Nessus from Tenable, customers may have another tool of their preference. Examples of other tools that could be worth exploring are:
CIS-CSAT
Rapid7
Qualys
Tenable.io
In the next blog post, I will talk about how to apply strict network policy rules to your namespaces so your applications and microservices only talk with who they are meant to in order to cover what we couldn’t harden via our Ansible playbook!
Platform9 is the only container management and orchestration solution that supports the complete lifecycle of the cloud-native journey on infrastructure anywhere. Our unique management plane simplifies operations, including observability & monitoring, increases uptime, minimizes costs, and scales as you scale. Built on open foundations, our platform enables enterprises to keep up with the ever-changing cloud-native ecosystem. Our cloud-native wizards provide Always-on Assurance™ every step of the way to ensure a smooth, simplified journey. Innovative enterprises like Juniper, Kingfisher Plc, Mavenir, Redfin, and Cloudera achieve 4x faster time-to-market, up to 90% reduction in operational costs, and 99.99% uptime.Platform9 is a better way to go cloud native, paving the way for an open distributed cloud
Latest posts by Platform9 (see all)
- Beyond Kubernetes Operations: Discover Platform9’s Always-On Assurance™ - November 29, 2023
- KubeCon 2023 Through Platform9’s Lens: Key Takeaways and Innovative Demos - November 14, 2023
- Getting to know Nate Conger: A candid conversation - June 12, 2023