Proxmox 8.1 released

Proxmox PVE 8.1: Proxmox has released version 8.1 of its virtualization environment. Version 8.1 is a consistent evolution of version 8.0.

Proxmox Virtual Environment, the leading open-source platform for server virtualization and enterprise cloud, has released version 8.1. This version brings many new features and improvements for managing virtual machines, containers, storage, networking, and clusters.

• Secure Boot Compatibility: Proxmox VE is now compatible with Secure Boot, a security feature that protects a computer’s boot process. Secure Boot ensures that only software with a valid digital signature is started. Proxmox VE now includes a signed shim bootloader that is accepted by most hardware UEFI implementations. This enables the installation of Proxmox VE in environments with Secure Boot enabled.

• Software-Defined Networking (SDN): Proxmox VE now includes the core packages for Software-Defined Networking (SDN) by default, a technology that enables dynamic and automated configuration of network resources. With SDN, network administrators can create and manage virtual zones and networks (VNets), controlling complex network configurations and multi-tenancy setups directly via the web interface at the “Datacenter” level. The use cases for SDN are diverse, from an isolated private network on each individual node to complex overlay networks spanning multiple Proxmox VE clusters across different locations. The benefits are a faster and more adaptable network infrastructure that can scale according to business requirements.

• New Flexible Notification System: Proxmox VE now offers a new framework that controls the routing of notifications. It allows selecting different targets as notification recipients. In addition to the previously standard local Postfix MTA, notifications can now also be sent to a Gotify server or an email directly via an SMTP server, including SMTP authentication. With predefined rules, matchers determine which targets receive which notifications for specific events. The new notification system is very flexible and allows precise configuration of when, where, and how notifications are sent.

• Support for Ceph Reef and Ceph Quincy: Proxmox VE supports Ceph Reef 18.2.0 as well as continuing support for Ceph Quincy 17.2.7. The preferred Ceph version can be selected during the installation process. Ceph Reef offers better default settings that improve performance and increase read speeds.

We are happy to assist you with the implementation of your Proxmox project.

You can find the detailed changelog here.

Enhancements in the web interface (GUI)

• Improvements to bulk actions:
  • Add a new “Bulk Suspend” action to suspend a selection of guests in one action.
  • Add a new section above the guest list for configuring guest filters and add a button for clearing filters.
  • Allow to filter guests by their tags.
  • Reorder fields and drop obvious warning about local storage to improve on screen-space utilization.
  • Reword the message displayed for bulk actions in the task log to “Bulk Start/Stop/Migrate”. The message shown previously was “Start/Stop/Migrate all” and could be misleading in case not all guests were affected (issue 2336).
  • The “Bulk Migrate” action is now hidden on standalone nodes, as there is no valid migration target in that case.
• Improvements to the node summary panel:
  • The summary now indicates whether the node was booted in legacy (BIOS) mode, EFI mode, or EFI mode with Secure Boot enabled.
  • The currently running kernel is now reported more compactly by indicating only the version and the build date.
• Allow to automatically decompress an ISO file when downloading from a URL to a storage. The decompression algorithm can be set in the GUI (issue 4849).
• Allow moving VMs and containers from one pool to a different pool in one operation.
• Avoid needlessly reloading the GUI after ordering a certificate via ACME for a different cluster node.
• The permission editor now also shows the ACL paths for notifications and PCI/USB mappings.
• The resource tree now displays the usage in percent when hovering over a storage.
• If the configured tree shape for tags is not “Full”, the resource tree now displays a tooltip with the tag name when hovering over the configured shape.
• Ensure the SPICE config is downloaded with the correct file extension on Safari to avoid problems with macOS application association (issue 4947)
• Fix an issue where the “Migrate” button stayed disabled even if selecting a valid target node.
• Fix a bug where the backup job editor window would add an invalid entry for VMID 0 when using selection mode “all”.
• Improve error message when creating a VM with an invalid hostname: Clarify that a valid hostname, not a valid DNS name, is expected (issue 4874).
• When uploading a subscription, ignore surrounding whitespace in the subscription key to avoid confusing errors.
• Improve the focus handling when editing tags to allow tabbing though editable tag fields.
• Allow adding tags already when creating VMs and containers.
• Increase height of the VM and container creation wizard to obtain a 4:3 ratio.
• When creating an IP/CIDR inside an IPSet, the GUI now requires that an IP/CIDR is provided. Previously, the GUI accepted an empty field, but the API threw an error.
• Update external links to proxmox.com that changed during the website redesign.
• Fix an issue where the OK button would stay disabled when editing an ACME DNS challenge plugin (issue 4531).
• Fix an issue where clicking “Reset” in the zpool creation window could cause an error when re-opening the window (issue 4951).
• Fix an issue where users could write notes with links containing JavaScript code. This JavaScript code would be executed when a (different) user clicked on such a link.
• HTML-encode API results before rendering as additional hardening against XSS.
• Improved translations, among others:
  • Croatian (NEW!)
  • Georgian (NEW!)
  • Arabic
  • Catalan
  • German
  • Italian
  • Polish
  • Simplified Chinese
  • Traditional Chinese
  • Ukrainian
  • Several remaining occurrences of the GiB unit in the GUI can now be translated (issue 4551).

Virtual machines (KVM/QEMU)

• New QEMU version 8.1.

See the upstream changelog for details.

• Add clipboard support to the VNC console in the GUI. For now the feature cannot be enabled in the GUI and has to be manually enabled via API or CLI. After installing the SPICE guest tools, text can be copied from/to the guest clipboard using the noVNC clipboard button.
• When creating a Windows VM, allow adding a second CD/DVD drive directly in the creation wizard.

This makes it easier to add the Windows VirtIO drivers ISO before starting the VM.

• Remove the 10-minute timeout for allocating VM disks when restoring from backup, as this timeout may be exceeded if disks are large or network storage is involved (issue 2817).
• Log a warning when starting a VM with a deprecated machine version.
• Fix issues where shutdown and reboot commands would time out (instead of failing immediately) on ACPI-suspended VMs.
• Enabling or disabling CPU hotplug for a running VM did not work reliably and is not allowed anymore.
• Avoid leaving potentially large amounts of memory assigned to the QEMU process after backup.
• Fix an issue where heavy network traffic or connection issues during a backup to Proxmox Backup Server could cause an unsolicited write to the first sector of a backed-up SATA disk, which usually contains the boot-sector (issue 2874).
• Fix an issue where a race condition could cause a VM crash during backup if iothread is enabled.
• Fix an issue where each pause and resume operation (for example when taking a snapshot) would increase the number of open file descriptors of the QEMU process, which could eventually lead to crashes.
• Fix an issue where starting a VM with machine type q35 and multiple IDE drives would fail.
• cloud-init: Fix issues where non-root users could not regenerate the cloud-init drive or set the ciupgrade option.
• Start VMs using PCI passthrough with a higher timeout that is calculated from the configured memory. Previously, the timeout was reported to be too short when using PCI passthrough.
• Fix an issue where qmeventd failed to retrieve VMIDs from processes on hybrid cgroup systems and logged errors to the journal.
• Fix an issue where remote migration would fail for certain combinations of source/target storage, for example from qcow2 on directory to LVM-thin.
• Fix an issue where backup of a VM template with a TPM would fail (issue 3963).
• Fix an issue where the VNC proxy would fail if the LC_PVE_TICKET was not set (issue 4522).
• Backports of several upstream kernel patches:
  • Fix an issue where VMs with a restricted CPU type could get stuck after live-migration from a host with kernel 5.15 to a host with kernel 6.2.
  • Fix an issue where VMs could get stuck after several days of uptime if KSM, ballooning, or both, were enabled.
  • The FLUSHBYASID flag is now exposed to nested VMs when running on an AMD CPU. This fixes an issue where some hypervisors running in a VM would fail to start nested VMs.
  • Fix an issue with recovering potential NX huge pages that resulted in a warning logged to the journal (issue 4833).
  • Fix an issue where only one NVMe device would be recognized even though multiple are present (issue 4770).

Containers (LXC)

• Support device passthrough for containers. The new dev0/dev1/… options take the path of host device. Optionally, owner and permission settings for the device node inside the container can be given. For now, the option cannot be set in the GUI and has to be manually set via API or CLI.
• Allow specifying multiple SSH keys in the container creation wizard (issue 4758).
• Show privileged status as a separate row in the guest status view in the GUI.
• Show distribution logo and name in the guest status view in the GUI.
• Fix an issue where network would fail to come up for Fedora containers.
• Add an API endpoint /nodes/{node}/lxc/{vmid}/interfaces for querying network interfaces of a running container.
• Improve architecture detection for NixOS containers, which would previously produce a warning and default to x86_64 in case /bin/sh did not (yet) exist in the container.
• The pct status command does not report guest CPU usage anymore, as there is currently no fast way to measure it (issue 4765).
• Restoring a container from a PBS backup now honors the ignore-unpack-errors flag (issue 3460).
• Fix an issue where Fedora containers would not have a container-getty on first boot.

General improvements for virtual guests

• Show progress of offline disk migration in the migration task log by use of dd’s status=progress argument (issue 3004).
• Proxmox VE now has an officially assigned OUI from the IEEE BC:24:11 to be used as product-specific MAC prefix. This is now used by default instead of assigning purely random MACs. (issue 4764).

HA Manager

• Notification for HA events, like fencing, are now configurable via the new modular notification system.
• An issue with the target selection during service recovery, where a fenced node was selected as target, was fixed (issue 4984).

Improved management for Proxmox VE clusters

• New flexible notification system.

Allows sending notifications to different targets. The local Postfix MTA, previously the sole notification option, is now one of several target types available.

Two new target types include: smtp allowing direct notification emails via authenticated SMTP, and gotify, which sends notifications to a Gotify instance.

Flexible notification routing is possible through matcher-based rules that determine which targets receive notifications for specific events.

Match rules can select events based on their severity, time of occurrence, or event-specific metadata fields (such as the event type). Multiple rules can be combined to implement more complex routing scenarios.

• Name resolution to find an IP for a node’s name now looks at all IPs associated with the name, only excluding loopback addresses. Additionally, a better warning is logged in case of a failed lookup.
• pvecm updatecerts, which is used to ensure a consistent state of the certificates in a cluster, was reworked for increased robustness:

Files not being stored in the cluster filesystem are now created earlier.

The call now waits for the node to join the quorate partition of the cluster instead of failing. This is especially helpful during node-bootup, when running before starting pveproxy.service.

The error message in case the call fails due to missing quorum was reworded for better clarity.

• The MAC addresses of the guests under SDN are now cached in the cluster filesystem for improved lookup speed in a cluster.

Backup/Restore

• Backups and backup jobs can now be configured with a notification mode for a smooth migration to the new notification system.

The legacy-sendmail mode replicates the previous behavior of sending an email via the local Postfix MTA if an email is configured.

The notification-system mode sends notifications exclusively using the new notification system.

The default mode auto behaves like legacy-sendmail if an email address is configured, and like notification-system if no email address is configured.

Thus, existing backup jobs without a configured email address will default to sending notification emails to the root user after the upgrade to Proxmox VE 8.1.

• Allow setting the pbs-entries-max parameter. In order to prevent failing container backups with a huge number of files in a directory, it can help to set it to a higher value than the default (issue 3069).
• Improvements to the vma CLI tool that handles VMA backup files:
  • The vma extract command now optionally takes a filter to only extract specific disks from the backup (issue 1534).
  • Fix an issue where the vma create command could not write to tmpfs (issue 4710).
• Improvements to file restore:
  • Fix an issue where the settings for ZFC ARC minimum and maximum were not properly set for the temporary file-restore VM.
  • Fix an issue where debug log messages were not printed even though the PBS_QEMU_DEBUG environment variable was set.
• Fix an issue with backups of diskless VMs to Proxmox Backup Server: Even though encryption was enabled, such backups would not be encrypted. Since the backup contained no disks, this did not reveal any VM data, but the VM configuration was stored in plaintext (issue 4822).
• File restore now allows downloading .tar.zst archives as an alternative to .zip archives.
• Improved handling of backups with master key:
  • Abort the backup if the the running QEMU binary does not support master keys, instead of just printing a warning. Master keys are supported in QEMU builds of Proxmox VE since version 6.4.
  • If no encryption key is configured, the backup task will explicitly warn that the backup will be unencrypted.
  • The backup log now prints only one message that encryption is enabled, instead of previously two messages.

Storage

• Improvements to the iSCSI storage backend:
  • Try to log into all discovered portals for a target, instead of just the single portal initially configured for the storage. This way, the storage can now become available in a multipath setup if at least one portal is online, even if the single configured portal is offline.
  • The backend is now usable immediately after installing Open-iSCSI. Previously, some services needed to be restarted first.
• Fix an issue where a replication job could not be run or deleted if it referred to a storage that does not exist anymore.
• SMB/CIFS: Fix connection check in case an empty domain is provided.
• The BTRFS plugin received a fix for creating base templates when falling back to the standard directory variant.

Ceph

• Support installing Ceph 18.2 Reef and make it the default release for new setups.
• Allow creating multiple OSDs per physical device via API and CLI, and display such setups properly in the GUI. Multiple OSDs on one device can be useful when using fast NVMe drives that would be bottle-necked by a single OSD service (issue 4631).
• When creating a pool, read the default values for size/min_size from the Ceph configuration instead of using hard-coded default values 3/2 (issue 2515).

There are use cases where different values for size/min_size make sense, for example 4/2 if a cluster spans two rooms.

• The pveceph install commands now asks the user to confirm the Ceph version to be installed (issue 4364).
• Improve discoverability of Ceph warnings by providing a tabular view and a button to copy warning details.
• Report OSD memory usage more accurately by using the Proportional Set Size (PSS) of the OSD process. Previously, memory usage was read from the OSD service and thus included the page cache, leading to extremely high values shown in the GUI.
• Use snake_case when setting options in Ceph config files to ensure consistency within that file (issue 4808).
• Mark global pg_bits setting as deprecated and make it a no-op. The setting has been deprecated since Ceph 13.
• Improve reporting of cluster health:
  • Replace “Error” category for PG states with “Warning” and “Critical” categories to allow more fine-grained assessment of the cluster state.
  • Rename “Working” state to “Busy” state to better convey its meaning.

Access control

• Support nested pools up to a nesting depth of 3 levels for greater flexibility in structuring VMs and containers (issue 1148).

Pool names can now contain at most two slashes (allowing to structure them as parent/child/grandchild).

Permissions are inherited along the path according to the usual inheritance rules.

• Improvements to LDAP/AD realms:
  • When adding/updating an LDAP/AD realm, there is now the option to directly check if the bind works, instead of having to wait for the first sync. This check is enabled by default in the GUI and can be disabled in the advanced options if needed.
  • Forbid specifying a Bind DN without a password in the GUI, which is already forbidden by the API.
  • Expose the mode option in the GUI that allows switching between LDAP, LDAPS and LDAP via STARTTLS. This option was already supported by the backend and succeeds the secure option which allowed switching between LDAP and LDAPS only.
  • Fix an issue with enforced TFA where certain sync settings would cause the TFA restriction to not be enforced.
  • It is now possible to update only the password field for the bind-user of an LDAP realm, this failed previously.
  • Allow setting the case-sensitive option of AD realms, which was previously only editable via CLI, in the GUI.
• Existing-but-disabled TFA factors can no longer circumvent realm-mandated TFA.
• The list of SDN-related ACL paths now reflects all SDN objects, ensuring that there are no settings which remain root@pam only.
• A mistyped entry of pools instead of pool in the default set of ACL paths was fixed.

Firewall & Software-Defined Networking

• The core of Proxmox VE’s Software-Defined Network stack has been lifted from experimental to supported.
• New DHCP plugin for SDN (tech preview).

Enabling DHCP for a zone will start a DHCP server that can automatically assign IP addresses to associated virtual guests (VMs and containers).

Currently, only Simple zones are supported, and dnsmasq is the only supported DHCP server.

Each subnet of a Simple zone can now be configured with DHCP ranges.

When a virtual guest associated to the zone starts, the DHCP plugin queries the zone’s IPAM for an IP address and offers it to the virtual guest.

If the built-in Proxmox VE IPAM is used, active DHCP leases can be viewed and edited conveniently on the web UI.

• IS-IS was added as a further SDN controller, next to EVPN and BGP.
• The interfaces section of the frr is now parsed in order to support multiple underlay networks (like IS-IS).
• MAC learning on SDN bridges can now selectively be disabled for individual plugins. This is implemented for the EVP plugin.
• A warning is logged if the main network configuration (/etc/network/interfaces) does not source the SDN controlled configuration (/etc/network/interfaces.d/sdn), because the SDN configuration would be ignored in that case.
• The error reporting for problems with vnet generation was improved, by pointing to the relevant task log.
• The firewall log can now be also displayed for a specific timespan instead of showing the live-view (issue 4442).
• Fix an issue where scoped alias resolution would fail with an error.
• Enabling VLAN-awareness for an EVPN zone is unsupported and now fails instead of just printing a warning (issue 4917).
• Fix an issue where an empty subnet could not be deleted if it has a gateway defined.
• The IPAM selector, which is a required choice, is not hidden behind the Advanced checkbox in the UI anymore.
• The identifying CIDR for a vnet is now named Subnet to improve clarity.
• A systemd.link(5) configuration is now shipped to both keep bridges up even if there’s no port connected, and to prevent that a random MAC-address is assigned to bridges or bond interfaces.
• A ethtool is now a hard dependency of ifupdown2, matching the common need of disabling offloading features of certain NICs.
• Prevent a crash in ifupdown2 caused by an error in a third-party plugin in /etc/network/ifup.d/.
• The accept_ra and autoconf sysctl settings are now also applied for bridge interfaces.
• ifupdown2 now correctly recognizes when remote IPs for vxlan are configured by external sources and does not remove them on reconfiguration.

Improved management of Proxmox VE nodes

• Secure Boot support.

Proxmox VE now ships a shim bootloader signed by a CA trusted by most hardware’s UEFI implementation. In addition, it ships variants of the GRUB bootloader, MOK utilities and kernel images signed by Proxmox and trusted by the shim bootloader.

New installation will support Secure Boot out of the box if it is enabled.

Existing installations can be adapted to Secure Boot by installing optional packages, and possibly reformatting and re-initializing the ESP(s), without the need for a complete reinstallation. See the wiki article for more details.

• The kernel shipped by Proxmox is shared for all products. This is now reflected in the renaming from pve-kernel and pve-headers to proxmox-kernel and proxmox-headers respectively in all relevant packages.
• The new proxmox-default-kernel and proxmox-default-headers meta-packages will depend on the currently recommended kernel-series.
  • Avoid logging benign but confusing warnings about a segfault in pverados.
• Many edge-cases encountered during the upgrade from PVE 7.4 to 8 by our user-base are now detected and warned about in the improved pve7to8 checks:
  • Warn if DKMS modules are detected, as many of them do not upgrade smoothly to the newer kernel versions in PVE 8.
  • Warn if the PVE 7 system does not have the correct meta-package of grub installed ensures to actually upgrade the installed bootloader to the newest version.
  • The check for old cgroupv1 containers was adapted to not cause false positives on current containers (for example Fedora 38).
• Support for adding custom ACME enabled CA’s which require authentication through External Account Binding (EAB) on the commandline (issue 4497).
• Using the Console/Shell on a PVE node is now possible for all users with the appropriate permissions (Sys.Console). The restriction to the pam was removed. The users will still need to login as a system user on the shell though.
• With the Proxmox repositories having support for fetching them directly the changelogs for new package versions shown in the UI are now all gathered with apt changelog.
• The pvesh debug tool now also supports yielding output for streaming API calls, like for example the syslog.
• The documentation on firmware updates provided by the operating system has been extended and revised, helping administrators to identify if their setup is optimal.

Installation ISO

• The ISO is able to run on Secure Boot enabled machines.
• The text-based UI got significant improvement based on the feedback received from the first release in PVE 8.0.
• The current link-state of each network interface is now displayed in the network configuration view, helping in identifying the correct NIC for the management interface (issue 4869).
• If provided by the DHCP server, the hostname field is already filled out with the information from the lease.
• The arc_max parameter for installations on ZFS can now be set in the Advanced Options. If not explicitly set by the user, it is set to a value targeting 10% of system memory instead of 50%, which is a better fit for a virtualization workload (issue 4829).
• The correct meta-package of grub is now installed based on the boot mode (grub-pc or grub-efi-amd64). This ensures that the bootloader on disk gets updated when there is an upgrade for the grub package.
• The text-based UI is now also available over a serial console, for headless systems with a serial port.
• /var/lib/vz backing the local storage is now created as separate dataset for installations on ZFS (issue 1410).
• The root dataset on ZFS installations now uses acltype=posixacl in line with upstream’s recommendation.
• Kernel parameters passed on the commandline during install are now also set in the target system (issue 4747).
• Fix the warning that is shown in case the address family (IPv4, IPv6) of the host IP and DNS server do not match.
• The text-based UI now sets the correct disk-size for the selected disk, instead of limiting the installation to the size of the first disk in the list (issue 4856).
• For better UX, the text-based UI now also displays a count-down before automatically rebooting.
• The screensaver in the graphical installer is now disabled.
• The graphical installer now displays the units used for disk-based options.
• The kernel commandline parameter vga788 is now set for both the graphical debug and all text-based UI installation options. This improves compatibility of the installer with certain hardware combinations.

Other Notable changes

• Existing backup jobs without a configured email address did not send email notifications before the upgrade, but will default to sending email notifications to the root user via the new notification system after the upgrade to Proxmox VE 8.1.

In order to disable notification emails, either change the job’s notification mode to legacy-sendmail or configure the notification system to ignore backup job notifications.

Known Issues & Breaking Changes

Kernel

• With Kernel 6.5 and ZFS it can happen that the host hits a kernel bug when starting a VM with hugepages, and the host must be rebooted.

More information can be found in the forum and in the bug reports for ZFS and Linux kernel.

• Some users with Intel Wi-Fi cards, like the AX201 model, reported that initialization of the card failed with Linux kernel 6.5.

This is still being investigated. You should avoid booting into the new kernel if you have no physical access to your server and an Intel Wi-Fi device is used as its only connection. See the documentation for how to pin a kernel version.

• Some SAS2008 controllers need a workaround to get detected since kernel 6.2, see the forum thread for details.

DATAZONE supports you with implementation — contact us for a no-obligation consultation.