A new technical preview feature brought to my attention by William Lam (see his blog post here: https://williamlam.com/2024/08/nvme-tiering-in-vsphere-8-0-update-3-is-a-homelab-game-changer.html) allows the use of locally installed NVMe devices to be leverage for Memory Tiering.

The techincal definition taken from the Memory Tiering over NVMe Tech Preview – vSphere 8.0U3 Techincal Guide describes this feature as this:

“Memory tiering over NVMe optimizes performance by intelligently directing VM memory allocations to either NVMe devices or faster dynamic random access memory (DRAM) in the host and performing hot and cold memory page placements. This allows customers to increase their memory footprint, while increasing workload capacity and reducing the overall total cost of ownership (TCO).”

Also, “Memory Tiering is recommended for use by customers who are running specific workload types in test / lab environments and not for use in production environments.”

As soon as I heard of this feature, I immediately realized this is a feature I wanted to try out right away in my home lab. Because I run older Dell T3600 hosts, I often find I am short of memory as those models physically support a maximum of 64GB of RAM. With is new tiering feature, this would allow me to add a significant amount of RAM, via NVMe device, allowing my home lab to allow more VM’s…a.k.a. workload capacity. Plus, the hardware to do this for my host(s) is inexpensive.

Now, let me state that in the guide, the NVMe device recommendation states that “An NVMe device must have the performance and endurance characteristics comparable
to Enterprise-Class SLC SSD devices.” and it points to hardware that is compatible with vSAN. Also, only 1 PCIe-based NVMe device per host can be configured as tiered memory. Plus, there are other compatibility criteria to be followed…

This being a home lab…I found that a simple NVMe to PCI-e 3.0 adapter would most likely fit the bill. This particular adapter is the one I found to try out, as there is an open PCI-e slot open and available. As far as an NVMe device, I went with a Samsung 980 M.2 SSD that I had kicking around on a shelf in my office.

The process of installing the M.2 drive in the adapter and installing the adapter in the server was straightforward and took all about 12 minutes including powering the host and booting ESXi.

For the rest of the process and configuration, I followed a combination of William’s blog post and the technical guide (links above), which helped with the config commands. The host was also previously put into maintenance mode, so no VM’s were running etc.

After powering up the host and booting ESXi, and I could see it picked up the new adapter just as expected, and I was able to ID the NVMe device:

And verified the M.2 SSD does display as Flash drive type and the transport is PCIE, all of which is correct. At the same time I also verified no other partitions had been previously configured…none were, so all was good with this step:

Then I SSH’d into the host, and just because I’m curious, ran the following vSphere esxcli function:

esxcli system tierdevice list

This illustrates there are no devices configured for NVMe tiering…which was totally expected:

Enabling the NVMe memory tiering feature is easy enough, and was the next step using vSphere esxcli (refer to page 23 in the guide):

esxcli system settings kernel set -s MemoryTiering -v TRUE

The next step is to reboot the host.

After the reboot and reconnecting, the next step is to configure the correct disk to use – I found the needed information of location listed in the Storage Devices -> Properties details:

I reconnected via SSH and ran the following command to create the single tier partition on the NVMe device (refer to page 16 in the guide):

esxcli system tierdevice create -d /vmfs/devices/disks/t10.NVMe____Samsung_SSD_980_1TB_____________________E62BA021D8382500

Note, that currently in the tech preview, this step can only be done either using the vSphere ESXCLI command (or vSphere PowerCLI). There is no current support to do this step from the vSphere Client, however, as the guide states, future vSphere releases will improve this process as part of configuring the NVMe devices within the vSphere Client.

Then I ran the list command again, and the NVMe device is now showing in the list:

esxcli system tierdevice list

I initiated one additional reboot, and after the reboot, and host came back up, I was able to verify the Memory Tiering is now set and reporting as Software…and the amount of memory is reporting as 319.79GB capacity!

After a few minutes DRS ran and rebalanced my running VM’s and even after the rebalance, I still have plenty of Memory capacity available for additional workloads.

There are a few additional advanced settings to play around with, and William does get into one of those settings in his blog, so please go check out his blog if you are reading this and want to know more.

Overall, this is an incredibly good techincal preview feature that will move the needle for the TCO of my home lab!

Cannot wait to see more of this feature in future releases!

EDIT:

Hey, just a quick edit to add this blog. One limitation I ran into is that the current Memory Tiering feature does NOT currently support snapshots – see the error below. The error mentions “Snapshots are not YET supported on a system with Software Memory Tiering enabled.” So, hopefully this will be support in the next iteration, or when the feature is not in techincal preview status. So, please take this into consideration before enabling!