Whenever buying new hardware, it’s all too easy to get caught in the “bigger number better” mantra and lose sight of which specifications are the important ones. I’m not immune to that misstep, and when I expanded my home lab earlier this year, I was on the hunt for some NVMe SSDs to cache data for my existing NAS, and to fill the empty slots in my new all-flash NAS.
And you know what, I got stuck thinking I needed the fastest drives. It’s an easy trap to fall into, because that’s the number that marketing always points at, but it’s also one of the last numbers you should worry about when buying NVMe drives for home lab use, and I’ll show you why.
Speed advantages are often theoretical
There are few home lab situations where NVMe speeds matter
Listen, I’m a speed demon about the drives that go in my PCs, and I’m here to say SSD speeds don’t matter anymore. Before you jump on me in the comments, I want to say that I know some home lab tasks do benefit from faster storage. Even then, you shouldn’t rely on single NVMe drives to provide that speed; they should be in RAID or ZFS pools or even Ceph clusters if you want the fastest speeds and largest capacity.
The fact is that once you get to PCIe 3.0 speeds for NVMe, so many other factors in your home lab become the deciding factor over how fast you can pass data. You might be using ex-enterprise hardware, which will be a few generations behind any of the new NVMe standards, and can’t get the speeds of PCIe 5.0 SSDs. Even the 7,000MB/s speeds of PCIe 4.0 might be a bridge too far, and that’s only considering on-device data transfer.
Your home lab network is the bottleneck — not your NVMe
Your NVMe SSD speeds are not what is holding your home lab back in terms of performance. At least, not if you’re using PCIe 3.0 and above. The first four screenshots in the gallery are of two PCIe 5.0 SSDs at 12,491 MB/s and 9,605 MB/s read speeds, a PCIe 3.0 SSD at 2,491 MB/s, and a PCIe 4.0 SSD at 3,488 MB/s. The last screenshot shows a speed test over a 2.5GbE network link, which gets just over 2,400 MB/s.
Mind you, those disk tests were done on the same device, so you’ll not get those speeds across the network, even when using NVMe-over-fabric. I thought I’d be limited by the PCIe 3.0 x1 slots on my Beelink ME mini, but I can saturate a 2.5GbE link easily without even using all six drives in RAID, and even if I had a 10GbE NIC in that mini PC, I’d still be network-limited.
Unless you’re using faster than 10GbE at home, like Nvidia’s BlueField, which can go up to 800GbE, you won’t be limited by NVMe speeds in your home lab while transferring data across your network.
Home lab tasks require different specifications
It’s not all about straight-line speed for your NVMe SSDs
Here’s a fun fact for you. When you build storage clusters with hard drives, the drives are the bottleneck, as read and write operations are substantially slower than the rest of the compute pipeline. With NVMe, that paradigm is flipped, and it’s more likely that the CPU, system memory, and the network are the slower parts that need to be accounted for.
That’s just as true in the home lab as it is in the data center, and the types of tasks are similar:
- Virtual machines: These generate continuous write loads, which are increased when doing snapshots and other backup processes
- Containers: Docker, Kubernetes, LXC and other containerized workloads generate frequent writes from logging, temp files, application data, and replacing ephemeral container data
- Database operations: Many home lab applications, like Home Assistant, create local databases and involve regular write operations
- Backup and sync: Automated backup systems and file sync services constantly write and read data in predictible but often heavy patterns
Home labs are even more write-heavy than gaming PCs or other home computing tasks. Virtualization platforms like Proxmox or VMWare constantly have VM disk operations, logging, and snapshot management to further add to your NVMe drives’ tasks. And if you’re using Kubernetes or other container orchestration platforms, built-in timeouts are based on minimum I/O performance levels.
What to look for instead
While consumer-level NVMe SSDs use total bytes written (TBW) to give you a sense of the overall endurance of the SSD, it’s only part of the equation. Say you are looking at an NVMe drive with 300TBW per 1TB of capacity. For a 2TB SSD, that means 600TBW, which is the equivalent of writing 300GB/day, every day, for 2,000 days (or 5.5 years).
Even in my heaviest usage days, my home lab would never get up to that, and realistically, most users won’t ever use their NVMe drive enough to wear out the NAND. I have had controllers fail, which is another problem entirely, and why I will only buy SSDs from trusted brands. Enterprise NVMe drives can be rated for 5,000TBW or higher, and all but the heaviest home lab LLM enthusiast will never get to those wear levels.
It’s still smart to look for higher TBW ratings, because it carries an implicit quality indicator. Enterprise drives also have power loss protection (PLP) via onboard capacitors in case your home lab temporarily blips, which guards against minor data loss that you might otherwise attribute to wearing NAND, and you probably want to look for NVMe SSDs that have DRAM cache, to reduce the reliance on system memory as cache, more than anything else. And high IOPS are more beneficial for small file access than overall speed, so look for that as well.
NVMe endurance ratings matter more than speed for the home lab
For home lab users, the capacity of your NVMe matters more than the speed it runs at. I mean, every bit of NAND used for NVMe drives is fast enough for real-world use, and benchmarks on sequential transfer speed are a vanity metric at best. But capacity doesn’t mean anything without endurance, so TBW (or drive writes per day DWPD) are more important to look for when considering NVMe SSDs for your lab.