TrueNAS is the best NAS operating system if you’re building a NAS from scratch, packed with handy features and tools that will get you into data storage nirvana in no time at all. Sure, going with a pre-built NAS enclosure makes the initial setup quick, but it’s really not that much harder to build your own, and then you don’t get stuck in vendor lock-in.
I recently spun up TrueNAS in a VM inside Proxmox, and I’ve been enjoying how well everything is laid out and the great performance, even while passing through drives from Proxmox. But I’m severely limited in system memory, as the mini PC I’m using has soldered-on RAM, and only 12GB of it. That limits how much data TrueNAS can cache in memory, but there’s a fix, if I want to sacrifice one of my SSDs (or part of one as a virtual drive).
Related
7 underrated TrueNAS features I use and swear by
These powerful tools are hiding in plain sight
What is L2ARC, and why would you use it
Speed up frequently accessed data requests without more RAM
TrueNAS is set to use the ZFS file system by default, and you shouldn’t try to mess with it; it’s perfect for keeping your data safe. One of the big features is Adaptive Replacement Cache, or ARC, which uses the server memory for frequently accessed data. This is great for speedy storage access, but what happens if you don’t have much system memory for it to use? Once requests fail to hit cached data, they’ll go into the slow lane for access from the pools, which could be an issue.
TrueNAS has a secondary caching algorithm called L2ARC, which uses an SSD to cache any read requests that overflow from the ARC. That way, they’re still faster than normal requests, even if they’re not quite as fast as system memory, and you can have more space for them, which can benefit some systems. For example, like the mini PC I’m using for my home lab has 12GB of memory that’s soldered on, and I can’t increase it.
Having L2ARC enabled reduces latency and mechanical disk load (if your storage pools are HDDs). It’s also beneficial for any workload that might not fit fully in memory, like virtualization, database backends, VM storage, or similar tasks. It’s worth noting that it only makes a difference for reads, and won’t do anything to help bottlenecks in write operations, those are best served with SLOG or orther write caching solutions.
It’s still better to have more RAM, but some things have changed
L2ARC has improved and changed, and some common misconceptions no longer apply. Having more system memory is still better, but even on 16GB of memory, L2ARC can make things faster. It’s worth knowing that L2ARC shouldn’t be added until RAM is maxed out, as it’s better to let ARC fill up available memory and allocate dynamically first.
While you should consider ARC efficient when it has a 90%+ hit rate, your expectations for L2ARC should be lowered to around a 20% hit rate.
L2ARC takes up some RAM overhead, so you might see a lower amount of ARC used, but that’s okay. And L2ARC is now persistent by default, so it no longer clears on a reboot, but that does mean it can take some time to re-read the L2ARC record pointers into RAM when TrueNAS starts back up, so be aware it’ll likely be sluggish for a bit.
Related
Here’s how I fixed my TrueNAS disk errors by properly doing HDD passthrough in Proxmox
Don’t make the mistakes I did
The best part is that it only takes a minute to set up
You can do it during pool creation or at any time afterwards
Adding L2ARC to your TrueNAS system takes only a few clicks, but you’ll need one important thing first: an unassigned drive per storage pool you have running. This doesn’t have to be a physical drive like I used. If you’re virtualizing TrueNAS, you can pass a virtual disk through and use that for L2ARC. If the virtual drive is on NVMe, you won’t notice much of a difference in speed.
While creating a new storage pool, there’s a section for cache. Picking an unallocated drive here will set it for L2ARC use. Or, clicking on an existing pool then add vdev will bring up the menu options, and again, select the L2ARC drive in the cache section.
More devices for L2ARC will improve latency and overall performance, and faster, large-capacity NVMe SSDs are suggested for improving random read-heavy workloads. NVMe is a best fit here because it’s likely faster than the storage pool drives, which is important for streaming or sequential workloads.
My TrueNAS system runs in a VM inside Proxmox, so I first had to pass through some storage drives. After finding the drive IDs, I powered down the TrueNAS VM, and went into the Proxmox CLI to add the drives. The following command adds one drive, with the 101 being the VM number in my Proxmox list (yours may vary), scsi1 could be any number higher than 0 (which is the virtual drive TrueNAS is installed on), and the /disk/by-id/ could be nvme or ata or scsi (if passing through a RAID controller, while using virtio1 instead of scsi1):
qm set 101 -scsi1 /dev/disk/by-id/nvme-xxxxxxxxx-xxxxx_xxx
After doing this for both of the NVMe drives I would use, I started the TrueNAS VM again, and logged in via the web interface to add the vdev and L2ARC drive. That’s it: L2ARC is now running. I’ll monitor the hit rate for ARC and L2ARC for a few weeks to see how effective this setup is, and decide if I need it, or if I want to make a virtual drive and use that instead of passing through a physical drive.
Related
5 reasons to use SSD cache inside your NAS
Give your NAS a boost with the power of flash storage
L2ARC is a handy tool, but it’s no replacement for adding more RAM
Using L2ARC to turn a spare SSD into a cache drive for TrueNAS can be beneficial, but it depends on how you use your NAS and what drives are inside. If you’re already using all-flash storage, the gains will be marginal and you’ll be taking one drive that could be used for storage space out of the equation. But if your RAM is already filling up with ARC hits and you need a little boost, an L2ARC SSD is a fantastic use of a few minutes of your time to set up.