Modern enterprise storage is defined by OpenZFS. For mission-critical homelabs and business servers, architecting the storage layout correctly from day one avoids painful resilvering bottlenecks, pool fragmentation, and unrecoverable data loss.
This master runbook details complete OpenZFS architecture on TrueNAS SCALE and Linux: contrasting mirrored vdevs against RAIDZ2, configuring NVMe storage tiers, sizing the Adaptive Replacement Cache (ARC), tuning dataset recordsizes (1M for media vs 16K for databases/VMs), and scheduling non-disruptive scrubs.
⚡ Quick Runbook: 60-Second Implementation
TL;DR: Create and optimize high-performance OpenZFS datasets with custom record sizes:
# 1. Dataset for Media/Streaming (Large Sequential I/O: 1M recordsize, zstd)
sudo zfs create -o recordsize=1M -o compression=zstd -o atime=off datapool/media
# 2. Dataset for PostgreSQL/MariaDB Containers (Small Random I/O: 16K recordsize)
sudo zfs create -o recordsize=16K -o compression=lz4 -o atime=off fastpool/databases
# 3. Restrict ZFS ARC RAM to prevent starvation of Docker/KVM services
# Set ARC max to 32GB on host (/etc/modprobe.d/zfs.conf)
echo "options zfs zfs_arc_max=34359738368" | sudo tee /etc/modprobe.d/zfs.conf
sudo update-initramfs -u1. Vdev Topologies: Mirrored Pairs vs. RAIDZ2
| Topology | IOPS Performance | Resilver Time | Expansion Flexibility | Usable Capacity |
|---|---|---|---|---|
| Mirrored vdevs (RAID 10) | Excellent (Multiplies with pairs) | Fastest (Linear drive read) | Add 2 drives at a time | 50% |
| RAIDZ2 (Dual Parity) | Single drive equivalent | Long (Calculates parity) | Full vdev replacement required | 66% – 80% |
Architecture Rule: Use mirrored vdevs for application datasets, VM disks, and transactional databases where random read/write IOPS are paramount. Use RAIDZ2 (typically 6 to 8 wide vdevs) for sequential mass archives and cold storage pools.
2. Storage Tiering (Fastpool vs. Datapool)
Modern homelab servers split storage across two distinct pools:
fastpool(NVMe / Enterprise SSD Mirror): Houses the host OS, Docker appdata, PostgreSQL/MariaDB databases, search indices, and VM disks. Highly responsive sub-millisecond latencies.datapool(Spinning HDD Topologies): Houses raw photo libraries (Immich), video media (Emby/Jellyfin), document archives (Paperless-ngx), and backup snapshots.
3. Dataset Tuning: Recordsize and Compression
ZFS recordsizes must match application block sizes to prevent write amplification:
- Media & Video (
recordsize=1M): Compresses video and audio into massive 1MB blocks, maximizing throughput and minimizing metadata overhead. - Databases (
recordsize=16K): PostgreSQL and MariaDB write in 8K/16K pages. Setting a 16K recordsize matches database block writes, eliminating read-modify-write performance penalties. - General Appdata (
recordsize=128K): Default general-purpose recordsize suitable for configuration files and logs.
4. ARC Memory Governance & Scrub Maintenance
ZFS Adaptive Replacement Cache (ARC) will consume up to 50% (on modern Linux) or 90% (on TrueNAS Core/SCALE) of host RAM by default. On hosts running Docker containers, constrain ARC memory:
# Monitor real-time ARC hit rates (Target: >85% hit rate)
arcstat 2
# Schedule bi-weekly non-disruptive scrub
sudo systemctl enable --now zfs-scrub-weekly.timer