homelab-configs/truenas-hardware-inventory.md

TrueNAS Hardware Inventory & SMART Analysis

Last updated: 2026-05-19
TrueNAS version: 25.04.2.6 (TrueNAS SCALE Fangtooth)
Kernel: 6.12.15-debug+truenas
Uptime: 11 days, 22 hours at time of audit
Hostname: truenas

---

System Board

• Motherboard: Gigabyte B450M DS3H WIFI-CF (AMD Ryzen AM4)

---

Storage Controllers

PCI Slot	Controller	Driver	Notes
03:00.0	Broadcom/LSI SAS3008 (SAS 9300-16i)	mpt3sas	LANPAN HBA card (installed Apr 2026)
05:00.0	Broadcom/LSI SAS3008 (SAS 9300-16i)	mpt3sas	YiBaoTong SAS9300-16i (original replacement)
06:00.1	AMD 400 Series Chipset SATA	ahci	Onboard, 8 native ports
0f:00.0	AMD FCH SATA [AHCI mode]	ahci	Onboard
0f:00.1	AMD FCH SATA [AHCI mode]	ahci	Onboard

Note: All pool drives connect via the SAS3008 HBAs. The boot SSD connects to the onboard SATA (ata10). SMART data is read via SAT (SCSI-ATA Translation) through the SAS controllers.

---

Drive Inventory

Dev	Serial	Model	Size	HCTL (Host)	Pool Role	SMART Status
sda	9JG47UTT	WDC WUH721414ALE600	12.7 TB	12:0:1:0 (SAS)	pool1 data	✅ Clean
sdb	0007A62D	HGST OOS16000G	14.6 TB	0:0:0:0 (SAS)	pool1 spare	✅ Clean
sdc	9RG9L8BC	WDC WUH721414ALE604	12.7 TB	12:0:0:0 (SAS)	pool1 data	✅ Clean
sdd	9JG47Y2T	WDC WUH721414ALE600	12.7 TB	12:0:2:0 (SAS)	pool1 data	✅ Clean
sde	ZX3014DP	Seagate ST16000NM000D	14.6 TB	12:0:3:0 (SAS)	pool1 data	✅ Clean
sdf	9JJ5DJET	WDC WUH721414ALE604	12.7 TB	12:0:5:0 (SAS)	pool1 data	✅ Clean
sdg	9JGV4MLT	WDC WUH721414ALE604	12.7 TB	12:0:6:0 (SAS)	pool1 data	✅ Clean
sdh	9JH31S2T	WDC WUH721414ALE604	12.7 TB	12:0:7:0 (SAS)	pool1 data	⚠️ See below
sdi	9JGZL01T	WDC WUH721414ALE604	12.7 TB	12:0:4:0 (SAS)	pool1 data	⚠️ See below
sdj	AA000000000000001308	EDILOCA EN206 512GB SSD	477 GB	10:0:0:0 (SATA)	Boot drive	✅ Clean

Pool drives: 8 data + 1 hot spare
Available space: ~56 TB
Pool configuration: RAIDZ2 (tolerates 2 simultaneous drive failures)

---

SMART Health Summary

✅ Clean Drives (7 drives)

All of the following have zero pending sectors, zero uncorrectable sectors, zero reallocated sectors, and no ATA errors:

• sda — 9JG47UTT (WDC WUH721414ALE600)
• sdb — 0007A62D (HGST OOS16000G) — hot spare
• sdc — 9RG9L8BC (WDC WUH721414ALE604)
• sdd — 9JG47Y2T (WDC WUH721414ALE600)
• sde — ZX3014DP (Seagate ST16000NM000D) — previously thought incompatible with SAS; now clean
• sdf — 9JJ5DJET (WDC WUH721414ALE604)
• sdg — 9JGV4MLT (WDC WUH721414ALE604)

⚠️ Drives Requiring Attention

sdh — Serial: 9JH31S2T (WDC WUH721414ALE604, 12.7 TB)

SMART Attribute	Value
Reallocated Sectors (ID 5)	0
Current Pending Sectors (ID 197)	40
Offline Uncorrectable (ID 198)	0
ATA Error Count	85
Self-Test Errors	21
Power-On Hours	~53,029 hrs (~6.1 years)

Important context: The 21 self-test errors are almost entirely caused by a misconfigured SMART schedule (see Fix section below) — a LONG self-test was set to run every hour. A long test on a 12.7 TB drive takes ~12 hours; starting a new one every hour continuously aborts the previous test, and each abortion is logged as a self-test failure. This generated endless alerts.

The 40 pending sectors and 85 ATA errors are real and need monitoring. Pending sectors are sectors the drive couldn't read; they haven't been confirmed bad yet (zero reallocated, zero uncorrectable). The drive is still functional but showing age-related wear.

sdi — Serial: 9JGZL01T (WDC WUH721414ALE604, 12.7 TB)

SMART Attribute	Value
Reallocated Sectors (ID 5)	2
Current Pending Sectors (ID 197)	8
Offline Uncorrectable (ID 198)	6
ATA Error Count	17
Self-Test Errors	2
Power-On Hours	~53,503 hrs (~6.1 years)

This drive has confirmed bad sectors: 6 offline uncorrectable and 2 already reallocated. This is a stronger warning sign than sdh. The drive is still online and ZFS is protecting data via RAIDZ2, but this drive should be prioritized for replacement.

---

ZFS Pool Status

Pool:    pool1
Type:    RAIDZ2 (8 data drives + 1 hot spare)
Status:  ONLINE — healthy
Warning: true (due to SMART alerts, not data errors)

Last Scrub:
  Completed: ~May 7, 2026
  Result:    0 errors (all data intact)
  Duration:  ~5.8 hours
  Data read: 24.96 TB

The scrub result is the most important number: ZFS read every byte of data across all drives and found zero checksum errors. This confirms RAIDZ2 is protecting all data correctly despite the SMART errors on sdh and sdi.

---

Root Cause Analysis: SMART Alert Flood

What was observed

A large number of SMART CRITICAL alerts across multiple drives, across multiple controller changes, leading to the theory that the controller hardware was causing errors.

Actual root cause (two separate issues)

Issue 1: Drive age (real hardware wear)
sdh and sdi have ~6.1 years of continuous runtime (53,000+ power-on hours). Enterprise WD Gold drives have a 5-year MTBF rating. These drives are past their design life and showing normal end-of-life bad sector accumulation. This is not a controller issue — the sectors on the drives themselves are worn.

Issue 2: Misconfigured SMART test schedule (false alarm amplifier)
Someone configured a LONG SMART self-test to run every hour on only one drive (9JH31S2T/sdh). This setting is still present in TrueNAS:

Schedule ID 1: LONG test, disk 9JH31S2T, schedule: every hour (all wildcards)

This caused an alert every single hour saying "new Self-Test Log error at hour timestamp XXXXX" — not because the drive was actively failing, but because each new hourly test was aborting the previous one.

Why the "three controllers showing the same errors" was misleading

• Linux device names (sda, sdb, sdh...) are not stable — they can be reassigned on reboot
• Alert messages recorded serials like 9JH31S2T appearing on sda, sdd, sde, sdf, sdh, sdj at different times — this was the same physical drive being seen under different device names
• The controller changes never fixed the errors because the problem was always in the drives themselves, not the controllers

Why different controllers also appeared to show errors

• The SATA controller was replaced with LANPAN SAS3008, and the old SAS9300-16i (YiBaoTong) may still be installed
• Both SAS3008 controllers read SMART data via SAT (SCSI-ATA Translation) — the elevated ATA error counts partially reflect SAT overhead, not pure drive errors
• But the pending and uncorrectable sectors are stored on the drive itself and cannot be faked by a controller

---

Action Items

🔴 Immediate — Fix SMART test schedule

Delete the hourly LONG test schedule for 9JH31S2T (TrueNAS UI: Data Protection → S.M.A.R.T. Tests).
Replace it with a monthly LONG test (or use the global TrueNAS SMART schedule for all drives).

This will immediately stop the hourly alert flood.

🟡 Medium-term — Plan drive replacements

Both sdh (9JH31S2T) and sdi (9JGZL01T) are showing real bad sectors after ~6 years of operation. RAIDZ2 currently protects the data, but losing a third drive while these two are degraded would be data-threatening.

Replacement priority:

1. sdi (9JGZL01T) — first priority; has confirmed reallocated + uncorrectable sectors
2. sdh (9JH31S2T) — second priority; 40 pending sectors, high ATA error count

The hot spare sdb (0007A62D, HGST OOS16000G, 14.6 TB) is clean and ready to take over automatically if either drive fails completely.

Replacement drives recommended: WD Gold or Seagate Exos 14TB+ (WDC WUH721414ALE604 or newer).

🟢 Ongoing — Monthly scrubs

Run a ZFS scrub monthly to verify data integrity. TrueNAS default schedule handles this. The May 2026 scrub showed 0 errors — continue monitoring.

---

Controller Replacement History

Date	Event
~2025	Original generic ASMedia ASM1061 SATA controller running
April 2026	ASMedia ASM1061 diagnosed as faulty (ata10, sda/sdh errors)
April 18, 2026	LANPAN SAS3008 ordered ($89.80)
April 19, 2026	LANPAN SAS3008 installed; Seagate Exos X20 16TB drives (0007A62D, ZX3014DP) found incompatible with SAS
April 19, 2026	2x WD 12.7TB drives replaced with HGST OOS16000G (sdb) and Seagate (sde); resilver started
April 20, 2026	Resilver completed; pool returned to ONLINE
May 2026	YiBaoTong SAS9300-16i also present (two SAS3008 controllers total)
May 7, 2026	Scrub completed — 0 errors
May 19, 2026	Full hardware audit performed; root cause identified (drive age + bad SMART schedule)

---

SSH Config Note

• SSH key: ~/.ssh/id_rsa (RSA key — TrueNAS authorized_keys uses RSA, not ed25519)
• SSH alias: truenas
• Admin user cannot run zpool or smartctl directly — use midclt call for TrueNAS API access