# TrueNAS Hardware Inventory & SMART Analysis

**Last updated:** 2026-05-19  
**TrueNAS version:** 25.04.2.6 (TrueNAS SCALE Fangtooth)  
**Kernel:** 6.12.15-debug+truenas  
**Uptime:** 11 days, 22 hours at time of audit  
**Hostname:** truenas  

---

## System Board

- **Motherboard:** Gigabyte B450M DS3H WIFI-CF (AMD Ryzen AM4)

---

## Storage Controllers

| PCI Slot | Controller | Driver | Notes |
|----------|-----------|--------|-------|
| 03:00.0 | Broadcom/LSI SAS3008 (SAS 9300-16i) | mpt3sas | LANPAN HBA card (installed Apr 2026) |
| 05:00.0 | Broadcom/LSI SAS3008 (SAS 9300-16i) | mpt3sas | YiBaoTong SAS9300-16i (original replacement) |
| 06:00.1 | AMD 400 Series Chipset SATA | ahci | Onboard, 8 native ports |
| 0f:00.0 | AMD FCH SATA [AHCI mode] | ahci | Onboard |
| 0f:00.1 | AMD FCH SATA [AHCI mode] | ahci | Onboard |

> Note: All pool drives connect via the SAS3008 HBAs. The boot SSD connects to the onboard SATA (ata10). SMART data is read via SAT (SCSI-ATA Translation) through the SAS controllers.

---

## Drive Inventory

| Dev | Serial | Model | Size | HCTL (Host) | Pool Role | SMART Status |
|-----|--------|-------|------|-------------|-----------|--------------|
| sda | 9JG47UTT | WDC WUH721414ALE600 | 12.7 TB | 12:0:1:0 (SAS) | pool1 data | ✅ Clean |
| sdb | 0007A62D | HGST OOS16000G | 14.6 TB | 0:0:0:0 (SAS) | pool1 **spare** | ✅ Clean |
| sdc | 9RG9L8BC | WDC WUH721414ALE604 | 12.7 TB | 12:0:0:0 (SAS) | pool1 data | ✅ Clean |
| sdd | 9JG47Y2T | WDC WUH721414ALE600 | 12.7 TB | 12:0:2:0 (SAS) | pool1 data | ✅ Clean |
| sde | ZX3014DP | Seagate ST16000NM000D | 14.6 TB | 12:0:3:0 (SAS) | pool1 data | ✅ Clean |
| sdf | 9JJ5DJET | WDC WUH721414ALE604 | 12.7 TB | 12:0:5:0 (SAS) | pool1 data | ✅ Clean |
| sdg | 9JGV4MLT | WDC WUH721414ALE604 | 12.7 TB | 12:0:6:0 (SAS) | pool1 data | ✅ Clean |
| sdh | 9JH31S2T | WDC WUH721414ALE604 | 12.7 TB | 12:0:7:0 (SAS) | pool1 data | ⚠️ See below |
| sdi | 9JGZL01T | WDC WUH721414ALE604 | 12.7 TB | 12:0:4:0 (SAS) | pool1 data | ⚠️ See below |
| sdj | AA000000000000001308 | EDILOCA EN206 512GB SSD | 477 GB | 10:0:0:0 (SATA) | Boot drive | ✅ Clean |

**Pool drives:** 8 data + 1 hot spare  
**Available space:** ~56 TB  
**Pool configuration:** RAIDZ2 (tolerates 2 simultaneous drive failures)

---

## SMART Health Summary

### ✅ Clean Drives (7 drives)

All of the following have **zero** pending sectors, zero uncorrectable sectors, zero reallocated sectors, and no ATA errors:

- sda — 9JG47UTT (WDC WUH721414ALE600)
- sdb — 0007A62D (HGST OOS16000G) — hot spare
- sdc — 9RG9L8BC (WDC WUH721414ALE604)
- sdd — 9JG47Y2T (WDC WUH721414ALE600)
- sde — ZX3014DP (Seagate ST16000NM000D) — previously thought incompatible with SAS; now clean
- sdf — 9JJ5DJET (WDC WUH721414ALE604)
- sdg — 9JGV4MLT (WDC WUH721414ALE604)

### ⚠️ Drives Requiring Attention

#### sdh — Serial: 9JH31S2T (WDC WUH721414ALE604, 12.7 TB)

| SMART Attribute | Value |
|----------------|-------|
| Reallocated Sectors (ID 5) | 0 |
| Current Pending Sectors (ID 197) | **40** |
| Offline Uncorrectable (ID 198) | 0 |
| ATA Error Count | **85** |
| Self-Test Errors | **21** |
| Power-On Hours | ~53,029 hrs (~6.1 years) |

**Important context:** The 21 self-test errors are almost entirely caused by a misconfigured SMART schedule (see Fix section below) — a LONG self-test was set to run **every hour**. A long test on a 12.7 TB drive takes ~12 hours; starting a new one every hour continuously aborts the previous test, and each abortion is logged as a self-test failure. This generated endless alerts.

The **40 pending sectors** and **85 ATA errors** are real and need monitoring. Pending sectors are sectors the drive couldn't read; they haven't been confirmed bad yet (zero reallocated, zero uncorrectable). The drive is still functional but showing age-related wear.

#### sdi — Serial: 9JGZL01T (WDC WUH721414ALE604, 12.7 TB)

| SMART Attribute | Value |
|----------------|-------|
| Reallocated Sectors (ID 5) | **2** |
| Current Pending Sectors (ID 197) | **8** |
| Offline Uncorrectable (ID 198) | **6** |
| ATA Error Count | **17** |
| Self-Test Errors | 2 |
| Power-On Hours | ~53,503 hrs (~6.1 years) |

This drive has **confirmed bad sectors**: 6 offline uncorrectable and 2 already reallocated. This is a stronger warning sign than sdh. The drive is still online and ZFS is protecting data via RAIDZ2, but this drive should be prioritized for replacement.

---

## ZFS Pool Status

```
Pool:    pool1
Type:    RAIDZ2 (8 data drives + 1 hot spare)
Status:  ONLINE — healthy
Warning: true (due to SMART alerts, not data errors)

Last Scrub:
  Completed: ~May 7, 2026
  Result:    0 errors (all data intact)
  Duration:  ~5.8 hours
  Data read: 24.96 TB
```

**The scrub result is the most important number**: ZFS read every byte of data across all drives and found zero checksum errors. This confirms RAIDZ2 is protecting all data correctly despite the SMART errors on sdh and sdi.

---

## Root Cause Analysis: SMART Alert Flood

### What was observed
A large number of SMART CRITICAL alerts across multiple drives, across multiple controller changes, leading to the theory that the controller hardware was causing errors.

### Actual root cause (two separate issues)

**Issue 1: Drive age (real hardware wear)**  
sdh and sdi have ~6.1 years of continuous runtime (53,000+ power-on hours). Enterprise WD Gold drives have a 5-year MTBF rating. These drives are past their design life and showing normal end-of-life bad sector accumulation. This is not a controller issue — the sectors on the drives themselves are worn.

**Issue 2: Misconfigured SMART test schedule (false alarm amplifier)**  
Someone configured a LONG SMART self-test to run **every hour** on only one drive (9JH31S2T/sdh). This setting is still present in TrueNAS:

```
Schedule ID 1: LONG test, disk 9JH31S2T, schedule: every hour (all wildcards)
```

This caused an alert every single hour saying "new Self-Test Log error at hour timestamp XXXXX" — not because the drive was actively failing, but because each new hourly test was aborting the previous one. 

### Why the "three controllers showing the same errors" was misleading
- Linux device names (sda, sdb, sdh...) are **not stable** — they can be reassigned on reboot
- Alert messages recorded serials like 9JH31S2T appearing on sda, sdd, sde, sdf, sdh, sdj at different times — this was the **same physical drive** being seen under different device names
- The controller changes never fixed the errors because the problem was always in the drives themselves, not the controllers

### Why different controllers also appeared to show errors
- The SATA controller was replaced with LANPAN SAS3008, and the old SAS9300-16i (YiBaoTong) may still be installed
- Both SAS3008 controllers read SMART data via SAT (SCSI-ATA Translation) — the elevated ATA error counts partially reflect SAT overhead, not pure drive errors
- But the pending and uncorrectable sectors are **stored on the drive itself** and cannot be faked by a controller

---

## Action Items

### 🔴 Immediate — Fix SMART test schedule

Delete the hourly LONG test schedule for 9JH31S2T (TrueNAS UI: Data Protection → S.M.A.R.T. Tests).  
Replace it with a **monthly** LONG test (or use the global TrueNAS SMART schedule for all drives).

This will **immediately stop the hourly alert flood**.

### 🟡 Medium-term — Plan drive replacements

Both sdh (9JH31S2T) and sdi (9JGZL01T) are showing real bad sectors after ~6 years of operation. RAIDZ2 currently protects the data, but losing a third drive while these two are degraded would be data-threatening.

Replacement priority:
1. **sdi (9JGZL01T)** — first priority; has confirmed reallocated + uncorrectable sectors
2. **sdh (9JH31S2T)** — second priority; 40 pending sectors, high ATA error count

The hot spare **sdb (0007A62D, HGST OOS16000G, 14.6 TB)** is clean and ready to take over automatically if either drive fails completely.

Replacement drives recommended: WD Gold or Seagate Exos 14TB+ (WDC WUH721414ALE604 or newer).

### 🟢 Ongoing — Monthly scrubs

Run a ZFS scrub monthly to verify data integrity. TrueNAS default schedule handles this. The May 2026 scrub showed 0 errors — continue monitoring.

---

## Controller Replacement History

| Date | Event |
|------|-------|
| ~2025 | Original generic ASMedia ASM1061 SATA controller running |
| April 2026 | ASMedia ASM1061 diagnosed as faulty (ata10, sda/sdh errors) |
| April 18, 2026 | LANPAN SAS3008 ordered ($89.80) |
| April 19, 2026 | LANPAN SAS3008 installed; Seagate Exos X20 16TB drives (0007A62D, ZX3014DP) found incompatible with SAS |
| April 19, 2026 | 2x WD 12.7TB drives replaced with HGST OOS16000G (sdb) and Seagate (sde); resilver started |
| April 20, 2026 | Resilver completed; pool returned to ONLINE |
| May 2026 | YiBaoTong SAS9300-16i also present (two SAS3008 controllers total) |
| May 7, 2026 | Scrub completed — 0 errors |
| May 19, 2026 | Full hardware audit performed; root cause identified (drive age + bad SMART schedule) |

---

## SSH Config Note

- SSH key: `~/.ssh/id_rsa` (RSA key — TrueNAS authorized_keys uses RSA, not ed25519)
- SSH alias: `truenas`
- Admin user cannot run `zpool` or `smartctl` directly — use `midclt call` for TrueNAS API access