196 lines
9.1 KiB
Markdown
196 lines
9.1 KiB
Markdown
# TrueNAS Hardware Inventory & SMART Analysis
|
|
|
|
**Last updated:** 2026-05-19
|
|
**TrueNAS version:** 25.04.2.6 (TrueNAS SCALE Fangtooth)
|
|
**Kernel:** 6.12.15-debug+truenas
|
|
**Uptime:** 11 days, 22 hours at time of audit
|
|
**Hostname:** truenas
|
|
|
|
---
|
|
|
|
## System Board
|
|
|
|
- **Motherboard:** Gigabyte B450M DS3H WIFI-CF (AMD Ryzen AM4)
|
|
|
|
---
|
|
|
|
## Storage Controllers
|
|
|
|
| PCI Slot | Controller | Driver | Notes |
|
|
|----------|-----------|--------|-------|
|
|
| 03:00.0 | Broadcom/LSI SAS3008 (SAS 9300-16i) | mpt3sas | LANPAN HBA card (installed Apr 2026) |
|
|
| 05:00.0 | Broadcom/LSI SAS3008 (SAS 9300-16i) | mpt3sas | YiBaoTong SAS9300-16i (original replacement) |
|
|
| 06:00.1 | AMD 400 Series Chipset SATA | ahci | Onboard, 8 native ports |
|
|
| 0f:00.0 | AMD FCH SATA [AHCI mode] | ahci | Onboard |
|
|
| 0f:00.1 | AMD FCH SATA [AHCI mode] | ahci | Onboard |
|
|
|
|
> Note: All pool drives connect via the SAS3008 HBAs. The boot SSD connects to the onboard SATA (ata10). SMART data is read via SAT (SCSI-ATA Translation) through the SAS controllers.
|
|
|
|
---
|
|
|
|
## Drive Inventory
|
|
|
|
| Dev | Serial | Model | Size | HCTL (Host) | Pool Role | SMART Status |
|
|
|-----|--------|-------|------|-------------|-----------|--------------|
|
|
| sda | 9JG47UTT | WDC WUH721414ALE600 | 12.7 TB | 12:0:1:0 (SAS) | pool1 data | ✅ Clean |
|
|
| sdb | 0007A62D | HGST OOS16000G | 14.6 TB | 0:0:0:0 (SAS) | pool1 **spare** | ✅ Clean |
|
|
| sdc | 9RG9L8BC | WDC WUH721414ALE604 | 12.7 TB | 12:0:0:0 (SAS) | pool1 data | ✅ Clean |
|
|
| sdd | 9JG47Y2T | WDC WUH721414ALE600 | 12.7 TB | 12:0:2:0 (SAS) | pool1 data | ✅ Clean |
|
|
| sde | ZX3014DP | Seagate ST16000NM000D | 14.6 TB | 12:0:3:0 (SAS) | pool1 data | ✅ Clean |
|
|
| sdf | 9JJ5DJET | WDC WUH721414ALE604 | 12.7 TB | 12:0:5:0 (SAS) | pool1 data | ✅ Clean |
|
|
| sdg | 9JGV4MLT | WDC WUH721414ALE604 | 12.7 TB | 12:0:6:0 (SAS) | pool1 data | ✅ Clean |
|
|
| sdh | 9JH31S2T | WDC WUH721414ALE604 | 12.7 TB | 12:0:7:0 (SAS) | pool1 data | ⚠️ See below |
|
|
| sdi | 9JGZL01T | WDC WUH721414ALE604 | 12.7 TB | 12:0:4:0 (SAS) | pool1 data | ⚠️ See below |
|
|
| sdj | AA000000000000001308 | EDILOCA EN206 512GB SSD | 477 GB | 10:0:0:0 (SATA) | Boot drive | ✅ Clean |
|
|
|
|
**Pool drives:** 8 data + 1 hot spare
|
|
**Available space:** ~56 TB
|
|
**Pool configuration:** RAIDZ2 (tolerates 2 simultaneous drive failures)
|
|
|
|
---
|
|
|
|
## SMART Health Summary
|
|
|
|
### ✅ Clean Drives (7 drives)
|
|
|
|
All of the following have **zero** pending sectors, zero uncorrectable sectors, zero reallocated sectors, and no ATA errors:
|
|
|
|
- sda — 9JG47UTT (WDC WUH721414ALE600)
|
|
- sdb — 0007A62D (HGST OOS16000G) — hot spare
|
|
- sdc — 9RG9L8BC (WDC WUH721414ALE604)
|
|
- sdd — 9JG47Y2T (WDC WUH721414ALE600)
|
|
- sde — ZX3014DP (Seagate ST16000NM000D) — previously thought incompatible with SAS; now clean
|
|
- sdf — 9JJ5DJET (WDC WUH721414ALE604)
|
|
- sdg — 9JGV4MLT (WDC WUH721414ALE604)
|
|
|
|
### ⚠️ Drives Requiring Attention
|
|
|
|
#### sdh — Serial: 9JH31S2T (WDC WUH721414ALE604, 12.7 TB)
|
|
|
|
| SMART Attribute | Value |
|
|
|----------------|-------|
|
|
| Reallocated Sectors (ID 5) | 0 |
|
|
| Current Pending Sectors (ID 197) | **40** |
|
|
| Offline Uncorrectable (ID 198) | 0 |
|
|
| ATA Error Count | **85** |
|
|
| Self-Test Errors | **21** |
|
|
| Power-On Hours | ~53,029 hrs (~6.1 years) |
|
|
|
|
**Important context:** The 21 self-test errors are almost entirely caused by a misconfigured SMART schedule (see Fix section below) — a LONG self-test was set to run **every hour**. A long test on a 12.7 TB drive takes ~12 hours; starting a new one every hour continuously aborts the previous test, and each abortion is logged as a self-test failure. This generated endless alerts.
|
|
|
|
The **40 pending sectors** and **85 ATA errors** are real and need monitoring. Pending sectors are sectors the drive couldn't read; they haven't been confirmed bad yet (zero reallocated, zero uncorrectable). The drive is still functional but showing age-related wear.
|
|
|
|
#### sdi — Serial: 9JGZL01T (WDC WUH721414ALE604, 12.7 TB)
|
|
|
|
| SMART Attribute | Value |
|
|
|----------------|-------|
|
|
| Reallocated Sectors (ID 5) | **2** |
|
|
| Current Pending Sectors (ID 197) | **8** |
|
|
| Offline Uncorrectable (ID 198) | **6** |
|
|
| ATA Error Count | **17** |
|
|
| Self-Test Errors | 2 |
|
|
| Power-On Hours | ~53,503 hrs (~6.1 years) |
|
|
|
|
This drive has **confirmed bad sectors**: 6 offline uncorrectable and 2 already reallocated. This is a stronger warning sign than sdh. The drive is still online and ZFS is protecting data via RAIDZ2, but this drive should be prioritized for replacement.
|
|
|
|
---
|
|
|
|
## ZFS Pool Status
|
|
|
|
```
|
|
Pool: pool1
|
|
Type: RAIDZ2 (8 data drives + 1 hot spare)
|
|
Status: ONLINE — healthy
|
|
Warning: true (due to SMART alerts, not data errors)
|
|
|
|
Last Scrub:
|
|
Completed: ~May 7, 2026
|
|
Result: 0 errors (all data intact)
|
|
Duration: ~5.8 hours
|
|
Data read: 24.96 TB
|
|
```
|
|
|
|
**The scrub result is the most important number**: ZFS read every byte of data across all drives and found zero checksum errors. This confirms RAIDZ2 is protecting all data correctly despite the SMART errors on sdh and sdi.
|
|
|
|
---
|
|
|
|
## Root Cause Analysis: SMART Alert Flood
|
|
|
|
### What was observed
|
|
A large number of SMART CRITICAL alerts across multiple drives, across multiple controller changes, leading to the theory that the controller hardware was causing errors.
|
|
|
|
### Actual root cause (two separate issues)
|
|
|
|
**Issue 1: Drive age (real hardware wear)**
|
|
sdh and sdi have ~6.1 years of continuous runtime (53,000+ power-on hours). Enterprise WD Gold drives have a 5-year MTBF rating. These drives are past their design life and showing normal end-of-life bad sector accumulation. This is not a controller issue — the sectors on the drives themselves are worn.
|
|
|
|
**Issue 2: Misconfigured SMART test schedule (false alarm amplifier)**
|
|
Someone configured a LONG SMART self-test to run **every hour** on only one drive (9JH31S2T/sdh). This setting is still present in TrueNAS:
|
|
|
|
```
|
|
Schedule ID 1: LONG test, disk 9JH31S2T, schedule: every hour (all wildcards)
|
|
```
|
|
|
|
This caused an alert every single hour saying "new Self-Test Log error at hour timestamp XXXXX" — not because the drive was actively failing, but because each new hourly test was aborting the previous one.
|
|
|
|
### Why the "three controllers showing the same errors" was misleading
|
|
- Linux device names (sda, sdb, sdh...) are **not stable** — they can be reassigned on reboot
|
|
- Alert messages recorded serials like 9JH31S2T appearing on sda, sdd, sde, sdf, sdh, sdj at different times — this was the **same physical drive** being seen under different device names
|
|
- The controller changes never fixed the errors because the problem was always in the drives themselves, not the controllers
|
|
|
|
### Why different controllers also appeared to show errors
|
|
- The SATA controller was replaced with LANPAN SAS3008, and the old SAS9300-16i (YiBaoTong) may still be installed
|
|
- Both SAS3008 controllers read SMART data via SAT (SCSI-ATA Translation) — the elevated ATA error counts partially reflect SAT overhead, not pure drive errors
|
|
- But the pending and uncorrectable sectors are **stored on the drive itself** and cannot be faked by a controller
|
|
|
|
---
|
|
|
|
## Action Items
|
|
|
|
### 🔴 Immediate — Fix SMART test schedule
|
|
|
|
Delete the hourly LONG test schedule for 9JH31S2T (TrueNAS UI: Data Protection → S.M.A.R.T. Tests).
|
|
Replace it with a **monthly** LONG test (or use the global TrueNAS SMART schedule for all drives).
|
|
|
|
This will **immediately stop the hourly alert flood**.
|
|
|
|
### 🟡 Medium-term — Plan drive replacements
|
|
|
|
Both sdh (9JH31S2T) and sdi (9JGZL01T) are showing real bad sectors after ~6 years of operation. RAIDZ2 currently protects the data, but losing a third drive while these two are degraded would be data-threatening.
|
|
|
|
Replacement priority:
|
|
1. **sdi (9JGZL01T)** — first priority; has confirmed reallocated + uncorrectable sectors
|
|
2. **sdh (9JH31S2T)** — second priority; 40 pending sectors, high ATA error count
|
|
|
|
The hot spare **sdb (0007A62D, HGST OOS16000G, 14.6 TB)** is clean and ready to take over automatically if either drive fails completely.
|
|
|
|
Replacement drives recommended: WD Gold or Seagate Exos 14TB+ (WDC WUH721414ALE604 or newer).
|
|
|
|
### 🟢 Ongoing — Monthly scrubs
|
|
|
|
Run a ZFS scrub monthly to verify data integrity. TrueNAS default schedule handles this. The May 2026 scrub showed 0 errors — continue monitoring.
|
|
|
|
---
|
|
|
|
## Controller Replacement History
|
|
|
|
| Date | Event |
|
|
|------|-------|
|
|
| ~2025 | Original generic ASMedia ASM1061 SATA controller running |
|
|
| April 2026 | ASMedia ASM1061 diagnosed as faulty (ata10, sda/sdh errors) |
|
|
| April 18, 2026 | LANPAN SAS3008 ordered ($89.80) |
|
|
| April 19, 2026 | LANPAN SAS3008 installed; Seagate Exos X20 16TB drives (0007A62D, ZX3014DP) found incompatible with SAS |
|
|
| April 19, 2026 | 2x WD 12.7TB drives replaced with HGST OOS16000G (sdb) and Seagate (sde); resilver started |
|
|
| April 20, 2026 | Resilver completed; pool returned to ONLINE |
|
|
| May 2026 | YiBaoTong SAS9300-16i also present (two SAS3008 controllers total) |
|
|
| May 7, 2026 | Scrub completed — 0 errors |
|
|
| May 19, 2026 | Full hardware audit performed; root cause identified (drive age + bad SMART schedule) |
|
|
|
|
---
|
|
|
|
## SSH Config Note
|
|
|
|
- SSH key: `~/.ssh/id_rsa` (RSA key — TrueNAS authorized_keys uses RSA, not ed25519)
|
|
- SSH alias: `truenas`
|
|
- Admin user cannot run `zpool` or `smartctl` directly — use `midclt call` for TrueNAS API access
|