diff --git a/truenas-hardware-inventory.md b/truenas-hardware-inventory.md new file mode 100644 index 0000000..c1de2a0 --- /dev/null +++ b/truenas-hardware-inventory.md @@ -0,0 +1,195 @@ +# TrueNAS Hardware Inventory & SMART Analysis + +**Last updated:** 2026-05-19 +**TrueNAS version:** 25.04.2.6 (TrueNAS SCALE Fangtooth) +**Kernel:** 6.12.15-debug+truenas +**Uptime:** 11 days, 22 hours at time of audit +**Hostname:** truenas + +--- + +## System Board + +- **Motherboard:** Gigabyte B450M DS3H WIFI-CF (AMD Ryzen AM4) + +--- + +## Storage Controllers + +| PCI Slot | Controller | Driver | Notes | +|----------|-----------|--------|-------| +| 03:00.0 | Broadcom/LSI SAS3008 (SAS 9300-16i) | mpt3sas | LANPAN HBA card (installed Apr 2026) | +| 05:00.0 | Broadcom/LSI SAS3008 (SAS 9300-16i) | mpt3sas | YiBaoTong SAS9300-16i (original replacement) | +| 06:00.1 | AMD 400 Series Chipset SATA | ahci | Onboard, 8 native ports | +| 0f:00.0 | AMD FCH SATA [AHCI mode] | ahci | Onboard | +| 0f:00.1 | AMD FCH SATA [AHCI mode] | ahci | Onboard | + +> Note: All pool drives connect via the SAS3008 HBAs. The boot SSD connects to the onboard SATA (ata10). SMART data is read via SAT (SCSI-ATA Translation) through the SAS controllers. + +--- + +## Drive Inventory + +| Dev | Serial | Model | Size | HCTL (Host) | Pool Role | SMART Status | +|-----|--------|-------|------|-------------|-----------|--------------| +| sda | 9JG47UTT | WDC WUH721414ALE600 | 12.7 TB | 12:0:1:0 (SAS) | pool1 data | ✅ Clean | +| sdb | 0007A62D | HGST OOS16000G | 14.6 TB | 0:0:0:0 (SAS) | pool1 **spare** | ✅ Clean | +| sdc | 9RG9L8BC | WDC WUH721414ALE604 | 12.7 TB | 12:0:0:0 (SAS) | pool1 data | ✅ Clean | +| sdd | 9JG47Y2T | WDC WUH721414ALE600 | 12.7 TB | 12:0:2:0 (SAS) | pool1 data | ✅ Clean | +| sde | ZX3014DP | Seagate ST16000NM000D | 14.6 TB | 12:0:3:0 (SAS) | pool1 data | ✅ Clean | +| sdf | 9JJ5DJET | WDC WUH721414ALE604 | 12.7 TB | 12:0:5:0 (SAS) | pool1 data | ✅ Clean | +| sdg | 9JGV4MLT | WDC WUH721414ALE604 | 12.7 TB | 12:0:6:0 (SAS) | pool1 data | ✅ Clean | +| sdh | 9JH31S2T | WDC WUH721414ALE604 | 12.7 TB | 12:0:7:0 (SAS) | pool1 data | ⚠️ See below | +| sdi | 9JGZL01T | WDC WUH721414ALE604 | 12.7 TB | 12:0:4:0 (SAS) | pool1 data | ⚠️ See below | +| sdj | AA000000000000001308 | EDILOCA EN206 512GB SSD | 477 GB | 10:0:0:0 (SATA) | Boot drive | ✅ Clean | + +**Pool drives:** 8 data + 1 hot spare +**Available space:** ~56 TB +**Pool configuration:** RAIDZ2 (tolerates 2 simultaneous drive failures) + +--- + +## SMART Health Summary + +### ✅ Clean Drives (7 drives) + +All of the following have **zero** pending sectors, zero uncorrectable sectors, zero reallocated sectors, and no ATA errors: + +- sda — 9JG47UTT (WDC WUH721414ALE600) +- sdb — 0007A62D (HGST OOS16000G) — hot spare +- sdc — 9RG9L8BC (WDC WUH721414ALE604) +- sdd — 9JG47Y2T (WDC WUH721414ALE600) +- sde — ZX3014DP (Seagate ST16000NM000D) — previously thought incompatible with SAS; now clean +- sdf — 9JJ5DJET (WDC WUH721414ALE604) +- sdg — 9JGV4MLT (WDC WUH721414ALE604) + +### ⚠️ Drives Requiring Attention + +#### sdh — Serial: 9JH31S2T (WDC WUH721414ALE604, 12.7 TB) + +| SMART Attribute | Value | +|----------------|-------| +| Reallocated Sectors (ID 5) | 0 | +| Current Pending Sectors (ID 197) | **40** | +| Offline Uncorrectable (ID 198) | 0 | +| ATA Error Count | **85** | +| Self-Test Errors | **21** | +| Power-On Hours | ~53,029 hrs (~6.1 years) | + +**Important context:** The 21 self-test errors are almost entirely caused by a misconfigured SMART schedule (see Fix section below) — a LONG self-test was set to run **every hour**. A long test on a 12.7 TB drive takes ~12 hours; starting a new one every hour continuously aborts the previous test, and each abortion is logged as a self-test failure. This generated endless alerts. + +The **40 pending sectors** and **85 ATA errors** are real and need monitoring. Pending sectors are sectors the drive couldn't read; they haven't been confirmed bad yet (zero reallocated, zero uncorrectable). The drive is still functional but showing age-related wear. + +#### sdi — Serial: 9JGZL01T (WDC WUH721414ALE604, 12.7 TB) + +| SMART Attribute | Value | +|----------------|-------| +| Reallocated Sectors (ID 5) | **2** | +| Current Pending Sectors (ID 197) | **8** | +| Offline Uncorrectable (ID 198) | **6** | +| ATA Error Count | **17** | +| Self-Test Errors | 2 | +| Power-On Hours | ~53,503 hrs (~6.1 years) | + +This drive has **confirmed bad sectors**: 6 offline uncorrectable and 2 already reallocated. This is a stronger warning sign than sdh. The drive is still online and ZFS is protecting data via RAIDZ2, but this drive should be prioritized for replacement. + +--- + +## ZFS Pool Status + +``` +Pool: pool1 +Type: RAIDZ2 (8 data drives + 1 hot spare) +Status: ONLINE — healthy +Warning: true (due to SMART alerts, not data errors) + +Last Scrub: + Completed: ~May 7, 2026 + Result: 0 errors (all data intact) + Duration: ~5.8 hours + Data read: 24.96 TB +``` + +**The scrub result is the most important number**: ZFS read every byte of data across all drives and found zero checksum errors. This confirms RAIDZ2 is protecting all data correctly despite the SMART errors on sdh and sdi. + +--- + +## Root Cause Analysis: SMART Alert Flood + +### What was observed +A large number of SMART CRITICAL alerts across multiple drives, across multiple controller changes, leading to the theory that the controller hardware was causing errors. + +### Actual root cause (two separate issues) + +**Issue 1: Drive age (real hardware wear)** +sdh and sdi have ~6.1 years of continuous runtime (53,000+ power-on hours). Enterprise WD Gold drives have a 5-year MTBF rating. These drives are past their design life and showing normal end-of-life bad sector accumulation. This is not a controller issue — the sectors on the drives themselves are worn. + +**Issue 2: Misconfigured SMART test schedule (false alarm amplifier)** +Someone configured a LONG SMART self-test to run **every hour** on only one drive (9JH31S2T/sdh). This setting is still present in TrueNAS: + +``` +Schedule ID 1: LONG test, disk 9JH31S2T, schedule: every hour (all wildcards) +``` + +This caused an alert every single hour saying "new Self-Test Log error at hour timestamp XXXXX" — not because the drive was actively failing, but because each new hourly test was aborting the previous one. + +### Why the "three controllers showing the same errors" was misleading +- Linux device names (sda, sdb, sdh...) are **not stable** — they can be reassigned on reboot +- Alert messages recorded serials like 9JH31S2T appearing on sda, sdd, sde, sdf, sdh, sdj at different times — this was the **same physical drive** being seen under different device names +- The controller changes never fixed the errors because the problem was always in the drives themselves, not the controllers + +### Why different controllers also appeared to show errors +- The SATA controller was replaced with LANPAN SAS3008, and the old SAS9300-16i (YiBaoTong) may still be installed +- Both SAS3008 controllers read SMART data via SAT (SCSI-ATA Translation) — the elevated ATA error counts partially reflect SAT overhead, not pure drive errors +- But the pending and uncorrectable sectors are **stored on the drive itself** and cannot be faked by a controller + +--- + +## Action Items + +### 🔴 Immediate — Fix SMART test schedule + +Delete the hourly LONG test schedule for 9JH31S2T (TrueNAS UI: Data Protection → S.M.A.R.T. Tests). +Replace it with a **monthly** LONG test (or use the global TrueNAS SMART schedule for all drives). + +This will **immediately stop the hourly alert flood**. + +### 🟡 Medium-term — Plan drive replacements + +Both sdh (9JH31S2T) and sdi (9JGZL01T) are showing real bad sectors after ~6 years of operation. RAIDZ2 currently protects the data, but losing a third drive while these two are degraded would be data-threatening. + +Replacement priority: +1. **sdi (9JGZL01T)** — first priority; has confirmed reallocated + uncorrectable sectors +2. **sdh (9JH31S2T)** — second priority; 40 pending sectors, high ATA error count + +The hot spare **sdb (0007A62D, HGST OOS16000G, 14.6 TB)** is clean and ready to take over automatically if either drive fails completely. + +Replacement drives recommended: WD Gold or Seagate Exos 14TB+ (WDC WUH721414ALE604 or newer). + +### 🟢 Ongoing — Monthly scrubs + +Run a ZFS scrub monthly to verify data integrity. TrueNAS default schedule handles this. The May 2026 scrub showed 0 errors — continue monitoring. + +--- + +## Controller Replacement History + +| Date | Event | +|------|-------| +| ~2025 | Original generic ASMedia ASM1061 SATA controller running | +| April 2026 | ASMedia ASM1061 diagnosed as faulty (ata10, sda/sdh errors) | +| April 18, 2026 | LANPAN SAS3008 ordered ($89.80) | +| April 19, 2026 | LANPAN SAS3008 installed; Seagate Exos X20 16TB drives (0007A62D, ZX3014DP) found incompatible with SAS | +| April 19, 2026 | 2x WD 12.7TB drives replaced with HGST OOS16000G (sdb) and Seagate (sde); resilver started | +| April 20, 2026 | Resilver completed; pool returned to ONLINE | +| May 2026 | YiBaoTong SAS9300-16i also present (two SAS3008 controllers total) | +| May 7, 2026 | Scrub completed — 0 errors | +| May 19, 2026 | Full hardware audit performed; root cause identified (drive age + bad SMART schedule) | + +--- + +## SSH Config Note + +- SSH key: `~/.ssh/id_rsa` (RSA key — TrueNAS authorized_keys uses RSA, not ed25519) +- SSH alias: `truenas` +- Admin user cannot run `zpool` or `smartctl` directly — use `midclt call` for TrueNAS API access