Home Linux Stuff

SYS:ONLINE

Predictive Drive Failure & S.M.A.R.T. Health Monitoring with smartd and Scrutiny

4 min read

Storage drives do not fail without warning. In over 80% of hardware failures, mechanical hard drives and NVMe SSDs exhibit distinct telemetry degradation—such as reallocated sector growth, pending sector queues, or NAND wear delta increases—weeks before catastrophic failure. Relying on filesystem errors to discover a bad drive often leads to corrupted RAID pools and unrecoverable data loss. This guide details deploying smartd for automated background diagnostic polling, scheduled non-destructive drive self-tests, and running Scrutiny in Docker for historical wear analytics and webhook alerting.

⚡ Quick Runbook: 60-Second Implementation

Deploy baseline SMART monitoring and automated test scheduling:

# 1. Install smartmontools
sudo apt-get update && sudo apt-get install -y smartmontools

# 2. Inspect connected drives
sudo smartctl --scan

# 3. Configure automated daemon monitoring with webhook alerting
sudo tee /etc/smartd.conf <<'CONFIG'
# Monitor all SATA, SAS, and NVMe drives automatically
# -a: Monitor all standard failure and warning metrics
# -o on: Enable automatic offline data collection
# -S on: Enable attribute autosave
# -s: Schedule Short Self-Test daily at 02:00, Extended Long Test Saturdays at 03:00
# -m: Alert recipient command
DEVICESCAN -H -m <nomailer> -M exec /usr/local/bin/smartd-notify.sh -o on -S on -s (S/../.././02|L/../../6/03)
CONFIG

# 4. Create the webhook alert script
sudo tee /usr/local/bin/smartd-notify.sh <<'CONFIG'
#!/usr/bin/env bash
MESSAGE="[SMART WARNING] Drive ${SMARTD_DEVICE} reported an issue: ${SMARTD_MESSAGE}"
curl -s -d "${MESSAGE}" https://ntfy.example.com/homelab-alerts
CONFIG
sudo chmod +x /usr/local/bin/smartd-notify.sh

# 5. Enable and start smartd
sudo systemctl enable --now smartd
sudo smartd -q onecheck

Key S.M.A.R.T. Metrics Explained

When reviewing drive telemetry, prioritize the following critical attributes:

Metric ID	Attribute Name	Failure Risk Threshold	Recommended Action
05	Reallocated Sector Count	Any value > 0 that increases over time	Plan replacement immediately; drive is developing surface defects.
C5	Current Pending Sector	Any value > 0	Unstable sectors waiting to be remapped; high probability of read failure.
C6	Offline Uncorrectable	Any value > 0	Defective sectors unrecoverable by ECC; data corruption likely.
NVMe 0x01	Critical Warning	Any bit set (non-zero)	Hardware reliability degraded; replace drive.
NVMe 0x05	Percentage Used	> 90%	NAND endurance exhausted; schedule SSD replacement before write lock.

Deploying Scrutiny Web Dashboard (Docker Compose)

For fleet-wide visual drive health dashboards and historical attribute tracking, deploy Scrutiny with raw drive access:

# /opt/stacks/scrutiny/compose.yaml
services:
  scrutiny:
    image: ghcr.io/analogj/scrutiny:master-omnibus
    container_name: scrutiny
    restart: unless-stopped
    ports:
      - "8080:8080"
    volumes:
      - /opt/stacks/scrutiny/config:/opt/scrutiny/config
      - /opt/stacks/scrutiny/influxdb:/opt/scrutiny/influxdb
      - /run/udev:/run/udev:ro
    cap_add:
      - SYS_RAWIO
      - SYS_ADMIN
    devices:
      - "/dev/sda:/dev/sda"
      - "/dev/sdb:/dev/sdb"
      - "/dev/nvme0n1:/dev/nvme0n1"

Manual Drive Health Testing Commands

Perform on-demand drive diagnostics when investigating suspected hardware issues:

# Check overall drive health assessment
sudo smartctl -H /dev/nvme0n1
sudo smartctl -H /dev/sda

# Display detailed attribute tables and error log history
sudo smartctl -A /dev/sda
sudo smartctl -l error /dev/sda

# Manually execute a non-destructive short background test (takes ~2 minutes)
sudo smartctl -t short /dev/sda

# Monitor test progress and results
sudo smartctl -l selftest /dev/sda

Related Runbooks in This Series

Tags: