Fix NS_I_DISK_STOP 0X400D0198 Disk Offline

Yeah, that disk alert is annoying. Here’s the fix.

You’re staring at NS_I_DISK_STOP (0X400D0198) - Disk %1 ( %2 ) on Content Server %3, has been offlined.

It means your NetStorage or Content Server node lost contact with a physical disk. The server didn’t crash — it just marked that disk as offline to protect data integrity. The cause is almost always I/O timeouts — the disk took too long to respond. Here’s how you bring it back without restarting the whole server.

Step 1 — Confirm which disk is actually offline

diskutil list
df -h | grep storage

Look for the mount point referenced in the alert. It’s usually something like /storage/disk1 or /data/disk2. If the mount point is missing or shows errors, that’s your disk. Also check dmesg | tail -20 for I/O errors — you’ll see task abort or device not ready messages.

Step 2 — Rescan the disk (the real fix)

The fastest way to get the disk back is to rescan for new or removed devices. On the Content Server, run:

echo 1 > /sys/class/scsi_disk/<disk_device>/device/rescan

To find <disk_device>, check ls -la /dev/disk/by-path/ or lsscsi. For example, if the disk is /dev/sdb, the path looks like /sys/class/scsi_disk/0:0:0:1/device/rescan. After rescanning, wait 10 seconds and check dmesg again. You should see the disk being re-added.

Don’t have a scsi_disk path? Use lsblk to confirm the block device name first.

Step 3 — Force the disk back online in the storage engine

Even after the OS sees the disk, the Content Server’s storage service might still think it’s offline. You need to tell it to re-check. Run:

nsstorage --disk-online <disk_id>

You’ll find <disk_id> in the original alert — it’s the %1 field. If you don’t have the exact ID, list disks with:

nsstorage --list-disks

That shows each disk’s status. Look for one with offline status. Then run --disk-online against its ID.

After that, the alert should clear. Check with nsstorage --status.

Why this works

The error 0X400D0198 isn’t a hardware death sentence — it’s a timeout guard. Content Servers run a heartbeat check against each disk. If a disk doesn’t respond within the configured timeout (usually 30-60 seconds), the server sets it offline. That keeps the rest of the storage group from hanging on a stuck I/O request.

What causes the timeout? Common triggers:

Disk controller saturation — too many concurrent reads/writes on a single SAS/SATA backplane.
Bad cable or backplane slot — intermittent signal loss that doesn’t fully disconnect.
Firmware bug on the disk — some SSDs from 2019-2020 had a known issue where they’d drop offline after 30 consecutive seconds of high queue depth.
File system metadata operation stuck — an fsck or extended stat on a directory with millions of small files can block the disk for longer than the timeout.

The rescan forces the SCSI layer to re-probe the device. The --disk-online command tells the storage engine “yes, this device is back, trust it again.” If you skip step 3, the OS sees the disk but the Content Server ignores it.

Less common variations

Variation 1 — The disk never came back after rescan

If dmesg shows I/O error, dev sdb, sector X after rescanning, the disk has bad sectors. The Content Server will offline it again immediately. In that case, you need to replace the disk. Run smartctl -a /dev/sdb to check. If Reallocated_Sector_Ct is above 100 or Pending_Sector is non-zero, it’s time for a swap.

After replacement, run nsstorage --add-disk <new_disk_path> to integrate it into the storage group.

Variation 2 — Multiple disks offline simultaneously

This points to a backplane or controller issue. If three disks on the same HBA go offline, it’s not three independent failures. Rescan all of them, but also check lspci | grep LSI (or your HBA model). If the controller itself is degraded, replace the HBA first. Some Content Servers use mpt3sas drivers — a firmware mismatch can cause mass timeouts.

Variation 3 — The disk is a solid-state drive (NVMe)

NVMe drives use a different rescan path. Instead of /sys/class/scsi_disk, use:

echo 1 > /sys/class/nvme/nvme0/device/rescan_controller

Then nvme list to verify. After that, the same nsstorage --disk-online command applies.

Prevention — stop it from happening again

The pattern is always the same: timeout → offline → rescan → back. To break that loop:

Increase the disk timeout on the Content Server. Edit /etc/nsstorage.conf and change disk_timeout=60 to disk_timeout=120. That gives the disk twice as long to respond before being marked offline. Don’t go above 180 — you don’t want a truly dead disk holding up writes for three minutes.
Monitor I/O queue depth. Use iostat -xd 5 during normal load. If avgqu-sz stays above 4 for any disk, you’re saturating the controller. Spread load across more disks or upgrade to faster media.
Update disk firmware. Check the vendor’s site for your model. Many “offline during burst” bugs were fixed in firmware releases from 2021 and later.
Enable persistent device names. Use /dev/disk/by-id/ or /dev/disk/by-uuid/ in your storage config. That way, if a disk reappears on a different SCSI address after rescan, the server still recognizes it.

One last thing — don’t ignore this alert. If it fires more than once a month for the same disk, that disk is failing. Replace it. The Content Server is designed to tolerate single-disk outages, but you lose redundancy. Two offline disks in the same storage group means data loss risk.