You're mid-deploy on a production EC2 instance, or maybe just saving a config file, and suddenly bash throws Read-only file system. Every write fails. touch /tmp/test fails. Your app logs stop. The first time this happens it looks like a permissions problem, but it isn't. The kernel did this on purpose.
What's actually happening here
When the Linux kernel spots an I/O error on a mounted filesystem — or the filesystem's errors= policy triggers — it flips that mount to read-only. The mount option in /etc/fstab is usually errors=remount-ro for EXT4 root filesystems, which is the default. The kernel doesn't want to keep writing to a disk that might be corrupting data, so it protects you by refusing writes.
The error is EROFS (Error 30, "Read-only file system"). It's not a disk failure by itself — it's the kernel's response to one. Underneath, you'll usually find one of these in dmesg:
EXT4-fs error (device sda1): ext4_journal_check_start:83: Detected aborted journal
EXT4-fs (sda1): Remounting filesystem read-only
The trigger is almost always one of: an unclean shutdown (power loss on a VM), a full disk where the journal couldn't commit, a failing SSD throwing I/O errors, or a snapshot restore that left the journal in a bad state. I've seen it most often on older Xen-based EC2 instances after the hypervisor reboots the host mid-write.
The fix
Don't just remount rw and keep going. That hides the corruption and it will bite harder next time. Here's the sequence:
- Check what the kernel is complaining about. This tells you whether it's a transient I/O blip or real corruption.
dmesg -T | grep -iE 'ext4|I/O error|remount|journal' | tail -40- Identify the mount and its state. Find the device behind the read-only mount.
mount | grep 'ro,'
df -hT | grep -v tmpfsYou'll see something like /dev/sda1 on / type ext4 (ro,relatime,errors=remount-ro). Note the device.
- Try a remount as a quick diagnostic. If it remounts cleanly and the journal wasn't aborted, you might get lucky — the error was transient.
sudo mount -o remount,rw /If that returns mount: /: cannot remount /dev/sda1 read-write, is write-protected, or dmesg immediately logs another ext4 error, the journal is aborted. Stop here and do step 4.
- Drop to single-user or rescue mode. You can't fsck a mounted rw filesystem. Either boot into rescue/recovery mode from the hypervisor console, or if it's a remote box, use
systemctl rescue.
sudo systemctl rescue- Unmount the filesystem. If it's your root, you'll need to be booted from a live or rescue image.
sudo umount /dev/sda1- Run e2fsck on the device. The
-fforces a full check even if the superblock thinks the fs is clean.-yauto-answers yes. Don't skip -f.
sudo e2fsck -fy /dev/sda1Expect output like "Inode 12345, i_blocks is X, should be Y. Fix?" and "Free blocks count wrong." Let it fix. If the primary superblock is corrupted, e2fsck will fall back to a backup: e2fsck -b 32768 /dev/sda1 (32768 is the standard backup location on 4K-block EXT4).
- Remount and verify. After a clean fsck, mount it rw.
sudo mount -o rw /dev/sda1 /mnt
touch /mnt/test && rm /mnt/testIf you were in rescue mode, exit to return to normal boot.
If it still fails
If e2fsck keeps finding errors on every run, the disk itself is failing. Check SMART data:
sudo smartctl -a /dev/sda | grep -iE 'Reallocated|Pending|Uncorrectable'Any non-zero Reallocated_Sector_Ct or Current_Pending_Sector means the drive is on its way out. Replace it — no amount of fsck will fix dying NAND or a scratched platter.
If e2fsck reports superblock unreadable and refuses to use a backup, don't guess block sizes. Run dumpe2fs /dev/sda1 | grep -i superblock from a rescue image — it lists every backup superblock location. Try them in order: e2fsck -b 32768 -B 4096 /dev/sda1, then -b 98304, and so on.
For virtualized cloud instances, also check the underlying volume. On AWS, an EBS volume can hit its burst balance and throttle, which surfaces as I/O errors to the guest. aws ec2 describe-volume-status --volume-ids vol-xxxxx will show impaired if that's the cause — in that case, stop the instance, detach, snapshot, and rebuild from the snapshot. The read-only state is a symptom, not the disease.
Don't disable the errors=remount-ro policy to "fix" this. That option has saved more data than it's ever cost. If you're seeing it regularly, your disk or your power is the problem.