FreeNAS Boot Corruption Recovery

My FreeNAS system is back up and running – all is well with the world.

Corruption and Config Salvage

But it was pretty scary for a moment there. It started with this email from FreeNAS:

The boot volume state is ONLINE: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected.

Fortunately, it was just the freenas-boot pool. The corruption had rendered the web UI unaccessible, but I was able to gain root command line access from the console.

I tried a scrub, but it was no good:

pool: freenas-boot
state: DEGRADED
status: One or more devices has experienced an error resulting in data
corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
entire pool from backup.
see: http://illumos.org/msg/ZFS-8000-8A
scan: scrub repaired 262K in 0h20m with 120 errors on Thu Apr  2 13:30:38 2015
config:

NAME        STATE     READ WRITE CKSUM
freenas-boot  DEGRADED     0     0 3.08K
  da0p2     DEGRADED     0     0 6.24K  too many errors

A ton of files were permanently damaged.

FreeNAS is intended to be run from a USB that can be discarded if there is any failure and replaced with a fresh image on a new USB stick and backed up configuration. Unfortunately, my backed up configuration file was a few weeks old and did not include some recent changes. Fortunately, I was able to download a copy of the configuration database via SFTP, thanks to some help from members of the excellent FreeNAS Forum.

What Happened?

Based on what I can tell, one non-integrated USB chip on the motherboard has probably failed. That would explain the original boot drive corruption as well as why a two USBs with a fresh, pure image wouldn’t boot on the same USB port, but boots just fine with the USB drive in a different port (not managed by the separate chip).

In the future, I will be mirroring the boot device, just in case, and backing up the configuration file after every change. (Oops, I already made a little change that I haven’t yet backed up… not off to a good start.)

And, better still, since it’s ZFS, despite the data corruption on the boot drive, my actual important data wasn’t affected or ever in any real danger. 🙂 Thanks, FreeNAS.