My FreeNAS system is back up and running – all is well with the world.
Corruption and Config Salvage
But it was pretty scary for a moment there. It started with this email from FreeNAS:
The boot volume state is ONLINE: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected.
Fortunately, it was just the freenas-boot pool. The corruption had rendered the web UI unaccessible, but I was able to gain root command line access from the console.
I tried a scrub, but it was no good:
pool: freenas-boot state: DEGRADED status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://illumos.org/msg/ZFS-8000-8A scan: scrub repaired 262K in 0h20m with 120 errors on Thu Apr 2 13:30:38 2015 config: NAME STATE READ WRITE CKSUM freenas-boot DEGRADED 0 0 3.08K da0p2 DEGRADED 0 0 6.24K too many errors
A ton of files were permanently damaged.
FreeNAS is intended to be run from a USB that can be discarded if there is any failure and replaced with a fresh image on a new USB stick and backed up configuration. Unfortunately, my backed up configuration file was a few weeks old and did not include some recent changes. Fortunately, I was able to download a copy of the configuration database via SFTP, thanks to some help from members of the excellent FreeNAS Forum.
Based on what I can tell, one non-integrated USB chip on the motherboard has probably failed. That would explain the original boot drive corruption as well as why a two USBs with a fresh, pure image wouldn’t boot on the same USB port, but boots just fine with the USB drive in a different port (not managed by the separate chip).
In the future, I will be mirroring the boot device, just in case, and backing up the configuration file after every change. (Oops, I already made a little change that I haven’t yet backed up… not off to a good start.)
And, better still, since it’s ZFS, despite the data corruption on the boot drive, my actual important data wasn’t affected or ever in any real danger. 🙂 Thanks, FreeNAS.