Project

General

Profile

Bug #7068

After upgrade with Webinterface from 9.2.1.9 to 9.3-stable locked volumes can't be decrypted

Added by Pascal Baumgardt almost 6 years ago. Updated about 3 years ago.

Status:
Closed: User Config Issue
Priority:
Nice to have
Assignee:
Xin Li
Category:
OS
Target version:
Seen in:
Severity:
New
Reason for Closing:
Reason for Blocked:
Needs QA:
Yes
Needs Doc:
Yes
Needs Merging:
Yes
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Description

After upgrade with Webinterface from 9.2.1.9 to 9.3-stable locked volumes can't be decrypted. If you try to decrypt with the passphrase and/or geli.key it gives out following error:

Error: Volume could not be imported: 2 devices failed to decrypt

History

#1 Updated by Jordan Hubbard almost 6 years ago

  • Assignee set to Xin Li
  • Target version changed from 9.3-RELEASE to Unspecified

I thought this was fixed. Xin, can you investigate? Thanks.

#2 Updated by Josh Paetzel almost 6 years ago

What is your pool config? eg: mitrror, RAIDZ, how many devices etc etc etc.

#3 Updated by Pascal Baumgardt almost 6 years ago

I got a 2x1x2.0 TB Mirror (ada0/ada1).

I'm using the built-in software raid from freenas.

Perhaps I explain what I did exactly: I was running freenas 9.2.1.8 (or maybe 9.2.1.9, I can't remember surely) on a pretty old Pentium 4 32bit/386 machine. So I was limited to 4 GB RAM.
Freenas started on a 8 GB USB-Stick and I've installed two 2 TB HDDs as "Mirror" in freenas.

I've downloaded the Webinterface-Update for freenas 9.3 (FreeNAS-9.3-RELEASE.GUI_Upgrade.txz an installed it from the GUI/Webinterface. Before I've started I saved the config-db as it said. I started the update. It said, everthing went ok and the system would reboot. It shut down, but didn't start again, there was only the cursor blinking.

I guessed, my old motherboard with a bios from 2005 wouldn't be able to handle the GPT Partition. So I've started on the machine itself (not via GUI/Webinterface) the FreeNAS-9.3-RELEASE.iso on CD. (I think this is a 64-bit only version of what I wasn't aware) The installer asked me to upgrade. So I choose to "upgrade". But then it didn't start anymore, I guess it was the problem with the bios. So I moved my disks and installation to a newer 64-bit machine and now it started, but there was a mounting error.

So I've formated the usb-start-stick. I hope this wasn't a terrible fault, but I thought I have the config.db and there was a info in a readme-file, I could also try it with a fresh install, import the config.db and all fine. So I did.

Finally I got it work again with freenas 9.3 (on the newer 64bit /Core 2 machine) and also on the old machine (Pentium 4, 32bit/386) with 9.2.1.9. Everthing looks nice, of my config. But if I want to decrypt my volume(s) it gives me the above mentioned errors.

#4 Updated by Pascal Baumgardt almost 6 years ago

Do you need any further infos? If yes, where can I get them? Sorry, I'm a newbie on freenas.

#5 Updated by Xin Li almost 6 years ago

  • Status changed from Unscreened to 15

Do you have backups of your encryption key? Sounds like you have lost your keys during the re-install?

#6 Updated by Pascal Baumgardt almost 6 years ago

I have, but it doesn't work. Perhaps is an old version of it, even though I don't think so. Is it possible, that it doesn't work because of an error? The keys are not safed in the config.db, right? I just have the config.db and the passphrase. I also have the geli.key and geli_recovery.key, but I'm not 100% sure that they are the latest version. I can't say. I didn't back up the keys before the update, it would be could, if there would be at least a notice to the user, that he has to backup them before updating. I thought backup of config.db and the passphrase would be enough.

#7 Updated by Pascal Baumgardt almost 6 years ago

Or is there any other chance to unlock the volume I still can see in the FreeNAS GUI? Or to find out, if I got the latest keys? (the keys size is 64 KB, right?)

#8 Updated by Jordan Hubbard almost 6 years ago

  • Status changed from 15 to Screened

#9 Updated by Xin Li almost 6 years ago

Pascal Baumgardt wrote:

I have, but it doesn't work. Perhaps is an old version of it, even though I don't think so. Is it possible, that it doesn't work because of an error? The keys are not safed in the config.db, right? I just have the config.db and the passphrase. I also have the geli.key and geli_recovery.key, but I'm not 100% sure that they are the latest version. I can't say. I didn't back up the keys before the update, it would be could, if there would be at least a notice to the user, that he has to backup them before updating. I thought backup of config.db and the passphrase would be enough.

No, the keys are not part of config.db. However, geli.key and geli_recovery.key should be the right files that you should have kept.

Once set, the key(s) never expire until you explicitly do a "rekey", a change of passphrase, or delete the backup key which will overwrite the key slot.

Note that you should be able to use your backed up geli.key in place of recovery key, is it possible for you to use a new USB device and import the pool with that geli.key?

#10 Updated by Pascal Baumgardt almost 6 years ago

If I try to import the pool with the geli.key ("Import Volume" in the GUI) I get following message:

The following disks failed to attach: gptid/a1a11b02-5e98-11e4-95e3-00300598d47b, gptid/a1fce450-5e98-11e4-95e3-00300598d47b

#11 Updated by Jordan Hubbard almost 6 years ago

BRB: When you upgraded, you used a new thumb drive, correct? Can you still import the two drives on the old system (pre-9.3) configuration? If not, I'm afraid there's probably nothing we can do here. Your data is gone.

#12 Updated by Pascal Baumgardt over 5 years ago

No, I used the same thumb drive, I did an upgrade on the existing installation (9.2.1.8 -> 9.3 via Web-GUI). Because my machine didn't start anymore I've tried a fresh install on the same thumbdrive (as I was told in the readme from the upgrade information!), without having backed it up before - this was a big mistake as I see now. I guess the latest geli.key and geli_recovery.key where only on the thumbdrive, but I've overwritten it with the new installation (9.3) after the upgrade process failed.

I can load the config.db into an fresh 9.2.1.8 installation and I can see my two drives then. But if I detach the drives and try to "auto import volume" it fails with the error from the very first posting here (Error: Volume could not be imported: 2 devices failed to decrypt). Same if I try to "auto import volume" on a fresh 9.2.1.8 installation without prior loading the config.db from my former installation.

#13 Updated by Jordan Hubbard over 5 years ago

  • Priority changed from Important to Nice to have

#14 Updated by Jordan Hubbard over 5 years ago

  • Status changed from Screened to Closed: User Config Issue

Yeah, nothing for us to do here I'm afraid - the data is gone. Using encryption on FreeNAS is definitely "advanced class" and there are no seatbelts to save you if you don't have key backups elsewhere. This is why we usually tell folks not to even try to use it unless they're comfortable with self-supported key escrow methodologies.

#15 Updated by Pascal Baumgardt over 5 years ago

Hm, thanks for your info. But I have the key backups for

a) masterkeys from ada0p2 and ada1p2
b) Userkey and Passphrase (geli.key)
c) Recovery key (geli_recovery.key)

Also if I read this post on Feature #3206, #11 I still got hope:

Von Dusan Lacko vor 11 Monaten aktualisiert

Andrew Johnson wrote:

Is the workaround just shell into freenas, run geli backup myself on all my drives, then back up those master key files somewhere safe?
And to restore a master key in the event of corruption, I'd need to run geli restore, providing the appropriate backup file?

I use geli and that's what I do. I have backups of the geli metadata from all my drives -- I use the drive serial numbers as file names so that if I need to restore I can easily identify which file goes with which drive (http://forums.freenas.org/threads/please-validate-my-backup-plan-rotating-offsite-backup-disks-from-single-freenas-primary-storage.17316/#post-93073).
There is one security implication you need to be aware of. The "user visisble" key in FreeNAS is the keyfile & passphrase. If those two get compromised you can rekey the encryption & change the passphrase. However, if your geli metadata gets also compromised you basically need to destroy the pool. Because even if you "rekey" the encryption the attacker has all the information he needs to recover the master key. See this description on how the encryption in FreeNAS works: http://forums.freenas.org/threads/recover-encryption-key.16593/#post-85497
However, keep in mind that even in the first scenario (attacker has "only" the keyfile & passphrase) if the attacker manages to take a peek at your drive before you rekey then you also lost -- he will be able to read your data even after you rekey.

If I summarize that correctly I should be able to reconstruct a working recovery-key or user-key and passphrase if I got the actual:

a) masterkeys from my raidz mirrored two drives
b) any version of a user-key and the correspondending correct passphrase for this pool (even if it would be an outdated version).

Am I correct?

#16 Updated by Xin Li over 5 years ago

Pascal Baumgardt wrote:

Hm, thanks for your info. But I have the key backups for

a) masterkeys from ada0p2 and ada1p2
b) Userkey and Passphrase (geli.key)
c) Recovery key (geli_recovery.key)

Assuming you mean 'metadata' in 'masterkey' (which you can not backup: master key is encrypted with your passphrase and/or user keys and possibly also with the recovery key/passphrase; when you backup it you only get the encrypted copy of it) on the above, and either the key or geli_recover.key is created before your backup (i.e. in sync), yes you can recover the data on ada0p2 and ada1p2.

If I summarize that correctly I should be able to reconstruct a working recovery-key or user-key and passphrase if I got the actual:

a) masterkeys from my raidz mirrored two drives
b) any version of a user-key and the correspondending correct passphrase for this pool (even if it would be an outdated version).

Am I correct?

There is no such thing 'raidz mirrored' (it can be single disk, raidz[,2,3], mirrored, or some mixture of these stripped) but if your pool is a mirrored configuration AND it consists only ada0p2 and ada1p2, then yes, you can use your old metadata combined with your old passphrase plus geli.key, or your recovery key to recover data by restoring the metadata to their drives, then use GUI to unlock.

So in general: yes I think your data is still recoverable, but be advised to use extra caution, let us know if you need further information and good luck!

#17 Updated by Pascal Baumgardt over 5 years ago

Thanks so much for your fast answer which gives me a bit hope.

Oops, you're right, I mixed it up with the raidz, mirror and so on. I set up a "simple" mirror within the GUI of FreeNAS unter "Volume Manager". I used two disks, ada0 and ada1. ada0 has the two partitions ada0p1 and ada0p2 and ada1 got the two partitions ada1p1 and ada1p2.

I used the command "geli backup /dev/ada0p2 /home/master_ada0p2.key" and "geli backup /dev/ada1p2 /home/master_ada1p2.key". I got two (encrypted) keys now. So these are only the masterkeys, not the metadata, right? How could I get the metadata? Is it possible at this state?

Furthermore I got a geli.key and a geli_recovery.key, which both were created before the backup of the masterkeys.

I also noticed this error message on my server:

Cron <root@nas> PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/root/bin" /usr/local/sbin/scrub -t 35 nas

cannot open 'nas': no such pool
cannot open 'nas': no such pool
usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ...
[-f fmt date | [[[[[cc]yy]mm]dd]HH]MM[.ss]] [+format]
expr: syntax error
[: -lt: unexpected operator
cannot open 'nas': no such pool
starting scrub of pool 'nas'
cannot open 'nas': no such pool

where 'nas' is the name of my mirrored volume. Perhaps there is something different wrong, not the keys?

I would be very very grateful if you could give me further information how to go on and perhaps recover my data! THANKS SO MUCH IN ADVANCE!

#18 Updated by Pascal Baumgardt over 5 years ago

I did now a "geli dump /dev/ada0p2" and a "geli dump /dev/ada1p2" and got the masterkeys and salt (or metadata?!). Does this help?

#19 Updated by Pascal Baumgardt over 5 years ago

Happy New Year to everyone! :) I really got no clue on how I should proceed. Perhaps anyone can give me a hint?

#20 Updated by Pascal Baumgardt over 5 years ago

Pascal Baumgardt wrote:

Happy New Year to everyone! :) I really got no clue on how I should proceed. Perhaps anyone can give me a hint?

I still couldn't solve my problem :-(. Is there anyone out there who could possibly help me? That would be very appreciated!

#21 Avatar?id=14398&size=24x24 Updated by Kris Moore about 3 years ago

  • Target version changed from Unspecified to N/A

Also available in: Atom PDF