[gentoo-user] Fileserver with Raid + Crypto + BtrFS

[gentoo-user] Fileserver with Raid + Crypto + BtrFS

[Ralf]

Hi,

at the moment i'm running a gentoo box as fileserver, the setup is as
follows:

 -----------------------------------
|               Ext4                |
 -----------------------------------
                  ^
                  |
 -----------------------------------
|               LUKS                |
 -----------------------------------
                  ^
                  |
 -----------------------------------
|            MD Raid 10             |
 -----------------------------------
   ^         ^         ^         ^
   |         |         |         |
 -----     -----     -----     -----
| hd0 |   | hd1 |   | hd2 |   | hd3 |
 -----     -----     -----     -----


Besides other features, I'd like to have a snapshot functionality, and I
don't want to introduce an additional layer by using LVM. So I'm
thinking about to migrate to Btrfs.

Btrfs supports Raid10 but no block-crypto.

If I would use a HD->MD Raid->Luks->Btrfs stack, I don't benefit from
the Raid implementation of Btrfs.
If I would use a HD->Luks->Btrfs stack, then I would have to use four
different LUKS devices, which results in four individual encryptions
(and I don't have AES-NI, so this would be a tremendous slowdown).

What would be the best way to have a Raid 10 together with a encrypted
Btrfs?

--

Besides that I would have to live-migrate the Raid10 as I don't have any
spare hdd to cache the data. So I would have to degrade my Raid10.
Is it possible to create a degraded Btrfs?

Please don't tell me that this is insanely dangerous, I know it is ;-)

Cheers
  Ralf

Re: [gentoo-user] Fileserver with Raid + Crypto + BtrFS

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 793 bytes --]

On Wed, 11 Nov 2015 18:09:39 +0100, Ralf wrote:

> Btrfs supports Raid10 but no block-crypto.
> 
> If I would use a HD->MD Raid->Luks->Btrfs stack, I don't benefit from
> the Raid implementation of Btrfs.

Nor do you get the automatic repair of corruption that btrfs RAID offers.

> If I would use a HD->Luks->Btrfs stack, then I would have to use four
> different LUKS devices, which results in four individual encryptions
> (and I don't have AES-NI, so this would be a tremendous slowdown).

It would definitely be slower, but maybe not "tremendously".
 
> What would be the best way to have a Raid 10 together with a encrypted
> Btrfs?

What about crypto on top of btrfs using a stacked filesystem like
ecryptfs?


-- 
Neil Bothwick

A good pun is its own reword.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-user] Fileserver with Raid + Crypto + BtrFS

[Ralf]

On 11/11/2015 07:19 PM, Neil Bothwick wrote:
> On Wed, 11 Nov 2015 18:09:39 +0100, Ralf wrote:
>
>> Btrfs supports Raid10 but no block-crypto.
>>
>> If I would use a HD->MD Raid->Luks->Btrfs stack, I don't benefit from
>> the Raid implementation of Btrfs.
> Nor do you get the automatic repair of corruption that btrfs RAID offers.
Oh cool, nice, I didn't know about that feature. But as you say, it's
definitely better using btrfs's raid instead of using stacked md raid.
>
>> If I would use a HD->Luks->Btrfs stack, then I would have to use four
>> different LUKS devices, which results in four individual encryptions
>> (and I don't have AES-NI, so this would be a tremendous slowdown).
> It would definitely be slower, but maybe not "tremendously".
Well yes, I would say so. My Box doesn't have AES-NI instruction set and
it 'only' has to relatively slow cores.
4x independent Luks results in 4x independent (en|de)cryption. Even now,
in my current configuration AES slows everything extremely down.
(before setting up my disks a few years ago, i benchmarked the setup
with and without luks. Afair, without Luks I had about Read:80Mib/s,
with Luks it's about 50MiB/s, and yes, everything is aligned correctly)
>  
>> What would be the best way to have a Raid 10 together with a encrypted
>> Btrfs?
> What about crypto on top of btrfs using a stacked filesystem like
> ecryptfs?
Nope, I also thought about that, but this is not elegant. Besides that,
it would also slow down the system as ecryptfs runs in the VFS layer and
is yet another layer which operates on top of an existing filesystem.
(and not like luks, which would run a layer below btrfs). So that's a
lot of overhead.

Ecryptfs is really nice for encrypting dedicated files or directories
but I don't think that it is a good solution for encrypting a _whole_
general purpose filesystem.

And thinking about btrfs snapshot feature, using some 'btrfs history
tool', i would probably only be able to see a lot of crypto garbage when
going through my history (which can for sure be accessed by ecryptfs,
but not by standard btrfs tools).

Cheers
  Ralf

Re: [gentoo-user] Fileserver with Raid + Crypto + BtrFS

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 970 bytes --]

On Wed, 11 Nov 2015 22:18:15 +0100, Ralf wrote:

> >> What would be the best way to have a Raid 10 together with a
> >> encrypted Btrfs?  
> > What about crypto on top of btrfs using a stacked filesystem like
> > ecryptfs?  
> Nope, I also thought about that, but this is not elegant. Besides that,
> it would also slow down the system as ecryptfs runs in the VFS layer and
> is yet another layer which operates on top of an existing filesystem.
> (and not like luks, which would run a layer below btrfs). So that's a
> lot of overhead.

But you're only doing it once.
 
> Ecryptfs is really nice for encrypting dedicated files or directories
> but I don't think that it is a good solution for encrypting a _whole_
> general purpose filesystem.

Do you need to encrypt everything? Being able to only encrypt the
directories that need it may be an advantage.


-- 
Neil Bothwick

Everything should be made as simple as possible, but no simpler.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-user] Fileserver with Raid + Crypto + BtrFS

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 583 bytes --]

On Wed, 11 Nov 2015 22:18:15 +0100, Ralf wrote:

> And thinking about btrfs snapshot feature, using some 'btrfs history
> tool', i would probably only be able to see a lot of crypto garbage when
> going through my history (which can for sure be accessed by ecryptfs,
> but not by standard btrfs tools).

That's a good point, you'd need to mount each snapshot before you could
read it.

I *really* wish btrfs had its own encryption, but I suspect this would be
far from trivia to implement.


-- 
Neil Bothwick

Nothing is foolproof to a sufficiently talented fool.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-user] Fileserver with Raid + Crypto + BtrFS

[Marc Joliet]

[-- Attachment #1: Type: text/plain, Size: 935 bytes --]

On Wednesday 11 November 2015 23:08:44 Neil Bothwick wrote:
>On Wed, 11 Nov 2015 22:18:15 +0100, Ralf wrote:
>> And thinking about btrfs snapshot feature, using some 'btrfs history
>> tool', i would probably only be able to see a lot of crypto garbage when
>> going through my history (which can for sure be accessed by ecryptfs,
>> but not by standard btrfs tools).
>
>That's a good point, you'd need to mount each snapshot before you could
>read it.
>
>I *really* wish btrfs had its own encryption, but I suspect this would be
>far from trivia to implement.

From a presentation [0] earlier this year I understand that Chris Mason really 
wants it, too, but has something else he wanted to finish first (I don't 
remember what, though).

[0] https://www.youtube.com/watch?v=W3QRWUfBua8

-- 
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-user] Fileserver with Raid + Crypto + BtrFS

[Marc Stürmer]

Am 11.11.2015 um 18:09 schrieb Ralf:

> Besides that I would have to live-migrate the Raid10 as I don't have any
> spare hdd to cache the data. So I would have to degrade my Raid10.
> Is it possible to create a degraded Btrfs?

You can create your shiny, new Btrfs on one device only first and then 
add more devices later as needed of course.

Re: [gentoo-user] Fileserver with Raid + Crypto + BtrFS

[Nuno Magalhães]

On Wed, Nov 11, 2015 at 5:09 PM, Ralf
<ralf+gentoo@ramses-pyramidenbau.de> wrote:
> So I'm
> thinking about to migrate to Btrfs.

Have you considered ZFS?
I currently have some disks with {fs}+LVM+RAID1 and others with a ZFS
mirror (no extra disks for ARC or anything), both approaches seem
manageable. To me btrfs still seems "not-ready-yet", but that's just
me.

Can't offer any real benchmarks, i'm just starting out, but the
correct comparison seems to be btrfs vs ZFS, not btrfs vs fs+LVM+RAID.

Cheers,
Nuno

Re: [gentoo-user] Fileserver with Raid + Crypto + BtrFS

[J. Roeleveld]

On 12 November 2015 00:14:15 CET, "Nuno Magalhães" <nunomagalhaes@eu.ipp.pt> wrote:
>On Wed, Nov 11, 2015 at 5:09 PM, Ralf
><ralf+gentoo@ramses-pyramidenbau.de> wrote:
>> So I'm
>> thinking about to migrate to Btrfs.
>
>Have you considered ZFS?
>I currently have some disks with {fs}+LVM+RAID1 and others with a ZFS
>mirror (no extra disks for ARC or anything), both approaches seem
>manageable. To me btrfs still seems "not-ready-yet", but that's just
>me.
>
>Can't offer any real benchmarks, i'm just starting out, but the
>correct comparison seems to be btrfs vs ZFS, not btrfs vs fs+LVM+RAID.
>
>Cheers,
>Nuno

I think for small amount of disks (around 4) btrfs is a better option.
For larger amounts (think 10+) ZFS is a better option.

This is based on the design ideas and due to the lack of a robust raid5+ implementation in btrfs.

ZFS requires more memory to perform well, compared to other filesystems.
I believe btrfs doesn't have this requirement. I don't have any systems with this yet, but am planning on implementing btrfs on desktops and seevers with small amount of disks.

For the servers with higher disk-counts, I am planning on implementing ZFS.

--
Joost
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.