[gentoo-dev] GSoC proposal: cp --reflink support for zfs.

[gentoo-dev] GSoC proposal: cp --reflink support for zfs.

[Yuxuan Shui]

[-- Attachment #1: Type: text/plain, Size: 1005 bytes --]

Hi,

I would like to implement cp --reflink support for ZFSOnLinux as my GSoC
project.

cp --reflink is used to create a COW copy of a file, so the file will not
take any disk space if it's not modified. This feature is very useful for
cases like storing a lot of almost identical virtual machine images. Also
this is a frequently requested feature for ZoL. [1][2][3]

Currently only btrfs support this feature, so my goal it to bring it to ZoL
as well.

I think the only way to do it (without changing too many parts of ZoL) is
to use the deduplication feature of zfs. A COW copy could be done by create
a new entry in ddt for the old file, and create a new file which points to
the ddt entry.

Please let me know if this proposal makes sense, and if that's the right
way to do it.

Thanks.

[1]:
https://groups.google.com/a/zfsonlinux.org/forum/#!topic/zfs-discuss/mvGB7QEpt3w
[2]: https://github.com/zfsonlinux/zfs/issues/405
[3]: https://github.com/zfsonlinux/zfs/issues/1063
-- 

Regards
Yuxuan Shui

[-- Attachment #2: Type: text/html, Size: 1466 bytes --]

Re: [gentoo-dev] GSoC proposal: cp --reflink support for zfs.

[Alex Xu]

[-- Attachment #1: Type: text/plain, Size: 1764 bytes --]

On 12/03/14 03:15 AM, Yuxuan Shui wrote:
> Hi,
> 
> I would like to implement cp --reflink support for ZFSOnLinux as my GSoC
> project.
> 
> cp --reflink is used to create a COW copy of a file, so the file will not
> take any disk space if it's not modified. This feature is very useful for
> cases like storing a lot of almost identical virtual machine images. Also
> this is a frequently requested feature for ZoL. [1][2][3]
> 
> Currently only btrfs support this feature, so my goal it to bring it to ZoL
> as well.
> 
> I think the only way to do it (without changing too many parts of ZoL) is
> to use the deduplication feature of zfs. A COW copy could be done by create
> a new entry in ddt for the old file, and create a new file which points to
> the ddt entry.
> 
> Please let me know if this proposal makes sense, and if that's the right
> way to do it.
> 
> Thanks.
> 
> [1]:
> https://groups.google.com/a/zfsonlinux.org/forum/#!topic/zfs-discuss/mvGB7QEpt3w
> [2]: https://github.com/zfsonlinux/zfs/issues/405
> [3]: https://github.com/zfsonlinux/zfs/issues/1063
> 

While I can't comment too much on the technical aspects, they seem to be
relatively sound.

However, there are some issues with the, er... other aspects, for lack
of better terminology.

1. This is possibly out of scope as a Gentoo project, since ZOL is not
really part of Gentoo. If it's not, then you're out of luck, because ZOL
is not an accepted organization.

2. This is likely too small to be a GSoC project. Perhaps see [0] for a
list of example ideas, if only so you can get a grasp on the size of a
good project.

It does sound like a good idea though, and even if you can't do it as
part of GSoC, you should pursue it anyways.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] GSoC proposal: cp --reflink support for zfs.

[Richard Yao]

[-- Attachment #1: Type: text/plain, Size: 2295 bytes --]

On 03/12/2014 08:45 AM, Alex Xu wrote:
> On 12/03/14 03:15 AM, Yuxuan Shui wrote:
>> Hi,
>>
>> I would like to implement cp --reflink support for ZFSOnLinux as my GSoC
>> project.
>>
>> cp --reflink is used to create a COW copy of a file, so the file will not
>> take any disk space if it's not modified. This feature is very useful for
>> cases like storing a lot of almost identical virtual machine images. Also
>> this is a frequently requested feature for ZoL. [1][2][3]
>>
>> Currently only btrfs support this feature, so my goal it to bring it to ZoL
>> as well.
>>
>> I think the only way to do it (without changing too many parts of ZoL) is
>> to use the deduplication feature of zfs. A COW copy could be done by create
>> a new entry in ddt for the old file, and create a new file which points to
>> the ddt entry.
>>
>> Please let me know if this proposal makes sense, and if that's the right
>> way to do it.
>>
>> Thanks.
>>
>> [1]:
>> https://groups.google.com/a/zfsonlinux.org/forum/#!topic/zfs-discuss/mvGB7QEpt3w
>> [2]: https://github.com/zfsonlinux/zfs/issues/405
>> [3]: https://github.com/zfsonlinux/zfs/issues/1063
>>
> 
> While I can't comment too much on the technical aspects, they seem to be
> relatively sound.
> 
> However, there are some issues with the, er... other aspects, for lack
> of better terminology.
> 
> 1. This is possibly out of scope as a Gentoo project, since ZOL is not
> really part of Gentoo. If it's not, then you're out of luck, because ZOL
> is not an accepted organization.

Things that provide us with improvements over what we have are
definitely worth consideration as GSoC projects. However, what is
accepted ultimately depends on not only feedback from a potential
mentor, but also a vote of Gentoo developers.

> 2. This is likely too small to be a GSoC project. Perhaps see [0] for a
> list of example ideas, if only so you can get a grasp on the size of a
> good project.
> 
> It does sound like a good idea though, and even if you can't do it as
> part of GSoC, you should pursue it anyways.

Leaning on my understanding of ZFS internals, I can say that this is
large enough. However, I do not think it will accomplish the desired
result if implemented in the manner suggested.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 901 bytes --]

Re: [gentoo-dev] GSoC proposal: cp --reflink support for zfs.

[Yuxuan Shui]

Hi,

On Wed, Mar 12, 2014 at 9:30 PM, Richard Yao <ryao@gentoo.org> wrote:
>
> On 03/12/2014 08:45 AM, Alex Xu wrote:
> > On 12/03/14 03:15 AM, Yuxuan Shui wrote:
> >> Hi,
> >>
> >> I would like to implement cp --reflink support for ZFSOnLinux as my GSoC
> >> project.
> >>
> >> cp --reflink is used to create a COW copy of a file, so the file will not
> >> take any disk space if it's not modified. This feature is very useful for
> >> cases like storing a lot of almost identical virtual machine images. Also
> >> this is a frequently requested feature for ZoL. [1][2][3]
> >>
> >> Currently only btrfs support this feature, so my goal it to bring it to ZoL
> >> as well.
> >>
> >> I think the only way to do it (without changing too many parts of ZoL) is
> >> to use the deduplication feature of zfs. A COW copy could be done by create
> >> a new entry in ddt for the old file, and create a new file which points to
> >> the ddt entry.
> >>
> >> Please let me know if this proposal makes sense, and if that's the right
> >> way to do it.
> >>
> >> Thanks.
> >>
> >> [1]:
> >> https://groups.google.com/a/zfsonlinux.org/forum/#!topic/zfs-discuss/mvGB7QEpt3w
> >> [2]: https://github.com/zfsonlinux/zfs/issues/405
> >> [3]: https://github.com/zfsonlinux/zfs/issues/1063
> >>
> >
> > While I can't comment too much on the technical aspects, they seem to be
> > relatively sound.
> >
> > However, there are some issues with the, er... other aspects, for lack
> > of better terminology.
> >
> > 1. This is possibly out of scope as a Gentoo project, since ZOL is not
> > really part of Gentoo. If it's not, then you're out of luck, because ZOL
> > is not an accepted organization.
>
> Things that provide us with improvements over what we have are
> definitely worth consideration as GSoC projects. However, what is
> accepted ultimately depends on not only feedback from a potential
> mentor, but also a vote of Gentoo developers.

Maybe I could propose this project to illumos? Since they seems to
accept proposal for OpenZFS.

>
> > 2. This is likely too small to be a GSoC project. Perhaps see [0] for a
> > list of example ideas, if only so you can get a grasp on the size of a
> > good project.
> >
> > It does sound like a good idea though, and even if you can't do it as
> > part of GSoC, you should pursue it anyways.
>
> Leaning on my understanding of ZFS internals, I can say that this is
> large enough. However, I do not think it will accomplish the desired
> result if implemented in the manner suggested.

Could you elaborate why this won't work? Thanks.

>

-- 

Regards
Yuxuan Shui

Re: [gentoo-dev] GSoC proposal: cp --reflink support for zfs.

[Rich Freeman]

On Wed, Mar 12, 2014 at 9:30 AM, Richard Yao <ryao@gentoo.org> wrote:
> Things that provide us with improvements over what we have are
> definitely worth consideration as GSoC projects. However, what is
> accepted ultimately depends on not only feedback from a potential
> mentor, but also a vote of Gentoo developers.

Honestly, this seems like a less-than-ideal fit for Gentoo.  We don't
really use ZFS at all, other than as yet another package in our tree.
Any of the Mozilla/ GSoC ideas would make as much sense to deliver as
part of the Gentoo GSoC.

Now, if this were about integrating auto-snapshots into the package
manager, (like can be done with snapper), etc, then I could see more
relevance (though it probably wouldn't rise to a full project).  I
could also see some kind of project to integrate advanced filesystem
features into core elements of our distro across many filesystems
(though I don't really see the relevance for reflinks in particular,
and they only apply to COW filesystems anyway).

This just seems like a ZFS project, and not like a Gentoo project.
I'd say the same if this were about adding a mute button to tabs in
Chromium, or fixing the offline btrfsck, or whatever.  All of those
things would be useful to Gentoo, but only insofar as they'd be useful
to anybody.

Rich

Re: [gentoo-dev] GSoC proposal: cp --reflink support for zfs.

[Paul Tagliamonte]

[-- Attachment #1: Type: text/plain, Size: 838 bytes --]

On Wed, Mar 12, 2014 at 01:19:08PM -0400, Rich Freeman wrote:
> This just seems like a ZFS project, and not like a Gentoo project.
> I'd say the same if this were about adding a mute button to tabs in
> Chromium, or fixing the offline btrfsck, or whatever.  All of those
> things would be useful to Gentoo, but only insofar as they'd be useful
> to anybody.

Agreed. Your dear friends in Debian would enjoy such changes as well, for
those using zfs with kfreebsd or a home build of the zol package. Perhaps
this would be a fit for upstream?

> Rich

Cheers,
  Paul

-- 
 .''`.  Paul Tagliamonte <paultag@debian.org>  |   Proud Debian Developer
: :'  : 4096R / 8F04 9AD8 2C92 066C 7352  D28A 7B58 5B30 807C 2A87
`. `'`  http://people.debian.org/~paultag
 `-     http://people.debian.org/~paultag/conduct-statement.txt

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] GSoC proposal: cp --reflink support for zfs.

[Richard Yao]

I do not think the original project idea is feasible as stated.

On Mar 12, 2014, at 1:22 PM, Paul Tagliamonte <paultag@debian.org> wrote:

> On Wed, Mar 12, 2014 at 01:19:08PM -0400, Rich Freeman wrote:
>> This just seems like a ZFS project, and not like a Gentoo project.
>> I'd say the same if this were about adding a mute button to tabs in
>> Chromium, or fixing the offline btrfsck, or whatever.  All of those
>> things would be useful to Gentoo, but only insofar as they'd be useful
>> to anybody.
> 
> Agreed. Your dear friends in Debian would enjoy such changes as well, for
> those using zfs with kfreebsd or a home build of the zol package. Perhaps
> this would be a fit for upstream?
> 
>> Rich
> 
> Cheers,
>  Paul
> 
> -- 
> .''`.  Paul Tagliamonte <paultag@debian.org>  |   Proud Debian Developer
> : :'  : 4096R / 8F04 9AD8 2C92 066C 7352  D28A 7B58 5B30 807C 2A87
> `. `'`  http://people.debian.org/~paultag
> `-     http://people.debian.org/~paultag/conduct-statement.txt

Re: [gentoo-dev] GSoC proposal: cp --reflink support for zfs.

[Richard Yao]

It is a moot point because I do not think this project idea is feasible.

On Mar 12, 2014, at 1:19 PM, Rich Freeman <rich0@gentoo.org> wrote:

> On Wed, Mar 12, 2014 at 9:30 AM, Richard Yao <ryao@gentoo.org> wrote:
>> Things that provide us with improvements over what we have are
>> definitely worth consideration as GSoC projects. However, what is
>> accepted ultimately depends on not only feedback from a potential
>> mentor, but also a vote of Gentoo developers.
> 
> Honestly, this seems like a less-than-ideal fit for Gentoo.  We don't
> really use ZFS at all, other than as yet another package in our tree.
> Any of the Mozilla/ GSoC ideas would make as much sense to deliver as
> part of the Gentoo GSoC.
> 
> Now, if this were about integrating auto-snapshots into the package
> manager, (like can be done with snapper), etc, then I could see more
> relevance (though it probably wouldn't rise to a full project).  I
> could also see some kind of project to integrate advanced filesystem
> features into core elements of our distro across many filesystems
> (though I don't really see the relevance for reflinks in particular,
> and they only apply to COW filesystems anyway).
> 
> This just seems like a ZFS project, and not like a Gentoo project.
> I'd say the same if this were about adding a mute button to tabs in
> Chromium, or fixing the offline btrfsck, or whatever.  All of those
> things would be useful to Gentoo, but only insofar as they'd be useful
> to anybody.
> 
> Rich
> 

[gentoo-dev] Re: GSoC proposal: cp --reflink support for zfs.

[Richard Yao]

[-- Attachment #1: Type: text/plain, Size: 2503 bytes --]

A key feature of reflinks is that they operate on any data in a mountpoint, but what you described only applies to data with a deduplication table entry. In such cases, it do not see what it accomplishes over simply using data deduplication. In specific, there is no efficiency advantage. It is not clear to me that trying to cut corners to obtain one is possible without incurring consistency issues with the deduplication table falling out of sync. In the case that there is no deduplication table entry, you would need to rewrite the file. reflinks are intended to be as done as quickly as hard links, so this would seem to negate the benefit.

Matthew Ahrens and I have discussed reflinks in the past and we both agree that they would be non-trivial to implement. That does not mean it cannot be done, but I do not think this particular approach would succeed. However, I encourage you to keep thinking about such things. If you think of a way of doing this that seems workable, it would likely be something that could be made into a GSoC project.

With that said, if you want to do a ZoL-related project for Gentoo, I have some other things that I could suggest that I believe are workable. Such ideas are things that I was asked to prepare on extremely short notice and I have not yet had time to publish them. Let me know if you would be interested.

On Mar 12, 2014, at 3:15 AM, Yuxuan Shui <yshuiv7@gmail.com> wrote:

> Hi,
> 
> I would like to implement cp --reflink support for ZFSOnLinux as my GSoC project.
> 
> cp --reflink is used to create a COW copy of a file, so the file will not take any disk space if it's not modified. This feature is very useful for cases like storing a lot of almost identical virtual machine images. Also this is a frequently requested feature for ZoL. [1][2][3]
> 
> Currently only btrfs support this feature, so my goal it to bring it to ZoL as well.
> 
> I think the only way to do it (without changing too many parts of ZoL) is to use the deduplication feature of zfs. A COW copy could be done by create a new entry in ddt for the old file, and create a new file which points to the ddt entry.
> 
> Please let me know if this proposal makes sense, and if that's the right way to do it.
> 
> Thanks.
> 
> [1]: https://groups.google.com/a/zfsonlinux.org/forum/#!topic/zfs-discuss/mvGB7QEpt3w
> [2]: https://github.com/zfsonlinux/zfs/issues/405
> [3]: https://github.com/zfsonlinux/zfs/issues/1063
> -- 
> 
> Regards
> Yuxuan Shui

[-- Attachment #2: Type: text/html, Size: 4361 bytes --]