[gentoo-dev] [PATCH multibuild.eclass] multibuild_merge_root: use a more portable, simpler 'cp -PpR' call.

[gentoo-dev] [PATCH multibuild.eclass] multibuild_merge_root: use a more portable, simpler 'cp -PpR' call.

[Michał Górny]

In the original multibuild.eclass code I tried to somehow achieve very
fast merging via avoiding actually copying anything. I used the 'cp -al'
call that used hardlinks to avoid copying data but that actually made
copying non-clobbering and less portable (BSD used a tar fallback, for
example, due to some bugs at the time).

While the original solution worked and was quite good for its initial
use, the fact that it is non-clobbering is a bit confusing and resulted
in late breakage in llvm ebuild (where I assumed it will clobber). I
think it's time to replace it with something simpler, more portable (no
more userland checks) and fully clobbering.

For this reason, the patch replaces the old code with a plain 'cp -PpR'
that should be POSIX-compliant. It also tries to use '--reflink=auto' if
available to make use of btrfs CoW support.

I'd appreciate very much of someone could put the code into thorough
testing. The consumers include all distutils-r1 ebuilds, multilib
ebuilds, PyQt4 and the reverted version of llvm :).
---
 eclass/multibuild.eclass | 24 ++++++++----------------
 1 file changed, 8 insertions(+), 16 deletions(-)

diff --git a/eclass/multibuild.eclass b/eclass/multibuild.eclass
index 0a2771e..e3e5e21 100644
--- a/eclass/multibuild.eclass
+++ b/eclass/multibuild.eclass
@@ -265,24 +265,16 @@ multibuild_merge_root() {
 	done
 	rm "${lockfile_l}" || die
 
-	if use userland_BSD; then
-		# 'cp -a -n' is broken:
-		# http://www.freebsd.org/cgi/query-pr.cgi?pr=174489
-		# using tar instead which is universal but terribly slow.
-
-		tar -C "${src}" -f - -c . \
-			| tar -x -f - -C "${dest}"
-		[[ ${PIPESTATUS[*]} == '0 0' ]]
-		ret=${?}
-	elif use userland_GNU; then
-		# cp works with '-a -n'.
-
-		cp -a -l -n "${src}"/. "${dest}"/
-		ret=${?}
-	else
-		die "Unsupported userland (${USERLAND}), please report."
+	local cp_args=()
+
+	if cp --reflink=auto --version &>/dev/null; then
+		# enable reflinking if possible to make this faster
+		cp_args+=( --reflink=auto )
 	fi
 
+	cp -P -R -p "${cp_args[@]}" "${src}"/. "${dest}"/
+	ret=${?}
+
 	# Remove the lock.
 	rm "${lockfile}" || die
 
-- 
1.9.1

Re: [gentoo-dev] [PATCH multibuild.eclass] multibuild_merge_root: use a more portable, simpler 'cp -PpR' call.

[Robin H. Johnson]

On Fri, Apr 04, 2014 at 12:30:31AM +0200, Michał Górny wrote:
> +	if cp --reflink=auto --version &>/dev/null; then
> +		# enable reflinking if possible to make this faster
> +		cp_args+=( --reflink=auto )
>  	fi
>  
> +	cp -P -R -p "${cp_args[@]}" "${src}"/. "${dest}"/
> +	ret=${?}
-p in GNU cp is --preserve=mode,ownership,timestamps which will cause
the loss of selinux contexts, xattr and hardlinks.

I know you want to be POSIX compatible, but I think passing
--preserve=all or -a is important in this case.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85

[gentoo-dev] [PATCH multibuild.eclass] multibuild_merge_root: use a more portable, simpler 'cp -a' call.

[Michał Górny]

In the original multibuild.eclass code I tried to somehow achieve very
fast merging via avoiding actually copying anything. I used the 'cp -al'
call that used hardlinks to avoid copying data but that actually made
copying non-clobbering and less portable (BSD used a tar fallback, for
example, due to some bugs at the time).

While the original solution worked and was quite good for its initial
use, the fact that it is non-clobbering is a bit confusing and resulted
in late breakage in llvm ebuild (where I assumed it will clobber). I
think it's time to replace it with something simpler, more portable (no
more userland checks) and fully clobbering.

For this reason, the patch replaces the old code with a plain 'cp -a'
or 'cp -PpR' that should be POSIX-compliant. It also tries to use
'--reflink=auto' if available to make use of btrfs CoW support.

I'd appreciate very much of someone could put the code into thorough
testing. The consumers include all distutils-r1 ebuilds, multilib
ebuilds, PyQt4 and the reverted version of llvm :).
---
 eclass/multibuild.eclass | 28 +++++++++++++---------------
 1 file changed, 13 insertions(+), 15 deletions(-)

diff --git a/eclass/multibuild.eclass b/eclass/multibuild.eclass
index 0a2771e..be3ecf3 100644
--- a/eclass/multibuild.eclass
+++ b/eclass/multibuild.eclass
@@ -265,24 +265,22 @@ multibuild_merge_root() {
 	done
 	rm "${lockfile_l}" || die
 
-	if use userland_BSD; then
-		# 'cp -a -n' is broken:
-		# http://www.freebsd.org/cgi/query-pr.cgi?pr=174489
-		# using tar instead which is universal but terribly slow.
-
-		tar -C "${src}" -f - -c . \
-			| tar -x -f - -C "${dest}"
-		[[ ${PIPESTATUS[*]} == '0 0' ]]
-		ret=${?}
-	elif use userland_GNU; then
-		# cp works with '-a -n'.
-
-		cp -a -l -n "${src}"/. "${dest}"/
-		ret=${?}
+	local cp_args=()
+
+	if cp -a --version &>/dev/null; then
+		cp_args+=( -a )
 	else
-		die "Unsupported userland (${USERLAND}), please report."
+		cp_args+=( -P -R -p )
+	fi
+
+	if cp --reflink=auto --version &>/dev/null; then
+		# enable reflinking if possible to make this faster
+		cp_args+=( --reflink=auto )
 	fi
 
+	cp "${cp_args[@]}" "${src}"/. "${dest}"/
+	ret=${?}
+
 	# Remove the lock.
 	rm "${lockfile}" || die
 
-- 
1.9.1

Re: [gentoo-dev] [PATCH multibuild.eclass] multibuild_merge_root: use a more portable, simpler 'cp -a' call.

[Rich Freeman]

On Fri, Apr 4, 2014 at 1:50 AM, Michał Górny <mgorny@gentoo.org> wrote:
> It also tries to use '--reflink=auto' if available to make use of
> btrfs CoW support.

++

Rich