public inbox for gentoo-dev@lists.gentoo.org
 help / color / mirror / Atom feed
* [gentoo-dev] [PATCHES] Header wrapping support for multilib
@ 2013-03-23 16:25 Michał Górny
  2013-03-23 16:26 ` [gentoo-dev] [PATCH 1/2] Introduce multibuild_merge_root() to merge interim installs Michał Górny
                   ` (3 more replies)
  0 siblings, 4 replies; 30+ messages in thread
From: Michał Górny @ 2013-03-23 16:25 UTC (permalink / raw
  To: Gentoo Developer Mailing List

[-- Attachment #1: Type: text/plain, Size: 1293 bytes --]

Hello,

I've finally got around to writing the header wrapping functions for
multilib. That's an initial yet working draft. I will send patches in
reply to this mail.

The first patch just converts internal distutils-r1 root merging
function to globally available multibuild_merge_root(). This is used
in distutils-r1 to merge fakerooted distutils installs, and will be
used to merge wrapped headers on top of regular install.

The second one adds the actual code. The headers to wrap are listed
in a global variable. After each install, the headers are moved to
temporary location and wrapper is being prepared there. When all
installs are done, all those files are merged on top of them.

The per-ABI headers are installed onto /usr/include/$CHOST, repeating
the directory structure from original /usr/include install. Wrapper
includes the headers by full <$CHOST/foo/bar.h> path.

Potential issues:

- only /usr/include is supported (to be extended if necessary),

- no 'fallback' code in header (to be added in the future), if no ABIs
  match, no code is given,

- not tested thoroughly. I'd appreciate if someone could play with it
  a bit and see whether proper headers are actually included and it
  works everywhere.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

^ permalink raw reply	[flat|nested] 30+ messages in thread

* [gentoo-dev] [PATCH 1/2] Introduce multibuild_merge_root() to merge interim installs.
  2013-03-23 16:25 [gentoo-dev] [PATCHES] Header wrapping support for multilib Michał Górny
@ 2013-03-23 16:26 ` Michał Górny
  2013-03-23 17:00   ` Ulrich Mueller
  2013-03-23 17:44   ` [gentoo-dev] [PATCH 1/2] " Alec Warner
  2013-03-23 16:26 ` [gentoo-dev] [PATCH 2/2] Support wrapping headers for multilib ABIs Michał Górny
                   ` (2 subsequent siblings)
  3 siblings, 2 replies; 30+ messages in thread
From: Michał Górny @ 2013-03-23 16:26 UTC (permalink / raw
  To: gentoo-dev; +Cc: Michał Górny

This is mostly a copy from distutils-r1. The function does copy all the
files from one location onto another, preserving whatever possible. It
can be run in parallel too without the risk of race conditions.
---
 gx86/eclass/distutils-r1.eclass | 41 +------------------------------------
 gx86/eclass/multibuild.eclass   | 45 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 46 insertions(+), 40 deletions(-)

diff --git a/gx86/eclass/distutils-r1.eclass b/gx86/eclass/distutils-r1.eclass
index 0982e6c..3c21741 100644
--- a/gx86/eclass/distutils-r1.eclass
+++ b/gx86/eclass/distutils-r1.eclass
@@ -449,49 +449,10 @@ distutils-r1_python_install() {
 
 	if [[ ! ${DISTUTILS_SINGLE_IMPL} ]]; then
 		_distutils-r1_rename_scripts "${root}"
-		_distutils-r1_merge_root "${root}" "${D}"
+		multibuild_merge_root "${root}" "${D}"
 	fi
 }
 
-# @FUNCTION: distutils-r1_merge_root
-# @USAGE: <src-root> <dest-root>
-# @INTERNAL
-# @DESCRIPTION:
-# Merge the directory tree from <src-root> to <dest-root>, removing
-# the <src-root> in the process.
-_distutils-r1_merge_root() {
-	local src=${1}
-	local dest=${2}
-
-	local lockfile=${T}/distutils-r1-merge-lock
-
-	if type -P lockf &>/dev/null; then
-		# On BSD, we have 'lockf' wrapper.
-		tar -C "${src}" -f - -c . \
-			| lockf "${lockfile}" tar -x -f - -C "${dest}"
-	else
-		local lock_fd
-		if type -P flock &>/dev/null; then
-			# On Linux, we have 'flock' which can lock fd.
-			redirect_alloc_fd lock_fd "${lockfile}" '>>'
-			flock ${lock_fd}
-		else
-			ewarn "distutils-r1: no locking service found, please report."
-		fi
-
-		cp -a -l -n "${src}"/. "${dest}"/
-
-		if [[ ${lock_fd} ]]; then
-			# Close the lock file when we are done with it.
-			# Prevents deadlock if we aren't in a subshell.
-			eval "exec ${lock_fd}>&-"
-		fi
-	fi
-	[[ ${?} == 0 ]] || die "Merging ${EPYTHON} image failed."
-
-	rm -rf "${src}"
-}
-
 # @FUNCTION: distutils-r1_python_install_all
 # @DESCRIPTION:
 # The default python_install_all(). It installs the documentation.
diff --git a/gx86/eclass/multibuild.eclass b/gx86/eclass/multibuild.eclass
index a3f1402..ed837d3 100644
--- a/gx86/eclass/multibuild.eclass
+++ b/gx86/eclass/multibuild.eclass
@@ -238,5 +238,50 @@ run_in_build_dir() {
 	return ${ret}
 }
 
+# @FUNCTION: multibuild_merge_root
+# @USAGE: <src-root> <dest-root>
+# @DESCRIPTION:
+# Merge the directory tree (fake root) from <src-root> to <dest-root>
+# (the real root). Both directories have to be real, absolute paths
+# (i.e. including ${D}). Source root will be removed.
+#
+# This functions uses locking to support merging during parallel
+# installs.
+multibuild_merge_root() {
+	local src=${1}
+	local dest=${2}
+
+	local lockfile=${T}/multibuild_merge_lock
+
+	if type -P lockf &>/dev/null; then
+		# On BSD, we have 'lockf' wrapper.
+		tar -C "${src}" -f - -c . \
+			| lockf "${lockfile}" tar -x -f - -C "${dest}"
+	else
+		local lock_fd
+		if type -P flock &>/dev/null; then
+			# On Linux, we have 'flock' which can lock fd.
+			redirect_alloc_fd lock_fd "${lockfile}" '>>'
+			flock ${lock_fd}
+		else
+			ewarn "multibuild: no locking service found, please report."
+		fi
+
+		cp -a -l -n "${src}"/. "${dest}"/
+
+		if [[ ${lock_fd} ]]; then
+			# Close the lock file when we are done with it.
+			# Prevents deadlock if we aren't in a subshell.
+			eval "exec ${lock_fd}>&-"
+		fi
+	fi
+
+	if [[ ${?} -ne 0 ]]; then
+		die "${MULTIBUILD_VARIANT:-(unknown)}: merging image failed."
+	fi
+
+	rm -rf "${src}"
+}
+
 _MULTIBUILD=1
 fi
-- 
1.8.1.5



^ permalink raw reply related	[flat|nested] 30+ messages in thread

* [gentoo-dev] [PATCH 2/2] Support wrapping headers for multilib ABIs.
  2013-03-23 16:25 [gentoo-dev] [PATCHES] Header wrapping support for multilib Michał Górny
  2013-03-23 16:26 ` [gentoo-dev] [PATCH 1/2] Introduce multibuild_merge_root() to merge interim installs Michał Górny
@ 2013-03-23 16:26 ` Michał Górny
  2013-03-24 15:14   ` Alexis Ballier
  2013-03-23 16:46 ` [gentoo-dev] [PATCHES] Header wrapping support for multilib Diego Elio Pettenò
  2013-04-01  9:19 ` [gentoo-dev] [PATCHES] Header wrapping support for multilib Michał Górny
  3 siblings, 1 reply; 30+ messages in thread
From: Michał Górny @ 2013-03-23 16:26 UTC (permalink / raw
  To: gentoo-dev; +Cc: Michał Górny

---
 gx86/eclass/autotools-multilib.eclass | 82 +++++++++++++++++++++++++++++++++++
 1 file changed, 82 insertions(+)

diff --git a/gx86/eclass/autotools-multilib.eclass b/gx86/eclass/autotools-multilib.eclass
index d7372b0..c65c777 100644
--- a/gx86/eclass/autotools-multilib.eclass
+++ b/gx86/eclass/autotools-multilib.eclass
@@ -33,6 +33,28 @@ inherit autotools-utils multilib-build
 
 EXPORT_FUNCTIONS src_prepare src_configure src_compile src_test src_install
 
+# @ECLASS-VARIABLE: MULTILIB_WRAPPED_HEADERS
+# @DESCRIPTION:
+# A list of headers to wrap for multilib support. The listed headers
+# will be moved to a non-standard location and replace with a file
+# including them conditionally to current ABI.
+#
+# This variable has to be a bash array. Paths shall be relative to
+# installation root (${D}), and name regular files. Recursive wrapping
+# is not supported.
+#
+# Please note that header wrapping is *discouraged*. It is preferred to
+# install all headers in a subdirectory of libdir and use pkg-config to
+# locate the headers. Some C preprocessors will not work with wrapped
+# headers.
+#
+# Example:
+# @CODE
+# MULTILIB_WRAPPED_HEADERS=(
+#	/usr/include/foobar/config.h
+# )
+# @CODE
+
 autotools-multilib_src_prepare() {
 	autotools-utils_src_prepare "${@}"
 }
@@ -49,13 +71,73 @@ autotools-multilib_src_test() {
 	multilib_foreach_abi autotools-utils_src_test "${@}"
 }
 
+_autotools-multilib_wrap_headers() {
+	debug-print-function ${FUNCNAME} "$@"
+	local f
+
+	for f in "${MULTILIB_WRAPPED_HEADERS[@]}"; do
+		# drop leading slash if it's there
+		f=${f#/}
+
+		if [[ ${f} != usr/include/* ]]; then
+			die "Wrapping headers outside of /usr/include is not supported at the moment."
+		fi
+		# and then usr/include
+		f=${f#usr/include/}
+
+		local dir=${f%/*}
+
+		# $CHOST shall be set by multilib_toolchain_setup
+		dodir "/tmp/multilib-include/${CHOST}/${dir}"
+		mv "${ED}/usr/include/${f}" "${ED}/tmp/multilib-include/${CHOST}/${dir}/" || die
+
+		if [[ ! -f ${ED}/tmp/multilib-include/${f} ]]; then
+			dodir "/tmp/multilib-include/${dir}"
+			cat > "${ED}/tmp/multilib-include/${f}" <<_EOF_ || die
+/* This file is auto-generated by autotools-multilib.eclass
+ * as a multilib-friendly wrapper to /${f}. For the original content,
+ * please see the files that are #included below.
+ */
+_EOF_
+		fi
+
+		local defs
+		case "${ABI}" in
+			amd64)
+				defs='defined(__x86_64__) && !defined(__ILP32__)';;
+			x86)
+				defs='defined(__i386__)';;
+			x32)
+				defs='defined(__x86_64__) && defined(__ILP32__)';;
+			*)
+				die "Header wrapping for ${ABI} not supported yet";;
+		esac
+
+		cat >> "${ED}/tmp/multilib-include/${f}" <<_EOF_ || die
+
+#if ${defs}
+#	include <${CHOST}/${f}>
+#endif
+_EOF_
+	done
+}
+
 autotools-multilib_src_install() {
 	autotools-multilib_secure_install() {
 		autotools-utils_src_install "${@}"
 
+		_autotools-multilib_wrap_headers
 		# Make sure all headers are the same for each ABI.
 		multilib_check_headers
 	}
 
 	multilib_foreach_abi autotools-multilib_secure_install "${@}"
+
+	# merge the wrapped headers
+	if [[ -d "${ED}"/tmp/multilib-include ]]; then
+		multibuild_merge_root \
+			"${ED}"/tmp/multilib-include "${ED}"/usr/include
+		# it can fail if something else uses /tmp
+		rmdir "${ED}"/tmp &>/dev/null
+	fi
 }
-- 
1.8.1.5



^ permalink raw reply related	[flat|nested] 30+ messages in thread

* Re: [gentoo-dev] [PATCHES] Header wrapping support for multilib
  2013-03-23 16:25 [gentoo-dev] [PATCHES] Header wrapping support for multilib Michał Górny
  2013-03-23 16:26 ` [gentoo-dev] [PATCH 1/2] Introduce multibuild_merge_root() to merge interim installs Michał Górny
  2013-03-23 16:26 ` [gentoo-dev] [PATCH 2/2] Support wrapping headers for multilib ABIs Michał Górny
@ 2013-03-23 16:46 ` Diego Elio Pettenò
  2013-03-23 17:32   ` Michał Górny
  2013-03-23 19:56   ` [gentoo-dev] [PATCH] Support wrapping headers for multilib ABIs Michał Górny
  2013-04-01  9:19 ` [gentoo-dev] [PATCHES] Header wrapping support for multilib Michał Górny
  3 siblings, 2 replies; 30+ messages in thread
From: Diego Elio Pettenò @ 2013-03-23 16:46 UTC (permalink / raw
  To: gentoo-dev

[-- Attachment #1: Type: text/plain, Size: 361 bytes --]

On 23/03/2013 17:25, Michał Górny wrote:
> - no 'fallback' code in header (to be added in the future), if no ABIs
>   match, no code is given,

Please don't commit this without a fallback that at the very least
#errors out — I don't want silent issues.

-- 
Diego Elio Pettenò — Flameeyes
flameeyes@flameeyes.eu — http://blog.flameeyes.eu/


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 555 bytes --]

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [gentoo-dev] [PATCH 1/2] Introduce multibuild_merge_root() to merge interim installs.
  2013-03-23 16:26 ` [gentoo-dev] [PATCH 1/2] Introduce multibuild_merge_root() to merge interim installs Michał Górny
@ 2013-03-23 17:00   ` Ulrich Mueller
  2013-03-23 17:28     ` Michał Górny
  2013-03-25 22:42     ` [gentoo-dev] [PATCH] " Michał Górny
  2013-03-23 17:44   ` [gentoo-dev] [PATCH 1/2] " Alec Warner
  1 sibling, 2 replies; 30+ messages in thread
From: Ulrich Mueller @ 2013-03-23 17:00 UTC (permalink / raw
  To: gentoo-dev; +Cc: Michał Górny

>>>>> On Sat, 23 Mar 2013, Michał Górny wrote:

> +multibuild_merge_root() {
> + [...]
> +	if type -P lockf &>/dev/null; then
> +		# On BSD, we have 'lockf' wrapper.
> +		tar -C "${src}" -f - -c . \
> +			| lockf "${lockfile}" tar -x -f - -C "${dest}"
> +	else
> + [...]
> +		cp -a -l -n "${src}"/. "${dest}"/

You assume that cp is GNU, if there's no lockf. Is it guaranteed that
this is true on all platforms?

Ulrich


^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [gentoo-dev] [PATCH 1/2] Introduce multibuild_merge_root() to merge interim installs.
  2013-03-23 17:00   ` Ulrich Mueller
@ 2013-03-23 17:28     ` Michał Górny
  2013-03-24  7:47       ` Ulrich Mueller
  2013-03-25 22:42     ` [gentoo-dev] [PATCH] " Michał Górny
  1 sibling, 1 reply; 30+ messages in thread
From: Michał Górny @ 2013-03-23 17:28 UTC (permalink / raw
  To: gentoo-dev; +Cc: ulm

[-- Attachment #1: Type: text/plain, Size: 657 bytes --]

On Sat, 23 Mar 2013 18:00:41 +0100
Ulrich Mueller <ulm@gentoo.org> wrote:

> >>>>> On Sat, 23 Mar 2013, Michał Górny wrote:
> 
> > +multibuild_merge_root() {
> > + [...]
> > +	if type -P lockf &>/dev/null; then
> > +		# On BSD, we have 'lockf' wrapper.
> > +		tar -C "${src}" -f - -c . \
> > +			| lockf "${lockfile}" tar -x -f - -C "${dest}"
> > +	else
> > + [...]
> > +		cp -a -l -n "${src}"/. "${dest}"/
> 
> You assume that cp is GNU, if there's no lockf. Is it guaranteed that
> this is true on all platforms?

They're all just assumptions. If it breaks somewhere, I'll improve
the function.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [gentoo-dev] [PATCHES] Header wrapping support for multilib
  2013-03-23 16:46 ` [gentoo-dev] [PATCHES] Header wrapping support for multilib Diego Elio Pettenò
@ 2013-03-23 17:32   ` Michał Górny
  2013-03-23 17:32     ` Diego Elio Pettenò
  2013-03-23 19:56   ` [gentoo-dev] [PATCH] Support wrapping headers for multilib ABIs Michał Górny
  1 sibling, 1 reply; 30+ messages in thread
From: Michał Górny @ 2013-03-23 17:32 UTC (permalink / raw
  To: gentoo-dev; +Cc: flameeyes

[-- Attachment #1: Type: text/plain, Size: 517 bytes --]

On Sat, 23 Mar 2013 17:46:18 +0100
Diego Elio Pettenò <flameeyes@flameeyes.eu> wrote:

> On 23/03/2013 17:25, Michał Górny wrote:
> > - no 'fallback' code in header (to be added in the future), if no ABIs
> >   match, no code is given,
> 
> Please don't commit this without a fallback that at the very least
> #errors out — I don't want silent issues.

While at it, is '#elif' portable enough here? I see it in C99 but you
probably know more about it than I do.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [gentoo-dev] [PATCHES] Header wrapping support for multilib
  2013-03-23 17:32   ` Michał Górny
@ 2013-03-23 17:32     ` Diego Elio Pettenò
  0 siblings, 0 replies; 30+ messages in thread
From: Diego Elio Pettenò @ 2013-03-23 17:32 UTC (permalink / raw
  To: Michał Górny; +Cc: gentoo-dev

[-- Attachment #1: Type: text/plain, Size: 405 bytes --]

On 23/03/2013 18:32, Michał Górny wrote:
> While at it, is '#elif' portable enough here? I see it in C99 but you
> probably know more about it than I do.

I'd say it's portable enough, I'm pretty sure it exists before C99 as
well — and worst case scenario it'll explode in our face, not silently.

-- 
Diego Elio Pettenò — Flameeyes
flameeyes@flameeyes.eu — http://blog.flameeyes.eu/


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 555 bytes --]

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [gentoo-dev] [PATCH 1/2] Introduce multibuild_merge_root() to merge interim installs.
  2013-03-23 16:26 ` [gentoo-dev] [PATCH 1/2] Introduce multibuild_merge_root() to merge interim installs Michał Górny
  2013-03-23 17:00   ` Ulrich Mueller
@ 2013-03-23 17:44   ` Alec Warner
  2013-03-23 18:57     ` Michał Górny
  2013-03-23 18:57     ` [gentoo-dev] " Jonathan Callen
  1 sibling, 2 replies; 30+ messages in thread
From: Alec Warner @ 2013-03-23 17:44 UTC (permalink / raw
  To: gentoo-dev; +Cc: Michał Górny

On Sat, Mar 23, 2013 at 9:26 AM, Michał Górny <mgorny@gentoo.org> wrote:
> This is mostly a copy from distutils-r1. The function does copy all the
> files from one location onto another, preserving whatever possible. It
> can be run in parallel too without the risk of race conditions.
> ---
>  gx86/eclass/distutils-r1.eclass | 41 +------------------------------------
>  gx86/eclass/multibuild.eclass   | 45 +++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 46 insertions(+), 40 deletions(-)
>
> diff --git a/gx86/eclass/distutils-r1.eclass b/gx86/eclass/distutils-r1.eclass
> index 0982e6c..3c21741 100644
> --- a/gx86/eclass/distutils-r1.eclass
> +++ b/gx86/eclass/distutils-r1.eclass
> @@ -449,49 +449,10 @@ distutils-r1_python_install() {
>
>         if [[ ! ${DISTUTILS_SINGLE_IMPL} ]]; then
>                 _distutils-r1_rename_scripts "${root}"
> -               _distutils-r1_merge_root "${root}" "${D}"
> +               multibuild_merge_root "${root}" "${D}"
>         fi
>  }
>
> -# @FUNCTION: distutils-r1_merge_root
> -# @USAGE: <src-root> <dest-root>
> -# @INTERNAL
> -# @DESCRIPTION:
> -# Merge the directory tree from <src-root> to <dest-root>, removing
> -# the <src-root> in the process.
> -_distutils-r1_merge_root() {
> -       local src=${1}
> -       local dest=${2}
> -
> -       local lockfile=${T}/distutils-r1-merge-lock
> -
> -       if type -P lockf &>/dev/null; then
> -               # On BSD, we have 'lockf' wrapper.
> -               tar -C "${src}" -f - -c . \
> -                       | lockf "${lockfile}" tar -x -f - -C "${dest}"
> -       else
> -               local lock_fd
> -               if type -P flock &>/dev/null; then
> -                       # On Linux, we have 'flock' which can lock fd.
> -                       redirect_alloc_fd lock_fd "${lockfile}" '>>'
> -                       flock ${lock_fd}
> -               else
> -                       ewarn "distutils-r1: no locking service found, please report."
> -               fi
> -
> -               cp -a -l -n "${src}"/. "${dest}"/
> -
> -               if [[ ${lock_fd} ]]; then
> -                       # Close the lock file when we are done with it.
> -                       # Prevents deadlock if we aren't in a subshell.
> -                       eval "exec ${lock_fd}>&-"
> -               fi
> -       fi
> -       [[ ${?} == 0 ]] || die "Merging ${EPYTHON} image failed."
> -
> -       rm -rf "${src}"
> -}
> -
>  # @FUNCTION: distutils-r1_python_install_all
>  # @DESCRIPTION:
>  # The default python_install_all(). It installs the documentation.
> diff --git a/gx86/eclass/multibuild.eclass b/gx86/eclass/multibuild.eclass
> index a3f1402..ed837d3 100644
> --- a/gx86/eclass/multibuild.eclass
> +++ b/gx86/eclass/multibuild.eclass
> @@ -238,5 +238,50 @@ run_in_build_dir() {
>         return ${ret}
>  }
>
> +# @FUNCTION: multibuild_merge_root
> +# @USAGE: <src-root> <dest-root>
> +# @DESCRIPTION:
> +# Merge the directory tree (fake root) from <src-root> to <dest-root>
> +# (the real root). Both directories have to be real, absolute paths
> +# (i.e. including ${D}). Source root will be removed.
> +#
> +# This functions uses locking to support merging during parallel
> +# installs.
> +multibuild_merge_root() {
> +       local src=${1}
> +       local dest=${2}
> +
> +       local lockfile=${T}/multibuild_merge_lock
> +
> +       if type -P lockf &>/dev/null; then
> +               # On BSD, we have 'lockf' wrapper.
> +               tar -C "${src}" -f - -c . \
> +                       | lockf "${lockfile}" tar -x -f - -C "${dest}"
> +       else
> +               local lock_fd
> +               if type -P flock &>/dev/null; then
> +                       # On Linux, we have 'flock' which can lock fd.
> +                       redirect_alloc_fd lock_fd "${lockfile}" '>>'
> +                       flock ${lock_fd}
> +               else
> +                       ewarn "multibuild: no locking service found, please report."
> +               fi
> +
> +               cp -a -l -n "${src}"/. "${dest}"/
> +
> +               if [[ ${lock_fd} ]]; then
> +                       # Close the lock file when we are done with it.
> +                       # Prevents deadlock if we aren't in a subshell.
> +                       eval "exec ${lock_fd}>&-"

Not following this bit.

The bash manpage says N>&DIGIT- redirects fd DIGIT to fd N. N may be
unspecified and defaults to 1.

It doesn't say what happens if DIGIT is unspecified. Nor does it say
it closes any file descriptors.

-A

> +               fi
> +       fi
> +
> +       if [[ ${?} -ne 0 ]]; then
> +               die "${MULTIBUILD_VARIANT:-(unknown)}: merging image failed."
> +       fi
> +
> +       rm -rf "${src}"
> +}
> +
>  _MULTIBUILD=1
>  fi
> --
> 1.8.1.5
>
>


^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [gentoo-dev] [PATCH 1/2] Introduce multibuild_merge_root() to merge interim installs.
  2013-03-23 17:44   ` [gentoo-dev] [PATCH 1/2] " Alec Warner
@ 2013-03-23 18:57     ` Michał Górny
  2013-03-23 18:57     ` [gentoo-dev] " Jonathan Callen
  1 sibling, 0 replies; 30+ messages in thread
From: Michał Górny @ 2013-03-23 18:57 UTC (permalink / raw
  To: gentoo-dev; +Cc: antarus

[-- Attachment #1: Type: text/plain, Size: 1194 bytes --]

On Sat, 23 Mar 2013 10:44:59 -0700
Alec Warner <antarus@gentoo.org> wrote:

> On Sat, Mar 23, 2013 at 9:26 AM, Michał Górny <mgorny@gentoo.org> wrote:
> > +               if [[ ${lock_fd} ]]; then
> > +                       # Close the lock file when we are done with it.
> > +                       # Prevents deadlock if we aren't in a subshell.
> > +                       eval "exec ${lock_fd}>&-"
> 
> Not following this bit.
> 
> The bash manpage says N>&DIGIT- redirects fd DIGIT to fd N. N may be
> unspecified and defaults to 1.
> It doesn't say what happens if DIGIT is unspecified.

Hm, that's interesting. It looks like they forgot to document it. I've
just tested it with bash-3.2 and it works as expected.

> Nor does it say it closes any file descriptors.

Well, if we assume that this fragment is correct, then it does:

  DIGIT is closed after being duplicated to N.


There's also the following in bash-4.2 info page:

  If >&- or <&- is preceded by {VARNAME}, the value of VARNAME defines
  the file descriptor to close.

But IMO it's more like describing a specific case, rather than introducing
>&-.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

^ permalink raw reply	[flat|nested] 30+ messages in thread

* [gentoo-dev] Re: [PATCH 1/2] Introduce multibuild_merge_root() to merge interim installs.
  2013-03-23 17:44   ` [gentoo-dev] [PATCH 1/2] " Alec Warner
  2013-03-23 18:57     ` Michał Górny
@ 2013-03-23 18:57     ` Jonathan Callen
  2013-03-23 19:02       ` Alec Warner
  1 sibling, 1 reply; 30+ messages in thread
From: Jonathan Callen @ 2013-03-23 18:57 UTC (permalink / raw
  To: gentoo-dev; +Cc: Alec Warner, Michał Górny

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 03/23/2013 01:44 PM, Alec Warner wrote:
> On Sat, Mar 23, 2013 at 9:26 AM, Michał Górny <mgorny@gentoo.org>
> wrote:
>> +                       # Close the lock file when we are done
>> with it. +                       # Prevents deadlock if we aren't
>> in a subshell. +                       eval "exec ${lock_fd}>&-"
> 
> Not following this bit.
> 
> The bash manpage says N>&DIGIT- redirects fd DIGIT to fd N. N may
> be unspecified and defaults to 1.
> 
> It doesn't say what happens if DIGIT is unspecified. Nor does it
> say it closes any file descriptors.
> 

The relevant section of bash(1) is reproduced below:

   Duplicating File Descriptors
       The redirection operator

              [n]<&word

       is used to duplicate input file descriptors.  If word expands to
       one or more digits, the file descriptor denoted by n is made to
       be a copy of that file descriptor.  If the digits in word do not
       specify a file descriptor open for input, a redirection error
       occurs.  If word evaluates to -, file descriptor n is closed.
       If n is not specified, the standard input (file descriptor 0) is
       used.

       The operator

              [n]>&word

       is used similarly to duplicate output file descriptors.  If n is
       not specified, the standard output (file descriptor 1) is used.
       If the digits in word do not specify a file descriptor open for
       output, a redirection error occurs.  As a special case, if n is
       omitted, and word does not expand to one or more digits, the
       standard output and standard error are redirected as described
       previously.

While the outcome of n>&- is not specified directly in the man page,
it is treated exactly as n<&- is.

- -- 
Jonathan Callen
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCgAGBQJRTfsWAAoJELHSF2kinlg4MO0P+wTQovWzYdfSgUP8H4WFmO1y
fpERqd1ZIoPle5Gl5+IUOWIHoMqkjS3mIAYUKF9cYwkufHRjbIs9l6NFGZg4wbRj
baCW6dRTbnbaM/YSiHLlstb4I5fcSwldwhtJrMuO9O6qLq2IFupJYkaSLnYJMs6C
HspU9W+bw4Vuvlk2wnhjGDGUj06ZIifQEOE6NYUVqF4GQaZlZpCsgA5BADPQL8zL
RAb90dJuRIuPPwgzuFka1+h0z4Sg5xxSlV7FS4c+r+HzrZCHHIHSyi9lOpOPB/8M
P+ZzJyDD3weW5AnAIuDjhJ3Rt0wI/2oYRRoVmC6hZ1CNSZuSjUCdWi6jvIvg99cR
TscJ0dKBl7bMcrrIILcXYgHtyKC/QRZWp6r0JxqOwX4L7FbPhF3qbtano+TxoCSr
X7hfCuRab1FEPE+jXeprH0c+N31SHrVBxfkJDG39MVx8hdJqXYzKCXah5/6lMJ63
Td2lc7Y3UBuXpxvQBoBCWTEV3vmKBrNX6v5Bu9TTLmrhfkcMsjCMX6TdU54tUoEU
YE37njpI4Thsl24/LPQtzkR8v9fVyYqVvvXylSq5HcM48VlqlhcI/EP0SRo9eUXu
CFRkwb7u5nX4YOdUMj+L0P59/N7Zobdfs/dCvGB4PDeJTqzFJowxVhm7Dh1Uxf6l
a1CBWFm9+S7kUPDTfwV6
=tFvf
-----END PGP SIGNATURE-----


^ permalink raw reply	[flat|nested] 30+ messages in thread

* [gentoo-dev] Re: [PATCH 1/2] Introduce multibuild_merge_root() to merge interim installs.
  2013-03-23 18:57     ` [gentoo-dev] " Jonathan Callen
@ 2013-03-23 19:02       ` Alec Warner
  0 siblings, 0 replies; 30+ messages in thread
From: Alec Warner @ 2013-03-23 19:02 UTC (permalink / raw
  To: Jonathan Callen; +Cc: gentoo-dev, Michał Górny

On Sat, Mar 23, 2013 at 11:57 AM, Jonathan Callen <abcd@gentoo.org> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
>
> On 03/23/2013 01:44 PM, Alec Warner wrote:
>> On Sat, Mar 23, 2013 at 9:26 AM, Michał Górny <mgorny@gentoo.org>
>> wrote:
>>> +                       # Close the lock file when we are done
>>> with it. +                       # Prevents deadlock if we aren't
>>> in a subshell. +                       eval "exec ${lock_fd}>&-"
>>
>> Not following this bit.
>>
>> The bash manpage says N>&DIGIT- redirects fd DIGIT to fd N. N may
>> be unspecified and defaults to 1.
>>
>> It doesn't say what happens if DIGIT is unspecified. Nor does it
>> say it closes any file descriptors.
>>
>
> The relevant section of bash(1) is reproduced below:
>
>    Duplicating File Descriptors
>        The redirection operator
>
>               [n]<&word
>
>        is used to duplicate input file descriptors.  If word expands to
>        one or more digits, the file descriptor denoted by n is made to
>        be a copy of that file descriptor.  If the digits in word do not
>        specify a file descriptor open for input, a redirection error
>        occurs.  If word evaluates to -, file descriptor n is closed.
>        If n is not specified, the standard input (file descriptor 0) is
>        used.
>
>        The operator
>
>               [n]>&word
>
>        is used similarly to duplicate output file descriptors.  If n is
>        not specified, the standard output (file descriptor 1) is used.
>        If the digits in word do not specify a file descriptor open for
>        output, a redirection error occurs.  As a special case, if n is
>        omitted, and word does not expand to one or more digits, the
>        standard output and standard error are redirected as described
>        previously.
>
> While the outcome of n>&- is not specified directly in the man page,
> it is treated exactly as n<&- is.

Ahh, I was reading:

   Moving File Descriptors
       The redirection operator

              [n]<&digit-

       moves the file descriptor digit to file descriptor n, or the
standard input (file descriptor 0) if n is not specified.  digit is
closed
       after being duplicated to n.

       Similarly, the redirection operator

              [n]>&digit-

       moves the file descriptor digit to file descriptor n, or the
standard output (file descriptor 1) if n is not specified.

I see where i went wrong now ;)

>
> - --
> Jonathan Callen
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.19 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQIcBAEBCgAGBQJRTfsWAAoJELHSF2kinlg4MO0P+wTQovWzYdfSgUP8H4WFmO1y
> fpERqd1ZIoPle5Gl5+IUOWIHoMqkjS3mIAYUKF9cYwkufHRjbIs9l6NFGZg4wbRj
> baCW6dRTbnbaM/YSiHLlstb4I5fcSwldwhtJrMuO9O6qLq2IFupJYkaSLnYJMs6C
> HspU9W+bw4Vuvlk2wnhjGDGUj06ZIifQEOE6NYUVqF4GQaZlZpCsgA5BADPQL8zL
> RAb90dJuRIuPPwgzuFka1+h0z4Sg5xxSlV7FS4c+r+HzrZCHHIHSyi9lOpOPB/8M
> P+ZzJyDD3weW5AnAIuDjhJ3Rt0wI/2oYRRoVmC6hZ1CNSZuSjUCdWi6jvIvg99cR
> TscJ0dKBl7bMcrrIILcXYgHtyKC/QRZWp6r0JxqOwX4L7FbPhF3qbtano+TxoCSr
> X7hfCuRab1FEPE+jXeprH0c+N31SHrVBxfkJDG39MVx8hdJqXYzKCXah5/6lMJ63
> Td2lc7Y3UBuXpxvQBoBCWTEV3vmKBrNX6v5Bu9TTLmrhfkcMsjCMX6TdU54tUoEU
> YE37njpI4Thsl24/LPQtzkR8v9fVyYqVvvXylSq5HcM48VlqlhcI/EP0SRo9eUXu
> CFRkwb7u5nX4YOdUMj+L0P59/N7Zobdfs/dCvGB4PDeJTqzFJowxVhm7Dh1Uxf6l
> a1CBWFm9+S7kUPDTfwV6
> =tFvf
> -----END PGP SIGNATURE-----


^ permalink raw reply	[flat|nested] 30+ messages in thread

* [gentoo-dev] [PATCH] Support wrapping headers for multilib ABIs.
  2013-03-23 16:46 ` [gentoo-dev] [PATCHES] Header wrapping support for multilib Diego Elio Pettenò
  2013-03-23 17:32   ` Michał Górny
@ 2013-03-23 19:56   ` Michał Górny
  2013-03-23 20:03     ` Michał Górny
  2013-03-23 22:01     ` [gentoo-dev] " Jonathan Callen
  1 sibling, 2 replies; 30+ messages in thread
From: Michał Górny @ 2013-03-23 19:56 UTC (permalink / raw
  To: gentoo-dev; +Cc: flameeyes, Michał Górny

---
 gx86/eclass/autotools-multilib.eclass | 86 +++++++++++++++++++++++++++++++++++
 1 file changed, 86 insertions(+)

diff --git a/gx86/eclass/autotools-multilib.eclass b/gx86/eclass/autotools-multilib.eclass
index d7372b0..e96fdaf 100644
--- a/gx86/eclass/autotools-multilib.eclass
+++ b/gx86/eclass/autotools-multilib.eclass
@@ -33,6 +33,28 @@ inherit autotools-utils multilib-build
 
 EXPORT_FUNCTIONS src_prepare src_configure src_compile src_test src_install
 
+# @ECLASS-VARIABLE: MULTILIB_WRAPPED_HEADERS
+# @DESCRIPTION:
+# A list of headers to wrap for multilib support. The listed headers
+# will be moved to a non-standard location and replace with a file
+# including them conditionally to current ABI.
+#
+# This variable has to be a bash array. Paths shall be relative to
+# installation root (${D}), and name regular files. Recursive wrapping
+# is not supported.
+#
+# Please note that header wrapping is *discouraged*. It is preferred to
+# install all headers in a subdirectory of libdir and use pkg-config to
+# locate the headers. Some C preprocessors will not work with wrapped
+# headers.
+#
+# Example:
+# @CODE
+# MULTILIB_WRAPPED_HEADERS=(
+#	/usr/include/foobar/config.h
+# )
+# @CODE
+
 autotools-multilib_src_prepare() {
 	autotools-utils_src_prepare "${@}"
 }
@@ -49,13 +71,77 @@ autotools-multilib_src_test() {
 	multilib_foreach_abi autotools-utils_src_test "${@}"
 }
 
+_autotools-multilib_wrap_headers() {
+	debug-print-function ${FUNCNAME} "$@"
+	local f
+
+	for f in "${MULTILIB_WRAPPED_HEADERS[@]}"; do
+		# drop leading slash if it's there
+		f=${f#/}
+
+		if [[ ${f} != usr/include/* ]]; then
+			die "Wrapping headers outside of /usr/include is not supported at the moment."
+		fi
+		# and then usr/include
+		f=${f#usr/include/}
+
+		local dir=${f%/*}
+
+		# $CHOST shall be set by multilib_toolchain_setup
+		dodir "/tmp/multilib-include/${CHOST}/${dir}"
+		mv "${ED}/usr/include/${f}" "${ED}/tmp/multilib-include/${CHOST}/${dir}/" || die
+
+		local defs
+		case "${ABI}" in
+			amd64)
+				defs='defined(__x86_64__) && !defined(__ILP32__)';;
+			x86)
+				defs='defined(__i386__)';;
+			x32)
+				defs='defined(__x86_64__) && defined(__ILP32__)';;
+			*)
+				die "Header wrapping for ${ABI} not supported yet";;
+		esac
+
+		if [[ ! -f ${ED}/tmp/multilib-include/${f} ]]; then
+			dodir "/tmp/multilib-include/${dir}"
+			cat > "${ED}/tmp/multilib-include/${f}" <<_EOF_ || die
+/* This file is auto-generated by autotools-multilib.eclass
+ * as a multilib-friendly wrapper. For the original content,
+ * please see the files that are #included below.
+ */
+
+#if ${defs}
+#	include <${CHOST}/${f}>
+#else
+#	error "No ABI matched, please report a bug to bugs.gentoo.org"
+#endif
+_EOF_
+		else
+			sed -e "/^#else/i\
+#elif ${defs}\n\
+#	include <${CHOST}/${f}>" \
+				-i "${ED}/tmp/multilib-include/${f}" || die
+		fi
+	done
+}
+
 autotools-multilib_src_install() {
 	autotools-multilib_secure_install() {
 		autotools-utils_src_install "${@}"
 
+		_autotools-multilib_wrap_headers
 		# Make sure all headers are the same for each ABI.
 		multilib_check_headers
 	}
 
 	multilib_foreach_abi autotools-multilib_secure_install "${@}"
+
+	# merge the wrapped headers
+	if [[ -d "${ED}"/tmp/multilib-include ]]; then
+		multibuild_merge_root \
+			"${ED}"/tmp/multilib-include "${ED}"/usr/include
+		# it can fail if something else uses /tmp
+		rmdir "${ED}"/tmp &>/dev/null
+	fi
 }
-- 
1.8.1.5



^ permalink raw reply related	[flat|nested] 30+ messages in thread

* Re: [gentoo-dev] [PATCH] Support wrapping headers for multilib ABIs.
  2013-03-23 19:56   ` [gentoo-dev] [PATCH] Support wrapping headers for multilib ABIs Michał Górny
@ 2013-03-23 20:03     ` Michał Górny
  2013-03-23 22:01     ` [gentoo-dev] " Jonathan Callen
  1 sibling, 0 replies; 30+ messages in thread
From: Michał Górny @ 2013-03-23 20:03 UTC (permalink / raw
  To: gentoo-dev; +Cc: flameeyes

[-- Attachment #1: Type: text/plain, Size: 501 bytes --]

Example result:

$ cat /usr/include/X11/X.h 
/* This file is auto-generated by autotools-multilib.eclass
 * as a multilib-friendly wrapper. For the original content,
 * please see the files that are #included below.
 */

#if defined(__i386__)
#	include <i686-pc-linux-gnu/X11/X.h>
#elif defined(__x86_64__) && !defined(__ILP32__)
#	include <x86_64-pc-linux-gnu/X11/X.h>
#else
#	error "No ABI matched, please report a bug to bugs.gentoo.org"
#endif

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

^ permalink raw reply	[flat|nested] 30+ messages in thread

* [gentoo-dev] Re: [PATCH] Support wrapping headers for multilib ABIs.
  2013-03-23 19:56   ` [gentoo-dev] [PATCH] Support wrapping headers for multilib ABIs Michał Górny
  2013-03-23 20:03     ` Michał Górny
@ 2013-03-23 22:01     ` Jonathan Callen
  1 sibling, 0 replies; 30+ messages in thread
From: Jonathan Callen @ 2013-03-23 22:01 UTC (permalink / raw
  To: gentoo-dev; +Cc: Michał Górny, flameeyes

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 03/23/2013 03:56 PM, Michał Górny wrote:
> +# This variable has to be a bash array. Paths shall be relative
> to +# installation root (${D}), and name regular files. Recursive
> wrapping +# is not supported.

Based on the code, this comment should read "Paths shall be relative
to installation root (${ED}), and name regular files."

- -- 
Jonathan Callen
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCgAGBQJRTiYdAAoJELHSF2kinlg4uC4QAJbQzbpgwBvrADDurGm8LID0
X8VuffHQovpV6m5Z5YM02u1NjJvVb9qXquGzlGowEjJg37u+M3JdfA6/ewNi4h7o
0RQIwRCv2TJNSfKeoce1hzFvmLAZlk7F8YZn9xJEDBBq37srbhWkugQJLNmmBlI2
UjLJXN3COxj3Z+biuqRmGPN7NoH/BpxRbaHd3y0E71ANesqpOxcoBIMztjJwJXI8
3WCmzKkwvpz0Suo4ycB4IQWIBH6+i3yg+iHw7CPz3kd4aCmraVWv5O2wm/pow9dR
X84m8JdJ5ZHhOl+Ic7obFiLf2yXeyQlFcyhwZ04+sHwhgpSdlNpJtVtjj//ncQPq
ghVTyNgpSBYixEwQl1cHTYYp28H9lIRoTa9l1hvfYfRkAMW9Skr5yRpCnAOC+2T5
HLb5hOX9nokcGjA9VyN8hN/nEOKX3FDlFDdIbRJDtmUhngE2yAcaveobSCD80iSN
dL5xacYTANQuT2s4jydqPtHHe6Iu+jJkLxjRV0nFldccosE/A2HJ0HbAfLC3jqBV
g+ZyBRq1mZoXYldmR7Rely77lfQ8UcMTPt7WhUImQ7qaSkt3czCZdMsClb6RFfN4
FEOd58+9xocYPz1SAtt0N+wDnDueoIeKxo4zjr5fisoUnqGjt8eI/y5NjbQXFmEz
dvU74R0WDh9lqbCqLg0m
=4Uls
-----END PGP SIGNATURE-----


^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [gentoo-dev] [PATCH 1/2] Introduce multibuild_merge_root() to merge interim installs.
  2013-03-23 17:28     ` Michał Górny
@ 2013-03-24  7:47       ` Ulrich Mueller
  2013-03-24 10:09         ` Michał Górny
  0 siblings, 1 reply; 30+ messages in thread
From: Ulrich Mueller @ 2013-03-24  7:47 UTC (permalink / raw
  To: Michał Górny; +Cc: gentoo-dev

>>>>> On Sat, 23 Mar 2013, Michał Górny wrote:

>> > +multibuild_merge_root() {
>> > + [...]
>> > +	if type -P lockf &>/dev/null; then
>> > +		# On BSD, we have 'lockf' wrapper.
>> > +		tar -C "${src}" -f - -c . \
>> > +			| lockf "${lockfile}" tar -x -f - -C "${dest}"
>> > +	else
>> > + [...]
>> > +		cp -a -l -n "${src}"/. "${dest}"/
>> 
>> You assume that cp is GNU, if there's no lockf. Is it guaranteed that
>> this is true on all platforms?

> They're all just assumptions. If it breaks somewhere, I'll improve
> the function.

Maybe explicitly testing for USERLAND would be less fragile? Or use
tar everywhere?

Ulrich


^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [gentoo-dev] [PATCH 1/2] Introduce multibuild_merge_root() to merge interim installs.
  2013-03-24  7:47       ` Ulrich Mueller
@ 2013-03-24 10:09         ` Michał Górny
  2013-03-24 13:40           ` Ulrich Mueller
  0 siblings, 1 reply; 30+ messages in thread
From: Michał Górny @ 2013-03-24 10:09 UTC (permalink / raw
  To: gentoo-dev; +Cc: ulm

[-- Attachment #1: Type: text/plain, Size: 914 bytes --]

On Sun, 24 Mar 2013 08:47:41 +0100
Ulrich Mueller <ulm@gentoo.org> wrote:

> >>>>> On Sat, 23 Mar 2013, Michał Górny wrote:
> 
> >> > +multibuild_merge_root() {
> >> > + [...]
> >> > +	if type -P lockf &>/dev/null; then
> >> > +		# On BSD, we have 'lockf' wrapper.
> >> > +		tar -C "${src}" -f - -c . \
> >> > +			| lockf "${lockfile}" tar -x -f - -C "${dest}"
> >> > +	else
> >> > + [...]
> >> > +		cp -a -l -n "${src}"/. "${dest}"/
> >> 
> >> You assume that cp is GNU, if there's no lockf. Is it guaranteed that
> >> this is true on all platforms?
> 
> > They're all just assumptions. If it breaks somewhere, I'll improve
> > the function.
> 
> Maybe explicitly testing for USERLAND would be less fragile? Or use
> tar everywhere?

I prefer something that can boom into face and let us know that it is
broken rather than silent, poor catch-all.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [gentoo-dev] [PATCH 1/2] Introduce multibuild_merge_root() to merge interim installs.
  2013-03-24 10:09         ` Michał Górny
@ 2013-03-24 13:40           ` Ulrich Mueller
  2013-03-24 14:13             ` Michał Górny
  0 siblings, 1 reply; 30+ messages in thread
From: Ulrich Mueller @ 2013-03-24 13:40 UTC (permalink / raw
  To: Michał Górny; +Cc: gentoo-dev

>>>>> On Sun, 24 Mar 2013, Michał Górny wrote:

>> >> > +multibuild_merge_root() {
>> >> > + [...]
>> >> > +	if type -P lockf &>/dev/null; then
>> >> > +		# On BSD, we have 'lockf' wrapper.
>> >> > +		tar -C "${src}" -f - -c . \
>> >> > +			| lockf "${lockfile}" tar -x -f - -C "${dest}"
>> >> > +	else
>> >> > + [...]
>> >> > +		cp -a -l -n "${src}"/. "${dest}"/

>> Maybe explicitly testing for USERLAND would be less fragile? Or use
>> tar everywhere?

> I prefer something that can boom into face and let us know that it
> is broken rather than silent, poor catch-all.

How about adding a couple of "assert" and "die" to tar and cp then?

Ulrich


^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [gentoo-dev] [PATCH 1/2] Introduce multibuild_merge_root() to merge interim installs.
  2013-03-24 13:40           ` Ulrich Mueller
@ 2013-03-24 14:13             ` Michał Górny
  2013-03-24 20:18               ` Zac Medico
  0 siblings, 1 reply; 30+ messages in thread
From: Michał Górny @ 2013-03-24 14:13 UTC (permalink / raw
  To: gentoo-dev; +Cc: ulm

[-- Attachment #1: Type: text/plain, Size: 1318 bytes --]

On Sun, 24 Mar 2013 14:40:58 +0100
Ulrich Mueller <ulm@gentoo.org> wrote:

> >>>>> On Sun, 24 Mar 2013, Michał Górny wrote:
> 
> >> >> > +multibuild_merge_root() {
> >> >> > + [...]
> >> >> > +	if type -P lockf &>/dev/null; then
> >> >> > +		# On BSD, we have 'lockf' wrapper.
> >> >> > +		tar -C "${src}" -f - -c . \
> >> >> > +			| lockf "${lockfile}" tar -x -f - -C "${dest}"
> >> >> > +	else
> >> >> > + [...]
> >> >> > +		cp -a -l -n "${src}"/. "${dest}"/
> 
> >> Maybe explicitly testing for USERLAND would be less fragile? Or use
> >> tar everywhere?
> 
> > I prefer something that can boom into face and let us know that it
> > is broken rather than silent, poor catch-all.
> 
> How about adding a couple of "assert" and "die" to tar and cp then?

Oh, I now see that the 'die' following the block does no longer catch
all cases properly. I will fix that.

Can you think of more assertions I should have there?

Oh, I didn't reply to use of userland. I thought about using that but
that would mean that I'd have to check all the userlands we support.
But that's probably better than assuming that a system having 'lockf'
has broken 'cp', true.

I will send an updated patch as soon as I clean up the mess Samuli made
with udev/systemd.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [gentoo-dev] [PATCH 2/2] Support wrapping headers for multilib ABIs.
  2013-03-23 16:26 ` [gentoo-dev] [PATCH 2/2] Support wrapping headers for multilib ABIs Michał Górny
@ 2013-03-24 15:14   ` Alexis Ballier
  2013-03-24 15:41     ` Michał Górny
  0 siblings, 1 reply; 30+ messages in thread
From: Alexis Ballier @ 2013-03-24 15:14 UTC (permalink / raw
  To: gentoo-dev; +Cc: mgorny

On Sat, 23 Mar 2013 17:26:38 +0100
Michał Górny <mgorny@gentoo.org> wrote:

> ---
>  gx86/eclass/autotools-multilib.eclass | 82
> +++++++++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+)
> 
> diff --git a/gx86/eclass/autotools-multilib.eclass
> b/gx86/eclass/autotools-multilib.eclass index d7372b0..c65c777 100644
> --- a/gx86/eclass/autotools-multilib.eclass
> +++ b/gx86/eclass/autotools-multilib.eclass


why not multilib-build ?


> @@ -33,6 +33,28 @@ inherit autotools-utils multilib-build
>  
>  EXPORT_FUNCTIONS src_prepare src_configure src_compile src_test
> src_install 
> +# @ECLASS-VARIABLE: MULTILIB_WRAPPED_HEADERS
> +# @DESCRIPTION:
> +# A list of headers to wrap for multilib support. The listed headers
> +# will be moved to a non-standard location and replace with a file
> +# including them conditionally to current ABI.
> +#
> +# This variable has to be a bash array. Paths shall be relative to
> +# installation root (${D}), and name regular files. Recursive
> wrapping +# is not supported.
> +#
> +# Please note that header wrapping is *discouraged*. It is preferred
> to +# install all headers in a subdirectory of libdir and use
> pkg-config to +# locate the headers. Some C preprocessors will not
> work with wrapped +# headers.
> +#
> +# Example:
> +# @CODE
> +# MULTILIB_WRAPPED_HEADERS=(
> +#	/usr/include/foobar/config.h
> +# )
> +# @CODE
> +
>  autotools-multilib_src_prepare() {
>  	autotools-utils_src_prepare "${@}"
>  }
> @@ -49,13 +71,73 @@ autotools-multilib_src_test() {
>  	multilib_foreach_abi autotools-utils_src_test "${@}"
>  }
>  
> +_autotools-multilib_wrap_headers() {
> +	debug-print-function ${FUNCNAME} "$@"
> +	local f
> +
> +	for f in "${MULTILIB_WRAPPED_HEADERS[@]}"; do
> +		# drop leading slash if it's there
> +		f=${f#/}
> +
> +		if [[ ${f} != usr/include/* ]]; then
> +			die "Wrapping headers outside
> of /usr/include is not supported at the moment."
> +		fi

Do you really want to support this at some point? Why?
I'd just go for paths relative to $EPREFIX/usr/include (or
$ED/usr/include) for MULTILIB_WRAPPED_HEADERS. That would simplify the
code.


> +		# and then usr/include
> +		f=${f#usr/include/}
> +
> +		local dir=${f%/*}
> +
> +		# $CHOST shall be set by multilib_toolchain_setup
> +		dodir "/tmp/multilib-include/${CHOST}/${dir}"
> +		mv "${ED}/usr/include/${f}"
> "${ED}/tmp/multilib-include/${CHOST}/${dir}/" || die +

why not use $T rather than $ED/tmp ?

> +		if [[ ! -f ${ED}/tmp/multilib-include/${f} ]]; then
> +			dodir "/tmp/multilib-include/${dir}"
> +			cat > "${ED}/tmp/multilib-include/${f}"
> <<_EOF_ || die +/* This file is auto-generated by
> autotools-multilib.eclass
> + * as a multilib-friendly wrapper to /${f}. For the original content,
> + * please see the files that are #included below.
> + */
> +_EOF_
> +		fi
> +
> +		local defs
> +		case "${ABI}" in

didn't you say $ABI may have name collisions?
considering the code below, it seems much safer to match on $CHOST

[...]

It'd be nice to have an attempt to support all the ABIs gentoo supports
in that file: I'd prefer to spot possible problems with this solution
vs. sedding a template for example to be seen before having to rewrite
it.
For example, IIRC, ppc64 defines both __powerpc__ and __powerpc64__,
the natural way would be:
#if defined(__powerpc64__)
ppc64 stuff
#elif defined(__powerpc__)
ppc stuff
#endif

with your approach that'd be:
#if defined(__powerpc__) && !defined(__powerpc64__)
ppc stuff
#elif defined(__powerpc64__)
ppc64 stuff
#endif


doing with a template has its disadvantages but allows more flexibility
in how the #ifery is written; I don't want to realize your approach
cannot deal with some weird arches after it has been deployed.


thanks for working on this,

Alexis.


^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [gentoo-dev] [PATCH 2/2] Support wrapping headers for multilib ABIs.
  2013-03-24 15:14   ` Alexis Ballier
@ 2013-03-24 15:41     ` Michał Górny
  2013-03-24 20:01       ` Alexis Ballier
  0 siblings, 1 reply; 30+ messages in thread
From: Michał Górny @ 2013-03-24 15:41 UTC (permalink / raw
  To: gentoo-dev; +Cc: aballier

[-- Attachment #1: Type: text/plain, Size: 6018 bytes --]

On Sun, 24 Mar 2013 16:14:52 +0100
Alexis Ballier <aballier@gentoo.org> wrote:

> On Sat, 23 Mar 2013 17:26:38 +0100
> Michał Górny <mgorny@gentoo.org> wrote:
> 
> > ---
> >  gx86/eclass/autotools-multilib.eclass | 82
> > +++++++++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+)
> > 
> > diff --git a/gx86/eclass/autotools-multilib.eclass
> > b/gx86/eclass/autotools-multilib.eclass index d7372b0..c65c777 100644
> > --- a/gx86/eclass/autotools-multilib.eclass
> > +++ b/gx86/eclass/autotools-multilib.eclass
> 
> 
> why not multilib-build ?

Because it has two parts which have to be called at the right time.
It doesn't have a public API yet, so I'm putting it where I can test it
sanely.

> > @@ -33,6 +33,28 @@ inherit autotools-utils multilib-build
> >  
> >  EXPORT_FUNCTIONS src_prepare src_configure src_compile src_test
> > src_install 
> > +# @ECLASS-VARIABLE: MULTILIB_WRAPPED_HEADERS
> > +# @DESCRIPTION:
> > +# A list of headers to wrap for multilib support. The listed headers
> > +# will be moved to a non-standard location and replace with a file
> > +# including them conditionally to current ABI.
> > +#
> > +# This variable has to be a bash array. Paths shall be relative to
> > +# installation root (${D}), and name regular files. Recursive
> > wrapping +# is not supported.
> > +#
> > +# Please note that header wrapping is *discouraged*. It is preferred
> > to +# install all headers in a subdirectory of libdir and use
> > pkg-config to +# locate the headers. Some C preprocessors will not
> > work with wrapped +# headers.
> > +#
> > +# Example:
> > +# @CODE
> > +# MULTILIB_WRAPPED_HEADERS=(
> > +#	/usr/include/foobar/config.h
> > +# )
> > +# @CODE
> > +
> >  autotools-multilib_src_prepare() {
> >  	autotools-utils_src_prepare "${@}"
> >  }
> > @@ -49,13 +71,73 @@ autotools-multilib_src_test() {
> >  	multilib_foreach_abi autotools-utils_src_test "${@}"
> >  }
> >  
> > +_autotools-multilib_wrap_headers() {
> > +	debug-print-function ${FUNCNAME} "$@"
> > +	local f
> > +
> > +	for f in "${MULTILIB_WRAPPED_HEADERS[@]}"; do
> > +		# drop leading slash if it's there
> > +		f=${f#/}
> > +
> > +		if [[ ${f} != usr/include/* ]]; then
> > +			die "Wrapping headers outside
> > of /usr/include is not supported at the moment."
> > +		fi
> 
> Do you really want to support this at some point? Why?

Honestly? No. But if people need it, I don't see much of a problem to
add the support in the future.

> I'd just go for paths relative to $EPREFIX/usr/include (or
> $ED/usr/include) for MULTILIB_WRAPPED_HEADERS. That would simplify the
> code.

That's true. But on the other hand, that would introduce yet another
root for paths which would probably end up being confusing.
Using /usr/include/... feels more natural IMO.

And of course, it leaves the possibility of supporting other locations
in the future.

> > +		# and then usr/include
> > +		f=${f#usr/include/}
> > +
> > +		local dir=${f%/*}
> > +
> > +		# $CHOST shall be set by multilib_toolchain_setup
> > +		dodir "/tmp/multilib-include/${CHOST}/${dir}"
> > +		mv "${ED}/usr/include/${f}"
> > "${ED}/tmp/multilib-include/${CHOST}/${dir}/" || die +
> 
> why not use $T rather than $ED/tmp ?

I prefer using $D rather than $T for files which are intended to be
installed. Purely theoretically, $T could be on another filesystem than
$D. Then, moving the file back and forth could cause it to lose some
of the metadata -- and will be slower, of course.

> > +		if [[ ! -f ${ED}/tmp/multilib-include/${f} ]]; then
> > +			dodir "/tmp/multilib-include/${dir}"
> > +			cat > "${ED}/tmp/multilib-include/${f}"
> > <<_EOF_ || die +/* This file is auto-generated by
> > autotools-multilib.eclass
> > + * as a multilib-friendly wrapper to /${f}. For the original content,
> > + * please see the files that are #included below.
> > + */
> > +_EOF_
> > +		fi
> > +
> > +		local defs
> > +		case "${ABI}" in
> 
> didn't you say $ABI may have name collisions?
> considering the code below, it seems much safer to match on $CHOST

Yes, that's why I've put the whole matching inline instead of
introducing a function to obtain the values. The current design
of the eclass -- which was quite stupid of me -- doesn't provide
a way to access that ABI_* thing. I am going to change that
in the future but don't want to step into two different issues
at a time.

At a first glance, $CHOST sounds like a neat idea. But I'm afraid it's
not a really safe choice. From a quick glance, CHOST values for x86
were:

i*86-pc-linux-gnu
x86_64-pc-linux-gnu
x86_64-pc-linux-gnux32

I feel like there's a lot of variables here, ends up in a bit ugly
pattern matching IMO.

> [...]
> 
> It'd be nice to have an attempt to support all the ABIs gentoo supports
> in that file: I'd prefer to spot possible problems with this solution
> vs. sedding a template for example to be seen before having to rewrite
> it.
> For example, IIRC, ppc64 defines both __powerpc__ and __powerpc64__,
> the natural way would be:
> #if defined(__powerpc64__)
> ppc64 stuff
> #elif defined(__powerpc__)
> ppc stuff
> #endif
> 
> with your approach that'd be:
> #if defined(__powerpc__) && !defined(__powerpc64__)
> ppc stuff
> #elif defined(__powerpc64__)
> ppc64 stuff
> #endif

I had the same problem with x32. I've chosen the solution which worked
and was easy to implement.

> doing with a template has its disadvantages but allows more flexibility
> in how the #ifery is written; I don't want to realize your approach
> cannot deal with some weird arches after it has been deployed.

To be honest, I can't really imagine how we could work with a template
here. It would at least require the function to be aware of other ABIs
which I really wanted to avoid. Of course, then there's the whole code
to handle it, and honestly I don't have a vision how it would work.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [gentoo-dev] [PATCH 2/2] Support wrapping headers for multilib ABIs.
  2013-03-24 15:41     ` Michał Górny
@ 2013-03-24 20:01       ` Alexis Ballier
  2013-03-25 22:22         ` [gentoo-dev] [PATCH] " Michał Górny
  0 siblings, 1 reply; 30+ messages in thread
From: Alexis Ballier @ 2013-03-24 20:01 UTC (permalink / raw
  To: gentoo-dev; +Cc: mgorny

On Sun, 24 Mar 2013 16:41:15 +0100
Michał Górny <mgorny@gentoo.org> wrote:

> On Sun, 24 Mar 2013 16:14:52 +0100
> Alexis Ballier <aballier@gentoo.org> wrote:
> 
> > On Sat, 23 Mar 2013 17:26:38 +0100
> > Michał Górny <mgorny@gentoo.org> wrote:
> > 
> > > ---
> > >  gx86/eclass/autotools-multilib.eclass | 82
> > > +++++++++++++++++++++++++++++++++++ 1 file changed, 82
> > > insertions(+)
> > > 
> > > diff --git a/gx86/eclass/autotools-multilib.eclass
> > > b/gx86/eclass/autotools-multilib.eclass index d7372b0..c65c777
> > > 100644 --- a/gx86/eclass/autotools-multilib.eclass
> > > +++ b/gx86/eclass/autotools-multilib.eclass
> > 
> > 
> > why not multilib-build ?
> 
> Because it has two parts which have to be called at the right time.
> It doesn't have a public API yet, so I'm putting it where I can test
> it sanely.


Well, it has nothing to do with autotools :/

I'm not sure how to do it properly though... the function should be
"shared" but multilib-build doesn't export any phase function by design.
Maybe put the function and the variable documentation in
multilib-build, call it from autotools-multilib and add documentation
to autotools-multilib like: "this eclass supports header wrapping, see
multilib-build documentation about MULTILIB_WRAPPED_HEADERS variable"

[...]
> > I'd just go for paths relative to $EPREFIX/usr/include (or
> > $ED/usr/include) for MULTILIB_WRAPPED_HEADERS. That would simplify
> > the code.
> 
> That's true. But on the other hand, that would introduce yet another
> root for paths which would probably end up being confusing.
> Using /usr/include/... feels more natural IMO.

if you have to strip /usr/include to use the input, then just don't
require /usr/include as input ;)

> And of course, it leaves the possibility of supporting other locations
> in the future.

But you're right: There's no need to unnecessarily ban corner cases.


> > > +		# and then usr/include
> > > +		f=${f#usr/include/}
> > > +
> > > +		local dir=${f%/*}
> > > +
> > > +		# $CHOST shall be set by multilib_toolchain_setup
> > > +		dodir "/tmp/multilib-include/${CHOST}/${dir}"
> > > +		mv "${ED}/usr/include/${f}"
> > > "${ED}/tmp/multilib-include/${CHOST}/${dir}/" || die +
> > 
> > why not use $T rather than $ED/tmp ?
> 
> I prefer using $D rather than $T for files which are intended to be
> installed. Purely theoretically, $T could be on another filesystem
> than $D. Then, moving the file back and forth could cause it to lose
> some of the metadata -- and will be slower, of course.

and if some file is left behind it will install stuff in /tmp :/

I usually tend to consider $D as a write-only file system and consider
it cleaner that way, but that's true, moving files around is likely
cleaner if done within $D

> 
> > > +		if [[ ! -f ${ED}/tmp/multilib-include/${f} ]];
> > > then
> > > +			dodir "/tmp/multilib-include/${dir}"
> > > +			cat > "${ED}/tmp/multilib-include/${f}"
> > > <<_EOF_ || die +/* This file is auto-generated by
> > > autotools-multilib.eclass
> > > + * as a multilib-friendly wrapper to /${f}. For the original
> > > content,
> > > + * please see the files that are #included below.
> > > + */
> > > +_EOF_
> > > +		fi
> > > +
> > > +		local defs
> > > +		case "${ABI}" in
> > 
> > didn't you say $ABI may have name collisions?
> > considering the code below, it seems much safer to match on $CHOST
> 
> Yes, that's why I've put the whole matching inline instead of
> introducing a function to obtain the values. The current design
> of the eclass -- which was quite stupid of me -- doesn't provide
> a way to access that ABI_* thing. I am going to change that
> in the future but don't want to step into two different issues
> at a time.
> 
> At a first glance, $CHOST sounds like a neat idea. But I'm afraid it's
> not a really safe choice. From a quick glance, CHOST values for x86
> were:
> 
> i*86-pc-linux-gnu
> x86_64-pc-linux-gnu
> x86_64-pc-linux-gnux32
> 
> I feel like there's a lot of variables here, ends up in a bit ugly
> pattern matching IMO.

Well, almost every matching I've seen uses CHOST, because it's the
safest choice. For your example: 
i?86*) ;;
x86_64*gnux32) ;;
x86_64*) ;;

I haven't checked the details, but that'd be even better: freebsd
defines a different ABI (which is correct because it's incompatible)
but the #ifery for this wrapper is the same, so with this CHOST,
matching freebsd (or other OSes) comes for free. In the end it'll be
shorter :)


> > [...]
> > 
> > It'd be nice to have an attempt to support all the ABIs gentoo
> > supports in that file: I'd prefer to spot possible problems with
> > this solution vs. sedding a template for example to be seen before
> > having to rewrite it.
> > For example, IIRC, ppc64 defines both __powerpc__ and __powerpc64__,
> > the natural way would be:
> > #if defined(__powerpc64__)
> > ppc64 stuff
> > #elif defined(__powerpc__)
> > ppc stuff
> > #endif
> > 
> > with your approach that'd be:
> > #if defined(__powerpc__) && !defined(__powerpc64__)
> > ppc stuff
> > #elif defined(__powerpc64__)
> > ppc64 stuff
> > #endif
> 
> I had the same problem with x32. I've chosen the solution which worked
> and was easy to implement.

it's easy too :p


> > doing with a template has its disadvantages but allows more
> > flexibility in how the #ifery is written; I don't want to realize
> > your approach cannot deal with some weird arches after it has been
> > deployed.
> 
> To be honest, I can't really imagine how we could work with a template
> here. It would at least require the function to be aware of other ABIs
> which I really wanted to avoid. Of course, then there's the whole code
> to handle it, and honestly I don't have a vision how it would work.

Template (short version):

#if defined(__powerpc64__)
ABI=ppc64
#elif defined(__powerpc__)
ABI=ppc
#else
#error "fail"
#endif

When installing a header for $ABI:
sed -e "s/ABI=${ABI}\$/#include <foo>/"

After installing everything:
sed -e "s/ABI=.*/#error \"sorry, no support for \0\"/"


See, there's no need to be aware of any ABI and you even get more
meaningful #errors since you now know the #if path it used (because of
the \0) :)

The clear disadvantage of this approach is that every wrapped header
will have a long list of #if #elif #endif for every gentoo ABI, most of
the paths through these conditions will be invalid, and it will be
harder to read.
I really prefer generating the wrapper on the fly like you do but I
want to be sure it's good enough for all our use cases.

Alexis.


^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [gentoo-dev] [PATCH 1/2] Introduce multibuild_merge_root() to merge interim installs.
  2013-03-24 14:13             ` Michał Górny
@ 2013-03-24 20:18               ` Zac Medico
  0 siblings, 0 replies; 30+ messages in thread
From: Zac Medico @ 2013-03-24 20:18 UTC (permalink / raw
  To: gentoo-dev; +Cc: Michał Górny

On 03/24/2013 07:13 AM, Michał Górny wrote:
> Oh, I didn't reply to use of userland. I thought about using that but
> that would mean that I'd have to check all the userlands we support.

BSD and GNU should be the only relevant USERLAND values (no others are
listed in profiles/desc/userland.desc). In portage code, I always do the
USERLAND conditionals like this:

if [[ ${USERLAND} == BSD ]] ; then
  # bsd stuff
else
  # gnu stuff
fi

That way, if USERLAND happens to be unset for whatever reason, it
defaults to the GNU case. GNU seems like a reasonable default, since for
prefix portage we use a GNU userland even for *BSD kernels.
-- 
Thanks,
Zac


^ permalink raw reply	[flat|nested] 30+ messages in thread

* [gentoo-dev] [PATCH] Support wrapping headers for multilib ABIs.
  2013-03-24 20:01       ` Alexis Ballier
@ 2013-03-25 22:22         ` Michał Górny
  0 siblings, 0 replies; 30+ messages in thread
From: Michał Górny @ 2013-03-25 22:22 UTC (permalink / raw
  To: gentoo-dev; +Cc: aballier, Michał Górny

This time using a template like Alexis suggested. I've only added
the ABIs currently supported by multilib-build. Feel free to provide me
with a more complete one.
---
 gx86/eclass/autotools-multilib.eclass | 93 +++++++++++++++++++++++++++++++++++
 1 file changed, 93 insertions(+)

diff --git a/gx86/eclass/autotools-multilib.eclass b/gx86/eclass/autotools-multilib.eclass
index d7372b0..ccbaac6 100644
--- a/gx86/eclass/autotools-multilib.eclass
+++ b/gx86/eclass/autotools-multilib.eclass
@@ -33,6 +33,28 @@ inherit autotools-utils multilib-build
 
 EXPORT_FUNCTIONS src_prepare src_configure src_compile src_test src_install
 
+# @ECLASS-VARIABLE: MULTILIB_WRAPPED_HEADERS
+# @DESCRIPTION:
+# A list of headers to wrap for multilib support. The listed headers
+# will be moved to a non-standard location and replace with a file
+# including them conditionally to current ABI.
+#
+# This variable has to be a bash array. Paths shall be relative to
+# installation root (${ED}), and name regular files. Recursive wrapping
+# is not supported.
+#
+# Please note that header wrapping is *discouraged*. It is preferred to
+# install all headers in a subdirectory of libdir and use pkg-config to
+# locate the headers. Some C preprocessors will not work with wrapped
+# headers.
+#
+# Example:
+# @CODE
+# MULTILIB_WRAPPED_HEADERS=(
+#	/usr/include/foobar/config.h
+# )
+# @CODE
+
 autotools-multilib_src_prepare() {
 	autotools-utils_src_prepare "${@}"
 }
@@ -49,13 +71,84 @@ autotools-multilib_src_test() {
 	multilib_foreach_abi autotools-utils_src_test "${@}"
 }
 
+_autotools-multilib_wrap_headers() {
+	debug-print-function ${FUNCNAME} "$@"
+	local f
+
+	for f in "${MULTILIB_WRAPPED_HEADERS[@]}"; do
+		# drop leading slash if it's there
+		f=${f#/}
+
+		if [[ ${f} != usr/include/* ]]; then
+			die "Wrapping headers outside of /usr/include is not supported at the moment."
+		fi
+		# and then usr/include
+		f=${f#usr/include/}
+
+		local dir=${f%/*}
+
+		# $CHOST shall be set by multilib_toolchain_setup
+		dodir "/tmp/multilib-include/${CHOST}/${dir}"
+		mv "${ED}/usr/include/${f}" "${ED}/tmp/multilib-include/${CHOST}/${dir}/" || die
+
+		if [[ ! -f ${ED}/tmp/multilib-include/${f} ]]; then
+			dodir "/tmp/multilib-include/${dir}"
+			# a generic template
+			cat > "${ED}/tmp/multilib-include/${f}" <<_EOF_ || die
+/* This file is auto-generated by autotools-multilib.eclass
+ * as a multilib-friendly wrapper. For the original content,
+ * please see the files that are #included below.
+ */
+
+#if defined(__x86_64__) /* amd64 */
+#	if defined(__ILP32__) /* x32 ABI */
+#		error "abi_x86_x32 not supported by the package."
+#	else /* 64-bit ABI */
+#		error "abi_x86_64 not supported by the package."
+#	endif
+#elif defined(__i386__) /* plain x86 */
+#	error "abi_x86_32 not supported by the package."
+#else
+#	error "No ABI matched, please report a bug to bugs.gentoo.org"
+#endif
+_EOF_
+		fi
+
+		# XXX: get abi_* directly
+		local abi_flag
+		case "${ABI}" in
+			amd64)
+				abi_flag=abi_x86_64;;
+			x86)
+				abi_flag=abi_x86_32;;
+			x32)
+				abi_flag=abi_x86_x32;;
+			*)
+				die "Header wrapping for ${ABI} not supported yet";;
+		esac
+
+		# Note: match a space afterwards to avoid collision potential.
+		sed -e "/${abi_flag} /s&error.*&include <${CHOST}/${f}>&" \
+			-i "${ED}/tmp/multilib-include/${f}" || die
+	done
+}
+
 autotools-multilib_src_install() {
 	autotools-multilib_secure_install() {
 		autotools-utils_src_install "${@}"
 
+		_autotools-multilib_wrap_headers
 		# Make sure all headers are the same for each ABI.
 		multilib_check_headers
 	}
 
 	multilib_foreach_abi autotools-multilib_secure_install "${@}"
+
+	# merge the wrapped headers
+	if [[ -d "${ED}"/tmp/multilib-include ]]; then
+		multibuild_merge_root \
+			"${ED}"/tmp/multilib-include "${ED}"/usr/include
+		# it can fail if something else uses /tmp
+		rmdir "${ED}"/tmp &>/dev/null
+	fi
 }
-- 
1.8.1.5



^ permalink raw reply related	[flat|nested] 30+ messages in thread

* [gentoo-dev] [PATCH] Introduce multibuild_merge_root() to merge interim installs.
  2013-03-23 17:00   ` Ulrich Mueller
  2013-03-23 17:28     ` Michał Górny
@ 2013-03-25 22:42     ` Michał Górny
  1 sibling, 0 replies; 30+ messages in thread
From: Michał Górny @ 2013-03-25 22:42 UTC (permalink / raw
  To: gentoo-dev; +Cc: ulm, zmedico, Michał Górny

This is mostly a copy from distutils-r1. The function does copy all the
files from one location onto another, preserving whatever possible. It
can be run in parallel too without the risk of race conditions.

As suggested by Ulrich and Zac, I've modified the code to use checks
based on userland rather than tool checking. The code supports BSD and
GNU userlands as defined in the profiles. With any other userland, it
explicitly dies.
---
 gx86/eclass/distutils-r1.eclass | 41 +-------------------------------
 gx86/eclass/multibuild.eclass   | 52 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 53 insertions(+), 40 deletions(-)

diff --git a/gx86/eclass/distutils-r1.eclass b/gx86/eclass/distutils-r1.eclass
index 0982e6c..3c21741 100644
--- a/gx86/eclass/distutils-r1.eclass
+++ b/gx86/eclass/distutils-r1.eclass
@@ -449,49 +449,10 @@ distutils-r1_python_install() {
 
 	if [[ ! ${DISTUTILS_SINGLE_IMPL} ]]; then
 		_distutils-r1_rename_scripts "${root}"
-		_distutils-r1_merge_root "${root}" "${D}"
+		multibuild_merge_root "${root}" "${D}"
 	fi
 }
 
-# @FUNCTION: distutils-r1_merge_root
-# @USAGE: <src-root> <dest-root>
-# @INTERNAL
-# @DESCRIPTION:
-# Merge the directory tree from <src-root> to <dest-root>, removing
-# the <src-root> in the process.
-_distutils-r1_merge_root() {
-	local src=${1}
-	local dest=${2}
-
-	local lockfile=${T}/distutils-r1-merge-lock
-
-	if type -P lockf &>/dev/null; then
-		# On BSD, we have 'lockf' wrapper.
-		tar -C "${src}" -f - -c . \
-			| lockf "${lockfile}" tar -x -f - -C "${dest}"
-	else
-		local lock_fd
-		if type -P flock &>/dev/null; then
-			# On Linux, we have 'flock' which can lock fd.
-			redirect_alloc_fd lock_fd "${lockfile}" '>>'
-			flock ${lock_fd}
-		else
-			ewarn "distutils-r1: no locking service found, please report."
-		fi
-
-		cp -a -l -n "${src}"/. "${dest}"/
-
-		if [[ ${lock_fd} ]]; then
-			# Close the lock file when we are done with it.
-			# Prevents deadlock if we aren't in a subshell.
-			eval "exec ${lock_fd}>&-"
-		fi
-	fi
-	[[ ${?} == 0 ]] || die "Merging ${EPYTHON} image failed."
-
-	rm -rf "${src}"
-}
-
 # @FUNCTION: distutils-r1_python_install_all
 # @DESCRIPTION:
 # The default python_install_all(). It installs the documentation.
diff --git a/gx86/eclass/multibuild.eclass b/gx86/eclass/multibuild.eclass
index a3f1402..1152245 100644
--- a/gx86/eclass/multibuild.eclass
+++ b/gx86/eclass/multibuild.eclass
@@ -238,5 +238,57 @@ run_in_build_dir() {
 	return ${ret}
 }
 
+# @FUNCTION: multibuild_merge_root
+# @USAGE: <src-root> <dest-root>
+# @DESCRIPTION:
+# Merge the directory tree (fake root) from <src-root> to <dest-root>
+# (the real root). Both directories have to be real, absolute paths
+# (i.e. including ${D}). Source root will be removed.
+#
+# This functions uses locking to support merging during parallel
+# installs.
+multibuild_merge_root() {
+	local src=${1}
+	local dest=${2}
+
+	local lockfile=${T}/multibuild_merge_lock
+	local ret
+
+	if use userland_BSD; then
+		# Locking is done by 'lockf' which can wrap a command.
+		# 'cp -a -n' is broken:
+		# http://www.freebsd.org/cgi/query-pr.cgi?pr=174489
+		# using tar instead which is universal but terribly slow.
+
+		tar -C "${src}" -f - -c . \
+			| lockf "${lockfile}" tar -x -f - -C "${dest}"
+		[[ ${PIPESTATUS[*]} == '0 0' ]]
+		ret=${?}
+	elif use userland_GNU; then
+		# GNU has 'flock' which can't wrap commands but can lock
+		# a fd which is good enough for us.
+		# and cp works with '-a -n'.
+
+		local lock_fd
+		redirect_alloc_fd lock_fd "${lockfile}" '>>'
+		flock ${lock_fd}
+
+		cp -a -l -n "${src}"/. "${dest}"/
+		ret=${?}
+
+		# Close the lock file when we are done with it.
+		# Prevents deadlock if we aren't in a subshell.
+		eval "exec ${lock_fd}>&-"
+	else
+		die "Unsupported userland (${USERLAND}), please report."
+	fi
+
+	if [[ ${ret} -ne 0 ]]; then
+		die "${MULTIBUILD_VARIANT:-(unknown)}: merging image failed."
+	fi
+
+	rm -rf "${src}"
+}
+
 _MULTIBUILD=1
 fi
-- 
1.8.1.5



^ permalink raw reply related	[flat|nested] 30+ messages in thread

* Re: [gentoo-dev] [PATCHES] Header wrapping support for multilib
  2013-03-23 16:25 [gentoo-dev] [PATCHES] Header wrapping support for multilib Michał Górny
                   ` (2 preceding siblings ...)
  2013-03-23 16:46 ` [gentoo-dev] [PATCHES] Header wrapping support for multilib Diego Elio Pettenò
@ 2013-04-01  9:19 ` Michał Górny
  2013-04-02 10:59   ` Alexis Ballier
  3 siblings, 1 reply; 30+ messages in thread
From: Michał Górny @ 2013-04-01  9:19 UTC (permalink / raw
  To: gentoo-dev

[-- Attachment #1: Type: text/plain, Size: 301 bytes --]

On Sat, 23 Mar 2013 17:25:32 +0100
Michał Górny <mgorny@gentoo.org> wrote:

> I've finally got around to writing the header wrapping functions for
> multilib. That's an initial yet working draft. I will send patches in
> reply to this mail.

Committed.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [gentoo-dev] [PATCHES] Header wrapping support for multilib
  2013-04-01  9:19 ` [gentoo-dev] [PATCHES] Header wrapping support for multilib Michał Górny
@ 2013-04-02 10:59   ` Alexis Ballier
  2013-04-02 11:47     ` Michał Górny
  0 siblings, 1 reply; 30+ messages in thread
From: Alexis Ballier @ 2013-04-02 10:59 UTC (permalink / raw
  To: gentoo-dev; +Cc: mgorny

On Mon, 1 Apr 2013 11:19:51 +0200
Michał Górny <mgorny@gentoo.org> wrote:

> On Sat, 23 Mar 2013 17:25:32 +0100
> Michał Górny <mgorny@gentoo.org> wrote:
> 
> > I've finally got around to writing the header wrapping functions for
> > multilib. That's an initial yet working draft. I will send patches
> > in reply to this mail.
> 
> Committed.
> 

Thanks. Sorry for not keeping up with the discussions, I was very busy
last week.

Some comments:
- I would have put the template in a separate file for better clarity
  but I guess having it online in the eclass is more error-proof.
- Header wrapping really has nothing to do with autotools and I
  strongly believe this should be moved to multilib-build.

Alexis.


^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [gentoo-dev] [PATCHES] Header wrapping support for multilib
  2013-04-02 10:59   ` Alexis Ballier
@ 2013-04-02 11:47     ` Michał Górny
  2013-04-04  8:07       ` Alexis Ballier
  0 siblings, 1 reply; 30+ messages in thread
From: Michał Górny @ 2013-04-02 11:47 UTC (permalink / raw
  To: gentoo-dev; +Cc: aballier

[-- Attachment #1: Type: text/plain, Size: 1147 bytes --]

On Tue, 2 Apr 2013 12:59:43 +0200
Alexis Ballier <aballier@gentoo.org> wrote:

> On Mon, 1 Apr 2013 11:19:51 +0200
> Michał Górny <mgorny@gentoo.org> wrote:
> 
> > On Sat, 23 Mar 2013 17:25:32 +0100
> > Michał Górny <mgorny@gentoo.org> wrote:
> > 
> > > I've finally got around to writing the header wrapping functions for
> > > multilib. That's an initial yet working draft. I will send patches
> > > in reply to this mail.
> > 
> > Committed.
> > 
> 
> Thanks. Sorry for not keeping up with the discussions, I was very busy
> last week.
> 
> Some comments:
> - I would have put the template in a separate file for better clarity
>   but I guess having it online in the eclass is more error-proof.

That would require establishing a bit of structure inside gx86 I'd
rather avoid doing myself. It would feel a bit weird to introduce
a directory just for one file.

> - Header wrapping really has nothing to do with autotools and I
>   strongly believe this should be moved to multilib-build.

It will be when there's a safe, public API for it. Right now, it's too
fragile.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [gentoo-dev] [PATCHES] Header wrapping support for multilib
  2013-04-02 11:47     ` Michał Górny
@ 2013-04-04  8:07       ` Alexis Ballier
  2013-04-04 17:53         ` Michał Górny
  0 siblings, 1 reply; 30+ messages in thread
From: Alexis Ballier @ 2013-04-04  8:07 UTC (permalink / raw
  To: gentoo-dev; +Cc: mgorny

On Tue, 2 Apr 2013 13:47:07 +0200
Michał Górny <mgorny@gentoo.org> wrote:
[...]
> > - Header wrapping really has nothing to do with autotools and I
> >   strongly believe this should be moved to multilib-build.
> 
> It will be when there's a safe, public API for it. Right now, it's too
> fragile.
> 

The problem is that at the moment it prevents non autotools{/,-utils}
based packages to do the wrapping. If _autotools-multilib_wrap_headers
were in multilib-build, it'd be trivial to add header wrapping support
in multilib-minimal.

What exactly is the unsafe part of the API of this function ? You could
very well make it functional by replacing MULTILIB_WRAPPED_HEADERS by
arguments to the function and then it'd do what it's supposed to: wrap
headers from its arguments into a multilib-safe header, move the unsafe
headers to "${ED}"/tmp/multilib-include ; if this function is called,
then multibuild_merge_root "${ED}"/tmp/multilib-include
"${ED}"/usr/include
_must_ be called at the end of src_install.

It's a weird API but after all it's not supposed to be used by
everyone ;)

Alexis.


^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [gentoo-dev] [PATCHES] Header wrapping support for multilib
  2013-04-04  8:07       ` Alexis Ballier
@ 2013-04-04 17:53         ` Michał Górny
  0 siblings, 0 replies; 30+ messages in thread
From: Michał Górny @ 2013-04-04 17:53 UTC (permalink / raw
  To: gentoo-dev; +Cc: aballier

[-- Attachment #1: Type: text/plain, Size: 1126 bytes --]

On Thu, 4 Apr 2013 10:07:06 +0200
Alexis Ballier <aballier@gentoo.org> wrote:

> > It will be when there's a safe, public API for it. Right now, it's too
> > fragile.
> 
> What exactly is the unsafe part of the API of this function ?

Well, the first thing was using $ABI directly, and I sent a patch for
that one already.

> You could
> very well make it functional by replacing MULTILIB_WRAPPED_HEADERS by
> arguments to the function and then it'd do what it's supposed to:

MULTILIB_WRAPPED_HEADERS will go public, I think. It's kind of nice to
have a single way to list them.

> wrap
> headers from its arguments into a multilib-safe header, move the unsafe
> headers to "${ED}"/tmp/multilib-include ; if this function is called,
> then multibuild_merge_root "${ED}"/tmp/multilib-include
> "${ED}"/usr/include
> _must_ be called at the end of src_install.

That part is fragile. I'd rather keep both steps encapsulated into two
functions. Especially that if we decide to support wrapping headers
outside /usr/include, the latter part would have to change.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

^ permalink raw reply	[flat|nested] 30+ messages in thread

end of thread, other threads:[~2013-04-04 18:08 UTC | newest]

Thread overview: 30+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-03-23 16:25 [gentoo-dev] [PATCHES] Header wrapping support for multilib Michał Górny
2013-03-23 16:26 ` [gentoo-dev] [PATCH 1/2] Introduce multibuild_merge_root() to merge interim installs Michał Górny
2013-03-23 17:00   ` Ulrich Mueller
2013-03-23 17:28     ` Michał Górny
2013-03-24  7:47       ` Ulrich Mueller
2013-03-24 10:09         ` Michał Górny
2013-03-24 13:40           ` Ulrich Mueller
2013-03-24 14:13             ` Michał Górny
2013-03-24 20:18               ` Zac Medico
2013-03-25 22:42     ` [gentoo-dev] [PATCH] " Michał Górny
2013-03-23 17:44   ` [gentoo-dev] [PATCH 1/2] " Alec Warner
2013-03-23 18:57     ` Michał Górny
2013-03-23 18:57     ` [gentoo-dev] " Jonathan Callen
2013-03-23 19:02       ` Alec Warner
2013-03-23 16:26 ` [gentoo-dev] [PATCH 2/2] Support wrapping headers for multilib ABIs Michał Górny
2013-03-24 15:14   ` Alexis Ballier
2013-03-24 15:41     ` Michał Górny
2013-03-24 20:01       ` Alexis Ballier
2013-03-25 22:22         ` [gentoo-dev] [PATCH] " Michał Górny
2013-03-23 16:46 ` [gentoo-dev] [PATCHES] Header wrapping support for multilib Diego Elio Pettenò
2013-03-23 17:32   ` Michał Górny
2013-03-23 17:32     ` Diego Elio Pettenò
2013-03-23 19:56   ` [gentoo-dev] [PATCH] Support wrapping headers for multilib ABIs Michał Górny
2013-03-23 20:03     ` Michał Górny
2013-03-23 22:01     ` [gentoo-dev] " Jonathan Callen
2013-04-01  9:19 ` [gentoo-dev] [PATCHES] Header wrapping support for multilib Michał Górny
2013-04-02 10:59   ` Alexis Ballier
2013-04-02 11:47     ` Michał Górny
2013-04-04  8:07       ` Alexis Ballier
2013-04-04 17:53         ` Michał Górny

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox