[gentoo-dev] [PATCH 1/2] dev-lang/jimtcl: Remove parallel econf code

[gentoo-dev] [PATCH 1/2] dev-lang/jimtcl: Remove parallel econf code

[Michał Górny]

The parallel econf code is used only with USE=static-libs, and even in
that case provides negligible speed gain. At the same time, it adds
a lot of complexity, causes the build logs to be unreadable mix of
output from both configure scripts and violates PMS by calling econf
in parallel which can cause issues with different package managers.
---
 dev-lang/jimtcl/jimtcl-0.75-r1.ebuild | 10 ++++------
 dev-lang/jimtcl/jimtcl-0.76.ebuild    | 10 ++++------
 dev-lang/jimtcl/jimtcl-9999.ebuild    | 10 ++++------
 3 files changed, 12 insertions(+), 18 deletions(-)

diff --git a/dev-lang/jimtcl/jimtcl-0.75-r1.ebuild b/dev-lang/jimtcl/jimtcl-0.75-r1.ebuild
index d05b5040644f..180745bbe7da 100644
--- a/dev-lang/jimtcl/jimtcl-0.75-r1.ebuild
+++ b/dev-lang/jimtcl/jimtcl-0.75-r1.ebuild
@@ -1,9 +1,9 @@
-# Copyright 1999-2014 Gentoo Foundation
+# Copyright 1999-2017 Gentoo Foundation
 # Distributed under the terms of the GNU General Public License v2
 
 EAPI="5"
 
-inherit multiprocessing eutils
+inherit eutils
 
 if [[ ${PV} == "9999" ]] ; then
 	EGIT_REPO_URI="http://repo.or.cz/r/jimtcl.git"
@@ -40,15 +40,13 @@ src_prepare() {
 }
 
 src_configure() {
-	multijob_init
-	CCACHE=None multijob_child_init econf --with-jim-shared
+	CCACHE=None econf --with-jim-shared
 	if use static-libs ; then
 		# The build does not support doing both simultaneously.
 		mkdir static-libs
 		cd static-libs
-		CCACHE=None ECONF_SOURCE=${S} multijob_child_init econf
+		CCACHE=None ECONF_SOURCE=${S} econf
 	fi
-	multijob_finish
 }
 
 src_compile() {
diff --git a/dev-lang/jimtcl/jimtcl-0.76.ebuild b/dev-lang/jimtcl/jimtcl-0.76.ebuild
index a82e24c9b204..ecdcdc96fac5 100644
--- a/dev-lang/jimtcl/jimtcl-0.76.ebuild
+++ b/dev-lang/jimtcl/jimtcl-0.76.ebuild
@@ -1,9 +1,9 @@
-# Copyright 1999-2015 Gentoo Foundation
+# Copyright 1999-2017 Gentoo Foundation
 # Distributed under the terms of the GNU General Public License v2
 
 EAPI="5"
 
-inherit multiprocessing eutils
+inherit eutils
 
 if [[ ${PV} == "9999" ]] ; then
 	EGIT_REPO_URI="http://repo.or.cz/r/jimtcl.git"
@@ -40,15 +40,13 @@ src_prepare() {
 }
 
 src_configure() {
-	multijob_init
-	CCACHE=None multijob_child_init econf --with-jim-shared
+	CCACHE=None econf --with-jim-shared
 	if use static-libs ; then
 		# The build does not support doing both simultaneously.
 		mkdir static-libs
 		cd static-libs
-		CCACHE=None ECONF_SOURCE=${S} multijob_child_init econf
+		CCACHE=None ECONF_SOURCE=${S} econf
 	fi
-	multijob_finish
 }
 
 src_compile() {
diff --git a/dev-lang/jimtcl/jimtcl-9999.ebuild b/dev-lang/jimtcl/jimtcl-9999.ebuild
index d05b5040644f..180745bbe7da 100644
--- a/dev-lang/jimtcl/jimtcl-9999.ebuild
+++ b/dev-lang/jimtcl/jimtcl-9999.ebuild
@@ -1,9 +1,9 @@
-# Copyright 1999-2014 Gentoo Foundation
+# Copyright 1999-2017 Gentoo Foundation
 # Distributed under the terms of the GNU General Public License v2
 
 EAPI="5"
 
-inherit multiprocessing eutils
+inherit eutils
 
 if [[ ${PV} == "9999" ]] ; then
 	EGIT_REPO_URI="http://repo.or.cz/r/jimtcl.git"
@@ -40,15 +40,13 @@ src_prepare() {
 }
 
 src_configure() {
-	multijob_init
-	CCACHE=None multijob_child_init econf --with-jim-shared
+	CCACHE=None econf --with-jim-shared
 	if use static-libs ; then
 		# The build does not support doing both simultaneously.
 		mkdir static-libs
 		cd static-libs
-		CCACHE=None ECONF_SOURCE=${S} multijob_child_init econf
+		CCACHE=None ECONF_SOURCE=${S} econf
 	fi
-	multijob_finish
 }
 
 src_compile() {
-- 
2.12.0

[gentoo-dev] [PATCH 2/2] sys-libs/ncurses: Remove parallel econf logic

[Michał Górny]

Remove the parallel econf logic that adds a lot of complexity for minor
gain. It results in the output from different configure scripts being
mixed in the build log, making it unreadable. It causes econf to be run
in a subshell which is a PMS violation and can cause issues with some of
package manager implementations.  Furthermore, the multijob parallel
processes are interleaved with multilib-build logic which is unsupported
and a very bad idea.
---
 sys-libs/ncurses/ncurses-5.9-r101.ebuild | 11 ++++-------
 sys-libs/ncurses/ncurses-6.0-r1.ebuild   |  9 +++------
 2 files changed, 7 insertions(+), 13 deletions(-)

diff --git a/sys-libs/ncurses/ncurses-5.9-r101.ebuild b/sys-libs/ncurses/ncurses-5.9-r101.ebuild
index 76b8a76d3a72..c722d68f3208 100644
--- a/sys-libs/ncurses/ncurses-5.9-r101.ebuild
+++ b/sys-libs/ncurses/ncurses-5.9-r101.ebuild
@@ -1,11 +1,11 @@
-# Copyright 1999-2015 Gentoo Foundation
+# Copyright 1999-2017 Gentoo Foundation
 # Distributed under the terms of the GNU General Public License v2
 
 # This version is just for the ABI .5 library
 
 EAPI="5"
 
-inherit eutils toolchain-funcs multilib-minimal multiprocessing
+inherit eutils toolchain-funcs multilib-minimal
 
 MY_PV=${PV:0:3}
 MY_P=${PN}-${MY_PV}
@@ -55,8 +55,6 @@ src_configure() {
 		$(usex unicode 'ncursesw' '')
 	)
 
-	multijob_init
-
 	# When installing ncurses, we have to use a compatible version of tic.
 	# This comes up when cross-compiling, doing multilib builds, upgrading,
 	# or installing for the first time.  Build a local copy of tic whenever
@@ -69,16 +67,15 @@ src_configure() {
 		CXXFLAGS=${BUILD_CXXFLAGS} \
 		CPPFLAGS=${BUILD_CPPFLAGS} \
 		LDFLAGS="${BUILD_LDFLAGS} -static" \
-		multijob_child_init do_configure cross --without-shared --with-normal
+		do_configure cross --without-shared --with-normal
 	fi
 	multilib-minimal_src_configure
-	multijob_finish
 }
 
 multilib_src_configure() {
 	local t
 	for t in "${NCURSES_TARGETS[@]}" ; do
-		multijob_child_init do_configure "${t}"
+		do_configure "${t}"
 	done
 }
 
diff --git a/sys-libs/ncurses/ncurses-6.0-r1.ebuild b/sys-libs/ncurses/ncurses-6.0-r1.ebuild
index 2ab63de41d4c..d98b23afb751 100644
--- a/sys-libs/ncurses/ncurses-6.0-r1.ebuild
+++ b/sys-libs/ncurses/ncurses-6.0-r1.ebuild
@@ -3,7 +3,7 @@
 
 EAPI="5"
 
-inherit eutils flag-o-matic toolchain-funcs multilib-minimal multiprocessing
+inherit eutils flag-o-matic toolchain-funcs multilib-minimal
 
 MY_PV=${PV:0:3}
 PV_SNAP=${PV:4}
@@ -61,8 +61,6 @@ src_configure() {
 		$(use unicode && usex threads 'ncursestw' '')
 	)
 
-	multijob_init
-
 	# When installing ncurses, we have to use a compatible version of tic.
 	# This comes up when cross-compiling, doing multilib builds, upgrading,
 	# or installing for the first time.  Build a local copy of tic whenever
@@ -87,16 +85,15 @@ src_configure() {
 		CXXFLAGS=${BUILD_CXXFLAGS} \
 		CPPFLAGS=${BUILD_CPPFLAGS} \
 		LDFLAGS="${BUILD_LDFLAGS} ${lbuildflags}" \
-		multijob_child_init do_configure cross --without-shared --with-normal
+		do_configure cross --without-shared --with-normal
 	fi
 	multilib-minimal_src_configure
-	multijob_finish
 }
 
 multilib_src_configure() {
 	local t
 	for t in "${NCURSES_TARGETS[@]}" ; do
-		multijob_child_init do_configure "${t}"
+		do_configure "${t}"
 	done
 }
 
-- 
2.12.0

Re: [gentoo-dev] [PATCH 1/2] dev-lang/jimtcl: Remove parallel econf code

[Sergei Trofimovich]

[-- Attachment #1: Type: text/plain, Size: 3943 bytes --]

On Mon, 20 Mar 2017 20:35:51 +0100
Michał Górny <mgorny@gentoo.org> wrote:

CCing maintainer

> The parallel econf code is used only with USE=static-libs, and even in
> that case provides negligible speed gain. At the same time, it adds
> a lot of complexity, causes the build logs to be unreadable mix of
> output from both configure scripts and violates PMS by calling econf
> in parallel which can cause issues with different package managers.
> ---
>  dev-lang/jimtcl/jimtcl-0.75-r1.ebuild | 10 ++++------
>  dev-lang/jimtcl/jimtcl-0.76.ebuild    | 10 ++++------
>  dev-lang/jimtcl/jimtcl-9999.ebuild    | 10 ++++------
>  3 files changed, 12 insertions(+), 18 deletions(-)
> 
> diff --git a/dev-lang/jimtcl/jimtcl-0.75-r1.ebuild b/dev-lang/jimtcl/jimtcl-0.75-r1.ebuild
> index d05b5040644f..180745bbe7da 100644
> --- a/dev-lang/jimtcl/jimtcl-0.75-r1.ebuild
> +++ b/dev-lang/jimtcl/jimtcl-0.75-r1.ebuild
> @@ -1,9 +1,9 @@
> -# Copyright 1999-2014 Gentoo Foundation
> +# Copyright 1999-2017 Gentoo Foundation
>  # Distributed under the terms of the GNU General Public License v2
>  
>  EAPI="5"
>  
> -inherit multiprocessing eutils
> +inherit eutils
>  
>  if [[ ${PV} == "9999" ]] ; then
>  	EGIT_REPO_URI="http://repo.or.cz/r/jimtcl.git"
> @@ -40,15 +40,13 @@ src_prepare() {
>  }
>  
>  src_configure() {
> -	multijob_init
> -	CCACHE=None multijob_child_init econf --with-jim-shared
> +	CCACHE=None econf --with-jim-shared
>  	if use static-libs ; then
>  		# The build does not support doing both simultaneously.
>  		mkdir static-libs
>  		cd static-libs
> -		CCACHE=None ECONF_SOURCE=${S} multijob_child_init econf
> +		CCACHE=None ECONF_SOURCE=${S} econf
>  	fi
> -	multijob_finish
>  }
>  
>  src_compile() {
> diff --git a/dev-lang/jimtcl/jimtcl-0.76.ebuild b/dev-lang/jimtcl/jimtcl-0.76.ebuild
> index a82e24c9b204..ecdcdc96fac5 100644
> --- a/dev-lang/jimtcl/jimtcl-0.76.ebuild
> +++ b/dev-lang/jimtcl/jimtcl-0.76.ebuild
> @@ -1,9 +1,9 @@
> -# Copyright 1999-2015 Gentoo Foundation
> +# Copyright 1999-2017 Gentoo Foundation
>  # Distributed under the terms of the GNU General Public License v2
>  
>  EAPI="5"
>  
> -inherit multiprocessing eutils
> +inherit eutils
>  
>  if [[ ${PV} == "9999" ]] ; then
>  	EGIT_REPO_URI="http://repo.or.cz/r/jimtcl.git"
> @@ -40,15 +40,13 @@ src_prepare() {
>  }
>  
>  src_configure() {
> -	multijob_init
> -	CCACHE=None multijob_child_init econf --with-jim-shared
> +	CCACHE=None econf --with-jim-shared
>  	if use static-libs ; then
>  		# The build does not support doing both simultaneously.
>  		mkdir static-libs
>  		cd static-libs
> -		CCACHE=None ECONF_SOURCE=${S} multijob_child_init econf
> +		CCACHE=None ECONF_SOURCE=${S} econf
>  	fi
> -	multijob_finish
>  }
>  
>  src_compile() {
> diff --git a/dev-lang/jimtcl/jimtcl-9999.ebuild b/dev-lang/jimtcl/jimtcl-9999.ebuild
> index d05b5040644f..180745bbe7da 100644
> --- a/dev-lang/jimtcl/jimtcl-9999.ebuild
> +++ b/dev-lang/jimtcl/jimtcl-9999.ebuild
> @@ -1,9 +1,9 @@
> -# Copyright 1999-2014 Gentoo Foundation
> +# Copyright 1999-2017 Gentoo Foundation
>  # Distributed under the terms of the GNU General Public License v2
>  
>  EAPI="5"
>  
> -inherit multiprocessing eutils
> +inherit eutils
>  
>  if [[ ${PV} == "9999" ]] ; then
>  	EGIT_REPO_URI="http://repo.or.cz/r/jimtcl.git"
> @@ -40,15 +40,13 @@ src_prepare() {
>  }
>  
>  src_configure() {
> -	multijob_init
> -	CCACHE=None multijob_child_init econf --with-jim-shared
> +	CCACHE=None econf --with-jim-shared
>  	if use static-libs ; then
>  		# The build does not support doing both simultaneously.
>  		mkdir static-libs
>  		cd static-libs
> -		CCACHE=None ECONF_SOURCE=${S} multijob_child_init econf
> +		CCACHE=None ECONF_SOURCE=${S} econf
>  	fi
> -	multijob_finish
>  }
>  
>  src_compile() {
> -- 
> 2.12.0
> 
> 


-- 

  Sergei

[-- Attachment #2: Цифровая подпись OpenPGP --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] [PATCH 2/2] sys-libs/ncurses: Remove parallel econf logic

[Sergei Trofimovich]

[-- Attachment #1: Type: text/plain, Size: 3732 bytes --]

On Mon, 20 Mar 2017 20:35:52 +0100
Michał Górny <mgorny@gentoo.org> wrote:

CCing maintainer

> Remove the parallel econf logic that adds a lot of complexity for minor
> gain. It results in the output from different configure scripts being
> mixed in the build log, making it unreadable. It causes econf to be run
> in a subshell which is a PMS violation and can cause issues with some of
> package manager implementations.  Furthermore, the multijob parallel
> processes are interleaved with multilib-build logic which is unsupported
> and a very bad idea.
> ---
>  sys-libs/ncurses/ncurses-5.9-r101.ebuild | 11 ++++-------
>  sys-libs/ncurses/ncurses-6.0-r1.ebuild   |  9 +++------
>  2 files changed, 7 insertions(+), 13 deletions(-)
> 
> diff --git a/sys-libs/ncurses/ncurses-5.9-r101.ebuild b/sys-libs/ncurses/ncurses-5.9-r101.ebuild
> index 76b8a76d3a72..c722d68f3208 100644
> --- a/sys-libs/ncurses/ncurses-5.9-r101.ebuild
> +++ b/sys-libs/ncurses/ncurses-5.9-r101.ebuild
> @@ -1,11 +1,11 @@
> -# Copyright 1999-2015 Gentoo Foundation
> +# Copyright 1999-2017 Gentoo Foundation
>  # Distributed under the terms of the GNU General Public License v2
>  
>  # This version is just for the ABI .5 library
>  
>  EAPI="5"
>  
> -inherit eutils toolchain-funcs multilib-minimal multiprocessing
> +inherit eutils toolchain-funcs multilib-minimal
>  
>  MY_PV=${PV:0:3}
>  MY_P=${PN}-${MY_PV}
> @@ -55,8 +55,6 @@ src_configure() {
>  		$(usex unicode 'ncursesw' '')
>  	)
>  
> -	multijob_init
> -
>  	# When installing ncurses, we have to use a compatible version of tic.
>  	# This comes up when cross-compiling, doing multilib builds, upgrading,
>  	# or installing for the first time.  Build a local copy of tic whenever
> @@ -69,16 +67,15 @@ src_configure() {
>  		CXXFLAGS=${BUILD_CXXFLAGS} \
>  		CPPFLAGS=${BUILD_CPPFLAGS} \
>  		LDFLAGS="${BUILD_LDFLAGS} -static" \
> -		multijob_child_init do_configure cross --without-shared --with-normal
> +		do_configure cross --without-shared --with-normal
>  	fi
>  	multilib-minimal_src_configure
> -	multijob_finish
>  }
>  
>  multilib_src_configure() {
>  	local t
>  	for t in "${NCURSES_TARGETS[@]}" ; do
> -		multijob_child_init do_configure "${t}"
> +		do_configure "${t}"
>  	done
>  }
>  
> diff --git a/sys-libs/ncurses/ncurses-6.0-r1.ebuild b/sys-libs/ncurses/ncurses-6.0-r1.ebuild
> index 2ab63de41d4c..d98b23afb751 100644
> --- a/sys-libs/ncurses/ncurses-6.0-r1.ebuild
> +++ b/sys-libs/ncurses/ncurses-6.0-r1.ebuild
> @@ -3,7 +3,7 @@
>  
>  EAPI="5"
>  
> -inherit eutils flag-o-matic toolchain-funcs multilib-minimal multiprocessing
> +inherit eutils flag-o-matic toolchain-funcs multilib-minimal
>  
>  MY_PV=${PV:0:3}
>  PV_SNAP=${PV:4}
> @@ -61,8 +61,6 @@ src_configure() {
>  		$(use unicode && usex threads 'ncursestw' '')
>  	)
>  
> -	multijob_init
> -
>  	# When installing ncurses, we have to use a compatible version of tic.
>  	# This comes up when cross-compiling, doing multilib builds, upgrading,
>  	# or installing for the first time.  Build a local copy of tic whenever
> @@ -87,16 +85,15 @@ src_configure() {
>  		CXXFLAGS=${BUILD_CXXFLAGS} \
>  		CPPFLAGS=${BUILD_CPPFLAGS} \
>  		LDFLAGS="${BUILD_LDFLAGS} ${lbuildflags}" \
> -		multijob_child_init do_configure cross --without-shared --with-normal
> +		do_configure cross --without-shared --with-normal
>  	fi
>  	multilib-minimal_src_configure
> -	multijob_finish
>  }
>  
>  multilib_src_configure() {
>  	local t
>  	for t in "${NCURSES_TARGETS[@]}" ; do
> -		multijob_child_init do_configure "${t}"
> +		do_configure "${t}"
>  	done
>  }
>  
> -- 
> 2.12.0
> 
> 


-- 

  Sergei

[-- Attachment #2: Цифровая подпись OpenPGP --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] [PATCH 2/2] sys-libs/ncurses: Remove parallel econf logic

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 627 bytes --]

On 20 Mar 2017 20:35, Michał Górny wrote:
> Remove the parallel econf logic that adds a lot of complexity for minor
> gain. It results in the output from different configure scripts being
> mixed in the build log, making it unreadable. It causes econf to be run
> in a subshell which is a PMS violation and can cause issues with some of
> package manager implementations.  Furthermore, the multijob parallel
> processes are interleaved with multilib-build logic which is unsupported
> and a very bad idea.

NAK.  you still haven't backed up your claims wrt to PMS, and this slows
things down significantly.
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] [PATCH 2/2] sys-libs/ncurses: Remove parallel econf logic

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2636 bytes --]

On pon, 2017-03-20 at 21:12 -0400, Mike Frysinger wrote:
> On 20 Mar 2017 20:35, Michał Górny wrote:
> > Remove the parallel econf logic that adds a lot of complexity for minor
> > gain. It results in the output from different configure scripts being
> > mixed in the build log, making it unreadable. It causes econf to be run
> > in a subshell which is a PMS violation and can cause issues with some of
> > package manager implementations.  Furthermore, the multijob parallel
> > processes are interleaved with multilib-build logic which is unsupported
> > and a very bad idea.
> 
> NAK.  you still haven't backed up your claims wrt to PMS, and this slows
> things down significantly.

You have found the appropriate PMS bit yourself, and chose to ignore it.
What can I do about that? How am I going to convince you that '*not*
guaranteed to work correctly if called from a subshell environment'
means you're not supposed to do that? If I submit a PMS patch changing
the wording to state the obvious you're going to accuse me once again of
changing PMS wording to target you.

Furthermore, I have linked an *explicit Portage bug* causing breakage
with your code. However, you've presumed that it's fine as long as
it will soon be fixed in the git version of Portage.

Finally, I should point out that PMS was *not* written with parallel
execution in mind. It was not deemed useful. It states no requirements
on parallel run guarantees, and so you can't expect package managers to
implement those commands in parallel-friendly way. It's bigger than you.

How about we turn things around, and start expecting maintainers to
justify their non-standard solutions instead? Please tell me, do you
think every other Gentoo developer is wrong not to do parallel econf? Do
you think I was stupid that I've removed that can of worms out of
multibuild?

Could you back your 'significant slow down' claim with any data? Here's
my results, MAKEOPTS=-j3 USE='cxx debug threads tinfo unicode'
ABI_X86='32 64', warm cache. The whole build takes around 25 minutes,
the configure phase (as measures by 'ebuild ... prepare; time ebuild ...
configure':

parallel: 2m35s

serial: 3m47s

We can dispute whether 1 minute gain out of 25 is meaningful. But
instead, I'd like to propose a better solution -- to use --cache-file=
to avoid redoing the same checks in subsequent runs.

cached: 2m24s

...which proves it's faster than the parallel solution, and has
the major advantage of saving both CPU and wall clock time. Any reason
not do that? A patch will follow.

-- 
Best regards,
Michał Górny

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 963 bytes --]