public inbox for gentoo-dev@lists.gentoo.org
 help / color / mirror / Atom feed
* [gentoo-dev] [PATCH 1/8] kernel-2.eclass: decompress xz in parallel
@ 2022-10-11 22:54 Sam James
  2022-10-11 22:54 ` [gentoo-dev] [PATCH 2/8] linux-mod.eclass: compress xz/zstd " Sam James
                   ` (6 more replies)
  0 siblings, 7 replies; 15+ messages in thread
From: Sam James @ 2022-10-11 22:54 UTC (permalink / raw
  To: gentoo-dev; +Cc: dist-kernel, kernel, Sam James

No-op until >=app-arch/xz-utils-5.3.3_alpha. Recently added support
for this in Portage and pkgcore too for unpacking.

Signed-off-by: Sam James <sam@gentoo.org>
---
 eclass/kernel-2.eclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/eclass/kernel-2.eclass b/eclass/kernel-2.eclass
index 50464470a9bb..c18ef21099d1 100644
--- a/eclass/kernel-2.eclass
+++ b/eclass/kernel-2.eclass
@@ -281,7 +281,7 @@
 # If you do change them, there is a chance that we will not fix resulting bugs;
 # that of course does not mean we're not willing to help.
 
-inherit estack toolchain-funcs
+inherit estack multiprocessing toolchain-funcs
 
 case ${EAPI} in
 	7|8) ;;
@@ -1065,7 +1065,7 @@ unipatch() {
 			extention=${extention/:*/}
 			PIPE_CMD=""
 			case ${extention} in
-				     xz) PIPE_CMD="xz -dc";;
+				     xz) PIPE_CMD="xz -T$(makeopts_jobs) -dc";;
 				   lzma) PIPE_CMD="lzma -dc";;
 				    bz2) PIPE_CMD="bzip2 -dc";;
 				 patch*) PIPE_CMD="cat";;
-- 
2.38.0



^ permalink raw reply related	[flat|nested] 15+ messages in thread

* [gentoo-dev] [PATCH 2/8] linux-mod.eclass: compress xz/zstd in parallel
  2022-10-11 22:54 [gentoo-dev] [PATCH 1/8] kernel-2.eclass: decompress xz in parallel Sam James
@ 2022-10-11 22:54 ` Sam James
  2022-10-12  8:27   ` Ulrich Mueller
  2022-10-11 22:54 ` [gentoo-dev] [PATCH 3/8] linux-mod.eclass: use pigz for parallel compression if available Sam James
                   ` (5 subsequent siblings)
  6 siblings, 1 reply; 15+ messages in thread
From: Sam James @ 2022-10-11 22:54 UTC (permalink / raw
  To: gentoo-dev; +Cc: dist-kernel, kernel, Sam James

Signed-off-by: Sam James <sam@gentoo.org>
---
 eclass/linux-mod.eclass | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/eclass/linux-mod.eclass b/eclass/linux-mod.eclass
index b7c13cbf7e76..8f511ba8ff05 100644
--- a/eclass/linux-mod.eclass
+++ b/eclass/linux-mod.eclass
@@ -163,7 +163,7 @@ _LINUX_MOD_ECLASS=1
 
 # TODO: When adding support for future EAPIs, please audit this list
 # for unused inherits and conditionalise them.
-inherit linux-info multilib toolchain-funcs
+inherit linux-info multilib multiprocessing toolchain-funcs
 
 case ${MODULES_OPTIONAL_USE_IUSE_DEFAULT:-n} in
   [nNfF]*|[oO][fF]*|0|-) _modules_optional_use_iuse_default='' ;;
@@ -712,17 +712,17 @@ linux-mod_src_install() {
 		cd "${objdir}" || die "${objdir} does not exist"
 		insinto "${INSTALL_MOD_PATH}"/lib/modules/${KV_FULL}/${libdir}
 
-		# check here for CONFIG_MODULE_COMPRESS_<compression option> (NONE, GZIP, XZ, ZSTD) 
+		# check here for CONFIG_MODULE_COMPRESS_<compression option> (NONE, GZIP, XZ, ZSTD)
 		# and similarily compress the module being built if != NONE.
 
 		if linux_chkconfig_present MODULE_COMPRESS_XZ; then
-			xz ${modulename}.${KV_OBJ} 
+			xz -T$(makeopts_jobs) ${modulename}.${KV_OBJ}
 			doins ${modulename}.${KV_OBJ}.xz || die "doins ${modulename}.${KV_OBJ}.xz failed"
 		elif linux_chkconfig_present MODULE_COMPRESS_GZIP; then
 			gzip ${modulename}.${KV_OBJ}
 			doins ${modulename}.${KV_OBJ}.gz || die "doins ${modulename}.${KV_OBJ}.gz failed"
 		elif linux_chkconfig_present MODULE_COMPRESS_ZSTD; then
-			zstd ${modulename}.${KV_OBJ}
+			zstd -T$(makeopts_jobs) ${modulename}.${KV_OBJ}
 			doins ${modulename}.${KV_OBJ}.zst || die "doins ${modulename}.${KV_OBJ}.zst failed"
 		else
 			doins ${modulename}.${KV_OBJ} || die "doins ${modulename}.${KV_OBJ} failed"
-- 
2.38.0



^ permalink raw reply related	[flat|nested] 15+ messages in thread

* [gentoo-dev] [PATCH 3/8] linux-mod.eclass: use pigz for parallel compression if available
  2022-10-11 22:54 [gentoo-dev] [PATCH 1/8] kernel-2.eclass: decompress xz in parallel Sam James
  2022-10-11 22:54 ` [gentoo-dev] [PATCH 2/8] linux-mod.eclass: compress xz/zstd " Sam James
@ 2022-10-11 22:54 ` Sam James
  2022-10-11 22:54 ` [gentoo-dev] [PATCH 4/8] linux-mod.eclass: improve error handling for module compression Sam James
                   ` (4 subsequent siblings)
  6 siblings, 0 replies; 15+ messages in thread
From: Sam James @ 2022-10-11 22:54 UTC (permalink / raw
  To: gentoo-dev; +Cc: dist-kernel, kernel, Sam James

Signed-off-by: Sam James <sam@gentoo.org>
---
 eclass/linux-mod.eclass | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/eclass/linux-mod.eclass b/eclass/linux-mod.eclass
index 8f511ba8ff05..e837916b881a 100644
--- a/eclass/linux-mod.eclass
+++ b/eclass/linux-mod.eclass
@@ -719,7 +719,11 @@ linux-mod_src_install() {
 			xz -T$(makeopts_jobs) ${modulename}.${KV_OBJ}
 			doins ${modulename}.${KV_OBJ}.xz || die "doins ${modulename}.${KV_OBJ}.xz failed"
 		elif linux_chkconfig_present MODULE_COMPRESS_GZIP; then
-			gzip ${modulename}.${KV_OBJ}
+			if type -P pigz ; then
+				pigz -n$(makeopts_jobs) ${modulename}.${KV_OBJ}
+			else
+				gzip ${modulename}.${KV_OBJ}
+			fi
 			doins ${modulename}.${KV_OBJ}.gz || die "doins ${modulename}.${KV_OBJ}.gz failed"
 		elif linux_chkconfig_present MODULE_COMPRESS_ZSTD; then
 			zstd -T$(makeopts_jobs) ${modulename}.${KV_OBJ}
-- 
2.38.0



^ permalink raw reply related	[flat|nested] 15+ messages in thread

* [gentoo-dev] [PATCH 4/8] linux-mod.eclass: improve error handling for module compression
  2022-10-11 22:54 [gentoo-dev] [PATCH 1/8] kernel-2.eclass: decompress xz in parallel Sam James
  2022-10-11 22:54 ` [gentoo-dev] [PATCH 2/8] linux-mod.eclass: compress xz/zstd " Sam James
  2022-10-11 22:54 ` [gentoo-dev] [PATCH 3/8] linux-mod.eclass: use pigz for parallel compression if available Sam James
@ 2022-10-11 22:54 ` Sam James
  2022-10-11 22:54 ` [gentoo-dev] [PATCH 5/8] linux-mod.eclass: cleanup whitespace Sam James
                   ` (3 subsequent siblings)
  6 siblings, 0 replies; 15+ messages in thread
From: Sam James @ 2022-10-11 22:54 UTC (permalink / raw
  To: gentoo-dev; +Cc: dist-kernel, kernel, Sam James

doins will 'die for us' in newer EAPIs, and add missing || dies to compressor calls.

Signed-off-by: Sam James <sam@gentoo.org>
---
 eclass/linux-mod.eclass | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/eclass/linux-mod.eclass b/eclass/linux-mod.eclass
index e837916b881a..e4c71cdf2c15 100644
--- a/eclass/linux-mod.eclass
+++ b/eclass/linux-mod.eclass
@@ -716,22 +716,22 @@ linux-mod_src_install() {
 		# and similarily compress the module being built if != NONE.
 
 		if linux_chkconfig_present MODULE_COMPRESS_XZ; then
-			xz -T$(makeopts_jobs) ${modulename}.${KV_OBJ}
-			doins ${modulename}.${KV_OBJ}.xz || die "doins ${modulename}.${KV_OBJ}.xz failed"
+			xz -T$(makeopts_jobs) ${modulename}.${KV_OBJ} || die "Compressing ${modulename}.${KV_OBJ} with xz failed"
+			doins ${modulename}.${KV_OBJ}.xz
 		elif linux_chkconfig_present MODULE_COMPRESS_GZIP; then
 			if type -P pigz ; then
-				pigz -n$(makeopts_jobs) ${modulename}.${KV_OBJ}
+				pigz -n$(makeopts_jobs) ${modulename}.${KV_OBJ} || die "Compressing ${modulename}.${KV_OBJ} with pigz failed"
 			else
-				gzip ${modulename}.${KV_OBJ}
+				gzip ${modulename}.${KV_OBJ} || die "Compressing ${modulename}.${KV_OBJ} with gzip failed"
 			fi
-			doins ${modulename}.${KV_OBJ}.gz || die "doins ${modulename}.${KV_OBJ}.gz failed"
+			doins ${modulename}.${KV_OBJ}.gz
 		elif linux_chkconfig_present MODULE_COMPRESS_ZSTD; then
-			zstd -T$(makeopts_jobs) ${modulename}.${KV_OBJ}
-			doins ${modulename}.${KV_OBJ}.zst || die "doins ${modulename}.${KV_OBJ}.zst failed"
+			zstd -T$(makeopts_jobs) ${modulename}.${KV_OBJ} || "Compressing ${modulename}.${KV_OBJ} with zstd failed"
+			doins ${modulename}.${KV_OBJ}.zst
 		else
-			doins ${modulename}.${KV_OBJ} || die "doins ${modulename}.${KV_OBJ} failed"
+			doins ${modulename}.${KV_OBJ}
 		fi
-		cd "${OLDPWD}"
+		cd "${OLDPWD}" || die "${OLDPWD} does not exist"
 
 		generate_modulesd "${objdir}/${modulename}"
 	done
-- 
2.38.0



^ permalink raw reply related	[flat|nested] 15+ messages in thread

* [gentoo-dev] [PATCH 5/8] linux-mod.eclass: cleanup whitespace
  2022-10-11 22:54 [gentoo-dev] [PATCH 1/8] kernel-2.eclass: decompress xz in parallel Sam James
                   ` (2 preceding siblings ...)
  2022-10-11 22:54 ` [gentoo-dev] [PATCH 4/8] linux-mod.eclass: improve error handling for module compression Sam James
@ 2022-10-11 22:54 ` Sam James
  2022-10-11 22:54 ` [gentoo-dev] [PATCH 6/8] kernel-build.eclass: compress xz/zstd in parallel Sam James
                   ` (2 subsequent siblings)
  6 siblings, 0 replies; 15+ messages in thread
From: Sam James @ 2022-10-11 22:54 UTC (permalink / raw
  To: gentoo-dev; +Cc: dist-kernel, kernel, Sam James

Signed-off-by: Sam James <sam@gentoo.org>
---
 eclass/linux-mod.eclass | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/eclass/linux-mod.eclass b/eclass/linux-mod.eclass
index e4c71cdf2c15..7f4ddb9b8ee6 100644
--- a/eclass/linux-mod.eclass
+++ b/eclass/linux-mod.eclass
@@ -150,8 +150,8 @@
 # It's a read-only variable. It contains the extension of the kernel modules.
 
 case ${EAPI:-0} in
-	[67]) 
-		inherit eutils 
+	[67])
+		inherit eutils
 		;;
 	8)
 		;;
@@ -361,8 +361,8 @@ generate_modulesd() {
 	debug-print-function ${FUNCNAME} $*
 	[ -n "${MODULES_OPTIONAL_USE}" ] && use !${MODULES_OPTIONAL_USE} && return
 
-	local 	currm_path currm currm_t t myIFS myVAR
-	local 	module_docs module_enabled module_aliases \
+	local currm_path currm currm_t t myIFS myVAR
+	local module_docs module_enabled module_aliases \
 			module_additions module_examples module_modinfo module_opts
 
 	for currm_path in ${@}
@@ -380,9 +380,9 @@ generate_modulesd() {
 		module_additions="$(eval echo \${#MODULESD_${currm_t}_ADDITIONS[*]})"
 		module_examples="$(eval echo \${#MODULESD_${currm_t}_EXAMPLES[*]})"
 
-		[[ ${module_aliases} -eq 0 ]] 	&& unset module_aliases
+		[[ ${module_aliases} -eq 0 ]]	&& unset module_aliases
 		[[ ${module_additions} -eq 0 ]]	&& unset module_additions
-		[[ ${module_examples} -eq 0 ]] 	&& unset module_examples
+		[[ ${module_examples} -eq 0 ]]  && unset module_examples
 
 		# If we specify we dont want it, then lets exit, otherwise we assume
 		# that if its set, we do want it.
-- 
2.38.0



^ permalink raw reply related	[flat|nested] 15+ messages in thread

* [gentoo-dev] [PATCH 6/8] kernel-build.eclass: compress xz/zstd in parallel
  2022-10-11 22:54 [gentoo-dev] [PATCH 1/8] kernel-2.eclass: decompress xz in parallel Sam James
                   ` (3 preceding siblings ...)
  2022-10-11 22:54 ` [gentoo-dev] [PATCH 5/8] linux-mod.eclass: cleanup whitespace Sam James
@ 2022-10-11 22:54 ` Sam James
  2022-10-12  4:46   ` Michał Górny
  2022-10-11 22:54 ` [gentoo-dev] [PATCH 7/8] kernel-build.eclass: use pigz for parallel compression if available Sam James
  2022-10-11 22:54 ` [gentoo-dev] [PATCH 8/8] kernel-build.eclass: use pbzip2/lbzip2 " Sam James
  6 siblings, 1 reply; 15+ messages in thread
From: Sam James @ 2022-10-11 22:54 UTC (permalink / raw
  To: gentoo-dev; +Cc: dist-kernel, kernel, Sam James

Note that this will only take effect if compressing
modules with xz or zstd (COMPRESS_MODULE_XZ is set or
COMPRESS_MODULE_ZSTD is set respectively)

... or if compiling the kernel with xz or zstd (COMPRESS_KERNEL_XZ
or COMPRESS_KERNEL_ZSTD is set respectiely).

Signed-off-by: Sam James <sam@gentoo.org>
---
 eclass/kernel-build.eclass | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/eclass/kernel-build.eclass b/eclass/kernel-build.eclass
index 98a385bf72c0..936258e9e8af 100644
--- a/eclass/kernel-build.eclass
+++ b/eclass/kernel-build.eclass
@@ -29,7 +29,7 @@ esac
 
 PYTHON_COMPAT=( python3_{8..11} )
 
-inherit python-any-r1 savedconfig toolchain-funcs kernel-install
+inherit multiprocessing python-any-r1 savedconfig toolchain-funcs kernel-install
 
 BDEPEND="
 	${PYTHON_DEPS}
@@ -90,6 +90,14 @@ kernel-build_src_configure() {
 		ARCH=$(tc-arch-kernel)
 	)
 
+	if type -P xz ; then
+		export XZ_OPT="-T$(makeopts_jobs)"
+	fi
+
+	if type -P zstd ; then
+		export ZSTD_NBTHREADS="$(makeopts_jobs)"
+	fi
+
 	restore_config .config
 	[[ -f .config ]] || die "Ebuild error: please copy default config into .config"
 
-- 
2.38.0



^ permalink raw reply related	[flat|nested] 15+ messages in thread

* [gentoo-dev] [PATCH 7/8] kernel-build.eclass: use pigz for parallel compression if available
  2022-10-11 22:54 [gentoo-dev] [PATCH 1/8] kernel-2.eclass: decompress xz in parallel Sam James
                   ` (4 preceding siblings ...)
  2022-10-11 22:54 ` [gentoo-dev] [PATCH 6/8] kernel-build.eclass: compress xz/zstd in parallel Sam James
@ 2022-10-11 22:54 ` Sam James
  2022-10-11 22:54 ` [gentoo-dev] [PATCH 8/8] kernel-build.eclass: use pbzip2/lbzip2 " Sam James
  6 siblings, 0 replies; 15+ messages in thread
From: Sam James @ 2022-10-11 22:54 UTC (permalink / raw
  To: gentoo-dev; +Cc: dist-kernel, kernel, Sam James

Note that this will only take effect if compressing modules with gzip
(COMPRESS_MODULE_GZIP is set) or compressing the kernel with gzip
(CONFIG_KERNEL_GZIP is set).

Signed-off-by: Sam James <sam@gentoo.org>
---
 eclass/kernel-build.eclass | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/eclass/kernel-build.eclass b/eclass/kernel-build.eclass
index 936258e9e8af..faf36d763d08 100644
--- a/eclass/kernel-build.eclass
+++ b/eclass/kernel-build.eclass
@@ -98,6 +98,13 @@ kernel-build_src_configure() {
 		export ZSTD_NBTHREADS="$(makeopts_jobs)"
 	fi
 
+	# pigz needs to take an argument, not an env var,
+	# for its options, which won't work because of how the kernel build system
+	# uses the variables (e.g. passes directly to tar as an executable).
+	if type -P pigz ; then
+		MAKEARGS+=( KGZIP="pigz" )
+	fi
+
 	restore_config .config
 	[[ -f .config ]] || die "Ebuild error: please copy default config into .config"
 
-- 
2.38.0



^ permalink raw reply related	[flat|nested] 15+ messages in thread

* [gentoo-dev] [PATCH 8/8] kernel-build.eclass: use pbzip2/lbzip2 for parallel compression if available
  2022-10-11 22:54 [gentoo-dev] [PATCH 1/8] kernel-2.eclass: decompress xz in parallel Sam James
                   ` (5 preceding siblings ...)
  2022-10-11 22:54 ` [gentoo-dev] [PATCH 7/8] kernel-build.eclass: use pigz for parallel compression if available Sam James
@ 2022-10-11 22:54 ` Sam James
  2022-10-12  0:20   ` [gentoo-dev] " Sam James
  6 siblings, 1 reply; 15+ messages in thread
From: Sam James @ 2022-10-11 22:54 UTC (permalink / raw
  To: gentoo-dev; +Cc: dist-kernel, kernel, Sam James

Note that this will only take effect if compressing
modules with bzip2 (COMPRESS_MODULE_BZIP2 is set) or
compressing the kernel with bzip2 (CONFIG_KERNEL_BZIP2 is set).

Signed-off-by: Sam James <sam@gentoo.org>
---
 eclass/kernel-build.eclass | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/eclass/kernel-build.eclass b/eclass/kernel-build.eclass
index faf36d763d08..cc99298811bc 100644
--- a/eclass/kernel-build.eclass
+++ b/eclass/kernel-build.eclass
@@ -98,13 +98,19 @@ kernel-build_src_configure() {
 		export ZSTD_NBTHREADS="$(makeopts_jobs)"
 	fi
 
-	# pigz needs to take an argument, not an env var,
-	# for its options, which won't work because of how the kernel build system
+	# pigz/pbzip2/lbzip2 all need to take an argument, not an env var,
+	# for their options, which won't work because of how the kernel build system
 	# uses the variables (e.g. passes directly to tar as an executable).
 	if type -P pigz ; then
 		MAKEARGS+=( KGZIP="pigz" )
 	fi
 
+	if type -P pbzip2 ; then
+		MAKEARGS+=( KBZIP2="pbzip2" )
+	elif type -P lbzip2 ; then
+		MAKEARGS+=( KBZIP2="lbzip2" )
+	fi
+
 	restore_config .config
 	[[ -f .config ]] || die "Ebuild error: please copy default config into .config"
 
-- 
2.38.0



^ permalink raw reply related	[flat|nested] 15+ messages in thread

* [gentoo-dev] Re: [PATCH 8/8] kernel-build.eclass: use pbzip2/lbzip2 for parallel compression if available
  2022-10-11 22:54 ` [gentoo-dev] [PATCH 8/8] kernel-build.eclass: use pbzip2/lbzip2 " Sam James
@ 2022-10-12  0:20   ` Sam James
  0 siblings, 0 replies; 15+ messages in thread
From: Sam James @ 2022-10-12  0:20 UTC (permalink / raw
  To: gentoo-dev; +Cc: dist-kernel, kernel



> On 11 Oct 2022, at 23:55, Sam James <sam@gentoo.org> wrote:
> 
> Note that this will only take effect if compressing
> modules with bzip2 (COMPRESS_MODULE_BZIP2 is set) or
> compressing the kernel with bzip2 (CONFIG_KERNEL_BZIP2 is set).
> 
> Signed-off-by: Sam James <sam@gentoo.org>
> ---
> eclass/kernel-build.eclass | 10 ++++++++--
> 1 file changed, 8 insertions(+), 2 deletions(-)
> 
> diff --git a/eclass/kernel-build.eclass b/eclass/kernel-build.eclass
> index faf36d763d08..cc99298811bc 100644
> --- a/eclass/kernel-build.eclass
> +++ b/eclass/kernel-build.eclass
> @@ -98,13 +98,19 @@ kernel-build_src_configure() {
>        export ZSTD_NBTHREADS="$(makeopts_jobs)"
>    fi
> 
> -    # pigz needs to take an argument, not an env var,
> -    # for its options, which won't work because of how the kernel build system
> +    # pigz/pbzip2/lbzip2 all need to take an argument, not an env var,
> +    # for their options, which won't work because of how the kernel build system
>    # uses the variables (e.g. passes directly to tar as an executable).
>    if type -P pigz ; then
>        MAKEARGS+=( KGZIP="pigz" )
>    fi
> 
> +    if type -P pbzip2 ; then

Oh, all of these need a redirection to silence them, which I'll fix locally.

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [gentoo-dev] [PATCH 6/8] kernel-build.eclass: compress xz/zstd in parallel
  2022-10-11 22:54 ` [gentoo-dev] [PATCH 6/8] kernel-build.eclass: compress xz/zstd in parallel Sam James
@ 2022-10-12  4:46   ` Michał Górny
  2022-10-12  4:51     ` Sam James
                       ` (2 more replies)
  0 siblings, 3 replies; 15+ messages in thread
From: Michał Górny @ 2022-10-12  4:46 UTC (permalink / raw
  To: gentoo-dev; +Cc: dist-kernel, kernel, Sam James

On Tue, 2022-10-11 at 23:54 +0100, Sam James wrote:
> Note that this will only take effect if compressing
> modules with xz or zstd (COMPRESS_MODULE_XZ is set or
> COMPRESS_MODULE_ZSTD is set respectively)
> 
> ... or if compiling the kernel with xz or zstd (COMPRESS_KERNEL_XZ
> or COMPRESS_KERNEL_ZSTD is set respectiely).
> 
> Signed-off-by: Sam James <sam@gentoo.org>
> ---
>  eclass/kernel-build.eclass | 10 +++++++++-
>  1 file changed, 9 insertions(+), 1 deletion(-)
> 
> diff --git a/eclass/kernel-build.eclass b/eclass/kernel-build.eclass
> index 98a385bf72c0..936258e9e8af 100644
> --- a/eclass/kernel-build.eclass
> +++ b/eclass/kernel-build.eclass
> @@ -29,7 +29,7 @@ esac
>  
>  PYTHON_COMPAT=( python3_{8..11} )
>  
> -inherit python-any-r1 savedconfig toolchain-funcs kernel-install
> +inherit multiprocessing python-any-r1 savedconfig toolchain-funcs kernel-install
>  
>  BDEPEND="
>  	${PYTHON_DEPS}
> @@ -90,6 +90,14 @@ kernel-build_src_configure() {
>  		ARCH=$(tc-arch-kernel)
>  	)
>  
> +	if type -P xz ; then
> +		export XZ_OPT="-T$(makeopts_jobs)"
> +	fi
> +
> +	if type -P zstd ; then
> +		export ZSTD_NBTHREADS="$(makeopts_jobs)"
> +	fi
> +
>  	restore_config .config
>  	[[ -f .config ]] || die "Ebuild error: please copy default config into .config"
>  

I'm think that we need an override for these values.  Just because
someone is using -j12 to compile source files in parallel (which has no
real drawback), it doesn't mean that that person wants to use 12-thread
parallel xz/zstd that will actually reduce compression ratio for minimal
gain (I mean, most of these files won't even use parallel compression
because they're too small).

Not to mention that someone using -j32 distcc setup necessarily wants
32-threaded local compression.

-- 
Best regards,
Michał Górny



^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [gentoo-dev] [PATCH 6/8] kernel-build.eclass: compress xz/zstd in parallel
  2022-10-12  4:46   ` Michał Górny
@ 2022-10-12  4:51     ` Sam James
  2022-10-12  8:39     ` Ulrich Mueller
  2022-10-12 16:43     ` Sam James
  2 siblings, 0 replies; 15+ messages in thread
From: Sam James @ 2022-10-12  4:51 UTC (permalink / raw
  To: Michał Górny; +Cc: gentoo-dev, dist-kernel, kernel



> On 12 Oct 2022, at 05:46, Michał Górny <mgorny@gentoo.org> wrote:
> 
> On Tue, 2022-10-11 at 23:54 +0100, Sam James wrote:
>> Note that this will only take effect if compressing
>> modules with xz or zstd (COMPRESS_MODULE_XZ is set or
>> COMPRESS_MODULE_ZSTD is set respectively)
>> 
>> ... or if compiling the kernel with xz or zstd (COMPRESS_KERNEL_XZ
>> or COMPRESS_KERNEL_ZSTD is set respectiely).
>> 
>> Signed-off-by: Sam James <sam@gentoo.org>
>> ---
>> eclass/kernel-build.eclass | 10 +++++++++-
>> 1 file changed, 9 insertions(+), 1 deletion(-)
>> 
>> diff --git a/eclass/kernel-build.eclass b/eclass/kernel-build.eclass
>> index 98a385bf72c0..936258e9e8af 100644
>> --- a/eclass/kernel-build.eclass
>> +++ b/eclass/kernel-build.eclass
>> @@ -29,7 +29,7 @@ esac
>> 
>> PYTHON_COMPAT=( python3_{8..11} )
>> 
>> -inherit python-any-r1 savedconfig toolchain-funcs kernel-install
>> +inherit multiprocessing python-any-r1 savedconfig toolchain-funcs kernel-install
>> 
>> BDEPEND="
>>    ${PYTHON_DEPS}
>> @@ -90,6 +90,14 @@ kernel-build_src_configure() {
>>        ARCH=$(tc-arch-kernel)
>>    )
>> 
>> +    if type -P xz ; then
>> +        export XZ_OPT="-T$(makeopts_jobs)"
>> +    fi
>> +
>> +    if type -P zstd ; then
>> +        export ZSTD_NBTHREADS="$(makeopts_jobs)"
>> +    fi
>> +
>>    restore_config .config
>>    [[ -f .config ]] || die "Ebuild error: please copy default config into .config"
>> 
> 
> I'm think that we need an override for these values.  Just because
> someone is using -j12 to compile source files in parallel (which has no
> real drawback), it doesn't mean that that person wants to use 12-thread
> parallel xz/zstd that will actually reduce compression ratio for minimal
> gain (I mean, most of these files won't even use parallel compression
> because they're too small).
> 
> Not to mention that someone using -j32 distcc setup necessarily wants
> 32-threaded local compression.

I'll make it use XZ_OPT if present in environment to avoid clobbering, ditto zstd.


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [gentoo-dev] [PATCH 2/8] linux-mod.eclass: compress xz/zstd in parallel
  2022-10-11 22:54 ` [gentoo-dev] [PATCH 2/8] linux-mod.eclass: compress xz/zstd " Sam James
@ 2022-10-12  8:27   ` Ulrich Mueller
  2022-10-12 16:42     ` Sam James
  0 siblings, 1 reply; 15+ messages in thread
From: Ulrich Mueller @ 2022-10-12  8:27 UTC (permalink / raw
  To: Sam James; +Cc: gentoo-dev, dist-kernel, kernel

[-- Attachment #1: Type: text/plain, Size: 227 bytes --]

What is the motivation for this? The typical kernel module on my system
has a size of about 20 KiB uncompressed, with the largest about 2 MiB.
Compression times are negligible for these, so is there a need to
optimize?

Ulrich

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 507 bytes --]

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [gentoo-dev] [PATCH 6/8] kernel-build.eclass: compress xz/zstd in parallel
  2022-10-12  4:46   ` Michał Górny
  2022-10-12  4:51     ` Sam James
@ 2022-10-12  8:39     ` Ulrich Mueller
  2022-10-12 16:43     ` Sam James
  2 siblings, 0 replies; 15+ messages in thread
From: Ulrich Mueller @ 2022-10-12  8:39 UTC (permalink / raw
  To: Michał Górny; +Cc: gentoo-dev, dist-kernel, kernel, Sam James

[-- Attachment #1: Type: text/plain, Size: 405 bytes --]

>>>>> On Wed, 12 Oct 2022, Michał Górny wrote:

> (I mean, most of these files won't even use parallel compression
> because they're too small).

xz(1) says about the block size: "The default size is three times the
LZMA2 dictionary size or 1 MiB, whichever is more." The default (-6)
dictionary size is 8 MiB.

So IIUC only files larger than 24 MiB would be compressed in parallel?

Ulrich

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 507 bytes --]

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [gentoo-dev] [PATCH 2/8] linux-mod.eclass: compress xz/zstd in parallel
  2022-10-12  8:27   ` Ulrich Mueller
@ 2022-10-12 16:42     ` Sam James
  0 siblings, 0 replies; 15+ messages in thread
From: Sam James @ 2022-10-12 16:42 UTC (permalink / raw
  To: gentoo-dev; +Cc: dist-kernel, kernel, Ulrich Mueller

[-- Attachment #1: Type: text/plain, Size: 553 bytes --]


Ulrich Mueller <ulm@gentoo.org> writes:

> [[PGP Signed Part:Undecided]]
> What is the motivation for this? The typical kernel module on my system
> has a size of about 20 KiB uncompressed, with the largest about 2 MiB.
> Compression times are negligible for these, so is there a need to
> optimize?

Note that this is only for external kernel modules. The motivation
for these is mostly for slower systems with many cores, but
slower per-core speeds (I have a bunch of ARM64 devices
like that).

>
> Ulrich
>
> [[End of PGP Signed Part]]

thanks,
sam

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 377 bytes --]

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [gentoo-dev] [PATCH 6/8] kernel-build.eclass: compress xz/zstd in parallel
  2022-10-12  4:46   ` Michał Górny
  2022-10-12  4:51     ` Sam James
  2022-10-12  8:39     ` Ulrich Mueller
@ 2022-10-12 16:43     ` Sam James
  2 siblings, 0 replies; 15+ messages in thread
From: Sam James @ 2022-10-12 16:43 UTC (permalink / raw
  To: gentoo-dev; +Cc: dist-kernel, kernel, Michał Górny

[-- Attachment #1: Type: text/plain, Size: 1935 bytes --]


Michał Górny <mgorny@gentoo.org> writes:

> On Tue, 2022-10-11 at 23:54 +0100, Sam James wrote:
>> Note that this will only take effect if compressing
>> modules with xz or zstd (COMPRESS_MODULE_XZ is set or
>> COMPRESS_MODULE_ZSTD is set respectively)
>> 
>> ... or if compiling the kernel with xz or zstd (COMPRESS_KERNEL_XZ
>> or COMPRESS_KERNEL_ZSTD is set respectiely).
>> 
>> Signed-off-by: Sam James <sam@gentoo.org>
>> ---
>>  eclass/kernel-build.eclass | 10 +++++++++-
>>  1 file changed, 9 insertions(+), 1 deletion(-)
>> 
>> diff --git a/eclass/kernel-build.eclass b/eclass/kernel-build.eclass
>> index 98a385bf72c0..936258e9e8af 100644
>> --- a/eclass/kernel-build.eclass
>> +++ b/eclass/kernel-build.eclass
>> @@ -29,7 +29,7 @@ esac
>>  
>>  PYTHON_COMPAT=( python3_{8..11} )
>>  
>> -inherit python-any-r1 savedconfig toolchain-funcs kernel-install
>> +inherit multiprocessing python-any-r1 savedconfig toolchain-funcs kernel-install
>>  
>>  BDEPEND="
>>  	${PYTHON_DEPS}
>> @@ -90,6 +90,14 @@ kernel-build_src_configure() {
>>  		ARCH=$(tc-arch-kernel)
>>  	)
>>  
>> +	if type -P xz ; then
>> +		export XZ_OPT="-T$(makeopts_jobs)"
>> +	fi
>> +
>> +	if type -P zstd ; then
>> +		export ZSTD_NBTHREADS="$(makeopts_jobs)"
>> +	fi
>> +
>>  	restore_config .config
>>  	[[ -f .config ]] || die "Ebuild error: please copy default config into .config"
>>  
>
> I'm think that we need an override for these values.  Just because
> someone is using -j12 to compile source files in parallel (which has no
> real drawback), it doesn't mean that that person wants to use 12-thread
> parallel xz/zstd that will actually reduce compression ratio for minimal
> gain (I mean, most of these files won't even use parallel compression
> because they're too small).

Note that >= xz-5.3.3_alpha defaults to the threaded compressor
anyway, just with -j1 and the difference is tiny.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 377 bytes --]

^ permalink raw reply	[flat|nested] 15+ messages in thread

end of thread, other threads:[~2022-10-12 16:44 UTC | newest]

Thread overview: 15+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2022-10-11 22:54 [gentoo-dev] [PATCH 1/8] kernel-2.eclass: decompress xz in parallel Sam James
2022-10-11 22:54 ` [gentoo-dev] [PATCH 2/8] linux-mod.eclass: compress xz/zstd " Sam James
2022-10-12  8:27   ` Ulrich Mueller
2022-10-12 16:42     ` Sam James
2022-10-11 22:54 ` [gentoo-dev] [PATCH 3/8] linux-mod.eclass: use pigz for parallel compression if available Sam James
2022-10-11 22:54 ` [gentoo-dev] [PATCH 4/8] linux-mod.eclass: improve error handling for module compression Sam James
2022-10-11 22:54 ` [gentoo-dev] [PATCH 5/8] linux-mod.eclass: cleanup whitespace Sam James
2022-10-11 22:54 ` [gentoo-dev] [PATCH 6/8] kernel-build.eclass: compress xz/zstd in parallel Sam James
2022-10-12  4:46   ` Michał Górny
2022-10-12  4:51     ` Sam James
2022-10-12  8:39     ` Ulrich Mueller
2022-10-12 16:43     ` Sam James
2022-10-11 22:54 ` [gentoo-dev] [PATCH 7/8] kernel-build.eclass: use pigz for parallel compression if available Sam James
2022-10-11 22:54 ` [gentoo-dev] [PATCH 8/8] kernel-build.eclass: use pbzip2/lbzip2 " Sam James
2022-10-12  0:20   ` [gentoo-dev] " Sam James

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox