[gentoo-dev] Re: [gentoo-commits] repo/gentoo:master commit in: media-libs/x265/

[gentoo-dev] Re: [gentoo-commits] repo/gentoo:master commit in: media-libs/x265/

[Alexis Ballier]

On Sun, 24 May 2020 20:25:11 +0000 (UTC)
"Thomas Deutschmann" <whissi@gentoo.org> wrote:

> commit:     eba596db8a926adb18595549c89294ed0a1e929e
> Author:     Thomas Deutschmann <whissi <AT> gentoo <DOT> org>
> AuthorDate: Sun May 24 15:07:04 2020 +0000
> Commit:     Thomas Deutschmann <whissi <AT> gentoo <DOT> org>
> CommitDate: Sun May 24 20:23:50 2020 +0000
> URL:
> https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=eba596db
> 
> media-libs/x265: rework assembly support
> 
> Closes: https://bugs.gentoo.org/681878
> Package-Manager: Portage-2.3.99, Repoman-2.3.22
> Signed-off-by: Thomas Deutschmann <whissi <AT> gentoo.org>
> 
>  media-libs/x265/metadata.xml    |  1 +
>  media-libs/x265/x265-3.3.ebuild | 66
> ++++++++++++++++++++--------------------- 2 files changed, 34
> insertions(+), 33 deletions(-)
> 
> diff --git a/media-libs/x265/metadata.xml
> b/media-libs/x265/metadata.xml index 22a07293b83..c585d553631 100644
> --- a/media-libs/x265/metadata.xml
> +++ b/media-libs/x265/metadata.xml
> @@ -5,6 +5,7 @@
>      <email>media-video@gentoo.org</email>
>    </maintainer>
>    <use>
> +    <flag name="asm">Enable x86_64 assembly optimizations.</flag>



This should not even be an useflag.
Either it works or does not. Individual features support is controlled
by cpu_flags_* (or built-in and autodetected at runtime).
Please fix.



>      <flag name="10bit">Add support for producing 10bits HEVC.</flag>
>      <flag name="12bit">Add support for producing 12bits HEVC.</flag>
>      <flag name="numa">Build with support for NUMA nodes.</flag>
> 
> diff --git a/media-libs/x265/x265-3.3.ebuild
> b/media-libs/x265/x265-3.3.ebuild index 9fc0159bc00..f5c4fee6d97
> 100644 --- a/media-libs/x265/x265-3.3.ebuild
> +++ b/media-libs/x265/x265-3.3.ebuild
> @@ -19,15 +19,17 @@ HOMEPAGE="http://x265.org/
> https://bitbucket.org/multicoreware/x265/wiki/Home" LICENSE="GPL-2"
>  # subslot = libx265 soname
>  SLOT="0/188"
> -IUSE="+10bit +12bit cpu_flags_arm_neon numa pic power8 test"
> +IUSE="+asm +10bit +12bit cpu_flags_arm_neon numa pic power8 test"
>  
>  # Test suite requires assembly support and is known to be broken
>  RESTRICT="test"
>  
>  ASM_DEPEND=">=dev-lang/yasm-1.2.0"
>  
> -BDEPEND="abi_x86_32? ( ${ASM_DEPEND} )
> -	abi_x86_64? ( ${ASM_DEPEND} )"
> +BDEPEND="asm? (
> +		abi_x86_32? ( ${ASM_DEPEND} )
> +		abi_x86_64? ( ${ASM_DEPEND} )
> +	)"
>  
>  RDEPEND="numa? ( >=sys-process/numactl-2.0.10-r1[${MULTILIB_USEDEP}]
> )" 
> @@ -85,17 +87,6 @@ x265_variant_src_configure() {
>  				-DENABLE_CLI=OFF
>  				-DMAIN12=ON
>  			)
> -			if [[ ${ABI} = x86 ]] ; then
> -				mycmakeargs+=( -DENABLE_ASSEMBLY=OFF
> )
> -			fi
> -			if [[ ${ABI} = arm ]] ; then
> -				# 589674
> -				mycmakeargs+=( -DENABLE_ASSEMBLY=OFF
> )
> -			fi
> -			if [[ ${ABI} = ppc64 ]] ; then
> -				#
> https://bugs.gentoo.org/show_bug.cgi?id=607802#c5
> -				mycmakeargs+=( -DENABLE_ASSEMBLY=OFF
> -DENABLE_ALTIVEC=OFF )
> -			fi
>  			;;
>  		"main10")
>  			mycmakeargs+=(
> @@ -104,17 +95,6 @@ x265_variant_src_configure() {
>  				-DENABLE_SHARED=OFF
>  				-DENABLE_CLI=OFF
>  			)
> -			if [[ ${ABI} = x86 ]] ; then
> -				mycmakeargs+=( -DENABLE_ASSEMBLY=OFF
> )
> -			fi
> -			if [[ ${ABI} = arm ]] ; then
> -				# 589674
> -				mycmakeargs+=( -DENABLE_ASSEMBLY=OFF
> )
> -			fi
> -			if [[ ${ABI} = ppc64 ]] ; then
> -				#
> https://bugs.gentoo.org/show_bug.cgi?id=607802#c5
> -				mycmakeargs+=( -DENABLE_ASSEMBLY=OFF
> -DENABLE_ALTIVEC=OFF )
> -			fi
>  			;;


What are you trying to fix here ?
This sounds like a regression to me: some asm will not work for
10/12bits variants while 8bit is fine. You are removing this work here.



>  		"main")
>  			if (( "${#MULTIBUILD_VARIANTS[@]}" > 1 )) ;
> then @@ -146,7 +126,6 @@ multilib_src_configure() {
>  	append-cxxflags -fPIC
>  
>  	local myabicmakeargs=(
> -		-DENABLE_TESTS=$(usex test ON OFF)
>  		$(multilib_is_native_abi || echo "-DENABLE_CLI=OFF")
>  		-DENABLE_LIBNUMA=$(usex numa ON OFF)
>  		-DCPU_POWER8=$(usex power8 ON OFF)
> @@ -154,18 +133,39 @@ multilib_src_configure() {
>  		-DLIB_INSTALL_DIR="$(get_libdir)"
>  	)
>  
> +	local supports_asm=yes
> +
>  	if [[ ${ABI} = x86 ]] ; then
> -		# Bug #528202
> -		if use pic ; then
> +		if use asm && use pic ; then
> +			# Bug #528202
>  			ewarn "PIC has been requested but x86 asm is
> not PIC-safe, disabling it."
> -			myabicmakeargs+=( -DENABLE_ASSEMBLY=OFF )
> +			supports_asm=no
>  		fi
>  	elif [[ ${ABI} = x32 ]] ; then
> -		# bug #510890
> -		myabicmakeargs+=( -DENABLE_ASSEMBLY=OFF )
> +		if use asm ; then
> +			# bug #510890
> +			ewarn "x32 ABI doesn't support asm"
> +			supports_asm=no
> +		fi
>  	elif [[ ${ABI} = arm ]] ; then
> -		myabicmakeargs+=( -DENABLE_ASSEMBLY=$(usex pic OFF
> $(usex cpu_flags_arm_neon ON OFF)) )
> -		use cpu_flags_arm_neon && use pic && ewarn "PIC has
> been requested but arm neon asm is not PIC-safe, disabling it."
> +		if use asm && use pic ; then
> +			ewarn "PIC has been requested but arm neon
> asm is not PIC-safe, disabling it."
> +			supports_asm=no
> +		elif use asm && use cpu_flags_arm_neon ; then
> +			supports_asm=yes
> +		elif use asm ; then
> +			supports_asm=no
> +		fi
> +	fi
> +
> +	if [[ "${supports_asm}" = yes ]] && use asm ; then
> +		myabicmakeargs+=( -DENABLE_ASSEMBLY=ON )
> +
> +		if multilib_is_native_abi ; then
> +			myabicmakeargs+=( -DENABLE_TESTS=$(usex test
> ON OFF) )
> +		fi


Tying tests to native ABI seems wrong to me.



Alexis.

[gentoo-dev] Re: [gentoo-commits] repo/gentoo:master commit in: media-libs/x265/

[Alexis Ballier]

On Sun, 24 May 2020 20:25:11 +0000 (UTC)
"Thomas Deutschmann" <whissi@gentoo.org> wrote:

> commit:     6e149596cc76f1bbcee6720828c8c8c92420f2a3
> Author:     Thomas Deutschmann <whissi <AT> gentoo <DOT> org>
> AuthorDate: Sun May 24 19:47:08 2020 +0000
> Commit:     Thomas Deutschmann <whissi <AT> gentoo <DOT> org>
> CommitDate: Sun May 24 20:23:53 2020 +0000
> URL:
> https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=6e149596
> 
> media-libs/x265: drop USE=pic
> 
> Gentoo's toolchain uses PIC by default. Since USE=asm was added,
> we no longer need a USE flag to control that behavior.


You got it wrong here it seems: USE=pic does not control whether
the toolchain produces PIC or not. Shared libs always are, and have
always been, built that way on Gentoo.
In this case, USE=pic means "no matter what it costs, I do not want
textrels", for the cases of hand written assembly that has to be
rewritten to support PIC. And, still in this case, this costs a lot of
performance, so it is enabled by default on hardened profiles and not
others.
Textrels work fine (on some architectures), they disallow W^X and force
each process using the shared lib to make a "copy" at runtime in order
to resolve relocations, so are not desirable but sometimes the cost
outweights the gain.

Plus, profiles/features/hardened enables pic by default but knows
nothing about USE=asm so this is a regression for them.


Alexis.

Re: [gentoo-dev] Re: [gentoo-commits] repo/gentoo:master commit in: media-libs/x265/

[Mike Gilbert]

On Mon, May 25, 2020 at 9:13 AM Alexis Ballier <aballier@gentoo.org> wrote:
>
> On Sun, 24 May 2020 20:25:11 +0000 (UTC)
> "Thomas Deutschmann" <whissi@gentoo.org> wrote:
>
> > commit:     6e149596cc76f1bbcee6720828c8c8c92420f2a3
> > Author:     Thomas Deutschmann <whissi <AT> gentoo <DOT> org>
> > AuthorDate: Sun May 24 19:47:08 2020 +0000
> > Commit:     Thomas Deutschmann <whissi <AT> gentoo <DOT> org>
> > CommitDate: Sun May 24 20:23:53 2020 +0000
> > URL:
> > https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=6e149596
> >
> > media-libs/x265: drop USE=pic
> >
> > Gentoo's toolchain uses PIC by default. Since USE=asm was added,
> > we no longer need a USE flag to control that behavior.
>
>
> You got it wrong here it seems: USE=pic does not control whether
> the toolchain produces PIC or not. Shared libs always are, and have
> always been, built that way on Gentoo.
> In this case, USE=pic means "no matter what it costs, I do not want
> textrels", for the cases of hand written assembly that has to be
> rewritten to support PIC. And, still in this case, this costs a lot of
> performance, so it is enabled by default on hardened profiles and not
> others.
> Textrels work fine (on some architectures), they disallow W^X and force
> each process using the shared lib to make a "copy" at runtime in order
> to resolve relocations, so are not desirable but sometimes the cost
> outweights the gain.
>
> Plus, profiles/features/hardened enables pic by default but knows
> nothing about USE=asm so this is a regression for them.

The USE flag toggles use of assembly, not use of PIC. The default USE
value in the hardened profile should not drive decisions on what we
name USE flags.

You can add the flag to package.use or package.use.mask in the
hardened profile if necessary.

Re: [gentoo-dev] Re: [gentoo-commits] repo/gentoo:master commit in: media-libs/x265/

[Alexis Ballier]

On Mon, 25 May 2020 11:26:26 -0400
Mike Gilbert <floppym@gentoo.org> wrote:

> On Mon, May 25, 2020 at 9:13 AM Alexis Ballier <aballier@gentoo.org>
> wrote:
> >
> > On Sun, 24 May 2020 20:25:11 +0000 (UTC)
> > "Thomas Deutschmann" <whissi@gentoo.org> wrote:
> >
> > > commit:     6e149596cc76f1bbcee6720828c8c8c92420f2a3
> > > Author:     Thomas Deutschmann <whissi <AT> gentoo <DOT> org>
> > > AuthorDate: Sun May 24 19:47:08 2020 +0000
> > > Commit:     Thomas Deutschmann <whissi <AT> gentoo <DOT> org>
> > > CommitDate: Sun May 24 20:23:53 2020 +0000
> > > URL:
> > > https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=6e149596
> > >
> > > media-libs/x265: drop USE=pic
> > >
> > > Gentoo's toolchain uses PIC by default. Since USE=asm was added,
> > > we no longer need a USE flag to control that behavior.
> >
> >
> > You got it wrong here it seems: USE=pic does not control whether
> > the toolchain produces PIC or not. Shared libs always are, and have
> > always been, built that way on Gentoo.
> > In this case, USE=pic means "no matter what it costs, I do not want
> > textrels", for the cases of hand written assembly that has to be
> > rewritten to support PIC. And, still in this case, this costs a lot
> > of performance, so it is enabled by default on hardened profiles
> > and not others.
> > Textrels work fine (on some architectures), they disallow W^X and
> > force each process using the shared lib to make a "copy" at runtime
> > in order to resolve relocations, so are not desirable but sometimes
> > the cost outweights the gain.
> >
> > Plus, profiles/features/hardened enables pic by default but knows
> > nothing about USE=asm so this is a regression for them.
> 
> The USE flag toggles use of assembly, not use of PIC. The default USE
> value in the hardened profile should not drive decisions on what we
> name USE flags.

... but using a global well documented useflag instead of a local
invention should drive such decisions.

> 
> You can add the flag to package.use or package.use.mask in the
> hardened profile if necessary.


Sure but it was not added there and it's not really relevant here since
 USE=asm does not make any sense either. (aka: this was better
 before this series of commits)

Re: [gentoo-dev] Re: [gentoo-commits] repo/gentoo:master commit in: media-libs/x265/

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2225 bytes --]

On Mon, 2020-05-25 at 19:49 +0200, Alexis Ballier wrote:
> On Mon, 25 May 2020 11:26:26 -0400
> Mike Gilbert <floppym@gentoo.org> wrote:
> 
> > On Mon, May 25, 2020 at 9:13 AM Alexis Ballier <aballier@gentoo.org>
> > wrote:
> > > On Sun, 24 May 2020 20:25:11 +0000 (UTC)
> > > "Thomas Deutschmann" <whissi@gentoo.org> wrote:
> > > 
> > > > commit:     6e149596cc76f1bbcee6720828c8c8c92420f2a3
> > > > Author:     Thomas Deutschmann <whissi <AT> gentoo <DOT> org>
> > > > AuthorDate: Sun May 24 19:47:08 2020 +0000
> > > > Commit:     Thomas Deutschmann <whissi <AT> gentoo <DOT> org>
> > > > CommitDate: Sun May 24 20:23:53 2020 +0000
> > > > URL:
> > > > https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=6e149596
> > > > 
> > > > media-libs/x265: drop USE=pic
> > > > 
> > > > Gentoo's toolchain uses PIC by default. Since USE=asm was added,
> > > > we no longer need a USE flag to control that behavior.
> > > 
> > > You got it wrong here it seems: USE=pic does not control whether
> > > the toolchain produces PIC or not. Shared libs always are, and have
> > > always been, built that way on Gentoo.
> > > In this case, USE=pic means "no matter what it costs, I do not want
> > > textrels", for the cases of hand written assembly that has to be
> > > rewritten to support PIC. And, still in this case, this costs a lot
> > > of performance, so it is enabled by default on hardened profiles
> > > and not others.
> > > Textrels work fine (on some architectures), they disallow W^X and
> > > force each process using the shared lib to make a "copy" at runtime
> > > in order to resolve relocations, so are not desirable but sometimes
> > > the cost outweights the gain.
> > > 
> > > Plus, profiles/features/hardened enables pic by default but knows
> > > nothing about USE=asm so this is a regression for them.
> > 
> > The USE flag toggles use of assembly, not use of PIC. The default USE
> > value in the hardened profile should not drive decisions on what we
> > name USE flags.
> 
> ... but using a global well documented useflag instead of a local
> invention should drive such decisions.

What 'global well documented useflag'?

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] Re: [gentoo-commits] repo/gentoo:master commit in: media-libs/x265/

[Mike Gilbert]

On Mon, May 25, 2020 at 3:18 PM Michał Górny <mgorny@gentoo.org> wrote:
>
> On Mon, 2020-05-25 at 19:49 +0200, Alexis Ballier wrote:
> > On Mon, 25 May 2020 11:26:26 -0400
> > Mike Gilbert <floppym@gentoo.org> wrote:
> >
> > > On Mon, May 25, 2020 at 9:13 AM Alexis Ballier <aballier@gentoo.org>
> > > wrote:
> > > > On Sun, 24 May 2020 20:25:11 +0000 (UTC)
> > > > "Thomas Deutschmann" <whissi@gentoo.org> wrote:
> > > >
> > > > > commit:     6e149596cc76f1bbcee6720828c8c8c92420f2a3
> > > > > Author:     Thomas Deutschmann <whissi <AT> gentoo <DOT> org>
> > > > > AuthorDate: Sun May 24 19:47:08 2020 +0000
> > > > > Commit:     Thomas Deutschmann <whissi <AT> gentoo <DOT> org>
> > > > > CommitDate: Sun May 24 20:23:53 2020 +0000
> > > > > URL:
> > > > > https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=6e149596
> > > > >
> > > > > media-libs/x265: drop USE=pic
> > > > >
> > > > > Gentoo's toolchain uses PIC by default. Since USE=asm was added,
> > > > > we no longer need a USE flag to control that behavior.
> > > >
> > > > You got it wrong here it seems: USE=pic does not control whether
> > > > the toolchain produces PIC or not. Shared libs always are, and have
> > > > always been, built that way on Gentoo.
> > > > In this case, USE=pic means "no matter what it costs, I do not want
> > > > textrels", for the cases of hand written assembly that has to be
> > > > rewritten to support PIC. And, still in this case, this costs a lot
> > > > of performance, so it is enabled by default on hardened profiles
> > > > and not others.
> > > > Textrels work fine (on some architectures), they disallow W^X and
> > > > force each process using the shared lib to make a "copy" at runtime
> > > > in order to resolve relocations, so are not desirable but sometimes
> > > > the cost outweights the gain.
> > > >
> > > > Plus, profiles/features/hardened enables pic by default but knows
> > > > nothing about USE=asm so this is a regression for them.
> > >
> > > The USE flag toggles use of assembly, not use of PIC. The default USE
> > > value in the hardened profile should not drive decisions on what we
> > > name USE flags.
> >
> > ... but using a global well documented useflag instead of a local
> > invention should drive such decisions.
>
> What 'global well documented useflag'?

It's neither global, nor well-documented, but several packages do
define it locally.

profiles % grep ":pic " use.local.desc
app-arch/gzip:pic - disable optimized assembly code that is not PIC friendly
app-benchmarks/ramspeed:pic - Force shared libraries to be built as
PIC (this is slower)
dev-libs/gmp:pic - Force static libraries to be built as PIC to avoid TEXTRELs.
gnome-base/orbit:pic - Force libname-server-2 to be built as PIC;
needed on hardened systems
media-libs/x264:pic - disable optimized assembly code that is not PIC friendly
media-libs/x265:pic - Disable optimized assembly code that is not PIC friendly
media-libs/xvid:pic - disable optimized assembly code that is not PIC friendly
media-video/ffmpeg:pic - Force shared libraries to be built as PIC
(this is slower)
media-video/transcode:pic - disable optimized assembly code that is
not PIC friendly
www-client/chromium:pic - Disable optimized assembly code that is not
PIC friendly

I suspect this flag got copy/pasted between various packages related
to ffmpeg. That's certainly how chromium ended up with it.

Personally, I think it should be renamed to "asm" or something similar
in the majority of cases where it actually disables all use of
assembly code.

Re: [gentoo-dev] Re: [gentoo-commits] repo/gentoo:master commit in: media-libs/x265/

[Alexis Ballier]

On Mon, 2020-05-25 at 17:04 -0400, Mike Gilbert wrote:
> On Mon, May 25, 2020 at 3:18 PM Michał Górny <mgorny@gentoo.org>
> wrote:
> > On Mon, 2020-05-25 at 19:49 +0200, Alexis Ballier wrote:
> > > On Mon, 25 May 2020 11:26:26 -0400
> > > Mike Gilbert <floppym@gentoo.org> wrote:
> > > 
> > > > On Mon, May 25, 2020 at 9:13 AM Alexis Ballier <
> > > > aballier@gentoo.org>
> > > > wrote:
> > > > > On Sun, 24 May 2020 20:25:11 +0000 (UTC)
> > > > > "Thomas Deutschmann" <whissi@gentoo.org> wrote:
> > > > > 
> > > > > > commit:     6e149596cc76f1bbcee6720828c8c8c92420f2a3
> > > > > > Author:     Thomas Deutschmann <whissi <AT> gentoo <DOT>
> > > > > > org>
> > > > > > AuthorDate: Sun May 24 19:47:08 2020 +0000
> > > > > > Commit:     Thomas Deutschmann <whissi <AT> gentoo <DOT>
> > > > > > org>
> > > > > > CommitDate: Sun May 24 20:23:53 2020 +0000
> > > > > > URL:
> > > > > > https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=6e149596
> > > > > > 
> > > > > > media-libs/x265: drop USE=pic
> > > > > > 
> > > > > > Gentoo's toolchain uses PIC by default. Since USE=asm was
> > > > > > added,
> > > > > > we no longer need a USE flag to control that behavior.
> > > > > 
> > > > > You got it wrong here it seems: USE=pic does not control
> > > > > whether
> > > > > the toolchain produces PIC or not. Shared libs always are,
> > > > > and have
> > > > > always been, built that way on Gentoo.
> > > > > In this case, USE=pic means "no matter what it costs, I do
> > > > > not want
> > > > > textrels", for the cases of hand written assembly that has to
> > > > > be
> > > > > rewritten to support PIC. And, still in this case, this costs
> > > > > a lot
> > > > > of performance, so it is enabled by default on hardened
> > > > > profiles
> > > > > and not others.
> > > > > Textrels work fine (on some architectures), they disallow W^X
> > > > > and
> > > > > force each process using the shared lib to make a "copy" at
> > > > > runtime
> > > > > in order to resolve relocations, so are not desirable but
> > > > > sometimes
> > > > > the cost outweights the gain.
> > > > > 
> > > > > Plus, profiles/features/hardened enables pic by default but
> > > > > knows
> > > > > nothing about USE=asm so this is a regression for them.
> > > > 
> > > > The USE flag toggles use of assembly, not use of PIC. The
> > > > default USE
> > > > value in the hardened profile should not drive decisions on
> > > > what we
> > > > name USE flags.
> > > 
> > > ... but using a global well documented useflag instead of a local
> > > invention should drive such decisions.
> > 
> > What 'global well documented useflag'?
> 
> It's neither global, nor well-documented, but several packages do
> define it locally.
> 

https://gitweb.gentoo.org/repo/gentoo/historical.git/commit/profiles/use.desc?id=103236c295aa30e5e42cfc8a7429e4eea5f0d680

https://gitweb.gentoo.org/repo/gentoo/historical.git/commit/profiles/use.desc?id=784deb7134b9d430546557a8f8a0877bf35c02ba

I guess this hasn't been really discussed back then.

It is also used in a global way in profiles (make.defaults).

> Personally, I think it should be renamed to "asm" or something
> similar
> in the majority of cases where it actually disables all use of
> assembly code.

Thankfully these days there's usually no need to disable asm to have
pic. hardened has no mention of that flag, and I think that e.g. for
openssl they would have noticed long ago.
And again, 'asm' as a useflag makes no sense: if it works and simply
replaces a C function by a faster one then it shouldn't even be an
useflag. 'pic' on the other hand conveys the tradeoff idea.

Re: [gentoo-dev] Re: [gentoo-commits] repo/gentoo:master commit in: media-libs/x265/

[Mike Gilbert]

On Mon, May 25, 2020 at 7:35 PM Alexis Ballier <aballier@gentoo.org> wrote:
>
> On Mon, 2020-05-25 at 17:04 -0400, Mike Gilbert wrote:
> > On Mon, May 25, 2020 at 3:18 PM Michał Górny <mgorny@gentoo.org>
> > wrote:
> > > On Mon, 2020-05-25 at 19:49 +0200, Alexis Ballier wrote:
> > > > On Mon, 25 May 2020 11:26:26 -0400
> > > > Mike Gilbert <floppym@gentoo.org> wrote:
> > > >
> > > > > On Mon, May 25, 2020 at 9:13 AM Alexis Ballier <
> > > > > aballier@gentoo.org>
> > > > > wrote:
> > > > > > On Sun, 24 May 2020 20:25:11 +0000 (UTC)
> > > > > > "Thomas Deutschmann" <whissi@gentoo.org> wrote:
> > > > > >
> > > > > > > commit:     6e149596cc76f1bbcee6720828c8c8c92420f2a3
> > > > > > > Author:     Thomas Deutschmann <whissi <AT> gentoo <DOT>
> > > > > > > org>
> > > > > > > AuthorDate: Sun May 24 19:47:08 2020 +0000
> > > > > > > Commit:     Thomas Deutschmann <whissi <AT> gentoo <DOT>
> > > > > > > org>
> > > > > > > CommitDate: Sun May 24 20:23:53 2020 +0000
> > > > > > > URL:
> > > > > > > https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=6e149596
> > > > > > >
> > > > > > > media-libs/x265: drop USE=pic
> > > > > > >
> > > > > > > Gentoo's toolchain uses PIC by default. Since USE=asm was
> > > > > > > added,
> > > > > > > we no longer need a USE flag to control that behavior.
> > > > > >
> > > > > > You got it wrong here it seems: USE=pic does not control
> > > > > > whether
> > > > > > the toolchain produces PIC or not. Shared libs always are,
> > > > > > and have
> > > > > > always been, built that way on Gentoo.
> > > > > > In this case, USE=pic means "no matter what it costs, I do
> > > > > > not want
> > > > > > textrels", for the cases of hand written assembly that has to
> > > > > > be
> > > > > > rewritten to support PIC. And, still in this case, this costs
> > > > > > a lot
> > > > > > of performance, so it is enabled by default on hardened
> > > > > > profiles
> > > > > > and not others.
> > > > > > Textrels work fine (on some architectures), they disallow W^X
> > > > > > and
> > > > > > force each process using the shared lib to make a "copy" at
> > > > > > runtime
> > > > > > in order to resolve relocations, so are not desirable but
> > > > > > sometimes
> > > > > > the cost outweights the gain.
> > > > > >
> > > > > > Plus, profiles/features/hardened enables pic by default but
> > > > > > knows
> > > > > > nothing about USE=asm so this is a regression for them.
> > > > >
> > > > > The USE flag toggles use of assembly, not use of PIC. The
> > > > > default USE
> > > > > value in the hardened profile should not drive decisions on
> > > > > what we
> > > > > name USE flags.
> > > >
> > > > ... but using a global well documented useflag instead of a local
> > > > invention should drive such decisions.
> > >
> > > What 'global well documented useflag'?
> >
> > It's neither global, nor well-documented, but several packages do
> > define it locally.
> >
>
> https://gitweb.gentoo.org/repo/gentoo/historical.git/commit/profiles/use.desc?id=103236c295aa30e5e42cfc8a7429e4eea5f0d680
>
> https://gitweb.gentoo.org/repo/gentoo/historical.git/commit/profiles/use.desc?id=784deb7134b9d430546557a8f8a0877bf35c02ba
>
> I guess this hasn't been really discussed back then.
>
> It is also used in a global way in profiles (make.defaults).
>
> > Personally, I think it should be renamed to "asm" or something
> > similar
> > in the majority of cases where it actually disables all use of
> > assembly code.
>
> Thankfully these days there's usually no need to disable asm to have
> pic. hardened has no mention of that flag, and I think that e.g. for
> openssl they would have noticed long ago.
> And again, 'asm' as a useflag makes no sense: if it works and simply
> replaces a C function by a faster one then it shouldn't even be an
> useflag. 'pic' on the other hand conveys the tradeoff idea.

If I understand you correctly, we should just drop the USE="pic" logic
from the remaining packages that have it? Or are you trying to say
something else?

Re: [gentoo-dev] Re: [gentoo-commits] repo/gentoo:master commit in: media-libs/x265/

[Alexis Ballier]

On Mon, 2020-05-25 at 21:09 -0400, Mike Gilbert wrote:
> On Mon, May 25, 2020 at 7:35 PM Alexis Ballier <aballier@gentoo.org>
> wrote:
> > On Mon, 2020-05-25 at 17:04 -0400, Mike Gilbert wrote:
> > > On Mon, May 25, 2020 at 3:18 PM Michał Górny <mgorny@gentoo.org>
> > > wrote:
> > > > On Mon, 2020-05-25 at 19:49 +0200, Alexis Ballier wrote:
> > > > > On Mon, 25 May 2020 11:26:26 -0400
> > > > > Mike Gilbert <floppym@gentoo.org> wrote:
> > > > > 
> > > > > > On Mon, May 25, 2020 at 9:13 AM Alexis Ballier <
> > > > > > aballier@gentoo.org>
> > > > > > wrote:
> > > > > > > On Sun, 24 May 2020 20:25:11 +0000 (UTC)
> > > > > > > "Thomas Deutschmann" <whissi@gentoo.org> wrote:
> > > > > > > 
> > > > > > > > commit:     6e149596cc76f1bbcee6720828c8c8c92420f2a3
> > > > > > > > Author:     Thomas Deutschmann <whissi <AT> gentoo
> > > > > > > > <DOT>
> > > > > > > > org>
> > > > > > > > AuthorDate: Sun May 24 19:47:08 2020 +0000
> > > > > > > > Commit:     Thomas Deutschmann <whissi <AT> gentoo
> > > > > > > > <DOT>
> > > > > > > > org>
> > > > > > > > CommitDate: Sun May 24 20:23:53 2020 +0000
> > > > > > > > URL:
> > > > > > > > https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=6e149596
> > > > > > > > 
> > > > > > > > media-libs/x265: drop USE=pic
> > > > > > > > 
> > > > > > > > Gentoo's toolchain uses PIC by default. Since USE=asm
> > > > > > > > was
> > > > > > > > added,
> > > > > > > > we no longer need a USE flag to control that behavior.
> > > > > > > 
> > > > > > > You got it wrong here it seems: USE=pic does not control
> > > > > > > whether
> > > > > > > the toolchain produces PIC or not. Shared libs always
> > > > > > > are,
> > > > > > > and have
> > > > > > > always been, built that way on Gentoo.
> > > > > > > In this case, USE=pic means "no matter what it costs, I
> > > > > > > do
> > > > > > > not want
> > > > > > > textrels", for the cases of hand written assembly that
> > > > > > > has to
> > > > > > > be
> > > > > > > rewritten to support PIC. And, still in this case, this
> > > > > > > costs
> > > > > > > a lot
> > > > > > > of performance, so it is enabled by default on hardened
> > > > > > > profiles
> > > > > > > and not others.
> > > > > > > Textrels work fine (on some architectures), they disallow
> > > > > > > W^X
> > > > > > > and
> > > > > > > force each process using the shared lib to make a "copy"
> > > > > > > at
> > > > > > > runtime
> > > > > > > in order to resolve relocations, so are not desirable but
> > > > > > > sometimes
> > > > > > > the cost outweights the gain.
> > > > > > > 
> > > > > > > Plus, profiles/features/hardened enables pic by default
> > > > > > > but
> > > > > > > knows
> > > > > > > nothing about USE=asm so this is a regression for them.
> > > > > > 
> > > > > > The USE flag toggles use of assembly, not use of PIC. The
> > > > > > default USE
> > > > > > value in the hardened profile should not drive decisions on
> > > > > > what we
> > > > > > name USE flags.
> > > > > 
> > > > > ... but using a global well documented useflag instead of a
> > > > > local
> > > > > invention should drive such decisions.
> > > > 
> > > > What 'global well documented useflag'?
> > > 
> > > It's neither global, nor well-documented, but several packages do
> > > define it locally.
> > > 
> > 
> > https://gitweb.gentoo.org/repo/gentoo/historical.git/commit/profiles/use.desc?id=103236c295aa30e5e42cfc8a7429e4eea5f0d680
> > 
> > https://gitweb.gentoo.org/repo/gentoo/historical.git/commit/profiles/use.desc?id=784deb7134b9d430546557a8f8a0877bf35c02ba
> > 
> > I guess this hasn't been really discussed back then.
> > 
> > It is also used in a global way in profiles (make.defaults).
> > 
> > > Personally, I think it should be renamed to "asm" or something
> > > similar
> > > in the majority of cases where it actually disables all use of
> > > assembly code.
> > 
> > Thankfully these days there's usually no need to disable asm to
> > have
> > pic. hardened has no mention of that flag, and I think that e.g.
> > for
> > openssl they would have noticed long ago.
> > And again, 'asm' as a useflag makes no sense: if it works and
> > simply
> > replaces a C function by a faster one then it shouldn't even be an
> > useflag. 'pic' on the other hand conveys the tradeoff idea.
> 
> If I understand you correctly, we should just drop the USE="pic"
> logic
> from the remaining packages that have it? Or are you trying to say
> something else?


Drop USE=asm unless there's a real reason to it: Such a useflag is,
IMHO, at the same level of a useflag on dev-lang/python that would
toggle dict's underlying implementations but not the semantics of the
language.
Have USE=pic for its historical meaning, aka, sacrificing everything to
have PIC shared libs because your system enforces this (pax).

Note that having the 'pic' useflag should be considered something to be
fixed: rewrite the asm in a PIC way. But these days nobody has the will
to do it since this is mostly an issue on x86+pax, both being slowly
decreasing.

Re: [gentoo-dev] Re: [gentoo-commits] repo/gentoo:master commit in: media-libs/x265/

[Alexis Ballier]

[...]
> > If I understand you correctly, we should just drop the USE="pic"
> > logic
> > from the remaining packages that have it? Or are you trying to say
> > something else?
> 

[...]

> Note that having the 'pic' useflag should be considered something to
> be
> fixed: rewrite the asm in a PIC way. But these days nobody has the
> will
> to do it since this is mostly an issue on x86+pax, both being slowly
> decreasing.


As a side note on this: if USE=-pic has no textrel then obviously the
useflag should be removed. This can happen over time if we don't pay
enough attention.

Re: [gentoo-dev] Re: [gentoo-commits] repo/gentoo:master commit in: media-libs/x265/

[Mike Gilbert]

On Tue, May 26, 2020 at 5:30 AM Alexis Ballier <aballier@gentoo.org> wrote:
> On Mon, 2020-05-25 at 21:09 -0400, Mike Gilbert wrote:
> > If I understand you correctly, we should just drop the USE="pic"
> > logic
> > from the remaining packages that have it? Or are you trying to say
> > something else?
>
>
> Drop USE=asm unless there's a real reason to it: Such a useflag is,
> IMHO, at the same level of a useflag on dev-lang/python that would
> toggle dict's underlying implementations but not the semantics of the
> language.
> Have USE=pic for its historical meaning, aka, sacrificing everything to
> have PIC shared libs because your system enforces this (pax).

Thanks, this last sentence gives me a better understanding of why this
"pic" USE flag exists at all.

> Note that having the 'pic' useflag should be considered something to be
> fixed: rewrite the asm in a PIC way. But these days nobody has the will
> to do it since this is mostly an issue on x86+pax, both being slowly
> decreasing.

Given that PaX has been stripped out of official Gentoo kernels due to
the grsecurity licensing issue, I wonder if there is any other good
reason to keep the "pic" USE flag today. Surely this affects a very
small population of users.

Re: [gentoo-dev] Re: [gentoo-commits] repo/gentoo:master commit in: media-libs/x265/

[Alexis Ballier]

On Tue, 26 May 2020 10:45:39 -0400
Mike Gilbert <floppym@gentoo.org> wrote:

> > Note that having the 'pic' useflag should be considered something
> > to be fixed: rewrite the asm in a PIC way. But these days nobody
> > has the will to do it since this is mostly an issue on x86+pax,
> > both being slowly decreasing.
> 
> Given that PaX has been stripped out of official Gentoo kernels due to
> the grsecurity licensing issue, I wonder if there is any other good
> reason to keep the "pic" USE flag today. Surely this affects a very
> small population of users.
> 

Yeah that was my thought when I saw pax/grsec beginning to be more
hostile to open source. That's not my call but the hardened team's,
however I'm all for removing these workarounds entirely if there's no
point in having them anymore.

Re: [gentoo-dev] Re: [gentoo-commits] repo/gentoo:master commit in: media-libs/x265/

[Alexis Ballier]

On Tue, 26 May 2020 10:45:39 -0400
Mike Gilbert <floppym@gentoo.org> wrote:
> > Note that having the 'pic' useflag should be considered something
> > to be fixed: rewrite the asm in a PIC way. But these days nobody
> > has the will to do it since this is mostly an issue on x86+pax,
> > both being slowly decreasing.
> 
> Given that PaX has been stripped out of official Gentoo kernels due to
> the grsecurity licensing issue, I wonder if there is any other good
> reason to keep the "pic" USE flag today. Surely this affects a very
> small population of users.


I couldn't find any recent reference, but PIC shared libs used to be a
QA policy. There's mainly two reasons to it: First is W^X enforcement;
non PIC shared libs are refused by the x86_64 linker so a non-issue
there, on x86 you need pax to emulate it because the mmu doesn't support
the X part; I don't know about other arches.
Then there is the small memory waste done because those libs will be
loaded COW and thus their "code" is not shared anymore between
processes. And the small startup performance hit to
perform the relocations.

The latter part affects everyone, and the rule of thumb for having a
pic useflag (instead of always pic) is that the gain for non-pic asm is
better than the loss of non-pic shared libs. This is subjective but
usually a no-brainer for multimedia packages.

This is probably something to bring up to QA too.

Re: [gentoo-dev] Re: [gentoo-commits] repo/gentoo:master commit in: media-libs/x265/

[Mike Gilbert]

On Tue, May 26, 2020 at 2:03 PM Alexis Ballier <aballier@gentoo.org> wrote:
>
> On Tue, 26 May 2020 10:45:39 -0400
> Mike Gilbert <floppym@gentoo.org> wrote:
> > > Note that having the 'pic' useflag should be considered something
> > > to be fixed: rewrite the asm in a PIC way. But these days nobody
> > > has the will to do it since this is mostly an issue on x86+pax,
> > > both being slowly decreasing.
> >
> > Given that PaX has been stripped out of official Gentoo kernels due to
> > the grsecurity licensing issue, I wonder if there is any other good
> > reason to keep the "pic" USE flag today. Surely this affects a very
> > small population of users.
>
>
> I couldn't find any recent reference, but PIC shared libs used to be a
> QA policy. There's mainly two reasons to it: First is W^X enforcement;
> non PIC shared libs are refused by the x86_64 linker so a non-issue
> there, on x86 you need pax to emulate it because the mmu doesn't support
> the X part; I don't know about other arches.
> Then there is the small memory waste done because those libs will be
> loaded COW and thus their "code" is not shared anymore between
> processes. And the small startup performance hit to
> perform the relocations.
>
> The latter part affects everyone, and the rule of thumb for having a
> pic useflag (instead of always pic) is that the gain for non-pic asm is
> better than the loss of non-pic shared libs. This is subjective but
> usually a no-brainer for multimedia packages.

Assuming that the pic performance penalty is really only relevant on
legacy arches (like x86), here are a couple of options:

1. Disable pic on arches where tie performance penalty is small.
2. Force pic everywhere, performance be damned.

Re: [gentoo-dev] Re: [gentoo-commits] repo/gentoo:master commit in: media-libs/x265/

[Mike Gilbert]

On Tue, May 26, 2020 at 2:13 PM Mike Gilbert <floppym@gentoo.org> wrote:
>
> On Tue, May 26, 2020 at 2:03 PM Alexis Ballier <aballier@gentoo.org> wrote:
> >
> > On Tue, 26 May 2020 10:45:39 -0400
> > Mike Gilbert <floppym@gentoo.org> wrote:
> > > > Note that having the 'pic' useflag should be considered something
> > > > to be fixed: rewrite the asm in a PIC way. But these days nobody
> > > > has the will to do it since this is mostly an issue on x86+pax,
> > > > both being slowly decreasing.
> > >
> > > Given that PaX has been stripped out of official Gentoo kernels due to
> > > the grsecurity licensing issue, I wonder if there is any other good
> > > reason to keep the "pic" USE flag today. Surely this affects a very
> > > small population of users.
> >
> >
> > I couldn't find any recent reference, but PIC shared libs used to be a
> > QA policy. There's mainly two reasons to it: First is W^X enforcement;
> > non PIC shared libs are refused by the x86_64 linker so a non-issue
> > there, on x86 you need pax to emulate it because the mmu doesn't support
> > the X part; I don't know about other arches.
> > Then there is the small memory waste done because those libs will be
> > loaded COW and thus their "code" is not shared anymore between
> > processes. And the small startup performance hit to
> > perform the relocations.
> >
> > The latter part affects everyone, and the rule of thumb for having a
> > pic useflag (instead of always pic) is that the gain for non-pic asm is
> > better than the loss of non-pic shared libs. This is subjective but
> > usually a no-brainer for multimedia packages.
>
> Assuming that the pic performance penalty is really only relevant on
> legacy arches (like x86), here are a couple of options:
>
> 1. Disable pic on arches where tie performance penalty is small.
> 2. Force pic everywhere, performance be damned.

Option 1 should say "Disable pic on arches where the performance
penalty is large."

Re: [gentoo-dev] Re: [gentoo-commits] repo/gentoo:master commit in: media-libs/x265/

[zorry]

tisdag 26 maj 2020 kl. 19:54:51 CEST skrev  Alexis Ballier:
> On Tue, 26 May 2020 10:45:39 -0400
> 
> Mike Gilbert <floppym@gentoo.org> wrote:
> > > Note that having the 'pic' useflag should be considered something
> > > to be fixed: rewrite the asm in a PIC way. But these days nobody
> > > has the will to do it since this is mostly an issue on x86+pax,
> > > both being slowly decreasing.
> > 
> > Given that PaX has been stripped out of official Gentoo kernels due to
> > the grsecurity licensing issue, I wonder if there is any other good
> > reason to keep the "pic" USE flag today. Surely this affects a very
> > small population of users.
> 
> Yeah that was my thought when I saw pax/grsec beginning to be more
> hostile to open source. That's not my call but the hardened team's,
> however I'm all for removing these workarounds entirely if there's no
> point in having them anymore.
Is not only PaX/Grsec that don't allow textrel. SELinux do it to and that is 
manline kernel.

Re: [gentoo-dev] Re: [gentoo-commits] repo/gentoo:master commit in: media-libs/x265/

[Alexis Ballier]

On Tue, 2020-05-26 at 14:14 -0400, Mike Gilbert wrote:
[...]
> 
> > 
> > Assuming that the pic performance penalty is really only relevant
> > on
> > legacy arches (like x86), here are a couple of options:
> > 
> > 1. Disable pic on arches where tie performance penalty is small.
> > 2. Force pic everywhere, performance be damned.
> 
> Option 1 should say "Disable pic on arches where the performance
> penalty is large."

Performance penalty is not really a matter of arch. Of course -fPIC
from C/C++ code has a small performance impact on x86 (bigger than on
amd64), but the gain is worth it. Here it is a matter of hand written
assembly that is not relocatable. The performance impact thus depends
on the gain of said assembly, which in the case of multimedia apps/libs
is huge because that's usually about vectorizing inner loops bodys
(somewhere between x4 or x8 speedups is not unusual for the part in
question).

That's why the policy has always been to have PIC everywhere with
exceptions/workarounds tied to the pic useflag, giving users and
profiles the choice.

Again, the real fix is to rewrite the assembly in a PIC way, or at
least have ifdefs to allow it, but that is hard. Packages that have the
option to have PIC or slightly faster non-PIC assembly do/should not
have a pic useflag and always use the PIC variant.

Re: [gentoo-dev] Re: [gentoo-commits] repo/gentoo:master commit in: media-libs/x265/

[Alexis Ballier]

On Tue, 26 May 2020 23:41:28 +0200
zorry@gentoo.org wrote:

> tisdag 26 maj 2020 kl. 19:54:51 CEST skrev  Alexis Ballier:
> > On Tue, 26 May 2020 10:45:39 -0400
> > 
> > Mike Gilbert <floppym@gentoo.org> wrote:
> > > > Note that having the 'pic' useflag should be considered
> > > > something to be fixed: rewrite the asm in a PIC way. But these
> > > > days nobody has the will to do it since this is mostly an issue
> > > > on x86+pax, both being slowly decreasing.
> > > 
> > > Given that PaX has been stripped out of official Gentoo kernels
> > > due to the grsecurity licensing issue, I wonder if there is any
> > > other good reason to keep the "pic" USE flag today. Surely this
> > > affects a very small population of users.
> > 
> > Yeah that was my thought when I saw pax/grsec beginning to be more
> > hostile to open source. That's not my call but the hardened team's,
> > however I'm all for removing these workarounds entirely if there's
> > no point in having them anymore.
> Is not only PaX/Grsec that don't allow textrel. SELinux do it to and
> that is manline kernel.


k so that means there's no dropping the pic useflag then i guess