[gentoo-dev] Review: news item and script for CPU_FLAGS_X86

[gentoo-dev] Review: news item and script for CPU_FLAGS_X86

[Michał Górny]

[-- Attachment #1.1: Type: text/plain, Size: 861 bytes --]

Dnia 2015-01-18, o godz. 21:44:05
Michał Górny <mgorny@gentoo.org> napisał(a):

> Hello,
> 
> I would like to commit the following flags as cpu_flags_x86_desc.
> The list combines global USE flags with some local USE flags I've been
> able to find.

Following your suggestions, I'm attaching three files:

1. updated cpu_flags_x86.desc, with:

 a. fma3/fma4 distinction,

 b. aes-ni renamed to aes for consistency with cpuinfo, we no longer
 have to worry about USE flag collision with other USE=aes uses,

 c. complete and machine-parseable listing of cpuinfo flags.

2. cpuinfo2cpuflags-x86.py script that parses cpu_flags_x86.desc
and /proc/cpuinfo and generates a nice CPU_FLAGS_X86 value for you. We
need to decide where to put it (scripts/?).

3. Initial text of the news item for review.

-- 
Best regards,
Michał Górny

[-- Attachment #1.2: cpu_flags_x86.desc --]
[-- Type: application/octet-stream, Size: 1361 bytes --]

# Copyright 1999-2014 Gentoo Foundation.
# Distributed under the terms of the GNU General Public License v2
# $Header$

# Whenever the flag name does not correspond to /proc/cpuinfo flags,
# please put the cpuinfo flag in square brackets. If more than one flag
# is specified, any of them will enable the respective USE flag.

3dnow - Use the 3DNow! instruction set
3dnowext - Use the Enhanced 3DNow! instruction set
aes - Enable support for Intel's AES instruction set (AES-NI)
avx - Adds support for Advanced Vector Extensions instructions
avx2 - Adds support for Advanced Vector Extensions 2 instructions
fma3 - Use the Fused Multiply Add 3 instruction set ([fma] in cpuinfo)
fma4 - Use the Fused Multiply Add 4 instruction set
mmx - Use the MMX instruction set
mmxext - Use the Extended MMX instruction set (intersection of Enhanced 3DNow! and SSE instruction sets) ([mmxext], [3dnowext] or [sse] in cpuinfo)
padlock - Use VIA padlock instructions ([phe] in cpuinfo)
popcnt - Enable popcnt instruction support ([abm] or [popcnt] in cpuinfo)
sse - Use the SSE instruction set
sse2 - Use the SSE2 instruction set
sse3 - Use the SSE3 instruction set ([pni] in cpuinfo, NOT ssse3)
sse4_1 - Enable SSE4.1 instruction support
sse4_2 - Enable SSE4.2 instruction support
sse4a - Enable SSE4a instruction support
ssse3 - Use the SSSE3 instruction set (NOT sse3/pni)

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.3: cpuinfo2cpuflags-x86.py --]
[-- Type: text/x-python, Size: 1860 bytes --]

#!/usr/bin/env python

import collections, errno, os, os.path, re, sys

def main(*argv):
	if 'PORTDIR' in os.environ:
		portdir = os.environ['PORTDIR']
	else:
		try:
			import portage
		except ImportError:
			return 'Unable to determine PORTDIR, please set it before calling this script'
		else:
			trees = portage.create_trees(
					config_root = os.environ.get('PORTAGE_CONFIGROOT'),
					target_root = os.environ.get('ROOT'))
			tree = trees[max(trees)]
			portdir = tree['porttree'].dbapi.repositories.get_location_for_name('gentoo')

	# dict of cpuinfo -> flag mappings
	flag_dict = collections.defaultdict(list)

	desc = os.path.join(portdir, 'profiles', 'desc', 'cpu_flags_x86.desc')
	# flag - description ; discards empty lines and comments
	l_regexp = re.compile(r'^(?P<flag>[a-z0-9_]+)\s+-\s+(?P<desc>.+)$')
	# [cpuinfo-flag] ; [] are discarded for easy use
	cpuinfo_regexp = re.compile(r'(?<=\[).+?(?=\])')
	try:
		with open(desc, 'r') as f:
			for l in f:
				m = l_regexp.match(l)
				if m:
					# grep for any [cpuinfo] or [cpuinfo2]
					flags = cpuinfo_regexp.findall(l)
					for fl in flags:
						flag_dict[fl].append(m.group('flag'))
					# fallback to exact flag match
					if not flags:
						flag_dict[m.group('flag')].append(m.group('flag'))
	except OSError as e:
		if e.errno == errno.ENOENT:
			return '%s does not exist, set PORTDIR to correct gentoo repository location' % desc

	out = set()

	flags_regexp = re.compile(r'^flags\s+:\s+(?P<flags>.*)$')
	with open('/proc/cpuinfo', 'r') as f:
		for l in f:
			m = flags_regexp.match(l)
			if m:
				for fl in m.group('flags').split():
					if fl in flag_dict:
						out.update(flag_dict[fl])
				break

	print('CPU_FLAGS_X86="%s"' % ' '.join(sorted(out)))
	return 0

if __name__ == '__main__':
	sys.exit(main(*sys.argv[1:]))

[-- Attachment #1.4: 2015-01-xx-CPU_FLAGS_X86-introduction.en.txt --]
[-- Type: text/plain, Size: 1127 bytes --]

Title: CPU_FLAGS_X86 introduction
Author: Michał Górny <mgorny@gentoo.org>
Content-Type: text/plain
Posted: 2015-01-xx
Revision: 1
News-Item-Format: 1.0
Display-If-Keyword: amd64 ~amd64 x86 ~x86

The USE flags corresponding to intruction sets and other features
specific to the x86 architecture are being moved into a separate USE
flag group called CPU_FLAGS_X86.

In order not to lose CPU-specific optimizations, users will be required
to update their make.conf (and package.use) file. For example, if
the following USE flags were present:

  USE="mmx mmxext sse sse2 sse3"

Those flags need to be copied into:

  CPU_FLAGS_X86="mmx mmxext sse sse2 sse3"

When in doubt, consult profiles/desc/cpu_flags_x86.desc. Most of
the flag names match /proc/cpuinfo names, with the notable exception
of SSE3 which is called 'pni' in /proc/cpuinfo (and which isn't the same
as SSSE3).

To help you enable correct USE flags, you can use the
cpuinfo2cpuflags-x86.py script provided in scripts/ directory of the
repository. It will parse your /proc/cpuinfo and print an accurate
CPU_FLAGS_X86 value for it.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] Review: news item and script for CPU_FLAGS_X86

[Alexis Ballier]

On Tue, 20 Jan 2015 10:40:17 +0100
Michał Górny <mgorny@gentoo.org> wrote:

> # Copyright 1999-2014 Gentoo Foundation.
> # Distributed under the terms of the GNU General Public License v2
> # $Header$
> 
> # Whenever the flag name does not correspond to /proc/cpuinfo flags,
> # please put the cpuinfo flag in square brackets. If more than one
> flag # is specified, any of them will enable the respective USE flag.
> 

nice


now, letmegrepitforyou:


> avx - Adds support for Advanced Vector Extensions instructions

media-video/ffmpeg:avx - Adds support for Advanced Vector Extensions
instructions (Intel Sandy Bridge, AMD Bulldozer and later chips)

> avx2 - Adds support for Advanced Vector Extensions 2 instructions

media-video/ffmpeg:avx2 - Adds support for Advanced Vector Extensions 2
instructions (Intel Haswell and later chips)

> fma3 - Use the Fused Multiply Add 3 instruction set ([fma] in cpuinfo)

media-video/ffmpeg:fma3 - Enables FMA3 optimizations: AMD processors
starting with Piledriver architecture and Intel Haswell based
processors or later.

> fma4 - Use the Fused Multiply Add 4 instruction set

media-video/ffmpeg:fma4 - Enables FMA4 optimizations: AMD processors
starting with Bulldozer architecture.

> sse4_1 - Enable SSE4.1 instruction support

media-libs/libvpx:sse4_1 - Enable optimization for SSE4_1 capable
processors (Intel Core 2 Penryn and later chips)


> sse4_2 - Enable SSE4.2 instruction support

media-video/ffmpeg:sse4_2 - Enables SSE4.2 optimizations: Nehalem-based
Intel Core i7 or later.

> ssse3 - Use the SSSE3 instruction set (NOT sse3/pni)

media-libs/libvpx:ssse3 - faster floating point optimization for SSSE3
capable chips (Intel Core 2 and later chips)






I like the "available in cpu $foo and later" because this influences me
in what to buy: If I know that I will have noticeable gain with hand
written asm in a package that is critical to me, I know it will perform
better for my needs despite benchmarks might be saying otherwise.


Alexis.

[gentoo-dev] Re: Review: news item and script for CPU_FLAGS_X86

[Duncan]

Michał Górny posted on Tue, 20 Jan 2015 10:40:17 +0100 as excerpted:

> Display-If-Keyword: amd64 ~amd64 x86 ~x86
> 
> The USE flags corresponding to intruction sets and other features
> specific to the x86 architecture are being moved into a separate USE
> flag group called CPU_FLAGS_X86.

So based on this preliminary news item, I decided to get ahead of the 
game and add the use_expand to the appropriate make.conf sourced file...

... and IMMEDIATELY ran into this:

CPU_FLAGS_X86 ??  I'm ~amd64, do I leave it as-is, change it to 
CPU_FLAGS_X86_64 (kernel style), or change it to CPU_FLAGS_AMD64 (gentoo 
style)?

Knowing what I know about gentoo's use of use_expand, I'm almost certain 
I leave it as-is: CPU_FLAGS_X86 .  But if it's triggering the slightest 
doubt and hesitation here, and it is, it's certain to be a rather larger 
source of confusion to many.  So...

Please update the news item to specify that yes, it's _X86 for amd64 
users as well (assuming my near-certainty above is correct, else make 
explicit whatever actually applies).

Here's a second-draft pass at more explicit wording (also corrects a 
typo, "intruction", that spell-check flagged when I select-pasted):

The USE flags corresponding to instruction sets and other features
specific to the x86 (and amd64) architecture are being moved into a 
separate USE flag group called CPU_FLAGS_X86.  (Because amd64 is an 
extension of x86, amd64 users use the same group, CPU_FLAGS_X86.)


Additionally, unless the change is going to be atomic across the tree, 
including existing stable packages /and/ the news item at exactly the 
same time, I'd suggest an additional paragraph in the news item, 
suggesting keeping the existing USE flags for some period (the 
traditional 1-year supported update time period?) as well.  First-draft 
wording (please suggest updates, I'm not entirely comfortable with this 
but it suffices for first-draft):

In ordered to cover the transition period, you may wish to keep existing 
copies of any USE flags transfered to CPU_FLAGS_X86 for a suggested 
period of one year, or until you've merged updates with the new 
use_expand form of all affected packages, after which you can remove the 
old form from your normal USE.


Thanks.  The extra work that this whole change entails is appreciated! 
=:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: Review: news item and script for CPU_FLAGS_X86

[Michał Górny]

[-- Attachment #1.1: Type: text/plain, Size: 615 bytes --]

Dnia 2015-01-21, o godz. 00:18:50
Duncan <1i5t5.duncan@cox.net> napisał(a):

> Michał Górny posted on Tue, 20 Jan 2015 10:40:17 +0100 as excerpted:
> 
> > Display-If-Keyword: amd64 ~amd64 x86 ~x86
> > 
> > The USE flags corresponding to intruction sets and other features
> > specific to the x86 architecture are being moved into a separate USE
> > flag group called CPU_FLAGS_X86.
> 
> So based on this preliminary news item, I decided to get ahead of the 
> game and add the use_expand to the appropriate make.conf sourced file...

Attaching a new version.

-- 
Best regards,
Michał Górny

[-- Attachment #1.2: 2015-01-xx-CPU_FLAGS_X86-introduction.en.txt --]
[-- Type: text/plain, Size: 1571 bytes --]

Title: CPU_FLAGS_X86 introduction
Author: Michał Górny <mgorny@gentoo.org>
Content-Type: text/plain
Posted: 2015-01-xx
Revision: 1
News-Item-Format: 1.0
Display-If-Keyword: amd64 ~amd64 x86 ~x86

The USE flags corresponding to intruction sets and other features
specific to the x86 architecture are being moved into a separate USE
flag group called CPU_FLAGS_X86.

In order not to lose CPU-specific optimizations, users will be required
to update their make.conf (and package.use) file. For example, if
the following USE flags were present:

  USE="mmx mmxext sse sse2 sse3"

Those flags need to be copied into:

  CPU_FLAGS_X86="mmx mmxext sse sse2 sse3"

Please note that CPU_FLAGS_X86 is used both on x86 and amd64 systems.

When in doubt, please consult profiles/desc/cpu_flags_x86.desc. Most of
the flag names match /proc/cpuinfo names, with the notable exception
of SSE3 which is called 'pni' in /proc/cpuinfo (please also do not
confuse it with distinct SSSE3).

To help you enable the correct USE flags, we are providing a Python
script which generates the correct value from your /proc/cpuinfo [1].
The Python script can be downloaded and executed using the following
command:

  $ wget -O - dev.gentoo.org/~mgorny/cpuinfo2cpuflags-x86.py | python

In order to ensure safe migration and maintain compatibility with
external repositories, it is recommended to preserve the old USE
settings for a period of one year or until no package of interest is
still using them.

[1]:https://bitbucket.org/mgorny/cpuinfo2cpuflags

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] Re: Review: news item and script for CPU_FLAGS_X86

[Michael Orlitzky]

On 01/23/2015 02:13 PM, Michał Górny wrote:
> To help you enable the correct USE flags, we are providing a Python
> script which generates the correct value from your /proc/cpuinfo [1].
> The Python script can be downloaded and executed using the following
> command:
> 
>   $ wget -O - dev.gentoo.org/~mgorny/cpuinfo2cpuflags-x86.py | python

Can we not encourage people to pipe stuff from a plain-http website into
an interpreter?

But that's not the biggest problem with the wget thing: the first time
someone runs this it's going to fail.

  $ wget -O - dev.gentoo.org/~mgorny/cpuinfo2cpuflags-x86.py | python
  ...
  Unable to determine PORTDIR, please set it before calling this script

Now they get to download it again. Perhaps a bigger problem:

  $ echo $PORTDIR
  /var/cache/portage/repositories/gentoo
  $ python cpuinfo2cpuflags-x86.py
  Unable to determine PORTDIR, please set it before calling this script

ಠ_ಠ

Re: [gentoo-dev] Re: Review: news item and script for CPU_FLAGS_X86

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1205 bytes --]

Dnia 2015-01-23, o godz. 14:26:48
Michael Orlitzky <mjo@gentoo.org> napisał(a):

> On 01/23/2015 02:13 PM, Michał Górny wrote:
> > To help you enable the correct USE flags, we are providing a Python
> > script which generates the correct value from your /proc/cpuinfo [1].
> > The Python script can be downloaded and executed using the following
> > command:
> > 
> >   $ wget -O - dev.gentoo.org/~mgorny/cpuinfo2cpuflags-x86.py | python
> 
> Can we not encourage people to pipe stuff from a plain-http website into
> an interpreter?

Find a better solution.

> But that's not the biggest problem with the wget thing: the first time
> someone runs this it's going to fail.
> 
>   $ wget -O - dev.gentoo.org/~mgorny/cpuinfo2cpuflags-x86.py | python
>   ...
>   Unable to determine PORTDIR, please set it before calling this script

Don't you have Portage installed?

> Now they get to download it again. Perhaps a bigger problem:
> 
>   $ echo $PORTDIR
>   /var/cache/portage/repositories/gentoo
>   $ python cpuinfo2cpuflags-x86.py
>   Unable to determine PORTDIR, please set it before calling this script

Did you export PORTDIR?

-- 
Best regards,
Michał Górny

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] Re: Review: news item and script for CPU_FLAGS_X86

[Michael Orlitzky]

On 01/23/2015 03:22 PM, Michał Górny wrote:
> Dnia 2015-01-23, o godz. 14:26:48
> Michael Orlitzky <mjo@gentoo.org> napisał(a):
> 
>> On 01/23/2015 02:13 PM, Michał Górny wrote:
>>> To help you enable the correct USE flags, we are providing a Python
>>> script which generates the correct value from your /proc/cpuinfo [1].
>>> The Python script can be downloaded and executed using the following
>>> command:
>>>
>>>   $ wget -O - dev.gentoo.org/~mgorny/cpuinfo2cpuflags-x86.py | python
>>
>> Can we not encourage people to pipe stuff from a plain-http website into
>> an interpreter?
> 
> Find a better solution.
> 

Is there an easy way for users to verify our signatures against the keys
in LDAP?

Even `wget --no-check-certificate` would be a big improvement. Or since
Firefox seems happy with the dev.gentoo.org certificate, we could just
ask them to download it with their browsers.

Longer term: can we make wget like our SSL certificate?


>> But that's not the biggest problem with the wget thing: the first time
>> someone runs this it's going to fail.
>>
>>   $ wget -O - dev.gentoo.org/~mgorny/cpuinfo2cpuflags-x86.py | python
>>   ...
>>   Unable to determine PORTDIR, please set it before calling this script
> 
> Don't you have Portage installed?

I do.

  $ emerge --info | grep PORTDIR
  PORTDIR="/var/cache/portage/repositories/gentoo"


>> Now they get to download it again. Perhaps a bigger problem:
>>
>>   $ echo $PORTDIR
>>   /var/cache/portage/repositories/gentoo
>>   $ python cpuinfo2cpuflags-x86.py
>>   Unable to determine PORTDIR, please set it before calling this script
> 
> Did you export PORTDIR?
> 

No, but the error says to set it, not export it =)

Re: [gentoo-dev] Re: Review: news item and script for CPU_FLAGS_X86

[Michael Orlitzky]

On 01/23/2015 03:22 PM, Michał Górny wrote:
> 
>> But that's not the biggest problem with the wget thing: the first time
>> someone runs this it's going to fail.
>>
>>   $ wget -O - dev.gentoo.org/~mgorny/cpuinfo2cpuflags-x86.py | python
>>   ...
>>   Unable to determine PORTDIR, please set it before calling this script
> 
> Don't you have Portage installed?
> 

This one depends on the eselected version of python. I had python-3.4
eselected, but portage was built with the default
PYTHON_TARGETS="python2_7 python3_3".

Re: [gentoo-dev] Re: Review: news item and script for CPU_FLAGS_X86

[Gordon Pettey]

[-- Attachment #1: Type: text/plain, Size: 1409 bytes --]

On Fri, Jan 23, 2015 at 3:16 PM, Michael Orlitzky <mjo@gentoo.org> wrote:

> On 01/23/2015 03:22 PM, Michał Górny wrote:
> > Dnia 2015-01-23, o godz. 14:26:48
> > Michael Orlitzky <mjo@gentoo.org> napisał(a):
> >
> >> On 01/23/2015 02:13 PM, Michał Górny wrote:
> >>> To help you enable the correct USE flags, we are providing a Python
> >>> script which generates the correct value from your /proc/cpuinfo [1].
> >>> The Python script can be downloaded and executed using the following
> >>> command:
> >>>
> >>>   $ wget -O - dev.gentoo.org/~mgorny/cpuinfo2cpuflags-x86.py | python
> >>
> >> Can we not encourage people to pipe stuff from a plain-http website into
> >> an interpreter?
> >
> > Find a better solution.
>

If you trust GitHub, put it on a Gist, and it'll be accessible via HTTPS
and SSH.
If the "raw" URL is too ugly, and you trust Google, use the HTTPS version
of goo.gl to shorten it.


> Even `wget --no-check-certificate` would be a big improvement. Or since
> Firefox seems happy with the dev.gentoo.org certificate, we could just
> ask them to download it with their browsers.
>
> Longer term: can we make wget like our SSL certificate?


"DigiCert SHA2 High Assurance Server CA" is not in ca-certificates.
Funnily, DigiCert's download link for it is via plain HTTP so reasonable
paranoia demands manually verifying the chain after downloading.

[-- Attachment #2: Type: text/html, Size: 2380 bytes --]

Re: [gentoo-dev] Re: Review: news item and script for CPU_FLAGS_X86

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 761 bytes --]

On Fri, 23 Jan 2015 21:22:31 +0100 Michał Górny wrote:
> > > To help you enable the correct USE flags, we are providing a Python
> > > script which generates the correct value from your /proc/cpuinfo [1].
> > > The Python script can be downloaded and executed using the following
> > > command:
> > > 
> > >   $ wget -O - dev.gentoo.org/~mgorny/cpuinfo2cpuflags-x86.py | python
> > 
> > Can we not encourage people to pipe stuff from a plain-http website into
> > an interpreter?
> 
> Find a better solution.

Why not to ship this script together with portage (using some
USE=tools)? Alternatively it may be added as an independent ebuild
in app-admin (like we already have different cleaners and updaters).

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] Re: Review: news item and script for CPU_FLAGS_X86

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 969 bytes --]

Dnia 2015-01-24, o godz. 00:59:24
Andrew Savchenko <bircoph@gentoo.org> napisał(a):

> On Fri, 23 Jan 2015 21:22:31 +0100 Michał Górny wrote:
> > > > To help you enable the correct USE flags, we are providing a Python
> > > > script which generates the correct value from your /proc/cpuinfo [1].
> > > > The Python script can be downloaded and executed using the following
> > > > command:
> > > > 
> > > >   $ wget -O - dev.gentoo.org/~mgorny/cpuinfo2cpuflags-x86.py | python
> > > 
> > > Can we not encourage people to pipe stuff from a plain-http website into
> > > an interpreter?
> > 
> > Find a better solution.
> 
> Why not to ship this script together with portage (using some
> USE=tools)? Alternatively it may be added as an independent ebuild
> in app-admin (like we already have different cleaners and updaters).

Because it's a tiny script people are supposed to run once and be done
with it.

-- 
Best regards,
Michał Górny

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] Re: Review: news item and script for CPU_FLAGS_X86

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1693 bytes --]

Dnia 2015-01-23, o godz. 16:16:25
Michael Orlitzky <mjo@gentoo.org> napisał(a):

> On 01/23/2015 03:22 PM, Michał Górny wrote:
> > Dnia 2015-01-23, o godz. 14:26:48
> > Michael Orlitzky <mjo@gentoo.org> napisał(a):
> > 
> >> On 01/23/2015 02:13 PM, Michał Górny wrote:
> >>> To help you enable the correct USE flags, we are providing a Python
> >>> script which generates the correct value from your /proc/cpuinfo [1].
> >>> The Python script can be downloaded and executed using the following
> >>> command:
> >>>
> >>>   $ wget -O - dev.gentoo.org/~mgorny/cpuinfo2cpuflags-x86.py | python
> >>
> >> Can we not encourage people to pipe stuff from a plain-http website into
> >> an interpreter?
> > 
> > Find a better solution.
> > 
> 
> Is there an easy way for users to verify our signatures against the keys
> in LDAP?

No. At least not that I'm aware of. Maybe gentoo-keys has something.

> Even `wget --no-check-certificate` would be a big improvement.

...because?

> Or since
> Firefox seems happy with the dev.gentoo.org certificate, we could just
> ask them to download it with their browsers.

The link to the project page is below. If someone wants to download by
hand, he can clone the repo as well.

> >> Now they get to download it again. Perhaps a bigger problem:
> >>
> >>   $ echo $PORTDIR
> >>   /var/cache/portage/repositories/gentoo
> >>   $ python cpuinfo2cpuflags-x86.py
> >>   Unable to determine PORTDIR, please set it before calling this script
> > 
> > Did you export PORTDIR?
> > 
> 
> No, but the error says to set it, not export it =)

Thanks, mr troll. REALLY HELPFUL.

-- 
Best regards,
Michał Górny

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] Re: Review: news item and script for CPU_FLAGS_X86

[Michael Orlitzky]

On 01/23/2015 05:07 PM, Michał Górny wrote:
> 
>> Even `wget --no-check-certificate` would be a big improvement.
> 
> ...because?
> 

You rule out some 13-year-olds in coffee shops as attackers. I
overstated -- since the user isn't replying with any data, it's not a
big improvement, but it's still a little better.


>>
>> No, but the error says to set it, not export it =)
> 
> Thanks, mr troll. REALLY HELPFUL.
> 

Ok it looks stupid, but I was serious. I source my make.conf in
~/.bashrc, so I have PORTDIR and friends set in my shell. I have PORTDIR
set, and the thing is telling me to set PORTDIR? It took me a moment to
realize what was wrong. It will certainly stump others.

If what you really want them to do is prepend PORTDIR="..." before the
wget command, then why give them the wget command without it? Or if the
environment variable needs to be set and then exported, why not just say
that? The news items go out to tens of thousands of people so the more
explicit the instructions are, the better. Because then people won't bug
you about all the ways in which they misinterpreted the instructions.

Re: [gentoo-dev] Re: Review: news item and script for CPU_FLAGS_X86

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2164 bytes --]

Dnia 2015-01-23, o godz. 17:27:16
Michael Orlitzky <mjo@gentoo.org> napisał(a):

> On 01/23/2015 05:07 PM, Michał Górny wrote:
> > 
> >> Even `wget --no-check-certificate` would be a big improvement.
> > 
> > ...because?
> > 
> 
> You rule out some 13-year-olds in coffee shops as attackers. I
> overstated -- since the user isn't replying with any data, it's not a
> big improvement, but it's still a little better.

So I have a few options at hand:

1. put a checksum in the news item. The item is GPG-signed, so
the checksum will be protected. Of course, nobody bothers to check
the signature but anyway... Bad news is that I don't know of any
command to check signature that would really fit in 80 chars.

2. Put it .gpg-protected. Then oneliner is simple 'wget -O - ... | gpg
-d | python' (I have no idea how far gpg verifies there). But it means
that people who don't care and don't have GPG won't be able to use
the one-liner.

3. Put it in an ebuild, after all. This will add a lot of complexity
but GPG comes for free, plus some people will actually test
and stabilize it.

Do you like 3.?

> >>
> >> No, but the error says to set it, not export it =)
> > 
> > Thanks, mr troll. REALLY HELPFUL.
> > 
> 
> Ok it looks stupid, but I was serious. I source my make.conf in
> ~/.bashrc, so I have PORTDIR and friends set in my shell. I have PORTDIR
> set, and the thing is telling me to set PORTDIR? It took me a moment to
> realize what was wrong. It will certainly stump others.
> 
> If what you really want them to do is prepend PORTDIR="..." before the
> wget command, then why give them the wget command without it? Or if the
> environment variable needs to be set and then exported, why not just say
> that? The news items go out to tens of thousands of people so the more
> explicit the instructions are, the better. Because then people won't bug
> you about all the ways in which they misinterpreted the instructions.

People usually won't need to do that. Unless they do something stupid
like switching to a Python interpreter they disabled in the eclass...

-- 
Best regards,
Michał Górny

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] Re: Review: news item and script for CPU_FLAGS_X86

[Brian Dolbec]

On Fri, 23 Jan 2015 16:16:25 -0500
Michael Orlitzky <mjo@gentoo.org> wrote:

> On 01/23/2015 03:22 PM, Michał Górny wrote:
> > Dnia 2015-01-23, o godz. 14:26:48
> > Michael Orlitzky <mjo@gentoo.org> napisał(a):
> > 
> >> On 01/23/2015 02:13 PM, Michał Górny wrote:
> >>> To help you enable the correct USE flags, we are providing a
> >>> Python script which generates the correct value from
> >>> your /proc/cpuinfo [1]. The Python script can be downloaded and
> >>> executed using the following command:
> >>>
> >>>   $ wget -O - dev.gentoo.org/~mgorny/cpuinfo2cpuflags-x86.py |
> >>> python
> >>
> >> Can we not encourage people to pipe stuff from a plain-http
> >> website into an interpreter?
> > 
> > Find a better solution.
> > 
> 
> Is there an easy way for users to verify our signatures against the
> keys in LDAP?
> 
> Even `wget --no-check-certificate` would be a big improvement. Or
> since Firefox seems happy with the dev.gentoo.org certificate, we
> could just ask them to download it with their browsers.
> 
> Longer term: can we make wget like our SSL certificate?
>

Yes, if this data or python code is to be downloaded routinely, then
api.gentoo.org is the new https service specifically designed for this.

Please talk to infra for a subdirectory assignment for this data/code.

Also, for python based apps, dev-python/ssl-fetch was specifically
designed for retrieving from api.g.o (or any url) with certificate
authentication.

Taking it one step further, the gentoo-keys project (uses ssl-fetch) is
just entering the tree and can be used to download and verify files and
gpg signatures of those files.  app-crypt/gkeys-0.1-r1 is in the tree
and installs with the gentoo release media gpg keys and downloads the
current gentoo-devs seed file.  The developers gpg keys must be
installed for them to be verified against.

eg:

$ gkeys verify -F dev.gentoo.org/~mgorny/cpuinfo2cpuflags-x86.py

should do it, it will automatically look for a matching *.sig to use to
verify with against the installed gpg keys.  It can even save and use
timestamps to prevent unneeded downloads for unchanged data in a local
cache.

But, for gkeys to become commonplace in usage, it also requires devs to
fix their current keys and LDAP data, or generate new GLEP 63 compliant
keys.  But that is an off topic discussion

-- 
Brian Dolbec <dolsen>

Re: [gentoo-dev] Re: Review: news item and script for CPU_FLAGS_X86

[Ben Kohler]

[-- Attachment #1: Type: text/plain, Size: 690 bytes --]

On Fri, Jan 23, 2015 at 4:38 PM, Michał Górny <mgorny@gentoo.org> wrote:

>
> 3. Put it in an ebuild, after all. This will add a lot of complexity
> but GPG comes for free, plus some people will actually test
> and stabilize it.
>
>
I think this should be in an ebuild.  You mentioned that it's only needed
ONCE, but it's needed ONCE for everytime one install gentoos, along the
same lines as mirrorselect.  A couple of years from now, do we want users
to have to dig through several dozen old news items to get this tool?

I know the ebuild's a bit of extra work but then the python issues can also
be properly handled.  And bugs can be properly handled, etc etc.

-Ben

[-- Attachment #2: Type: text/html, Size: 1130 bytes --]

Re: [gentoo-dev] Re: Review: news item and script for CPU_FLAGS_X86

[Michael Orlitzky]

On 01/23/2015 05:38 PM, Michał Górny wrote:
> 
> 3. Put it in an ebuild, after all. This will add a lot of complexity
> but GPG comes for free, plus some people will actually test
> and stabilize it.
> 

I do. It seems like a lot of work, but we can test and quick-stable it.
Thank you.


>> If what you really want them to do is prepend PORTDIR="..." before the
>> wget command, then why give them the wget command without it? Or if the
>> environment variable needs to be set and then exported, why not just say
>> that? The news items go out to tens of thousands of people so the more
>> explicit the instructions are, the better. Because then people won't bug
>> you about all the ways in which they misinterpreted the instructions.
> 
> People usually won't need to do that. Unless they do something stupid
> like switching to a Python interpreter they disabled in the eclass...
> 

It's not my fault this time. Remember when python-3.4 went stable for a
minute and this news item went out?

  Python 3.4 is now enabled by default, replacing Python 3.3 as the
  default Python 3 interpreter.
  ...
  Once the changes have taken place, a world update should take care of
  reinstalling any python libraries you have installed. You should also
  switch your default python3 interpreter using eselect python.

  For example:

  eselect python set --python3 python3.4


A few hours later, PYTHON_TARGETS was reverted, and no news item went
out telling me to undo the eselect.

Re: [gentoo-dev] Re: Review: news item and script for CPU_FLAGS_X86

[Michał Górny]

[-- Attachment #1.1: Type: text/plain, Size: 489 bytes --]

Dnia 2015-01-23, o godz. 17:56:49
Michael Orlitzky <mjo@gentoo.org> napisał(a):

> On 01/23/2015 05:38 PM, Michał Górny wrote:
> > 
> > 3. Put it in an ebuild, after all. This will add a lot of complexity
> > but GPG comes for free, plus some people will actually test
> > and stabilize it.
> > 
> 
> I do. It seems like a lot of work, but we can test and quick-stable it.
> Thank you.

Ebuild committed. Now review the news item :P.

-- 
Best regards,
Michał Górny

[-- Attachment #1.2: 2015-01-xx-CPU_FLAGS_X86-introduction.en.txt --]
[-- Type: text/plain, Size: 1495 bytes --]

Title: CPU_FLAGS_X86 introduction
Author: Michał Górny <mgorny@gentoo.org>
Content-Type: text/plain
Posted: 2015-01-xx
Revision: 1
News-Item-Format: 1.0
Display-If-Keyword: amd64 ~amd64 x86 ~x86

The USE flags corresponding to intruction sets and other features
specific to the x86 architecture are being moved into a separate USE
flag group called CPU_FLAGS_X86.

In order not to lose CPU-specific optimizations, users will be required
to update their make.conf (and package.use) file. For example, if
the following USE flags were present:

  USE="mmx mmxext sse sse2 sse3"

Those flags need to be copied into:

  CPU_FLAGS_X86="mmx mmxext sse sse2 sse3"

Please note that CPU_FLAGS_X86 is used both on x86 and amd64 systems.

When in doubt, please consult profiles/desc/cpu_flags_x86.desc. Most of
the flag names match /proc/cpuinfo names, with the notable exception
of SSE3 which is called 'pni' in /proc/cpuinfo (please also do not
confuse it with distinct SSSE3).

To help you enable the correct USE flags, we are providing a Python
script tool generates the correct value using your /proc/cpuinfo.
It can be found in app-portage/cpuinfo2cpuflags package:

  $ emerge -1v app-portage/cpuinfo2cpuflags
  $ cpuinfo2cpuflags-x86.py

In order to ensure safe migration and maintain compatibility with
external repositories, it is recommended to preserve the old USE
settings for a period of one year or until no package of interest is
still using them.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] Re: Review: news item and script for CPU_FLAGS_X86

[Michael Orlitzky]

On 01/23/2015 06:12 PM, Michał Górny wrote:
> To help you enable the correct USE flags, we are providing a Python
> script tool generates the correct value using your /proc/cpuinfo.
> It can be found in app-portage/cpuinfo2cpuflags package:

I think you accidentally a word here, but otherwise it looks good to me.
The only other suggestion I have is to include $PORTDIR in the path
before profiles/desc/cpu_flags_x86.desc because the profiles/desc
directory isn't familiar to most users.

I see you updated the error message, too. Thanks.

Re: [gentoo-dev] Re: Review: news item and script for CPU_FLAGS_X86

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 498 bytes --]

On Sat, 24 Jan 2015 00:12:15 +0100 Michał Górny wrote:
> To help you enable the correct USE flags, we are providing a Python
> script tool generates the correct value using your /proc/cpuinfo.
> It can be found in app-portage/cpuinfo2cpuflags package:
> 
>   $ emerge -1v app-portage/cpuinfo2cpuflags
>   $ cpuinfo2cpuflags-x86.py

A bit pedantry here: emerge should be preceded by # instead of $,
since normally one can't install packages as a user.

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] Re: Review: news item and script for CPU_FLAGS_X86

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 673 bytes --]

Dnia 2015-01-24, o godz. 02:29:26
Andrew Savchenko <bircoph@gentoo.org> napisał(a):

> On Sat, 24 Jan 2015 00:12:15 +0100 Michał Górny wrote:
> > To help you enable the correct USE flags, we are providing a Python
> > script tool generates the correct value using your /proc/cpuinfo.
> > It can be found in app-portage/cpuinfo2cpuflags package:
> > 
> >   $ emerge -1v app-portage/cpuinfo2cpuflags
> >   $ cpuinfo2cpuflags-x86.py
> 
> A bit pedantry here: emerge should be preceded by # instead of $,
> since normally one can't install packages as a user.

As far as I'm concerned, # is a comment in shell script :P.

-- 
Best regards,
Michał Górny

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] Review: news item and script for CPU_FLAGS_X86

[Michał Górny]

[-- Attachment #1.1: Type: text/plain, Size: 1057 bytes --]

Dnia 2015-01-20, o godz. 10:40:17
Michał Górny <mgorny@gentoo.org> napisał(a):

> Dnia 2015-01-18, o godz. 21:44:05
> Michał Górny <mgorny@gentoo.org> napisał(a):
> 
> > Hello,
> > 
> > I would like to commit the following flags as cpu_flags_x86_desc.
> > The list combines global USE flags with some local USE flags I've been
> > able to find.
> 
> Following your suggestions, I'm attaching three files:
> 
> 1. updated cpu_flags_x86.desc, with:
> 
>  a. fma3/fma4 distinction,
> 
>  b. aes-ni renamed to aes for consistency with cpuinfo, we no longer
>  have to worry about USE flag collision with other USE=aes uses,
> 
>  c. complete and machine-parseable listing of cpuinfo flags.
> 
> 2. cpuinfo2cpuflags-x86.py script that parses cpu_flags_x86.desc
> and /proc/cpuinfo and generates a nice CPU_FLAGS_X86 value for you. We
> need to decide where to put it (scripts/?).
> 
> 3. Initial text of the news item for review.

Ok, the hopefully-final text for the news item attached.

-- 
Best regards,
Michał Górny

[-- Attachment #1.2: 2015-01-xx-CPU_FLAGS_X86-introduction.en.txt --]
[-- Type: text/plain, Size: 1569 bytes --]

Title: CPU_FLAGS_X86 introduction
Author: Michał Górny <mgorny@gentoo.org>
Content-Type: text/plain
Posted: 2015-01-xx
Revision: 1
News-Item-Format: 1.0
Display-If-Keyword: amd64 ~amd64 x86 ~x86

The USE flags corresponding to the instruction sets and other features
specific to the x86 architecture are being moved into a separate USE
flag group called CPU_FLAGS_X86.

In order not to lose CPU-specific optimizations, users will be required
to update their make.conf (and package.use) file. For example, if
the following USE flags were present:

  USE="mmx mmxext sse sse2 sse3"

Those flags need to be copied into:

  CPU_FLAGS_X86="mmx mmxext sse sse2 sse3"

Please note that the CPU_FLAGS_X86 variable is used both on x86 and
amd64 systems.

When in doubt, please consult the profiles/desc/cpu_flags_x86.desc file
available in the Gentoo repository checkout. Most of the flag names
match /proc/cpuinfo names, with the notable exception of SSE3 which is
called 'pni' in /proc/cpuinfo (please also do not confuse it with
distinct SSSE3).

To help users enable the correct USE flags, we are providing a Python
script that generates the correct value using /proc/cpuinfo. It can be
found in the app-portage/cpuinfo2cpuflags package:

  $ emerge -1v app-portage/cpuinfo2cpuflags
  $ cpuinfo2cpuflags-x86.py

In order to ensure safe migration and maintain compatibility with
external repositories, it is recommended to preserve the old USE
settings for a period of one year or until no package of interest is
still using them.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

[gentoo-dev] Re: Review: news item and script for CPU_FLAGS_X86

[Duncan]

Michał Górny posted on Fri, 23 Jan 2015 20:13:34 +0100 as excerpted:

> Dnia 2015-01-21, o godz. 00:18:50 Duncan <1i5t5.duncan@cox.net>
> napisał(a):
> 
>> Michał Górny posted on Tue, 20 Jan 2015 10:40:17 +0100 [as snipped]
>> 
>> So based on this preliminary news item, I decided to get ahead of the
>> game and add the use_expand to the appropriate make.conf sourced
>> file...
> 
> Attaching a new version.

Thanks.  The general notification and use-expand instructions look much 
better now (and I like your wording FAR more than my suggested wording 
=:^).

It looks like the script details are being dealt with elsewhere, so all 
that remains is ...

This should be yet another minor improvement to gentoo, but it's all 
those minor improvements that add up and make the gentoo experience 
better over time, so really, /thanks/ for the extra work you're putting 
into this.  =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: Review: news item and script for CPU_FLAGS_X86

[Diamond]

On Fri, 23 Jan 2015 16:45:34 -0600
Ben Kohler <bkohler@gmail.com> wrote:

> 
> I think this should be in an ebuild.  You mentioned that it's only
> needed ONCE, but it's needed ONCE for everytime one install gentoos,
> along the same lines as mirrorselect.  A couple of years from now, do
> we want users to have to dig through several dozen old news items to
> get this tool?
> 
Maybe it's better to integrate this new tool with eselect tool?

Re: [gentoo-dev] Re: Review: news item and script for CPU_FLAGS_X86

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 641 bytes --]

Dnia 2015-01-24, o godz. 10:27:47
Diamond <diamond@hi-net.ru> napisał(a):

> On Fri, 23 Jan 2015 16:45:34 -0600
> Ben Kohler <bkohler@gmail.com> wrote:
> 
> > 
> > I think this should be in an ebuild.  You mentioned that it's only
> > needed ONCE, but it's needed ONCE for everytime one install gentoos,
> > along the same lines as mirrorselect.  A couple of years from now, do
> > we want users to have to dig through several dozen old news items to
> > get this tool?
> > 
> Maybe it's better to integrate this new tool with eselect tool?

No. It doesn't resemble eselect in any way.

-- 
Best regards,
Michał Górny

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] Review: news item and script for CPU_FLAGS_X86

[Alexis Ballier]

On Sat, 24 Jan 2015 00:35:39 +0100
Michał Górny <mgorny@gentoo.org> wrote:

> Title: CPU_FLAGS_X86 introduction
> Author: Michał Górny <mgorny@gentoo.org>
> Content-Type: text/plain
> Posted: 2015-01-xx
> Revision: 1
> News-Item-Format: 1.0
> Display-If-Keyword: amd64 ~amd64 x86 ~x86

but.... why ?
will you write another news item for other arches ?


> When in doubt, please consult the profiles/desc/cpu_flags_x86.desc
> file available in the Gentoo repository checkout.

usually we don't point people at reading profiles or ebuilds
directly; maybe 'equery uses' ?

> Most of the flag
> names match /proc/cpuinfo names, with the notable exception of SSE3
> which is called 'pni' in /proc/cpuinfo (please also do not confuse it
> with distinct SSSE3).

IMHO this is too much into details but I don't really mind.


Alexis.

Re: [gentoo-dev] Review: news item and script for CPU_FLAGS_X86

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1435 bytes --]

Dnia 2015-01-26, o godz. 12:41:00
Alexis Ballier <aballier@gentoo.org> napisał(a):

> On Sat, 24 Jan 2015 00:35:39 +0100
> Michał Górny <mgorny@gentoo.org> wrote:
> 
> > Title: CPU_FLAGS_X86 introduction
> > Author: Michał Górny <mgorny@gentoo.org>
> > Content-Type: text/plain
> > Posted: 2015-01-xx
> > Revision: 1
> > News-Item-Format: 1.0
> > Display-If-Keyword: amd64 ~amd64 x86 ~x86
> 
> but.... why ?
> will you write another news item for other arches ?

Are there other arches using CPU_FLAGS_X86? ;) But seriously, the item
is quite arch-specific. Other arches are likely to have kinda specific
flags with rules for choosing them, another script etc.

> > When in doubt, please consult the profiles/desc/cpu_flags_x86.desc
> > file available in the Gentoo repository checkout.
> 
> usually we don't point people at reading profiles or ebuilds
> directly; maybe 'equery uses' ?

I guess that kinda works. So far 'quse -D mmx' has the nicest output
but I don't want people to rely on portage-utils.

> > Most of the flag
> > names match /proc/cpuinfo names, with the notable exception of SSE3
> > which is called 'pni' in /proc/cpuinfo (please also do not confuse it
> > with distinct SSSE3).
> 
> IMHO this is too much into details but I don't really mind.

I just recall it's a frequent issue, people missing SSE3 and mixing it
up with SSSE3.

-- 
Best regards,
Michał Górny

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] Review: news item and script for CPU_FLAGS_X86

[Alexis Ballier]

On Mon, 26 Jan 2015 16:20:10 +0100
Michał Górny <mgorny@gentoo.org> wrote:

> Dnia 2015-01-26, o godz. 12:41:00
> Alexis Ballier <aballier@gentoo.org> napisał(a):
> 
> > On Sat, 24 Jan 2015 00:35:39 +0100
> > Michał Górny <mgorny@gentoo.org> wrote:
> > 
> > > Title: CPU_FLAGS_X86 introduction
> > > Author: Michał Górny <mgorny@gentoo.org>
> > > Content-Type: text/plain
> > > Posted: 2015-01-xx
> > > Revision: 1
> > > News-Item-Format: 1.0
> > > Display-If-Keyword: amd64 ~amd64 x86 ~x86
> > 
> > but.... why ?
> > will you write another news item for other arches ?
> 
> Are there other arches using CPU_FLAGS_X86? ;) But seriously, the item
> is quite arch-specific. Other arches are likely to have kinda specific
> flags with rules for choosing them, another script etc.

I think it is better to have it done all in one pass: even if there is
no script, it is just as good as it is today.

My concern is: This will clutter e.g. ffmpeg ebuild by having two ways
to pass cpu flags, depending on the arch, and will give a kind of silly
output with "altivec" or "neon" as standard useflags but x86 cpu flags
as USE_EXPAND. This is too much inconsistent to me.


> > > Most of the flag
> > > names match /proc/cpuinfo names, with the notable exception of
> > > SSE3 which is called 'pni' in /proc/cpuinfo (please also do not
> > > confuse it with distinct SSSE3).
> > 
> > IMHO this is too much into details but I don't really mind.
> 
> I just recall it's a frequent issue, people missing SSE3 and mixing it
> up with SSSE3.

Yep; I don't think this kind of obvious and not new warning belongs to a
_news_ item but that's up to you.

Alexis.

Re: [gentoo-dev] Review: news item and script for CPU_FLAGS_X86

[Michał Górny]

[-- Attachment #1.1: Type: text/plain, Size: 789 bytes --]

Dnia 2015-01-26, o godz. 12:41:00
Alexis Ballier <aballier@gentoo.org> napisał(a):

> On Sat, 24 Jan 2015 00:35:39 +0100
> Michał Górny <mgorny@gentoo.org> wrote:
> 
> > Title: CPU_FLAGS_X86 introduction
> > Author: Michał Górny <mgorny@gentoo.org>
> > Content-Type: text/plain
> > Posted: 2015-01-xx
> > Revision: 1
> > News-Item-Format: 1.0
> > Display-If-Keyword: amd64 ~amd64 x86 ~x86
> 
> but.... why ?
> will you write another news item for other arches ?
> 
> 
> > When in doubt, please consult the profiles/desc/cpu_flags_x86.desc
> > file available in the Gentoo repository checkout.
> 
> usually we don't point people at reading profiles or ebuilds
> directly; maybe 'equery uses' ?

Here's the updated text.


-- 
Best regards,
Michał Górny

[-- Attachment #1.2: 2015-01-xx-CPU_FLAGS_X86-introduction.en.txt --]
[-- Type: text/plain, Size: 1637 bytes --]

Title: CPU_FLAGS_X86 introduction
Author: Michał Górny <mgorny@gentoo.org>
Content-Type: text/plain
Posted: 2015-01-xx
Revision: 1
News-Item-Format: 1.0
Display-If-Keyword: amd64 ~amd64 x86 ~x86

The USE flags corresponding to the instruction sets and other features
specific to the x86 (amd64) architecture are being moved into a separate
USE flag group called CPU_FLAGS_X86.

In order not to lose CPU-specific optimizations, users will be required
to update their make.conf (and package.use) file. For example, if
the following USE flags were present:

  USE="mmx mmxext sse sse2 sse3"

Those flags need to be copied into:

  CPU_FLAGS_X86="mmx mmxext sse sse2 sse3"

Please note that the same CPU_FLAGS_X86 variable is used both on x86
and amd64 systems.

When in doubt, you can consult the flag descriptions using one of
the commonly available tools, e.g. `equery uses` from gentoolkit:

  $ equery uses media-video/ffmpeg

Most of the flag names match /proc/cpuinfo names, with the notable
exception of SSE3 which is called 'pni' in /proc/cpuinfo (please also
do not confuse it with distinct SSSE3).

To help users enable the correct USE flags, we are providing a Python
script that generates the correct value using /proc/cpuinfo. It can be
found in the app-portage/cpuinfo2cpuflags package:

  $ emerge -1v app-portage/cpuinfo2cpuflags
  $ cpuinfo2cpuflags-x86.py

In order to ensure safe migration and maintain compatibility with
external repositories, it is recommended to preserve the old USE
settings for a period of one year or until no package of interest is
still using them.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] Review: news item and script for CPU_FLAGS_X86

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1630 bytes --]

Dnia 2015-01-26, o godz. 16:40:35
Alexis Ballier <aballier@gentoo.org> napisał(a):

> On Mon, 26 Jan 2015 16:20:10 +0100
> Michał Górny <mgorny@gentoo.org> wrote:
> 
> > Dnia 2015-01-26, o godz. 12:41:00
> > Alexis Ballier <aballier@gentoo.org> napisał(a):
> > 
> > > On Sat, 24 Jan 2015 00:35:39 +0100
> > > Michał Górny <mgorny@gentoo.org> wrote:
> > > 
> > > > Title: CPU_FLAGS_X86 introduction
> > > > Author: Michał Górny <mgorny@gentoo.org>
> > > > Content-Type: text/plain
> > > > Posted: 2015-01-xx
> > > > Revision: 1
> > > > News-Item-Format: 1.0
> > > > Display-If-Keyword: amd64 ~amd64 x86 ~x86
> > > 
> > > but.... why ?
> > > will you write another news item for other arches ?
> > 
> > Are there other arches using CPU_FLAGS_X86? ;) But seriously, the item
> > is quite arch-specific. Other arches are likely to have kinda specific
> > flags with rules for choosing them, another script etc.
> 
> I think it is better to have it done all in one pass: even if there is
> no script, it is just as good as it is today.
> 
> My concern is: This will clutter e.g. ffmpeg ebuild by having two ways
> to pass cpu flags, depending on the arch, and will give a kind of silly
> output with "altivec" or "neon" as standard useflags but x86 cpu flags
> as USE_EXPAND. This is too much inconsistent to me.

I understand your concern but unless someone's going to do the work for
other arches, I doubt there's a point in waiting forever. Script is
a minor issue, but someone has to figure out how various packages
behave and what flags to use.

-- 
Best regards,
Michał Górny

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] Re: Review: news item and script for CPU_FLAGS_X86

[vivo75]

Il 24/01/2015 00:12, Michał Górny ha scritto:
>   $ emerge -1v app-portage/cpuinfo2cpuflags
>   $ cpuinfo2cpuflags-x86.py
it's

cpuinfo2cpuflags-x86

w/o the .py here

Re: [gentoo-dev] Re: Review: news item and script for CPU_FLAGS_X86

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 393 bytes --]

Dnia 2015-01-26, o godz. 23:11:35
"vivo75@gmail.com" <vivo75@gmail.com> napisał(a):

> Il 24/01/2015 00:12, Michał Górny ha scritto:
> >   $ emerge -1v app-portage/cpuinfo2cpuflags
> >   $ cpuinfo2cpuflags-x86.py
> it's
> 
> cpuinfo2cpuflags-x86
> 
> w/o the .py here

floppym already told me, I updated it locally but didn't reupload ;P.

-- 
Best regards,
Michał Górny

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] Review: news item and script for CPU_FLAGS_X86

[Alexis Ballier]

On Mon, 26 Jan 2015 20:09:18 +0100
Michał Górny <mgorny@gentoo.org> wrote:

> Dnia 2015-01-26, o godz. 16:40:35
> Alexis Ballier <aballier@gentoo.org> napisał(a):
> 
> > On Mon, 26 Jan 2015 16:20:10 +0100
> > Michał Górny <mgorny@gentoo.org> wrote:
> > 
> > > Dnia 2015-01-26, o godz. 12:41:00
> > > Alexis Ballier <aballier@gentoo.org> napisał(a):
> > > 
> > > > On Sat, 24 Jan 2015 00:35:39 +0100
> > > > Michał Górny <mgorny@gentoo.org> wrote:
> > > > 
> > > > > Title: CPU_FLAGS_X86 introduction
> > > > > Author: Michał Górny <mgorny@gentoo.org>
> > > > > Content-Type: text/plain
> > > > > Posted: 2015-01-xx
> > > > > Revision: 1
> > > > > News-Item-Format: 1.0
> > > > > Display-If-Keyword: amd64 ~amd64 x86 ~x86
> > > > 
> > > > but.... why ?
> > > > will you write another news item for other arches ?
> > > 
> > > Are there other arches using CPU_FLAGS_X86? ;) But seriously, the
> > > item is quite arch-specific. Other arches are likely to have
> > > kinda specific flags with rules for choosing them, another script
> > > etc.
> > 
> > I think it is better to have it done all in one pass: even if there
> > is no script, it is just as good as it is today.
> > 
> > My concern is: This will clutter e.g. ffmpeg ebuild by having two
> > ways to pass cpu flags, depending on the arch, and will give a kind
> > of silly output with "altivec" or "neon" as standard useflags but
> > x86 cpu flags as USE_EXPAND. This is too much inconsistent to me.
> 
> I understand your concern but unless someone's going to do the work
> for other arches, I doubt there's a point in waiting forever. Script
> is a minor issue, but someone has to figure out how various packages
> behave and what flags to use.
> 

It doesn't have to be perfect, just consistent. As of figuring out how
to have such flags, I already gave you the link: profiles/base/use.mask.

Let's see:

# ppc arch specific USE flags
altivec
pbbuttonsd
ppcsha1

# mips arch specific USE flags
n32
n64
fixed-point
loongson2f
mips32r2
mipsdspr1
mipsdspr2
mipsfpu

# sparc arch specific USE flags
vis
ultra1

etc.

grep their desc in use.desc or .local.desc and paste these to
profiles/desc/cpu_flags_xxx.desc, and you're done.
if you want to do things better, open a bug for relevant arch team to
review it, improve it or remove useless stuff; it'd be better tracked
than a discussion here.



Anyway, flags renamings will have to be done on a per-package basis,
probably with a bug openened and certainly with proper review done, so
that being x86* only or all arches makes little to no difference. Even
better: you won't have me on your back ranting against pointless
inconsistencies :)

Alexis.

Re: [gentoo-dev] Review: news item and script for CPU_FLAGS_X86

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 3663 bytes --]

Dnia 2015-01-27, o godz. 10:18:37
Alexis Ballier <aballier@gentoo.org> napisał(a):

> On Mon, 26 Jan 2015 20:09:18 +0100
> Michał Górny <mgorny@gentoo.org> wrote:
> 
> > Dnia 2015-01-26, o godz. 16:40:35
> > Alexis Ballier <aballier@gentoo.org> napisał(a):
> > 
> > > On Mon, 26 Jan 2015 16:20:10 +0100
> > > Michał Górny <mgorny@gentoo.org> wrote:
> > > 
> > > > Dnia 2015-01-26, o godz. 12:41:00
> > > > Alexis Ballier <aballier@gentoo.org> napisał(a):
> > > > 
> > > > > On Sat, 24 Jan 2015 00:35:39 +0100
> > > > > Michał Górny <mgorny@gentoo.org> wrote:
> > > > > 
> > > > > > Title: CPU_FLAGS_X86 introduction
> > > > > > Author: Michał Górny <mgorny@gentoo.org>
> > > > > > Content-Type: text/plain
> > > > > > Posted: 2015-01-xx
> > > > > > Revision: 1
> > > > > > News-Item-Format: 1.0
> > > > > > Display-If-Keyword: amd64 ~amd64 x86 ~x86
> > > > > 
> > > > > but.... why ?
> > > > > will you write another news item for other arches ?
> > > > 
> > > > Are there other arches using CPU_FLAGS_X86? ;) But seriously, the
> > > > item is quite arch-specific. Other arches are likely to have
> > > > kinda specific flags with rules for choosing them, another script
> > > > etc.
> > > 
> > > I think it is better to have it done all in one pass: even if there
> > > is no script, it is just as good as it is today.
> > > 
> > > My concern is: This will clutter e.g. ffmpeg ebuild by having two
> > > ways to pass cpu flags, depending on the arch, and will give a kind
> > > of silly output with "altivec" or "neon" as standard useflags but
> > > x86 cpu flags as USE_EXPAND. This is too much inconsistent to me.
> > 
> > I understand your concern but unless someone's going to do the work
> > for other arches, I doubt there's a point in waiting forever. Script
> > is a minor issue, but someone has to figure out how various packages
> > behave and what flags to use.
> > 
> 
> It doesn't have to be perfect, just consistent. As of figuring out how
> to have such flags, I already gave you the link: profiles/base/use.mask.

I'm afraid this mail pretty much proves why we shouldn't do this.

> Let's see:
> 
> # ppc arch specific USE flags
> altivec
> pbbuttonsd

Is this even a USE flag? Maybe you meant USE=macbook or something like
this?

> ppcsha1

This is not a CPU feature, and i'm not sure if this should be
an explicit flag at all. This sounds like 'use ppc' + 'use asm'.

> # mips arch specific USE flags
> n32
> n64

Those are rather covered in ABI_MIPS.

> fixed-point
> loongson2f
> mips32r2
> mipsdspr1
> mipsdspr2
> mipsfpu

The MIPS team will likely want to drop some of the prefixes.

> # sparc arch specific USE flags
> vis
> ultra1
> 
> etc.
> 
> grep their desc in use.desc or .local.desc and paste these to
> profiles/desc/cpu_flags_xxx.desc, and you're done.
> if you want to do things better, open a bug for relevant arch team to
> review it, improve it or remove useless stuff; it'd be better tracked
> than a discussion here.

True, a bug is a good idea.

> Anyway, flags renamings will have to be done on a per-package basis,
> probably with a bug openened and certainly with proper review done, so
> that being x86* only or all arches makes little to no difference. Even
> better: you won't have me on your back ranting against pointless
> inconsistencies :)

The point is not to publish a news item telling people to update flags
before we decide on the final flags. The list pretty much covered x86
flag review, and found issues that we were able to fix before
committing.

-- 
Best regards,
Michał Górny

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]