[gentoo-dev] Move x86/amd64 CPU extensions USE flags to a new USE_EXPAND variable

[gentoo-dev] Move x86/amd64 CPU extensions USE flags to a new USE_EXPAND variable

[Jeroen Roovers]

     Hi all,


bugs like [1] makes clear to me that the increasing number of CPU
extensions USE flags is getting more and more confusing.

ffmpeg-0.6_p25767.ebuild : IUSE="+3dnow +3dnowext alsa altivec amr
bindist +bzip2 cpudetection custom-cflags debug dirac doc +encode faac
frei0r gsm +hardcoded-tables ieee1394 jack jpeg2k +mmx +mmxext mp3
network oss pic qt-faststart rtmp schroedinger sdl speex +ssse3
static-libs test theora threads v4l v4l2 vaapi vdpau vorbis vpx X x264
xvid +zlib video_cards_nvidia"

(I am not going to argue here that USE defaults shouldn't be used for
CPU extensions.)

Among all CPU extensions USE flags you'll find:

3dnow
3dnowext
mmx
mmxext
sse
sse2
sse3
sse4
sse4a
sse5
ssse3

I probably missed a few, there.

I think it would look a lot less confusing if, using the example above,
the output of emerge would not look like this:

[ebuild  N    ] media-video/ffmpeg-0.6_p25767  USE="3dnow 3dnowext
bzip2 encode hardcoded-tables mmx mmxext ssse3 zlib -X -alsa (-altivec)
-amr -bindist -cpudetection -custom-cflags -debug -dirac -doc -faac
-frei0r -gsm -ieee1394 -jack -jpeg2k -mp3 -network -oss -pic
-qt-faststart -rtmp -schroedinger -sdl -speex -static-libs -test
-theora -threads -v4l -v4l2 -vaapi -vdpau -vorbis -vpx -x264 -xvid"
VIDEO_CARDS="-nvidia" 0 kB

but more like this:

[ebuild  N    ] media-video/ffmpeg-0.6_p25767  USE="bzip2 encode
hardcoded-tables zlib -X -alsa (-altivec) -amr -bindist -cpudetection
-custom-cflags -debug -dirac -doc -faac -frei0r -gsm -ieee1394 -jack
-jpeg2k -mp3 -network -oss -pic -qt-faststart -rtmp -schroedinger -sdl
-speex -static-libs -test -theora -threads -v4l -v4l2 -vaapi -vdpau
-vorbis -vpx -x264 -xvid" VIDEO_CARDS="-nvidia" CPU_EXTS="3dnow
3dnowext mmx mmxext ssse3" 0 kB


     jer


[1] https://bugs.gentoo.org/show_bug.cgi?id=340539
    "media-video/ffmpeg-0.6_p25423 - libswscale/swscale.c:1269: error:
     implicit declaration of function 'sws_init_swScale_*'"

Re: [gentoo-dev] Move x86/amd64 CPU extensions USE flags to a new USE_EXPAND variable

[Patrick Lauer]

On 12/11/10 18:57, Jeroen Roovers wrote:
>      Hi all,
> 
> 
> bugs like [1] makes clear to me that the increasing number of CPU
> extensions USE flags is getting more and more confusing.
> 
[snip]

> 
> Among all CPU extensions USE flags you'll find:
> 
> 3dnow
> 3dnowext
> mmx
> mmxext
> sse
> sse2
> sse3
> sse4
> sse4a
> sse5
> ssse3
> 
> I probably missed a few, there.
> 

How many packages are affected by those?
I like the idea, but I think changing it around for 2 packages might be
a bit silly. If there's a reasonable amount of packages involved I'm all
for it.

Patrick

Re: [gentoo-dev] Move x86/amd64 CPU extensions USE flags to a new USE_EXPAND variable

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 450 bytes --]

On Sat, 11 Dec 2010 19:19:05 +0100
Patrick Lauer <patrick@gentoo.org> wrote:

> How many packages are affected by those?
> I like the idea, but I think changing it around for 2 packages might
> be a bit silly. If there's a reasonable amount of packages involved
> I'm all for it.

How many packages use VIDEO_CARDS? How many packages use LIRC_DEVICES?
I think the number doesn't matter that much here.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Move x86/amd64 CPU extensions USE flags to a new USE_EXPAND variable

[Matt Turner]

On Sat, Dec 11, 2010 at 5:57 PM, Jeroen Roovers <jer@gentoo.org> wrote:
[snip]

I agree that this could be better. To me, most of the problems with
this are due to users not knowing which of these should be set for
their particular CPU.

Instead of having defaults set by a profile, I'd like to figure out a
way we can have these flags set by default dependent on the user's
CPU. This might require some additional logic in portage; I don't
know.

Matt

Re: [gentoo-dev] Move x86/amd64 CPU extensions USE flags to a new USE_EXPAND variable

[Dirkjan Ochtman]

On Sat, Dec 11, 2010 at 20:01, Matt Turner <mattst88@gentoo.org> wrote:
> I agree that this could be better. To me, most of the problems with
> this are due to users not knowing which of these should be set for
> their particular CPU.
>
> Instead of having defaults set by a profile, I'd like to figure out a
> way we can have these flags set by default dependent on the user's
> CPU. This might require some additional logic in portage; I don't
> know.

Yeah, I think setting the best possible default is important here.

Cheers,

Dirkjan

Re: [gentoo-dev] Move x86/amd64 CPU extensions USE flags to a new USE_EXPAND variable

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 413 bytes --]

On Sat, 11 Dec 2010 18:57:58 +0100
Jeroen Roovers <jer@gentoo.org> wrote:
> bugs like [1] makes clear to me that the increasing number of CPU
> extensions USE flags is getting more and more confusing.

Another big confusion is that x86 and amd64 use the same names for many
extensions, but often the archs require different handling. Why not have
X86_CPU_EXTS, AMD64_CPU_EXTS etc?

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Move x86/amd64 CPU extensions USE flags to a new USE_EXPAND variable

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 606 bytes --]

On Saturday, December 11, 2010 12:57:58 Jeroen Roovers wrote:
> but more like this:
> 
> [ebuild  N    ] media-video/ffmpeg-0.6_p25767  USE="bzip2 encode
> hardcoded-tables zlib -X -alsa (-altivec) -amr -bindist -cpudetection
> -custom-cflags -debug -dirac -doc -faac -frei0r -gsm -ieee1394 -jack
> -jpeg2k -mp3 -network -oss -pic -qt-faststart -rtmp -schroedinger -sdl
> -speex -static-libs -test -theora -threads -v4l -v4l2 -vaapi -vdpau
> -vorbis -vpx -x264 -xvid" VIDEO_CARDS="-nvidia" CPU_EXTS="3dnow
> 3dnowext mmx mmxext ssse3" 0 kB

nice idea.  now put together some patches :P.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[gentoo-dev] Re: Move x86/amd64 CPU extensions USE flags to a new USE_EXPAND variable

[Duncan]

Dirkjan Ochtman posted on Sat, 11 Dec 2010 20:03:39 +0100 as excerpted:

> On Sat, Dec 11, 2010 at 20:01, Matt Turner <mattst88@gentoo.org> wrote:
>> I agree that this could be better. To me, most of the problems with
>> this are due to users not knowing which of these should be set for
>> their particular CPU.
>>
>> Instead of having defaults set by a profile, I'd like to figure out a
>> way we can have these flags set by default dependent on the user's CPU.
>> This might require some additional logic in portage; I don't know.
> 
> Yeah, I think setting the best possible default is important here.

FWIW, I don't think that's a good idea -- at least if you're suggesting 
real-time build-machine detection (see below for the alternative).  Not 
only will implementing it be a lot of work, but then you'll have people 
filing bugs because they intended to compile for several different 
machines and set generic CFLAGS accordingly, but still ended up with 
broken packages because the default CPU extensions logic looked at the 
build machine only, and they thought they were done when they set the 
profile, CFLAGS and CHOST to the level of generic they intended.

Unless the logic is based on the -march set in C(XX)FLAGS.  That could 
work, as people should already be setting that as specific or generic as 
they want.

I do like the USE_EXPAND idea, tho.

Meanwhile, would it be possible to have altivec in the same USE_EXPAND?  
It's the same CPU extensions type of thing on PPC, if I'm not mistaken.  
Does profile-mask work for USE_EXPAND?  Because it just seems strange to 
me to see all those CPU extensions in their own USE_EXPAND and see altivec 
still in USE.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

[gentoo-dev] Re: Move x86/amd64 CPU extensions USE flags to a new USE_EXPAND variable

[Ryan Hill]

[-- Attachment #1: Type: text/plain, Size: 579 bytes --]

On Sat, 11 Dec 2010 18:57:58 +0100
Jeroen Roovers <jer@gentoo.org> wrote:

> Among all CPU extensions USE flags you'll find:
> 
> 3dnow
> 3dnowext
> mmx
> mmxext
> sse
> sse2
> sse3
> sse4
> sse4a
> sse5
> ssse3
> 
> I probably missed a few, there.

sse4.1, sse4.2, avx

sse5 was a draft, it was never implemented.

-- 
fonts, gcc-porting,                  it makes no sense how it makes no sense
toolchain, wxwidgets                           but i'll take it free anytime
@ gentoo.org                EFFD 380E 047A 4B51 D2BD C64F 8AA8 8346 F9A4 0662

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-dev] Re: Move x86/amd64 CPU extensions USE flags to a new USE_EXPAND variable

[Ryan Hill]

[-- Attachment #1: Type: text/plain, Size: 977 bytes --]

On Sat, 11 Dec 2010 19:01:16 +0000
Matt Turner <mattst88@gentoo.org> wrote:

> On Sat, Dec 11, 2010 at 5:57 PM, Jeroen Roovers <jer@gentoo.org> wrote:
> [snip]
> 
> I agree that this could be better. To me, most of the problems with
> this are due to users not knowing which of these should be set for
> their particular CPU.
> 
> Instead of having defaults set by a profile, I'd like to figure out a
> way we can have these flags set by default dependent on the user's
> CPU. This might require some additional logic in portage; I don't
> know.

I think the fewer sources of magic USE flags the better.  Maybe we could
document how to figure out what instruction sets a processor supports in the
handbook instead.


-- 
fonts, gcc-porting,                  it makes no sense how it makes no sense
toolchain, wxwidgets                           but i'll take it free anytime
@ gentoo.org                EFFD 380E 047A 4B51 D2BD C64F 8AA8 8346 F9A4 0662

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Re: Move x86/amd64 CPU extensions USE flags to a new USE_EXPAND variable

["Sergio D. Rodríguez Inclan"]

  El 12/12/2010 02:46 a.m., Ryan Hill escribió:
> On Sat, 11 Dec 2010 19:01:16 +0000
> Matt Turner<mattst88@gentoo.org>  wrote:
>
>> On Sat, Dec 11, 2010 at 5:57 PM, Jeroen Roovers<jer@gentoo.org>  wrote:
>> [snip]
>>
>> I agree that this could be better. To me, most of the problems with
>> this are due to users not knowing which of these should be set for
>> their particular CPU.
>>
>> Instead of having defaults set by a profile, I'd like to figure out a
>> way we can have these flags set by default dependent on the user's
>> CPU. This might require some additional logic in portage; I don't
>> know.
> I think the fewer sources of magic USE flags the better.  Maybe we could
> document how to figure out what instruction sets a processor supports in the
> handbook instead.
>
>
A good manual would be greatly appreciated :)

Regards,
Sergio.

Re: [gentoo-dev] Re: Move x86/amd64 CPU extensions USE flags to a new USE_EXPAND variable

[Matt Turner]

On Sun, Dec 12, 2010 at 6:34 AM, Ryan Hill <dirtyepic@gentoo.org> wrote:
> On Sat, 11 Dec 2010 18:57:58 +0100
> sse5 was a draft, it was never implemented.

I don't think it's quite as simple as that. According to Wikipedia,
AMD has significantly scaled back SSE5 so that it won't interfere with
AVX, but it will be implemented in Bulldozer and will consist of the
new  XOP, FMA4, and CVT16 instruction sets.

Strangely enough, media-libs/spandsp already has an SSE5 USE flag.

Matt

Re: [gentoo-dev] Move x86/amd64 CPU extensions USE flags to a new USE_EXPAND variable

[ross smith]

[-- Attachment #1: Type: text/plain, Size: 1107 bytes --]

On Sat, Dec 11, 2010 at 14:03, Dirkjan Ochtman <djc@gentoo.org> wrote:

> On Sat, Dec 11, 2010 at 20:01, Matt Turner <mattst88@gentoo.org> wrote:
> > I agree that this could be better. To me, most of the problems with
> > this are due to users not knowing which of these should be set for
> > their particular CPU.
> >
> > Instead of having defaults set by a profile, I'd like to figure out a
> > way we can have these flags set by default dependent on the user's
> > CPU. This might require some additional logic in portage; I don't
> > know.
>
> Yeah, I think setting the best possible default is important here.
>

+1 to good defaults.

Perhaps  something along the lines of: If CPU_FLAGS is empty or not defined,
set flags based on the -march and -mtune variables.  If CPU_FLAGS is set,
respect what has been set there and ignore the other logic for defaults.

This would allow the average user, and the users building for multiple
machines to continue like nothing changed, and users that want to
selectively want to enable processor flags can in an easy way.

Best,

Ross

>
> Cheers,
>
> Dirkjan
>
>

[-- Attachment #2: Type: text/html, Size: 1775 bytes --]

[gentoo-dev] Re: Move x86/amd64 CPU extensions USE flags to a new USE_EXPAND variable

[Ryan Hill]

[-- Attachment #1: Type: text/plain, Size: 1147 bytes --]

On Sun, 12 Dec 2010 13:09:08 -0500
ross smith <rjsm@umich.edu> wrote:

> Perhaps  something along the lines of: If CPU_FLAGS is empty or not defined,
> set flags based on the -march and -mtune variables.  If CPU_FLAGS is set,
> respect what has been set there and ignore the other logic for defaults.

-march flags are the lowest common denominator for an architecture.  For
example, core2 enables MMX, SSE, SSE2, SSE3, and SSSE3 while a Core i7
processor also supports SSE4.1, SSE4.2, and AES instructions.  It might be
useful to use -march as a base and leave it up to the user to set any
additional flags, but I don't see how that is any different than what we have
now.

In addition it would require us to maintain accurate mappings of what -march
enables what flags and deal with the fact that these flags may not exist
between gcc versions.  It also falls apart with -march=native.

-- 
fonts, gcc-porting,                  it makes no sense how it makes no sense
toolchain, wxwidgets                           but i'll take it free anytime
@ gentoo.org                EFFD 380E 047A 4B51 D2BD C64F 8AA8 8346 F9A4 0662

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-dev] Re: Move x86/amd64 CPU extensions USE flags to a new USE_EXPAND variable

[Ryan Hill]

[-- Attachment #1: Type: text/plain, Size: 748 bytes --]

On Sun, 12 Dec 2010 09:01:13 -0400
"Sergio D. Rodríguez Inclan" <srinclan@gmail.com> wrote:

>   El 12/12/2010 02:46 a.m., Ryan Hill escribió:
> > I think the fewer sources of magic USE flags the better.  Maybe we could
> > document how to figure out what instruction sets a processor supports in the
> > handbook instead.

> A good manual would be greatly appreciated :)

I wrote a guide a couple weeks ago that might be a good starting point.

http://en.gentoo-wiki.com/wiki/Hardware_CFLAGS

-- 
fonts, gcc-porting,                  it makes no sense how it makes no sense
toolchain, wxwidgets                           but i'll take it free anytime
@ gentoo.org                EFFD 380E 047A 4B51 D2BD C64F 8AA8 8346 F9A4 0662

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Re: Move x86/amd64 CPU extensions USE flags to a new USE_EXPAND variable

[Robin H. Johnson]

On Sun, Dec 12, 2010 at 11:09:01PM -0600, Ryan Hill wrote:
> On Sun, 12 Dec 2010 09:01:13 -0400
> "Sergio D. Rodríguez Inclan" <srinclan@gmail.com> wrote:
> 
> >   El 12/12/2010 02:46 a.m., Ryan Hill escribió:
> > > I think the fewer sources of magic USE flags the better.  Maybe we could
> > > document how to figure out what instruction sets a processor supports in the
> > > handbook instead.
> > A good manual would be greatly appreciated :)
> I wrote a guide a couple weeks ago that might be a good starting point.
> http://en.gentoo-wiki.com/wiki/Hardware_CFLAGS
For older stuff and esp. alternative architectures, dig out the genflags
utility that I wrote for the 2004.0 release, at drobbin's request.

The data would be the main stuff of use, the application itself not so
much.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

Re: [gentoo-dev] Re: Move x86/amd64 CPU extensions USE flags to a new USE_EXPAND variable

[Francesco R]

[-- Attachment #1: Type: text/plain, Size: 4120 bytes --]

2010/12/13 Ryan Hill <dirtyepic@gentoo.org>

> On Sun, 12 Dec 2010 09:01:13 -0400
> "Sergio D. Rodríguez Inclan" <srinclan@gmail.com> wrote:
>
> >   El 12/12/2010 02:46 a.m., Ryan Hill escribió:
> > > I think the fewer sources of magic USE flags the better.  Maybe we
> could
> > > document how to figure out what instruction sets a processor supports
> in the
> > > handbook instead.
>
> > A good manual would be greatly appreciated :)
>
> I wrote a guide a couple weeks ago that might be a good starting point.
>
> http://en.gentoo-wiki.com/wiki/Hardware_CFLAGS
>
>
if I read correctly the article on the wiki it does circa what the script
reported below does.
would be possible to adopt something similar for automatic C*FLAGS selection
if someone step in willing to take the pain to mantain it.

#!/usr/bin/python
# Copyright 2010 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# Author: Francesco Riosa
# extrapolated from http://en.gentoo-wiki.com/wiki/Hardware_CFLAGS, errors
are mine

# kate: encoding utf-8; eol unix
# kate: indent-width 4; mixedindent off; replace-tabs on;
# kate: remove-trailing-space on; space-indent on

# echo "int main() { return 0; }" | gcc -march=native -v -E - 2>&1 | grep
march
# echo "int main() { return 0; }" | gcc -march=core2 -v -Q -x c - 2>&1

"""
example output:
./hw-cflags.py
extrapolating flags for gcc-4.4.5
  useful flags: -march=core2 -msse4.1 --param l1-cache-size=32 --param
l1-cache-line-size=64 --param l2-cache-size=2048 -mtune=generic
  redundant:    -mcx16 -msahf

extrapolating flags for gcc-4.5.1
  useful flags: -march=core2 -msse4.1 --param l1-cache-size=32 --param
l1-cache-line-size=64 --param l2-cache-size=2048 -mtune=core2
  redundant:    -mcx16 -msahf
"""

import os
import time
import fnmatch
from subprocess import Popen, PIPE

GCC_PATH = '/usr/bin/'
GCC_LIST = fnmatch.filter(os.listdir(GCC_PATH), 'gcc-[0-9].*')
GCC_LIST.sort()

def extract_flags(gcccmd, header):
    # get output from gcc
    buf = ''
    devnul = open('/dev/null', 'w')
    p = Popen(gcccmd, stdin=PIPE, stdout=devnul, stderr=PIPE)
    p.stdin.write("""int main() { return 0; }""")
    p.stdin.close()
    while p.poll() is None:
        t = p.stderr.read()
        buf = "buf%s" % t
        time.sleep(0.01)
    p.stderr.close()
    devnul.close()

    # parse it
    flags = []
    add = False
    for line in buf.split('\n'):
        if line.startswith(header):
            add = True
            flags += line.strip().split(' ')
            continue
        if add:
            if line.startswith(' '):
                flags += line.strip().split(' ')
            else:
                break

    # extract flags we are interested in
    t = []
    march = ''
    mtune = '-mtune=generic'
    for i in xrange(len(flags)):
        if flags[i].startswith('-m'):
            if flags[i].startswith('-mtune'):
                mtune = flags[i]
            elif flags[i].startswith('-march'):
                march = flags[i]
            else:
                t.append(flags[i])
        elif flags[i] == '--param':
            t.append("%s %s" % (flags[i], flags[i+1]))
    flags = t

    return march, mtune, flags


for gcc in GCC_LIST:

    print "extrapolating flags for %s" % gcc

    gcccmd = [ GCC_PATH + gcc, '-march=native', '-v', '-E', '-', ]
    header='COLLECT_GCC_OPTIONS'
    march, mtune, flags_native = extract_flags(gcccmd, header)

    gcccmd = [ GCC_PATH + gcc, march, '-v', '-Q', '-x', 'c', '-', ]
    header='options enabled:'
    t, t, flags_enabled = extract_flags(gcccmd, header)

    redundant_flags = []
    useful_flags = []

    for x in flags_native:
        if x in flags_enabled:
            redundant_flags.append(x)
        else:
            useful_flags.append(x)

    if gcc < "gcc-4.5.0":
        mtune = '-mtune=generic'

    print "  useful flags: %s %s %s " % (march, " ".join(useful_flags),
mtune)
    print "  redundant:    %s" % " ".join(redundant_flags)
    print

[-- Attachment #2: Type: text/html, Size: 6218 bytes --]

Re: [gentoo-dev] Re: Move x86/amd64 CPU extensions USE flags to a new USE_EXPAND variable

[Konstantin Tokarev]

13.12.2010, 18:53, "Francesco R" <vivo75@gmail.com>:
> 2010/12/13 Ryan Hill <dirtyepic@gentoo.org>
>> On Sun, 12 Dec 2010 09:01:13 -0400
>> "Sergio D. Rodríguez Inclan" <srinclan@gmail.com> wrote:
>>
>>>   El 12/12/2010 02:46 a.m., Ryan Hill escribió:
>>> > I think the fewer sources of magic USE flags the better.  Maybe we could
>>> > document how to figure out what instruction sets a processor supports in the
>>> > handbook instead.
>>
>>> A good manual would be greatly appreciated :)
>>
>> I wrote a guide a couple weeks ago that might be a good starting point.
>>
>> http://en.gentoo-wiki.com/wiki/Hardware_CFLAGS
>
> if I read correctly the article on the wiki it does circa what the script reported below does.
> would be possible to adopt something similar for automatic C*FLAGS selection if someone step in willing to take the pain to mantain it.
>
> #!/usr/bin/python
> # Copyright 2010 Gentoo Foundation
> # Distributed under the terms of the GNU General Public License v2
> # Author: Francesco Riosa
> # extrapolated from http://en.gentoo-wiki.com/wiki/Hardware_CFLAGS, errors are mine
>
> # kate: encoding utf-8; eol unix
> # kate: indent-width 4; mixedindent off; replace-tabs on;
> # kate: remove-trailing-space on; space-indent on
>
> # echo "int main() { return 0; }" | gcc -march=native -v -E - 2>&1 | grep march
> # echo "int main() { return 0; }" | gcc -march=core2 -v -Q -x c - 2>&1
>
> """
> example output:
> ./hw-cflags.py
> extrapolating flags for gcc-4.4.5
>   useful flags: -march=core2 -msse4.1 --param l1-cache-size=32 --param l1-cache-line-size=64 --param l2-cache-size=2048 -mtune=generic
>   redundant:    -mcx16 -msahf
>
> extrapolating flags for gcc-4.5.1
>   useful flags: -march=core2 -msse4.1 --param l1-cache-size=32 --param l1-cache-line-size=64 --param l2-cache-size=2048 -mtune=core2
>   redundant:    -mcx16 -msahf
> """
>
> import os
> import time
> import fnmatch
> from subprocess import Popen, PIPE
>
> GCC_PATH = '/usr/bin/'
> GCC_LIST = fnmatch.filter(os.listdir(GCC_PATH), 'gcc-[0-9].*')
> GCC_LIST.sort()
>
> def extract_flags(gcccmd, header):
>     # get output from gcc
>     buf = ''
>     devnul = open('/dev/null', 'w')
>     p = Popen(gcccmd, stdin=PIPE, stdout=devnul, stderr=PIPE)
>     p.stdin.write("""int main() { return 0; }""")
>     p.stdin.close()
>     while p.poll() is None:
>         t = p.stderr.read()
>         buf = "buf%s" % t
>         time.sleep(0.01)
>     p.stderr.close()
>     devnul.close()
>
>     # parse it
>     flags = []
>     add = False
>     for line in buf.split('\n'):
>         if line.startswith(header):
>             add = True
>             flags += line.strip().split(' ')
>             continue
>         if add:
>             if line.startswith(' '):
>                 flags += line.strip().split(' ')
>             else:
>                 break
>
>     # extract flags we are interested in
>     t = []
>     march = ''
>     mtune = '-mtune=generic'
>     for i in xrange(len(flags)):
>         if flags[i].startswith('-m'):
>             if flags[i].startswith('-mtune'):
>                 mtune = flags[i]
>             elif flags[i].startswith('-march'):
>                 march = flags[i]
>             else:
>                 t.append(flags[i])
>         elif flags[i] == '--param':
>             t.append("%s %s" % (flags[i], flags[i+1]))
>     flags = t
>
>     return march, mtune, flags
>
> for gcc in GCC_LIST:
>
>     print "extrapolating flags for %s" % gcc
>
>     gcccmd = [ GCC_PATH + gcc, '-march=native', '-v', '-E', '-', ]
>     header='COLLECT_GCC_OPTIONS'
>     march, mtune, flags_native = extract_flags(gcccmd, header)
>
>     gcccmd = [ GCC_PATH + gcc, march, '-v', '-Q', '-x', 'c', '-', ]
>     header='options enabled:'
>     t, t, flags_enabled = extract_flags(gcccmd, header)
>
>     redundant_flags = []
>     useful_flags = []
>
>     for x in flags_native:
>         if x in flags_enabled:
>             redundant_flags.append(x)
>         else:
>             useful_flags.append(x)
>
>     if gcc < "gcc-4.5.0":
>         mtune = '-mtune=generic'
>
>     print "  useful flags: %s %s %s " % (march, " ".join(useful_flags), mtune)
>     print "  redundant:    %s" % " ".join(redundant_flags)
>     print

Note that for some architectures (e.g, PowerPC) you need to add "-mcpu" case
(-mcpu has the same meaning as -march on x86)


-- 
Regards,
Konstantin

Re: [gentoo-dev] Re: Move x86/amd64 CPU extensions USE flags to a new USE_EXPAND variable

[Francesco R]

[-- Attachment #1: Type: text/plain, Size: 5316 bytes --]

2010/12/13 Konstantin Tokarev <annulen@yandex.ru>

>
>
> 13.12.2010, 18:53, "Francesco R" <vivo75@gmail.com>:
> > 2010/12/13 Ryan Hill <dirtyepic@gentoo.org>
> >> On Sun, 12 Dec 2010 09:01:13 -0400
> >> "Sergio D. Rodríguez Inclan" <srinclan@gmail.com> wrote:
> >>
> >>>   El 12/12/2010 02:46 a.m., Ryan Hill escribió:
> >>> > I think the fewer sources of magic USE flags the better.  Maybe we
> could
> >>> > document how to figure out what instruction sets a processor supports
> in the
> >>> > handbook instead.
> >>
> >>> A good manual would be greatly appreciated :)
> >>
> >> I wrote a guide a couple weeks ago that might be a good starting point.
> >>
> >> http://en.gentoo-wiki.com/wiki/Hardware_CFLAGS
> >
> > if I read correctly the article on the wiki it does circa what the script
> reported below does.
> > would be possible to adopt something similar for automatic C*FLAGS
> selection if someone step in willing to take the pain to mantain it.
> >
> > #!/usr/bin/python
> > # Copyright 2010 Gentoo Foundation
> > # Distributed under the terms of the GNU General Public License v2
> > # Author: Francesco Riosa
> > # extrapolated from http://en.gentoo-wiki.com/wiki/Hardware_CFLAGS,
> errors are mine
> >
> > # kate: encoding utf-8; eol unix
> > # kate: indent-width 4; mixedindent off; replace-tabs on;
> > # kate: remove-trailing-space on; space-indent on
> >
> > # echo "int main() { return 0; }" | gcc -march=native -v -E - 2>&1 | grep
> march
> > # echo "int main() { return 0; }" | gcc -march=core2 -v -Q -x c - 2>&1
> >
> > """
> > example output:
> > ./hw-cflags.py
> > extrapolating flags for gcc-4.4.5
> >   useful flags: -march=core2 -msse4.1 --param l1-cache-size=32 --param
> l1-cache-line-size=64 --param l2-cache-size=2048 -mtune=generic
> >   redundant:    -mcx16 -msahf
> >
> > extrapolating flags for gcc-4.5.1
> >   useful flags: -march=core2 -msse4.1 --param l1-cache-size=32 --param
> l1-cache-line-size=64 --param l2-cache-size=2048 -mtune=core2
> >   redundant:    -mcx16 -msahf
> > """
> >
> > import os
> > import time
> > import fnmatch
> > from subprocess import Popen, PIPE
> >
> > GCC_PATH = '/usr/bin/'
> > GCC_LIST = fnmatch.filter(os.listdir(GCC_PATH), 'gcc-[0-9].*')
> > GCC_LIST.sort()
> >
> > def extract_flags(gcccmd, header):
> >     # get output from gcc
> >     buf = ''
> >     devnul = open('/dev/null', 'w')
> >     p = Popen(gcccmd, stdin=PIPE, stdout=devnul, stderr=PIPE)
> >     p.stdin.write("""int main() { return 0; }""")
> >     p.stdin.close()
> >     while p.poll() is None:
> >         t = p.stderr.read()
> >         buf = "buf%s" % t
> >         time.sleep(0.01)
> >     p.stderr.close()
> >     devnul.close()
> >
> >     # parse it
> >     flags = []
> >     add = False
> >     for line in buf.split('\n'):
> >         if line.startswith(header):
> >             add = True
> >             flags += line.strip().split(' ')
> >             continue
> >         if add:
> >             if line.startswith(' '):
> >                 flags += line.strip().split(' ')
> >             else:
> >                 break
> >
> >     # extract flags we are interested in
> >     t = []
> >     march = ''
> >     mtune = '-mtune=generic'
> >     for i in xrange(len(flags)):
> >         if flags[i].startswith('-m'):
> >             if flags[i].startswith('-mtune'):
> >                 mtune = flags[i]
> >             elif flags[i].startswith('-march'):
> >                 march = flags[i]
> >             else:
> >                 t.append(flags[i])
> >         elif flags[i] == '--param':
> >             t.append("%s %s" % (flags[i], flags[i+1]))
> >     flags = t
> >
> >     return march, mtune, flags
> >
> > for gcc in GCC_LIST:
> >
> >     print "extrapolating flags for %s" % gcc
> >
> >     gcccmd = [ GCC_PATH + gcc, '-march=native', '-v', '-E', '-', ]
> >     header='COLLECT_GCC_OPTIONS'
> >     march, mtune, flags_native = extract_flags(gcccmd, header)
> >
> >     gcccmd = [ GCC_PATH + gcc, march, '-v', '-Q', '-x', 'c', '-', ]
> >     header='options enabled:'
> >     t, t, flags_enabled = extract_flags(gcccmd, header)
> >
> >     redundant_flags = []
> >     useful_flags = []
> >
> >     for x in flags_native:
> >         if x in flags_enabled:
> >             redundant_flags.append(x)
> >         else:
> >             useful_flags.append(x)
> >
> >     if gcc < "gcc-4.5.0":
> >         mtune = '-mtune=generic'
> >
> >     print "  useful flags: %s %s %s " % (march, " ".join(useful_flags),
> mtune)
> >     print "  redundant:    %s" % " ".join(redundant_flags)
> >     print
>
> Note that for some architectures (e.g, PowerPC) you need to add "-mcpu"
> case
> (-mcpu has the same meaning as -march on x86)
>
>
> --
> Regards,
> Konstantin
>
>
Assuming there is some possibility of Gentoo using some automated method of
choice of CFLAGS the arch maintainers should really step in and tell what
works on their cpus and be able to modify/disable it.

There are other shortcomings in the script, for example it does not work
with <gcc-4.2 because it use -march=native.
Also it should at least consider what's in /etc/make.conf C*FLAGS/CHOST.
It's really a proof of concept

[-- Attachment #2: Type: text/html, Size: 7088 bytes --]

Re: [gentoo-dev] Re: Move x86/amd64 CPU extensions USE flags to a new USE_EXPAND variable

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 1372 bytes --]

Hi all,

On Mon, Dec 13, 2010 at 07:07:10PM +0100, Francesco R wrote:
> 2010/12/13 Konstantin Tokarev <annulen@yandex.ru>
> 
> >
> >
> > 13.12.2010, 18:53, "Francesco R" <vivo75@gmail.com>:
> > > 2010/12/13 Ryan Hill <dirtyepic@gentoo.org>
> > >> On Sun, 12 Dec 2010 09:01:13 -0400
> > >> "Sergio D. Rodr?guez Inclan" <srinclan@gmail.com> wrote:
> > >>
> > >>>   El 12/12/2010 02:46 a.m., Ryan Hill escribi?:
> > >>> > I think the fewer sources of magic USE flags the better.  Maybe we
> > could
> > >>> > document how to figure out what instruction sets a processor supports
> > in the
> > >>> > handbook instead.
> > >>
> > >>> A good manual would be greatly appreciated :)
> > >>
> > >> I wrote a guide a couple weeks ago that might be a good starting point.
> > >>
> > >> http://en.gentoo-wiki.com/wiki/Hardware_CFLAGS
> > >
> > > if I read correctly the article on the wiki it does circa what the script
> > reported below does.
> > > would be possible to adopt something similar for automatic C*FLAGS
> > selection if someone step in willing to take the pain to mantain it.

I've been reading this thread, but I don't understand why we need to
worry about this since the newer versions of gcc can figure it out automatically
by using -march=native? Does -mcpu also have a native option for those
who need to use -mcpu?

William


[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-dev] Re: Move x86/amd64 CPU extensions USE flags to a new USE_EXPAND variable

[Ryan Hill]

[-- Attachment #1: Type: text/plain, Size: 779 bytes --]

On Mon, 13 Dec 2010 18:13:08 -0600
William Hubbs <williamh@gentoo.org> wrote:

> I've been reading this thread, but I don't understand why we need to
> worry about this since the newer versions of gcc can figure it out automatically
> by using -march=native?

Absolutely, -march=native should be used if you aren't using distcc.  This is
about setting CPU-based USE flags (mmx, sse, etc.).

> Does -mcpu also have a native option for those who need to use -mcpu?

I think native is limited to amd64, x86, alpha, and mips.


-- 
fonts, gcc-porting,                  it makes no sense how it makes no sense
toolchain, wxwidgets                           but i'll take it free anytime
@ gentoo.org                EFFD 380E 047A 4B51 D2BD C64F 8AA8 8346 F9A4 0662

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Re: Move x86/amd64 CPU extensions USE flags to a new USE_EXPAND variable

[Jeroen Roovers]

On Mon, 13 Dec 2010 18:41:07 -0600
Ryan Hill <dirtyepic@gentoo.org> wrote:

> Absolutely, -march=native should be used if you aren't using distcc.
> This is about setting CPU-based USE flags (mmx, sse, etc.).

Exactly, or x86 CPU extensions based USE flags, more exactly.

It seems the thread has been somewhat hijacked in favour of automating
setting USE flags or at least helping to set them based on information
found here and there on the system that happens to be queried for it,
something we have never supported since the thankfully forgotten days of
<profiles/base/use.defaults>. And rather than trusting gcc
-march=native to do the work for you, you would probably be better off
checking `grep ^flags /proc/cpuinfo' anyway. :)


     jer

Re: [gentoo-dev] Re: Move x86/amd64 CPU extensions USE flags to a new USE_EXPAND variable

[Konstantin Tokarev]

> And rather than trusting gcc
> -march=native to do the work for you, you would probably be better off
> checking `grep ^flags /proc/cpuinfo' anyway. :)
>

Gentoo runs not only on Linux (remember about Gentoo Prefix), and this feature is 
system-dependend

-- 
Regards,
Konstantin