[gentoo-portage-dev] src_profile() request for comments

[gentoo-portage-dev] src_profile() request for comments

[Bastian Balthazar Bux]

>From man gcc:

-fprofile-generate
    Enable options usually used for instrumenting application to produce
    profile useful for later recompilation with profile
    feedback based optimization.  You must use "-fprofile-generate" both
    when compiling and when linking your program.

    The following options are enabled: "-fprofile-arcs",
    "-fprofile-values", "-fvpt".

-fprofile-use
    Enable profile feedback directed optimizations, and optimizations
    generally profitable only with profile feedback available.

    The following options are enabled: "-fbranch-probabilities",
    "-fvpt", "-funroll-loops", "-fpeel-loops", "-ftracer".


Starting saying that this staff have never been tried by me, make sense
to add a specific optional function src_profile() to manage the eventual
profiling?

With this option the flow of an emerge should be something like this:

 --------------
|  pkg_setup   |...src.....
 --------------           .
       .                  .
      bin          --------------
       .          |  src_unpack  |
       .           --------------
       .                  .
       .           --------------
       .          | src_compile  |........................
       .           --------------           .            .
       .                  .                 .            .
       .                  .           --------------     .
       .                  .          |   src_test   |.....
       .                  .           --------------     .
       .                  .                 .            .
       .                  .                 .     ################
       .                  .                 .     # src_profile  #
       .                  .                 .     ################
       .           --------------           .            .
       .          | src_install  |........................
       .           --------------
       .                  .
 --------------           .
| pkg_preinst  |...........
 --------------
       .
 --------------
| pkg_postinst |
 --------------


with src_profile() responsible to do basically two things:
- run the builded app with some fake data
- rebuild the source using the gathered profile info.

The first step within some ebuilds could be done by src_test, the second
is basically a reduced src_compile.

any thoughts ?

-- 
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] src_profile() request for comments

[Jason Stubbs]

On Wednesday 28 September 2005 23:42, Bastian Balthazar Bux wrote:
> with src_profile() responsible to do basically two things:
> - run the builded app with some fake data
> - rebuild the source using the gathered profile info.
>
> The first step within some ebuilds could be done by src_test, the second
> is basically a reduced src_compile.
>
> any thoughts ?

       -fno-guess-branch-probability
           Do not guess branch probabilities using a randomized model.

           Sometimes GCC will opt to use a randomized model to guess branch
           probabilities, when none are available from either profiling feed-
           back (-fprofile-arcs) or __builtin_expect.  This means that differ-
           ent runs of the compiler on the same program may produce different
           object code.

           In a hard real-time system, people don't want different runs of the
           compiler to produce code that has different behavior; minimizing
           non-determinism is of paramount import.  This switch allows users
           to reduce non-determinism, possibly at the expense of inferior
           optimization.

           The default is -fguess-branch-probability at levels -O, -O2, -O3,
           -Os.

Any "test data" that could be used to automatically profile is essentially 
just a randomized model. So it would seem gcc has it inbuilt already.

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] src_profile() request for comments

[Paul de Vrieze]

[-- Attachment #1: Type: text/plain, Size: 1990 bytes --]

On Wednesday 28 September 2005 17:10, Jason Stubbs wrote:
> On Wednesday 28 September 2005 23:42, Bastian Balthazar Bux wrote:
> > with src_profile() responsible to do basically two things:
> > - run the builded app with some fake data
> > - rebuild the source using the gathered profile info.
> >
> > The first step within some ebuilds could be done by src_test, the
> > second is basically a reduced src_compile.
> >
> > any thoughts ?
>
>        -fno-guess-branch-probability
>            Do not guess branch probabilities using a randomized model.
>
>            Sometimes GCC will opt to use a randomized model to guess
> branch probabilities, when none are available from either profiling
> feed- back (-fprofile-arcs) or __builtin_expect.  This means that
> differ- ent runs of the compiler on the same program may produce
> different object code.
>
>            In a hard real-time system, people don't want different runs
> of the compiler to produce code that has different behavior; minimizing
> non-determinism is of paramount import.  This switch allows users to
> reduce non-determinism, possibly at the expense of inferior
> optimization.
>
>            The default is -fguess-branch-probability at levels -O, -O2,
> -O3, -Os.
>
> Any "test data" that could be used to automatically profile is
> essentially just a randomized model. So it would seem gcc has it
> inbuilt already.

The difference is that the randomized model used by gcc is not based on 
possible input to the program (as that might be very big), but on 
possible input to more lowlevel concepts such as functions (or even the 
condition statement). As such test data, if reasonably representative of 
actual data for the application, would give a rather accurate profile of 
the branching probabilities and program paths. The more data the better 
of course.

Paul

-- 
Paul de Vrieze
Gentoo Developer
Mail: pauldv@gentoo.org
Homepage: http://www.devrieze.net

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-portage-dev] src_profile() request for comments

[Jason Stubbs]

On Thursday 29 September 2005 00:27, Paul de Vrieze wrote:
> On Wednesday 28 September 2005 17:10, Jason Stubbs wrote:
> > On Wednesday 28 September 2005 23:42, Bastian Balthazar Bux wrote:
> > > with src_profile() responsible to do basically two things:
> > > - run the builded app with some fake data
> > > - rebuild the source using the gathered profile info.
> > >
> > > The first step within some ebuilds could be done by src_test, the
> > > second is basically a reduced src_compile.
> > >
> > > any thoughts ?
> >
> >        -fno-guess-branch-probability
> >            Do not guess branch probabilities using a randomized model.
> >
> >            Sometimes GCC will opt to use a randomized model to guess
> > branch probabilities, when none are available from either profiling
> > feed- back (-fprofile-arcs) or __builtin_expect.  This means that
> > differ- ent runs of the compiler on the same program may produce
> > different object code.
> >
> >            In a hard real-time system, people don't want different runs
> > of the compiler to produce code that has different behavior; minimizing
> > non-determinism is of paramount import.  This switch allows users to
> > reduce non-determinism, possibly at the expense of inferior
> > optimization.
> >
> >            The default is -fguess-branch-probability at levels -O, -O2,
> > -O3, -Os.
> >
> > Any "test data" that could be used to automatically profile is
> > essentially just a randomized model. So it would seem gcc has it
> > inbuilt already.
>
> The difference is that the randomized model used by gcc is not based on
> possible input to the program (as that might be very big), but on
> possible input to more lowlevel concepts such as functions (or even the
> condition statement). As such test data, if reasonably representative of
> actual data for the application, would give a rather accurate profile of
> the branching probabilities and program paths. The more data the better
> of course.

Yes, but I can't think of an application that would be profileable through 
automated means and have a useful set of test data that is relevant to real 
world execution. My point was that if it's not based on real world then it's 
essentially random data.

Having it so that portage installs a profiled version which automatically 
collects data and stores it somewhere for portage to use in a recompile 
sounds like more of a worthwhile persuit to my ears.

--
Jason Stubbs

-- 
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] src_profile() request for comments

[Bastian Balthazar Bux]

Jason Stubbs wrote:
> On Thursday 29 September 2005 00:27, Paul de Vrieze wrote:
> 
>>On Wednesday 28 September 2005 17:10, Jason Stubbs wrote:
>>
>>>On Wednesday 28 September 2005 23:42, Bastian Balthazar Bux wrote:
>>>
>>>>with src_profile() responsible to do basically two things:
>>>>- run the builded app with some fake data
>>>>- rebuild the source using the gathered profile info.
>>>>
>>>>The first step within some ebuilds could be done by src_test, the
>>>>second is basically a reduced src_compile.
>>>>
>>>>any thoughts ?
>>>
>>>       -fno-guess-branch-probability
>>>           Do not guess branch probabilities using a randomized model.
>>>
>>>           Sometimes GCC will opt to use a randomized model to guess
>>>branch probabilities, when none are available from either profiling
>>>feed- back (-fprofile-arcs) or __builtin_expect.  This means that
>>>differ- ent runs of the compiler on the same program may produce
>>>different object code.
>>>
>>>           In a hard real-time system, people don't want different runs
>>>of the compiler to produce code that has different behavior; minimizing
>>>non-determinism is of paramount import.  This switch allows users to
>>>reduce non-determinism, possibly at the expense of inferior
>>>optimization.
>>>
>>>           The default is -fguess-branch-probability at levels -O, -O2,
>>>-O3, -Os.
>>>
>>>Any "test data" that could be used to automatically profile is
>>>essentially just a randomized model. So it would seem gcc has it
>>>inbuilt already.
>>
>>The difference is that the randomized model used by gcc is not based on
>>possible input to the program (as that might be very big), but on
>>possible input to more lowlevel concepts such as functions (or even the
>>condition statement). As such test data, if reasonably representative of
>>actual data for the application, would give a rather accurate profile of
>>the branching probabilities and program paths. The more data the better
>>of course.
> 
> 
> Yes, but I can't think of an application that would be profileable through 
> automated means and have a useful set of test data that is relevant to real 
> world execution. My point was that if it's not based on real world then it's 
> essentially random data.
> 
> Having it so that portage installs a profiled version which automatically 
> collects data and stores it somewhere for portage to use in a recompile 
> sounds like more of a worthwhile persuit to my ears.
> 
> --
> Jason Stubbs
> 


To say it all, the starting point of this mumbleing was to achieve
optimizations on MySQL.

IT's very difficult to me mind at how a c-code optimizer, with any
degree of intelligence, could predict the usage of such a big piece of
code (At least using a finite amount of time).

The idea was to select a certain number of tests from the included
test-suite, the ones more rappresentative of a normal usage and then
pass the gathered data again to gcc that could do a better work.

As said before, this is still mumbleing, not even tryed it ;)

IMHO other packages may want this expecially becouse seems that this
kind of profiling is going to become better and better with new releases
of gcc.
-- 
gentoo-portage-dev@gentoo.org mailing list