[gentoo-dev] debug/release builds extensions/clarification proposal

[gentoo-dev] debug/release builds extensions/clarification proposal

[Maciej Mrozowski]

[-- Attachment #1: Type: text/plain, Size: 4383 bytes --]

Hi

I would like to give some idea into consideration.

Abstract
In short, adding following new variables to make.conf and implement handling 
of them in eclasses:
- CFLAGS_DEBUG (and friends like CXXFLAGS_DEBUG) - use defined debug compiler 
flags - by default set to -O0 -ggdb (and maybe -Wall as well)
- LDFLAGS_DEBUG - user defined debug linker flags - default to "${LDFLAGS}"
- FEATURES_DEBUG - default to "${FEATURES} nostrip" (or splidebug, according 
to preference)

Background
Currently handling debug/release builds is incoherent and misleading to say 
the least. We have got in Gentoo:
- CFLAGS/LDFLAGS - user needs to take care of adding -O0 and/or -ggdb
- user needs to take care of FEATURES=nostrip or FEATURES=splitdebug (not 
both)
- user may set debug USE flag

The drawbacks are as follows:
- USE=debug is useless  when CFLAGS/LDFLAGS or FEATURES are not appropriate
- CFLAGS/LDFLAGS must be set globally when they are about to be "supported"
- those who don't want to set them globally, they are forced to use (very 
flexible and great indeed) /etc/portage/env hack - which is undocumented and 
unsupported, because everything user set there, is not shown by emerge --info, 
thus bug reports from such machines  are not taken into consideration, as 
virtually everything that breaks can be there
- too much choice leads to confusion, and many users who enabled some of those 
(but not all), still don't get useful backtraces and only fool themselves when 
reporting something upstream.

Motivation:
The idea is modeled after handling such situations in CMake build system. I'm 
one of contributors to official Gentoo KDE team experimental overlay (kde-
crazy) and we provide live ebuilds and betas/snapshots for KDE4 and related 
applications. It's quite probable that many of our users participate in KDE 
testing, it's vital to provide for them an easy way of setting up testing box 
(though it's not the motive here).
KDE4 uses CMake as build system and CMake out-of-the-box provides build 
configurations: Release, Debug, RelWithDebInfo, MinSizeRel (one can easily 
figure out what they mean). For typical use, user doesn't want nor needs more 
than two configurations - let's call them Release and Debug - it fits in 
existing USE=debug handling scheme, where there are two build types with 
"release" as default. Overlay team and Gentoo KDE developers already make use 
of this feature and we provide debug USE flag for all KDE4 packages. The 
motive is to make handling build type in more coherent, predictable and less 
confusing way and I guess this proposal addresses it quite well.

Implementation:
Implementation would be provided by build system eclasses - for far cmake 
eclasses could benefit from this extension, though new USE=debug capable 
eclass could be introduced for other build systems (including autotools). 
Implementation is trivial - eclass would be responsible for handling USE=debug 
flag, when debug is set:
- replace CFLAGS with CFLAGS_DEBUG, LDFLAGS with LDFLAGS_DEBUG and possibly 
others
- replace FEATURES with FEATURES_DEBUG
Implementation could as well automatically add "debug" to IUSE as well - 
specific packages that are not interested in such flag - would just override 
IUSE in its ebuilds.
For cmake-utils - handling debug IUSE is already done, only replacing 
CFLAGS/LDFLAGS/FEATURES would be requited.
For autoconf based packages - some of them already provide 'support' for debug 
builds ('a'ka --enable-debug), but making use of debug USE would need special 
support here or separate eclass that could be introduced for packages that can 
benefit. If could be as well enforced globally (by adding either --enable-
debug or --disable-debug apart from switching CFLAGS - as confgiure scipt 
ignores unknown arguments) but that's just a matter or implementation.

Backward compatibility
As extension operates on newly introduced variables - backward compatibility 
is preserved. Backward compatibility may be "broken" for those who utilize 
/etc/portage/env hack as strange compiler/iinker flag/FEATURE combinations may 
appear.

In similar scheme more build profiles could be implemented, (like libs/apps 
ready for profiling) but let's leave it alone for now.

Discussion and constructive criticism is welcome :)

-- 
regards
MM


[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[gentoo-dev] Re: debug/release builds extensions/clarification proposal

[Duncan]

Maciej Mrozowski <reavertm@poczta.fm> posted
200812010616.08357.reavertm@poczta.fm, excerpted below, on  Mon, 01 Dec
2008 06:16:07 +0100:

> Implementation:
> Implementation would be provided by build system eclasses [snip]
> - replace FEATURES with FEATURES_DEBUG

FEATURES are package-manager implemented, above the bash level where 
eclasses are parsed and executed, thus for portage, at the python level.  
As such, neither /etc/portage/env nor eclasses can effectively deal with 
FEATURES in general, tho there are a few specific exceptions that do 
happen to be implemented at the bash level.

Thus, your GLEP (Gentoo Linux Enhancement Proposal) needs to specifically 
address this problem, either stating that this FEATURE can be implemented 
100% at the bash/eclass level with details, or omitting/changing the 
FEATURE portion so it will work at the bash/eclass level, or outlining 
specifically what the package manager implementation must be.  (Of 
course, if it's the latter, it will need to be an official GLEP, and 
you'll have three separate package managers and their developers to push 
the proposal thru to at least to general agreement, or the council will 
almost certainly reject the GLEP, if it gets even that far.)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

[gentoo-dev] Re: debug/release builds extensions/clarification proposal

[Diego 'Flameeyes' Pettenò]

[-- Attachment #1: Type: text/plain, Size: 1394 bytes --]

Maciej Mrozowski <reavertm@poczta.fm> writes:

> - USE=debug is useless  when CFLAGS/LDFLAGS or FEATURES are not appropriate

What are you saying here? I'm afraid you're mistaken here.

For the most part, USE=debug means "enable debug code paths", which for
lots of projects simply means "enable assertions"; there are packages
that take this as "enable debug symbols too" but I don't think that's
very valid since users might want debug code paths but not symbols and
vice-versa (I indeed have debug symbols bug no debug codepaths enabled).

Now just to make sure the common misconceptions don't hit again:

- -ggdb *does not have any runtime performance hit*; neither in
   execution time nor in memory usage; the debug sections are not mapped
   into memory at all; this is true for both non-stripped and split
   executables;
- -O0 is not always a good idea; beside bugs in packages concealed by
   -O1+ [1], there are some further points: missing registers on x86
   causes build failures, and if ( 0 ) cases are not optimised away,
   resulting in stuff like FFmpeg not to link properly since undefined
   references are not pruned away; this means that using -O0
   unconditionally for any package for debug is not really an option;

[1] http://blog.flameeyes.eu/2008/09/02/testing-the-corner-cases

-- 
Diego "Flameeyes" Pettenò
http://blog.flameeyes.eu/

[-- Attachment #2: Type: application/pgp-signature, Size: 196 bytes --]

Re: [gentoo-dev] debug/release builds extensions/clarification proposal

[Peter Volkov]

В Пнд, 01/12/2008 в 06:16 +0100, Maciej Mrozowski пишет:
> Currently handling debug/release builds is incoherent and misleading to say 
> the least. We have got in Gentoo:

All that parts do their separate and quite a different work so I can't
say that it's incoherent (by idea at least).

> The drawbacks are as follows:
> - USE=debug is useless  when CFLAGS/LDFLAGS or FEATURES are not appropriate

USE=debug enables additional debug output or more assertions in the
code. It's hard to tell in advance in details what USE=debug does since
different packages enable different things. But generally it adds
additional code with -DDEBUG and this is independent of CFLAGS/LDFLAGS.
If you know packages where this is not true, fill bugs on them.

> - CFLAGS/LDFLAGS must be set globally when they are about to be "supported"
> - those who don't want to set them globally, they are forced to use (very 
> flexible and great indeed) /etc/portage/env hack - which is undocumented and 
> unsupported, because everything user set there, is not shown by emerge --info, 
> thus bug reports from such machines  are not taken into consideration, as 
> virtually everything that breaks can be there

This leads me to different conclusion. I was thinking about new portage
feature: emerge --info <pkg> . So to make portage show not only global
information but per-package either. In many cases this will simplify
analyzing of the problem.

> - too much choice leads to confusion

That's always true. But we use Gentoo because we enjoy our freedom to
choose... Rigth? :)

> Implementation is trivial - eclass would be responsible for handling USE=debug 
> flag, when debug is set:
> - replace CFLAGS with CFLAGS_DEBUG, LDFLAGS with LDFLAGS_DEBUG and possibly 
> others
> - replace FEATURES with FEATURES_DEBUG

USE flags should never change {C,LD}FLAGS or FEATURES as they are
different things and such relation between USE flags, {C,LD}FLAGS and/or
FEATURES will lead to even more confusion. (also there is complexity
Duncan told you about...)

Personally to get build with symbols I use a trivial wrapper around
emerge:

demerge() {
  env USE="debug" CFLAGS="-O2 -pipe -g -ggdb" PKGDIR="/vt/binpkg-debug" \
  FEATURES="buildpkg splitdebug collision-protect ccache noclean installsources" \
    emerge "$@"
}

and I use demerge whenever I need to debug package. I'm sure this is
just a quick hack which could be greatly improved to track which
packages are installed with or without symbols. But you got an idea:
such thing are better to do with separate, but very tiny and simple
wrappers.

P.S. I remember most of this was already discussed in this mailing list.
Try to search it and you'll find much more ideas and motivations.

-- 
Peter.

Re: [gentoo-dev] Re: debug/release builds extensions/clarification proposal

[Maciej Mrozowski]

On Monday 01 of December 2008 08:04:04 Duncan wrote:

Well, so far it's not GLEP, just an idea thrown to brainstorm.

> As such, neither /etc/portage/env nor eclasses can effectively deal with
> FEATURES in general, tho there are a few specific exceptions that do
> happen to be implemented at the bash level.

Those exceptions are nostrip and splitdebug at least, besides I intend to keep 
it bash (or ebuild) level only - to preserve simplicity and yet functionality. 
FEATURES_DEBUG was a clean and convenient approach of me being unaware of 
FEATURES internals - thanks for clarification. FEATURES little inconsistency 
problem needs to be addressed. The goal is to have only one, determined and 
always working way of "not-stripping" symbols. Of course it can be easily 
handled in eclass by something like this:

if use debug; then
   FEATURES=${FEATURES//splitdebug//}
   FEATURES=${FEATURES//nostrip//}
   FEATURES="${FEATURES} ${PREFERRED_NOSTRIP_METHOD}"

Dzwon tanio do wszystkich!
Sprawdz >> http://link.interia.pl/f1fa7

Re: [gentoo-dev] Re: debug/release builds extensions/clarification proposal

[Maciej Mrozowski]

[-- Attachment #1: Type: text/plain, Size: 3273 bytes --]

On Monday 01 of December 2008 09:36:12 Diego 'Flameeyes' Pettenò wrote:
> > - USE=debug is useless  when CFLAGS/LDFLAGS or FEATURES are not
> > appropriate
> What are you saying here? I'm afraid you're mistaken here.

The point is to look at this from users' (well, a bit) point of view - 
USE=debug variable is ambiguous in it's meaning. While it enables only 
codepaths (asserts, #ifdefs and similar) it suggests (by name and for some 
packages not only suggests) enabling debug symbols.
And policy is to enforce CFLAGS from make.conf and wipe out every package-
defined flags as far as I know.

> For the most part, USE=debug means "enable debug code paths", which for
> lots of projects simply means "enable assertions"; there are packages
> that take this as "enable debug symbols too" but I don't think that's
> very valid since users might want debug code paths but not symbols and
> vice-versa (I indeed have debug symbols bug no debug codepaths enabled).

That's correct, the problem is - Gentoo does not provide officially supported 
mechanism of enabling both or just debug symbols per package basis - it 
doesn't even provide any supported/documented mechanism for per package 
CFLAGS, FEATURES and similar.
If /etc/portage/env hack/feature could be made official (for CFLAGS,LDFLAGS 
and bash-domain FEATURES) - it could address this issue good enough, because 
with proper smart combination of symlinks/files the "ultimate configuration" 
power would be delivered, not just "cleaning/workaround" I am actually 
proposing. Per package debug/release/profile/or_any_other configuration is 
what I would pursue, and in my proposal I used USE=debug as existing and 
supported way of achieving this.

While I don't like hack @pve uses (I prefer portage/env as more convenient 
way), his idea about emerge --info <pkg> seems interesting.

> - -ggdb *does not have any runtime performance hit*; neither in

Yes, I'm well aware of that, though it increases disk space requirements a bit 
as it's applied to all libs/bins.

> - -O0 is not always a good idea; beside bugs in packages concealed by
>    -O1+ [1]

[1] is a pathology and should be fought against, -O1+ may leave frame stack 
useless for debugging due to inline optimizations in some places (especially 
debugging inline class implementations is limited, which affects Qt/KDE) - 
besides - I may not stated it clear - those default values would be defined in 
the very same make.conf, so it could be:

CHOST="x86_64-pc-linux-gnu"                                                  
CFLAGS="-march=nocona -O2 -pipe -msse3 -ftree-vectorize"
CXXFLAGS="${CFLAGS}"

CFLAGS_DEBUG="-O2 -ggdb"

Yet, I still cannot think of this proposal other way like of dirty workaround 
for the problem, that doesn't really exist (well, at least for developers, who 
have meta-distribution and ultimate freedom for user in mind).  For the users 
the problem is real, of course it's usually a consequence of either not being 
aware of those mechanisms or as a result of ambiguous semantics of USE=debug.

And what about pushing some bash-domain FEATURES to USE flags? Like nostrip, 
splitdebug? I guess being able to set it per package is important.

-- 
regards
MM

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] debug/release builds extensions/clarification proposal

[Marius Mauch]

On Mon, 01 Dec 2008 11:39:35 +0300
Peter Volkov <pva@gentoo.org> wrote:

> This leads me to different conclusion. I was thinking about new
> portage feature: emerge --info <pkg> . So to make portage show not
> only global information but per-package either. In many cases this
> will simplify analyzing of the problem.

That feature already exists (for installed packages at least).

Marius

Re: [gentoo-dev] Re: debug/release builds extensions/clarification proposal

[Maciej Mrozowski]

[-- Attachment #1: Type: text/plain, Size: 604 bytes --]

On Monday 01 of December 2008 08:04:04 Duncan wrote:
> (Of
> course, if it's the latter, it will need to be an official GLEP, and
> you'll have three separate package managers and their developers to push
> the proposal thru to at least to general agreement, or the council will
> almost certainly reject the GLEP, if it gets even that far.)

That I found interesting - what does any 3rd party package manager to do with 
setting policies and enhancements regarding official Gentoo package manager? 
Have you ever heard of "liberum veto"? But that's an off topic of course.

-- 
regards
MM

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Re: debug/release builds extensions/clarification proposal

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 505 bytes --]

On Mon, 1 Dec 2008 22:46:18 +0100
Maciej Mrozowski <reavertm@poczta.fm> wrote:
> That I found interesting - what does any 3rd party package manager to
> do with setting policies and enhancements regarding official Gentoo
> package manager?

Experience, manpower, the ability to try out potential enhancements
rapidly, a long track record of getting it right and the growing
recognition that most people doing package manager work for Gentoo
aren't doing it with Portage.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Re: debug/release builds extensions/clarification proposal

[Maciej Mrozowski]

[-- Attachment #1: Type: text/plain, Size: 1623 bytes --]

On Monday 01 of December 2008 22:51:57 Ciaran McCreesh wrote:

> Experience, manpower, the ability to try out potential enhancements
> rapidly, a long track record of getting it right and the growing
> recognition that most people doing package manager work for Gentoo
> aren't doing it with Portage.

While of course I agree that any input from 'outside' is welcome and valuable, 
yet to get things done, in my opinion the final decision should not be blocked 
by from any alternative package manager and some policies should be enforced. 

But on topic, what's a counter proposal for my idea then?
Quick search in archives gave me some results I don't particularly like, like 
the idea with /etc/portage/packages.cflags and /etc/portage/package.env, and 
they have been dropped for similar reasons - as the former needs special 
parsing instead just sourcing the script (the problem is that someone needs to 
implement this - this is usually the problem, especially in pure volunteer 
projects like Gentoo), the latter looks a bit messy to me. /etc/portage/env 
would be the best approach when made officially supported (recently it looks 
like /etc/portage/env is sourced multiple times and that should be fixed, for 
convenience, just in case user wants to put:
CFLAGS="-O0 -ggdb"
CXXFLAGS="${CFLAGS}"
FEATURES="${FEATURES} nostrip"
(or even USE="${USE} debug")
actually /etc/portage/env could easily replace package.keywords and 
package.use as well and introduce replacement for meybe-proposed-sometime 
package.features - I wonder whether it's been discussed already.

-- 
regards
MM

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Re: debug/release builds extensions/clarification proposal

[Alec Warner]

On Mon, Dec 1, 2008 at 3:14 PM, Maciej Mrozowski <reavertm@poczta.fm> wrote:
> On Monday 01 of December 2008 22:51:57 Ciaran McCreesh wrote:
>
>> Experience, manpower, the ability to try out potential enhancements
>> rapidly, a long track record of getting it right and the growing
>> recognition that most people doing package manager work for Gentoo
>> aren't doing it with Portage.
>
> While of course I agree that any input from 'outside' is welcome and valuable,
> yet to get things done, in my opinion the final decision should not be blocked
> by from any alternative package manager and some policies should be enforced.
>
> But on topic, what's a counter proposal for my idea then?

<mean hat>

You asked, so the counter proposal is to *do nothing*.

<very mean generic rant hat on>

Ideas (even good ones) don't always get implemented.

Sometimes that just isn't the direction the maintainers want to take
the project.
Sometimes it is harder to implement than most people realize.
Sometimes suggested implementations are just a hack and a bad idea all around.

I think starting with an implementation may have been a bad starting move.

Start with what you want to accomplish:
  - Get feedback on whether this is useful or not.
  - Get feedback on other features that may be available.
  - Get feedback on how some folks would accomplish this.

"I want to be able to turn debug builds on or off on a per-package
basis.  Debug builds entail both debugging symbols, split-debug, debug
CFLAGS and debug LDFLAGS."

Is that a fair summary of your request?

I am unsure how much you actually care about how each package manager
implements this feature (or if anyone implements it but portage, or
paludis, or whatever the majority of the KDE users are using).

I'm also unsure how useful this is when say, some part of KDE links
against libfoo and KDE is built with debug symbols but libfoo is not.
Is that really useful?  Are users actually asking for this proposed
feature or do you just think they want it?  Do you have any data to
back up why someone should implement this feature (mailing list posts,
forums threads, etc..)

Certainly for portage per-package features are possible with a minor
patch (to read the custom settings from your config and to inject the
FEATURES variables into the per-package config when necessary).  The
problem that has been stated in the past is that FEATURES were not
designed to be used in that manner (per-package).

We could design an separate system that let you define per-package
'things' and use these 'things' to trigger debug builds (completely
outside of FEATURES, leaving them alone).  FEATURES were in fact
specific features of portage that you want 'on' or 'off'
(metadata-transfer, parallel-fetch, userfetch, unmerge-orphans,
etc...)  These are examples of things you would not turn off
per-ebuild.

But the question is always 'is it really worth it' and can you get
someone to do it.
Sometimes, doing nothing is better than doing something badly.

<endrant>

-Alec

> Quick search in archives gave me some results I don't particularly like, like
> the idea with /etc/portage/packages.cflags and /etc/portage/package.env, and
> they have been dropped for similar reasons - as the former needs special
> parsing instead just sourcing the script (the problem is that someone needs to
> implement this - this is usually the problem, especially in pure volunteer
> projects like Gentoo), the latter looks a bit messy to me. /etc/portage/env
> would be the best approach when made officially supported (recently it looks
> like /etc/portage/env is sourced multiple times and that should be fixed, for
> convenience, just in case user wants to put:
> CFLAGS="-O0 -ggdb"
> CXXFLAGS="${CFLAGS}"
> FEATURES="${FEATURES} nostrip"
> (or even USE="${USE} debug")
> actually /etc/portage/env could easily replace package.keywords and
> package.use as well and introduce replacement for meybe-proposed-sometime
> package.features - I wonder whether it's been discussed already.

Not without causing a bunch of pain in figuring out the inheriting
order of stack USE variables.

>
> --
> regards
> MM
>

Re: [gentoo-dev] Re: debug/release builds extensions/clarification proposal

[Maciej Mrozowski]

[-- Attachment #1: Type: text/plain, Size: 7911 bytes --]

On Tuesday 02 of December 2008 10:40:19 Alec Warner wrote:

> <mean hat>
> You asked, so the counter proposal is to *do nothing*.
> <very mean generic rant hat on>
> Ideas (even good ones) don't always get implemented.
>
> Sometimes that just isn't the direction the maintainers want to take
> the project.
> Sometimes it is harder to implement than most people realize.
> Sometimes suggested implementations are just a hack and a bad idea all
> around.
>
> I think starting with an implementation may have been a bad starting move.
>
> Start with what you want to accomplish:
>   - Get feedback on whether this is useful or not.
>   - Get feedback on other features that may be available.
>   - Get feedback on how some folks would accomplish this.
>
> "I want to be able to turn debug builds on or off on a per-package
> basis.  Debug builds entail both debugging symbols, split-debug, debug
> CFLAGS and debug LDFLAGS."
>
> Is that a fair summary of your request?

Yes, precisely. But forget about this proposal, as I stated already it's just 
a workaround for inability to set CFLAGS/LDFLAGS and those two FEATURES per-
package basis in *official* way.

> I am unsure how much you actually care about how each package manager
> implements this feature (or if anyone implements it but portage, or
> paludis, or whatever the majority of the KDE users are using).
>
> I'm also unsure how useful this is when say, some part of KDE links
> against libfoo and KDE is built with debug symbols but libfoo is not.
> Is that really useful?  Are users actually asking for this proposed
> feature or do you just think they want it?  Do you have any data to
> back up why someone should implement this feature (mailing list posts,
> forums threads, etc..)

No, and I'm afraid I cannot provide any single evidence that users actually 
need features like:
- per package cflags/ldflags/features
- per category use flags, accept_keywords, cflags
- or tag clouds instead of hard coded categories
- user-defined packages sets (official)
- multiple portage configurations support to ease building binaries for 
several targets on a same host
- dynamic libraries tracking for safe package upgrade or removal
- real backwards dependencies
- maybe git driven Portage
- automatic kernel modules rebuilding
- mysql split ebuilds

Actually, I'm perfectly certain that users are way more interested in critical 
important aspects of their system like whether HOMEPAGE should be set in 
ebuilds or in metadata.xml :D

Please let me solve your little problem with HOMEPAGE for you...
Package's homepage obviously may be, and actually is - ${PN}-${PV} specific.
That being said it *would* needs to be specified either in every ebuild or as 
someone proposed - in metadata.xml in versioned/tagged way.
And no matter how many searches you run - it may be easy to predict that due 
to lazyness (less probable) or just to avoid copy/paste (copy/paste is bad - 
everyone knows that) - some developers used to put HOMEPAGE in eclasses - 
because it may be used to put in postinst message for some reasons, that being 
said it needs to be in ebuild domain in current implementation.
Mixing XML and bash (ebuild) in general isn't a bad idea but using bothe of 
them seems to be inconsistent - but some trade off needs to be paid sometimes.
When duplicating HOMEPAGE is such a pain for developer (as he needs to type it 
all over again, I agree, it is pain, especially when one need to put some 
things only to please repoman), why not invest some time and develop tools 
that could make it easier - like meta-ebuilds (or ebuild generators) and 
ebuild templates? I've done something like this to autogenerate plasma applet 
live ebuilds from KDE playground on their SVN (it's not yet commited to 
overlay as eclass is not yet ready to fetch/unpack and build packages from 
this location and I haven't got time yet to patch it).
If declaring HOMEPAGE in eclasses troubles you as you need BASH to process it 
properly (it may be pain for non-BASH search tools) and XML may be problematic 
to parse for bash tools - why not create such ebuild generator or 'compiler' - 
that could generate ebuild? Or for example as complete BASH script (no need 
for inherit anything) - and use eclasses ONLY like 'development library'.
This way - every ebuild could be:
- eclass-breakage free (overwriting eclasses don't take place so you are 
certain that user's emerge-problem is not him messing with eclasses - like 
mixing those from other overlays)
- every defined variable is there (no need for 'inherit' lookup) - so that one 
can easily find HOMEPAGE= using every kind of tool (unless it is enclosed with 
some condition - why would anyone need to do that btw?)
- much larger disk space requirements for Portage tree - but that could be 
compensated by for example gzipping every ebuild.
Of  course every problems with dichotomy ebuild vs metadata could be solved by 
some new Portage backend - better suited for queries and storage (maybe some 
relational database).
But so far - the solution in very simple - require HOMEPAGE to be in *every* 
ebuild and make repoman very angry otherwise and don't play with versioned 
metadata because it's waste of your time imho, and it could be invested on 
developing/deploying better backend for example or at least discussing on some 
not needed and never requested features like on the list above (and if 
requested, then probably by some clueless users...).
Glad that I could help.

> Certainly for portage per-package features are possible with a minor
> patch (to read the custom settings from your config and to inject the
> FEATURES variables into the per-package config when necessary).  The
> problem that has been stated in the past is that FEATURES were not
> designed to be used in that manner (per-package).
>
> We could design an separate system that let you define per-package
> 'things' and use these 'things' to trigger debug builds (completely
> outside of FEATURES, leaving them alone).  FEATURES were in fact
> specific features of portage that you want 'on' or 'off'
> (metadata-transfer, parallel-fetch, userfetch, unmerge-orphans,
> etc...)  These are examples of things you would not turn off
> per-ebuild.

> But the question is always 'is it really worth it' and can you get
> someone to do it.
> Sometimes, doing nothing is better than doing something badly.
> <endrant>

Yes, and developers in terms of Portage enhancements seem to be pretty good at 
this. Please don't get me wrong, I admire and respect *every* *singe* *effort* 
put in maintaining packages (maybe because I doing it as well), "wasting time" 
by bug fixing, and making it all alive. I really do.
And I understand that developers are free to spend their own precious free 
time (no sarcasm here) on whatever they want (as it's your time as you 
volunteer) and I would be perfectly fine with that if they wouldn't block the 
progress by miscalculated priorities and only because they have @gentoo.org in 
their email and they are the only ones authorized to introduce changes.

And as as side note - I anyone thinks that Gentoo is his private project and 
users are completely clueless (well, sometimes they are) and they are 
completely irrelevant, I guess the one should acknowledge the existence of 
some materials[1] (and maybe reconsider his further participation otherwise). 
There is some interesting discussion[2] on the forums related to user 
contribution to Gentoo and apart from many valid points by other users, there 
is my proposal[3] that aims to make your life a bit easier.

1. http://www.gentoo.org/foundation/en/#doc_chap2
2. http://forums.gentoo.org/viewtopic-t-702248.html
3. http://forums.gentoo.org/viewtopic-p-5296515.html#5296515

-- 
regards
MM

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Re: debug/release builds extensions/clarification proposal

[Marius Mauch]

On Wed, 3 Dec 2008 08:19:18 +0100
Maciej Mrozowski <reavertm@poczta.fm> wrote:

> No, and I'm afraid I cannot provide any single evidence that users
> actually need features like:
> - per package cflags/ldflags/features
> - per category use flags, accept_keywords, cflags
> - or tag clouds instead of hard coded categories
> - user-defined packages sets (official)
> - multiple portage configurations support to ease building binaries
> for several targets on a same host
> - dynamic libraries tracking for safe package upgrade or removal
> - real backwards dependencies
> - maybe git driven Portage
> - automatic kernel modules rebuilding
> - mysql split ebuilds

Assuming that's a list of feature requests, you know that half of them
are already available, right? (not counting the non-feature in there)

Marius

Re: [gentoo-dev] Re: debug/release builds extensions/clarification proposal

[Alec Warner]

On Tue, Dec 2, 2008 at 11:19 PM, Maciej Mrozowski <reavertm@poczta.fm> wrote:
> On Tuesday 02 of December 2008 10:40:19 Alec Warner wrote:
>
>> <mean hat>
>> You asked, so the counter proposal is to *do nothing*.
>> <very mean generic rant hat on>
>> Ideas (even good ones) don't always get implemented.
>>
>> Sometimes that just isn't the direction the maintainers want to take
>> the project.
>> Sometimes it is harder to implement than most people realize.
>> Sometimes suggested implementations are just a hack and a bad idea all
>> around.
>>
>> I think starting with an implementation may have been a bad starting move.
>>
>> Start with what you want to accomplish:
>>   - Get feedback on whether this is useful or not.
>>   - Get feedback on other features that may be available.
>>   - Get feedback on how some folks would accomplish this.
>>
>> "I want to be able to turn debug builds on or off on a per-package
>> basis.  Debug builds entail both debugging symbols, split-debug, debug
>> CFLAGS and debug LDFLAGS."
>>
>> Is that a fair summary of your request?
>
> Yes, precisely. But forget about this proposal, as I stated already it's just
> a workaround for inability to set CFLAGS/LDFLAGS and those two FEATURES per-
> package basis in *official* way.
>
>> I am unsure how much you actually care about how each package manager
>> implements this feature (or if anyone implements it but portage, or
>> paludis, or whatever the majority of the KDE users are using).
>>
>> I'm also unsure how useful this is when say, some part of KDE links
>> against libfoo and KDE is built with debug symbols but libfoo is not.
>> Is that really useful?  Are users actually asking for this proposed
>> feature or do you just think they want it?  Do you have any data to
>> back up why someone should implement this feature (mailing list posts,
>> forums threads, etc..)
>
> No, and I'm afraid I cannot provide any single evidence that users actually
> need features like:
> - per package cflags/ldflags/features
> - per category use flags, accept_keywords, cflags
> - or tag clouds instead of hard coded categories
> - user-defined packages sets (official)
> - multiple portage configurations support to ease building binaries for
> several targets on a same host
> - dynamic libraries tracking for safe package upgrade or removal
> - real backwards dependencies
> - maybe git driven Portage
> - automatic kernel modules rebuilding
> - mysql split ebuilds
>
> Actually, I'm perfectly certain that users are way more interested in critical
> important aspects of their system like whether HOMEPAGE should be set in
> ebuilds or in metadata.xml :D

Sorry, I can't prioritize email threads ;)

>
> Please let me solve your little problem with HOMEPAGE for you...
> Package's homepage obviously may be, and actually is - ${PN}-${PV} specific.
> That being said it *would* needs to be specified either in every ebuild or as
> someone proposed - in metadata.xml in versioned/tagged way.
> And no matter how many searches you run - it may be easy to predict that due
> to lazyness (less probable) or just to avoid copy/paste (copy/paste is bad -
> everyone knows that) - some developers used to put HOMEPAGE in eclasses -
> because it may be used to put in postinst message for some reasons, that being
> said it needs to be in ebuild domain in current implementation.
> Mixing XML and bash (ebuild) in general isn't a bad idea but using bothe of
> them seems to be inconsistent - but some trade off needs to be paid sometimes.
> When duplicating HOMEPAGE is such a pain for developer (as he needs to type it
> all over again, I agree, it is pain, especially when one need to put some
> things only to please repoman), why not invest some time and develop tools
> that could make it easier - like meta-ebuilds (or ebuild generators) and
> ebuild templates? I've done something like this to autogenerate plasma applet
> live ebuilds from KDE playground on their SVN (it's not yet commited to
> overlay as eclass is not yet ready to fetch/unpack and build packages from
> this location and I haven't got time yet to patch it).
> If declaring HOMEPAGE in eclasses troubles you as you need BASH to process it
> properly (it may be pain for non-BASH search tools) and XML may be problematic
> to parse for bash tools - why not create such ebuild generator or 'compiler' -
> that could generate ebuild? Or for example as complete BASH script (no need
> for inherit anything) - and use eclasses ONLY like 'development library'.
> This way - every ebuild could be:
> - eclass-breakage free (overwriting eclasses don't take place so you are
> certain that user's emerge-problem is not him messing with eclasses - like
> mixing those from other overlays)
> - every defined variable is there (no need for 'inherit' lookup) - so that one
> can easily find HOMEPAGE= using every kind of tool (unless it is enclosed with
> some condition - why would anyone need to do that btw?)
> - much larger disk space requirements for Portage tree - but that could be
> compensated by for example gzipping every ebuild.
> Of  course every problems with dichotomy ebuild vs metadata could be solved by
> some new Portage backend - better suited for queries and storage (maybe some
> relational database).
> But so far - the solution in very simple - require HOMEPAGE to be in *every*
> ebuild and make repoman very angry otherwise and don't play with versioned
> metadata because it's waste of your time imho, and it could be invested on
> developing/deploying better backend for example or at least discussing on some
> not needed and never requested features like on the list above (and if
> requested, then probably by some clueless users...).
> Glad that I could help.
>

Sweet, fixed!

>> Certainly for portage per-package features are possible with a minor
>> patch (to read the custom settings from your config and to inject the
>> FEATURES variables into the per-package config when necessary).  The
>> problem that has been stated in the past is that FEATURES were not
>> designed to be used in that manner (per-package).
>>
>> We could design an separate system that let you define per-package
>> 'things' and use these 'things' to trigger debug builds (completely
>> outside of FEATURES, leaving them alone).  FEATURES were in fact
>> specific features of portage that you want 'on' or 'off'
>> (metadata-transfer, parallel-fetch, userfetch, unmerge-orphans,
>> etc...)  These are examples of things you would not turn off
>> per-ebuild.
>
>> But the question is always 'is it really worth it' and can you get
>> someone to do it.
>> Sometimes, doing nothing is better than doing something badly.
>> <endrant>
>
> Yes, and developers in terms of Portage enhancements seem to be pretty good at
> this. Please don't get me wrong, I admire and respect *every* *singe* *effort*
> put in maintaining packages (maybe because I doing it as well), "wasting time"
> by bug fixing, and making it all alive. I really do.
> And I understand that developers are free to spend their own precious free
> time (no sarcasm here) on whatever they want (as it's your time as you
> volunteer) and I would be perfectly fine with that if they wouldn't block the
> progress by miscalculated priorities and only because they have @gentoo.org in
> their email and they are the only ones authorized to introduce changes.
>

One, I'm not a portage developer, so what I say regarding what goes in
actually doesn't matter, since I can't commit to the repo same as you
;)

Two, If there was some good data around what was the 'correct'
priority for items I would be all for it; but we don't have any good
means to collect it[1].

Three:
Portage is open source; so if a user feels that the developers are
unreliable or mis-prioritize requests they can just implement the
features they want themselves (no "@gentoo.org authorization"
required).  If people find the modification useful the author can even
contact the portage team to have their patch included in the official
version.  Sadly past experience has shown that very few authors take
this approach in practice; even though to me it is a pillar of open
source awesomeness.

> And as as side note - I anyone thinks that Gentoo is his private project and
> users are completely clueless (well, sometimes they are) and they are
> completely irrelevant, I guess the one should acknowledge the existence of
> some materials[1] (and maybe reconsider his further participation otherwise).
> There is some interesting discussion[2] on the forums related to user
> contribution to Gentoo and apart from many valid points by other users, there
> is my proposal[3] that aims to make your life a bit easier.

I don't think users are clueless. That doesn't mean I will implement
anything they ask of me though; each program I write has specific
goals associated with it (how it is used and what for) and if the
ideas presented are against those goals and the user cannot convince
me that it makes sense for the program to have that functionality then
I will not add that functionality.  I may be willing to write a patch
and give it the user for their own use (and they are free to
distribute said patch themselves.)

-Alec

[1] I guess we could crawl the forums for some sort of manual
prioritization, but I'm really looking for like, quarterly user
surveys, that help give us (the developers) an idea about what users
who take the survey say they want from us.

>
> 1. http://www.gentoo.org/foundation/en/#doc_chap2
> 2. http://forums.gentoo.org/viewtopic-t-702248.html
> 3. http://forums.gentoo.org/viewtopic-p-5296515.html#5296515
>
> --
> regards
> MM
>

[gentoo-dev] Re: Re: debug/release builds extensions/clarification proposal

[Steve Long]

Maciej Mrozowski wrote:

> On Monday 01 of December 2008 09:36:12 Diego 'Flameeyes' Pettenò wrote:
>> > - USE=debug is useless  when CFLAGS/LDFLAGS or FEATURES are not
>> > appropriate
>> What are you saying here? I'm afraid you're mistaken here.
> 
> The point is to look at this from users' (well, a bit) point of view -

Dunno who else would be using the software, so agreed ;)

> USE=debug variable is ambiguous in it's meaning. While it enables only
> codepaths (asserts, #ifdefs and similar) it suggests (by name and for some
> packages not only suggests) enabling debug symbols.

It'd be much saner if it did exactly what you're suggesting imo.

> And policy is to enforce CFLAGS from make.conf and wipe out every package-
> defined flags as far as I know.
> 
>> For the most part, USE=debug means "enable debug code paths", which for
>> lots of projects simply means "enable assertions"; there are packages
>> that take this as "enable debug symbols too" but I don't think that's
>> very valid since users might want debug code paths but not symbols and
>> vice-versa (I indeed have debug symbols bug no debug codepaths enabled).

"Let's address the common use case first" is often used as an excuse
for "let's not deal with any other use case." Why should your use-case
override the vast majority of users? It's not like you don't know how to
configure things exactly how you want or anything, so what's the issue?

If you're saying you want your extra feature (in the software engineering
sense) I agree it's perfectly valid; it's just lower priority, and I'm sure
you can do the work on that bit in a flash. (So far all i've heard is
adding -DDEBUG which is hardly ground-shaking.)

> That's correct, the problem is - Gentoo does not provide officially
> supported mechanism of enabling both or just debug symbols per package
> basis - it doesn't even provide any supported/documented mechanism for per
> package CFLAGS, FEATURES and similar.
Indeed; ain't it pathetic? 3 years arguing about PMS and still can't manage
the basics. (Answers to this point on -project please.)

> If /etc/portage/env hack/feature could be made official
It's the only sane solution; even if you advocate changing the name for
political reasons. I'd recommend looking at[1] and the previously-linked
post for a nice way to do both libs and apps (in the common case.)

> Yet, I still cannot think of this proposal other way like of dirty
> workaround for the problem, that doesn't really exist (well, at least for
> developers, who
> have meta-distribution and ultimate freedom for user in mind).  For the
> users the problem is real, of course it's usually a consequence of either
> not being aware of those mechanisms or as a result of ambiguous semantics
> of USE=debug.
Ahh but devs _are_ users (except when they're not.)

> And what about pushing some bash-domain FEATURES to USE flags? Like
> nostrip, splitdebug?
Good idea.

> I guess being able to set it per package is important.
You only get real choice if you have a commit bit ime. Then you're allowed
to ask all the inane luser questions you like too. No, I don't get it
either (it's hard to distinguish the respect from the flames sometimes, is
about as much mitigation as I can dredge up;) but it's off-topic, as was a
large part of this thread ;> Use gentoo-project for the non-technical
aspects (yes, _you_ have to separate your posts first: you don't have that
bit, remember?)

[1] http://forums.gentoo.org/viewtopic-p-5192821.html#5192821