Re: [gentoo-dev] RFC: New build types

Re: [gentoo-dev] RFC: New build types

[Rémi Cardona]

What would be the point of such a change? What problem are you trying to 
solve or to improve?

You'll need to answer those questions anyway should you ever need to 
write a GLEP for that.

Cheers

-- 
Rémi Cardona
LRI, INRIA
remi.cardona@lri.fr
remi@gentoo.org
-- 
gentoo-dev@lists.gentoo.org mailing list

[gentoo-dev] RFC: New build types

[Steve Long]

Something that's been discussed on IRC is the idea of a .pbuild file,
written in Python. I can also think of .cbuild (C) .Cbuild (C++) .sbuild
(Scheme) .hbuild (Haskell) and .jbuild (guess;) as being of immediate use,
(although I accept I might be the only one interested in the first ;)

The basic idea would be to replace ebuild.sh with an API equivalent (where
API is defined as stable portage or an approved PMS which adequately covers
this aspect.) How, or indeed whether or not, helper utilities equivalent to
eclasses (or an elib) would be provided, would of course be down to whoever
wrote and maintained the relevant code.

How do others feel about such an addition?


-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] RFC: New build types

[Jakub Moc]

Steve Long napsal(a):
> How do others feel about such an addition?


http://archives.gentoo.org/gentoo-dev/msg_a450013b143e01a25810f66ff345c9d4.xml

-- 
Best regards,

  Jakub Moc
  mailto:jakub@gentoo.org
  GPG signature:
  http://subkeys.pgp.net:11371/pks/lookup?op=get&search=0xCEBA3D9E
  Primary key fingerprint: D2D7 933C 9BA1 C95B 2C95  B30F 8717 D5FD CEBA 
3D9E

  ... still no signature   ;)
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] RFC: New build types

[Luca Barbato]

Steve Long wrote:
> Something that's been discussed on IRC is the idea of a .pbuild file,
> written in Python. I can also think of .cbuild (C) .Cbuild (C++) .sbuild
> (Scheme) .hbuild (Haskell) and .jbuild (guess;) as being of immediate use,
> (although I accept I might be the only one interested in the first ;)

I do not see any improvement per se.

> How do others feel about such an addition?

I think it's pointless.

lu

-- 

Luca Barbato
Gentoo Council Member
Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

-- 
gentoo-dev@lists.gentoo.org mailing list

[gentoo-dev] Re: RFC: New build types

[Steve Long]

Rémi Cardona wrote:

> What would be the point of such a change? What problem are you trying to
> solve or to improve?
>
First and foremost to give an environment wherein people can write their
installation scripts using the language they are most comfortable with.

Secondly efficiency; in the case of a pbuild it could be run from within the
PM; for something like a jbuild it would use the native tools and existing
libraries like ANT. For hbuild it would tie into Cabal. While these may be
used already, we go from PM -> BASH -> LangX. I'm just saying give the
_option_ to leave out the BASH bit when you have mature tools in langX.

> You'll need to answer those questions anyway should you ever need to
> write a GLEP for that.
> 
Yeah, that's a long way off; no point doing a GLEP without a working
implementation to show what you mean, imo. No point dedicating coder
resource to implement if it'd never get used in any case.

-- 
gentoo-dev@lists.gentoo.org mailing list

[gentoo-dev] Re: RFC: New build types

[Steve Long]

Luca Barbato wrote:

> Steve Long wrote:
>> Something that's been discussed on IRC is the idea of a .pbuild file,
>> written in Python. I can also think of .cbuild (C) .Cbuild (C++) .sbuild
>> (Scheme) .hbuild (Haskell) and .jbuild (guess;) as being of immediate
>> use, (although I accept I might be the only one interested in the first
>> ;)
> 
> I do not see any improvement per se.
> 
Well I agree C and C++ aren't very useful, since they are more than
adequately covered by make et al. With the others, there are setup tools
like distutils in the language already.

>> How do others feel about such an addition?
> 
> I think it's pointless.
> 
Fair enough. It's intended to make it easier to write install scripts.


-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] Re: RFC: New build types

[Rémi Cardona]

Steve Long a écrit :
> First and foremost to give an environment wherein people can write their
> installation scripts using the language they are most comfortable with.

If bash is not "easy" or straightforward enough for what you are trying 
to achieve, then I'd say the package is broken (ie, hand-made configure 
script, odd makefiles and whatnot). Better fix the package rather than 
rewriting ebuilds, make the world a better place.

> Secondly efficiency; in the case of a pbuild it could be run from within the
> PM; for something like a jbuild it would use the native tools and existing
> libraries like ANT. For hbuild it would tie into Cabal. While these may be
> used already, we go from PM -> BASH -> LangX. I'm just saying give the
> _option_ to leave out the BASH bit when you have mature tools in langX.

Care to back that up with any sort of figure or number? Is bash really 
the bottleneck? For 90% of the tree's ebuilds, I would _gcc_ is the 
bottleneck. Then I'd bet a big lump on libtool. Not portage, not bash.

But then again, I don't have any numbers to back that up either...

Honestly, maybe it could be a fun project, but I'm hardly convinced it 
would bring any sort of real advantage to the tree. In fact, having 
ebuilds in many languages would probably wreak havoc more than anything 
else.

My 2¢

Cheers,

Rémi
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] Re: RFC: New build types

[Marius Mauch]

[-- Attachment #1: Type: text/plain, Size: 1250 bytes --]

On Thu, 20 Mar 2008 03:59:01 +0000
Steve Long <slong@rathaus.eclipse.co.uk> wrote:

> Rémi Cardona wrote:
> 
> > What would be the point of such a change? What problem are you
> > trying to solve or to improve?
> >
> Secondly efficiency; in the case of a pbuild it could be run from
> within the PM; for something like a jbuild it would use the native
> tools and existing libraries like ANT. For hbuild it would tie into
> Cabal. While these may be used already, we go from PM -> BASH ->
> LangX. I'm just saying give the _option_ to leave out the BASH bit
> when you have mature tools in langX.

Generally those build tools are designed to be invoked from the shell,
not from within their implementation language. Of course you can invoke
use a distutils setup.py script from within a python interpreter, but
that's not how it's designed to be used and will probably end up being
more complicated to handle than invoking it from the shell.

Maybe it's 1% or 2% faster, but that's not worth the additional
complexity.

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[gentoo-dev] Re: Re: RFC: New build types

[Steve Long]

Rémi Cardona wrote:

> Steve Long a écrit :
>> First and foremost to give an environment wherein people can write their
>> installation scripts using the language they are most comfortable with.
> 
> If bash is not "easy" or straightforward enough for what you are trying
> to achieve, then I'd say the package is broken (ie, hand-made configure
> script, odd makefiles and whatnot). Better fix the package rather than
> rewriting ebuilds, make the world a better place.
>
Heh, I'm fine with BASH believe it or not ;p nor do I have that much
interest in the other scripting languages. I really just think it would
make porting stuff to Gentoo a lot simpler for people who don't know Cbut
do know their language of choice.
 
>> Secondly efficiency; in the case of a pbuild it could be run from within
>> the PM; for something like a jbuild it would use the native tools and
>> existing libraries like ANT. For hbuild it would tie into Cabal. While
>> these may be used already, we go from PM -> BASH -> LangX. I'm just
>> saying give the _option_ to leave out the BASH bit when you have mature
>> tools in langX.
> 
> Care to back that up with any sort of figure or number? Is bash really
> the bottleneck? For 90% of the tree's ebuilds, I would _gcc_ is the
> bottleneck. Then I'd bet a big lump on libtool. Not portage, not bash.
> 
> But then again, I don't have any numbers to back that up either...
>
I don't have figures, but my understanding is that one of the major factors
in pkgcore's speed (which *is* impressive, even if the UI isn't quite there
yet) is that it doesn't reload bash for every phase. (The whole
ebuild "daemon" or ebd thing.)

> Honestly, maybe it could be a fun project, but I'm hardly convinced it
> would bring any sort of real advantage to the tree. In fact, having
> ebuilds in many languages would probably wreak havoc more than anything
> else.
> 
I don't see how it would wreak more havoc than a novice using, eg ANT from
Java which s/he is comfortable with, and then further having to learn BASH
peculiarities when things don't fit with the eclass. But yeah, the fun is
what attracts me to the idea more than anything.

It's something I'd imagine would be used only for packages developed in the
relevant overlay, since that's where the people who know the language
develop stuff (and they'd be the ones maintaining their version.) However,
they'd need to know that, once they've signed off on it, the central tree
will support it without further code changes.


-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] Re: Re: RFC: New build types

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 2809 bytes --]

On Thu, Mar 20, 2008 at 06:51:13AM +0000, Steve Long wrote:
> Rémi Cardona wrote:
> 
> > Steve Long a écrit :
> >> First and foremost to give an environment wherein people can write their
> >> installation scripts using the language they are most comfortable with.
> > 
> > If bash is not "easy" or straightforward enough for what you are trying
> > to achieve, then I'd say the package is broken (ie, hand-made configure
> > script, odd makefiles and whatnot). Better fix the package rather than
> > rewriting ebuilds, make the world a better place.
> >
> Heh, I'm fine with BASH believe it or not ;p nor do I have that much
> interest in the other scripting languages. I really just think it would
> make porting stuff to Gentoo a lot simpler for people who don't know Cbut
> do know their language of choice.
>  
> >> Secondly efficiency; in the case of a pbuild it could be run from within
> >> the PM; for something like a jbuild it would use the native tools and
> >> existing libraries like ANT. For hbuild it would tie into Cabal. While
> >> these may be used already, we go from PM -> BASH -> LangX. I'm just
> >> saying give the _option_ to leave out the BASH bit when you have mature
> >> tools in langX.
> > 
> > Care to back that up with any sort of figure or number? Is bash really
> > the bottleneck? For 90% of the tree's ebuilds, I would _gcc_ is the
> > bottleneck. Then I'd bet a big lump on libtool. Not portage, not bash.
> > 
> > But then again, I don't have any numbers to back that up either...
> >
> I don't have figures, but my understanding is that one of the major factors
> in pkgcore's speed (which *is* impressive, even if the UI isn't quite there
> yet) is that it doesn't reload bash for every phase. (The whole
> ebuild "daemon" or ebd thing.)

From a speed standpoint, EBD is only relevant if we're talking about 
metadata regeneration-
http://gentooexperimental.org/~ferringb/blog/archives/2005-03.html#e2005-03-05T16_59_39.txt

Generally speaking, if you're sourcing to get metadata (regardless of 
the underlying format), you're already screwed- cache exists for a 
reason and is massively faster to rely on.  Pkgcore's speed comes 
about from careful design + a massive amount of JIT, EBD is faster 
then the alternatives but that's *only* relevant for metadata 
regeneration.

Finally, bear in mind we're talking about build phase here- even if 
the pkg is just a straight unpack/copy, the bottleneck there isn't 
going to be the bash bits for setting up the env, it's going to be the 
unpack, copy, multiple QA checks that do repeated find's across ${D}, 
multiple file invocations for same file, etc.  Seriously- profile a 
merge sometime, even on non-compilations the large time slices are 
never bash.
~brian

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Re: Re: RFC: New build types

[Petteri Räty]

[-- Attachment #1: Type: text/plain, Size: 757 bytes --]

Steve Long kirjoitti:
>>
> I don't see how it would wreak more havoc than a novice using, eg ANT from
> Java which s/he is comfortable with, and then further having to learn BASH
> peculiarities when things don't fit with the eclass. But yeah, the fun is
> what attracts me to the idea more than anything.
> 

Java needs to be compiled and ant is meant to be started from the 
command line. Of course you can invoke the main method from Java but 
what's the point? Developers have to be able to review ebuilds and 
having all those different languages would make the job harder and I 
don't really see benefits. If you need something bit more complex done 
in an ebuild, you can always use something like inline python.

Regards,
Petteri


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

[gentoo-dev] Re: Re: Re: RFC: New build types

[Steve Long]

Petteri Räty wrote:

> Steve Long kirjoitti:
>>>
>> I don't see how it would wreak more havoc than a novice using, eg ANT
>> from Java which s/he is comfortable with, and then further having to
>> learn BASH peculiarities when things don't fit with the eclass. But yeah,
>> the fun is what attracts me to the idea more than anything.
>> 
> 
> Java needs to be compiled and ant is meant to be started from the
> command line. Of course you can invoke the main method from Java but
> what's the point? Developers have to be able to review ebuilds and
> having all those different languages would make the job harder and I
> don't really see benefits. If you need something bit more complex done
> in an ebuild, you can always use something like inline python.
> 
Yeah, sorry I haven't used Java seriously since 1.1 (apart from some MIDP
stuff) so haven't used ANT. I'm thinking more in terms of how Java was
touted as network code, similar to tcl (which is one scripted setup I would
be interested in.) So where you have a VM already instantiated, along with
whatever SecurityManager and so on, you have a framework for user, shared
or system installs, according to privilege level, with dependency
resolution handled by the package manager. (The dependencies don't have to
be confined to what the language knows about.)

You're right though, that's not of so much interest for stuff where you
already have ebuilds with associated shell infra, which you're used to
maintaining.

Thanks,
igli.


-- 
gentoo-dev@lists.gentoo.org mailing list

[gentoo-dev] Re: Re: Re: RFC: New build types

[Steve Long]

Brian Harring wrote:

> On Thu, Mar 20, 2008 at 06:51:13AM +0000, Steve Long wrote:
>> I don't have figures, but my understanding is that one of the major
>> factors in pkgcore's speed (which *is* impressive, even if the UI isn't
>> quite there yet) is that it doesn't reload bash for every phase. (The
>> whole ebuild "daemon" or ebd thing.)
> 
> From a speed standpoint, EBD is only relevant if we're talking about
> metadata regeneration-
>
http://gentooexperimental.org/~ferringb/blog/archives/2005-03.html#e2005-03-05T16_59_39.txt
>
Ah OK; thanks, very interesting post.
 
> Generally speaking, if you're sourcing to get metadata (regardless of
> the underlying format), you're already screwed- cache exists for a
> reason and is massively faster to rely on.  Pkgcore's speed comes
> about from careful design + a massive amount of JIT, EBD is faster
> then the alternatives but that's *only* relevant for metadata
> regeneration.
>
Would the metadata regen be quicker if the relevant file were in python
rather than bash?

> Finally, bear in mind we're talking about build phase here- even if
> the pkg is just a straight unpack/copy, the bottleneck there isn't
> going to be the bash bits for setting up the env, it's going to be the
> unpack, copy, multiple QA checks that do repeated find's across ${D},
> multiple file invocations for same file, etc.  Seriously- profile a
> merge sometime, even on non-compilations the large time slices are
> never bash.

Understood; thanks for discussing.

I was under the impression that implicit in the design of portage/pkgcore,
was that build scripts wouldn't necessarily be in bash, and that ebuild was
simply the bash format. Other formats in scripting languages seemed a
no-brainer; sorry if it was off-base.


-- 
gentoo-dev@lists.gentoo.org mailing list