[gentoo-dev] The future of ebuild

[gentoo-dev] The future of ebuild

[Felipe Contreras]

Hi gentooists,

I've been reading news sites about some changes happening in Gentoo
and I thought it might be a good time to submit some ideas I've been
baking for several years.

I come from a Linux From Scratch background, I like the feeling of
knowing every single corner of my system and the fact that there isn't
anything that I don't want or need. However, typing every single
command by hand is far from ideal, so at first I started writing some
scripts and eventually I wrote a build system that suited my needs. I
did it in bash for several reasons.

After a while I realized bash wasn't exactly the best language to
write such thing. Mainly because:

a) The code ends up with a lot of stuff for handling strings properly
(like escaping sequences)
b) Error are difficult to handle since bash doesn't have exceptions
c) Persistent information is difficult to achieve (no database stuff)
d) Package information is difficult to fetch/store (no objects/struct)

A more featured language could allow for example: filtered output,
exception handling->state storage->resuming.

But the big deal is with the package definition, recently I learned
about Domain Specific Languages, and I think that is the best option.

A new dsl allows many interesting features in the package definition
itself like: inheritance, exceptions, arrays, hash tables, objects,
modules, documentation, information messages, etc.

Take this example:

package Binutils < Gnu
  definition
    @version = "2.17"
    @name = "binutils"
    super() # run the Gnu definition stuff

    @config_opts = "--disable-nls --with-sysroot=\"#{$sys_root}\"
--enable-shared --disable-multilib"
  end

  steps
    build
      cd #{$top_build_dir}
      mkdir -p #{@build_dir}
      cd #{@build_dir}
      :configure "script" => "../#{@dir}/configure", "opts" => @config_opts
      make configure-host
      make
    end

    install
      cd #{$top_build_dir}
      cd #{@build_dir}
      make install
    end
  end
end

This is based on an already working prototype made in Ruby, so it's
biased towards Ruby facilities.

I've tried different build systems: rpm, dpkg, autopackage.
Unfortunately I never tried ebuild because it was based on bash as far
as I could tell.

After almost a decade of using Linux I still haven't found a build
system that suits all my needs. AFAIK ebuild is the most advanced but
it's still relying on ancient technology (bash scripts) so there will
always be limitations despite the brilliant ideas.

The core of a distribution is the "packaging" system, and the core of
the packaging system is the building system, which has no reason not
to be distribution agnostic, and actually, packaging system agnostic.

Why not create a new build system with a state of the art programming
language, and an advanced DSL that actually other distributions could
use?

I would like to hear your opinions on this matter.

-- 
Felipe Contreras
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] The future of ebuild

[Duncan Coutts]

On Wed, 2008-02-20 at 22:40 +0200, Felipe Contreras wrote:

> The core of a distribution is the "packaging" system, and the core of
> the packaging system is the building system, which has no reason not
> to be distribution agnostic, and actually, packaging system agnostic.
> 
> Why not create a new build system with a state of the art programming
> language, and an advanced DSL that actually other distributions could
> use?
> 
> I would like to hear your opinions on this matter.

Take a look at Nix. It's a distribution-agnostic package manager that
uses a purely functional DSL for package specifications.
http://nix.cs.uu.nl/index.html

-- 
Duncan Coutts : Gentoo Developer (Haskell team)

-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] The future of ebuild

[Felipe Contreras]

On Thu, Feb 21, 2008 at 2:29 PM, Duncan Coutts <dcoutts@gentoo.org> wrote:
>
>  On Wed, 2008-02-20 at 22:40 +0200, Felipe Contreras wrote:
>
>  > The core of a distribution is the "packaging" system, and the core of
>  > the packaging system is the building system, which has no reason not
>  > to be distribution agnostic, and actually, packaging system agnostic.
>  >
>  > Why not create a new build system with a state of the art programming
>  > language, and an advanced DSL that actually other distributions could
>  > use?
>  >
>  > I would like to hear your opinions on this matter.
>
>  Take a look at Nix. It's a distribution-agnostic package manager that
>  uses a purely functional DSL for package specifications.
>  http://nix.cs.uu.nl/index.html

That's exactly what I'm taking about :) I'll try it out. Thanks for
sharing the link.

Is there any interest in the Gentoo community to migrate to Nix?

Best regards.

-- 
Felipe Contreras
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] The future of ebuild

[Santiago M. Mola]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=UTF-8, Size: 1036 bytes --]

On Sun, Feb 24, 2008 at 12:02 PM, Felipe Contreras
<felipe.contreras@gmail.com> wrote:
> On Thu, Feb 21, 2008 at 2:29 PM, Duncan Coutts <dcoutts@gentoo.org> wrote:
> >  >
> >  > Why not create a new build system with a state of the art programming
> >  > language, and an advanced DSL that actually other distributions could
> >  > use?
> >  >
> >  > I would like to hear your opinions on this matter.
> >
> >  Take a look at Nix. It's a distribution-agnostic package manager that
> >  uses a purely functional DSL for package specifications.
> >  http://nix.cs.uu.nl/index.html
>
> That's exactly what I'm taking about :) I'll try it out. Thanks for
> sharing the link.
>
> Is there any interest in the Gentoo community to migrate to Nix?
>

This is not going to happen. "Migrating to Nix" means "rewriting the
distro from scratch".  If you´re interested in a distro built on top
of Nix, you can try NixOS (which looks really nice IMO).

Regards,
Santiago

-- 
Santiago M. Mola
Jabber ID: cooldwind@gmail.com
éí¢‡^¾X¬¶È\x1ežÚ(¢¸&j)bž	b²

Re: [gentoo-dev] The future of ebuild

[Jakub Moc]

Santiago M. Mola napsal(a):
> This is not going to happen. "Migrating to Nix" means "rewriting the
> distro from scratch".  If you´re interested in a distro built on top
> of Nix, you can try NixOS (which looks really nice IMO).
> 
> Regards,
> Santiago

While we are switching the whole tree to Nix, I'd strongly advocate that:

1/ All ebuilds must be rewritten in whitespace [1]; that will have the 
great side-effect that all those pesky whitespace repoman checks will 
start to be meaningful finally.

2/ All eclasses must be rewritten in Piet [2] at the same time, so that 
we make dev's life a bit more colorful once the former step. is 
finished. There is already a platform-independent IDE ready for this 
purpose after all [3].

Since the benefits of the above are so obvious, I suppose this doesn't 
even require a GLEP and the porting work should therefore start 
immediately to make it in time for 2008.0.

Have a nice day. :P

[1] http://compsoc.dur.ac.uk/whitespace/
[2] http://en.wikipedia.org/wiki/Piet_(programming_language)
[3] http://www.rapapaing.com/piet/


-- 
Best regards,

  Jakub Moc
  mailto:jakub@gentoo.org
  GPG signature:
  http://subkeys.pgp.net:11371/pks/lookup?op=get&search=0xCEBA3D9E
  Primary key fingerprint: D2D7 933C 9BA1 C95B 2C95  B30F 8717 D5FD CEBA 
3D9E

  ... still no signature   ;)


-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] The future of ebuild

[Sebastian Volke]

[-- Attachment #1: Type: text/plain, Size: 1010 bytes --]

On Monday 25 February 2008 11:59:35 Jakub Moc wrote:
> While we are switching the whole tree to Nix, I'd strongly advocate that:
>
> 1/ All ebuilds must be rewritten in whitespace [1]; that will have the
> great side-effect that all those pesky whitespace repoman checks will
> start to be meaningful finally.
>
> 2/ All eclasses must be rewritten in Piet [2] at the same time, so that
> we make dev's life a bit more colorful once the former step. is
> finished. There is already a platform-independent IDE ready for this
> purpose after all [3].
>
> Since the benefits of the above are so obvious, I suppose this doesn't
> even require a GLEP and the porting work should therefore start
> immediately to make it in time for 2008.0.
>

Nice "mockery" :-)


-- 
*Sebastian Volke*
registered Linux user #426550

mail address: echo ozslxhmfw_gfxyn@lrc.sjy | perl -pe 'y/a-z/v-za-w/'

If your mail client doesn't support GPG signatures, please ignore this
little "signature.asc" file.

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] The future of ebuild

[Duncan Coutts]

On Sun, 2008-02-24 at 13:02 +0200, Felipe Contreras wrote:

> >  Take a look at Nix. It's a distribution-agnostic package manager that
> >  uses a purely functional DSL for package specifications.
> >  http://nix.cs.uu.nl/index.html
> 
> That's exactly what I'm taking about :) I'll try it out. Thanks for
> sharing the link.
> 
> Is there any interest in the Gentoo community to migrate to Nix?

It's quite a radical departure and requires more accurate information
about packages. We'll see how the NixOS people get on.

-- 
Duncan Coutts : Gentoo Developer (Haskell team)

-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] The future of ebuild

[Alec Warner]

On 2/20/08, Felipe Contreras <felipe.contreras@gmail.com> wrote:
> Hi gentooists,
>
>  I've been reading news sites about some changes happening in Gentoo
>  and I thought it might be a good time to submit some ideas I've been
>  baking for several years.
>
>  I come from a Linux From Scratch background, I like the feeling of
>  knowing every single corner of my system and the fact that there isn't
>  anything that I don't want or need. However, typing every single
>  command by hand is far from ideal, so at first I started writing some
>  scripts and eventually I wrote a build system that suited my needs. I
>  did it in bash for several reasons.
>
>  After a while I realized bash wasn't exactly the best language to
>  write such thing. Mainly because:
>
>  a) The code ends up with a lot of stuff for handling strings properly
>  (like escaping sequences)

Yes this is annoying

>  b) Error are difficult to handle since bash doesn't have exceptions

I disagree here: most errors are fatal anyway any non fatal errors can
be printed and saved via the elog facility.

>  c) Persistent information is difficult to achieve (no database stuff)

How is this a bash limitation?  Most languages don't have 'database
stuff' built in.
I don't see how saving stuff to files is much different than the env
dumping we do in bash?

>  d) Package information is difficult to fetch/store (no objects/struct)

Yes.

>
>  A more featured language could allow for example: filtered output,
>  exception handling->state storage->resuming.
>
>  But the big deal is with the package definition, recently I learned
>  about Domain Specific Languages, and I think that is the best option.
>
>  A new dsl allows many interesting features in the package definition
>  itself like: inheritance, exceptions, arrays, hash tables, objects,
>  modules, documentation, information messages, etc.

Note that most languages allow for the same things...so why would we
author our own language?  What runtime would it use?

>
>  Take this example:
>
>  package Binutils < Gnu
>   definition
>     @version = "2.17"
>     @name = "binutils"
>     super() # run the Gnu definition stuff
>
>     @config_opts = "--disable-nls --with-sysroot=\"#{$sys_root}\"
>  --enable-shared --disable-multilib"
>   end
>
>   steps
>     build
>       cd #{$top_build_dir}
>       mkdir -p #{@build_dir}
>       cd #{@build_dir}
>       :configure "script" => "../#{@dir}/configure", "opts" => @config_opts
>       make configure-host
>       make
>     end
>
>     install
>       cd #{$top_build_dir}
>       cd #{@build_dir}
>       make install
>     end
>   end
>  end
>
>  This is based on an already working prototype made in Ruby, so it's
>  biased towards Ruby facilities.

It looks Rakish ;)

>
>  I've tried different build systems: rpm, dpkg, autopackage.
>  Unfortunately I never tried ebuild because it was based on bash as far
>  as I could tell.

Typically a 'build system' would refer to 'autotools' or 'ant' or
'setuptools' not an ebuild.

>
>  After almost a decade of using Linux I still haven't found a build
>  system that suits all my needs. AFAIK ebuild is the most advanced but
>  it's still relying on ancient technology (bash scripts) so there will
>  always be limitations despite the brilliant ideas.

Bash is not 'ancient' and it works suprisingly well for the vast
majority of cases.  I realize using a high level language would
probably make some tasks easier (mmm dicts and real arrays).  There is
the matter of porting over 10000 ebuilds however ;)

>
>  The core of a distribution is the "packaging" system, and the core of
>  the packaging system is the building system, which has no reason not
>  to be distribution agnostic, and actually, packaging system agnostic.
>
>  Why not create a new build system with a state of the art programming
>  language, and an advanced DSL that actually other distributions could
>  use?

I would talk to Drobbins, he was talking about something like this to
manage complexity.

>
>  I would like to hear your opinions on this matter.
>
>  --
>  Felipe Contreras
>
> --
>  gentoo-dev@lists.gentoo.org mailing list
>
>
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] The future of ebuild

[Alec Warner]

On 2/23/08, Alec Warner <antarus@gentoo.org> wrote:
> On 2/20/08, Felipe Contreras <felipe.contreras@gmail.com> wrote:
>  > Hi gentooists,
>  >
>  >  I've been reading news sites about some changes happening in Gentoo
>  >  and I thought it might be a good time to submit some ideas I've been
>  >  baking for several years.
>  >
>  >  I come from a Linux From Scratch background, I like the feeling of
>  >  knowing every single corner of my system and the fact that there isn't
>  >  anything that I don't want or need. However, typing every single
>  >  command by hand is far from ideal, so at first I started writing some
>  >  scripts and eventually I wrote a build system that suited my needs. I
>  >  did it in bash for several reasons.
>  >
>  >  After a while I realized bash wasn't exactly the best language to
>  >  write such thing. Mainly because:
>  >
>  >  a) The code ends up with a lot of stuff for handling strings properly
>  >  (like escaping sequences)
>
>
> Yes this is annoying
>
>
>  >  b) Error are difficult to handle since bash doesn't have exceptions
>
>
> I disagree here: most errors are fatal anyway any non fatal errors can
>  be printed and saved via the elog facility.
>
>
>  >  c) Persistent information is difficult to achieve (no database stuff)
>
>
> How is this a bash limitation?  Most languages don't have 'database
>  stuff' built in.
>  I don't see how saving stuff to files is much different than the env
>  dumping we do in bash?

Er, I meant saving stuff to databases ;)

>
>
>  >  d) Package information is difficult to fetch/store (no objects/struct)
>
>
> Yes.
>
>
>  >
>  >  A more featured language could allow for example: filtered output,
>  >  exception handling->state storage->resuming.
>  >
>  >  But the big deal is with the package definition, recently I learned
>  >  about Domain Specific Languages, and I think that is the best option.
>  >
>  >  A new dsl allows many interesting features in the package definition
>  >  itself like: inheritance, exceptions, arrays, hash tables, objects,
>  >  modules, documentation, information messages, etc.
>
>
> Note that most languages allow for the same things...so why would we
>  author our own language?  What runtime would it use?
>
>
>  >
>  >  Take this example:
>  >
>  >  package Binutils < Gnu
>  >   definition
>  >     @version = "2.17"
>  >     @name = "binutils"
>  >     super() # run the Gnu definition stuff
>  >
>  >     @config_opts = "--disable-nls --with-sysroot=\"#{$sys_root}\"
>  >  --enable-shared --disable-multilib"
>  >   end
>  >
>  >   steps
>  >     build
>  >       cd #{$top_build_dir}
>  >       mkdir -p #{@build_dir}
>  >       cd #{@build_dir}
>  >       :configure "script" => "../#{@dir}/configure", "opts" => @config_opts
>  >       make configure-host
>  >       make
>  >     end
>  >
>  >     install
>  >       cd #{$top_build_dir}
>  >       cd #{@build_dir}
>  >       make install
>  >     end
>  >   end
>  >  end
>  >
>  >  This is based on an already working prototype made in Ruby, so it's
>  >  biased towards Ruby facilities.
>
>
> It looks Rakish ;)
>
>
>  >
>  >  I've tried different build systems: rpm, dpkg, autopackage.
>  >  Unfortunately I never tried ebuild because it was based on bash as far
>  >  as I could tell.
>
>
> Typically a 'build system' would refer to 'autotools' or 'ant' or
>  'setuptools' not an ebuild.
>
>
>  >
>  >  After almost a decade of using Linux I still haven't found a build
>  >  system that suits all my needs. AFAIK ebuild is the most advanced but
>  >  it's still relying on ancient technology (bash scripts) so there will
>  >  always be limitations despite the brilliant ideas.
>
>
> Bash is not 'ancient' and it works suprisingly well for the vast
>  majority of cases.  I realize using a high level language would
>  probably make some tasks easier (mmm dicts and real arrays).  There is
>  the matter of porting over 10000 ebuilds however ;)
>
>
>  >
>  >  The core of a distribution is the "packaging" system, and the core of
>  >  the packaging system is the building system, which has no reason not
>  >  to be distribution agnostic, and actually, packaging system agnostic.
>  >
>  >  Why not create a new build system with a state of the art programming
>  >  language, and an advanced DSL that actually other distributions could
>  >  use?
>
>
> I would talk to Drobbins, he was talking about something like this to
>  manage complexity.
>
>
>  >
>  >  I would like to hear your opinions on this matter.
>  >
>  >  --
>  >  Felipe Contreras
>  >
>  > --
>  >  gentoo-dev@lists.gentoo.org mailing list
>  >
>  >
>
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] The future of ebuild

[Felipe Contreras]

On Sat, Feb 23, 2008 at 10:45 PM, Alec Warner <antarus@gentoo.org> wrote:
> On 2/20/08, Felipe Contreras <felipe.contreras@gmail.com> wrote:
>  >  b) Error are difficult to handle since bash doesn't have exceptions
>
>  I disagree here: most errors are fatal anyway any non fatal errors can
>  be printed and saved via the elog facility.

Yes, for the most common usage that's true, but that think about this
example: I'm compiling gstreamer-plugins-good, which needs libraw1394,
but the compilation fails, perhaps that user is not interested in that
particular plugin so a dialog can pop up and the user can choose if to
continue without the libraw stuff or fail.

I'm sure that can be done without exceptions but as the complexity
increases properly checking/passing around error values/messages
becomes tedious.

>  >  c) Persistent information is difficult to achieve (no database stuff)
>
>  How is this a bash limitation?  Most languages don't have 'database
>  stuff' built in.
>  I don't see how saving stuff to files is much different than the env
>  dumping we do in bash?

I guess it's mostly the burden of serializing/parsing all that stuff.

>  >  A more featured language could allow for example: filtered output,
>  >  exception handling->state storage->resuming.
>  >
>  >  But the big deal is with the package definition, recently I learned
>  >  about Domain Specific Languages, and I think that is the best option.
>  >
>  >  A new dsl allows many interesting features in the package definition
>  >  itself like: inheritance, exceptions, arrays, hash tables, objects,
>  >  modules, documentation, information messages, etc.
>
>  Note that most languages allow for the same things...so why would we
>  author our own language?  What runtime would it use?

The language can be something very simple that has bash embedded. That
has the advantage that you can just copy paste what you are doing
already.

As more ideas appear the DSL can be updated, but still, the embedded
bash would always be there.

<snip/>
>  >  This is based on an already working prototype made in Ruby, so it's
>  >  biased towards Ruby facilities.
>
>  It looks Rakish ;)

Perhaps, the only interesting thing here is that those commands are
not defined internally. There's an auxiliary bash shell that is
running those commands. Not like "system" calls, which loose the
environment between calls, this is something that cooperates nicely.

>  >  I've tried different build systems: rpm, dpkg, autopackage.
>  >  Unfortunately I never tried ebuild because it was based on bash as far
>  >  as I could tell.
>
>  Typically a 'build system' would refer to 'autotools' or 'ant' or
>  'setuptools' not an ebuild.

Is there such a big difference?

I'm sure it's possible to by-pass autotools and write down all the
commands required to build something in an ebuild. Similarly it's
possible to use autotools to compile and install a bunch of packages.

Those "build systems" also need to be updated, but that's another story.

>  >  After almost a decade of using Linux I still haven't found a build
>  >  system that suits all my needs. AFAIK ebuild is the most advanced but
>  >  it's still relying on ancient technology (bash scripts) so there will
>  >  always be limitations despite the brilliant ideas.
>
>  Bash is not 'ancient' and it works suprisingly well for the vast
>  majority of cases.  I realize using a high level language would
>  probably make some tasks easier (mmm dicts and real arrays).  There is
>  the matter of porting over 10000 ebuilds however ;)

Yeah, bash is pretty good for many things, just doesn't scale that much.

At some point someone decided the current status was not good enough
and decided to create ebuild, even though he was well aware that
thousands of already existing instructions about how to build packages
would have to be re-written.

If it's easy to write people will possibly even write more of those.
Take for example ArchLinux which also has around 10,000 packages
simply because it's so easy to write them.

And with something that is distribution agnostic, different
communities can benefit from sharing the same rules.

>  >
>  >  The core of a distribution is the "packaging" system, and the core of
>  >  the packaging system is the building system, which has no reason not
>  >  to be distribution agnostic, and actually, packaging system agnostic.
>  >
>  >  Why not create a new build system with a state of the art programming
>  >  language, and an advanced DSL that actually other distributions could
>  >  use?
>
>  I would talk to Drobbins, he was talking about something like this to
>  manage complexity.

Will do.

Best regards.

-- 
Felipe Contreras
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] The future of ebuild

[Luis Francisco Araujo]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Felipe Contreras wrote:
| On Sat, Feb 23, 2008 at 10:45 PM, Alec Warner <antarus@gentoo.org> wrote:
|> On 2/20/08, Felipe Contreras <felipe.contreras@gmail.com> wrote:
|>  >  b) Error are difficult to handle since bash doesn't have exceptions
|>
|>  I disagree here: most errors are fatal anyway any non fatal errors can
|>  be printed and saved via the elog facility.
|
| Yes, for the most common usage that's true, but that think about this
| example: I'm compiling gstreamer-plugins-good, which needs libraw1394,
| but the compilation fails, perhaps that user is not interested in that
| particular plugin so a dialog can pop up and the user can choose if to
| continue without the libraw stuff or fail.
|
| I'm sure that can be done without exceptions but as the complexity
| increases properly checking/passing around error values/messages
| becomes tedious.
|

That's what USE flags are for.

- --

Luis F. Araujo "araujo at gentoo.org"
Gentoo Linux

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.7 (GNU/Linux)

iD8DBQFHwbyIBCmRZan6aegRArK/AJ9wBvqPZ9PErpxiVHgpkSLuCZIixQCdGiEB
wvdMli4taTHJFVBoHYIzyLs=
=/1k7
-----END PGP SIGNATURE-----
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] The future of ebuild

[Alec Warner]

On 2/24/08, Felipe Contreras <felipe.contreras@gmail.com> wrote:
> On Sat, Feb 23, 2008 at 10:45 PM, Alec Warner <antarus@gentoo.org> wrote:
>  > On 2/20/08, Felipe Contreras <felipe.contreras@gmail.com> wrote:
>
> >  >  b) Error are difficult to handle since bash doesn't have exceptions
>  >
>  >  I disagree here: most errors are fatal anyway any non fatal errors can
>  >  be printed and saved via the elog facility.
>
>
> Yes, for the most common usage that's true, but that think about this
>  example: I'm compiling gstreamer-plugins-good, which needs libraw1394,
>  but the compilation fails, perhaps that user is not interested in that
>  particular plugin so a dialog can pop up and the user can choose if to
>  continue without the libraw stuff or fail.
>
>  I'm sure that can be done without exceptions but as the complexity
>  increases properly checking/passing around error values/messages
>  becomes tedious.

This has nothing to do with the build system and everything to do with
the package manager actually processing information provided.

>
>
>  >  >  c) Persistent information is difficult to achieve (no database stuff)
>  >
>  >  How is this a bash limitation?  Most languages don't have 'database
>  >  stuff' built in.
>  >  I don't see how saving stuff to files is much different than the env
>  >  dumping we do in bash?
>
>
> I guess it's mostly the burden of serializing/parsing all that stuff.

But the code is written already...

>
>
>  >  >  A more featured language could allow for example: filtered output,
>  >  >  exception handling->state storage->resuming.
>  >  >
>  >  >  But the big deal is with the package definition, recently I learned
>  >  >  about Domain Specific Languages, and I think that is the best option.
>  >  >
>  >  >  A new dsl allows many interesting features in the package definition
>  >  >  itself like: inheritance, exceptions, arrays, hash tables, objects,
>  >  >  modules, documentation, information messages, etc.
>  >
>  >  Note that most languages allow for the same things...so why would we
>  >  author our own language?  What runtime would it use?
>
>
> The language can be something very simple that has bash embedded. That
>  has the advantage that you can just copy paste what you are doing
>  already.
>
>  As more ideas appear the DSL can be updated, but still, the embedded
>  bash would always be there.
>
>  <snip/>
>
> >  >  This is based on an already working prototype made in Ruby, so it's
>  >  >  biased towards Ruby facilities.
>  >
>  >  It looks Rakish ;)
>
>
> Perhaps, the only interesting thing here is that those commands are
>  not defined internally. There's an auxiliary bash shell that is
>  running those commands. Not like "system" calls, which loose the
>  environment between calls, this is something that cooperates nicely.
>
>
>  >  >  I've tried different build systems: rpm, dpkg, autopackage.
>  >  >  Unfortunately I never tried ebuild because it was based on bash as far
>  >  >  as I could tell.
>  >
>  >  Typically a 'build system' would refer to 'autotools' or 'ant' or
>  >  'setuptools' not an ebuild.
>
>
> Is there such a big difference?
>
>  I'm sure it's possible to by-pass autotools and write down all the
>  commands required to build something in an ebuild. Similarly it's
>  possible to use autotools to compile and install a bunch of packages.
>
>  Those "build systems" also need to be updated, but that's another story.

Updating for the sake of 'stuff being old' is not a compelling case.

>
>
>  >  >  After almost a decade of using Linux I still haven't found a build
>  >  >  system that suits all my needs. AFAIK ebuild is the most advanced but
>  >  >  it's still relying on ancient technology (bash scripts) so there will
>  >  >  always be limitations despite the brilliant ideas.
>  >
>  >  Bash is not 'ancient' and it works suprisingly well for the vast
>  >  majority of cases.  I realize using a high level language would
>  >  probably make some tasks easier (mmm dicts and real arrays).  There is
>  >  the matter of porting over 10000 ebuilds however ;)
>
>
> Yeah, bash is pretty good for many things, just doesn't scale that much.
>
>  At some point someone decided the current status was not good enough
>  and decided to create ebuild, even though he was well aware that
>  thousands of already existing instructions about how to build packages
>  would have to be re-written.
>
>  If it's easy to write people will possibly even write more of those.
>  Take for example ArchLinux which also has around 10,000 packages
>  simply because it's so easy to write them.
>
>  And with something that is distribution agnostic, different
>  communities can benefit from sharing the same rules.

What is 'distribution agnostic'?

Technically most things are; rpm is used by many distros and is
basically similar to what we have (phases, a meta language used to
move files around, bad dependencies).  So is a debian rules file.  How
are these 'distro specific'?

>
>
>  >  >
>  >  >  The core of a distribution is the "packaging" system, and the core of
>  >  >  the packaging system is the building system, which has no reason not
>  >  >  to be distribution agnostic, and actually, packaging system agnostic.
>  >  >
>  >  >  Why not create a new build system with a state of the art programming
>  >  >  language, and an advanced DSL that actually other distributions could
>  >  >  use?
>  >
>  >  I would talk to Drobbins, he was talking about something like this to
>  >  manage complexity.
>
>
> Will do.

So the biggest problem I see here is that this all seems like academic
handwaving.  You don't move from an established yet complex format
like ebuild to some other random format that no one has ever used
before.  Someone needs to come up with something, foster it, get 500
packages written that use it with a package manager that can actually
install them.. Spell out the advantages of why the new format kicks
ass and what it gives over the others (rpm, ebuild, deb, others...)
and then maybe people will move.

I would posit that most developers don't care because ebuild works for
them and there exists a small percentage of packages that could
benefit from a better system.

>
>  Best regards.
>
>
>  --
>
> Felipe Contreras
>  --
>  gentoo-dev@lists.gentoo.org mailing list
>
>
-- 
gentoo-dev@lists.gentoo.org mailing list

[gentoo-dev] Re: The future of ebuild

[Steve Long]

Felipe Contreras wrote:

> On Sat, Feb 23, 2008 at 10:45 PM, Alec Warner <antarus@gentoo.org> wrote:
>> On 2/20/08, Felipe Contreras <felipe.contreras@gmail.com> wrote:
>>  >  b) Error are difficult to handle since bash doesn't have exceptions
>>
>>  I disagree here: most errors are fatal anyway any non fatal errors can
>>  be printed and saved via the elog facility.
> 
> Yes, for the most common usage that's true, but that think about this
> example: I'm compiling gstreamer-plugins-good, which needs libraw1394,
> but the compilation fails, perhaps that user is not interested in that
> particular plugin so a dialog can pop up and the user can choose if to
> continue without the libraw stuff or fail.
>
That would typically be configured via USE flags, or for some things via a
USE expand, like CAMERAS for gphoto. I agree there is scope for UI work,
but I think that's better done by other apps calling the package manager,
in the same way as other distros do. For Gentoo, we use update from the
CLI, and himerge for a GUI, although portato (GTK) and kuroo (KDE) are both
nice too. Typically in such an instance a user would see the package fail,
and try with a different USE setting.

We're adding USE editing to update, should be done this week; for all the
GUIs that's already possible. In terms of what you're discussing, the
package manager has to be able to run without user interaction, but a next
generation build system is possible (eg pbuilds in python have been
discussed on IRC, and qmake et al are around); ebuild is more a meta-format
in that it can use a wide variety of base tools.

update: http://forums.gentoo.org/viewtopic-t-546828.html
http://www.haskell.org/himerge/
http://portato.origo.ethz.ch/
http://kuroo.org/
> I'm sure that can be done without exceptions but as the complexity
> increases properly checking/passing around error values/messages
> becomes tedious.
> 
>>  >  c) Persistent information is difficult to achieve (no database stuff)
>>
>>  How is this a bash limitation?  Most languages don't have 'database
>>  stuff' built in.
>>  I don't see how saving stuff to files is much different than the env
>>  dumping we do in bash?
> 
> I guess it's mostly the burden of serializing/parsing all that stuff.
>
Saving to flat files or databases gives the same persistence, and in any
case the ebuilds don't mess around with them, the package manager does.
It's perfectly possible to use a database to store the information ofc, but
at some point you have to deal with the fact that ebuilds are text files
edited by humans.
 
>>  >  A more featured language could allow for example: filtered output,
>>  >  exception handling->state storage->resuming.
>>  >
Portage does all those at the moment (filtering output can easily be done by
script, see update, or in an app like kuroo.)

>>  >  But the big deal is with the package definition, recently I learned
>>  >  about Domain Specific Languages, and I think that is the best option.
>>  >
>>  >  A new dsl allows many interesting features in the package definition
>>  >  itself like: inheritance, exceptions, arrays, hash tables, objects,
>>  >  modules, documentation, information messages, etc.
>>
>>  Note that most languages allow for the same things...so why would we
>>  author our own language?  What runtime would it use?
> 
> The language can be something very simple that has bash embedded. That
> has the advantage that you can just copy paste what you are doing
> already.
>
If it's got bash embedded it won't be simple ;) I see ebuilds as already
providing that DSL, in that you have full BASH capability and a library of
API functions you can use.
 
>>  >  I've tried different build systems: rpm, dpkg, autopackage.
>>  >  Unfortunately I never tried ebuild because it was based on bash as
>>  >  far as I could tell.
>>
>>  Typically a 'build system' would refer to 'autotools' or 'ant' or
>>  'setuptools' not an ebuild.
> 
> Is there such a big difference?
> 
> I'm sure it's possible to by-pass autotools and write down all the
> commands required to build something in an ebuild. Similarly it's
> possible to use autotools to compile and install a bunch of packages.
> 
You seem to be missing that ebuild is a support metastructure on top of
build systems like make, autotools or distutils?

> Those "build systems" also need to be updated, but that's another story.
>
What for?

>>  >  After almost a decade of using Linux I still haven't found a build
>>  >  system that suits all my needs. AFAIK ebuild is the most advanced but
>>  >  it's still relying on ancient technology (bash scripts) so there will
>>  >  always be limitations despite the brilliant ideas.
>>
>>  Bash is not 'ancient' and it works suprisingly well for the vast
>>  majority of cases.  I realize using a high level language would
>>  probably make some tasks easier (mmm dicts and real arrays).  There is
>>  the matter of porting over 10000 ebuilds however ;)
> 
> Yeah, bash is pretty good for many things, just doesn't scale that much.
>
I'm not sure where and how it needs to scale. The API provided is very easy
to work with and encompasses an amazing range of build systems.
 
> At some point someone decided the current status was not good enough
> and decided to create ebuild, even though he was well aware that
> thousands of already existing instructions about how to build packages
> would have to be re-written.
>
Not at all; ebuild utilises those instructions (eg makefiles). It just makes
every build system available and provides an extended BASH API for common
tasks.

> If it's easy to write people will possibly even write more of those.
> Take for example ArchLinux which also has around 10,000 packages
> simply because it's so easy to write them.
>
Ebuilds are pretty easy too imo. It's perfectly possible to write an ebuild
without a single function, for example, if the standard
configure --prefix=/usr and make system is in-place (and eclasses extend
that to other systems.)
 
> And with something that is distribution agnostic, different
> communities can benefit from sharing the same rules.
>
Ebuilds and portage can be used elsewhere, see
http://www.gentoo.org/proj/en/gentoo-alt/prefix/index.xml
and in any case Gentoo can be, and is, used as a basis for a binary distro.
 
>>  >
>>  >  The core of a distribution is the "packaging" system, and the core of
>>  >  the packaging system is the building system, which has no reason not
>>  >  to be distribution agnostic, and actually, packaging system agnostic.
>>  >
>>  >  Why not create a new build system with a state of the art programming
>>  >  language, and an advanced DSL that actually other distributions could
>>  >  use?
>>
I personally don't see the need.


-- 
gentoo-dev@lists.gentoo.org mailing list