[gentoo-portage-dev] portage-ng requirements doc

[gentoo-portage-dev] portage-ng requirements doc

[Daniel Robbins]

[-- Attachment #1: Type: text/plain, Size: 1915 bytes --]

Here is a draft of what we are looking for for the comprehensive
requirements document we need to develop. To help portage-ng
development, please expand on this document and post your
additions/improvements to this list. This doc will probably end up being
10-15 pages long...

portage-ng

This is the specification for portage-ng, a package manager/"ports"
system. portage-ng is intended to be able to perform the following tasks
on an individual host system: downloading sources or pre-compiled
packages from the Internet, configuration, compilation and installation
of sources according to user requirements, general package management.
In addition, portage-ng should be able to be easily extended to perform
tasks that would be expected of it in an enterprise, cluster, or other
kind of environment, which may include management of groups of systems
over the network.

architecture requirements:

1) facilitate parallel, community development
2) allow for easy extensibility and new feature additions over time
3) provide a coherent model for (meta)data representation and storage
4) will run on a wide variety of systems
5) will run efficiently even on modest hardware
6) in as much as possible, encourage and/or enforce the development of
high-quality, versatile and maintainable code
7) encourage/enforce separation of package metadata from specific build
steps and algorithms
8) allow for delivery of feature set that meets or exceeds expectations
of existing user community

design goals:

1) reflect the sensibilities of "ports" system designs
2) provide an easy-to-understand developer API
3) provide an open, transparent architecture that reflects UNIX design
philosophy
4) In every way possible, program should be malleable to allow
conformation to user needs and expectations,
both interface as well as the specific actions performed by the program.






[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-portage-dev] portage-ng requirements doc

[Paul de Vrieze]

[-- Attachment #1: signed data --]
[-- Type: text/plain, Size: 3258 bytes --]

Below I'll give my shot at it.

On Friday 28 November 2003 19:55, Daniel Robbins wrote:
> architecture requirements:
>
> 1) facilitate parallel, community development
> 2) allow for easy extensibility and new feature additions over time
> 3) provide a coherent model for (meta)data representation and storage
> 4) will run on a wide variety of systems
> 5) will run efficiently even on modest hardware
> 6) in as much as possible, encourage and/or enforce the development of
> high-quality, versatile and maintainable code
> 7) encourage/enforce separation of package metadata from specific build
> steps and algorithms
> 8) allow for delivery of feature set that meets or exceeds expectations
> of existing user community
>
> design goals:
>
> 1) reflect the sensibilities of "ports" system designs
> 2) provide an easy-to-understand developer API
> 3) provide an open, transparent architecture that reflects UNIX design
> philosophy
> 4) In every way possible, program should be malleable to allow
> conformation to user needs and expectations,
> both interface as well as the specific actions performed by the program.

implementation goals:
1) Support distfile renaming or distfile subdirectories or both to solve the
   name-clash problem.
2) Support useflag and slot dependencies as layed out by carpaski before and
   which will probably come into the current portage.
3) A full dependency tracker that can use the USEFLAGS from compile-time
   instead of check-time to determine dependencies and stale packages. (Should
   make depclean work)
4) Support partial overlap of same-version/same-package packages. This would
   require some extra variable like EXTRASLOT (for example php would specify
   a value like apache2 for the apache2 module and apache1 for the apache1
   module). This variable can be determined at the compilation time based on
   other parameters. The package database needs some name mangling (shouldn't
   be a big problem). Further there needs to be some overlap detection. What
   this would mean is that at the moment a second package of the same version
   but with a different EXTRASLOT would be merged, the overlapping filenames
   would be collected. Those would go in a general package. The ones from the
   original package that do not overlap go to a new package specific to that
   EXTRASLOT, and the files that are new go into the package with the new
   EXTRASLOT.

   Why is this important. It allows for a package like php to be compiled as
   both a plugin to apache1 as to apache2 without any difficulties. It is also
   the solution for modules like alsa and nvidia drivers, and it eases
   cross-compiler installation.
5) Make the ebuild format as extendable as possible (without breaking all
   current ones) to allow for future enhancements.
6) (sidestep) Make a clear system for specifying the locales that should be
   built, and if a specific locale needs to be chosen, what is the priority
   list for that. This will make locales in things like kde and openoffice a
   lot easier, but it also should be used by glibc and other ebuilds.

Paul

-- 
Paul de Vrieze
Researcher
Mail: pauldv@cs.kun.nl
Homepage: http://www.devrieze.net



[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-portage-dev] portage-ng requirements doc

[Andrew Gaffney]

Paul de Vrieze wrote:
> Below I'll give my shot at it.
> 
> On Friday 28 November 2003 19:55, Daniel Robbins wrote:
> 
>>architecture requirements:
>>
>>1) facilitate parallel, community development
>>2) allow for easy extensibility and new feature additions over time
>>3) provide a coherent model for (meta)data representation and storage
>>4) will run on a wide variety of systems
>>5) will run efficiently even on modest hardware
>>6) in as much as possible, encourage and/or enforce the development of
>>high-quality, versatile and maintainable code
>>7) encourage/enforce separation of package metadata from specific build
>>steps and algorithms
>>8) allow for delivery of feature set that meets or exceeds expectations
>>of existing user community
>>
>>design goals:
>>
>>1) reflect the sensibilities of "ports" system designs
>>2) provide an easy-to-understand developer API
>>3) provide an open, transparent architecture that reflects UNIX design
>>philosophy
>>4) In every way possible, program should be malleable to allow
>>conformation to user needs and expectations,
>>both interface as well as the specific actions performed by the program.
> 
> 
> implementation goals:
> 1) Support distfile renaming or distfile subdirectories or both to solve the
>    name-clash problem.
> 2) Support useflag and slot dependencies as layed out by carpaski before and
>    which will probably come into the current portage.
> 3) A full dependency tracker that can use the USEFLAGS from compile-time
>    instead of check-time to determine dependencies and stale packages. (Should
>    make depclean work)
> 4) Support partial overlap of same-version/same-package packages. This would
>    require some extra variable like EXTRASLOT (for example php would specify
>    a value like apache2 for the apache2 module and apache1 for the apache1
>    module). This variable can be determined at the compilation time based on
>    other parameters. The package database needs some name mangling (shouldn't
>    be a big problem). Further there needs to be some overlap detection. What
>    this would mean is that at the moment a second package of the same version
>    but with a different EXTRASLOT would be merged, the overlapping filenames
>    would be collected. Those would go in a general package. The ones from the
>    original package that do not overlap go to a new package specific to that
>    EXTRASLOT, and the files that are new go into the package with the new
>    EXTRASLOT.
> 
>    Why is this important. It allows for a package like php to be compiled as
>    both a plugin to apache1 as to apache2 without any difficulties. It is also
>    the solution for modules like alsa and nvidia drivers, and it eases
>    cross-compiler installation.
> 5) Make the ebuild format as extendable as possible (without breaking all
>    current ones) to allow for future enhancements.
> 6) (sidestep) Make a clear system for specifying the locales that should be
>    built, and if a specific locale needs to be chosen, what is the priority
>    list for that. This will make locales in things like kde and openoffice a
>    lot easier, but it also should be used by glibc and other ebuilds.

I've got something that should be in the new portage. It needs to be able to download 
source tarballs in a separate thread while compiling is happening to speed the overall 
process, much like apt-get for Debian.

-- 
Andrew Gaffney


--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] portage-ng requirements doc

[Daniel Robbins]

[-- Attachment #1: Type: text/plain, Size: 394 bytes --]

On Fri, 2003-11-28 at 17:47, Andrew Gaffney wrote:
> I've got something that should be in the new portage. It needs to be able to download 
> source tarballs in a separate thread while compiling is happening to speed the overall 
> process, much like apt-get for Debian.

OK, this can be generally phrased as "Support simultaneous execution of
steps for improved performance."

Daniel

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-portage-dev] portage-ng requirements doc

[Daniel Robbins]

[-- Attachment #1: Type: text/plain, Size: 743 bytes --]

On Fri, 2003-11-28 at 15:15, Paul de Vrieze wrote:
> implementation goals:

I do not mean any offense by this, but I will continue to
unapologetically reject anything implementation-focused at this point.
Requirements and design goals need to be written in such a way so that
they are not implementation-specific. So, words such as "USEFLAGS,"
"distfiles," "slots" or even "ebuild" should not be used in any of the
requirements, as they have a built-in bias towards a particular
implementation.

Figure out what types of problems you are trying to address in your list
of implementation goals, and distill them to a raw, generic requirement
or design goal. Then I can incorporate them into the design spec.

Regards,

Daniel

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-portage-dev] portage-ng requirements doc

[Lisa Seelye]

[-- Attachment #1: Type: text/plain, Size: 904 bytes --]

On Fri, 2003-11-28 at 13:55, Daniel Robbins wrote:
> Here is a draft of what we are looking for for the comprehensive
> requirements document we need to develop. To help portage-ng
> development, please expand on this document and post your
> additions/improvements to this list. This doc will probably end up being
> 10-15 pages long...

The design specs should be completely standalone from the
implementation.  That is, no language specific cruft in the specs. 
This, so the user can (if he or she so chooses) rewrite Portage(-ng) on
their own in any capable language.

On a related thread, for the sake of a short dependency tree (both in
code length and number of packages needed) the chosen language should be
a major deciding factor when picking a language to implement the final
project in.

-- 
Regards,
-Lisa
<Vix ulla tam iniqua pax, quin bello vel aequissimo sit potior>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-portage-dev] portage-ng requirements doc

[Marius Mauch]

[-- Attachment #1: Type: text/plain, Size: 2382 bytes --]

Hi,

Some of my points are probably already covered in your list, but let me
show my wishlist:

portage-ng should be ...

- completely based on exchangable components (I don't like the term
plugin here as plugins are IMO always optional) and each component
should have a well defined interface as well as metadata stating
dependencies on other components (e.g. a DB based tree backend needs a
DB compatible sync mechanism and parser).

- statically typed, so it can be better checked for errors before
released. I realize that this might not be possible in several
languages, but then at least we should enforce consistent typenames in
the API documentation (as currently you often have to read the source to
find out what parameter a function really wants).

- able to be used as a secondary package manager, this would also
include the ability to enable users to install packages in their
homedirs using system packages for the dependencies. One important part
of this is implementing pathspec.

- as secure as possible. This includes signed packages (both source and
binary) and automatic priorization of security related updates.

That's all for now, but I think my first point implicates many other
points mentioned by you. Let me elaborate a bit more by giving an
example what components could be used in the current portage:
- ebuild parser
- metadata parser
- profile parser
- config parser
- packagename/version parser
- dependency resolver
- rsync sync component
- cvs sync component
- http sync component (for snapshots)
- filesystem portdir storage component
- filesystem package database storage component
- searching component
...

So now the portagesql guys for example could just write different
portdir storage, sync and metadata parser components to use mysql as a
backend for the metadata instead of re-implementing portage completely.
For the dependency example above their storage component would need
their sync component or, more flexible, it implements some write()
functions that could be used by sync components implementing a suitable
interface (but for instance a storage component for cd-rom based
snapshots could not do that). I hope you get the idea.

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]