[gentoo-portage-dev] Updated portage-ng requirements doc

[gentoo-portage-dev] Updated portage-ng requirements doc

[Daniel Robbins]

[-- Attachment #1.1: Type: text/plain, Size: 138 bytes --]

Hi All,

Attach is a much-improved portage-ng requirements doc (about 7x more
verbose than the previous one...)

Regards,

Daniel

[-- Attachment #1.2: portage-systemspec --]
[-- Type: text/plain, Size: 8135 bytes --]

portage-ng

This is the specification for portage-ng, a package manager/"ports" system that
is intended to be a successor to the current portage package manager.  The goal
of this project isn't simply to rewrite portage, but to also significantly
expand the potential of this type of tool.

Like portage, portage-ng will need to be able to perform the following tasks on
an individual host system: download sources or pre-compiled packages from the
Internet, configure, compile and install  sources according to user
requirements, and general package management. In addition, portage-ng should be
able to be easily extended to perform tasks that would be expected of it in an
enterprise, cluster, or other kind of environment, which may include management
of groups of systems over the network. The following requirements are intended
to further define the qualities of portage-ng that will allow it to tackle
existing problems more effectively, as well as tackle new types of problems:

architecture requirements:

1) facilitate parallel, community development

portage-ng's internal architecture should allow for true community development
my multiple teams. Approaches such as modularity, consistent well-defined APIs,
and modern programming practices could be used to make the development of code
more manageable, scalable and accessible.

2) allow for easy extensibility and new feature additions over time

Building on the previous requirement, the internal architecture of portage-ng
should allow seamless addition of new functionalities and capabilities. This
could be implemented using paradigms such as inheritance, "plugin"
functionality, a component model or combination, or possibly others.

3) provide a coherent model for (meta)data representation and storage

In the current portage code, there is support for various types of data
repositories, including ebuilds, /var/db/pkg (installed) package data, and
others. portage-ng should have a coherent (defined as "an orderly, logical, and
aesthetically consistent relation of parts") data model. The intended goal
is to reduce code complexity, increase internal code re-use, reduce the amount
of code dedicated to "load/store" operations, and allow allow re-targeting of
the data-store for all aspects of portage-ng in one fell swoop.

This also implies that careful consideration is made in representing the
various namespaces within portage-ng.

4) will run on a wide variety of systems

Portage was initially designed as a Linux-based application. We are now
looking at ways to integrate BSD, MacOS, and other platform support into
portage. portage-ng should be designed from the ground-up to support multiple
platforms. In addition, portage-ng should be able to support a large number
of CPU architectures, such as x86, ppc, amd64, ia64, sparc, alpha and others.

5) will run efficiently even on modest hardware, and take full advantage of
the resources of modern hardware.

Portage-ng should be able to run efficiently on modest hardware. On more modern
hardware, portage-ng should be able to take full advantage of current execution
of multiple parallizable steps in order to improve performance. This may
include multiple simultaneous fetching of source files, or the parallelization
of any other type of parallizable step. This implies, of course, that
portage-ng must be capable of recognizing which steps can be run in parallel
and which can't.

6) in as much as possible, encourage and/or enforce the development of
high-quality, versatile and maintainable code

To meet this requirement, we will need to successfully meet requirements 1 and
2. Beyond this, portage-ng development and package handling should use modern
practices to avoid unwanted side-effects that can cause unpredictable behavior.
Concepts that should be explored should include (but not be limited to) the use
of logic and functional programming concepts, as well as immutable data types,
both in portage-ng development as well as the day-to-day addition of new
packages to portage-ng. Meeting this requirement should also include an
evaluation of the current portage eclass functionality, and how new portage-ng
functionality could be created to further address the types of problems that
eclasses are intended to address.

7) encourage/enforce separation of package metadata from specific build steps
and algorithms

In portage, ebuilds generally have two types of metadata -- the first contains
specific steps to build, unpack, or compile the package and is inherently
procedural in nature. This metadata can be called "stepwise metadata." The
second type of metadata is more abstract and does not contain specific steps
to perform. For example, DEPEND, RDEPEND and SRC_URI are of this type, and
this allows Portage to use this metadata for a variety of (rather than a 
single) purpose. Due to the abstract nature of the metadata, algorithms such
as dependency resolution can be changed without causing ebuilds to break.

Portage-ng should be designed to provide a clear separation between stepwise
and abstract metadata, and encourage or enforce metadata to be represented in
an abstract form when possible, so it is of more use to portage-ng. This will
allow internal portage-ng algorithms to be changed without causing package
building and installation to break, and allow the system as a whole to be much
more flexible.

An example of a type of metadata that should be avoided would be a REBUILD
variable which tells portage-ng what packages should be rebuilt after a
particular package is installed. Such a REBUILD variable would need to be
treated as a stepwise procedure by portage-ng, which would limit its
usefulness. Instead, this metadata should be represented in another way so that
portage-ng can understand the finer levels of underlying dependencies and
resolve these on its own rather than being given a list of things to do via a
REBUILD variable.

In addition, the ability to track what abstract metadata is used by the 
stepwise metadata should be carefully considered. This would allow for the
elimination of the equivalent of the IUSE variable in portage-ng, since
portage-ng would already be aware of what metadata was being used by the
various parts of the build script.

8) allow for delivery of feature set that meets or exceeds expectations of
existing user community

Careful attention should be paid to the needs of our user community. What
features are they looking for in portage-ng? What kinds of problems are they
interested in having portage-ng solve? These needs should be taken into account
for any implementation.

In addition, the following design goals should be met in any implementation:

1) reflect the sensibilities of "ports" system designs, as well as Portage

These sensibilities include ease of use, ease of addition of new packages, ease
of learning and ease of accessing and modifying data. Plain text seems to be
preferred as a default data store. New possibilities can be explored as long
as the design as a whole addresses the general preference of the plaintext
approach.

2) provide an easy-to-understand developer API

This includes self-documenting API calls, good documentation, as well as
possibly other aspects of design.

3) provide an open, transparent architecture that reflects UNIX design
philosophy

This is related to design goal 1. This item underscores the fact Portage
in some ways reflects a UNIX design philosophy and any implementation should
explore ways to retain or expand the openness and transparency of portage-ng
over portage.

4) In every way possible, program should be malleable to allow conformation to
user needs and expectations, both interface as well as the specific actions
performed by the program.

Portage-ng should be designed as a general-purpose tool, rather than one that
just performs a handful of specific tasks. It's our hope that portage-ng will
be used in a variety of innovative ways not originally anticipated by its
designers.




[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-portage-dev] Updated portage-ng requirements doc

[Jason Stubbs]

On Saturday 29 November 2003 12:47, Daniel Robbins wrote:
> Hi All,
>
> Attach is a much-improved portage-ng requirements doc (about 7x more
> verbose than the previous one...)

Would a list of required functionality (without any implementation details) be 
part of this document as well? It seems to me that such information would be 
useful at the commencement of the design phase.

Regards,
Jason Stubbs

--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] Updated portage-ng requirements doc

[Jason Stubbs]

[-- Attachment #1: Type: text/plain, Size: 1268 bytes --]

On Saturday 29 November 2003 13:28, Jason Stubbs wrote:
> On Saturday 29 November 2003 12:47, Daniel Robbins wrote:
> > Hi All,
> >
> > Attach is a much-improved portage-ng requirements doc (about 7x more
> > verbose than the previous one...)
>
> Would a list of required functionality (without any implementation details)
> be part of this document as well? It seems to me that such information
> would be useful at the commencement of the design phase.

I've taken the liberty of writing up all the requests for basic functionility 
that I can think of and put it into the attached file. I haven't completed 
the "enterprise/cluster management" section yet as it's far past bed-time. I 
ask that others can comment and expand/revise this document, after which it 
can enhance the brief second paragraph of Daniel's system-spec.

In light of the architectural requirements, I attempted to write functional 
requirements that aren't tied to any other piece of functionality. While I 
haven't stated it in the document, the intention is to make it easier to 
descern points of separation of functional components when it comes time to 
actually do the design - yeah I know, jumping the gun.

Anyway, constructive comments are always appreciated.

Regards,
Jason Stubbs

[-- Attachment #2: portage-functional-reqs --]
[-- Type: text/plain, Size: 2715 bytes --]

** Ports **

Download sources or precompiled packages

Mirrors should be fully configurable - number, order, transport, credentials, etc. Furthermore, any package specific mirrors should also be fully configurable.


Configure & compile sources

Any and all available compilation options should be available to the user. Compilation options should be remembered across recompilations. Compilation options should be specified in such a way that commonality can be found between packages. Each package should provide definitions of common compilations options where there is any difference to the global interpretation. During compilation and preparation for installation, no files on the live filesystem should be affected.


** General Package Management **

Querying available packages

Queries should provide all information possible before a package is installed, including but not limited to:
- full title
- long description
- package home page
- size of minimally required source files or patches
- what packages are required to compile
- what packages are required to run
- license(s) for the package or parts there-in
- compilation flags known to fail


Query installed packages

Queries should cover all facets of installed packages, including but not limited to:
- what package provides which file
- the amount of disk space a package is occupying
- the date and time at which a package was installed


Install packages

Packages should be installed by the same method whether using precompiled packages or compiled sources. On installation, files should be noted for later removal. Under no circumstances should files on the live filesystem be overwritten by the installation of a package unless the files were provided by a previous version of the same package. In the case where another package "absorbs" the functionality of a package, it should be noted in the (partially-) depracated package.


Upgrade packages

When packages are upgraded, existing files should be backed so that upgrades can be easily rolled back when necessary.


Uninstalling installed packages

Packages should be checked for reverse dependencies to ensure that removal does not affect system functionality. Uninstalling packages should remove all files except those in protected directories. Files in protected directories should be backed up and removed from the live filesystem.


Updating available package database

As with mirrors above, the method of updating the package database should be fully configurable - number, order, transport, credentials, etc.


** Enterprise/Cluster Management **

Create precompiled packages
Provide package database mirror
Provide package source mirror
Provide precompiled package mirror


[-- Attachment #3: Type: text/plain, Size: 45 bytes --]

--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] Updated portage-ng requirements doc

[Daniel Robbins]

[-- Attachment #1: Type: text/plain, Size: 3634 bytes --]

On Sat, 2003-11-29 at 07:34, Jason Stubbs wrote:
> In light of the architectural requirements, I attempted to write functional 
> requirements that aren't tied to any other piece of functionality. While I 
> haven't stated it in the document, the intention is to make it easier to 
> descern points of separation of functional components when it comes time to 
> actually do the design - yeah I know, jumping the gun.
> 
> Anyway, constructive comments are always appreciated.

> Configure & compile sources
>
> Any and all available compilation options should be available to the 
> user.

In this paragraph, there seem to be several points and I'm not sure I
understand all of them. What is the purpose of having "Compilation
options [...] remembered across compilations?" For what purpose? Can
this purpose be expressed as a general requirement or design goal rather
than a specific feature request?

> ** General Package Management **
> 
> Querying available packages

By "Querying", do you mean user queries, such as "emerge -s"? If so,
this should be listed as a functional requirement for the default
text-based user interface (which may end up being a component or
plugin.) The requirements for the default text-based interface can be a
subsection of the main spec document. This part should also be rewritten
to be more specific about what you want, such as "Responses to user
queries by the text-based interface should be able to display a
comprehensive set of related metadata, beyond that which "emerge -s"
currently displays, including..."

> Install packages
>
> Packages should be installed by the same method whether using 
> precompiled packages or compiled sources.

What do you mean by "method?" It sounds like you're refering to an
interface issue rather than an internal architecture issue.

> Upgrade packages
>
> When packages are upgraded, existing files should be backed so that 
> upgrades can be easily rolled back when necessary.

Some people may not want to create back-ups for roll-backs. I think what
is important is that the capability exists to write a portage-ng
component to provide this functionality. This can be translated into a
design goal:

"The process of installing files to disk should be fully upgradable so
that capabilities such as roll-backs of already-installed packages can
be easily integrated into portage-ng."

Or it could be added as a comment to the section talking about how
portage-ng should be modular, to give more specific parameters about
what this modularity should allow us to do.

It can also be added as a functional requirement as well. I just want to
look at ways to "expand on" the functionality we want and derive general
architecture requirements and design goals from them.

> Uninstalling installed packages
> 
> Packages should be checked for reverse dependencies to ensure that 
> removal does not affect system functionality.

I think this needs to be distilled too. In order to act on reverse
dependencies, they need to be recorded. So we can break this down into a
requirement:

"All potentially-useful metadata should be recorded for each installed
package, including items such as what specific versions of dependencies
were installed at the time the package was compiled. This will allow for
support of more advanced dependency algorithms, such as ones that handle
reverse dependencies." 

and then add reverse deps as a functional requirement in its own
section.

I'll add your stuff to the doc for its next revision, and then get the
XML online.

Regards,

Daniel

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-portage-dev] Updated portage-ng requirements doc

[Jason Stubbs]

On Sunday 30 November 2003 01:02, Daniel Robbins wrote:
> On Sat, 2003-11-29 at 07:34, Jason Stubbs wrote:
> > In light of the architectural requirements, I attempted to write
> > functional requirements that aren't tied to any other piece of
> > functionality. While I haven't stated it in the document, the intention
> > is to make it easier to descern points of separation of functional
> > components when it comes time to actually do the design - yeah I know,
> > jumping the gun.
> >
> > Anyway, constructive comments are always appreciated.
> >
> > Configure & compile sources
> >
> > Any and all available compilation options should be available to the
> > user.
>
> In this paragraph, there seem to be several points and I'm not sure I
> understand all of them. What is the purpose of having "Compilation
> options [...] remembered across compilations?" For what purpose? Can
> this purpose be expressed as a general requirement or design goal rather
> than a specific feature request?

For "compilation option" read "USE flag". This is an instance where using 
terminology of the current implementation is actually clearer. Perhaps 
stating it as "per-package options should be supported" is much clearer?

> > ** General Package Management **
> >
> > Querying available packages
>
> By "Querying", do you mean user queries, such as "emerge -s"? If so,
> this should be listed as a functional requirement for the default
> text-based user interface (which may end up being a component or
> plugin.) The requirements for the default text-based interface can be a
> subsection of the main spec document. This part should also be rewritten
> to be more specific about what you want, such as "Responses to user
> queries by the text-based interface should be able to display a
> comprehensive set of related metadata, beyond that which "emerge -s"
> currently displays, including..."

Actually, I was thinking more along the lines of "ebuilds" supporting variable 
amounts of information and that extra types of information should be able to 
be added without breaking other components, although I failed to state that 
at all. I also wanted to list minimally required information for a package, 
but now I realise that the only information an installed package needs to 
offer is what files are installed.

> > Install packages
> >
> > Packages should be installed by the same method whether using
> > precompiled packages or compiled sources.
>
> What do you mean by "method?" It sounds like you're refering to an
> interface issue rather than an internal architecture issue.

Yep, get rid of that.

> > Upgrade packages
> >
> > When packages are upgraded, existing files should be backed so that
> > upgrades can be easily rolled back when necessary.
>
> Some people may not want to create back-ups for roll-backs. I think what
> is important is that the capability exists to write a portage-ng
> component to provide this functionality. This can be translated into a
> design goal:
>
> "The process of installing files to disk should be fully upgradable so
> that capabilities such as roll-backs of already-installed packages can
> be easily integrated into portage-ng."
>
> Or it could be added as a comment to the section talking about how
> portage-ng should be modular, to give more specific parameters about
> what this modularity should allow us to do.
>
> It can also be added as a functional requirement as well. I just want to
> look at ways to "expand on" the functionality we want and derive general
> architecture requirements and design goals from them.

I've never been good at this step in the process. I can easily recognise 
quality but have trouble producing it. My point is that I read the systemspec 
you wrote and couldn't think of anything that it didn't cover without 
thoughts of an actual design. A design entered my head here too! ;-)  Like 
you've interpreted, my intention was to state that support for that 
functionality (even by extension or replacement) is required.

> > Uninstalling installed packages
> >
> > Packages should be checked for reverse dependencies to ensure that
> > removal does not affect system functionality.
>
> I think this needs to be distilled too. In order to act on reverse
> dependencies, they need to be recorded. So we can break this down into a
> requirement:
>
> "All potentially-useful metadata should be recorded for each installed
> package, including items such as what specific versions of dependencies
> were installed at the time the package was compiled. This will allow for
> support of more advanced dependency algorithms, such as ones that handle
> reverse dependencies."
>
> and then add reverse deps as a functional requirement in its own
> section.

I fail to see the difference between the two, but I'm sure that all will be 
made clear.

> I'll add your stuff to the doc for its next revision, and then get the
> XML online.

Sounds good. Once there's an XML version online, it will seem more formal and 
thus a lot more input should be received. Thanks for your consideration.

Regards,
Jason Stubbs

--
gentoo-portage-dev@gentoo.org mailing list