[gentoo-portage-dev] The purpose of USE flags

[gentoo-portage-dev] The purpose of USE flags

[Jason Stubbs]

Hello all,

At the risk of sounding precocious, I think I'm going to open another can of 
worms. ;-)

I've been thinking on the idea of USE flags and am wondering what the 
developers' perspective is. They are defined in "Gentoo Guide to USE flags" 
as "keywords that define options used on a system-wide basis to configure 
applications during their respective compilation procedures". Is there any 
other, more formal, definition?

My reason for asking is that my feel, from the way developers and experience 
users talk about USE flags, that their usage has been mostly restricted to 
options provided by configure scripts. This is natural as (I assume) that's 
the way that USE flags evolved. However, I think that the the purpose of USE 
flags needs to be rethought, when implemented in portage-ng, to include how 
USE flags pertain to the system as a whole.

I was half thinking about it when I wrote the following except from my last 
mail:

On Tuesday 09 December 2003 07:06, Jason Stubbs wrote:
> USE flags need to be able to be grouped into 'super-USE' flags. For example: 
> "multimedia", "workstation", "server", etc.
> where "multimedia" = "video-codecs", "audio-codecs", "media-recording", etc.
> where "video-codecs" = "avi", "divx", "dvd", etc.

I'm going to expand on this idea a little, although I think it might not be 
well received due to the unexplained fear of USE flag explosion. My idea is 
that USE flags should be somewhat the equivalent of Microsoft Windows 
Installer's Product Feature selection tree, the main difference being that it 
applies to the whole operating system.

To explain further, I will again use an example possibility:
Multimedia
+ Recording
- Playback
  + Video
  - Audio
    + Audio Codecs
    - Audio Providers
      - OSS
      - ALSA
      - ARTS
      - ESD
      - <default>

From this small excerpt of a possible tree, you can image how big a full tree 
would be. The benefits of this style of tree are manyfold, but I'll just go 
over a few of the obvious (to me):

* Finer grain of control

Depending on what is actually implement of course, this model can offer much 
more control. If the above example is extended to include ALSA drivers, the 
model is shown to become a unified way to model all option behaviour of 
packages. Mozilla, for example, could be done in exactly the same way.

* Better categorization

With this USE flag model, if an ebuild-ng specifies what functionality it 
provides, both intrisically and optionally, packages become very easy to find 
for the end-user. It will also be easy to find other packages that do same 
things as a package that the user is familiar with.

That's all that I can think of at the moment. The only negative thing I can 
see is the explosion of USE flags. However, this is only a bad thing if they 
are uncategoriezed. Proper categorization should actually make them easier to 
understand and easier to maintain.

Well, that's enough said for now. All constructive criticism and any additions 
welcome.

-- 
Regards,
Jason Stubbs

--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] The purpose of USE flags

[Sami Näätänen]

On Friday 12 December 2003 21:27, Jason Stubbs wrote:
> Hello all,
>
> I've been thinking on the idea of USE flags and am wondering what the
> developers' perspective is. They are defined in "Gentoo Guide to USE
> flags" as "keywords that define options used on a system-wide basis
> to configure applications during their respective compilation
> procedures". Is there any other, more formal, definition?

One thing that I think should be changed is the way that now they are 
USE flags ie binary valued variables. I would like to have these as 
multivalue variables, which then could be used in the following way.

use.gui.gtk 10,qt 1,curses allways

or using alternate more readable syntax

use.gui.gtk 10
+use.gui.qt 1
+use.gui.curses allways

So if I would have a packages that possibly has any of the three 
different possible gui's the above definition would compile the 
pacakges as follow.

If the package has curses support the support is allways included.
If it has qt and no gtk support then qt is used.
If gtk is supported it will take place as the higer priority gui.

If there is more than one allways qui's, but all of them are not 
supported at the same time so instead of allways we possibly could use 
to priority types look below.

> My reason for asking is that my feel, from the way developers and
> experience users talk about USE flags, that their usage has been
> mostly restricted to options provided by configure scripts. This is
> natural as (I assume) that's the way that USE flags evolved. However,
> I think that the the purpose of USE flags needs to be rethought, when
> implemented in portage-ng, to include how USE flags pertain to the
> system as a whole.

And here I give another possibility that could be used as a multi valued 
variable.

use.X never

This definition would simply block any tries to install something that 
would need an X-server to be runned.

So we would need a definition for the ebuilds like the following.

in xfree ebuild:

never_use=use.X

This would mark that this ebuild can't be installed when the useflag 
use.X is set to never. This same thing could be used to block say Gnome 
or KDE or ...


Here are some things to thought for this kind of system.

What kinds of values except priority we allow?
should we have type defined the use 'flags'?
  This is quite hard, but it would be better if we could manage using
  only one type of variables even if the type would be quite complex.

If yes what kind of different types we need?
  The earlier mentioned gui example could has two allways stated gui's,
  but if only one of them can be used then we would need two
  different numerical value types. As an example:
    use.gui.gtk allways 10
    -use.gui.qt
    +use.gui.curses allways
  This would add gtk and curses allways, but if only one of them can be
  used then it is gtk. Qt support is disabled in this example.

Do we need multiple numerical types, or can we use simply one numerical 
type and interpret it differently as needed.

How to make this efficient?
  In my mind this need's a database, but in my mind all portage stuff
  should be in a database, but that's another story at the moment ( I'm
  thinking it constantly, and the idea is heading out, but I need to
  write it down first).




I should state the difference of following two definitions to make sure 
everybody understands what the heck I'm talking about. :)

-use.X
use.X never

The former is mild forbid. So if the support is not required to use the 
application, then the application can be used. It will be without the 
support for X though. The second definition simply says that if one 
ever tried to install package that would install X, then this package 
can't be installed.

Applications that has optional support for X can be use though, because 
they can be compiled without X support. So in the first case X server 
itself can be installed excplicitly, but in the later it can't be 
installed at all.






And here are the examples of my definitions as they would be in a config 
file (remember that I would put these in the DB).

=use.gui.gtk prefer 10 # Any other gui variables, but gtk are forbiden
+use.gui.gtk prefer 10 # Only affects the value of the gtk variable
-use.gui.gtk           # Only disables gtk support
-use.gui.gtk prefer 10 # Same thing, because the value is ignored
!use.gui.gtk           # Forbids installation of gtk and gtk apps
+use.gui.gtk never     # Same thing
-use.gui.gtk never     # Not same thing the never is ignored

So this has a grammar of something like this

definition = <operation><variable>



And some dangerous examples to show that the powerfull absolute denial
definitions should be thought well before using.

!use.gui # Forbids all packages that use any GUI variables
!use     # Forbids all packages that use any USE variables


So now comments are welcome. :)


--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] The purpose of USE flags

[Marius Mauch]

[-- Attachment #1: Type: text/plain, Size: 419 bytes --]

Hi,

personally I really dislike this proposal as it adds a lot of complexity
for functionality that would rarely be used. Also IMO USE flags should
not prevent the installation of a package, that's the job of masking.

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-portage-dev] The purpose of USE flags

[Jason Stubbs]

On Saturday 13 December 2003 04:27, Jason Stubbs wrote:
> I've been thinking on the idea of USE flags and am wondering what the
> developers' perspective is. They are defined in "Gentoo Guide to USE flags"
> as "keywords that define options used on a system-wide basis to configure
> applications during their respective compilation procedures". Is there any
> other, more formal, definition?

Well, seeing there was no negative (read: any) feedback, I will summarize the 
motivation for this as a new requirement. Something to the effect of:

The meaning and function of USE flags should be immediately apparent to users 
regardless of their technical skill. This applies to individual USE flags, 
USE flags with respect to individual applications and also with respect to 
the system as a whole.

-- 
Regards,
Jason Stubbs

--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] The purpose of USE flags

[Jason Stubbs]

On Monday 15 December 2003 07:23, Marius Mauch wrote:
> personally I really dislike this proposal as it adds a lot of complexity
> for functionality that would rarely be used.

Agreed there.

> Also IMO USE flags should not prevent the installation of a package, that's 
> the job of masking. 

Many users often get frustrated by the fact that they have specified "-arts", 
for example, but still get arts installed. If USE flags specified optional 
and intrinsic functionality, this would not be hard to implement and would be 
much more intuitive than package masking.

On the other hand, if the current package masking was made more intuitive and 
advertised, there probably wouldn't be anywhere near as many users as there 
are complaining.

Either way, I think that having ebuilds specifiy their intrinsic functionality 
through USE flags or such some construct has many benefits.

--
Regards,
Jason Stubbs

--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] The purpose of USE flags

[Sami Näätänen]

On Monday 15 December 2003 00:23, Marius Mauch wrote:
> Hi,
>
> personally I really dislike this proposal as it adds a lot of
> complexity for functionality that would rarely be used. Also IMO USE
> flags should not prevent the installation of a package, that's the
> job of masking

Well at least in current state masking is very poorly implemented.

Besides the system I explined can be used as a tree use flags and thus 
it does not make it more complex on teh contrary.

Example using current implementation as the functionality basis:

Enable all gui USE 'flags'
=use.gui.*
Disable qt USE 'flag'
-use.gui.qt

So the only thing different is the amount of text and the fact that one 
needs to put multiple rows of stuff. But this is more readable than 
this:

USE="X acpi acpi4linux apache2 avi cdr crypt cups debug doc dvb dvd 
emacs encode esd examples fbcon flash foomaticdb gd gdbm ggi gif 
gphoto2 gpm gps gtk gtk2 imap imlib innodb java joystick jpeg junit 
ladcca lcms leim libg++ libwww lirc mad maildir memlimit mikmod mmx 
mozilla moznocompose moznoirc moznomail mpeg mpi mule ncurses nls oav 
oggvorbis opengl oss pam pdflib perl pic plotutils png postgres python 
qt quicktime readline samba scanner sdl slp sse ssl tcpd tetex tiff 
truetype wmf x86 xml xml2 xmms xv zlib"

And besides the USE 'flags' should be program controlled by default ala 
ufed and not in the make.conf.

Oh and this representation is simply to make it tree like.
Because in my opinion all the stuff has to be in database so these 
representations doesn't really matter, does they?

So we need a preferences program to set the configurations in the 
database in this way we can controll multiple Gentoo hosts in single 
portage database from single machine.


Now that I have something up from my final idea I can some what clarify 
the idea.

So we have single database where mutiple hosts information can be 
stored.

We have a portage 'daemon' to handle all the database access, package 
installation and uninstallation. It doesn't need to be allways in the 
background, but it can be launced when something needs it.

This one daemon controlls the dependency tree calculations etc and can 
compile multiple packages simultaneysly if all their dependencies are 
allready installed. Thi ofcourse depends wheter the multiple processes 
are allowed or not through SMP and/or distcc and/or openmosix

So now one could que new packages to the daemon at will and it would add 
these to it's dependency tree. The daemon installs new packages to the 
tree (It might even be a special graph, if there are packagages that 
genuinly can depend on eich other) and installs those packages that are 
not meeting the requirements. This way we can also see the situations 
where something can't be installed, because something else doesn't work 
with certain version of the common dependency of the two packages.



Now a litle more to the portage level. :)

So the portage daemon would need to be a modular concept to be really 
usefull. So here are my hasty propositions to be as the modules.

1. Main module, which controls the others. So is kind the current
   portage core. And the daemon really is only this the others are
   libraries.

2. Preferences module. Sets all the system options ie what database and
   the database URI to use etc. This is lot like the current
   make.profile and some parts of make.conf.

3. Database handling. This should be so that there can be one for every
   database one might use, but of course not all of them made, but given
   the possibility to users to make their own. The default should be a
   very light weigth solution like flat files. This module is
   responsible for database queries and alterings.

4. Build module. This builds a package in isolated environment

5. Install/uninstall module, which handles the file altering in the live
   system. 

Ebuilds wouldn't anymore be a shell scripts, but the build module will 
offer all necessary commands to build something. Ebuild will simply be 
a command sequence. Same goes for the install module.
This way it can be made more secure and easier to control.

Because this consept uses database abstaraction we can put only the 
staff that suits well to the database and use other type of data 
structures if more suitable to the needed task.


--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] The purpose of USE flags

[Marius Mauch]

[-- Attachment #1: Type: text/plain, Size: 6287 bytes --]

On 12/16/03  Sami Näätänen wrote:

> On Monday 15 December 2003 00:23, Marius Mauch wrote:
> > Hi,
> >
> > personally I really dislike this proposal as it adds a lot of
> > complexity for functionality that would rarely be used. Also IMO USE
> > flags should not prevent the installation of a package, that's the
> > job of masking
> 
> Well at least in current state masking is very poorly implemented.
> Besides the system I explined can be used as a tree use flags and thus
> it does not make it more complex on teh contrary.

I should have explained better: I'm not against the tree representation
of USE flags, but getting away from binary flags makes things complex.

> Example using current implementation as the functionality basis:
> 
> Enable all gui USE 'flags'
> =use.gui.*
> Disable qt USE 'flag'
> -use.gui.qt
> 
> So the only thing different is the amount of text and the fact that
> one needs to put multiple rows of stuff. But this is more readable
> than this:
> 
> USE="X acpi acpi4linux apache2 avi cdr crypt cups debug doc dvb dvd 
> emacs encode esd examples fbcon flash foomaticdb gd gdbm ggi gif 
> gphoto2 gpm gps gtk gtk2 imap imlib innodb java joystick jpeg junit 
> ladcca lcms leim libg++ libwww lirc mad maildir memlimit mikmod mmx 
> mozilla moznocompose moznoirc moznomail mpeg mpi mule ncurses nls oav 
> oggvorbis opengl oss pam pdflib perl pic plotutils png postgres python
> qt quicktime readline samba scanner sdl slp sse ssl tcpd tetex tiff 
> truetype wmf x86 xml xml2 xmms xv zlib"

These settings don't match, if you want to compare different
implementations please use examples that match. One problem I see with
the tree represenations (and a flag being present in multiple leafs) is
an ordering probem. E.g. we have use.gui.gnome.esd (as esd is part of
gnome) and use.sound.esd. Now a user has "-use.gui.gnome +use.sound", is
esd enabled then or not?

> And besides the USE 'flags' should be program controlled by default
> ala ufed and not in the make.conf.

Why?

> Oh and this representation is simply to make it tree like.
> Because in my opinion all the stuff has to be in database so these 
> representations doesn't really matter, does they?

I agree on the tree representation, but I don't agree on the "everything
should be in a database" part. One of the main benefits of a
plain-textfile approach is that it's easy to fix, using databases for
everything adds another layer of complexity.

> So we need a preferences program to set the configurations in the 
> database in this way we can controll multiple Gentoo hosts in single 
> portage database from single machine.
> Now that I have something up from my final idea I can some what
> clarify the idea.
> 
> So we have single database where mutiple hosts information can be 
> stored.
> 
> We have a portage 'daemon' to handle all the database access, package 
> installation and uninstallation. It doesn't need to be allways in the 
> background, but it can be launced when something needs it.
> 
> This one daemon controlls the dependency tree calculations etc and can
> 
> compile multiple packages simultaneysly if all their dependencies are 
> allready installed. Thi ofcourse depends wheter the multiple processes
> are allowed or not through SMP and/or distcc and/or openmosix
> 
> So now one could que new packages to the daemon at will and it would
> add these to it's dependency tree. The daemon installs new packages to
> the tree (It might even be a special graph, if there are packagages
> that genuinly can depend on eich other) and installs those packages
> that are not meeting the requirements. This way we can also see the
> situations where something can't be installed, because something else
> doesn't work with certain version of the common dependency of the two
> packages.

Now that's a bit more than your original mail ;)

> Now a litle more to the portage level. :)
> 
> So the portage daemon would need to be a modular concept to be really 
> usefull. So here are my hasty propositions to be as the modules.
> 
> 1. Main module, which controls the others. So is kind the current
>    portage core. And the daemon really is only this the others are
>    libraries.

No, the core should do only the initialization of the other modules, and
provide an interface to tools, t should not contain any portage
functionality itself.

> 2. Preferences module. Sets all the system options ie what database
> and
>    the database URI to use etc. This is lot like the current
>    make.profile and some parts of make.conf.

Ok, we can have different config modules, but IMO text configs should be
the default.

> 3. Database handling. This should be so that there can be one for
> every
>    database one might use, but of course not all of them made, but
>    given the possibility to users to make their own. The default
>    should be a very light weigth solution like flat files. This module
>    is responsible for database queries and alterings.

Different modules for portage tree's, package databases and so on, ok.
But we still should use text files as default for several reasons:
- easier to fix
- less dependencies
- current mirror system is based on textfiles, I don't know how
easy/hard it is to get the mirrors switched to a new scheme

> 4. Build module. This builds a package in isolated environment

ok

> 5. Install/uninstall module, which handles the file altering in the
> live system. 

ok

> Ebuilds wouldn't anymore be a shell scripts, but the build module will
> offer all necessary commands to build something. Ebuild will simply be
> a command sequence. Same goes for the install module.
> This way it can be made more secure and easier to control.

Yes and no, part of the ebuild-ng format will be shell commands as I
don't see how we can get away from it. Also we'll need a compability
module to parse old-style ebuilds as the current 10000 ebuilds or so
can't be rewritten over night.

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-portage-dev] The purpose of USE flags

[Pieter Van den Abeele]

On 16 Dec 2003, at 18:11, Sami Näätänen wrote:

> We have a portage 'daemon' to handle all the database access, package
> installation and uninstallation. It doesn't need to be allways in the
> background, but it can be launced when something needs it.
>
> This one daemon controlls the dependency tree calculations etc and can
> compile multiple packages simultaneysly if all their dependencies are
> allready installed. Thi ofcourse depends wheter the multiple processes
> are allowed or not through SMP and/or distcc and/or openmosix
>
> So now one could que new packages to the daemon at will and it would 
> add
> these to it's dependency tree.

The idea of having a deamon maintain a dependency tree is a good idea, 
but somewhat too detailed for a requirement. It doesn't necessarily 
need to be a daemon. It can be a simple standalone program too without 
using a socket or something.

I'd rephrase it as:

A subset (module) of portage is responsible for maintaining a 
representation of the system. We call the representation of a system a 
build plan.
Installing new components can be seen as merging the buildplan for the 
new components with the old build plan.
...

> Now a litle more to the portage level. :)
>
> So the portage daemon would need to be a modular concept to be really
> usefull. So here are my hasty propositions to be as the modules.

modularity is what we want for several reasons

> 1. Main module, which controls the others. So is kind the current
>    portage core. And the daemon really is only this the others are
>    libraries.

I wouldn't call this the portage core, as the existence of this module 
is not -really- crucial for portage to work. In a modular system, one 
should (theoretically, whether this is easy or not is another question) 
be able to communicate with each module, the main module should 
implement things like:

- interaction with the user (a user controls other portage module 
through executing a command like 'emerge --pretend blabla')
- userland modules (interaction with catalyst?) emerge --build-livecd 
asystemspec
...

> 2. Preferences module. Sets all the system options ie what database and
>    the database URI to use etc. This is lot like the current
>    make.profile and some parts of make.conf.

I would use 'knowledge base' instead of database, because it is a more 
generic term. A ports directory is not really a database as we know it 
(think SQL etc.). One of the benefits of formulating the core in 
prolog, would be to use a 'datalog' syntax, datalog is regarded as a 
universal query language. SQL can be expressed in datalog, just like it 
can be expressed in relational algebra/calculus.

Preferences are a good idea, there can be system preferences, user 
preferences ... containing statements about the knowledge base:

avoid pam,
filter unstable packages when the package is a core package
...

I'm also thinking about having regular users specify their own 
preferences, which can be taken into account by root when reasoning 
about the system, but which should be interpreted as wishes instead of 
'requirements'

> 3. Database handling. This should be so that there can be one for every
>    database one might use, but of course not all of them made, but 
> given
>    the possibility to users to make their own. The default should be a
>    very light weigth solution like flat files. This module is
>    responsible for database queries and alterings.

see 2.

> 4. Build module. This builds a package in isolated environment

more generic: 'interprets build plans'

> 5. Install/uninstall module, which handles the file altering in the 
> live
>    system.

I'd add this to 4, The idea is to - just like catalyst - use a strategy 
design pattern:

each concrete architecture/platform can implement its own 
'instal/uninstall' strategy.

> Ebuilds wouldn't anymore be a shell scripts, but the build module will
> offer all necessary commands to build something. Ebuild will simply be
> a command sequence. Same goes for the install module.

This makes eclasses overriding/adding some default behaviour or 
'meta-data' somewhat more difficult to implement.
I'd split ebuilds into two parts: the part detailing 'howto' and the 
part detailing 'about'.

about: dependencies, keywords, use flags (iuse)
howto: command sequences.

howto data can take into account about data, but not the other way 
around. Still, I'd abstract away from the concrete ebuild syntax and 
the way to interpret it. The key is to design a system which is 
powerfull enough to represent other (ports) systems in.

> This way it can be made more secure and easier to control.

right now when you write:

expoit() {
	rm -rf /
}
expoit

at the top of the ebuild, this gets executed when reading data 'about' 
the ebuild. Similarily one could write

DEPEND=""
setdepend() {
	DEPEND=`date`
}
setdepend

which returns another value for DEPEND every time the ebuild is 
interpreted.

I would however allow:

src_compile() {
	sequence of commands
	if (use foo) {
		more commands	
	}
}

because that is really powerfull.


> Because this consept uses database abstaraction we can put only the
> staff that suits well to the database and use other type of data
> structures if more suitable to the needed task.

I certainly wouldn't exclude using a database (like in sql databases, 
etc.), but wouldn't exclude other possibilities too. If anyone wants to 
create one huge file containing all data in his own syntax, that should 
be possible because of the modular design.

Pieter



--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] The purpose of USE flags

[Sami Näätänen]

On Tuesday 16 December 2003 21:59, Marius Mauch wrote:
> On 12/16/03  Sami Näätänen wrote:
> >
> > Well at least in current state masking is very poorly implemented.
> > Besides the system I explined can be used as a tree use flags and
> > thus it does not make it more complex on teh contrary.
>
> I should have explained better: I'm not against the tree
> representation of USE flags, but getting away from binary flags makes
> things complex.

And things that doesn't well fit in binary world will be complex if 
represented in binary. So that's why I would see that makeing something
that looks like binary, but can get other values if explicitly added 
would be best choice.

> > Example using current implementation as the functionality basis:
> >
> > Enable all gui USE 'flags'
> > =use.gui.*
> > Disable qt USE 'flag'
> > -use.gui.qt
> >
> > So the only thing different is the amount of text and the fact that
> > one needs to put multiple rows of stuff. But this is more readable
> > than this:
> >
> > USE="X acpi acpi4linux apache2 avi cdr crypt cups debug doc dvb dvd
> > emacs encode esd examples fbcon flash foomaticdb gd gdbm ggi gif
> > gphoto2 gpm gps gtk gtk2 imap imlib innodb java joystick jpeg junit
> > ladcca lcms leim libg++ libwww lirc mad maildir memlimit mikmod mmx
> > mozilla moznocompose moznoirc moznomail mpeg mpi mule ncurses nls
> > oav oggvorbis opengl oss pam pdflib perl pic plotutils png postgres
> > python qt quicktime readline samba scanner sdl slp sse ssl tcpd
> > tetex tiff truetype wmf x86 xml xml2 xmms xv zlib"
>
> These settings don't match, if you want to compare different
> implementations please use examples that match. One problem I see
> with the tree represenations (and a flag being present in multiple
> leafs) is an ordering probem. E.g. we have use.gui.gnome.esd (as esd
> is part of gnome) and use.sound.esd. Now a user has "-use.gui.gnome
> +use.sound", is esd enabled then or not?

I think this is problem only if the thing is graph. What it is if the 
flags can be present in many places.

> > And besides the USE 'flags' should be program controlled by default
> > ala ufed and not in the make.conf.
>
> Why?

Because even at current state these flags are not very self explanatory 
and thus can make a really big mess to control. I although agree that I 
am a fan of config files, but only to the point where they are clean an 
ordered, what the whole punch of use flags simply are not.

> > Oh and this representation is simply to make it tree like.
> > Because in my opinion all the stuff has to be in database so these
> > representations doesn't really matter, does they?
>
> I agree on the tree representation, but I don't agree on the
> "everything should be in a database" part. One of the main benefits
> of a plain-textfile approach is that it's easy to fix, using
> databases for everything adds another layer of complexity.

I would keep a flat file backup, which then could be parsed to the 
database. This gives the speed and reliability.
So one file for the system one for the package tree etc.

> > So we need a preferences program to set the configurations in the
> > database in this way we can controll multiple Gentoo hosts in
> > single portage database from single machine.
> > Now that I have something up from my final idea I can some what
> > clarify the idea.
> >
> > So we have single database where mutiple hosts information can be
> > stored.
> >
> > We have a portage 'daemon' to handle all the database access,
> > package installation and uninstallation. It doesn't need to be
> > allways in the background, but it can be launced when something
> > needs it.
> >
> > This one daemon controlls the dependency tree calculations etc and
> > can
> >
> > compile multiple packages simultaneysly if all their dependencies
> > are allready installed. Thi ofcourse depends wheter the multiple
> > processes are allowed or not through SMP and/or distcc and/or
> > openmosix
> >
> > So now one could que new packages to the daemon at will and it
> > would add these to it's dependency tree. The daemon installs new
> > packages to the tree (It might even be a special graph, if there
> > are packagages that genuinly can depend on eich other) and installs
> > those packages that are not meeting the requirements. This way we
> > can also see the situations where something can't be installed,
> > because something else doesn't work with certain version of the
> > common dependency of the two packages.
>
> Now that's a bit more than your original mail ;)

Yeah sorry that one was just part of my portage dream... :)

> > Now a litle more to the portage level. :)
> >
> > So the portage daemon would need to be a modular concept to be
> > really usefull. So here are my hasty propositions to be as the
> > modules.
> >
> > 1. Main module, which controls the others. So is kind the current
> >    portage core. And the daemon really is only this the others are
> >    libraries.
>
> No, the core should do only the initialization of the other modules,
> and provide an interface to tools, t should not contain any portage
> functionality itself.

That I meant. the core was simply a bad choice of word.

> > 2. Preferences module. Sets all the system options ie what database
> > and
> >    the database URI to use etc. This is lot like the current
> >    make.profile and some parts of make.conf.
>
> Ok, we can have different config modules, but IMO text configs should
> be the default.

Well I meant more like we could only upgrade this part to suit the needs
of possible new modules withouth breaking all the other stuff.
and besides I definitely want to use text config files for this.

> > 3. Database handling. This should be so that there can be one for
> > every
> >    database one might use, but of course not all of them made, but
> >    given the possibility to users to make their own. The default
> >    should be a very light weigth solution like flat files. This
> > module is responsible for database queries and alterings.
>
> Different modules for portage tree's, package databases and so on,
> ok. But we still should use text files as default for several
> reasons: - easier to fix
> - less dependencies
> - current mirror system is based on textfiles, I don't know how
> easy/hard it is to get the mirrors switched to a new scheme

The current sync would be a portage daemon connecting to another one.
The way to do it could be simply rsyncing the tree 'backup' and then 
localy parsing it and making the changes. The only problem is how to 
make package name/category transitions easily, but that could be made 
say using package identifiers instead their names as the basis of the 
package then package transformation would be a simple field change. 
using say 64 bit integer will give quite many possible packages. :)

Default database would be some custom made 'database' using flat files 
or similar. The alternatives that I would include to the core portage 
teams shoulders would be berkely DB, postgresgl and mysql.

And I would make the flatfile backup system, which could be made when 
syncing or after bunch of (un)installations. Ofcourse also when asked 
to backup the system.
 
> > 4. Build module. This builds a package in isolated environment
>
> ok
>
> > 5. Install/uninstall module, which handles the file altering in the
> > live system.
>
> ok
>
> > Ebuilds wouldn't anymore be a shell scripts, but the build module
> > will offer all necessary commands to build something. Ebuild will
> > simply be a command sequence. Same goes for the install module.
> > This way it can be made more secure and easier to control.
>
> Yes and no, part of the ebuild-ng format will be shell commands as I
> don't see how we can get away from it. Also we'll need a compability
> module to parse old-style ebuilds as the current 10000 ebuilds or so
> can't be rewritten over night.

I would put all the stuff to database like this.

All dependency and fetching etc info from the ebuilds to database 
fields.

All the building, installation etc functions to database so that every 
needed function could be a either shell command sequence from the 
current ebuilds or a new command sequence. These new command sequences 
can be interpreted differently to make the process more secure and 
possibly faster.

These old-style ebuilds can be parsed to the database. After all we 
would need a way to make new ebuilds and for that flat file is the 
simple and efective solution. So we would need to be able to parse 
theese things to the database any way.

One more bonus is that we can put some default command sequences to the 
database and if those are enough to the package then use one of them. 
If not write new one.


--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] The purpose of USE flags

[Jason Stubbs]

Hmmm, I didn't get this email.

On Tuesday 16 December 2003 21:59, Marius Mauch wrote:
> One problem I see with the tree represenations (and a flag being present in 
> multiple leafs) is an ordering probem. E.g. we have use.gui.gnome.esd (as 
> esd is part of gnome) and use.sound.esd. Now a user has "-use.gui.gnome 
> +use.sound", is esd enabled then or not?   

Personally, I think that flags being present in multiple leafs of a tree would 
be a bad thing. It adds unneeded complexity on the dev side and confusion on 
the user side. With a well designed tree, it shouldn't occur anyway. To take 
your example, use.gui.gnome.esd shouldn't exist because esd is not a gui.

-- 
Regards,
Jason Stubbs

--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] The purpose of USE flags

[Paul de Vrieze]

[-- Attachment #1: signed data --]
[-- Type: text/plain, Size: 1811 bytes --]

On Tuesday 16 December 2003 22:36, Pieter Van den Abeele wrote:

> I would use 'knowledge base' instead of database, because it is a more
> generic term. A ports directory is not really a database as we know it
> (think SQL etc.). One of the benefits of formulating the core in
> prolog, would be to use a 'datalog' syntax, datalog is regarded as a
> universal query language. SQL can be expressed in datalog, just like it
> can be expressed in relational algebra/calculus.

Well, knowledge base makes me think of logic, so settings repository might be 
even a better word. It could however also be that I had too many encounters 
with logicians.
> each concrete architecture/platform can implement its own
> 'instal/uninstall' strategy.
>
> > Ebuilds wouldn't anymore be a shell scripts, but the build module will
> > offer all necessary commands to build something. Ebuild will simply be
> > a command sequence. Same goes for the install module.
>
> This makes eclasses overriding/adding some default behaviour or
> 'meta-data' somewhat more difficult to implement.
> I'd split ebuilds into two parts: the part detailing 'howto' and the
> part detailing 'about'.

My idea is basically that there is a structure with embedded shellscripts. 
This way we don't need bash for the metadata, only for actual unpacking, 
configuring, compiling and installing

>
> I certainly wouldn't exclude using a database (like in sql databases,
> etc.), but wouldn't exclude other possibilities too. If anyone wants to
> create one huge file containing all data in his own syntax, that should
> be possible because of the modular design.

Me neither, it should be an option however.

Paul

-- 
Paul de Vrieze
Gentoo Developer
Mail: pauldv@gentoo.org
Homepage: http://www.devrieze.net

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-portage-dev] The purpose of USE flags

[Pieter Van den Abeele]

On 16 Dec 2003, at 23:23, Paul de Vrieze wrote:

> Well, knowledge base makes me think of logic, so settings repository 
> might be
> even a better word. It could however also be that I had too many 
> encounters
> with logicians.

"settings" is a bit too concrete imho: My view is that an ebuild (or a 
port file in general) defines possible settings for some system 
component(s). It is the reasoning systems task to use that knowledge to 
come up with a system consisting of components with a specification 
(=setting) that satisfies the users needs, constraints ...

'metadata repository', 'configuration knowledge' are two other 
posibilities that come to mind.

> My idea is basically that there is a structure with embedded 
> shellscripts.
> This way we don't need bash for the metadata, only for actual 
> unpacking,
> configuring, compiling and installing

indeed, good idea

>> I certainly wouldn't exclude using a database (like in sql databases,
>> etc.), but wouldn't exclude other possibilities too. If anyone wants 
>> to
>> create one huge file containing all data in his own syntax, that 
>> should
>> be possible because of the modular design.
>
> Me neither, it should be an option however.

yep

Pieter


--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] The purpose of USE flags

[Marius Mauch]

[-- Attachment #1: Type: text/plain, Size: 1176 bytes --]

On 12/17/03  Jason Stubbs wrote:

> Hmmm, I didn't get this email.
> 
> On Tuesday 16 December 2003 21:59, Marius Mauch wrote:
> > One problem I see with the tree represenations (and a flag being
> > present in multiple leafs) is an ordering probem. E.g. we have
> > use.gui.gnome.esd (as esd is part of gnome) and use.sound.esd. Now a
> > user has "-use.gui.gnome +use.sound", is esd enabled then or not?   
> 
> Personally, I think that flags being present in multiple leafs of a
> tree would be a bad thing. It adds unneeded complexity on the dev side
> and confusion on the user side. With a well designed tree, it
> shouldn't occur anyway. To take your example, use.gui.gnome.esd
> shouldn't exist because esd is not a gui.

Hmm, but AFAIK esd is considered a part of the Gnome desktop, same for
arts/KDE. So if I (as a user) disable gnome I'd expect that esd is
disabled too. Actually it's the same problem we have with categories,
things can fit in multiple branches. 

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-portage-dev] The purpose of USE flags

[Jason Stubbs]

On Wednesday 17 December 2003 09:10, Marius Mauch wrote:
> On 12/17/03  Jason Stubbs wrote:
> > Hmmm, I didn't get this email.
> >
> > On Tuesday 16 December 2003 21:59, Marius Mauch wrote:
> > > One problem I see with the tree represenations (and a flag being
> > > present in multiple leafs) is an ordering probem. E.g. we have
> > > use.gui.gnome.esd (as esd is part of gnome) and use.sound.esd. Now a
> > > user has "-use.gui.gnome +use.sound", is esd enabled then or not?
> >
> > Personally, I think that flags being present in multiple leafs of a
> > tree would be a bad thing. It adds unneeded complexity on the dev side
> > and confusion on the user side. With a well designed tree, it
> > shouldn't occur anyway. To take your example, use.gui.gnome.esd
> > shouldn't exist because esd is not a gui.
>
> Hmm, but AFAIK esd is considered a part of the Gnome desktop, same for
> arts/KDE. So if I (as a user) disable gnome I'd expect that esd is
> disabled too. Actually it's the same problem we have with categories,
> things can fit in multiple branches.

You're right. That could be confusing for the user to. It makes it more 
difficult to develop, but one possibility would be to have groupings (apart 
from the tree). For example:

gui.gtk
sound.esd
@de.gnome = (gui.gtk sound.esd)

Actually, this isn't much different to having a flag present in multiple 
leafs, except maybe easier to maintain. As for ordering, I would say that the 
last directive specified should be followed as is the case with the current 
implementation. It's not the most intuitive, but at least it's easily 
predictable. For example:

"-de.gnome +sound.esd" = "-gui.gtk +sound.esd"
"-gui.gtk -sound.esd +de.gnome" = "+gui.gtk +sound.esd"



I've just thought of another possibility with regard to USE flag defaults. 
Presently, they're specified globally and are set depending on the arch, 
right? Do they actually differ for the archs at all? I tried diffing the 
use.defaults files but not much immediate results.

Anyway, my proposal is that defaults are moved from a global setting to the 
ebuilds. Integrating with my other proposal, ebuilds would have something to 
the effect of PRODIVES, PREFERS, SUPPORTS with the following definitions:

PROVIDES - intrinsic functionality to the package. If the use has specified 
the directive "-" for this flag then the package is masked.
PREFERS - optional functionality that is usually required/expected. If this 
flag has a directive specified then it is followed. Otherwise, it is assumed 
to be "+".
SUPPORTS - optional functionality that is usually not required/expected. If 
this flag has a directive specified then it followed. Otherwise, if a package 
is installed that provides this flag then it is assumed to be "+". In all 
other cases, it is assumed to be "-".

What prompted this is packages that have a lot of options such as mplayer. 
mplayer has support for esd. If gnome PREFERS esd then, with the above rules, 
mplayer would be compiled with esd by default. Similarly, gcc could be 
compiled without java support by default where mozilla would be compiled with 
it.

I wont bother giving many examples to back it up, but this would get around 
the need for users having to specify 10s to 100s of flags. The behaviour of 
SUPPORTS when no directive is specified should probably be optional. Other 
than that, what do you think?

--
Regards,
Jason Stubbs

--
gentoo-portage-dev@gentoo.org mailing list