[gentoo-dev] Portage Sets

[gentoo-dev] Portage Sets

[Alec Warner]

So I have implemented merging of sets (more or less) in a local portage
branch.  However there are some use cases for which the appopriate
action is ambiguous.

Use Case #1:
Set1 = { postfix, gentoolkit, lsof, bind-utils, vixie-cron }

A set of standard tools to be on a machine.  Assume a new install (ie
none of the packages in Set1 are installed).  Is it an error if one of
the packages in the set is masked or unavailable?  In this case I would
say yes; if you instead just skip the masked package (say postfix in
this case because it's convenient ) vixie-cron will pull ssmtp instead
of postfix.

Of course this will also occur if postfix is after vixie-cron in the
set; but for our purposes we will assume the administrator put them in
this order such that postfix will get pulled in.

One could also say that the user should have used emerge -pv set1 and
noticed that ssmtp is being pulled in instead of postfix.

Use Case #2:

Set1 = cat /var/lib/portage/world (the world set)

Assume the world file has 100 packages in it, two which are masked, and
three of which there are no ebuilds in the tree for.  If missing
packages cause an error (which in use case 1 they should imho) then the
user cannot update world without unmasking things properly.  The
packages for which ebuilds are missing in the tree is in fact a portage
bug(I think), and will probably need to be fixed.

Other use cases for sets would be appreciated, as far as behavior.  It
would probably be best to provide some sort of switch.

Unmerging sets is also a fun one, I'm not sure if it's entirely
appropriate yet.
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Portage Sets

[Kevin F. Quinn]

[-- Attachment #1: Type: text/plain, Size: 4382 bytes --]

On Tue, 29 Aug 2006 07:57:43 -0400
Alec Warner <antarus@gentoo.org> wrote:

> So I have implemented merging of sets (more or less) in a local
> portage branch.

Could you elaborate how the implementation of sets would differ from:

# emerge $(cat /var/lib/portage/myset)

where /var/lib/portage/myset is a file containing a list of atoms?
That would help in thinking what the behaviour could be.

I'm thinking that perhaps you do everything up to but not including
qmerge for all packages then do the qmerge phase for the set in one go,
provided they all got to install ok.  It might be useful to try to move
all actions that might fail during qmerge to the end of the install
phase (I'm thinking things like collision-detection, qa checks), so
that the only reason qmerge on the set would fail is lack of disk space.

Obviously that's simplifying somewhat - if there are build-time DEPEND
relationships between elements of the set, the set has to be broken
down into sub-sets that don't have such internal dependencies.  However
I can't think of much else you would do with sets that the $(cat
<file>) approach doesn't already supply.  Alternatively you could
require that sets must not have such internal dependencies.

>  However there are some use cases for which the
> appopriate action is ambiguous.
> 
> Use Case #1:
> Set1 = { postfix, gentoolkit, lsof, bind-utils, vixie-cron }
> 
> A set of standard tools to be on a machine.  Assume a new install (ie
> none of the packages in Set1 are installed).  Is it an error if one of
> the packages in the set is masked or unavailable?  In this case I
> would say yes; if you instead just skip the masked package (say
> postfix in this case because it's convenient ) vixie-cron will pull
> ssmtp instead of postfix.
> 
> Of course this will also occur if postfix is after vixie-cron in the
> set; but for our purposes we will assume the administrator put them in
> this order such that postfix will get pulled in.
> 
> One could also say that the user should have used emerge -pv set1 and
> noticed that ssmtp is being pulled in instead of postfix.
> 
> Use Case #2:
> 
> Set1 = cat /var/lib/portage/world (the world set)
> 
> Assume the world file has 100 packages in it, two which are masked,
> and three of which there are no ebuilds in the tree for.  If missing
> packages cause an error (which in use case 1 they should imho) then
> the user cannot update world without unmasking things properly.  The
> packages for which ebuilds are missing in the tree is in fact a
> portage bug(I think), and will probably need to be fixed.

For the initial merge case then an error before anything is merged is
good.  For an upgrade merge, a warning would be more appropriate
(perhaps becoming an error with FEATURES=stricter or similar).

> Other use cases for sets would be appreciated, as far as behavior.  It
> would probably be best to provide some sort of switch.

The most obvious use is to have a related group of packages that may
otherwise include a virtual, making the choice of that virtual explicit
(like your cron example above).

Other sets would be simply groups of packages that make functional
sense together, where perhaps one package might make little sense
without others in the set.  For example:

sylpheed-kev = ( sylpheed-claws sylpheed-claws-mailmbox \
	sylpheed-claws-smime sylpheed-claws-rssyl sylpheed-claws-smime \
	sylpheed-claws-vcalendar)

toolchain4 = ( \
	~sys-devel/gcc-4.1.1 \
	~sys-devel/binutils-2.16.1 \
	~sys-libs/glibc-2.4 )

Are you considering allowing sets to contain other sets?  Then world
would contain the names of sets, not just CPs.

> Unmerging sets is also a fun one, I'm not sure if it's entirely
> appropriate yet.

Would it do anything more than:

# emerge -C $(cat /var/lib/portage/myset)

Perhaps it would unmerge any packages merged as dependencies of the
set that are no longer dependencies of anything else currently
installed - but I think that's better handled separately.

Ah; it occurs to me that unmerging a set should only unmerge elements
of the set that are not part of any other installed set (including
world).  So behaviour is less like 'emerge $(cat <set>)' and more like
emerge sets/<set> where <sets>/set/set-V.ebuild is like a meta-ebuild.

-- 
Kevin F. Quinn

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Portage Sets

[Chris White]

[-- Attachment #1: Type: text/plain, Size: 460 bytes --]

On Tuesday 29 August 2006 07:22, Kevin F. Quinn wrote:
> # emerge $(cat /var/lib/portage/myset)

y'all are killing me

emerge $(< /var/lib/portage/myset )

</pet_peeve> :p

-- 
Chris White
Gentoo Developer aka:
ChrisWhite
cpw
ChrisWhite|Work
WhiteChocolate
VanillaWhite
Whitey
WhiteLight
WhiteCheese
WhiteSugar
WhiteButter
WhiteWall
WhiteLemon
WhiteApple
WhiteBlanket
WhiteEnergy
WhiteWhite
<cab> go shorten your sig ChrisWhite ;)

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Portage Sets

[Simon Stelling]

Chris White wrote:
> y'all are killing me

So are you:

> --
> Chris White
> Gentoo Developer aka:
> ChrisWhite
> cpw
> ChrisWhite|Work
> WhiteChocolate
> VanillaWhite
> Whitey
> WhiteLight
> WhiteCheese
> WhiteSugar
> WhiteButter
> WhiteWall
> WhiteLemon
> WhiteApple
> WhiteBlanket
> WhiteEnergy
> WhiteWhite
> <cab> go shorten your sig ChrisWhite

head -n4 $(<~/.sig) > ~/.sig

-- 
Kind Regards,

Simon Stelling
Gentoo/AMD64 developer
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Portage Sets

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 221 bytes --]

On Tuesday 29 August 2006 10:44, Simon Stelling wrote:
> head -n4 $(<~/.sig) > ~/.sig

seriously, when did this turn into the forums ?  well at least he doesnt have 
a 640x480 animated gif in there (yet?) ...
-mike

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] Portage Sets

[Wernfried Haas]

[-- Attachment #1: Type: text/plain, Size: 493 bytes --]

On Tue, Aug 29, 2006 at 01:41:26PM -0400, Mike Frysinger wrote:
> seriously, when did this turn into the forums ?  well at least he doesnt have 
> a 640x480 animated gif in there (yet?) ...

Actually there's an 80x80 px limit for avatars and the possibility to
turn them off via your profile. ;-)

cheers,
	Wernfried

-- 
Wernfried Haas (amne) - amne at gentoo dot org
Gentoo Forums: http://forums.gentoo.org
IRC: #gentoo-forums on freenode - email: forum-mods at gentoo dot org

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Portage Sets

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 391 bytes --]

On Tuesday 29 August 2006 14:22, Wernfried Haas wrote:
> On Tue, Aug 29, 2006 at 01:41:26PM -0400, Mike Frysinger wrote:
> > seriously, when did this turn into the forums ?  well at least he doesnt
> > have a 640x480 animated gif in there (yet?) ...
>
> Actually there's an 80x80 px limit for avatars and the possibility to
> turn them off via your profile. ;-)

avatars != signatures
-mike

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] Portage Sets

[Alec Warner]

Kevin F. Quinn wrote:
> On Tue, 29 Aug 2006 07:57:43 -0400
> Alec Warner <antarus@gentoo.org> wrote:
> 
>> So I have implemented merging of sets (more or less) in a local
>> portage branch.
> 
> Could you elaborate how the implementation of sets would differ from:
> 
> # emerge $(cat /var/lib/portage/myset)
> 
> where /var/lib/portage/myset is a file containing a list of atoms?
> That would help in thinking what the behaviour could be.
> 
> I'm thinking that perhaps you do everything up to but not including
> qmerge for all packages then do the qmerge phase for the set in one go,
> provided they all got to install ok.  It might be useful to try to move
> all actions that might fail during qmerge to the end of the install
> phase (I'm thinking things like collision-detection, qa checks), so
> that the only reason qmerge on the set would fail is lack of disk space.
> 

one word, no ;)  This is designed (for now) to be rather non-intrusive.
 Here you are talking about major changes with code and behavior.  At
this point you could implement all of that for any number of packages
(not just sets).

> Obviously that's simplifying somewhat - if there are build-time DEPEND
> relationships between elements of the set, the set has to be broken
> down into sub-sets that don't have such internal dependencies.  However
> I can't think of much else you would do with sets that the $(cat
> <file>) approach doesn't already supply.  Alternatively you could
> require that sets must not have such internal dependencies.

Dynamic sets (World,All,GLSA,System), these are not 'static' in the
sense of normal files.

> 
>>  However there are some use cases for which the
>> appopriate action is ambiguous.
>>
>> Use Case #1:
>> Set1 = { postfix, gentoolkit, lsof, bind-utils, vixie-cron }
>>
>> A set of standard tools to be on a machine.  Assume a new install (ie
>> none of the packages in Set1 are installed).  Is it an error if one of
>> the packages in the set is masked or unavailable?  In this case I
>> would say yes; if you instead just skip the masked package (say
>> postfix in this case because it's convenient ) vixie-cron will pull
>> ssmtp instead of postfix.
>>
>> Of course this will also occur if postfix is after vixie-cron in the
>> set; but for our purposes we will assume the administrator put them in
>> this order such that postfix will get pulled in.
>>
>> One could also say that the user should have used emerge -pv set1 and
>> noticed that ssmtp is being pulled in instead of postfix.
>>
>> Use Case #2:
>>
>> Set1 = cat /var/lib/portage/world (the world set)
>>
>> Assume the world file has 100 packages in it, two which are masked,
>> and three of which there are no ebuilds in the tree for.  If missing
>> packages cause an error (which in use case 1 they should imho) then
>> the user cannot update world without unmasking things properly.  The
>> packages for which ebuilds are missing in the tree is in fact a
>> portage bug(I think), and will probably need to be fixed.
> 
> For the initial merge case then an error before anything is merged is
> good.  For an upgrade merge, a warning would be more appropriate
> (perhaps becoming an error with FEATURES=stricter or similar).
> 
>> Other use cases for sets would be appreciated, as far as behavior.  It
>> would probably be best to provide some sort of switch.
> 
> The most obvious use is to have a related group of packages that may
> otherwise include a virtual, making the choice of that virtual explicit
> (like your cron example above).
> 
> Other sets would be simply groups of packages that make functional
> sense together, where perhaps one package might make little sense
> without others in the set.  For example:
> 
> sylpheed-kev = ( sylpheed-claws sylpheed-claws-mailmbox \
> 	sylpheed-claws-smime sylpheed-claws-rssyl sylpheed-claws-smime \
> 	sylpheed-claws-vcalendar)
> 
> toolchain4 = ( \
> 	~sys-devel/gcc-4.1.1 \
> 	~sys-devel/binutils-2.16.1 \
> 	~sys-libs/glibc-2.4 )
> 
> Are you considering allowing sets to contain other sets?  Then world
> would contain the names of sets, not just CPs.

At this point, no.

> 
>> Unmerging sets is also a fun one, I'm not sure if it's entirely
>> appropriate yet.
> 
> Would it do anything more than:
> 
> # emerge -C $(cat /var/lib/portage/myset)
> 
> Perhaps it would unmerge any packages merged as dependencies of the
> set that are no longer dependencies of anything else currently
> installed - but I think that's better handled separately.

I don't want to differ from normal portage behavior in this regard.
When portage can do this, sets should as well, if portage doesn't for
normal packages, I don't want to special case sets.

> 
> Ah; it occurs to me that unmerging a set should only unmerge elements
> of the set that are not part of any other installed set (including
> world).  So behaviour is less like 'emerge $(cat <set>)' and more like
> emerge sets/<set> where <sets>/set/set-V.ebuild is like a meta-ebuild.
> 

This is an interseting idea.  But then not all sets are equal; world,
system, and the "All" set change fairly often. Plus everything installed
is in "All" so you would want to ignore that set, as well as probably
the world set.
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Portage Sets

[Chris White]

[-- Attachment #1: Type: text/plain, Size: 736 bytes --]

On Tuesday 29 August 2006 04:57, Alec Warner wrote:
> So I have implemented merging of sets (more or less) in a local portage
> branch.  However there are some use cases for which the appopriate
> action is ambiguous.
>
> Use Case #1:
> Set1 = { postfix, gentoolkit, lsof, bind-utils, vixie-cron }

Where exactly would these sets be defined?  Reading through it doesn't seem to 
be all that apparent.

> head -n4 $(<~/.sig) > ~/.sig

$ head -n $(<~/.sig) > ~/.sig
Segmentation Fault

-- 
Chris White
Gentoo Developer aka:
ChrisWhite		cpw
ChrisWhite|Work	WhiteChocolate
VanillaWhite		Whitey
WhiteLight		WhiteCheese
WhiteSugar		WhiteButter
WhiteWall		WhiteLemon
WhiteApple		WhiteBlanket
WhiteEnergy		WhiteWhite

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Portage Sets

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 980 bytes --]

On Tue, 29 Aug 2006 07:57:43 -0400 Alec Warner <antarus@gentoo.org>
wrote:
| So I have implemented merging of sets (more or less) in a local
| portage branch.

Aah, good to see that you're picking up one of the more useful Paludis
features.

For those who aren't aware of how Paludis handles this... File sets are
a text file that live in repodir/sets/ or confdir/sets/ and are named
setname.conf. Lines can be in the form *atom , which means "include
atom in the set", or ?atom , which means "include atom in the set only
if the package represented by atom is already installed".

The big difference between paludis --install setname and paludis
--install $(< setfile), aside from the *? support, is that the former
won't reinstall packages that don't need upgrading, and the latter
will. It's the same as emerge world vs emerge $(< worldfile), the
latter being similar to emerge -e world...

-- 
Ciaran McCreesh
Mail            : ciaranm at ciaranm.org


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Portage Sets

[Ciaran McCreesh]

On Tue, 29 Aug 2006 18:56:05 +0200 Simon Stelling <blubb@gentoo.org>
wrote:
| Ciaran McCreesh wrote:
| > For those who aren't aware of how Paludis handles this... File sets
| > are a text file that live in repodir/sets/ or confdir/sets/ and are
| > named setname.conf. Lines can be in the form *atom , which means
| > "include atom in the set", or ?atom , which means "include atom in
| > the set only if the package represented by atom is already
| > installed".
| 
| That paragraph sounds like if it came from Duncan. No offense to
| anyone, just couldn't resist *g*.

Naah, it was only one paragraph.

| > The big difference between paludis --install setname and paludis
| > --install $(< setfile), aside from the *? support, is that the
| > former won't reinstall packages that don't need upgrading, and the
| > latter will. It's the same as emerge world vs emerge $(<
| > worldfile), the latter being similar to emerge -e world...
| 
| So it's the same as 'emerge -n $(< worldfile)'?

It behaves like emerge -n works, yes (and not the way emerge --help
says that -n works, which implies that emerge -n foo won't upgrade foo
if foo is already installed at a lower version).

-- 
Ciaran McCreesh
Mail            : ciaranm at ciaranm.org

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Portage Sets

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 244 bytes --]

On Tuesday 29 August 2006 11:25, Ciaran McCreesh wrote:
> Aah, good to see that you're picking up one of the more useful Paludis
> features.

the idea has been around before Paludis, it just implemented custom sets 
before portage
-mike

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]