[gentoo-dev] RFC: storing predefined INSTALL_MASK directory lists in repos

[gentoo-dev] RFC: storing predefined INSTALL_MASK directory lists in repos

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2089 bytes --]

Hello, all.

Some time ago I wrote a small tool called install-mask. As the name
suggests, it was intended to help with manipulating INSTALL_MASK.
I think that its most important feature was ability to use pre-defined
mask lists to easily mask unwanted directories without getting into
the details.

For example, you could do:

  install-mask -a systemd

and it masked all directories related to systemd that were safe to be
masked.

The original solution had a few weak points, however, and since it was
an external tool it didn't help most of the Gentoo users. For this
reason, I wanted to implement the necessary bits in portage itself. Now
kensington reminded me of it and offered to work on it, so I'd like to
revive the topic.

What we need is an agreement where to put the 'mask lists'. That is, we
need a mapping of pre-defined keys to descriptions & lists of paths.
I have two ideas.


IDEA ONE

I'm thinking of doing something similar to USE_EXPAND. That is,
a structure like:

- a top-level install-mask.desc that keeps mapping of keys to
  descriptions,

- install-mask directory where each file name matches the install-mask
  key and file contents list all the paths.

For example:

install-mask.desc:

  bash-completion: app-shells/bash completions
  systemd: sys-apps/systemd unit files and auxiliary files

install-mask/bash-completion:

  /usr/share/bash-completion

install-mask/systemd:

  /usr/lib/systemd/system
  /usr/lib/systemd/user


IDEA TWO

A single file with ini-like syntax (alike layout.conf) that holds all
the info. Such file can be found already in
$PORTDIR/app-portage/install-mask/files/location-db.conf:

Example contents:

  [systemd]
  path=/usr/lib/systemd/system
  path=/usr/lib/systemd/user
  description=systemd unit and auxiliary files.

  [bash-completion]
  path=/usr/share/bash-completion


In both cases, the keys could be put into INSTALL_MASK in make.conf:

  INSTALL_MASK="systemd bash-completion"

What are your thoughts?

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] RFC: storing predefined INSTALL_MASK directory lists in repos

[Peter Stuge]

[-- Attachment #1: Type: text/plain, Size: 524 bytes --]

Michał Górny wrote:
>   INSTALL_MASK="systemd bash-completion"
> 
> What are your thoughts?

It seems like this will generally duplicate all -USE flags.

Would it make sense to instead have a single setting which changes
the meaning of USE to be that everything not USEd is install-masked?

Rather than adding another distinct step into the pipeline, perhaps
the trigger for doing the filtering can instead be integrated with an
existing mechanism, to optimize for low complexity and high reuse?


//Peter

[-- Attachment #2: Type: application/pgp-signature, Size: 190 bytes --]

[gentoo-dev] Re: RFC: storing predefined INSTALL_MASK directory lists in repos

[Duncan]

Peter Stuge posted on Sat, 11 Jan 2014 12:53:38 +0100 as excerpted:

> Michał Górny wrote:
>>   INSTALL_MASK="systemd bash-completion"
>> 
>> What are your thoughts?
> 
> It seems like this will generally duplicate all -USE flags.
> 
> Would it make sense to instead have a single setting which changes the
> meaning of USE to be that everything not USEd is install-masked?
> 
> Rather than adding another distinct step into the pipeline, perhaps the
> trigger for doing the filtering can instead be integrated with an
> existing mechanism, to optimize for low complexity and high reuse?

No, this would not be a duplicate.  Gentoo policy is that the mere 
installation of a few small and harmless if not used files should not be 
controlled by USE flags, as that will force an entire package rebuild 
just to get them.  So files such as systemd units and bash-completion 
triggers are always installed, since if the target isn't used anyway, 
their existence isn't going to cause any harm.  The justification is that 
few people will care more about a couple of small files than they would 
about the hassle of having to rebuild an entire package just to get them 
if they change their mind, and for those on small systems or embedded, 
where the space really /does/ matter, or for those who REALLY don't want 
them, install-mask is an existing general control mechanism fit for the 
task.

But then you get people potentially breaking their systems because they 
naively masked anything with systemd in the name, for example, including 
the upstream standard name /usr/$LIBDIR/systemd/systemd-udevd, which 
gentoo currently renames to /sbin/udevd.  Now we have to patch not only 
the udev package, but any package that calls systemd-udevd.

So the next step in automation and safety is as proposed here, provide a 
standard location for pre-created "safe" mask files that a user can then 
choose to activate, or not, as they please, very likely with an eselect 
module to follow that provides a nice gentoo-standard GUI for doing so, 
thus both exposing more browsable mask choices to the user than the user 
may otherwise be aware of, and letting the user activate them safely 
without messing with the "raw" and potentially unsafe if they don't 
really know what they're doing INSTALL_MASK settings.

Of course the raw INSTALL_MASK settings would still be there for users 
who want/need to use them.  This won't remove them and users with enough 
expertise can still mask as they always have.  This will simply give 
users that need it a less sharp and hazardous way of activating mask 
settings pre-cleared as "safe" by gentoo devs, who in turn now get the 
ability to change what's in those safe settings (but not whether the user 
has them activated) as upstream moves things around, making formerly safe 
and effective values either no longer safe, or no longer effective.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] RFC: storing predefined INSTALL_MASK directory lists in repos

[Luis Ressel]

[-- Attachment #1: Type: text/plain, Size: 337 bytes --]

I've got an additional proposal: It would be interesting if this
feature could also make use of the LINGUAS var for selectively
filtering /usr/share/man and and /usr/share/locale, as most ebuilds
don't respect this variable natively.

-- 
Luis Ressel <aranea@aixah.de>
GPG fpr: F08D 2AF6 655E 25DE 52BC  E53D 08F5 7F90 3029 B5BD

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] RFC: storing predefined INSTALL_MASK directory lists in repos

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 522 bytes --]

Dnia 2014-01-11, o godz. 16:56:37
Luis Ressel <aranea@aixah.de> napisał(a):

> I've got an additional proposal: It would be interesting if this
> feature could also make use of the LINGUAS var for selectively
> filtering /usr/share/man and and /usr/share/locale, as most ebuilds
> don't respect this variable natively.

That's a side goal.

I was thinking of something like:

  INSTALL_MASK="linguas -linguas_XX"

that would remove all linguas except for language XX.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] RFC: storing predefined INSTALL_MASK directory lists in repos

[Alan McKinnon]

On 11/01/2014 18:01, Michał Górny wrote:
> Dnia 2014-01-11, o godz. 16:56:37
> Luis Ressel <aranea@aixah.de> napisał(a):
> 
>> I've got an additional proposal: It would be interesting if this
>> feature could also make use of the LINGUAS var for selectively
>> filtering /usr/share/man and and /usr/share/locale, as most ebuilds
>> don't respect this variable natively.
> 
> That's a side goal.
> 
> I was thinking of something like:
> 
>   INSTALL_MASK="linguas -linguas_XX"
> 
> that would remove all linguas except for language XX.


That is similar to USE="no<something>" with all the attendant user
support problems it brings.

A far better method from a user point of view is to install the linguas
the user explicitly asked for. Your proposal as worded will be taken at
first glance to mean "install all linguas, but not XX" as most users
won't see the MASK portion and forget to flip the logic around in their
head.

How much work is it to get native support for LINGUAS into all ebuilds?
That would be the intuitive place considering there is already USE flags
for LINGUAS.

-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-dev] RFC: storing predefined INSTALL_MASK directory lists in repos

[Luis Ressel]

[-- Attachment #1: Type: text/plain, Size: 621 bytes --]

On Sat, 11 Jan 2014 17:01:34 +0100
Michał Górny <mgorny@gentoo.org> wrote:

> That's a side goal.
> 
> I was thinking of something like:
> 
>   INSTALL_MASK="linguas -linguas_XX"
> 
> that would remove all linguas except for language XX.

That would be enough for me. A bit of a duplication of information,
but if it eases the implementation, that shouldn't be much of a problem.

But imho it'd be nice if this approach didn't require a separate config
entry for each language (that'd be 233 entries).


-- 
Luis Ressel <aranea@aixah.de>
GPG fpr: F08D 2AF6 655E 25DE 52BC  E53D 08F5 7F90 3029 B5BD

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] RFC: storing predefined INSTALL_MASK directory lists in repos

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 937 bytes --]

Dnia 2014-01-11, o godz. 17:21:46
Luis Ressel <aranea@aixah.de> napisał(a):

> On Sat, 11 Jan 2014 17:01:34 +0100
> Michał Górny <mgorny@gentoo.org> wrote:
> 
> > That's a side goal.
> > 
> > I was thinking of something like:
> > 
> >   INSTALL_MASK="linguas -linguas_XX"
> > 
> > that would remove all linguas except for language XX.
> 
> That would be enough for me. A bit of a duplication of information,
> but if it eases the implementation, that shouldn't be much of a problem.
> 
> But imho it'd be nice if this approach didn't require a separate config
> entry for each language (that'd be 233 entries).

I agree. But I think it's cleaner to have 233 extra files than
third special case there :).

OTOH, we could just make portage use LINGUAS to create implicit
INSTALL_MASK using directory paths. That wouldn't actually require any
magic in INSTALL_MASK itself.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] RFC: storing predefined INSTALL_MASK directory lists in repos

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1291 bytes --]

Dnia 2014-01-11, o godz. 18:15:09
Alan McKinnon <alan.mckinnon@gmail.com> napisał(a):

> A far better method from a user point of view is to install the linguas
> the user explicitly asked for. Your proposal as worded will be taken at
> first glance to mean "install all linguas, but not XX" as most users
> won't see the MASK portion and forget to flip the logic around in their
> head.

As said on the other mail, I think we could just make portage
implicitly convert LINGUAS into INSTALL_MASK. That is, use the old
variable and give it a bit of new behavior.

> How much work is it to get native support for LINGUAS into all ebuilds?
> That would be the intuitive place considering there is already USE flags
> for LINGUAS.

Honestly? I'm all limbs against LINGUAS in its current form. It's just
extra dumb.

We have basically two cases:

1. packages that make LINGUAS into USE flags and use them to control
l10n. It's just useless extra work and extra rebuilds for locale
change.

2. packages that respect LINGUAS implicitly. That is, install only some
of the files silently and you don't even know which were enabled.

install-mask provides a clean framework to strip linguas with
binpackage friendliness potential.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] RFC: storing predefined INSTALL_MASK directory lists in repos

[Alan McKinnon]

On 11/01/2014 18:52, Michał Górny wrote:
> Dnia 2014-01-11, o godz. 18:15:09
> Alan McKinnon <alan.mckinnon@gmail.com> napisał(a):
> 
>> A far better method from a user point of view is to install the linguas
>> the user explicitly asked for. Your proposal as worded will be taken at
>> first glance to mean "install all linguas, but not XX" as most users
>> won't see the MASK portion and forget to flip the logic around in their
>> head.
> 
> As said on the other mail, I think we could just make portage
> implicitly convert LINGUAS into INSTALL_MASK. That is, use the old
> variable and give it a bit of new behavior.

Do you mean retain LINGUAS in make.conf and remove it from "emerge -p"
output?

I don't know much about how LINGUAS works behind the scenes, but you
seem to be proposing a scheme that works something like this:

1. User specifics what LINGUAS they want in make.conf
2. Portage magically and invisibly installs files only for that LINGUA

That seems a good approach, it unclutters emerge -p output [the amount
of clutter that causes, together with APACHE2_MODULES, CAMERAS,
PHP_MODULES etc is quite unbelievable] and gives the user what they
asked for. If you hide the negative logic in the implementation that is
double bonus

> 
>> How much work is it to get native support for LINGUAS into all ebuilds?
>> That would be the intuitive place considering there is already USE flags
>> for LINGUAS.
> 
> Honestly? I'm all limbs against LINGUAS in its current form. It's just
> extra dumb.
> 
> We have basically two cases:
> 
> 1. packages that make LINGUAS into USE flags and use them to control
> l10n. It's just useless extra work and extra rebuilds for locale
> change.
> 
> 2. packages that respect LINGUAS implicitly. That is, install only some
> of the files silently and you don't even know which were enabled.
> 
> install-mask provides a clean framework to strip linguas with
> binpackage friendliness potential.
> 


-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-dev] RFC: storing predefined INSTALL_MASK directory lists in repos

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1496 bytes --]

Dnia 2014-01-11, o godz. 19:11:21
Alan McKinnon <alan.mckinnon@gmail.com> napisał(a):

> On 11/01/2014 18:52, Michał Górny wrote:
> > Dnia 2014-01-11, o godz. 18:15:09
> > Alan McKinnon <alan.mckinnon@gmail.com> napisał(a):
> > 
> >> A far better method from a user point of view is to install the linguas
> >> the user explicitly asked for. Your proposal as worded will be taken at
> >> first glance to mean "install all linguas, but not XX" as most users
> >> won't see the MASK portion and forget to flip the logic around in their
> >> head.
> > 
> > As said on the other mail, I think we could just make portage
> > implicitly convert LINGUAS into INSTALL_MASK. That is, use the old
> > variable and give it a bit of new behavior.
> 
> Do you mean retain LINGUAS in make.conf and remove it from "emerge -p"
> output?

Yes, and no. Packages that explicitly use LINGUAS will still use it
as USE flags. However, other packages will get the stripping
implicitly.

> I don't know much about how LINGUAS works behind the scenes, but you
> seem to be proposing a scheme that works something like this:
> 
> 1. User specifics what LINGUAS they want in make.conf
> 2. Portage magically and invisibly installs files only for that LINGUA

Well, currently build system often invisibly strips LINGUAS. But this
sucks since we don't know if it did it or not afterwards. if portage
stripped them, portage will have all the details.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] Re: RFC: storing predefined INSTALL_MASK directory lists in repos

[Peter Stuge]

Duncan wrote:
> >>   INSTALL_MASK="systemd bash-completion"
> >> 
> >> What are your thoughts?
> > 
> > It seems like this will generally duplicate all -USE flags.
> > 
> > Would it make sense to instead have a single setting which changes the
> > meaning of USE to be that everything not USEd is install-masked?
> 
> No, this would not be a duplicate.

I did generalize, but think more about this - certainly for both
Michał's examples I have already either set or unset "systemd" and
"bash-completion" in USE.


> Gentoo policy is that the mere installation of a few small and
> harmless if not used files should not be controlled by USE flags

Policy doesn't matter if they doesn't make sense. Maybe I am asking
if Michał's idea could be implemented by changing that policy.

An INSTALL_MASK setting in make.conf doesn't require recompile any
less than USE setting already do.


> install-mask is an existing general control mechanism fit for the task.
..
> So the next step in automation and safety is as proposed here, provide a
> standard location for pre-created "safe" mask files that a user can then
> choose to activate
..
> a less sharp and hazardous way of activating mask settings pre-cleared
> as "safe" by gentoo devs

The mechanism wouldn't change. I think this discussion is only about
having the most expressive knobs.


//Peter

Re: [gentoo-dev] Re: RFC: storing predefined INSTALL_MASK directory lists in repos

[Donnie Berkholz]

[-- Attachment #1: Type: text/plain, Size: 1218 bytes --]

On 19:59 Sat 11 Jan     , Peter Stuge wrote:
> Duncan wrote:
> > >> Michał Górny wrote:
> > >>   INSTALL_MASK="systemd bash-completion"
> > >> 
> > >> What are your thoughts?
> > > 
> > > It seems like this will generally duplicate all -USE flags.
> > > 
> > > Would it make sense to instead have a single setting which changes 
> > > the meaning of USE to be that everything not USEd is 
> > > install-masked?
> > 
> > No, this would not be a duplicate.
> 
> I did generalize, but think more about this - certainly for both 
> Michał's examples I have already either set or unset "systemd" and 
> "bash-completion" in USE.

It would largely be a duplicate, since we've already made a decision 
about whether the ability to build without a feature is important enough 
to merit a USE flag. This is essentially rethinking the same decision 
and adding complexity to it. I think having this as an additional 
feature in the core PM would be confusing to users.

It would probably be a better fit in gentoolkit or a similar tool.

-- 
Thanks,
Donnie

Donnie Berkholz
Council Member / Sr. Developer, Gentoo Linux <http://dberkholz.com>
Analyst, RedMonk <http://redmonk.com/dberkholz/>

[-- Attachment #2: Type: application/pgp-signature, Size: 966 bytes --]