[gentoo-dev] [RFC] Masterplan for solving LINGUAS problems

[gentoo-dev] [RFC] Masterplan for solving LINGUAS problems

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2416 bytes --]

Hello, everyone.

Since the previous thread doesn't seem to have brought any good
solution to the problem other than stopping to (ab)use LINGUAS
as USE_EXPAND, I would like to start a RFC on a draft solution that
I'd like afterwards to propose to the Council.


Rationale
---------

The direct reason for this is that LINGUAS is treated as non-standard
special variable by multiple build systems. This includes the following
problems:

1. no localizations are installed if it is set to an empty value (which
happens in EAPI 5 when the ebuild does not use the flags),

2. there were historical cases where order of LINGUAS mattered.

Those problems can't be reasonably solved within the scope of
USE_EXPAND. Furthermore, the use of flags to control localizations is
causing the following problems:

a. maintaining correct flag list is a serious maintenance burden,
especially that differences in build systems make it hard to figure out
the 'most correct' set automatically,

b. missing flags result in localizations being silently dropped, with
no clear way (i.e. for QA check) to detect that,

c. large number of additional USE flags make it pretty much impossible
to limit localizations this way when using binary packages.


The plan
--------

1. Get approval on INSTALL_MASK GLEP [1] and finish implementing it
in Portage.

2. Introduce a new USE_EXPAND that can be used to control localizations
whenever this is really required (dependencies, large files, etc.).
Let's use L10N as a draft name for it.

3. Fix all packages using LINGUAS as USE_EXPAND, either by converting
to L10N or by removing the needless flags.

4. Remove LINGUAS from USE_EXPAND, therefore removing the special EAPI
rules from the variable.

5. Release a news item explaining the users the change,
and the necessary action. Request changing LINGUAS to L10N
in make.conf, and make LINGUAS considered an 'advanced variable' for
implicit localization control (i.e. passed through to build systems).
Recommend clean INSTALL_MASK solution instead.

The example 'new' make.conf would probably look like:

  # controlling e.g. langpacks
  L10N="en_US pl"
  # stripping unneeded files
  INSTALL_MASK="@linguas -@linguas_pl"


Your thoughts?


[1]:https://wiki.gentoo.org/wiki/User:MGorny/GLEP:INSTALL_MASK

-- 
Best regards,
Michał Górny
<http://dev.gentoo.org/~mgorny/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] [RFC] Masterplan for solving LINGUAS problems

[Chí-Thanh Christopher Nguyễn]

Michał Górny schrieb:
> Request changing LINGUAS to L10N
> in make.conf, and make LINGUAS considered an 'advanced variable' for
> implicit localization control (i.e. passed through to build systems).
> Recommend clean INSTALL_MASK solution instead.

Can we have this change happen (semi-)automatically, via config_protect?
Also, will there be a transition period where users need to have both 
variables set?


Best regards,
Chí-Thanh Christopher Nguyễn

Re: [gentoo-dev] [RFC] Masterplan for solving LINGUAS problems

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 943 bytes --]

On Tue, 31 May 2016 14:53:36 +0200
Chí-Thanh Christopher Nguyễn <chithanh@gentoo.org> wrote:

> Michał Górny schrieb:
> > Request changing LINGUAS to L10N
> > in make.conf, and make LINGUAS considered an 'advanced variable' for
> > implicit localization control (i.e. passed through to build systems).
> > Recommend clean INSTALL_MASK solution instead.  
> 
> Can we have this change happen (semi-)automatically, via config_protect?

I don't know. We'd have to bind it to some package then.

Maybe we should bind it to sys-apps/portage, along with the USE_EXPAND
behavior fix. Then users wouldn't directly be affected by LINGUAS
removing localizations from all packages.

> Also, will there be a transition period where users need to have both 
> variables set?

I'd rather avoid that, since that prolongs the problem with LINGUAS
misbehavior.

-- 
Best regards,
Michał Górny
<http://dev.gentoo.org/~mgorny/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] [RFC] Masterplan for solving LINGUAS problems

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 3749 bytes --]

Hi,

On Tue, 31 May 2016 14:49:26 +0200 Michał Górny wrote:
> Hello, everyone.
> 
> Since the previous thread doesn't seem to have brought any good
> solution to the problem other than stopping to (ab)use LINGUAS
> as USE_EXPAND, I would like to start a RFC on a draft solution that
> I'd like afterwards to propose to the Council.
> 
> 
> Rationale
> ---------
> 
> The direct reason for this is that LINGUAS is treated as non-standard
> special variable by multiple build systems. This includes the following
> problems:
> 
> 1. no localizations are installed if it is set to an empty value (which
> happens in EAPI 5 when the ebuild does not use the flags),
> 
> 2. there were historical cases where order of LINGUAS mattered.
> 
> Those problems can't be reasonably solved within the scope of
> USE_EXPAND. Furthermore, the use of flags to control localizations is
> causing the following problems:
> 
> a. maintaining correct flag list is a serious maintenance burden,
> especially that differences in build systems make it hard to figure out
> the 'most correct' set automatically,
> 
> b. missing flags result in localizations being silently dropped, with
> no clear way (i.e. for QA check) to detect that,
> 
> c. large number of additional USE flags make it pretty much impossible
> to limit localizations this way when using binary packages.
> 
> 
> The plan
> --------
> 
> 1. Get approval on INSTALL_MASK GLEP [1] and finish implementing it
> in Portage.
> 
> 2. Introduce a new USE_EXPAND that can be used to control localizations
> whenever this is really required (dependencies, large files, etc.).
> Let's use L10N as a draft name for it.
> 
> 3. Fix all packages using LINGUAS as USE_EXPAND, either by converting
> to L10N or by removing the needless flags.
> 
> 4. Remove LINGUAS from USE_EXPAND, therefore removing the special EAPI
> rules from the variable.
> 
> 5. Release a news item explaining the users the change,
> and the necessary action. Request changing LINGUAS to L10N
> in make.conf, and make LINGUAS considered an 'advanced variable' for
> implicit localization control (i.e. passed through to build systems).
> Recommend clean INSTALL_MASK solution instead.
> 
> The example 'new' make.conf would probably look like:
> 
>   # controlling e.g. langpacks
>   L10N="en_US pl"
>   # stripping unneeded files
>   INSTALL_MASK="@linguas -@linguas_pl"

Users will have extra burden because of this change. We should make
user experience simpler, not more and more difficult. With proposed
change users will need to manage two lists for the same
functionality: L10N and INSTALL_MASK.

The latter one is being used for many purposes and people may be
confused by requirement to put @linguas-like directives there. It's
like we'll force to put all USE_EXPAND flags explicitly in the USE
variable.

What about different locales for the same language? E.g. there is
ru, ru_RU and ru_RU_0 for Russian (I have no clue what ru_RU_0 is
supposed to mean). What are planes for this varians? Will
@lingual_ru contain all of them or only some?

What about man pages? They are also controlled by LINGUAS (at
least for some packages), but will be out of control with proposed
changes.

If we are going to do an instant move from LINGUAS to L10N (no
graced period as discussed here on another thread), why to rename
variable in the first place?

It will be a nice move towards our users if INSTALL_MASK will be
automatically populated based on L10N list with ability to override
it if really needed, e.g. user needs L10N for en only (e.g. large
docs), but wants some other languages for localization files.

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] [RFC] Masterplan for solving LINGUAS problems

[Daniel Campbell]

[-- Attachment #1.1: Type: text/plain, Size: 3512 bytes --]

On 05/31/2016 05:49 AM, Michał Górny wrote:
> Hello, everyone.
> 
> Since the previous thread doesn't seem to have brought any good
> solution to the problem other than stopping to (ab)use LINGUAS
> as USE_EXPAND, I would like to start a RFC on a draft solution that
> I'd like afterwards to propose to the Council.
> 
> 
> Rationale
> ---------
> 
> The direct reason for this is that LINGUAS is treated as non-standard
> special variable by multiple build systems. This includes the following
> problems:
> 
> 1. no localizations are installed if it is set to an empty value (which
> happens in EAPI 5 when the ebuild does not use the flags),
> 
> 2. there were historical cases where order of LINGUAS mattered.
> 
> Those problems can't be reasonably solved within the scope of
> USE_EXPAND. Furthermore, the use of flags to control localizations is
> causing the following problems:
> 
> a. maintaining correct flag list is a serious maintenance burden,
> especially that differences in build systems make it hard to figure out
> the 'most correct' set automatically,
> 
> b. missing flags result in localizations being silently dropped, with
> no clear way (i.e. for QA check) to detect that,
> 
> c. large number of additional USE flags make it pretty much impossible
> to limit localizations this way when using binary packages.
> 
> 
> The plan
> --------
> 
> 1. Get approval on INSTALL_MASK GLEP [1] and finish implementing it
> in Portage.
> 
> 2. Introduce a new USE_EXPAND that can be used to control localizations
> whenever this is really required (dependencies, large files, etc.).
> Let's use L10N as a draft name for it.
> 
> 3. Fix all packages using LINGUAS as USE_EXPAND, either by converting
> to L10N or by removing the needless flags.
> 
> 4. Remove LINGUAS from USE_EXPAND, therefore removing the special EAPI
> rules from the variable.
> 
> 5. Release a news item explaining the users the change,
> and the necessary action. Request changing LINGUAS to L10N
> in make.conf, and make LINGUAS considered an 'advanced variable' for
> implicit localization control (i.e. passed through to build systems).
> Recommend clean INSTALL_MASK solution instead.
> 
> The example 'new' make.conf would probably look like:
> 
>   # controlling e.g. langpacks
>   L10N="en_US pl"
>   # stripping unneeded files
>   INSTALL_MASK="@linguas -@linguas_pl"
> 
> 
> Your thoughts?
> 
> 
> [1]:https://wiki.gentoo.org/wiki/User:MGorny/GLEP:INSTALL_MASK
> 

I think this idea has some potential, but would there be a way for a
user to choose L10N *or* INSTALL_MASK instead of both? If I understand
correctly, a person who wanted all of their system to be en_US only, but
wanted to take part in translation of some other project, would need to
add the other locales directly to L10N, then somehow mask them out for
other packages. Or the reverse: leave L10N="en_US" or something, and
somehow enable other languages in that specific package.

Is there a package-level option for this? Users can set their locales in
/etc/locale.gen, and that handles things globally, but what about the
user that doesn't want to include that for all of their packages? This
seems like an all-or-nothing thing, lacking in granularity. If I'm
wrong, please clarify so I can understand better.

-- 
Daniel Campbell - Gentoo Developer
OpenPGP Key: 0x1EA055D6 @ hkp://keys.gnupg.net
fpr: AE03 9064 AE00 053C 270C  1DE4 6F7A 9091 1EA0 55D6


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] [RFC] Masterplan for solving LINGUAS problems

[Daniel Campbell]

[-- Attachment #1.1: Type: text/plain, Size: 3990 bytes --]

On 05/31/2016 10:31 AM, Daniel Campbell wrote:
> On 05/31/2016 05:49 AM, Michał Górny wrote:
>> Hello, everyone.
>>
>> Since the previous thread doesn't seem to have brought any good
>> solution to the problem other than stopping to (ab)use LINGUAS
>> as USE_EXPAND, I would like to start a RFC on a draft solution that
>> I'd like afterwards to propose to the Council.
>>
>>
>> Rationale
>> ---------
>>
>> The direct reason for this is that LINGUAS is treated as non-standard
>> special variable by multiple build systems. This includes the following
>> problems:
>>
>> 1. no localizations are installed if it is set to an empty value (which
>> happens in EAPI 5 when the ebuild does not use the flags),
>>
>> 2. there were historical cases where order of LINGUAS mattered.
>>
>> Those problems can't be reasonably solved within the scope of
>> USE_EXPAND. Furthermore, the use of flags to control localizations is
>> causing the following problems:
>>
>> a. maintaining correct flag list is a serious maintenance burden,
>> especially that differences in build systems make it hard to figure out
>> the 'most correct' set automatically,
>>
>> b. missing flags result in localizations being silently dropped, with
>> no clear way (i.e. for QA check) to detect that,
>>
>> c. large number of additional USE flags make it pretty much impossible
>> to limit localizations this way when using binary packages.
>>
>>
>> The plan
>> --------
>>
>> 1. Get approval on INSTALL_MASK GLEP [1] and finish implementing it
>> in Portage.
>>
>> 2. Introduce a new USE_EXPAND that can be used to control localizations
>> whenever this is really required (dependencies, large files, etc.).
>> Let's use L10N as a draft name for it.
>>
>> 3. Fix all packages using LINGUAS as USE_EXPAND, either by converting
>> to L10N or by removing the needless flags.
>>
>> 4. Remove LINGUAS from USE_EXPAND, therefore removing the special EAPI
>> rules from the variable.
>>
>> 5. Release a news item explaining the users the change,
>> and the necessary action. Request changing LINGUAS to L10N
>> in make.conf, and make LINGUAS considered an 'advanced variable' for
>> implicit localization control (i.e. passed through to build systems).
>> Recommend clean INSTALL_MASK solution instead.
>>
>> The example 'new' make.conf would probably look like:
>>
>>   # controlling e.g. langpacks
>>   L10N="en_US pl"
>>   # stripping unneeded files
>>   INSTALL_MASK="@linguas -@linguas_pl"
>>
>>
>> Your thoughts?
>>
>>
>> [1]:https://wiki.gentoo.org/wiki/User:MGorny/GLEP:INSTALL_MASK
>>
> 
> I think this idea has some potential, but would there be a way for a
> user to choose L10N *or* INSTALL_MASK instead of both? If I understand
> correctly, a person who wanted all of their system to be en_US only, but
> wanted to take part in translation of some other project, would need to
> add the other locales directly to L10N, then somehow mask them out for
> other packages. Or the reverse: leave L10N="en_US" or something, and
> somehow enable other languages in that specific package.
> 
> Is there a package-level option for this? Users can set their locales in
> /etc/locale.gen, and that handles things globally, but what about the
> user that doesn't want to include that for all of their packages? This
> seems like an all-or-nothing thing, lacking in granularity. If I'm
> wrong, please clarify so I can understand better.
> 

I forgot to include that I think the INSTALL_MASK groups, even if not
implemented for this issue, are a great idea. It would allow users to
target specific things like "get rid of info pages", "no systemd unit
files", etc, in a way that is controlled by the repo (or an override in
/etc/portage somewhere). It prevents more ebuild bloat, too.

-- 
Daniel Campbell - Gentoo Developer
OpenPGP Key: 0x1EA055D6 @ hkp://keys.gnupg.net
fpr: AE03 9064 AE00 053C 270C  1DE4 6F7A 9091 1EA0 55D6


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[gentoo-dev] Re: [RFC] Masterplan for solving LINGUAS problems

[Jörg Schaible]

How can I select different linguas for individual packages with this 
approach?

Michał Górny wrote:

> Hello, everyone.
> 
> Since the previous thread doesn't seem to have brought any good
> solution to the problem other than stopping to (ab)use LINGUAS
> as USE_EXPAND, I would like to start a RFC on a draft solution that
> I'd like afterwards to propose to the Council.
> 
> 
> Rationale
> ---------
> 
> The direct reason for this is that LINGUAS is treated as non-standard
> special variable by multiple build systems. This includes the following
> problems:
> 
> 1. no localizations are installed if it is set to an empty value (which
> happens in EAPI 5 when the ebuild does not use the flags),
> 
> 2. there were historical cases where order of LINGUAS mattered.
> 
> Those problems can't be reasonably solved within the scope of
> USE_EXPAND. Furthermore, the use of flags to control localizations is
> causing the following problems:
> 
> a. maintaining correct flag list is a serious maintenance burden,
> especially that differences in build systems make it hard to figure out
> the 'most correct' set automatically,
> 
> b. missing flags result in localizations being silently dropped, with
> no clear way (i.e. for QA check) to detect that,
> 
> c. large number of additional USE flags make it pretty much impossible
> to limit localizations this way when using binary packages.
> 
> 
> The plan
> --------
> 
> 1. Get approval on INSTALL_MASK GLEP [1] and finish implementing it
> in Portage.
> 
> 2. Introduce a new USE_EXPAND that can be used to control localizations
> whenever this is really required (dependencies, large files, etc.).
> Let's use L10N as a draft name for it.
> 
> 3. Fix all packages using LINGUAS as USE_EXPAND, either by converting
> to L10N or by removing the needless flags.
> 
> 4. Remove LINGUAS from USE_EXPAND, therefore removing the special EAPI
> rules from the variable.
> 
> 5. Release a news item explaining the users the change,
> and the necessary action. Request changing LINGUAS to L10N
> in make.conf, and make LINGUAS considered an 'advanced variable' for
> implicit localization control (i.e. passed through to build systems).
> Recommend clean INSTALL_MASK solution instead.
> 
> The example 'new' make.conf would probably look like:
> 
>   # controlling e.g. langpacks
>   L10N="en_US pl"
>   # stripping unneeded files
>   INSTALL_MASK="@linguas -@linguas_pl"
> 
> 
> Your thoughts?
> 
> 
> [1]:https://wiki.gentoo.org/wiki/User:MGorny/GLEP:INSTALL_MASK
> 

Re: [gentoo-dev] Re: [RFC] Masterplan for solving LINGUAS problems

[James Le Cuirot]

[-- Attachment #1: Type: text/plain, Size: 306 bytes --]

On Tue, 31 May 2016 23:34:07 +0200
Jörg Schaible <joerg.schaible@gmx.de> wrote:

> How can I select different linguas for individual packages with this 
> approach?

You could probably set INSTALL_MASK and L10N
via /etc/portage/package.env.

-- 
James Le Cuirot (chewi)
Gentoo Linux Developer

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 951 bytes --]

Re: [gentoo-dev] Re: [RFC] Masterplan for solving LINGUAS problems

[Mike Gilbert]

On Tue, May 31, 2016 at 5:34 PM, Jörg Schaible <joerg.schaible@gmx.de> wrote:
> How can I select different linguas for individual packages with this
> approach?

Why would you want to?

[gentoo-dev] Re: Re: [RFC] Masterplan for solving LINGUAS problems

[Jörg Schaible]

Mike Gilbert wrote:

> On Tue, May 31, 2016 at 5:34 PM, Jörg Schaible <joerg.schaible@gmx.de>
> wrote:
>> How can I select different linguas for individual packages with this
>> approach?
> 
> Why would you want to?

As programmer I am used to read English manuals and for most command line 
tools it is the only up-to-date documentation. Development tools or system 
configuration with translated manuals is simply awful for me. Therefore I've 
set LINGUAS to "en" in make.conf and run my shells with "LANG=en_US.UTF-8".

Situation changes for GUI stuff like office applications where I feel more 
comfortable as "normal" user. KDE is configured for German language and for 
a lot of those apps I've explicitly set linguas_de in package.use. For 
libreoffice and calligra I've even added linguas_fr to get French spelling 
support.

For me it is currently very convenient to look simply at the use flags of a 
package to see which languages will be supported. Gnucash is the only 
application I had to set environment variables for (using package.env) to 
get German language support (and it took me a while to find this and get it 
right). I may have missed other packages that do not use IUSE_EXPAND, simply 
because I am not aware of the support.

Cheers,
Jörg

Re: [gentoo-dev] Re: [RFC] Masterplan for solving LINGUAS problems

[Francesco Riosa]

[-- Attachment #1: Type: text/plain, Size: 402 bytes --]

2016-06-01 0:03 GMT+02:00 Mike Gilbert <floppym@gentoo.org>:

> On Tue, May 31, 2016 at 5:34 PM, Jörg Schaible <joerg.schaible@gmx.de>
> wrote:
> > How can I select different linguas for individual packages with this
> > approach?
>
> Why would you want to?
>
> In addition to  Jörg Schaible answer:
I do use my kde sessions with en_US language, my daughter prefer to use
it_IT, same PC.

[-- Attachment #2: Type: text/html, Size: 855 bytes --]

[gentoo-dev] Re: [RFC] Masterplan for solving LINGUAS problems

[Duncan]

Andrew Savchenko posted on Tue, 31 May 2016 20:08:10 +0300 as excerpted:

> If we are going to do an instant move from LINGUAS to L10N (no graced
> period as discussed here on another thread), why to rename variable in
> the first place?

Because the LINGUAS var is hopelessly tangled up in various build 
systems... in different ways for different build systems so it's nigh 
impossible to automatically untangle it.

By switching to a var (L10N being the proposed name) that's not actually 
used in strange ways by the build systems, gentoo's settings won't have 
unexpected implicit results due to getting tangled up with the build 
system, only the explicit results managed by the gentoo-level code in the 
ebuilds/eclasses.

That of course being the whole problem LINGUAS turned into such a mess in 
the first place.  In hindsight the mess could have been predicted, but as 
they say, hindsight is 20/20...

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

[gentoo-dev] Re: [RFC] Masterplan for solving LINGUAS problems

[Duncan]

Daniel Campbell posted on Tue, 31 May 2016 10:31:22 -0700 as excerpted:

> I think this idea has some potential, but would there be a way for a
> user to choose L10N *or* INSTALL_MASK instead of both? If I understand
> correctly, a person who wanted all of their system to be en_US only, but
> wanted to take part in translation of some other project, would need to
> add the other locales directly to L10N, then somehow mask them out for
> other packages. Or the reverse: leave L10N="en_US" or something, and
> somehow enable other languages in that specific package.
> 
> Is there a package-level option for this? Users can set their locales in
> /etc/locale.gen, and that handles things globally, but what about the
> user that doesn't want to include that for all of their packages? This
> seems like an all-or-nothing thing, lacking in granularity. If I'm
> wrong, please clarify so I can understand better.

For portage at least, pretty much all vars, including the proposed L10N, 
can be set per-package using /etc/portage/package.env and the files it 
points to in /etc/portage/env/.  

(The package.env method works during the python phase as well, but is 
primarily for direct variable setting and isn't as flexible as the older 
/etc/portage/env/cat-egory/pkg/file method, which only works during the 
bash/ebuild phase, but exposes the full richness of bash to be used as 
it's actually bash executing it.)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: [RFC] Masterplan for solving LINGUAS problems

[Michał Górny]

Dnia 31 maja 2016 23:34:07 CEST, "Jörg Schaible" <joerg.schaible@gmx.de> napisał(a):
>How can I select different linguas for individual packages with this 
>approach?

Using the currently available mechanisms you could use package.env to alter INSTALL_MASK, e.g.:

/etc/portage/env/german:
  INSTALL_MASK="@l10n -@l10n-de"

/etc/portage/package.env:
  dev-foo/* german

However, we will probably deploy a convenient package.install_mask file.

>Michał Górny wrote:
>
>> Hello, everyone.
>> 
>> Since the previous thread doesn't seem to have brought any good
>> solution to the problem other than stopping to (ab)use LINGUAS
>> as USE_EXPAND, I would like to start a RFC on a draft solution that
>> I'd like afterwards to propose to the Council.
>> 
>> 
>> Rationale
>> ---------
>> 
>> The direct reason for this is that LINGUAS is treated as non-standard
>> special variable by multiple build systems. This includes the
>following
>> problems:
>> 
>> 1. no localizations are installed if it is set to an empty value
>(which
>> happens in EAPI 5 when the ebuild does not use the flags),
>> 
>> 2. there were historical cases where order of LINGUAS mattered.
>> 
>> Those problems can't be reasonably solved within the scope of
>> USE_EXPAND. Furthermore, the use of flags to control localizations is
>> causing the following problems:
>> 
>> a. maintaining correct flag list is a serious maintenance burden,
>> especially that differences in build systems make it hard to figure
>out
>> the 'most correct' set automatically,
>> 
>> b. missing flags result in localizations being silently dropped, with
>> no clear way (i.e. for QA check) to detect that,
>> 
>> c. large number of additional USE flags make it pretty much
>impossible
>> to limit localizations this way when using binary packages.
>> 
>> 
>> The plan
>> --------
>> 
>> 1. Get approval on INSTALL_MASK GLEP [1] and finish implementing it
>> in Portage.
>> 
>> 2. Introduce a new USE_EXPAND that can be used to control
>localizations
>> whenever this is really required (dependencies, large files, etc.).
>> Let's use L10N as a draft name for it.
>> 
>> 3. Fix all packages using LINGUAS as USE_EXPAND, either by converting
>> to L10N or by removing the needless flags.
>> 
>> 4. Remove LINGUAS from USE_EXPAND, therefore removing the special
>EAPI
>> rules from the variable.
>> 
>> 5. Release a news item explaining the users the change,
>> and the necessary action. Request changing LINGUAS to L10N
>> in make.conf, and make LINGUAS considered an 'advanced variable' for
>> implicit localization control (i.e. passed through to build systems).
>> Recommend clean INSTALL_MASK solution instead.
>> 
>> The example 'new' make.conf would probably look like:
>> 
>>   # controlling e.g. langpacks
>>   L10N="en_US pl"
>>   # stripping unneeded files
>>   INSTALL_MASK="@linguas -@linguas_pl"
>> 
>> 
>> Your thoughts?
>> 
>> 
>> [1]:https://wiki.gentoo.org/wiki/User:MGorny/GLEP:INSTALL_MASK
>> 


-- 
Best regards,
Michał Górny (by phone)

Re: [gentoo-dev] [RFC] Masterplan for solving LINGUAS problems

[Mart Raudsepp]

Ühel kenal päeval, T, 31.05.2016 kell 14:49, kirjutas Michał Górny:
> Since the previous thread doesn't seem to have brought any good
> solution to the problem other than stopping to (ab)use LINGUAS
> as USE_EXPAND, I would like to start a RFC on a draft solution that
> I'd like afterwards to propose to the Council.
> 
> 
> Rationale
> ---------
> 
> The direct reason for this is that LINGUAS is treated as non-standard
> special variable by multiple build systems. This includes the
> following
> problems:
> 
> 1. no localizations are installed if it is set to an empty value
> (which
> happens in EAPI 5 when the ebuild does not use the flags),

Why not just add a USE_EXPAND_DONTTOUCH variable to profiles, like
there is already a USE_EXPAND_HIDDEN, which tells the PM to not play
with the envvar, and just use it for USE expansion when the ebuild does
use it?
Point being, it leaves it unset, when it's unset, and it leaves it set
to empty value or a value when it is so.

Obviously with a better name than USE_EXPAND_DONTTOUCH, but you get the idea.

I suppose this doesn't solve the case of PM not knowing about what is inside a binary package, but if said variable also affected the metadata of the result, this should be possible to handled with some PM work, while not duplicating places to set languages to be compiled/installed.

The common case should be to support language x, y and z, and not wanting to change that, and never or rarely build binary packages (just as a backup before upgrade).
This is how I believe Gentoo is used as a source based distribution.

And big roll-outs with staging and binary packages have capable overlords knowing what's up in this area.

As this idea is too obvious, I'm sure it has come up before and dismissed, but as I don't remember it mentioned in the previous thread, nor with a quick skim over them, here it is anyways.


Mart

[gentoo-dev] Re: [RFC] Masterplan for solving LINGUAS problems

[Martin Vaeth]

Michał Górny <mgorny@gentoo.org> wrote:
>
> 1. Get approval on INSTALL_MASK GLEP [1] and finish implementing it
> in Portage.

As already mentioned by some, INSTALL_MASK is something else than
LINGUAS, because LINGUAS involves also files which are generated
by the build system. Although I appreciate a more configurable
INSTALL_MASK, this should not be mixed with the LINGUAS problem.

> 2. Introduce a new USE_EXPAND that can be used to control localizations
> whenever this is really required (dependencies, large files, etc.).
> Let's use L10N as a draft name for it.

Just to be sure that there is no misunderstanding:
L10N should take the role which LINGUAS currently has for most
packages from the user perspective.
In other words, all ebuilds for packages whose build systems treat
LINGUAS in a clean way (not relying on order etc.)
and which currently have IUSE=linuguas_* will very likely contain
corresponding ISUE=l10n_* and the line

export LINGUAS=$L10N

Therefore, I suggest to put this line in l10n.eclass
(perhaps with a mechanism to avoid it for some exceptional packages
which have to treat LINGUAS differently.)
This way, none of these ebuilds inheriting l10n needs to be patched.

> 3. Fix all packages using LINGUAS as USE_EXPAND, either by converting
> to L10N or by removing the needless flags.

I would strongly discourage doing the latter (unless IUSE=linguas_*
was there only by mistake):  When users change their L10N (or
USE-flags), users should be able to see which packages need a
recompilation when using emerge -N. Also for binary packages, it
must be easily possible to see changes.
See the more lengthy explanation below.

> Request changing LINGUAS to L10N in make.conf,

+

> and make LINGUAS considered an 'advanced variable' for
> implicit localization control (i.e. passed through to build systems).

With only some rare exceptions (e.g. where the order matters),
ebuilds should better set this variable according to IUSE=l10n_*.
It would be better if these exceptions would not have to exist
at all, i.e. if e.g. the ebuilds for packages for which the order
of LINGUAS matters also use IUSE=l10n_* and introduce other
USE flags to specify the order somehow (AFAIK only mplayer is
involved, and for this only the first item is important, so that
one can introduce e.g. IUSE=default_l10n_* variables for which exactly
one must be set, or something similar.)
Indeed, these exceptions are very inconvenient for the user
as well as for the package manager:

1) In contrast to packages with L10N, the user cannot see with
tools like eix for which linguas a certain package is installed.
In fact, the user would have to analyze the compressed environment
file find this out (this is also very package manager specific):

2) Even worse for binary packages.

3) The package manager cannot see after a change of LINGUAS,
which packages need a recompilaten.

4) The same for binary packages.

> Recommend clean INSTALL_MASK solution instead.

I would also not mix these unrelated things in the documentation.
One can hint that this takes additional actions for build
systems not honouring L10N correctly, but usually it is
not a substitute for setting L10N (or LINGUAS for the
exceptional packages) but only a supplement.

Re: [gentoo-dev] [RFC] Masterplan for solving LINGUAS problems

[Mart Raudsepp]

Ühel kenal päeval, K, 01.06.2016 kell 12:18, kirjutas Mart Raudsepp:
> Ühel kenal päeval, T, 31.05.2016 kell 14:49, kirjutas Michał Górny:
> > Since the previous thread doesn't seem to have brought any good
> > solution to the problem other than stopping to (ab)use LINGUAS
> > as USE_EXPAND, I would like to start a RFC on a draft solution that
> > I'd like afterwards to propose to the Council.
> > 
> > 
> > Rationale
> > ---------
> > 
> > The direct reason for this is that LINGUAS is treated as non-
> > standard
> > special variable by multiple build systems. This includes the
> > following
> > problems:
> > 
> > 1. no localizations are installed if it is set to an empty value
> > (which
> > happens in EAPI 5 when the ebuild does not use the flags),
> 
> Why not just add a USE_EXPAND_DONTTOUCH variable to profiles, like
> there is already a USE_EXPAND_HIDDEN, which tells the PM to not play
> with the envvar, and just use it for USE expansion when the ebuild
> does
> use it?
> Point being, it leaves it unset, when it's unset, and it leaves it
> set
> to empty value or a value when it is so.
> 
> Obviously with a better name than USE_EXPAND_DONTTOUCH, but you get
> the idea.
> 
> I suppose this doesn't solve the case of PM not knowing about what is
> inside a binary package, but if said variable also affected the
> metadata of the result, this should be possible to handled with some
> PM work, while not duplicating places to set languages to be
> compiled/installed.
> 
> The common case should be to support language x, y and z, and not
> wanting to change that, and never or rarely build binary packages
> (just as a backup before upgrade).
> This is how I believe Gentoo is used as a source based distribution.
> 

That said, I didn't like this being made a USE_EXPAND from the start,
as it does something else - control what huge language packs get
installed, as opposed to what stripping of files and file contents is
done. And LINGUAS is quite a different beast, due to dialects, having
special rules of implicit handling (e.g, you have a dialect in LINGUAS,
but no specific translations exist for that language, you still get the
main one instead; or was it vice-versa).
But I don't think we need to wait for the INSTALL_MASK bits to swap the
USE_EXPAND over to L10N or some other name (maybe something that
denotes likely extra downloads of langpacks). We still have LINGUAS,
which we might want to document better (with the binary package caveats
mentioned, etc), and localepurge that end results in the same result as
with the proposed INSTALL_MASK.

Though this shouldn't demotivate to make this group feature for
INSTALL_MASK happen regardless. The lingua groups should probably be
with wildcards like <path>/pl/... AND <path>/pl_*/... to cover
dialects, when you don't have dialect groups.

Additionally I don't think there's harm in filtering languages out
manually based on LINGUAS after all this is done still (LINGUAS not
being a USE_EXPAND), if the upstream build system itself doesn't. Might
want to take more care about dialects though.

`
Mart

Re: [gentoo-dev] [RFC] Masterplan for solving LINGUAS problems

[Michał Górny]

Dnia 1 czerwca 2016 11:18:30 CEST, Mart Raudsepp <leio@gentoo.org> napisał(a):
>Ühel kenal päeval, T, 31.05.2016 kell 14:49, kirjutas Michał Górny:
>> Since the previous thread doesn't seem to have brought any good
>> solution to the problem other than stopping to (ab)use LINGUAS
>> as USE_EXPAND, I would like to start a RFC on a draft solution that
>> I'd like afterwards to propose to the Council.
>> 
>> 
>> Rationale
>> ---------
>> 
>> The direct reason for this is that LINGUAS is treated as non-standard
>> special variable by multiple build systems. This includes the
>> following
>> problems:
>> 
>> 1. no localizations are installed if it is set to an empty value
>> (which
>> happens in EAPI 5 when the ebuild does not use the flags),
>
>Why not just add a USE_EXPAND_DONTTOUCH variable to profiles, like
>there is already a USE_EXPAND_HIDDEN, which tells the PM to not play
>with the envvar, and just use it for USE expansion when the ebuild does
>use it?
>Point being, it leaves it unset, when it's unset, and it leaves it set
>to empty value or a value when it is so.
>
>Obviously with a better name than USE_EXPAND_DONTTOUCH, but you get the
>idea.

Excuse me but are you really serious? We are in this swamp because someone tried to be too smart. And what solution do you propose? Let's add another layer of complexity and smartness, for a single variable. Obviously nothing can go wrong with this.

>
>I suppose this doesn't solve the case of PM not knowing about what is
>inside a binary package, but if said variable also affected the
>metadata of the result, this should be possible to handled with some PM
>work, while not duplicating places to set languages to be
>compiled/installed.
>
>The common case should be to support language x, y and z, and not
>wanting to change that, and never or rarely build binary packages (just
>as a backup before upgrade).
>This is how I believe Gentoo is used as a source based distribution.
>
>And big roll-outs with staging and binary packages have capable
>overlords knowing what's up in this area.
>
>As this idea is too obvious, I'm sure it has come up before and
>dismissed, but as I don't remember it mentioned in the previous thread,
>nor with a quick skim over them, here it is anyways.
>
>
>Mart


-- 
Best regards,
Michał Górny (by phone)

Re: [gentoo-dev] [RFC] Masterplan for solving LINGUAS problems

[Dirkjan Ochtman]

On Wed, Jun 1, 2016 at 3:09 PM, Michał Górny <mgorny@gentoo.org> wrote:
> Excuse me but are you really serious? We are in this swamp because someone tried to be too smart. And what solution do you propose? Let's add another layer of complexity and smartness, for a single variable. Obviously nothing can go wrong with this.

Please stop the derogatory remarks and unnecessary cynicism. If you
think some proposed solution is a bad idea, just explain why you think
it's a bad idea. Not everyone has the same perspective as you on these
things; nor are they likely to be converted to your stance by your
scolding them.

Cheers,

Dirkjan

Re: [gentoo-dev] Re: [RFC] Masterplan for solving LINGUAS problems

[Michał Górny]

Dnia 1 czerwca 2016 11:23:09 CEST, Martin Vaeth <martin@mvath.de> napisał(a):
>Michał Górny <mgorny@gentoo.org> wrote:
>>
>> 1. Get approval on INSTALL_MASK GLEP [1] and finish implementing it
>> in Portage.
>
>As already mentioned by some, INSTALL_MASK is something else than
>LINGUAS, because LINGUAS involves also files which are generated
>by the build system. Although I appreciate a more configurable
>INSTALL_MASK, this should not be mixed with the LINGUAS problem.

LINGUAS will be passed through from the environment so nothing is lost.

>
>> 2. Introduce a new USE_EXPAND that can be used to control
>localizations
>> whenever this is really required (dependencies, large files, etc.).
>> Let's use L10N as a draft name for it.
>
>Just to be sure that there is no misunderstanding:
>L10N should take the role which LINGUAS currently has for most
>packages from the user perspective.
>In other words, all ebuilds for packages whose build systems treat
>LINGUAS in a clean way (not relying on order etc.)
>and which currently have IUSE=linuguas_* will very likely contain
>corresponding ISUE=l10n_* and the line
>
>export LINGUAS=$L10N
>
>Therefore, I suggest to put this line in l10n.eclass
>(perhaps with a mechanism to avoid it for some exceptional packages
>which have to treat LINGUAS differently.)
>This way, none of these ebuilds inheriting l10n needs to be patched.

This eclass should be killed with fire as ugly nonsense that makes some packages useless for binary packages. As far as I'm concerned, it may not exist, be broken or whatever.

As for LINGUAS, it should be left as a toy for advanced users and not presented as a recommended solution.

>
>> 3. Fix all packages using LINGUAS as USE_EXPAND, either by converting
>> to L10N or by removing the needless flags.
>
>I would strongly discourage doing the latter (unless IUSE=linguas_*
>was there only by mistake):  When users change their L10N (or
>USE-flags), users should be able to see which packages need a
>recompilation when using emerge -N. Also for binary packages, it
>must be easily possible to see changes.
>See the more lengthy explanation below.
>
>> Request changing LINGUAS to L10N in make.conf,
>
>+
>
>> and make LINGUAS considered an 'advanced variable' for
>> implicit localization control (i.e. passed through to build systems).
>
>With only some rare exceptions (e.g. where the order matters),
>ebuilds should better set this variable according to IUSE=l10n_*.
>It would be better if these exceptions would not have to exist
>at all, i.e. if e.g. the ebuilds for packages for which the order
>of LINGUAS matters also use IUSE=l10n_* and introduce other
>USE flags to specify the order somehow (AFAIK only mplayer is
>involved, and for this only the first item is important, so that
>one can introduce e.g. IUSE=default_l10n_* variables for which exactly
>one must be set, or something similar.)
>Indeed, these exceptions are very inconvenient for the user
>as well as for the package manager:
>
>1) In contrast to packages with L10N, the user cannot see with
>tools like eix for which linguas a certain package is installed.
>In fact, the user would have to analyze the compressed environment
>file find this out (this is also very package manager specific):

What's the use of this? Most of the packages don't have the flags anyway. For those who have, you can't trust them being up-to-date.

As for installed package, all files are listed in vdb (including those masked), so you can easily figure out the differences. In fact, app-portage/install-mask supports rebuilding due to IM changes for a long time.

>
>2) Even worse for binary packages.

Wrong. All localizations are included in the binary package, so it's perfectly reusable across all systems. Of course, as long as maintainer doesn't start playing with LINGUAS.

>
>3) The package manager cannot see after a change of LINGUAS,
>which packages need a recompilaten.

Wrong, as already explained above.

>
>4) The same for binary packages.

Likewise.

>
>> Recommend clean INSTALL_MASK solution instead.
>
>I would also not mix these unrelated things in the documentation.
>One can hint that this takes additional actions for build
>systems not honouring L10N correctly, but usually it is
>not a substitute for setting L10N (or LINGUAS for the
>exceptional packages) but only a supplement.


-- 
Best regards,
Michał Górny (by phone)

Re: [gentoo-dev] [RFC] Masterplan for solving LINGUAS problems

[Mart Raudsepp]

Ühel kenal päeval, K, 01.06.2016 kell 15:18, kirjutas Dirkjan Ochtman:
> On Wed, Jun 1, 2016 at 3:09 PM, Michał Górny <mgorny@gentoo.org>
> wrote:
> > Excuse me but are you really serious? We are in this swamp because
> > someone tried to be too smart. And what solution do you propose?
> > Let's add another layer of complexity and smartness, for a single
> > variable. Obviously nothing can go wrong with this.
> 
> Please stop the derogatory remarks and unnecessary cynicism. If you
> think some proposed solution is a bad idea, just explain why you
> think
> it's a bad idea. Not everyone has the same perspective as you on
> these
> things; nor are they likely to be converted to your stance by your
> scolding them.


+1
Especially given the followup mail.
Really unprofessional.

So lets have users set multiple variables for the same thing, so it's
done properly because users need to care about env variable intricacies
in package management deep guts.
That said, maybe indeed better to force them to set it twice, as
discussed in my followup mail.


Mart

Re: [gentoo-dev] Re: [RFC] Masterplan for solving LINGUAS problems

[Mart Raudsepp]

Ühel kenal päeval, K, 01.06.2016 kell 15:19, kirjutas Michał Górny:
> As for LINGUAS, it should be left as a toy for advanced users and not
> presented as a recommended solution.

There is nothing advanced in it for the user, only the mess we have
created with package manager behaviour and mis-use of it (the order
matters case; which I believe is long eradicated).
We are a source based distribution, and gettext/intltool upstream
LINGUAS behaviour is perfect advantage for our main use case of
customizing ones own system and almost always building things from
source, only using binary packages before an upgrade as a backup, if at
all.
So it's natural to use the way that really build only the support you
want. This is what LINGUAS gives you, when the PM doesn't happen to
munge it.

Hiding this away under some toy for advanced users is not in our spirit
of Gentoo, as far as I would judge.

But this is a matter of documentation at this point, in principle I
agree that SRC_URI extra downloads should be under a different naming.

INSTALL_MASK groups for locales is what I would consider a convenience
for binary package builders in a wide environment where language choice
to the end user preferably gets filtered on deployment in a site- or
machine-specific manner. Or a toy for advanced binary distribution
creators, if you will. A way for binary packages to provide almost as
good support for LINGUAS as source packages (but not quite).
That said, supporting our binary package ecosystem is very important,
and I applaud these efforts. The proposed INSTALL_MASK improvements are
very useful for many other cases as well. For source-based users as
well (openrc init scripts, systemd unit files, gtk-doc documentation,
etc)

Either way, the masterplan works out, I just don't think we need to
wait for INSTALL_MASK groups here in any way. The reminder is a matter
of documentation, a matter of perspective.
This l10n.eclass PLOCALES nonsense needs to go ASAP.


Mart

Re: [gentoo-dev] Re: [RFC] Masterplan for solving LINGUAS problems

[Michał Górny]

Dnia 1 czerwca 2016 16:03:40 CEST, Mart Raudsepp <leio@gentoo.org> napisał(a):
>Ühel kenal päeval, K, 01.06.2016 kell 15:19, kirjutas Michał Górny:
>> As for LINGUAS, it should be left as a toy for advanced users and not
>> presented as a recommended solution.
>
>There is nothing advanced in it for the user, only the mess we have
>created with package manager behaviour and mis-use of it (the order
>matters case; which I believe is long eradicated).
>We are a source based distribution, and gettext/intltool upstream
>LINGUAS behaviour is perfect advantage for our main use case of
>customizing ones own system and almost always building things from
>source, only using binary packages before an upgrade as a backup, if at
>all.
>So it's natural to use the way that really build only the support you
>want. This is what LINGUAS gives you, when the PM doesn't happen to
>munge it.
>
>Hiding this away under some toy for advanced users is not in our spirit
>of Gentoo, as far as I would judge.

You forget the important point that it's done silently and implicitly, with no clear way of knowing which localizations were actually discarded afterwards.

And the fact that currently LINGUAS affects both packages listing the flags and not doing so is causing even more confusion.

>
>But this is a matter of documentation at this point, in principle I
>agree that SRC_URI extra downloads should be under a different naming.
>
>INSTALL_MASK groups for locales is what I would consider a convenience
>for binary package builders in a wide environment where language choice
>to the end user preferably gets filtered on deployment in a site- or
>machine-specific manner. Or a toy for advanced binary distribution
>creators, if you will. A way for binary packages to provide almost as
>good support for LINGUAS as source packages (but not quite).
>That said, supporting our binary package ecosystem is very important,
>and I applaud these efforts. The proposed INSTALL_MASK improvements are
>very useful for many other cases as well. For source-based users as
>well (openrc init scripts, systemd unit files, gtk-doc documentation,
>etc)
>
>Either way, the masterplan works out, I just don't think we need to
>wait for INSTALL_MASK groups here in any way. The reminder is a matter
>of documentation, a matter of perspective.
>This l10n.eclass PLOCALES nonsense needs to go ASAP.
>
>
>Mart


-- 
Best regards,
Michał Górny (by phone)

Re: [gentoo-dev] Re: [RFC] Masterplan for solving LINGUAS problems

[Raymond Jennings]

[-- Attachment #1: Type: text/plain, Size: 3069 bytes --]

I'd honestly as a minor issue like ot know why we called it linguas in the
first place.  Linguas itself is latin/romance based in name, so gentoo at
least has been showing a bit of a bias.

Personally I think its a bad name on neutrality grounds alone.

I think though I should also point out...don't we already have existing
ebuilds that expose LINGUAS options in their USE flags?

On Wed, Jun 1, 2016 at 7:17 AM, Michał Górny <mgorny@gentoo.org> wrote:

> Dnia 1 czerwca 2016 16:03:40 CEST, Mart Raudsepp <leio@gentoo.org>
> napisał(a):
> >Ühel kenal päeval, K, 01.06.2016 kell 15:19, kirjutas Michał Górny:
> >> As for LINGUAS, it should be left as a toy for advanced users and not
> >> presented as a recommended solution.
> >
> >There is nothing advanced in it for the user, only the mess we have
> >created with package manager behaviour and mis-use of it (the order
> >matters case; which I believe is long eradicated).
> >We are a source based distribution, and gettext/intltool upstream
> >LINGUAS behaviour is perfect advantage for our main use case of
> >customizing ones own system and almost always building things from
> >source, only using binary packages before an upgrade as a backup, if at
> >all.
> >So it's natural to use the way that really build only the support you
> >want. This is what LINGUAS gives you, when the PM doesn't happen to
> >munge it.
> >
> >Hiding this away under some toy for advanced users is not in our spirit
> >of Gentoo, as far as I would judge.
>
> You forget the important point that it's done silently and implicitly,
> with no clear way of knowing which localizations were actually discarded
> afterwards.
>
> And the fact that currently LINGUAS affects both packages listing the
> flags and not doing so is causing even more confusion.
>
> >
> >But this is a matter of documentation at this point, in principle I
> >agree that SRC_URI extra downloads should be under a different naming.
> >
> >INSTALL_MASK groups for locales is what I would consider a convenience
> >for binary package builders in a wide environment where language choice
> >to the end user preferably gets filtered on deployment in a site- or
> >machine-specific manner. Or a toy for advanced binary distribution
> >creators, if you will. A way for binary packages to provide almost as
> >good support for LINGUAS as source packages (but not quite).
> >That said, supporting our binary package ecosystem is very important,
> >and I applaud these efforts. The proposed INSTALL_MASK improvements are
> >very useful for many other cases as well. For source-based users as
> >well (openrc init scripts, systemd unit files, gtk-doc documentation,
> >etc)
> >
> >Either way, the masterplan works out, I just don't think we need to
> >wait for INSTALL_MASK groups here in any way. The reminder is a matter
> >of documentation, a matter of perspective.
> >This l10n.eclass PLOCALES nonsense needs to go ASAP.
> >
> >
> >Mart
>
>
> --
> Best regards,
> Michał Górny (by phone)
>
>

[-- Attachment #2: Type: text/html, Size: 3793 bytes --]

[gentoo-dev] Re: [RFC] Masterplan for solving LINGUAS problems

[Martin Vaeth]

Michał Górny <mgorny@gentoo.org> wrote:
>>
>>Therefore, I suggest to put this line in l10n.eclass
>>(perhaps with a mechanism to avoid it for some exceptional packages
>>which have to treat LINGUAS differently.)
>>This way, none of these ebuilds inheriting l10n needs to be patched.
>
> This eclass should be killed with fire as ugly nonsense that
> makes some packages useless for binary packages.

Exactly the opposite is the case:
Letting a variabe like LINGUAS "hiddden from the package manager"
decide about the actual content of the package makes any handling
of binary packages broken.
The introduction of linguas_* (and of the l10n eclass to make this
handable for the ebuild writer) was finally fixing this long-broken
behaviour of ebuilds.
You now want to revert this fix and return to the earlier broken
state without any need.
If the only reason for the "need" is a badly
formulated/thought-through EAPI=5/6 specification to clean
LINGUAS for packages not having it in its IUSE, then please
fix this specification or make at least an exception for LINGUAS,
but don't break the working solution.

> As for LINGUAS, it should be left as a toy for advanced users

Selecting the languages which a package supports is an option
only for advanced users? You must be kidding!

> and not presented as a recommended solution.

The "recommended solution" is to rely on a hack which removs
files behind the back of upstreams installation program (which
likely installs other random language data in random files for
various languages)?
You are kidding again!

>>1) In contrast to packages with L10N, the user cannot see with
>>tools like eix for which linguas a certain package is installed.
>>In fact, the user would have to analyze the compressed environment
>>file find this out (this is also very package manager specific):
>
> What's the use of this?

That the package manager (and the user) is well aware of the
options actually used to compile/install the package. Not hiding
the user-selected configuration behind the package manager and
the user in some random variable.

> Most of the packages don't have the flags anyway.

Then fix the package not handling LINGUAS in a sane way yet.

> For those who have, you can't trust them being up-to-date.

That's a non-argument. No ebuild you can trust being
up-to-date as well as all ebuilds.

> As for installed package, all files are listed in vdb
> (including those masked), so you can easily figure out the
> differences.

So to find out whether a binary package is valid for the
current configuration (i.e. the current LINGUAS), the package
manager or user "just" has to recompile it and compare
the files and checksums. Great!
And because this reinstallation-solution is possible, it
is ok to rely on a hack instead of a solution visible to
the package manager and user.

>>2) Even worse for binary packages.
>
> Wrong. All localizations are included in the binary package,

but not in the metadata.

> Of course, as long as maintainer doesn't start playing with LINGUAS.

That's the point: As long as all systems and all packages have
the same LINGUAS settings. Which is not the case for many
users having various systems. E.g. for a laptop, it is very
natural to have different LINGUAS than for a desktop, even
if you likely compile for the laptop on the desktop.

>>3) The package manager cannot see after a change of LINGUAS,
>>which packages need a recompilaten.
>
> Wrong, as already explained above.

So? How can you see it? By checking hackishly the existence
of random files in */po/* while e.g. the actual effect of
LINGUAS is in some error message provided by the package
in god-knows-what manner?

*If* you want to keep only LINGUAS and not introduce L10N,
*then* at least you should make LINGUAS a metadata variable
to be stored in /var/db so that the user and package manager
can easily access it.

Re: [gentoo-dev] Re: [RFC] Masterplan for solving LINGUAS problems

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 6993 bytes --]

On Wed, 1 Jun 2016 16:27:16 +0000 (UTC)
Martin Vaeth <martin@mvath.de> wrote:

> Michał Górny <mgorny@gentoo.org> wrote:
> >>
> >>Therefore, I suggest to put this line in l10n.eclass
> >>(perhaps with a mechanism to avoid it for some exceptional packages
> >>which have to treat LINGUAS differently.)
> >>This way, none of these ebuilds inheriting l10n needs to be patched.  
> >
> > This eclass should be killed with fire as ugly nonsense that
> > makes some packages useless for binary packages.  
> 
> Exactly the opposite is the case:
> Letting a variabe like LINGUAS "hiddden from the package manager"
> decide about the actual content of the package makes any handling
> of binary packages broken.
> The introduction of linguas_* (and of the l10n eclass to make this
> handable for the ebuild writer) was finally fixing this long-broken
> behaviour of ebuilds.
> You now want to revert this fix and return to the earlier broken
> state without any need.

Do you have any numbers on how many ebuilds were exactly being fixed?
And how many were broken in the process because someone failed to
update linguas_*? How do you feel about the lot of Chromium users who
spent a few hours rebuilding it to discover that they did only for
a linguas_* flag change they didn't even care for?

We're seriously talking about broken fix to a non-existent problem that
actually introduces real problems (and the incidental locale removal
is the least of those problems).

> If the only reason for the "need" is a badly
> formulated/thought-through EAPI=5/6 specification to clean
> LINGUAS for packages not having it in its IUSE, then please
> fix this specification or make at least an exception for LINGUAS,
> but don't break the working solution.

How are you going to fix it? What should land in LINGUAS then? How
would you determine the correct value? How would you apply package.use
with flags that don't exist?

> > As for LINGUAS, it should be left as a toy for advanced users  
> 
> Selecting the languages which a package supports is an option
> only for advanced users? You must be kidding!

No, implicitly stripping installed files via insecure mechanism that
is outside of package manager control is.

> > and not presented as a recommended solution.  
> 
> The "recommended solution" is to rely on a hack which removs
> files behind the back of upstreams installation program (which
> likely installs other random language data in random files for
> various languages)?
> You are kidding again!

Yes, we know that all binary distributions were based on hacks for
years. Only Gentoo did it right by enabling quirks that incidentally
caused files not to be installed without knowing which files were
actually omitted. Except when they didn't. Or maybe they did.

> >>1) In contrast to packages with L10N, the user cannot see with
> >>tools like eix for which linguas a certain package is installed.
> >>In fact, the user would have to analyze the compressed environment
> >>file find this out (this is also very package manager specific):  
> >
> > What's the use of this?  
> 
> That the package manager (and the user) is well aware of the
> options actually used to compile/install the package. Not hiding
> the user-selected configuration behind the package manager and
> the user in some random variable.

It's as random as LINGUAS. Your point?

> > Most of the packages don't have the flags anyway.  
> 
> Then fix the package not handling LINGUAS in a sane way yet.

I'm not sure if you are aware of it but most of us is not getting paid
for working on Gentoo. So thank you, I'd rather not get another huge
amount of work for minor benefit of a few vocal users who can't stand
not having every single possible useless choice pretty printed on top of
every single package.

So how about you do it? I'm going to complain if you get around 80% of
ebuild fixed for this, and guarantee that the flags will always be
up-to-date.

> > For those who have, you can't trust them being up-to-date.  
> 
> That's a non-argument. No ebuild you can trust being
> up-to-date as well as all ebuilds.

Of course. Your arguments are important arguments, mine are
non-arguments. When people don't get needed locales installed, it's
a non-problem. When they get extra files on their precious hard drives,
it's a huge issue.

> > As for installed package, all files are listed in vdb
> > (including those masked), so you can easily figure out the
> > differences.  
> 
> So to find out whether a binary package is valid for the
> current configuration (i.e. the current LINGUAS), the package
> manager or user "just" has to recompile it and compare
> the files and checksums. Great!
> And because this reinstallation-solution is possible, it
> is ok to rely on a hack instead of a solution visible to
> the package manager and user.

I don't even understand what you're trying to imply.

*Every* binary package is valid because it doesn't have anything
stripped. It's stripped while the package is installed. That's how
INSTALL_MASK works.

And as I already pointed out, it *is* visible to the package manager
because it records it. It's real data. There. In the files. Seriously.

> >>2) Even worse for binary packages.  
> >
> > Wrong. All localizations are included in the binary package,  
> 
> but not in the metadata.

Your point?

> > Of course, as long as maintainer doesn't start playing with LINGUAS.  
> 
> That's the point: As long as all systems and all packages have
> the same LINGUAS settings. Which is not the case for many
> users having various systems. E.g. for a laptop, it is very
> natural to have different LINGUAS than for a desktop, even
> if you likely compile for the laptop on the desktop.

That's why you are not supposed to use it. Simple enough. Then you have
reusable binary packages that can get fitted to the target system using
INSTALL_MASK.

> >>3) The package manager cannot see after a change of LINGUAS,
> >>which packages need a recompilaten.  
> >
> > Wrong, as already explained above.  
> 
> So? How can you see it? By checking hackishly the existence
> of random files in */po/* while e.g. the actual effect of
> LINGUAS is in some error message provided by the package
> in god-knows-what manner?
> 
> *If* you want to keep only LINGUAS and not introduce L10N,
> *then* at least you should make LINGUAS a metadata variable
> to be stored in /var/db so that the user and package manager
> can easily access it.

Why are you making some random assumptions that I'm going to do exactly
the opposite of what I've proposed?

If LINGUAS is marked as advanced variable you're not supposed to use,
I don't care about its side effects. I care about L10N and files
stripped via INSTALL_MASKs. If you use LINGUAS, you're on your own.

-- 
Best regards,
Michał Górny
<http://dev.gentoo.org/~mgorny/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

[gentoo-dev] Re: [RFC] Masterplan for solving LINGUAS problems

[Martin Vaeth]

Michał Górny <mgorny@gentoo.org> wrote:
>
> Do you have any numbers on how many ebuilds were exactly being fixed?
> And how many were broken in the process because someone failed to
> update linguas_*?

Broken ebuilds are a reason to fix the ebuilds, but not a reason
to replace a working concept by a hack which only works sometimes.

> How do you feel about the lot of Chromium users who
> spent a few hours rebuilding it to discover that they did only for
> a linguas_* flag change they didn't even care for?

You mean a user who emits --newuse when he only wants --changed-use
and who sees the output of --ask, showing that the change is in
LINGUAS="..." and nevertheless does not --exclude the package?
In other words: Users who do not really know how to use portage?
Well, I would recommend them to learn about the tools they use...

> We're seriously talking about broken fix to a non-existent problem

No, we are talking about removing a working solution (l10n.eclass)
to return to a broken state which had caught me and probably many
other users who manage systems with various linguas needs -
until it was finally fixed at least for a certain number of packages
by introducing the l10n eclass.

> that actually introduces real problems

s/introduces/solves/
like seeing that installation of a binary package conflicts
with the machine's configuration.

> How are you going to fix it? What should land in LINGUAS then?

Leave LINGUAS unchanged for those packages which dont have linguas_*
in their IUSE.
Then still the packages which don't have IUSE=linguas_* are broken
(because they have LINGUAS take effect without knowing of the
user or the package manager), but at least those which currently
use IUSE=linguas_* are working.

> How would you determine the correct value?

For ebuilds which are not updated to use IUSE=languas_*
there is in principle no clean solution. Leave LINGUAS unchanged
for these as a temporary hack until hopefully they are fixed
some day. (Or change to l10n and modify l10n.eclass)

>> > As for LINGUAS, it should be left as a toy for advanced users
>> Selecting the languages which a package supports is an option
>> only for advanced users? You must be kidding!
>
> No, implicitly stripping installed files via insecure mechanism that
> is outside of package manager control is.

Here we have the same opinion. The point is that LINGUAS is a
variable which is not controlled by portage. This is the whole
point of my argument: There must be a clean way to control this
variable. The IUSE=linguas_* is one such clean way.
Another way might be to make LINGUAS really a /var/db variable.

> Yes, we know that all binary distributions were based on hacks for
> years. Only Gentoo did it right [...]

I do not understand what you mean here.
I was not talking about binary distributions; maybe there is a
misunderstanding.
Maybe we have the same opinion: That INSTALL_MASK is some sort of hack;
a very convenient one, but actually a hack.
In any case it is not capable to do the "right" thing concerning
linguas for every package; it only does it "right" for certain
packages.

>> That the package manager (and the user) is well aware of the
>> options actually used to compile/install the package. Not hiding
>> the user-selected configuration behind the package manager and
>> the user in some random variable.
>
> It's as random as LINGUAS. Your point?

Any variable which is not under the package manager's control
and which influences the building of the package is a problem.
LINGUAS is one, CFLAGS is another one. For CFLAGS there is at
least a variable in /var/db.

>> Then fix the package not handling LINGUAS in a sane way yet.
>
> I'm not sure if you are aware of it but most of us is not getting paid
> for working on Gentoo.

Not fixing the rest of the ebuild is not an excuse to remove a
mechanism which made at least those ebuilds work for which the
maintainers took the time to fix them (by using l10n.eclass
or doing an analogous thing manually).

>> For those who have, you can't trust them being up-to-date.
>> That's a non-argument. No ebuild you can trust being
>> up-to-date as well as all ebuilds.
>
> Of course. Your arguments are important arguments, mine are
> non-arguments.

I was not talking about "your arguments" but about your specific
argument "we must remove a working mechanism because an ebuild
author might do a mistake and e.g. bump without checking the
changes of the package".

> When people don't get needed locales installed, it's a non-problem.

Who said this? This is exactly the issue I am talking about with
binary packages:

> *Every* binary package is valid because it doesn't have anything
> stripped.

This is not true. If you compiled the binary package with
LINGUAS="en_US de_DE" and install it on a laptop with
LINGUAS="en_US de_DE cz_CZ", you will be missing the Czech
translations, because they are not in the binary package.
(They are not "stripped" but simply not there...)
So binary package compiled with LINGUAS="en_US de_DE" might
simply not be valid on a machine which has LINGUAS="en_US de_DE".
Well, the majority of packages probably still *is* valid,
because the majority of packages probably doesn't provide a Czech
translation.
That's why it should be known to the package manager which
LINGUAS setting have been used, and - at least for hopefully
as many packages as possible - which LINGUAS are actually
supported by the package.
The current solution with IUSE=linguas_* did this job very well.

> That's why you are not supposed to use it. Simple enough.

So you want to remove *completely* the possibility to use the
LINGUAS settings of some packages. Well, not remove it, but only
allow it in future as hack for users who are then obliged to do
the job of the package manager manually and without any support.

What are again the reasons for removing the working solution
with IUSE and showing the gentoo users the obscene stinky finger?
That removing it makes things simpler?
For the package manager maybe yes. For the user of course not.
As always: Removing features always apparently simplifies things.
Except for those who use these features.

> Why are you making some random assumptions that I'm going to do exactly
> the opposite of what I've proposed?
>
> If LINGUAS is marked as advanced variable you're not supposed to use,
> I don't care about its side effects. I care about L10N and files
> stripped via INSTALL_MASKs. If you use LINGUAS, you're on your own.

Then why do you claim that l10n.eclass (which then of course must
be updated to treat IUSE=l10n_* in the way it does currently with
instead of IUSE=linguas_*) should be removed?
l10n.eclass together with
export LINGUAS=$L10N
should be exactly the right behaviour for all packages currently
supporting IUSE=linguas_*. Why should this perfectly working thing
(at least for those ebuilds which support it) now be removed?
What would be the benefit for the user?  I see only the disadvantage
that some users now have to do the job of the package manager even
for these packages...

[gentoo-dev] Re: Re: [RFC] Masterplan for solving LINGUAS problems

[Jörg Schaible]

Michał Górny wrote:

> Dnia 31 maja 2016 23:34:07 CEST, "Jörg Schaible" <joerg.schaible@gmx.de>
> napisał(a):
>>How can I select different linguas for individual packages with this
>>approach?
> 
> Using the currently available mechanisms you could use package.env to
> alter INSTALL_MASK, e.g.:
> 
> /etc/portage/env/german:
>   INSTALL_MASK="@l10n -@l10n-de"
> 
> /etc/portage/package.env:
>   dev-foo/* german
> 
> However, we will probably deploy a convenient package.install_mask file.

OK, just to get this right. I always want English locale, for GUI apps also 
German and sometimes even French (office apps). Therefore I should set in 
future something along:

/etc/portage/make.conf:
  LINGUAS="en de fr"
  INSTALL_MASK="@l10n -@l10n-en"

/etc/portage/env/add-de:
  INSTALL_MASK="@l10n -@l10n-en -@l10n-de"

/etc/portage/env/add-de-fr:
  INSTALL_MASK="@l10n -@l10n-en -@l10n-de  -@l10n-fr"

/etc/portage/package.env:
  dev-foo-gui/* add-de
  dev-bar-office/* add-fr

Sounds reasonable to me. It's just no longer obvious(*) which packages 
actually support different languages.

(*) Boldly assuming all packages provide the correct info currently ;-)

Cheers,
Jörg

Re: [gentoo-dev] Re: [RFC] Masterplan for solving LINGUAS problems

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 963 bytes --]

On Wed, 1 Jun 2016 18:18:53 +0000 (UTC)
Martin Vaeth <martin@mvath.de> wrote:

> Michał Górny <mgorny@gentoo.org> wrote:
> >
> > Do you have any numbers on how many ebuilds were exactly being fixed?
> > And how many were broken in the process because someone failed to
> > update linguas_*?  
> 
> Broken ebuilds are a reason to fix the ebuilds, but not a reason
> to replace a working concept by a hack which only works sometimes.

If more than 95% of ebuilds are broken, this proves that a concept is
broken.

Well, feel free to live in your fairy land. I have better things to do.
I'm not going to continue the discussion if you are so blinded by
LINGUAS that you are even unable to understand what I'm talking about,
and consistently mix the LINGUAS concept and INSTALL_MASK concept,
and using the resulting mish-mash to prove whatever point you consider
correct.

-- 
Best regards,
Michał Górny
<http://dev.gentoo.org/~mgorny/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

[gentoo-dev] Re: [RFC] Masterplan for solving LINGUAS problems

[Martin Vaeth]

Michał Górny <mgorny@gentoo.org> wrote:
>
> If more than 95% of ebuilds are broken, this proves that a concept is
> broken.
>
> Well, feel free to live in your fairy land.

I prefer to have at least 5% of the ebuilds working and the other
being fixable (if their maintainers want to spend the effort)
than to remove a concept which breaks also these 5% and turns
all ebuilds non-fixable, in principle.

> I'm not going to continue the discussion if you are so blinded by
> LINGUAS that you are even unable to understand what I'm talking about,
> and consistently mix the LINGUAS concept and INSTALL_MASK concept,

...says the man who mixes this wildly in the first posting by
suggesting to recommend the user to not use LINGUAS and use
INSTALL_MASK *instead*; as a reply to my warning to not mix
these completely unrelated concepts.

Re: [gentoo-dev] Re: [RFC] Masterplan for solving LINGUAS problems

[Kent Fredric]

On 2 June 2016 at 07:33, Martin Vaeth <martin@mvath.de> wrote:
>
> I prefer to have at least 5% of the ebuilds working and the other
> being fixable (if their maintainers want to spend the effort)
> than to remove a concept which breaks also these 5% and turns
> all ebuilds non-fixable, in principle.

Changing the status-quo to "broken by default and needs 95% of the
tree to change to not be broken" is a bad precedent.

Its better to have 100% of the tree *not* broken by default, and then
we progressively whittle the tree into a flexible state.

( At least, this is what I've gathered so far )

>
>> I'm not going to continue the discussion if you are so blinded by
>> LINGUAS that you are even unable to understand what I'm talking about,
>> and consistently mix the LINGUAS concept and INSTALL_MASK concept,
>
> ...says the man who mixes this wildly in the first posting by
> suggesting to recommend the user to not use LINGUAS and use
> INSTALL_MASK *instead*; as a reply to my warning to not mix
> these completely unrelated concepts.


LINGUAS affects compilation due to weird toolchains.

INSTALL_MASK just nukes files.

-- 
Kent

KENTNL - https://metacpan.org/author/KENTNL

[gentoo-dev] Re: [RFC] Masterplan for solving LINGUAS problems

[Martin Vaeth]

Raymond Jennings <shentino@gmail.com> wrote:
>
> I'd honestly as a minor issue like ot know why we called it linguas in the
> first place.
> [...]
> I think though I should also point out...don't we already have existing
> ebuilds that expose LINGUAS options in their USE flags?

From this posting, it is obvious that the discussion between mgorny
and me was not well understandable, so I try to summarize the
situation:

The variable name LINGUAS was not chosen by gentoo:
autotools based build systems (and some others) support
this variable to control, for which languages some support is
installed. The effect can be that simply some files are (not)
installed, but the effect can also be much more complex:
It really depends on the package.

Gentoo has chosen this name so that as a side effect of setting
USE=linguas_* you also get a correct LINGUAS variable exported
(according to the USE-settings and your settings and according
to the ebuild's IUSE.)
These are the package with LINGUAS options in their USE flags
which you mentioned above.

This had the advantage that no additional code except a correct
IUSE is needed for these ebuilds. (That's why gentoo has chosen
this name).

It had the disadvantage that:
1. Ebuilds without handling IUSE=linguas_* explicitly (or
with wrong values, because maintainers did not care about the
values) had problems.
2. Some packages which needed a different handling of LINGUAS
(like respecting the order) also had problems.

There are at least the following solutions to these disadvantages:

a)
One _could_ solve problem 1 simply by not touching LINGUAS if
IUSE=linguas_* is not in the ebuild. (Main problem with this
solution: It is not PMS compatible; one needs e.g.
an exception for LINGUAS).
In a similar manner, one could solve problem 2 by allowing
ebuilds to modify LINGUAS at build time (which is perhaps
also not PMS compatible).

b)
Or one could, for all packages with LINGUAS in their USE-flag
simply rename IUSE=linguas_* to IUSE=l10n_* and set
export LINGUAS=$L10N
in these ebuilds (this would require practically no manual ebuild
editing if one does it in the l10n.eclass).

I had originally understood mgorny's suggestion such that b)
is meant, and I would complain neither against a) nor b).

But in his reply, mgorny says that, moreover, he wants to
remove the l10n.eclass, more precisely, that he considers
it as broken that packages use IUSE=l10n_* for setting
LINGUAS.

This suggestion means that setting LINGUAS can be done
*only* in a hackish way by the user, "hidden" from the
package manager, not in the clean way as it is currently
done by those ebuilds with LINGUAS in their use-falgs.

I agree with him that a hidden setting is a bad idea.
Where our opinions strongly differ is whether a way to
cleanly set LINGUAS should be provided (e.g. by allowing
IUSE=l10n_* to modify LINGUAS appropriately, and make this
the "good" way instead calling it a "broken" way which
should be avoided.)

The INSTALL_MASK is an independent thing which for _many_
packages "by accident" can be used with a similar effect as
setting LINGUAS (because many packages use LINGUAS only to
determine the files which they install).
However, it is a different thing, and for some packages
it is not a replacement at all.
For instance, the default language of mplayer will depend on
the first entry of LINGUAS, independent of INSTALL_MASK.
That's why I suggested that  INSTALL_MASK should actually
not be discussed at all in this context.

[gentoo-dev] Re: [RFC] Masterplan for solving LINGUAS problems

[Martin Vaeth]

Kent Fredric <kentfredric@gmail.com> wrote:
> On 2 June 2016 at 07:33, Martin Vaeth <martin@mvath.de> wrote:
>>
>> I prefer to have at least 5% of the ebuilds working and the other
>> being fixable (if their maintainers want to spend the effort)
>> than to remove a concept which breaks also these 5% and turns
>> all ebuilds non-fixable, in principle.
>
> Changing the status-quo to "broken by default and needs 95% of the
> tree to change to not be broken" is a bad precedent.

What you describe _is_ the status-quo. The 95% are broken and
remain broken (unless a lot of work is spent), no matter which
of the suggested solutions is chosen.

I am voting to *keep* better this than to change the status-quo
by breaking the 5% working percent, too:

mgorny's suggestion to kill l10n.eclass (or, more abstractly
speaking, to forbid setting LINGUAS based in USE-flags within
an ebuild) just influences the 5% of ebuilds which are
currently using this mechanism and thus are non-broken
(if their IUSE is correctly maintained).
Removing l10n.eclass would throw these 5% back to the broken
state of the other 95%, and even forbid by policy that any
other would be fixed.

I really cannot see an advantage in doing this, except for
some irrational feeling of "consistency" to have everything
equally broken.

Re: [gentoo-dev] Re: [RFC] Masterplan for solving LINGUAS problems

[Mart Raudsepp]

Ühel kenal päeval, K, 01.06.2016 kell 20:24, kirjutas Martin Vaeth:
> Gentoo has chosen this name so that as a side effect of setting
> USE=linguas_* you also get a correct LINGUAS variable exported
> (according to the USE-settings and your settings and according
> to the ebuild's IUSE.)
> These are the package with LINGUAS options in their USE flags
> which you mentioned above.

People already set LINGUAS for autotools control; using the same for
USE_EXPAND probably seemed natural to re-use that, but the effects on
PM variable mangling wasn't thought about at the time.
This was introduced only to have a natural way to refer to them in
SRC_URI and beyond, for extra language pack downloads (like firefox
langpacks).
Things escalated badly from there.

> This had the advantage that no additional code except a correct
> IUSE is needed for these ebuilds. (That's why gentoo has chosen
> this name).

> It had the disadvantage that:
> 1. Ebuilds without handling IUSE=linguas_* explicitly (or
> with wrong values, because maintainers did not care about the
> values) had problems.

There were no problems, and there still are no problem with implicit
LINGUAS handling (by means of not listing them in IUSE), as long as the
package manager doesn't modify the variable set in make.conf.
The user has set it explicitly somewhere (make.conf most likely) and
simply gets honored in upstream build system, just like CFLAGS.
The problem is when package manager mangles it per PMS rules, for which
I'm told portage has special PMS ignoring exceptions.

> 2. Some packages which needed a different handling of LINGUAS
> (like respecting the order) also had problems.

LINGUAS does not imply any order whatsoever. Packages that assumed so
were seriously buggy and if any still remain, this needs to be fixed
ASAP. We have LC_* and LANG environment variables to choose what the
localization is at runtime, not some arbitrary choice at build time.
This is not a concern whatsoever, forget about it.

> There are at least the following solutions to these disadvantages:
> 
> a)
> One _could_ solve problem 1 simply by not touching LINGUAS if
> IUSE=linguas_* is not in the ebuild. (Main problem with this
> solution: It is not PMS compatible; one needs e.g.
> an exception for LINGUAS).

This is a problem when they are in IUSE too. For example a package
provides translations via gettext, but has extra downloads for some
languages support (lets say grammar checking support for a language in
an office application). The latter would get marked up with IUSE for
usage in SRC_URI and install. Package manager will intersect them and
remove the LINGUAS values not found in IUSE, but simple UI translation
gettext catalogs for these languages might still exist upstream. They
now get removed due to IUSE=linguas_* not including them.

Including them all in IUSE for simple translation catalogs is not
feasible to maintain. It also clutters emerge --verbose --pretend or --
ask output hard with LINGUAS="long list of 196 language codes" for
these and mixes it all up for when the information is more useful when
it implies huge extra downloads.

And yes, there are packages that have 196 different language and
dialects translations. Even core GNOME packages would have such an
amount, see https://l10n.gnome.org/languages/

> In a similar manner, one could solve problem 2 by allowing
> ebuilds to modify LINGUAS at build time (which is perhaps
> also not PMS compatible).
> 
> b)
> Or one could, for all packages with LINGUAS in their USE-flag
> simply rename IUSE=linguas_* to IUSE=l10n_* and set
> export LINGUAS=$L10N
> in these ebuilds (this would require practically no manual ebuild
> editing if one does it in the l10n.eclass).

The idea is not to export this anywhere in ebuilds. We'd have to do
that in pretty much all.
The point is that if the USE_EXPAND is renamed, then a LINGUAS as found
in make.conf will just get passed verbatim into the environment (as
parsed in via shlex - make.conf is not bash sourced), and then honored
by upstream build system.
That's because it isn't in USE_EXPAND list anymore, so it doesn't have
such PMS rules to modify it.
One can modify the environment via package.env as well, to change
things per-package if one fancies for some reason.

> I had originally understood mgorny's suggestion such that b)
> is meant, and I would complain neither against a) nor b).
> 
> But in his reply, mgorny says that, moreover, he wants to
> remove the l10n.eclass, more precisely, that he considers
> it as broken that packages use IUSE=l10n_* for setting
> LINGUAS.
> 
> This suggestion means that setting LINGUAS can be done
> *only* in a hackish way by the user, "hidden" from the
> package manager, not in the clean way as it is currently
> done by those ebuilds with LINGUAS in their use-falgs.

I don't see anything hackish in just setting L10N for extra language
support downloads and LINGUAS for UI translations.

> I agree with him that a hidden setting is a bad idea.

You can find it in the VDB packed in environment.* in case of portage.
Exporting it specially like CFLAGS, CXXFLAGS, FEATURES and others are,
is a separate matter if one wants to propose that later.

> Where our opinions strongly differ is whether a way to
> cleanly set LINGUAS should be provided (e.g. by allowing
> IUSE=l10n_* to modify LINGUAS appropriately, and make this
> the "good" way instead calling it a "broken" way which
> should be avoided.)

I don't believe I have read such a claim from his e-mails anywhere, so
not sure what this is about.

> The INSTALL_MASK is an independent thing which for _many_
> packages "by accident" can be used with a similar effect as
> setting LINGUAS (because many packages use LINGUAS only to
> determine the files which they install).
> However, it is a different thing, and for some packages
> it is not a replacement at all.

I agree that INSTALL_MASK is a different thing, and that's why I don't
think we need to do step 1) in his masterplan; we can do that in
parallel and push out later, as a means to at least filter the specific
language files easily when installing a binary package which includes
all translations (was built with LINGUAS unset to not filter any with
autotools; or a superset of what you need on install on that machine).

> For instance, the default language of mplayer will depend on
> the first entry of LINGUAS, independent of INSTALL_MASK.
> That's why I suggested that  INSTALL_MASK should actually
> not be discussed at all in this context.

This is not the case since long ago as far as I look, as it was a bug
and fixed long ago.
USE_EXPAND can not express any ordering, portage probably sorts the
list alphabetically too.
LINGUAS as used upstream does not imply any order either.
Runtime locale environment variables like LC_MESSAGES are supposed to
affect what the "default" language is at startup; we are not windows.


Mart

Re: [gentoo-dev] [RFC] Masterplan for solving LINGUAS problems

[Daniel Campbell (zlg)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On June 1, 2016 6:46:28 AM PDT, Mart Raudsepp <leio@gentoo.org> wrote:
>Ühel kenal päeval, K, 01.06.2016 kell 15:18, kirjutas Dirkjan Ochtman:
>> On Wed, Jun 1, 2016 at 3:09 PM, Michał Górny <mgorny@gentoo.org>
>> wrote:
>> > Excuse me but are you really serious? We are in this swamp because
>> > someone tried to be too smart. And what solution do you propose?
>> > Let's add another layer of complexity and smartness, for a single
>> > variable. Obviously nothing can go wrong with this.
>>
>> Please stop the derogatory remarks and unnecessary cynicism. If you
>> think some proposed solution is a bad idea, just explain why you
>> think
>> it's a bad idea. Not everyone has the same perspective as you on
>> these
>> things; nor are they likely to be converted to your stance by your
>> scolding them.
>
>
>+1
>Especially given the followup mail.
>Really unprofessional.
>
>So lets have users set multiple variables for the same thing, so it's
>done properly because users need to care about env variable intricacies
>in package management deep guts.
>That said, maybe indeed better to force them to set it twice, as
>discussed in my followup mail.
>
>
>Mart

Unprofessional? We're volunteers. We're already bound by a CoC so there's no need to bring up a meaningless buzzword to describe behavior.

I'm not defending Michal's tone (sorry, not sure how to add the slash to the glyph on my phone), but 'unprofessional' is hardly an accusation to be taken seriously here. "Snarky", "rude", "unconstructive"; *something* that's descriptive and meaningful would make your point clearer.
- --
Sent from my Android device with K-9 Mail. Please excuse my brevity.
-----BEGIN PGP SIGNATURE-----

iQJRBAEBCgA7NBxEYW5pZWwgQ2FtcGJlbGwgKEdlbnRvbyBEZXZlbG9wZXIpIDx6
bGdAZ2VudG9vLm9yZz4FAldPTnUACgkQASQOlFA54XBZOg/8CRPXRWvH+GSQN9xG
JIybw5gNmDqfddmaHWTN4+8MzV8BEyeO2/n3NDfCV0awqgx2trdrBw8c1Us4JC2y
2PO6PWEsxIPWrUOI10VyF3nSFr7sPJuQcD6oOMpeUkzGSI7FjzsWuZ/BeBASAWDq
F6Z/kqJVCD3pHLxqBKPd9oUeLOND5hDYkx9il9GWW+Bl1xIP2wZLf8nnI73Uqk70
vaklt1rpIeh+CDJu6ToAg/yohUb0vFvUi/TffqLVvXXTa/IJCyhDHPbjQvqOfkJx
3Xc+IQvddwO3R04MG2Ram43Q3QB90NZ3fBA5lpGZo4lS/yFxFrYnEC2tQHe0sHFB
MGCJ642ly0pobbaFSkrXoXQl4NtSXSSPnab0OMo3XbHekDXcqwuRYYCQIzHljgND
lAZtYtBGgIsqAzyyJA+JezQAkM6SRgh+CQomzJZU7vZ68HQiwnJRZksYKwdC7d9S
W0ityGD6ECwFNV9+k+Oa2qoSEfGE2hF7fSIVh2HLReegdWKEajA6TyPpNrQ57mD5
0gFgqAi6F9r6vxUd7bTiWLu7aeJ94iWjr3eL+NbxPvqO/gIzggdyIWdGPYW76bgd
/Je99P9Q4uGy23WPNJ2P1844SjqpFAB/F3I3YVBekyhMvsOxaGpd8JQDVr5Q+45C
xTkvtwIPG/iImPgYOXkUvdWAB5U=
=cZZa
-----END PGP SIGNATURE-----

[gentoo-dev] Re: [RFC] Masterplan for solving LINGUAS problems

[Martin Vaeth]

Mart Raudsepp <leio@gentoo.org> wrote:
> Ühel kenal päeval, K, 01.06.2016 kell 20:24, kirjutas Martin Vaeth:
>> Gentoo has chosen this name so that as a side effect of setting
>> USE=linguas_* you also get a correct LINGUAS variable exported
>> (according to the USE-settings and your settings and according
>> to the ebuild's IUSE.)
>> These are the package with LINGUAS options in their USE flags
>> which you mentioned above.
>
> People already set LINGUAS for autotools control; using the same for
> USE_EXPAND probably seemed natural to re-use that, but the effects on
> PM variable mangling wasn't thought about at the time.

At least, it looks like it was well-thought:
It avoided errors like "language xxx not supported",
it gave the user control to set it easily on a per-peckage basis,
and most important, it made the package manager aware of the LINGUAS
used and handle binary packages correctly.
From the user perspective it had only advantages.
If all these positive side effects were not the original intention,
it was a very good accident that it happened.

> There were no problems, and there still are no problem with implicit
> LINGUAS handling (by means of not listing them in IUSE), as long as the
> package manager doesn't modify the variable set in make.conf.

As long as there is only one setting valid for all packages
and all systems on which one will ever install the binary package.
A quite unrealistic setting for quite some use cases.

> The user has set it explicitly somewhere (make.conf most likely) and
> simply gets honored in upstream build system, just like CFLAGS.

That handling of CFLAGS is bad enough - no reason to repeat this mistake:
This is usable only if you have only one architecture but not in a
hybrid setup with e.g. roughly compatible machines where e.g. you change
CFLAGS by /etc/portage/env or /etc/portage/package.env (or locally
in the environment) to compile for another machine or in a ways
compatible for another machine, etc.

Well, for CFLAGS, CXXFLAGS, LDFLAGS at least the
variables actually used are stored in
/var/db/cat/pkg/{CFLAGS,CXXFLAGS,LDFLAGS}
and similarly stored in the metadata of binary packages (not
hidden in some hard-to-parse compressed environment file),
so that the package manager _could_ check their validity
(and even if it does not - which AFAIK is the case currently -
it is not very hard to write a script which does this).

That's why I suggested that - as the very least - if you really want
to kill all the current advantages for no reason, at the very least
handle LNGUAS not even *worse* than CFLAGS and at least store them
in a new /var/db variable so that - if you already let the user do
the job which the package manager is supposed to do - at least the
user can write his own scripts to do portage's job.
(Of course, this scripts can never work as good as the current
solution if you really kill it, because these scripts are lacking
the information which LINGUAS are actually supported by the package.)

> LINGUAS does not imply any order whatsoever. Packages that assumed so
> were seriously buggy and if any still remain, this needs to be fixed
> ASAP.

So if you consider this as a non-issue, I see even less reason
why to remove the *transpararent* (to the package manager and user)
IUSE=linguas_* and go back to the intransparant LINGUAS=...
(I am not complaining about a renaming of the IUSE, as long as
this IUSE in the end affects LINUGAS)

> Including them all in IUSE for simple translation catalogs is not
> feasible to maintain.

I disagree. The list of packages is in most cases very easy to get
(e.g. ls po/*.po) and inserting it correctly is a usually a
one-time thing. For bumping, one will have to check for dependency
changes anyway, and that a package adds a new language without listing
it in its ChangeLog is also very rare.

> And yes, there are packages that have 196 different language and
> dialects translations. Even core GNOME packages [...]

Nevertheless these packages are very rare. One could also manage
*one* list of all languages globally and allow to use it for
convenience for such cases (if a few of the listed languages are
actually not provided, nothing critical will happen: The
user's output has perhaps a few more entries than necessary and
perhaps an unnecessary rebuild can happen if the user
selects/deselects an exotic language.)

> It also clutters emerge --verbose --pretend or --ask output hard with
> LINGUAS="long list of 196 language codes"

About the situation which we currently have (e.g. k3b): For some
packages the LINGUAS list is not 1 line but 2-4. So what?
If this really is an issue for somebody (I don't consider it some),
then one can discuss to add an option to suppress the LINGUAS output
for those not interested in it.
But it's certainly not a reason to make it *intransparent* for the
package manager and user.

> these and mixes it all up for when the information is more useful
> when it implies huge extra downloads.

This happens always with USE-flags: Unless you look at the
ebuild source, you never know whether it will trigger dependencies,
increase the download time, increase the build time (and how much),
increase mount of installed data (and how much) etc.
Just because that information is not listed, it does not make having
the USE-flags less useful. There is no difference to linguas-specific
USE-flags.

> The point is that if the USE_EXPAND is renamed, then a LINGUAS as found
> in make.conf will just get passed verbatim into the environment (as
> parsed in via shlex - make.conf is not bash sourced), and then honored
> by upstream build system.

Yes, exeactly: Intransparent implicit handling as a side effect,
instead of a transparent and reliable handling through the
package manager.
We had this hackish handling before IUSE=linguas_*
was introduced, which was a big relief for a lot of people.
Now the suggestion is to turn back to the hacks, excusing
the hack by declaring setting/changing LINGUAS a
thing for the "advanced" user, only.

> I don't see anything hackish in just setting L10N for extra language
> support downloads and LINGUAS for UI translations.

Setting LINGUAS without awareness of the package manager of it
*is* a hack.

> You can find it in the VDB packed in environment.* in case of portage.

Now, don't tell me that the necessity to write code which uncompresses
a library and parses the tricky escape handling of the environment
file to get eventually you own hackish script to do what the
package manager should be suppposed to do, is anything else than
an even bad hack.

> Exporting it specially like CFLAGS, CXXFLAGS, FEATURES and others are,
> is a separate matter if one wants to propose that later.

It would at least turn a really bad hack into a little bit less bad hack.
But it is of course never as good as a real transparent solution
with USE-flags.

>> Where our opinions strongly differ is whether a way to
>> cleanly set LINGUAS should be provided (e.g. by allowing
>> IUSE=l10n_* to modify LINGUAS appropriately, and make this
>> the "good" way instead calling it a "broken" way which
>> should be avoided.)
>
> I don't believe I have read such a claim from his e-mails anywhere, so
> not sure what this is about.

My the suggestion to make this (export LINGUAS based on
IUSE=linguas_*/l10n_*) as the default solution in l10n.eclass
(which is the current place where this thing happens implicitly
through the name) was replied with:

: This eclass should be killed with fire as ugly nonsense [...]
: As far as I'm concerned, it may not exist, be broken or whatever.

>> For instance, the default language of mplayer will depend on
>> the first entry of LINGUAS [...]
>
> This is not the case since long ago as far as I look, as it was a bug
> and fixed long ago.

Then even more, I think that LINGUAS should be implicitly set by through
USE-flags: If order does not matter, there is absolutely no reason to do
not do it transparatenly this way.