[gentoo-dev] LINGUAS handling

[gentoo-dev] LINGUAS handling

[Peter Volkov]

Hi.

We have LINGUAS support in portage for quite some time now, but how do
we handle LINGUAS? It's not as evident as it seems and thus we need some
guidelines. So I'll formulate few questions below but first let's look
how portage works with LINGUAS. If you know this skip to Questions
section below.


Portage/gettext
---------------

Like USE variable LINGUAS value could be set by user in /etc/make.conf,
package.keywords or environment. There are three different cases: 1)
LINGUAS set to some value, 2) LINGUAS set and contains '*'[1] 3) LINGUAS
unset. portage will treat this cases differently for ebuilds with
linguas_<lang> in IUSE and without. This makes 6 cases:

1. In case LINGUAS is set and ebuild has no linguas_<lang> in IUSE
portage exports LINGUAS as is.

2. In case LINGUAS is set and IUSE contains linguas_<lang> portage
exports intersection of languages in LINGUAS and IUSE. E.g. LINGUAS set
to "lang1 lang2" and IUSE contains "linguaus_<lang1> linguaus_<lang3>"
portage will export LINGUAS="lang1".

3. In case LINGUAS contains '*' and there is no linguaus_<lang> in IUSE
portage unsets LINGUAS.

4. In case LINGUAS contains '*' and IUSE contains linguas_<lang> portage
exports LINGUAS with langs defined in the IUSE's linguas_<lang>.

5. If LINGUAS is unset and ebuild has no linguas_<lang> flag portage
unsets LINGUAS.

6. In case LINGUAS is unset and IUSE contains linguas_<lang> portage
exports LINGUAS="".

                |    ebuild contains linguas_<lang> the IUSE      |
                --------------------------------------------------|
                |          no         |            yes            |
-------------------------------------------------------------------
LINGUAS unset   |    LINGUAS unset    |        LINGUAS=''         |
------------------------------------------------------------------|
LINGUAS has '*' |    LINGUAS unset    |      LINGUAS='lang'       |
-------------------------------------------------------------------
LINGUAS set     |LINGUAS set literally|LINGUAS set to intersection|
                |                     |of lang in IUSE and LINGUAS|
-------------------------------------------------------------------

BTW, I guess all above is applicable to all USE_EXPANDED variables.

But unlike other variables LINGUAS is used by gettext and in case it is
unset gettext installs all translations.


Questions
---------

1. Do we want all packages to support LINGUAS if possible? It is
possible to leave gettext based package without LINGUAS and everything
will just work, but I think that it's good idea to make supported
languages visible to user through linguas_<lang> flags.

2. How should we handle case of unset LINGUAS in ebuild? Should we mimic
gettext and install all supported languages, using code like

LINGUAS=${LINGUAS-%UNSET%}
if test "%UNSET%" == "$LINGUAS"; then
	# install all supported languages
fi

? If yes, it's easy to write such code and put in eclass. If no, how do
we live with inconsistency that some packages will install all languages
some, will install nothing (document in handbook and add portage warning
in case LINGUAS is unset?)?

3. What is the purpose of strip-linguas function (mentioned in
devmanual)? It's clear what it does but I have no ideas why. Probably
similar code could be used in QA function that will gather supported
languages from po directories and warn maintainer that it's time to
update linguas_<lang> in IUSE (and probably later it could be expanded
to support qt packages too).



[1] or equivalently linguas_* is set in package.use

-- 
Peter.

Re: [gentoo-dev] LINGUAS handling

[Harald van Dijk]

On Wed, Jun 09, 2010 at 11:38:03AM +0400, Peter Volkov wrote:
> 1. Do we want all packages to support LINGUAS if possible? It is
> possible to leave gettext based package without LINGUAS and everything
> will just work, but I think that it's good idea to make supported
> languages visible to user through linguas_<lang> flags.

I agree, but I'd like to point out this would be a visible change in
default behaviour: the default would change from "install everything" to
"install nothing". For gettext-based packages, "install everything" is a
sane default, in my opinion.

> 2. How should we handle case of unset LINGUAS in ebuild? Should we mimic
> gettext and install all supported languages, using code like
> 
> LINGUAS=${LINGUAS-%UNSET%}
> if test "%UNSET%" == "$LINGUAS"; then
> 	# install all supported languages
> fi

Firstly, don't use == with test. It's not portable. The bash built-in
test supports ==. Other test implementations don't, such as the one from
GNU coreutils, and the built-in test in other shells, don't. If you use
test in a context where you're absolutely sure the built-in version will
be used, and the executing shell is bash, then I suppose you can use ==,
but at that point you're better off using [[ ]] anyway.

That said, to test if a variable is set, use ${VAR+set} over
${VAR-unset}. ${VAR-unset} evaluates to "unset" if VAR is unset, and if
VAR is set to the string "unset". I suppose that's why you used %UNSET%,
to reduce the possibility of accidents. You can reduce it to 0, using
for example

if [[ -z ${LINGUAS+set} ]]; then
  # LINGUAS is unset
fi

> ? If yes, it's easy to write such code and put in eclass. If no, how do
> we live with inconsistency that some packages will install all languages
> some, will install nothing (document in handbook and add portage warning
> in case LINGUAS is unset?)?

Unfortunately, consistency either way is bad. Making unset LINGUAS
install nothing changes gettext's design, when the whole idea behind
LINGUAS was to mimic gettext's design. Making unset LINGUAS install
everything causes massive disk space requirements for the default
settings for some packages such as openoffice. In my opinion, either of
those would be worse than having LINGUAS behave inconsistently.

> 3. What is the purpose of strip-linguas function (mentioned in
> devmanual)? It's clear what it does but I have no ideas why. Probably
> similar code could be used in QA function that will gather supported
> languages from po directories and warn maintainer that it's time to
> update linguas_<lang> in IUSE (and probably later it could be expanded
> to support qt packages too).

It's used for some packages that fail to build with certain LINGUAS
values. If I recall correctly, binutils had a bug where putting en_GB in
your LINGUAS made the build fail, for example. binutils doesn't support
en_GB anyway, so it gets filtered out,

[gentoo-dev] Re: LINGUAS handling

[Duncan]

Harald van Dijk posted on Wed, 09 Jun 2010 21:21:44 +0200 as excerpted:

> Firstly, don't use == with test. It's not portable. The bash built-in
> test supports ==. Other test implementations don't, such as the one from
> GNU coreutils, and the built-in test in other shells, don't. If you use
> test in a context where you're absolutely sure the built-in version will
> be used, and the executing shell is bash, then I suppose you can use ==,
> but at that point you're better off using [[ ]] anyway.

Good point about [[ ]].

> That said, to test if a variable is set, use ${VAR+set} over
> ${VAR-unset}. ${VAR-unset} evaluates to "unset" if VAR is unset, and if
> VAR is set to the string "unset". I suppose that's why you used %UNSET%,
> to reduce the possibility of accidents. You can reduce it to 0, using
> for example
> 
> if [[ -z ${LINGUAS+set} ]]; then
>   # LINGUAS is unset
> fi

While we're talking style, a question (well, a couple):

I've seen code like that posted frequently, but always wonder why the -z 
is even used at all.  Also, why use the if/then form?  Is that Gentoo 
style guidelines (like always using the brace-var form apparently is, 
${LINGUAS} vs. $LINGUAS)?  Why not simply (: instead of # to avoid an 
empty subshell):

[[ ${LINGUAS} ]] || {
	: LINGUAS is unset
}


>> If yes, it's easy to write such code and put in eclass. If no, how do
>> we live with inconsistency that some packages will install all
>> languages some, will install nothing (document in handbook and add
>> portage warning in case LINGUAS is unset?)?
> 
> Unfortunately, consistency either way is bad. Making unset LINGUAS
> install nothing changes gettext's design, when the whole idea behind
> LINGUAS was to mimic gettext's design. Making unset LINGUAS install
> everything causes massive disk space requirements for the default
> settings for some packages such as openoffice. In my opinion, either of
> those would be worse than having LINGUAS behave inconsistently.

++

>> 3. What is the purpose of strip-linguas function

> It's used for some packages that fail to build with certain LINGUAS
> values. If I recall correctly, binutils had a bug where putting en_GB in
> your LINGUAS made the build fail, for example. binutils doesn't support
> en_GB anyway, so it gets filtered out,

Here's a simple question I've wondered for awhile, not documented anywhere 
I could find, but should be.

What do those of us who only speak English (whatever form, US is fine) put 
in LINGUAS, to ensure no "extras" we can't use anyway get installed?

Currently, I have in one of my make.conf include files: LINGUAS="en", 
hoping to avoid the "everything installed" scenario above.  But then on 
binutils, I get the message "en" is not supported, which is fine, it 
works, but with it stripped, does that mean it's installing unnecessary 
language stuff that's of no use to me anyway?  Should it be "en_US" or 
"en_US.UTF-8" instead?  Or maybe it should be simply "C"?

A line or two in the l10n guide, plus possibly the handbook, telling what 
to set if "plain English" is "plain OK" to avoid the "unset installs it 
all" problem, would be useful.

And if there is no such single value available presently, as looks likely, 
how big a project would it be to add it?  Or perhaps an alternate 
approach, a table of packages and the setting to be added to a file in 
/etc/portage/env (or whatever), would be more appropriate?  I haven't a 
clue on this.  Perhaps all I need is pointed at the right documentation, 
but if so, that "right documentation" should probably be a bit more 
visible, as I've certainly looked.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] LINGUAS handling

[Peter Volkov]

В Срд, 09/06/2010 в 21:21 +0200, Harald van Dijk пишет:
> On Wed, Jun 09, 2010 at 11:38:03AM +0400, Peter Volkov wrote:
> > 1. Do we want all packages to support LINGUAS if possible? It is
> > possible to leave gettext based package without LINGUAS and everything
> > will just work, but I think that it's good idea to make supported
> > languages visible to user through linguas_<lang> flags.
> 
> I agree, but I'd like to point out this would be a visible change in
> default behaviour: the default would change from "install everything" to
> "install nothing". For gettext-based packages, "install everything" is a
> sane default, in my opinion.

Yup, but 1) this change will be visible during emerge -pv since new
linguas value are visible there, 2) we already have gettext packages
with linguas supported.

> > 2. How should we handle case of unset LINGUAS in ebuild? Should we mimic
> > gettext and install all supported languages, using code like
> > 
> > LINGUAS=${LINGUAS-%UNSET%}
> > if test "%UNSET%" == "$LINGUAS"; then
> > 	# install all supported languages
> > fi
> 
> Firstly, don't use == with test.

Thank you for suggestions! Actually it was just an illustration loosely
taken from bug 148702.

> Unfortunately, consistency either way is bad. Making unset LINGUAS
> install nothing changes gettext's design, when the whole idea behind
> LINGUAS was to mimic gettext's design. Making unset LINGUAS install
> everything causes massive disk space requirements for the default
> settings for some packages such as openoffice. In my opinion, either of
> those would be worse than having LINGUAS behave inconsistently.

Ok.

> > 3. What is the purpose of strip-linguas function
> 
> It's used for some packages that fail to build with certain LINGUAS
> values. If I recall correctly, binutils had a bug where putting en_GB in
> your LINGUAS made the build fail, for example. binutils doesn't support
> en_GB anyway, so it gets filtered out,

I see.

But what is preferred way for gettext packages? Define supported
languages in IUSE or use strip-linguas if required?

-- 
Peter.