[gentoo-dev] better handling of multibyte characters (nls/cjk/unicode)

[gentoo-dev] better handling of multibyte characters (nls/cjk/unicode)

[Mike Frysinger]

http://bugs.gentoo.org/show_bug.cgi?id=26561
<summary>
At the moment the relevant USE flags seem to be:
cjk  -  Adds support for Multi-byte character languages (Chinese, Japanese, 
Korean)

nls  -  Adds Native Language Support (using gettext - GNU locale utilities) 

If I want to enable unicode support or similar for an application (say, vim) 
only cjk seems applicable at the moment, and is a tad misleading (there are
hundreds of other languages in unicode!).
</summary>

please post your input ... personally i'd prefer if everyone just learned 
ASCII compliant languages ;)
-mike

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] better handling of multibyte characters (nls/cjk/unicode)

[Heinrich Wendel]

On Thursday 14 August 2003 01:34, Mike Frysinger wrote:
> http://bugs.gentoo.org/show_bug.cgi?id=26561
> <summary>
> At the moment the relevant USE flags seem to be:
> cjk  -  Adds support for Multi-byte character languages (Chinese, Japanese,
> Korean)
>
> nls  -  Adds Native Language Support (using gettext - GNU locale utilities)
>
> If I want to enable unicode support or similar for an application (say,
> vim) only cjk seems applicable at the moment, and is a tad misleading
> (there are hundreds of other languages in unicode!).
> </summary>
>
> please post your input ... personally i'd prefer if everyone just learned
> ASCII compliant languages ;)

hehe ;)

/me votes for unicode use-flag


> -mike
mfg, Heinrich :)

>
> --
> gentoo-dev@gentoo.org mailing list


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] better handling of multibyte characters (nls/cjk/unicode)

[Heinrich Wendel]

On Thursday 14 August 2003 01:38, Heinrich Wendel wrote:
> On Thursday 14 August 2003 01:34, Mike Frysinger wrote:
> > http://bugs.gentoo.org/show_bug.cgi?id=26561
> > <summary>
> > At the moment the relevant USE flags seem to be:
> > cjk  -  Adds support for Multi-byte character languages (Chinese,
> > Japanese, Korean)
> >
> > nls  -  Adds Native Language Support (using gettext - GNU locale
> > utilities)
> >
> > If I want to enable unicode support or similar for an application (say,
> > vim) only cjk seems applicable at the moment, and is a tad misleading
> > (there are hundreds of other languages in unicode!).
> > </summary>
> >
> > please post your input ... personally i'd prefer if everyone just learned
> > ASCII compliant languages ;)
>
> hehe ;)
>
> /me votes for unicode use-flag

should also be activated in all profiles by default.

>
> > -mike
>
> mfg, Heinrich :)
>
> > --
> > gentoo-dev@gentoo.org mailing list
>
> --
> gentoo-dev@gentoo.org mailing list


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] better handling of multibyte characters (nls/cjk/unicode)

[Mamoru KOMACHI]

Hi all,

At Thu, 14 Aug 2003 01:40:40 +0200,
Heinrich Wendel wrote:

> On Thursday 14 August 2003 01:38, Heinrich Wendel wrote:
> > On Thursday 14 August 2003 01:34, Mike Frysinger wrote:
> > > http://bugs.gentoo.org/show_bug.cgi?id=26561
> > > <summary>

> > > If I want to enable unicode support or similar for an application (say,
> > > vim) only cjk seems applicable at the moment, and is a tad misleading
> > > (there are hundreds of other languages in unicode!).
> > > </summary>

> > /me votes for unicode use-flag
> 
> should also be activated in all profiles by default.

Any objection if I add unicode USE flag (and make it default in all
profiles) ? We cannot simply convert cjk USE flag to unicode, but in
some cases unicode USE flag seems appropriate and we definitely need
it. (currently dev-python/wxPython and x11-libs/wxGTK have unicode
local USE flag and others use cjk or nls for unicode support)

If nobody objects (and hopefully many approve ;) ) I'll create it this
weekend.

-- 
Mamoru KOMACHI <usata@gentoo.org>
http://www.gentoo.org/~usata/

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] better handling of multibyte characters (nls/cjk/unicode)

[Paul de Vrieze]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thursday 30 October 2003 12:44, Mamoru KOMACHI wrote:
>
> Any objection if I add unicode USE flag (and make it default in all
> profiles) ? We cannot simply convert cjk USE flag to unicode, but in
> some cases unicode USE flag seems appropriate and we definitely need
> it. (currently dev-python/wxPython and x11-libs/wxGTK have unicode
> local USE flag and others use cjk or nls for unicode support)
>
> If nobody objects (and hopefully many approve ;) ) I'll create it this
> weekend.

That seems like a good idea.

A thing a bit unrelated is that it might be a good idea to create some 
documentation on how to setup gentoo for foreign languages. 

Paul

- -- 
Paul de Vrieze
Gentoo Developer
Mail: pauldv@gentoo.org
Homepage: http://www.devrieze.net
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQE/oQN5bKx5DBjWFdsRAj4BAJ9JnywETKx8XphaJ2vvGGxo+L36CACfdO0k
ZOBIBNvR3eKjWeiFbGA+yT4=
=Aq26
-----END PGP SIGNATURE-----


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] better handling of multibyte characters (nls/cjk/unicode)

[Spider]

[-- Attachment #1: Type: text/plain, Size: 1126 bytes --]

begin  quote
On Thu, 30 Oct 2003 13:26:17 +0100
Paul de Vrieze <pauldv@gentoo.org> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On Thursday 30 October 2003 12:44, Mamoru KOMACHI wrote:
> >
> > Any objection if I add unicode USE flag (and make it default in all
> > profiles) ? We cannot simply convert cjk USE flag to unicode, but in
> > some cases unicode USE flag seems appropriate and we definitely need
> > it. (currently dev-python/wxPython and x11-libs/wxGTK have unicode
> > local USE flag and others use cjk or nls for unicode support)
> >
> > If nobody objects (and hopefully many approve ;) ) I'll create it
> > this weekend.

unicode, or utf-8 ?  There ought to be some magic to convert locales to
utf-8 as well if this is enabled.








> A thing a bit unrelated is that it might be a good idea to create some
> documentation on how to setup gentoo for foreign languages. 
like this :
http://www.gentoo.org/doc/en/guide-localization.xml


//Spider
-- 
begin  .signature
This is a .signature virus! Please copy me into your .signature!
See Microsoft KB Article Q265230 for more information.
end

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] better handling of multibyte characters (nls/cjk/unicode)

[Paul de Vrieze]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thursday 30 October 2003 13:41, Spider wrote:
> begin  quote
> On Thu, 30 Oct 2003 13:26:17 +0100
>
> like this :
> http://www.gentoo.org/doc/en/guide-localization.xml
>

Yes, (it looked to me like a document on translating the guides). I would 
also like sections on things like input methods etc.

Paul

- -- 
Paul de Vrieze
Gentoo Developer
Mail: pauldv@gentoo.org
Homepage: http://www.devrieze.net
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQE/oRGhbKx5DBjWFdsRAkxwAJ0ROxNPnFnTDvhGYLvn52B51Wk3PQCguwtY
UF8+XZ3zEOZRZNniPHgKWmY=
=8fn5
-----END PGP SIGNATURE-----


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] better handling of multibyte characters (nls/cjk/unicode)

[Spider]

[-- Attachment #1: Type: text/plain, Size: 539 bytes --]

begin  quote
On Thu, 30 Oct 2003 14:26:42 +0100
Paul de Vrieze <pauldv@gentoo.org> wrote:

> > http://www.gentoo.org/doc/en/guide-localization.xml
> 
> Yes, (it looked to me like a document on translating the guides). I
> would  also like sections on things like input methods etc.


True, that would be nice addition to it.

Anyone here who has experience with that sort of thing?


//Spider
-- 
begin  .signature
This is a .signature virus! Please copy me into your .signature!
See Microsoft KB Article Q265230 for more information.
end

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] better handling of multibyte characters (nls/cjk/unicode)

[Svyatogor]

On Thu, 30 Oct 2003 14:43:59 +0100
Spider <spider@gentoo.org> wrote:

> begin  quote
> On Thu, 30 Oct 2003 14:26:42 +0100
> Paul de Vrieze <pauldv@gentoo.org> wrote:
> 
> > > http://www.gentoo.org/doc/en/guide-localization.xml
> > 
> > Yes, (it looked to me like a document on translating the guides). I
> > would  also like sections on things like input methods etc.
> 
> 
> True, that would be nice addition to it.
> 
> Anyone here who has experience with that sort of thing?

I am currently working on a document explaning how to set up Unicode locale
convert whatever you need to UTF-8 and configure various apps. I anyone could
write a section on input methods, (got no idea about how this stuff works) I'll be
happy to include it.

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] better handling of multibyte characters (nls/cjk/unicode)

[Mamoru KOMACHI]

At Thu, 30 Oct 2003 13:41:51 +0100,
Spider wrote:

> unicode, or utf-8 ?

I personally vote for unicode, but if it is going to be enabled by
default (i.e. Gentoo support UTF-8 immediately after installation),
utf8 seems adequate rather than unicode. I just don't want to add
utf7, utf16, utf32 USE flags for each encoding but have unicode USE
flag which will support Unicode encodings (if any).

>  There ought to be some magic to convert locales to utf-8 as well if
> this is enabled.

Even if the USE flag is enabled by default, it doesn't take any effect
unless each ebuild actually uses the USE flag. It may break wxGTK
related ebuilds as darkspecter suggested (thanks for the comment) and
I'm thinking that it isn't time to make the USE flag default yet
(liquidx is working on getting UTF-8 down on Gentoo system). Anyhow,
creating unicode/utf8 USE flag and making it default are two different
matters and we can discuss them separately.

--
Mamoru KOMACHI <usata@gentoo.org>
http://www.gentoo.org/~usata/

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] better handling of multibyte characters (nls/cjk/unicode)

[Alastair Tse]

[-- Attachment #1: Type: text/plain, Size: 1636 bytes --]

On Thu, 2003-10-30 at 14:09, Mamoru KOMACHI wrote:
> > unicode, or utf-8 ?
> 
> I personally vote for unicode, but if it is going to be enabled by
> default (i.e. Gentoo support UTF-8 immediately after installation),
> utf8 seems adequate rather than unicode. I just don't want to add
> utf7, utf16, utf32 USE flags for each encoding but have unicode USE
> flag which will support Unicode encodings (if any).

Exactly. "unicode" should denote adding support for UCS-* or UTF-*
encodings and internal representations. 

> >  There ought to be some magic to convert locales to utf-8 as well if
> > this is enabled.

> I'm thinking that it isn't time to make the USE flag default yet
> (liquidx is working on getting UTF-8 down on Gentoo system). Anyhow,
> creating unicode/utf8 USE flag and making it default are two different
> matters and we can discuss them separately.

From my point of view, "unicode" USE flag encompasses UTF-8 support. I
think they are the essentially the same problem. Think of it this way,
treating them as separate issues would be like treating iso2022-jp as a
separate issue to Japanese support.

Anyhow, the generation of UTF-8 locales can be accomplished by a patch
to glibc which could be applied depending on the unicode USE flag. I've
got one such patch around, I'll submit a bug to track it in bugzilla
soon.

Cheers,

-- 
Alastair 'liquidx' Tse
 >> Gentoo Developer
 >> http://www.liquidx.net/ | http://dev.gentoo.org/~liquidx/
 >> GPG Key : http://dev.gentoo.org/~liquidx/liquidx_gentoo_org.asc
 >> FingerPrint : 579A 9B0E 43E8 0E40 EE93  BB1C 38CE 1C7B 3907 14F6

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] better handling of multibyte characters (nls/cjk/unicode)

[Spider]

[-- Attachment #1: Type: text/plain, Size: 1971 bytes --]

begin  quote

Combined reply below.


> > > unicode, or utf-8 ?
> > 
> > I personally vote for unicode, but if it is going to be enabled by
> > default (i.e. Gentoo support UTF-8 immediately after installation),
> > utf8 seems adequate rather than unicode. I just don't want to add
> > utf7, utf16, utf32 USE flags for each encoding but have unicode USE
> > flag which will support Unicode encodings (if any).
> 
> Exactly. "unicode" should denote adding support for UCS-* or UTF-*
> encodings and internal representations. 



I agree here, just wanted to get a statement out that it should
encompass both, since theres a few interesting aspects of UTF8 locales
and filenames. (Namely, gtk 1.2 doesn't like utf8 filenames , which
makes me want to deprecate it. I've been looking for a solution and for
all that I can see no -easy- one was avaiable. )




> 
> > >  There ought to be some magic to convert locales to utf-8 as well
> > >  if this is enabled.
> 
> > I'm thinking that it isn't time to make the USE flag default yet
> > (liquidx is working on getting UTF-8 down on Gentoo system). Anyhow,
> > creating unicode/utf8 USE flag and making it default are two
> > different> matters and we can discuss them separately.
> 
<cut>
 
> Anyhow, the generation of UTF-8 locales can be accomplished by a patch
> to glibc which could be applied depending on the unicode USE flag.
> I've got one such patch around, I'll submit a bug to track it in
> bugzilla soon.


Yeah, I haven't spoken for making it Default, even if I could go radical
 and demand it default and that we remove all legacy software that
doesn't cut it....  *G*   (No, I don't intend to ;)

However, as for the locale genereation patch, that would be very
welcome, and i think that alone would make a "unicode" useflag valid.
I'm all for this. 

//Spider


-- 
begin  .signature
This is a .signature virus! Please copy me into your .signature!
See Microsoft KB Article Q265230 for more information.
end

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] better handling of multibyte characters (nls/cjk/unicode)

[Bartosch Pixa]

[-- Attachment #1: Type: text/plain, Size: 654 bytes --]

On Thu, 2003-10-30 at 12:44, Mamoru KOMACHI wrote:

> Any objection if I add unicode USE flag (and make it default in all
> profiles) ? We cannot simply convert cjk USE flag to unicode, but in
> some cases unicode USE flag seems appropriate and we definitely need
> it. (currently dev-python/wxPython and x11-libs/wxGTK have unicode
> local USE flag and others use cjk or nls for unicode support)
> 
> If nobody objects (and hopefully many approve ;) ) I'll create it this
> weekend.

wxGTK with unicode enabled breaks most apps that link to it.

regards
-- 
Bartosch Pixa
Gentoo Linux Developer                    http://www.gentoo.org

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] better handling of multibyte characters (nls/cjk/unicode)

[Alastair Tse]

[-- Attachment #1: Type: text/plain, Size: 835 bytes --]

On Thu, 2003-10-30 at 11:44, Mamoru KOMACHI wrote:
> > > /me votes for unicode use-flag
> > 
> > should also be activated in all profiles by default.
> 
> Any objection if I add unicode USE flag (and make it default in all
> profiles) ? We cannot simply convert cjk USE flag to unicode, but in

wxGTK is currently using a local unicode useflag to enable unicode
support. However, this is experimental so I rather not have unicode
default in all profiles just yet. By compiling with the unicode useflag,
you'll end up with certain wxGTK apps failing to merge.

Cheers,
-- 
Alastair 'liquidx' Tse
 >> Gentoo Developer
 >> http://www.liquidx.net/ | http://dev.gentoo.org/~liquidx/
 >> GPG Key : http://dev.gentoo.org/~liquidx/liquidx_gentoo_org.asc
 >> FingerPrint : 579A 9B0E 43E8 0E40 EE93  BB1C 38CE 1C7B 3907 14F6

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] better handling of multibyte characters (nls/cjk/unicode)

[Mart Raudsepp]

Alastair Tse kirjutas N, 30.10.2003 kell 18:08:
> On Thu, 2003-10-30 at 11:44, Mamoru KOMACHI wrote:
> > > > /me votes for unicode use-flag
> > > 
> > > should also be activated in all profiles by default.
> > 
> > Any objection if I add unicode USE flag (and make it default in all
> > profiles) ? We cannot simply convert cjk USE flag to unicode, but in
> 
> wxGTK is currently using a local unicode useflag to enable unicode
> support. However, this is experimental so I rather not have unicode
> default in all profiles just yet. By compiling with the unicode useflag,
> you'll end up with certain wxGTK apps failing to merge.

Sorry for a really delayed answer.
To my knowledge unicode support in wxGTK isn't experimental but rather
suggested but not enforced.
The reason why certain wxGTK apps fail to merge is that they don't
comform with the unicode wxWindows prerequisites.
Quote from wxWindows docs:

"The important conclusion is that if you use wxChar instead of char,
avoid using C style strings and use wxString instead and don't forget to
enclose all string literals inside wxT() macro, your program
automatically becomes (almost) Unicode compliant!"

Those certain apps fail to follow the above. And to be honest, my own
wxWindows projects fail to follow it aswell, but that will be fixed soon
;)
Oh, and instead of wxT() one should use _() to automagically add i18n
support aswell.

So my conclusion is that we still should make USE=unicode be default
(put into the profiles), but ebuilds that depend upon wxGTK should be
checked if the wxWindows app is unicode aware or not. If it isn't then
it should simply remove the unicode flag in the ebuild. Personally I
would add the unicode flag to make.conf and wxGTK apps that don't
compile emerge with USE=-unicode, also I'd prod the app devs to make it
follow wxWindows unicode prerequisites that aren't hard to follow
(personal experience).


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] better handling of multibyte characters (nls/cjk/unicode)

[Alastair Tse]

[-- Attachment #1: Type: text/plain, Size: 1200 bytes --]


> > wxGTK is currently using a local unicode useflag to enable unicode
> > support. However, this is experimental so I rather not have unicode
> > default in all profiles just yet. By compiling with the unicode useflag,
> > you'll end up with certain wxGTK apps failing to merge.
> 
> Sorry for a really delayed answer.
> To my knowledge unicode support in wxGTK isn't experimental but rather
> suggested but not enforced.

Yes, I mean experimental is in for Gentoo, not for wxGTK. I've only
enabled that option a couple of weeks ago because some people wanted to
test wxGTK with unicode support.

> The reason why certain wxGTK apps fail to merge is that they don't
> comform with the unicode wxWindows prerequisites.

Yes, if you would like to submit patches against those programs that
don't conform to those requirements, that would be most appreciated by
the developers. Right now, because at least half of the applications
that use wxGTK don't support unicode, there is no way I'm going to
enable that as default for wxGTK until that is fixed.

Cheers,
-- 
Alastair 'liquidx' Tse
 >> Gentoo Developer
 >> http://www.liquidx.net/ | http://dev.gentoo.org/~liquidx/


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] better handling of multibyte characters (nls/cjk/unicode)

[Svyatogor]

On Thu, 30 Oct 2003 20:44:33 +0900
Mamoru KOMACHI <usata@gentoo.org> wrote:
> Any objection if I add unicode USE flag (and make it default in all
> profiles) ? We cannot simply convert cjk USE flag to unicode, but in
> some cases unicode USE flag seems appropriate and we definitely need
> it. (currently dev-python/wxPython and x11-libs/wxGTK have unicode
> local USE flag and others use cjk or nls for unicode support)
> 
> If nobody objects (and hopefully many approve ;) ) I'll create it this
> weekend.

I'd love to see this flag! However this should be *off* by default cause there are
some applications (like MC for example) which have troubles understanding unicode.

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] better handling of multibyte characters (nls/cjk/unicode)

[Mamoru KOMACHI]

Hi all, thanks for commenting on this issue ;)

At Thu, 30 Oct 2003 16:08:52 +0000,
Alastair Tse wrote:

> wxGTK is currently using a local unicode useflag to enable unicode
> support. However, this is experimental so I rather not have unicode
> default in all profiles just yet. By compiling with the unicode useflag,
> you'll end up with certain wxGTK apps failing to merge.

At Fri, 31 Oct 2003 15:43:05 +0200,
Svyatogor wrote:

> I'd love to see this flag! However this should be *off* by default
> cause there are some applications (like MC for example) which have
> troubles understanding unicode.

So what I will do is:

1. Create global USE flag named "unicode" which enables
   Unicode(including UTF-7/UTF-8/UTF-16 and so on) support.
2. Don't make it default in any profiles. (We will consider it later
   when those applications settle down.)

Everybody happy with this? I hope it will be a good starting point for
integrating Unicoded support into Gentoo Linux.

-- 
Mamoru KOMACHI <usata@gentoo.org>
http://www.gentoo.org/~usata/

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] better handling of multibyte characters (nls/cjk/unicode)

[Spider]

[-- Attachment #1: Type: text/plain, Size: 948 bytes --]

begin  quote
On Fri, 31 Oct 2003 23:24:52 +0900
Mamoru KOMACHI <usata@gentoo.org> wrote:

> 
> So what I will do is:
> 
> 1. Create global USE flag named "unicode" which enables
>    Unicode(including UTF-7/UTF-8/UTF-16 and so on) support.
> 2. Don't make it default in any profiles. (We will consider it later
>    when those applications settle down.)
> 
> Everybody happy with this? I hope it will be a good starting point for
> integrating Unicoded support into Gentoo Linux.
> 

Sounds great.


proposed addition to baselayout:

if [ `use unicode` ] ;
then
cat >> ${D}/etc/rc.conf << EOF
# This setting enables a default UTF-8 locale for your system. 
# please look inside /usr/share/locale for more examples
LANG="en_US.UTF-8"

EOF
fi

This is rough, but something like it perhaps?


//Spider



-- 
begin  .signature
This is a .signature virus! Please copy me into your .signature!
See Microsoft KB Article Q265230 for more information.
end

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] better handling of multibyte characters (nls/cjk/unicode)

[Mamoru KOMACHI]

Hi,

At Fri, 31 Oct 2003 16:05:30 +0100,
Spider wrote:

> > 1. Create global USE flag named "unicode" which enables
> >    Unicode(including UTF-7/UTF-8/UTF-16 and so on) support.
> > 2. Don't make it default in any profiles. (We will consider it later
> >    when those applications settle down.)

> Sounds great.

I've just added unicode USE flag :)

> proposed addition to baselayout:
> 
> if [ `use unicode` ] ;
> then
> cat >> ${D}/etc/rc.conf << EOF
> # This setting enables a default UTF-8 locale for your system. 
> # please look inside /usr/share/locale for more examples
> LANG="en_US.UTF-8"
> 
> EOF
> fi
> 
> This is rough, but something like it perhaps?

If http://bugs.gentoo.org/show_bug.cgi?id=9988 "setting LANG variable"
is implemented we may think about it, but I don't think the system
should set LANG variable (at least) until then.

-- 
Mamoru KOMACHI <usata@gentoo.org>
http://www.gentoo.org/~usata/

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] better handling of multibyte characters (nls/cjk/unicode)

[Spider]

[-- Attachment #1: Type: text/plain, Size: 645 bytes --]

begin  quote
On Sat, 01 Nov 2003 21:06:56 +0900
Mamoru KOMACHI <usata@gentoo.org> wrote:

> 
> I've just added unicode USE flag :)

Good.


> If http://bugs.gentoo.org/show_bug.cgi?id=9988 "setting LANG variable"
> is implemented we may think about it, but I don't think the system
> should set LANG variable (at least) until then.
> 


Well, ok.  Except that if you use utf-8 enabled locales and the like,
you really want to have LANG set to a utf-8 locale, rather than a
"normal" one. 
//Spider


-- 
begin  .signature
This is a .signature virus! Please copy me into your .signature!
See Microsoft KB Article Q265230 for more information.
end

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] better handling of multibyte characters (nls/cjk/unicode)

[Martin Schlemmer]

[-- Attachment #1: Type: text/plain, Size: 1390 bytes --]

On Fri, 2003-10-31 at 17:05, Spider wrote:
> begin  quote
> On Fri, 31 Oct 2003 23:24:52 +0900
> Mamoru KOMACHI <usata@gentoo.org> wrote:
> 
> > 
> > So what I will do is:
> > 
> > 1. Create global USE flag named "unicode" which enables
> >    Unicode(including UTF-7/UTF-8/UTF-16 and so on) support.
> > 2. Don't make it default in any profiles. (We will consider it later
> >    when those applications settle down.)
> > 
> > Everybody happy with this? I hope it will be a good starting point for
> > integrating Unicoded support into Gentoo Linux.
> > 
> 
> Sounds great.
> 
> 
> proposed addition to baselayout:
> 
> if [ `use unicode` ] ;
> then
> cat >> ${D}/etc/rc.conf << EOF
> # This setting enables a default UTF-8 locale for your system. 
> # please look inside /usr/share/locale for more examples
> LANG="en_US.UTF-8"
> 
> EOF
> fi
> 
> This is rough, but something like it perhaps?

I do not see why.  If you enable utf8 support, it does not
mean you want it in console.  The user should specify his
LANG himself, as well as the fact that it should rather
be:

ELANG="en_US.UTF-8"

and then get it to propagate only to the users, as there are
still a lot of things breaking during build with LANG, et el
set.


Thanks,

-- 

Martin Schlemmer
Gentoo Linux Developer, Desktop/System Team Developer
Cape Town, South Africa



[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] better handling of multibyte characters (nls/cjk/unicode)

[Spider]

[-- Attachment #1: Type: text/plain, Size: 845 bytes --]

begin  quote
On Sat, 01 Nov 2003 21:10:50 +0200
Martin Schlemmer <azarah@gentoo.org> wrote:



> I do not see why.  If you enable utf8 support, it does not
> mean you want it in console.  The user should specify his
> LANG himself, as well as the fact that it should rather be:
> 
> ELANG="en_US.UTF-8"
> 
> and then get it to propagate only to the users, as there are
> still a lot of things breaking during build with LANG, et el
> set.

Ahh, thats very true. Except that I'd still want it documented. Yep,
I'm having a drive for UTF-8 right now, I've ran into places where I
need filenames in UTF-8 and most things have started to break otherwise.
 unpleasant compability problems. 


//Spider


-- 
begin  .signature
This is a .signature virus! Please copy me into your .signature!
See Microsoft KB Article Q265230 for more information.
end

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] better handling of multibyte characters (nls/cjk/unicode)

[Stanislav Brabec]

V St, 05. 11. 2003 v 14:50, Spider píše:

> Ahh, thats very true. Except that I'd still want it documented. Yep,
> I'm having a drive for UTF-8 right now, I've ran into places where I
> need filenames in UTF-8 and most things have started to break otherwise.
>  unpleasant compability problems. 

These problems are currently discussed in glib bugzilla:
http://bugzilla.gnome.org/show_bug.cgi?id=114068


And there are my ideas, what needs to be done for seamless UTF-8
support:

0) baselayout: Add "official" way to set system default locale:
/etc/rc.conf or /etc/env.d/{number}locale (or similar).

1) glibc: Generate UTF-8 locales for all languages, if unicode USE flag
is on. Either as patch, or ex-post:
for i in ...... ; do
localedef -i $i -f UTF-8 -u charids.894 $i.UTF-8
done

2) Enable UTF-8 locales in GDM:

Two alternatives:

# Create UTF-8 for all locales.
sed 's:\([^ 	]*\)\([ 	][ 	]*\)\(.*\)UTF-8,\(.*\):\1	\2\4,\3UTF-8\
\1(UTF-8)\2\3UTF-8,\4:g' <config/locale.alias | sed 's:^\([^ 	][^ 	][^
	][^ 	][^ 	][^ 	][^ 	][^ 	]\)*(UTF-8):&	:' >config/locale.alias~
mv config/locale.alias~ config/locale.alias

# Prefer non UTF-8 for all locales.
sed 's:\([^ 	]*\)\([ 	][ 	]*\)\(.*\)UTF-8,\(.*\):\1	\2\4,\3UTF-8:g'
<config/locale.alias | sed 's:^\([^ 	][^ 	][^ 	][^ 	][^ 	][^ 	][^ 	][^
	]\)*(UTF-8):&	:' >config/locale.alias~
mv config/locale.alias~ config/locale.alias

3) Grab Redhat patches for most problematic applications (mc, slang,
maybe ncurses).



Setting language mini HOWTO:


LANG

LANG sets overall NLS support. Use LANG in form cs_CZ, cs_CZ.UTF-8 or
cs_CZ.UTF-8@variant. Avoid cs (X does not like it), cs_CZ.utf8 (again -
X doesn't like it) and czech (X and some old apps does not like it -
very old gtk). Test: run LANG={mysetup} xcalc. No warning must appear.

Note: localedef --list shows cs_CZ.utf8. It's because glibc ignores all
'-' characters and converts charset to lowercase.


LANGUAGE

LANGUAGE can be optionally set to list of languages for searching
messages, e. g. cs_CZ:sk_SK. Applications translated to cs will use cs,
applications translated to sk but not cs will use sk.

Warning: Some desktop managers with ability to set session language
(GDM) does not properly set/reset this variable and it is inherited to
all locale setups. It can cause strange setups.


LC_{category} - Set it for special purposes, if you want to use
different locales for messages, ctypes, collating etc.

LC_ALL - no need to set in environment. It nearly duplicates LANG. Nice
to use in scripts, not environment.

-- 
Stanislav Brabec
http://www.penguin.cz/~utx, ICQ 116020046

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] better handling of multibyte characters (nls/cjk/unicode)

[Stanislav Brabec]

gabor wrote:

> could you explain to me what's the difference between LC_ALL and LANG?
> 
> until now i'm using export LC_ALL=en_US.UTF-8 in /etc/profile...
> should i change it to LANG=en_US.UTF-8?

If you don't use other LC_{category}, there is no difference.

But if you use it with these variables, there it is:

LANG is fallback, overridden by all other LC_* variables.

LC_ALL is enforcement, and overrides LANG and all LC_{category}
variables. It means: if you define LC_ALL, sessing LC_{category} has no
effect.

There is an example:

utx$ ( export LANG=C LC_NUMERIC=C LC_ALL=C ; printf "%'5.3f\\n" 1 )
1.000
utx$ ( export LANG=C LC_NUMERIC=C LC_ALL=cs_CZ ; printf "%'5.3f\\n" 1 )
1,000
utx$ ( export LANG=cs_CZ LC_NUMERIC=C LC_ALL=C ; printf "%'5.3f\\n" 1 )
1.000
utx$ ( export LANG=C LC_NUMERIC=cs_CZ LC_ALL=C ; printf "%'5.3f\\n" 1 )
1.000
utx$ ( export LANG=C LC_NUMERIC=cs_CZ ; printf "%'5.3f\\n" 1 )
1,000

Note: Script programmers do often a mistake, using following
construction:
LC_TIME=C date

It works incorrectly, if you have LC_ALL set to any language.

-- 
Stanislav Brabec
http://www.penguin.cz/~utx, ICQ 116020046

--
gentoo-dev@gentoo.org mailing list