[gentoo-embedded] glibc+iconv

[gentoo-embedded] glibc+iconv

[Sergey Mironov]

Hello. I have my arm-unknown-linux-gnueabi system which uses glibc and
busybox. Recently I realised that iconv program doesn't exist in the
tree. I've compared host's and target's glibc file lists and found
that arm's version doesn't install anything to
/usr/arm-unknown-linux-gnueabi/usr/bin. What can be the cause of this?
Small C program shows that iconv_open function returns error code just
like if there is no iconv at all. How shoud I make arm's iconv work?

Thanks in advance,
Sergey

Re: [gentoo-embedded] glibc+iconv

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 704 bytes --]

On Thursday 13 October 2011 15:15:40 Sergey Mironov wrote:
> Hello. I have my arm-unknown-linux-gnueabi system which uses glibc and
> busybox. Recently I realised that iconv program doesn't exist in the
> tree. I've compared host's and target's glibc file lists and found
> that arm's version doesn't install anything to
> /usr/arm-unknown-linux-gnueabi/usr/bin. What can be the cause of this?
> Small C program shows that iconv_open function returns error code just
> like if there is no iconv at all. How shoud I make arm's iconv work?

see glibc's files/eblits/src_compile.eblit:
	is_crosscompile && sed -i '1ibuild-programs = no' config.make

i imagine disabling that line will get you `iconv`
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-embedded] glibc+iconv

[Sergey Mironov]

2011/10/14 Mike Frysinger <vapier@gentoo.org>:
> On Thursday 13 October 2011 15:15:40 Sergey Mironov wrote:
>> Hello. I have my arm-unknown-linux-gnueabi system which uses glibc and
>> busybox. Recently I realised that iconv program doesn't exist in the
>> tree. I've compared host's and target's glibc file lists and found
>> that arm's version doesn't install anything to
>> /usr/arm-unknown-linux-gnueabi/usr/bin. What can be the cause of this?
>> Small C program shows that iconv_open function returns error code just
>> like if there is no iconv at all. How shoud I make arm's iconv work?
>
> see glibc's files/eblits/src_compile.eblit:
>        is_crosscompile && sed -i '1ibuild-programs = no' config.make
>
> i imagine disabling that line will get you `iconv`
> -mike
>

Thanks, I will try it!  Am I understand correctly that iconv itself
(library, not program) is a part of glibc and it is not possible to
exclude it during the build? So despite the lack of program I should
still have fully functional library.

Re: [gentoo-embedded] glibc+iconv

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 1685 bytes --]

On Thursday 13 October 2011 19:30:14 Sergey Mironov wrote:
> 2011/10/14 Mike Frysinger:
> > On Thursday 13 October 2011 15:15:40 Sergey Mironov wrote:
> >> Hello. I have my arm-unknown-linux-gnueabi system which uses glibc and
> >> busybox. Recently I realised that iconv program doesn't exist in the
> >> tree. I've compared host's and target's glibc file lists and found
> >> that arm's version doesn't install anything to
> >> /usr/arm-unknown-linux-gnueabi/usr/bin. What can be the cause of this?
> >> Small C program shows that iconv_open function returns error code just
> >> like if there is no iconv at all. How shoud I make arm's iconv work?
> > 
> > see glibc's files/eblits/src_compile.eblit:
> >        is_crosscompile && sed -i '1ibuild-programs = no' config.make
> > 
> > i imagine disabling that line will get you `iconv`
> 
> Thanks, I will try it!  Am I understand correctly that iconv itself
> (library, not program) is a part of glibc and it is not possible to
> exclude it during the build? So despite the lack of program I should
> still have fully functional library.

your understanding is correct, however ........

iconv() is part of glibc but it relies on all of the gconv shared libs found 
in /usr/$CTARGET/usr/lib/gconv/ to do its actually work.  which we also 
incidentally delete when building the cross-compiler glibc (see 
src_install.eblit and look for "gconv").

you aren't the first person to find this behavior undesirable, and when i 
implemented it, it was more of "let's save space on things i don't think 
anyone will use".  but if people are using it, then installing these things 
probably makes sense.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-embedded] glibc+iconv

[Sergey Mironov]

2011/10/14 Mike Frysinger <vapier@gentoo.org>:
> On Thursday 13 October 2011 19:30:14 Sergey Mironov wrote:
>> 2011/10/14 Mike Frysinger:
>> > On Thursday 13 October 2011 15:15:40 Sergey Mironov wrote:
>> >> Hello. I have my arm-unknown-linux-gnueabi system which uses glibc and
>> >> busybox. Recently I realised that iconv program doesn't exist in the
>> >> tree. I've compared host's and target's glibc file lists and found
>> >> that arm's version doesn't install anything to
>> >> /usr/arm-unknown-linux-gnueabi/usr/bin. What can be the cause of this?
>> >> Small C program shows that iconv_open function returns error code just
>> >> like if there is no iconv at all. How shoud I make arm's iconv work?
>> >
>> > see glibc's files/eblits/src_compile.eblit:
>> >        is_crosscompile && sed -i '1ibuild-programs = no' config.make
>> >
>> > i imagine disabling that line will get you `iconv`
>>
>> Thanks, I will try it!  Am I understand correctly that iconv itself
>> (library, not program) is a part of glibc and it is not possible to
>> exclude it during the build? So despite the lack of program I should
>> still have fully functional library.
>
> your understanding is correct, however ........
>
> iconv() is part of glibc but it relies on all of the gconv shared libs found
> in /usr/$CTARGET/usr/lib/gconv/ to do its actually work.  which we also
> incidentally delete when building the cross-compiler glibc (see
> src_install.eblit and look for "gconv").
>
> you aren't the first person to find this behavior undesirable, and when i
> implemented it, it was more of "let's save space on things i don't think
> anyone will use".  but if people are using it, then installing these things
> probably makes sense.
> -mike
>

Well, maybe I really want something strange. I guess, people often
install another, non-cross-compiled version of glibc on top of initial
one. I've tried to do so, but found that cross-emerge complains about
conflicts - it simply doesn't treat initial glibc as a package
installed on target. I saw 2 ways - either edit package.provided and
don't install new glibc or disable conflicts detection and overwrite
some target's  /lib* and /usr/lib/* files. I've chosen first way since
I am afraid of getting a mess of two glibc's compiled with different
tools. But how do you (or other people) act in this situation?

By the way, I also had to handcopy libstdc++.so from
/usr/lib/gcc/$CTARGET to /usr/$CTARGET/lib to make C++ programs work.
It is another thing which makes me thinking of installig full glibc on
top of cross-one.

Sergey

Re: [gentoo-embedded] glibc+iconv

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 3504 bytes --]

On Friday 14 October 2011 06:11:40 Sergey Mironov wrote:
> 2011/10/14 Mike Frysinger:
> > On Thursday 13 October 2011 19:30:14 Sergey Mironov wrote:
> >> 2011/10/14 Mike Frysinger:
> >> > On Thursday 13 October 2011 15:15:40 Sergey Mironov wrote:
> >> >> Hello. I have my arm-unknown-linux-gnueabi system which uses glibc
> >> >> and busybox. Recently I realised that iconv program doesn't exist in
> >> >> the tree. I've compared host's and target's glibc file lists and
> >> >> found that arm's version doesn't install anything to
> >> >> /usr/arm-unknown-linux-gnueabi/usr/bin. What can be the cause of
> >> >> this? Small C program shows that iconv_open function returns error
> >> >> code just like if there is no iconv at all. How shoud I make arm's
> >> >> iconv work?
> >> > 
> >> > see glibc's files/eblits/src_compile.eblit:
> >> >        is_crosscompile && sed -i '1ibuild-programs = no' config.make
> >> > 
> >> > i imagine disabling that line will get you `iconv`
> >> 
> >> Thanks, I will try it!  Am I understand correctly that iconv itself
> >> (library, not program) is a part of glibc and it is not possible to
> >> exclude it during the build? So despite the lack of program I should
> >> still have fully functional library.
> > 
> > your understanding is correct, however ........
> > 
> > iconv() is part of glibc but it relies on all of the gconv shared libs
> > found in /usr/$CTARGET/usr/lib/gconv/ to do its actually work.  which we
> > also incidentally delete when building the cross-compiler glibc (see
> > src_install.eblit and look for "gconv").
> > 
> > you aren't the first person to find this behavior undesirable, and when i
> > implemented it, it was more of "let's save space on things i don't think
> > anyone will use".  but if people are using it, then installing these
> > things probably makes sense.
> 
> Well, maybe I really want something strange. I guess, people often
> install another, non-cross-compiled version of glibc on top of initial
> one. I've tried to do so, but found that cross-emerge complains about
> conflicts - it simply doesn't treat initial glibc as a package
> installed on target. I saw 2 ways - either edit package.provided and
> don't install new glibc or disable conflicts detection and overwrite
> some target's  /lib* and /usr/lib/* files. I've chosen first way since
> I am afraid of getting a mess of two glibc's compiled with different
> tools. But how do you (or other people) act in this situation?

indeed, this is the current wonky behavior.  i guess the thing to do would be 
two fold:
	- remove the disabling of tools/supplemental files so the cross-compiler 
glibc has all the same files as sys-libs/glibc
	- have crossdev automatically add a package.provided entry to the 
/usr/$CTARGET/etc/ tree for sys-libs/glibc

> By the way, I also had to handcopy libstdc++.so from
> /usr/lib/gcc/$CTARGET to /usr/$CTARGET/lib to make C++ programs work.
> It is another thing which makes me thinking of installig full glibc on
> top of cross-one.

yes, the /usr/$CTARGET tree has no gcc files installed at all.  in native 
installs, we've got /etc/env.d/ which adds the internal gcc paths via LDPATH 
to ld.so.conf, and then ldso at runtime finds libstdc++.so and friends.  but 
gcc is not installed at all in /usr/$CTARGET.  in this case, you should be 
able to emerge gcc into /usr/$CTARGET since none of the cross-compiler gcc 
files should be in there ...
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-embedded] glibc+iconv

[Sergey Mironov]

2011/10/14 Mike Frysinger <vapier@gentoo.org>:
> On Friday 14 October 2011 06:11:40 Sergey Mironov wrote:
>> 2011/10/14 Mike Frysinger:
>> > On Thursday 13 October 2011 19:30:14 Sergey Mironov wrote:
>> >> 2011/10/14 Mike Frysinger:
>> >> > On Thursday 13 October 2011 15:15:40 Sergey Mironov wrote:
>> >> >> Hello. I have my arm-unknown-linux-gnueabi system which uses glibc
>> >> >> and busybox. Recently I realised that iconv program doesn't exist in
>> >> >> the tree. I've compared host's and target's glibc file lists and
>> >> >> found that arm's version doesn't install anything to
>> >> >> /usr/arm-unknown-linux-gnueabi/usr/bin. What can be the cause of
>> >> >> this? Small C program shows that iconv_open function returns error
>> >> >> code just like if there is no iconv at all. How shoud I make arm's
>> >> >> iconv work?
>> >> >
>> >> > see glibc's files/eblits/src_compile.eblit:
>> >> >        is_crosscompile && sed -i '1ibuild-programs = no' config.make
>> >> >
>> >> > i imagine disabling that line will get you `iconv`
>> >>
>> >> Thanks, I will try it!  Am I understand correctly that iconv itself
>> >> (library, not program) is a part of glibc and it is not possible to
>> >> exclude it during the build? So despite the lack of program I should
>> >> still have fully functional library.
>> >
>> > your understanding is correct, however ........
>> >
>> > iconv() is part of glibc but it relies on all of the gconv shared libs
>> > found in /usr/$CTARGET/usr/lib/gconv/ to do its actually work.  which we
>> > also incidentally delete when building the cross-compiler glibc (see
>> > src_install.eblit and look for "gconv").
>> >
>> > you aren't the first person to find this behavior undesirable, and when i
>> > implemented it, it was more of "let's save space on things i don't think
>> > anyone will use".  but if people are using it, then installing these
>> > things probably makes sense.
>>
>> Well, maybe I really want something strange. I guess, people often
>> install another, non-cross-compiled version of glibc on top of initial
>> one. I've tried to do so, but found that cross-emerge complains about
>> conflicts - it simply doesn't treat initial glibc as a package
>> installed on target. I saw 2 ways - either edit package.provided and
>> don't install new glibc or disable conflicts detection and overwrite
>> some target's  /lib* and /usr/lib/* files. I've chosen first way since
>> I am afraid of getting a mess of two glibc's compiled with different
>> tools. But how do you (or other people) act in this situation?
>
> indeed, this is the current wonky behavior.  i guess the thing to do would be
> two fold:
>        - remove the disabling of tools/supplemental files so the cross-compiler
> glibc has all the same files as sys-libs/glibc
>        - have crossdev automatically add a package.provided entry to the
> /usr/$CTARGET/etc/ tree for sys-libs/glibc

Ok, thank you for explanations. I'll try to copy ebuild into my
target's overlay and make neccesary modifications.

>> By the way, I also had to handcopy libstdc++.so from
>> /usr/lib/gcc/$CTARGET to /usr/$CTARGET/lib to make C++ programs work.
>> It is another thing which makes me thinking of installig full glibc on
>> top of cross-one.
>
> yes, the /usr/$CTARGET tree has no gcc files installed at all.  in native
> installs, we've got /etc/env.d/ which adds the internal gcc paths via LDPATH
> to ld.so.conf, and then ldso at runtime finds libstdc++.so and friends.  but
> gcc is not installed at all in /usr/$CTARGET.  in this case, you should be
> able to emerge gcc into /usr/$CTARGET since none of the cross-compiler gcc
> files should be in there ...
> -mike
>

Yes, looks like I was wrong, libstdc++ is actually a library from gcc,
not from glibc. Unfortunately, installing gcc on target  may be
expensive in my case, since we have some image size constraints. And
we don't need compilers to be on target actually. Well, copying from
/usr/lib/gcc/$CTARGET is not too bad since it shouldn't  lead to
conflicts. Maybe it also will be fine to delegate this task to
crossdev one day.

Sergey

Re: [gentoo-embedded] glibc+iconv

[Ed W]

> you aren't the first person to find this behavior undesirable, and when i 
> implemented it, it was more of "let's save space on things i don't think 
> anyone will use".  but if people are using it, then installing these things 
> probably makes sense.
> -mike

I concede to being somewhat confused on how all the various pieces
integrate, but iconv seems like an increasingly useful piece, even if
only in a flawed capability that supports only 7 bit to UTF-8 and vice
versa.

Some experimentation with latest uclibc suggests that it has some
working iconv capability, although it appears to also cause a limited
locale implementation to be pulled in (which I really don't have a
requirement for).  I haven't tried to use it in anger, but for example
it allows a bunch of ebuilds to compile which previously moaned about
missing iconv implementations (admitedly those with tests claiming that
the uclibc iconv is incomplete...)

Just my (rather ignorant) 2p

Ed

Re: [gentoo-embedded] glibc+iconv

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 1464 bytes --]

On Tuesday 18 October 2011 05:27:32 Ed W wrote:
> > you aren't the first person to find this behavior undesirable, and when i
> > implemented it, it was more of "let's save space on things i don't think
> > anyone will use".  but if people are using it, then installing these
> > things probably makes sense.
> 
> I concede to being somewhat confused on how all the various pieces
> integrate, but iconv seems like an increasingly useful piece, even if
> only in a flawed capability that supports only 7 bit to UTF-8 and vice
> versa.

the issue is that the current cross-compilers delete the `iconv` binary that 
is cross-compile to run on CHOST, not CBUILD.  the native CBUILD `iconv` is 
still available.

if you want to use sys-libs/glibc for a target, atm you need to compile it for 
the target: CBUILD=x86_64 CHOST=arm CTARGET=arm.

> Some experimentation with latest uclibc suggests that it has some
> working iconv capability, although it appears to also cause a limited
> locale implementation to be pulled in (which I really don't have a
> requirement for).  I haven't tried to use it in anger, but for example
> it allows a bunch of ebuilds to compile which previously moaned about
> missing iconv implementations (admitedly those with tests claiming that
> the uclibc iconv is incomplete...)

this is iconv() from the C library, not the `iconv` binary.

i wish locale support in uClibc was more feature complete ...
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-embedded] glibc+iconv

[Ed W]

>> Some experimentation with latest uclibc suggests that it has some
>> working iconv capability, although it appears to also cause a limited
>> locale implementation to be pulled in (which I really don't have a
>> requirement for).  I haven't tried to use it in anger, but for example
>> it allows a bunch of ebuilds to compile which previously moaned about
>> missing iconv implementations (admitedly those with tests claiming that
>> the uclibc iconv is incomplete...)
> this is iconv() from the C library, not the `iconv` binary.
>
> i wish locale support in uClibc was more feature complete ...

On my uclibc box (no libiconv)

root@localhost $ ls -al /usr/bin/iconv
-rwxr-xr-x    1 root     root         10878 Oct 19 19:22 /usr/bin/iconv

root@localhost $ ldd /usr/bin/iconv
        libc.so.0 => /lib/libc.so.0 (0x9ff12000)
        ld-uClibc.so.0 => /lib/ld-uClibc.so.0 (0x9ff75000)


I would really like to get some proxy developer to help me get uclibc
updated and also some patches to a few ebuilds (we hardly need any with
latest uclibc, at least on x86)

See here:
    https://bugs.gentoo.org/show_bug.cgi?id=308477#c21
...for some notes on building uclibc iconv *under* a uclibc build root
(needs some tweaking to help the build succeed - would like to see those
tweaks hit upstream, but ...).  The main snag is that you get iconv
*and* locale... I don't yet have a use for locale, but setting it to
build only en_gb.utf8 seems to work and I think total incremental
library install size was around 40-50KB

(I also noticed a plausible implementation of a functioning gettext
libintl.h ... ?)

Seems like uclibc has *near* correct working iconv and could do with
word getting out so it gets more exposure and hopefully soon a fully
working implementation..

Cheers

Ed W