[gentoo-user] The CHOST variable

[gentoo-user] The CHOST variable

[Nils Holland]

Hi folks,

well, it's not that a certain thing I'm intending to do has pointed me
to it, but I've just noticed that something I've taken for granted for
years is something I probably fail to understand correctly. And as I'm
always eager to learn, I'm wondering if someone can point me in the
right direction.

I'm talking about the CHOST variable as defined in /etc/make.conf, and
what actually lies behind it.

Obviously, whatever is specified in make.conf as CHOST is passed to
./configure as --host when emerging packages. Ok, I know several
things about that: The contents of the CHOST variable is a "target
triplet" in the form of (for example) i686-pc-linux-gnu. According to
the autoconf info documentation, packages using the GNU build
environment use this triplet to properly configure the package at hand
for the specified target architecture. If no --host is given to
configure, it will try to determine the correct triplet for the
current system by the rules specified in "config.sub". Beside --host,
configure also accepts --build and --target, which specify the system
on which the package is being configured and built (--build) and the
the type of architecture for which a compiler being configured is
supposed to be able to generate code (--target, which - I believe -
only really seems to make sense when building a compiler like
GCC). These can also be influenced via make.conf, where they are
called CBUILD and CTARGET.

All of this is fine so far and relatively understandable. But a few
questions remain:

1) So a package using the GNU build system determines or is passed
(via --host aka. CHOST) a target triplet specifying the system on
which the resulting compiled code is supposed to run. What does the
package do with that information? Does it only use it to determine
what it has to compile (different / special code for different systems
/ architectures), or does this already have an influence on the
optimization of the resulting code for a certain (sub-)architecture?
Forthermore: Does configuring a package with, say,
--host=i686-pc-linux-gnu already automatically mean that said package
would not be able to run on (for example) an i486-pc-linux-gnu host?
Or can this only be said to be true if the package itself decides to
compile different / incompatible code for i686 and i486; in other
words: If the package itself does not make any distinction on the CPU
subtype, then the result would run "everywhere", as --host does not by
itself imply any such thing as --march=?

2) /etc/make.conf contains a note that one should not change the CHOST
lightly (not that I'm planning to) and links to a nice document
explaining how it can be done anyway (which, I have to admit, didn't
make me any wiser, however). The question is, out of curiosity, why
the CHOST should not be changed and what would happen if one did it
anyway. I willingly believe that it would lead to problems, but would
the actual cause of these problems actually be caused by the
configuration of the machine being mixed up (for example, by the GNU
build system / autoconf suddenly looking for a compiler or similiar
tools / libraries under a path or by a name involving, for example,
i486-pc-linux-gnu, which does not automatically exist of the
appropriate tools have not been installed accordingly. Or would
problems arise because code generated with the new CHOST does no
longer "fit" to code generated with the previous / old CHOST?

Any hints here would be welcome. Again, it's not that I need to know
in order to do something, but all of this stuff has always worked so
well every time I've built a package, and if feels kind of strange not
to know why / how it actually works. As neither the docs of autoconf,
binutils nor GCC could properly enlighten me, I thought I'd ask
here. ;-)

Greetings,
Nils


-- 
Nils Holland * Ti Systems, Wunstorf-Luthe (Germany)
Powered by GNU/Linux since 1998

Re: [gentoo-user] The CHOST variable

[Alan McKinnon]

Apparently, though unproven, at 01:43 on Friday 04 February 2011, Nils Holland 
did opine thusly:

I'm not in a position to give a fully definitive answer to 1) ...

> 2) /etc/make.conf contains a note that one should not change the CHOST
> lightly (not that I'm planning to) and links to a nice document
> explaining how it can be done anyway (which, I have to admit, didn't
> make me any wiser, however). The question is, out of curiosity, why
> the CHOST should not be changed and what would happen if one did it
> anyway. I willingly believe that it would lead to problems, but would
> the actual cause of these problems actually be caused by the
> configuration of the machine being mixed up (for example, by the GNU
> build system / autoconf suddenly looking for a compiler or similiar
> tools / libraries under a path or by a name involving, for example,
> i486-pc-linux-gnu, which does not automatically exist of the
> appropriate tools have not been installed accordingly. Or would
> problems arise because code generated with the new CHOST does no
> longer "fit" to code generated with the previous / old CHOST?

The warning is actually there to stop users doing stupid things like blindly 
trying to convert 32 bit systems to 64 bit. This is how that goes down:

1. Change CHOST
2. emerge -e world
3. ???
4. Fail!

Yes, if you are real smart it can be done. But "real smart" really does mean 
"real smart" i.e. not for the faint of heart and certainly not worth being 
officially supported.


-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] The CHOST variable

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 1868 bytes --]

Am 04.02.2011 01:27, schrieb Alan McKinnon:
> Apparently, though unproven, at 01:43 on Friday 04 February 2011, Nils Holland 
> did opine thusly:
> 
> I'm not in a position to give a fully definitive answer to 1) ...
> 
>> 2) /etc/make.conf contains a note that one should not change the CHOST
>> lightly (not that I'm planning to) and links to a nice document
>> explaining how it can be done anyway (which, I have to admit, didn't
>> make me any wiser, however). The question is, out of curiosity, why
>> the CHOST should not be changed and what would happen if one did it
>> anyway. I willingly believe that it would lead to problems, but would
>> the actual cause of these problems actually be caused by the
>> configuration of the machine being mixed up (for example, by the GNU
>> build system / autoconf suddenly looking for a compiler or similiar
>> tools / libraries under a path or by a name involving, for example,
>> i486-pc-linux-gnu, which does not automatically exist of the
>> appropriate tools have not been installed accordingly. Or would
>> problems arise because code generated with the new CHOST does no
>> longer "fit" to code generated with the previous / old CHOST?
> 
> The warning is actually there to stop users doing stupid things like blindly 
> trying to convert 32 bit systems to 64 bit. This is how that goes down:
> 
> 1. Change CHOST
> 2. emerge -e world
> 3. ???
> 4. Fail!
> 
> Yes, if you are real smart it can be done. But "real smart" really does mean 
> "real smart" i.e. not for the faint of heart and certainly not worth being 
> officially supported.
> 

Is the same true for more compatible arches like i486 -> i686? I have a
system where I used the wrong stage-3 and now I'm stuck with an i486
CHOST on an Atom netbook where I could use every bit of performance.

Regards,
Florian


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [gentoo-user] The CHOST variable

[Alan McKinnon]

Apparently, though unproven, at 11:29 on Friday 04 February 2011, Nils Holland 
did opine thusly:

> Florian Philipp wrote:
> > Am 04.02.2011 01:27, schrieb Alan McKinnon:
> >> Yes, if you are real smart it can be done. But "real smart" really does
> >> mean
> >> "real smart" i.e. not for the faint of heart and certainly not worth
> >> being
> >> officially supported.
> > 
> > Is the same true for more compatible arches like i486 -> i686? I have a
> > system where I used the wrong stage-3 and now I'm stuck with an i486
> > CHOST on an Atom netbook where I could use every bit of performance.
> 
> Yep, that's about the same direction I was wondering about when asking my
> question.
> 
> Specifically, I've been playing around with catalyst a bit in order to
> create my own stage3 tarballs and install discs. Works fine. But then I
> started to wonder about the following: I'd probably like to make my own
> stage3 and install disc i486 or even i386, so that it works "everywhere".
> Then, on whatever machine I install it, I would probably like to change
> the CHOST to something suitable for that machine, and that's actually what
> the comment in /etc/make.conf suggests against doing. As a result, I
> started to wonder why this is so, and what exactly would happen if done
> anyway.
> 
> Of course, this is not really a big problem. I could create multiple
> stage3's with catalyst whenever I really need one to install on a machine
> and have that stage3 by of the right type for the machine right from the
> start. But still, always eager to learn, I thought I'd ask about the
> background behind all that stuff. ;-)

Interestingly, Ubuntu has always built for basic arches, and they seem to get 
away with it.

IIRC they are now on i586 but for the longest time used i386. No performance 
issues. You might want to investigate how they do their builds and see if you 
can use their tricks.


-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] The CHOST variable

[Nils Holland]

Florian Philipp wrote:

> Am 04.02.2011 01:27, schrieb Alan McKinnon:
>>
>> Yes, if you are real smart it can be done. But "real smart" really does
>> mean
>> "real smart" i.e. not for the faint of heart and certainly not worth
>> being
>> officially supported.
>>
>
> Is the same true for more compatible arches like i486 -> i686? I have a
> system where I used the wrong stage-3 and now I'm stuck with an i486
> CHOST on an Atom netbook where I could use every bit of performance.

Yep, that's about the same direction I was wondering about when asking my
question.

Specifically, I've been playing around with catalyst a bit in order to
create my own stage3 tarballs and install discs. Works fine. But then I
started to wonder about the following: I'd probably like to make my own
stage3 and install disc i486 or even i386, so that it works "everywhere".
Then, on whatever machine I install it, I would probably like to change
the CHOST to something suitable for that machine, and that's actually what
the comment in /etc/make.conf suggests against doing. As a result, I
started to wonder why this is so, and what exactly would happen if done
anyway.

Of course, this is not really a big problem. I could create multiple
stage3's with catalyst whenever I really need one to install on a machine
and have that stage3 by of the right type for the machine right from the
start. But still, always eager to learn, I thought I'd ask about the
background behind all that stuff. ;-)

Greetings,
Nils

Re: [gentoo-user] The CHOST variable

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 814 bytes --]

On Fri, 04 Feb 2011 06:58:27 +0100, Florian Philipp wrote:

> Is the same true for more compatible arches like i486 -> i686? I have a
> system where I used the wrong stage-3 and now I'm stuck with an i486
> CHOST on an Atom netbook where I could use every bit of performance.

It's possible, there's even a Gentoo document on it,
http://www.gentoo.org/doc/en/change-chost.xml

Note the first line states

"Changing the CHOST is a big issue that can seriously screw up your
system"

Any time you consider a process that involves emerge -e world, you should
also consider a reinstall. A reinstall will certainly be quicker, the
only reason for an in place fix is if you cannot take the machine down
for that length of time.


-- 
Neil Bothwick

On the other hand, you have different fingers.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] The CHOST variable

[Nils Holland]

Alan McKinnon wrote:

> Interestingly, Ubuntu has always built for basic arches, and they seem to
> get away with it.
>
> IIRC they are now on i586 but for the longest time used i386. No
> performance issues. You might want to investigate how they do
> their builds and see if you can use their tricks.

The question is, I guess, if the target host, when of the same arch (i.e.
i[3456]86) does actually have any influence on the code that gets
generated in terms of performance or ability to run on other sub-arches.
This is what I really couldn't find out so far and would find highly
interesting to know.

For example, why not just go (and stay) with CHOST="i386-pc-linux-gnu" and
on an i686 machine, set march or mcpu = i686 via CFLAGS if you want to
optimize for the particular subarch at hand? Why should it be necessary /
what would the (dis)advantages be of of such a setup vs. also having CHOST
set to "i686-pc-linux-gnu"?

Concering the Gentoo doc about changing the CHOST that was mentioned: Yep,
I've read that. If I understood it correctly, problems when changing CHOST
mainly seem to arise because of the way GCC and related basic build utils
install themselves (using the host triplet as part of their path or
executable name), leaving wrong / messed up references when changing the
CHOST.

For example, as I've said previously, the CHOST value gets passed to
./configure as --host for each package that gets build. That would make
configure try to select a compiler called <CHOST>-gcc in order to compile
the package, i.e. when CHOST is i486-pc-linux-gnu, a compiler called
i486-pc-linux-gnu-gcc would be used. Include file directories for glibc
and / or glibc itself sems to be selected by a similiar mechanism. All
right, no problem, so far this only determines how things are called and
where they are located, but are there any other "real" effects of all this
stuff?

The Gentoo CHOST document that was mentioned says something about nptl not
being available on i386. If true, and if the host variable generally
influences the availability of features, things would slowly start to make
sens as to why the CHOST might be important. On the other hand, I've read
through some documentation of the GNU C library (glibc) and didn't even
find anything about nptl not being available on i386, not to mention
anything else about different features on different subarches.

You see, I'm probably not entirely "getting it" yet. ;-)

Greetings,
Nils

Re: [gentoo-user] The CHOST variable

[Marius Vaitiekunas]

On Fri, Feb 4, 2011 at 3:50 PM, Nils Holland <nhg@tisys.org> wrote:
> Alan McKinnon wrote:
>
>> Interestingly, Ubuntu has always built for basic arches, and they seem to
>> get away with it.
>>
>> IIRC they are now on i586 but for the longest time used i386. No
>> performance issues. You might want to investigate how they do
>> their builds and see if you can use their tricks.
>
> The question is, I guess, if the target host, when of the same arch (i.e.
> i[3456]86) does actually have any influence on the code that gets
> generated in terms of performance or ability to run on other sub-arches.
> This is what I really couldn't find out so far and would find highly
> interesting to know.
>
> For example, why not just go (and stay) with CHOST="i386-pc-linux-gnu" and
> on an i686 machine, set march or mcpu = i686 via CFLAGS if you want to
> optimize for the particular subarch at hand? Why should it be necessary /
> what would the (dis)advantages be of of such a setup vs. also having CHOST
> set to "i686-pc-linux-gnu"?
>
> Concering the Gentoo doc about changing the CHOST that was mentioned: Yep,
> I've read that. If I understood it correctly, problems when changing CHOST
> mainly seem to arise because of the way GCC and related basic build utils
> install themselves (using the host triplet as part of their path or
> executable name), leaving wrong / messed up references when changing the
> CHOST.
>
> For example, as I've said previously, the CHOST value gets passed to
> ./configure as --host for each package that gets build. That would make
> configure try to select a compiler called <CHOST>-gcc in order to compile
> the package, i.e. when CHOST is i486-pc-linux-gnu, a compiler called
> i486-pc-linux-gnu-gcc would be used. Include file directories for glibc
> and / or glibc itself sems to be selected by a similiar mechanism. All
> right, no problem, so far this only determines how things are called and
> where they are located, but are there any other "real" effects of all this
> stuff?
>
> The Gentoo CHOST document that was mentioned says something about nptl not
> being available on i386. If true, and if the host variable generally
> influences the availability of features, things would slowly start to make
> sens as to why the CHOST might be important. On the other hand, I've read
> through some documentation of the GNU C library (glibc) and didn't even
> find anything about nptl not being available on i386, not to mention
> anything else about different features on different subarches.
>
> You see, I'm probably not entirely "getting it" yet. ;-)
>
> Greetings,
> Nils
>
>
>

Hi,

I am not a big guru there, but i have changed CHOST variable
successfully few years ago. If I remember, the steps were like that:

Change CHOST variable.
Bootstrap system (like building from stage 1):
# /usr/portage/scripts/bootstrap.sh
# emerge -e system
# emerge -e world

Before gentoo has been providing daily stages, I was installing my
systems from stage1. It was a nice learning curve :)

-- 
mv

[gentoo-user] Re: The CHOST variable

[James]

Nils Holland <nhg <at> tisys.org> writes:

> Florian Philipp wrote:

> Specifically, I've been playing around with catalyst a bit in order to
> create my own stage3 tarballs and install discs. Works fine. But then I
> started to wonder about the following: I'd probably like to make my own
> stage3 and install disc i486 or even i386, so that it works "everywhere".
> Then, on whatever machine I install it, I would probably like to change
> the CHOST to something suitable for that machine, and that's actually what
> the comment in /etc/make.conf suggests against doing. As a result, I
> started to wonder why this is so, and what exactly would happen if done
> anyway.


Ah music to my ears!

Several folks I know have been pondering your very ideas,
except as to how to apply the build out code for an
embedded Gentoo (x86) based tree. ALIX, ATOM, old
K5 and K6 machines, on and on and on. Surely
there is common ground amongst our efforts......?

I know this list is very sharp, but there are some 
(deeply skilled) folks on the embedded-gentoo list:

gentoo-embedded@gentoo.org
http://www.gentoo.org/proj/en/base/embedded/handbook/

that if you pose your well-formed ideas and goals there
you'll find some of the best coding guidance
and pick up an ontourage of wanna_bee_follows (grin)
on your quest to build out for smaller x86 arches.

This aspect of gentoo's lineages is rich, but at
the moment, is in need of a champion......

Drop me some private email as I have lots
of links and resources, related to your quest.

sincerely,
James

Re: [gentoo-user] The CHOST variable

[Enrico Weigelt]

* Nils Holland <nhg@tisys.org> wrote:

> 1) So a package using the GNU build system determines or is passed
> (via --host aka. CHOST) a target triplet specifying the system on
> which the resulting compiled code is supposed to run. What does the
> package do with that information? Does it only use it to determine
> what it has to compile (different / special code for different systems
> / architectures), or does this already have an influence on the
> optimization of the resulting code for a certain (sub-)architecture?

Exactly. Some packages have arch- and subarch-specific optimizations.
Those things IMHO should be primarily taken from the target triplet
(instead of other ./configure options), as it quite exactly defines
what cputype, platform and toolchain type you're using. It's very
important for crosscompiling (even some toolchains, eg. gcc., could,
and should, be explicitly asked for their target triplet).

> Forthermore: Does configuring a package with, say,
> --host=i686-pc-linux-gnu already automatically mean that said package
> would not be able to run on (for example) an i486-pc-linux-gnu host?

Maybe. Or maybe it's just quite slow, eg. when it's using certain
opcodes that are not supported by the older subarch and handled by
the kernel via exceptions (back in i386 times, that was the case for
fpops, when no fpu was present). It all depends on how the individual
package is actually coded.

> 2) /etc/make.conf contains a note that one should not change the CHOST
> lightly (not that I'm planning to) and links to a nice document
> explaining how it can be done anyway (which, I have to admit, didn't
> make me any wiser, however). The question is, out of curiosity, why
> the CHOST should not be changed and what would happen if one did it
> anyway. I willingly believe that it would lead to problems, but would
> the actual cause of these problems actually be caused by the
> configuration of the machine being mixed up (for example, by the GNU
> build system / autoconf suddenly looking for a compiler or similiar
> tools / libraries under a path or by a name involving, for example,
> i486-pc-linux-gnu, which does not automatically exist of the
> appropriate tools have not been installed accordingly. Or would
> problems arise because code generated with the new CHOST does no
> longer "fit" to code generated with the previous / old CHOST?

One of the most evil things that can happen (also w/ certain optimization
flags) is a mismatching calling convention between certain libraries
and executables. For example, code built for a newer subarch could put
more call arguments into registers which the called functions dont
expect - stack corruptions will happen.


cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [gentoo-user] The CHOST variable

[Enrico Weigelt]

* Nils Holland <nhg@tisys.org> wrote:

> The question is, I guess, if the target host, when of the same arch (i.e.
> i[3456]86) does actually have any influence on the code that gets
> generated in terms of performance or ability to run on other sub-arches.
> This is what I really couldn't find out so far and would find highly
> interesting to know.

Yes, both. Passing arguments via register (instead of stack) can be
much faster in some cases, especially when you have a lot of calls
with small parameter sets, eg:

libfoo.c:

    int foo(int a, int b)
    {
	... <do-something> ...
    }
    
progbar.c:

    int main()
    {
	...
	for (int x=0; x<10000; x++)
	    foo(x, wurst);
	...
    }

On older subarchs, main() will have to push x and wurst on stack for
_each_ call, and foo() has to pop them down. With a newer subarch that
has more usable registers, the parameters a and b can directly sit in
registers (probably the counting of x might already happen in the same
one), so the parameters dont have to go through the stack. That all 
heavily depends on the target ABI.

> For example, why not just go (and stay) with CHOST="i386-pc-linux-gnu" and
> on an i686 machine, set march or mcpu = i686 via CFLAGS if you want to
> optimize for the particular subarch at hand? Why should it be necessary /
> what would the (dis)advantages be of of such a setup vs. also having CHOST
> set to "i686-pc-linux-gnu"?

I'm not sure right now if/how -march and -mcpu affect calling convention,
variable alignments, etc (IMHO -march does, while -mcpu doesn't), but
as soon as you change these ABI elements, it can get really dangerous.

> Concering the Gentoo doc about changing the CHOST that was mentioned: Yep,
> I've read that. If I understood it correctly, problems when changing CHOST
> mainly seem to arise because of the way GCC and related basic build utils
> install themselves (using the host triplet as part of their path or
> executable name), leaving wrong / messed up references when changing the
> CHOST.

You're lucky if the compiler fails early this way, so you don't have
the unpleasant experience of sudden runtime breakages ;-p
 
> For example, as I've said previously, the CHOST value gets passed to
> ./configure as --host for each package that gets build. That would make
> configure try to select a compiler called <CHOST>-gcc in order to compile

... if the toolchain commands aren't passed explicitly ...
(which you *SHOULD* do in any crosscompiling scenario)

> the package, i.e. when CHOST is i486-pc-linux-gnu, a compiler called
> i486-pc-linux-gnu-gcc would be used. Include file directories for glibc
> and / or glibc itself sems to be selected by a similiar mechanism.

AFAIK they're not selected by this parameter (even some illdesigned
configure script might do such crazy things on its own ;-o), but built
into the toolchain commands themselves. You can override/change them,
but then you really should know what you're doing ;-p

> The Gentoo CHOST document that was mentioned says something about nptl not
> being available on i386. If true, and if the host variable generally
> influences the availability of features, things would slowly start to make
> sens as to why the CHOST might be important. 

Exactly. nptl vs. linuxthreads is exactly an good example for
incompatible ABIs (while API more or less stays the same).

> On the other hand, I've read
> through some documentation of the GNU C library (glibc) and didn't even
> find anything about nptl not being available on i386, not to mention
> anything else about different features on different subarches.

I don't know much about their internals, but I can easily imagine
that nptl makes heavy use of newer registers which linuxthreads didn't.
For those things you should ask the gcc and libc folks.


cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [gentoo-user] The CHOST variable

[Nils Holland]

On 21:21 Fri 04 Feb     , Enrico Weigelt wrote:

> * Nils Holland <nhg@tisys.org> wrote:
> 
> > 1) So a package using the GNU build system determines or is passed
> > (via --host aka. CHOST) a target triplet specifying the system on
> > which the resulting compiled code is supposed to run. What does the
> > package do with that information? Does it only use it to determine
> > what it has to compile (different / special code for different systems
> > / architectures), or does this already have an influence on the
> > optimization of the resulting code for a certain (sub-)architecture?
> 
> Exactly. Some packages have arch- and subarch-specific optimizations.
> Those things IMHO should be primarily taken from the target triplet
> (instead of other ./configure options), as it quite exactly defines
> what cputype, platform and toolchain type you're using. It's very
> important for crosscompiling (even some toolchains, eg. gcc., could,
> and should, be explicitly asked for their target triplet).

Hi Enrico,

thanks very much for your in-depth explanation in this and also your
other mail you've sent in response to my question. That was really
helpful, as it cleared up some things I could previously only guess
about but was by no means certain about.

I've learnt something new (which was my highest priority here), and
can even use this knowlede to deduce what makes sense in my current
experiments with building my own stage3's and install boot discs via
catalyst.

So, thanks again for your insightful replies!

Greetings,
Nils


-- 
Nils Holland * Ti Systems, Wunstorf-Luthe (Germany)
Powered by GNU/Linux since 1998

Re: [gentoo-user] The CHOST variable

[Stroller]

On 4/2/2011, at 9:31am, Neil Bothwick wrote:
> ...
> Any time you consider a process that involves emerge -e world, you should
> also consider a reinstall. A reinstall will certainly be quicker, the
> only reason for an in place fix is if you cannot take the machine down
> for that length of time.

I disagree. The `emerge -e world` may take longer, but it will consume less of *my time* overseeing it.

Obviously this may not be the case if we're talking about a desktop system in constant use, and the update breaks your web-browser. But in all my use cases I'm going to be able to leave the `emerge -e world` running in the background and just check up on it periodically.

A Gentoo reinstall is not so easy as "click, click, click" and let it run. You should do a full backup first, because you can easily copy /etc over to the new machine and only afterwards realise you forgot to copy /var/bind/

An `emerge -e world` may break things, but it's not usually that likely to. 

Stroller.

Re: [gentoo-user] The CHOST variable

[Stroller]

On 4/2/2011, at 5:58am, Florian Philipp wrote:
>> ...
>> The warning is actually there to stop users doing stupid things like blindly 
>> trying to convert 32 bit systems to 64 bit. This is how that goes down:
>> 
>> 1. Change CHOST
>> 2. emerge -e world
>> 3. ???
>> 4. Fail!
> 
> Is the same true for more compatible arches like i486 -> i686? I have a
> system where I used the wrong stage-3 and now I'm stuck with an i486
> CHOST on an Atom netbook where I could use every bit of performance.

I'm in the process of this at the moment. 

I have a Cobalt Qube III, which has a K6-2 CPU, and which will never accommodate any other processor. 

The closest Gentoo stage was i486, and on such a slow old system it would be nice to squeeze out any extra performance I can.

From the i486 stage I used:

# cat /etc/make.conf.catalyst
# These settings were set by the catalyst build script that automatically
# built this stage.
CFLAGS="-O2 -march=i486 -pipe"
CXXFLAGS="${CFLAGS}"
# WARNING: Changing your CHOST ...
# Please consult http://www.gentoo.org/doc/en/change-chost.xml before changing.
CHOST="i486-pc-linux-gnu"

Following the instructions I changed the CHOST (to i586) and did the `emerge -1 binutils gcc glibc` part.

That went fine, so I changed CFLAGS="-march=native -O2 -pipe -fomit-frame-pointer", and followed the next steps up to 2.12 - `emerge -e world`.

This failed about 3 packages in and really messed things up - I quickly couldn't emerge anything else and running executables failed with a segfault or something.

I had to pull the drive back out of the Qube and copy most of the i486 stage back over the fouled-up installation.

This time around, I changed the CHOST again to i586 and ran `emerge -1 binutils gcc glibc`, then `lafilefixer --justfixit && revdep-rebuild` (didn't need to do anything, it said) then the `emerge -e world` part - but this time still with the original i486 CFLAGS (as the catalyst above). This is currently at "104 of 145" packages through the emerge process, so seems ok.

Hopefully when that's done I'll be able to change CFLAGS to "-march=native" and re-emerge the whole lot again. I'm now a bit nervous of that (I'm wondering if `-mcpu` or `-mtune` would be more suitable as a first step), so I'll take a disk image using dd (again) so that I have a restore point, should I need it.

Stroller.

Re: [gentoo-user] The CHOST variable

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 1871 bytes --]

On Mon, 7 Feb 2011 11:13:27 +0000, Stroller wrote:

> > Any time you consider a process that involves emerge -e world, you
> > should also consider a reinstall. A reinstall will certainly be
> > quicker, the only reason for an in place fix is if you cannot take
> > the machine down for that length of time.  
> 
> I disagree. The `emerge -e world` may take longer, but it will consume
> less of *my time* overseeing it.
> 
> Obviously this may not be the case if we're talking about a desktop
> system in constant use, and the update breaks your web-browser. But in
> all my use cases I'm going to be able to leave the `emerge -e world`
> running in the background and just check up on it periodically.
> 
> A Gentoo reinstall is not so easy as "click, click, click" and let it
> run.

To a large extent, it is. you've already partitioned the drive, set up
make.conf, created a world file, all of which can be reused. So a
reinstall isn't much more than emerge -e world and recompile the kernel
after emptying the filesystem.

> An `emerge -e world` may break things, but it's not usually that likely
> to. 

An emerge -e world is not likely to break things in itself, but the steps
that require it, such as changing CHOST, are. The extra steps of a
reinstall over trying to fix a machine with a borked
binutils/glibc/whatever can be far more time consuming, not to mention
frustrating, than a reinstall, and may only be fixed by a reinstall
anyway after all that.

I'm not an advocate of reinstalling normally, this installation is three
years older than the computer running it, but when performing drastic
low-level surgery, I believe it should be contemplated.


-- 
Neil Bothwick

Quality control, n.:
  Assuring that the quality of a product does not get out of hand
   and add to the cost of its manufacture or design.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] The CHOST variable

[Nils Holland]

On 12:24 Mon 07 Feb     , Stroller wrote:
> 
> 
> The closest Gentoo stage was i486, and on such a slow old system it would be nice to squeeze out any extra performance I can.

Well, what I'm currently in the process of trying to do (not because I
have an actual need for it, but rather out of experimentation) is
taking a Gentoo i486 stage3 tarball and turning it into a i586 one.

I realized that on the i686 host I am using, I can just let catalyst
loose on an official Gentoo i486 or i686 tarball and, with the
appropriate settings, have it build a new and more current tarball out
of it. Fine.

However, what certainly doesn't work is taking one of these stage3
tarballs (i486 or i686) and have catalyst build an i586 tarball out of
that. In my attemts when trying this, catalyst first compiled a new
glibc with i586, and the next thing it tried was building a new zlib
with using the i586 glibc and the still i486 gcc and other stuff,
which resulted in failure (don't have the exact error message anymore,
but it was along the lines of the linker complaining about not being
able to determine SONAME of libz.so).

So what would probably work and what I'll try in the next days is the
following: Unpack i486 tarball, chroot into it, and then manually try
to make a i586 tarball out of it. That would probably involve the
process described in the Gentoo "how to change your CHOST" document. I
guess that once I've "prepared" such an i586 stage3 manually, I can
regularely let catalyst handle updating it, just as it does fine with
the original i486 and i686 stage3's.

Again, I'm only doing this out of curiosity to see if it works and /
or what problems I encounter and to learn some stuff; I'm probably
never going to use the resulting stage3 tarball on my own machines,
as Gentoo's i486 and i686 ones suit me just fine and I'm not a big
performance tuner / tweaker. Still, if I succeed and if anyone is
interested, I could certainly make me i586 stage3 available for
download.

I've also, just for fun, looked at a i386 stage3. However, while that
would technically certainly also be possible, it seems that Gentoo's
glibc ebuild is specifically set up to bail out when building on
i386. That would probably even be fixable by not building with nptl,
but that way it would only be possible to built something that would
make problems when using the normal portage tree and coming into a
situation where it wants to build a new (nptl-enabled) glibc for the
first time, not to mention that I don't have a clue what other stuff
would break when the user tries to install it from portage on a system
that comes with a glibc I've "hacked" to come without nptl.

Greetings,
Nils

-- 
Nils Holland * Ti Systems, Wunstorf-Luthe (Germany)
Powered by GNU/Linux since 1998

Re: [gentoo-user] The CHOST variable

[Stroller]

On 7/2/2011, at 11:03pm, Nils Holland wrote:
> On 12:24 Mon 07 Feb     , Stroller wrote:
>> 
>> 
>> The closest Gentoo stage was i486, and on such a slow old system it would be nice to squeeze out any extra performance I can.
> 
> ...
> So what would probably work and what I'll try in the next days is the
> following: Unpack i486 tarball, chroot into it, and then manually try
> to make a i586 tarball out of it. That would probably involve the
> process described in the Gentoo "how to change your CHOST" document.

If my process wasn't clear from my last email: it looks like, following that document, you have to do the whole thing with changed CHOST, *before* making any changes to CFLAGS. It appears like only after you've `emerge -e world` with the new CHOST can you change CFLAGS.

> I guess that once I've "prepared" such an i586 stage3 manually, I can
> regularely let catalyst handle updating it, just as it does fine with
> the original i486 and i686 stage3's.

I haven't used catalyst at all. You make it sound interesting & useful.

Stroller.

Re: [gentoo-user] The CHOST variable

[Nils Holland]

On 12:41 Tue 08 Feb     , Stroller wrote:

> If my process wasn't clear from my last email: it looks like, following that document, you have to do the whole thing with changed CHOST, *before* making any changes to CFLAGS. It appears like only after you've `emerge -e world` with the new CHOST can you change CFLAGS.

Thanks for the hint, right now I've got the i486 stage3 in a chroot
and have changed both the CHOST and CFLAGS to i586. If that leads to
problems, as you've described, I'll start over with only setting CHOST
to i586 at first and then changing CFLAGS. Currently the build process
is running fine, however, but I'm mentally prepared for difficults
now. ;-)
 
> > I guess that once I've "prepared" such an i586 stage3 manually, I can
> > regularely let catalyst handle updating it, just as it does fine with
> > the original i486 and i686 stage3's.
> 
> I haven't used catalyst at all. You make it sound interesting & useful.

I've found it useful in the sense that it has allowed me to download
an original Gentoo stage3 way back, and then - say, once a month -
I've let catalyst build a new, updated stage3 out of it which I then
used for new installations whenever I had to do then. Of course, I
could just have grabbed one of the official Gentoo stage3's that get
updated weekly, but hey, somehow it just feels cooler to use a stage3
you've rolled yourself. ;-)

I'll post again when I have my i586 stage3 available for
download. Folks wanting to perform a new Gentoo installation on an
i586 kind of machine could then just grab and use that one if they
don't want to use Gentoo's i486 stage3 (and stay at i486 or change the
CHOST / CFLAGS to i586 after installation themselves).

Greetings,
Nils


-- 
Nils Holland * Ti Systems, Wunstorf-Luthe (Germany)
Powered by GNU/Linux since 1998

Re: [gentoo-user] The CHOST variable

[Alan McKinnon]

Apparently, though unproven, at 15:33 on Monday 07 February 2011, Neil 
Bothwick did opine thusly:

> > An `emerge -e world` may break things, but it's not usually that likely
> > to. 
> 
> An emerge -e world is not likely to break things in itself, but the steps
> that require it, such as changing CHOST, are. The extra steps of a
> reinstall over trying to fix a machine with a borked
> binutils/glibc/whatever can be far more time consuming, not to mention
> frustrating, than a reinstall, and may only be fixed by a reinstall
> anyway after all that.
> 
> I'm not an advocate of reinstalling normally, this installation is three
> years older than the computer running it, but when performing drastic
> low-level surgery, I believe it should be contemplated.


If you're a gambling man, play it by the numbers:

A re-install for a Gentoo user with a clue is a certain 1 hour of your life 
tops to get it redone with a recent stage 3, more likely 30 minutes. That will 
give you a working system albeit one a bit out of date that emerge -avunD 
world will fix nicely.

Trying to fix the existing system complete with it's borkage is very 
uncertain. Maybe it's 5 minutes, maybe it's 2 hours, maybe it's three days on 
this list scratching heads before eventually giving up and re-installing 
anyway.

Personally, I long ago proved I can do #2. I like the certainty from #1.


-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] The CHOST variable

[Stroller]

On 8/2/2011, at 9:55pm, Alan McKinnon wrote:
> ...
> If you're a gambling man, play it by the numbers:
> 
> A re-install for a Gentoo user with a clue is a certain 1 hour of your life 
> tops to get it redone with a recent stage 3, more likely 30 minutes. That will 
> give you a working system albeit one a bit out of date that emerge -avunD 
> world will fix nicely.

That makes no sense to me at all.

There are stage3s built on a daily basis - they won't be out of date at all. But they don't include system logger, cron, locale, or any of your personalisations.

On any kind of complex system it will take me more than a day to set those up - can I be sure that I'll get them right first time, every time? 

If we're talking about an older system then I need to back up everything so that I can copy files the files I missed from the /etc to the new one.

If you're talking about stuff like KDE then that alone is going to take more than an hour to compile.

I understood that a backup that was of your own system - that might be a bit out of date but which includes these things - was called a stage4. And in that case you're still faced with the problem that it's probably just as out of date as the install which is proving difficult to update in the first place.

My experience of a system which was a year or even 18 months old was that it did have some blockers and some bugs which had been addressed here months before. It did require me to pull some files out of Gentoo's CVS attic and emerge the old ebuilds from my local tree, before they could be updated to current. But I knew that the configuration was sound and that all I had to do was address the blocker and then resume the `emerge -ud world`- the emerging is something that can go on in the background whilst I'm not paying attention. 

Stroller.

Re: [gentoo-user] The CHOST variable

[Stroller]

On 8/2/2011, at 9:11pm, Nils Holland wrote:
> On 12:41 Tue 08 Feb     , Stroller wrote:
> 
>> If my process wasn't clear from my last email: it looks like, following that document, you have to do the whole thing with changed CHOST, *before* making any changes to CFLAGS. It appears like only after you've `emerge -e world` with the new CHOST can you change CFLAGS.
> 
> Thanks for the hint, right now I've got the i486 stage3 in a chroot
> and have changed both the CHOST and CFLAGS to i586. If that leads to
> problems, as you've described, I'll start over with only setting CHOST
> to i586 at first and then changing CFLAGS. Currently the build process
> is running fine, however, but I'm mentally prepared for difficults
> now. ;-)

I completed the `emerge -e world` with the new CHOST, ran `lafilefixer --justfixit && revdep-rebuild` and then Python and the Perl packages, as listed in the documentation. I'm pretty sure I ran `lafilefixer --justfixit && revdep-rebuild` again, finding nothing.

Changing CFLAGS="-march=native -O2 -pipe -fomit-frame-pointer" failed again on the first package, ncurses. CFLAGS="-mtune=native -O2 -pipe -fomit-frame-pointer" seems to be working (now on package 10 of 155).

Perhaps I should have tried CFLAGS="-O2 -march=i586 -pipe" instead. I don't really know what the "-fomit-frame-pointer" part does - I imagine someone suggested it, perhaps on here, years ago, and it has got copied from system to system.

Stroller.

Re: [gentoo-user] The CHOST variable

[Paul Hartman]

On Thu, Feb 10, 2011 at 9:24 AM, Stroller
<stroller@stellar.eclipse.co.uk> wrote:
> I don't really know what the "-fomit-frame-pointer" part does - I imagine someone suggested it, perhaps on here, years ago, and it has got copied from system to system.

I think it removes your ability to get a stack trace when programs
crash, in exchange for potentially more speed. If you're a programmer
or filing bug reports on crashing software you should not use it, I
don't think. But I could be wrong. :)

Re: [gentoo-user] The CHOST variable

[Stroller]

On 10/2/2011, at 4:22pm, Paul Hartman wrote:

> On Thu, Feb 10, 2011 at 9:24 AM, Stroller
> <stroller@stellar.eclipse.co.uk> wrote:
>> I don't really know what the "-fomit-frame-pointer" part does - I imagine someone suggested it, perhaps on here, years ago, and it has got copied from system to system.
> 
> I think it removes your ability to get a stack trace when programs
> crash, in exchange for potentially more speed. If you're a programmer
> or filing bug reports on crashing software you should not use it, I
> don't think. But I could be wrong. :)

Thanks! 
:D

I appreciate the informative & helpful answer.

Stroller.

Re: [gentoo-user] The CHOST variable

[Alan McKinnon]

Apparently, though unproven, at 17:18 on Thursday 10 February 2011, Stroller 
did opine thusly:

> On 8/2/2011, at 9:55pm, Alan McKinnon wrote:
> > ...
> > If you're a gambling man, play it by the numbers:
> > 
> > A re-install for a Gentoo user with a clue is a certain 1 hour of your
> > life tops to get it redone with a recent stage 3, more likely 30
> > minutes. That will give you a working system albeit one a bit out of
> > date that emerge -avunD world will fix nicely.
> 
> That makes no sense to me at all.
> 
> There are stage3s built on a daily basis - they won't be out of date at
> all. But they don't include system logger, cron, locale, or any of your
> personalisations.

All I'm saying is it is often better to go the reinstall route which will take 
a known amount of time for a certain result, rather than the try fix it route 
which is highly variable as to how long it will take or even if it will work 
at all.

It's assumed that one has backups of world and all configs, or can make 
backups before starting the reinstall.

As with all things YMMV and real circumstance trumps abstract theory every 
time

> 
> On any kind of complex system it will take me more than a day to set those
> up - can I be sure that I'll get them right first time, every time?
> 
> If we're talking about an older system then I need to back up everything so
> that I can copy files the files I missed from the /etc to the new one.
> 
> If you're talking about stuff like KDE then that alone is going to take
> more than an hour to compile.
> 
> I understood that a backup that was of your own system - that might be a
> bit out of date but which includes these things - was called a stage4. And
> in that case you're still faced with the problem that it's probably just
> as out of date as the install which is proving difficult to update in the
> first place.
> 
> My experience of a system which was a year or even 18 months old was that
> it did have some blockers and some bugs which had been addressed here
> months before. It did require me to pull some files out of Gentoo's CVS
> attic and emerge the old ebuilds from my local tree, before they could be
> updated to current. But I knew that the configuration was sound and that
> all I had to do was address the blocker and then resume the `emerge -ud
> world`- the emerging is something that can go on in the background whilst
> I'm not paying attention.
> 
> Stroller.

-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] The CHOST variable

[Nils Holland]

On 10:22 Thu 10 Feb     , Paul Hartman wrote:

> On Thu, Feb 10, 2011 at 9:24 AM, Stroller
> <stroller@stellar.eclipse.co.uk> wrote:
> > I don't really know what the "-fomit-frame-pointer" part does - I imagine someone suggested it, perhaps on here, years ago, and it has got copied from system to system.
> 
> I think it removes your ability to get a stack trace when programs
> crash, in exchange for potentially more speed. If you're a programmer
> or filing bug reports on crashing software you should not use it, I
> don't think. But I could be wrong. :)

Yep, that option leads to the frame pointer not being saved /
restored, making an additional register available (but obviously there
are differences between architectures; on some this option has no
effect at all). On architectures where it does have an effect, it
would make debugging impossible.

Well, after having written the last paragraph I've decided to check
out the GCC man page for more information and see: According to the
GCC man page -fomit-frame-pointer gets enabled by default when using
optimization levels -O or higher (including, of course -O2, which in
my experience is what most people seem to use).

As a matter of fact, I have -O2 but intentionally not set
-fomit-frame-pointer, not being aware that with -O2, I automatically
get it. ;-) I'll have to check out the Gentoo handbook, but if I
remember correctly, it actually suggests using -O2 but better stay
away from -fomit-frame-pointer. If it really suggests this, that might
be considered a bug I guess. ;-)

Greetings,
Nils


-- 
Nils Holland * Ti Systems, Wunstorf-Luthe (Germany)
Powered by GNU/Linux since 1998