[gentoo-user] Shared libraries in Gentoo

[gentoo-user] Shared libraries in Gentoo

[Al]

Hello,

I looked into many ebuilds, but didn't come to a final conclusion yet.
I am rather confused.

How does a program in Gentoo know, where to look for shared libraries?

Is this compiled into the programes by means of rpath?

Does Gentoo set up a general search path for libraries?

Does this rather differ on per package basis?

I hint to the right docs would help.

Thank you in advance

Al

[gentoo-user] Re: Shared libraries in Gentoo

[Nikos Chantziaras]

On 09/06/2010 09:28 PM, Al wrote:
> Hello,
>
> I looked into many ebuilds, but didn't come to a final conclusion yet.
> I am rather confused.
>
> How does a program in Gentoo know, where to look for shared libraries?

The program doesn't know.  But the runtime linker does.  And those paths 
are in /etc/ld.so.conf.  This file gets updated automatically by portage 
when needed.

But... sometimes the program also knows and can link against libraries 
long after it has started up using a dlopen() call:

http://linux.die.net/man/3/dlopen

Re: [gentoo-user] Re: Shared libraries in Gentoo

[Al]

>> How does a program in Gentoo know, where to look for shared libraries?
>
> The program doesn't know.  But the runtime linker does.  And those paths are
> in /etc/ld.so.conf.  This file gets updated automatically by portage when
> needed.
>
> But... sometimes the program also knows and can link against libraries long
> after it has started up using a dlopen() call:
>
> http://linux.die.net/man/3/dlopen
>

Thank you Nikos. I did read obout this in the Linux HOWTO:

http://www.faqs.org/docs/Linux-HOWTO/Program-Library-HOWTO.html#DL-LIBRARIES

But I was woundering if the /etc/ld.so.conf was only historical stuff.
O.K. is not it's up-to-date. Good to know this.

But it also writes that dlopen() is specific for Linux and Solaris.
There would be alternatives:

 1.) The glib library
 2.) libltdl, which is part of GNU libtool

Now I was woundering, which way would Gentoo choose or if that is not
package specific at all. Are you sure dlopen() is used as a general
approach on Gentoo?

Also I installed a few libries with Prefix Gentoo on Cygwin. On Cygwin
there is no /etc/ld.so.conf. Yet the libraries are found somehow. I
still have to find out how it works in that environment.

Al

Re: [gentoo-user] Re: Shared libraries in Gentoo

[Al]

> Also I installed a few libries with Prefix Gentoo on Cygwin. On Cygwin
> there is no /etc/ld.so.conf. Yet the libraries are found somehow. I
> still have to find out how it works in that environment.

Ah! Your manpage answers this question: "The directories /lib and
/usr/lib are searched" as last step.

Al

[gentoo-user] Re: Shared libraries in Gentoo

[Nikos Chantziaras]

On 09/07/2010 12:24 AM, Al wrote:
>>> How does a program in Gentoo know, where to look for shared libraries?
>>
>> The program doesn't know.  But the runtime linker does.  And those paths are
>> in /etc/ld.so.conf.  This file gets updated automatically by portage when
>> needed.
>>
>> But... sometimes the program also knows and can link against libraries long
>> after it has started up using a dlopen() call:
>>
>> http://linux.die.net/man/3/dlopen
>>
>
> Thank you Nikos. I did read obout this in the Linux HOWTO:
>
> http://www.faqs.org/docs/Linux-HOWTO/Program-Library-HOWTO.html#DL-LIBRARIES
>
> But I was woundering if the /etc/ld.so.conf was only historical stuff.
> O.K. is not it's up-to-date. Good to know this.
>
> But it also writes that dlopen() is specific for Linux and Solaris.
> There would be alternatives:
>
>   1.) The glib library
>   2.) libltdl, which is part of GNU libtool
>
> Now I was woundering, which way would Gentoo choose or if that is not
> package specific at all. Are you sure dlopen() is used as a general
> approach on Gentoo?

Gentoo doesn't choose anything; it's up to the programs to decide how 
they want to load libraries at runtime.  It's like asking whether Gentoo 
chooses to use Qt or Gtk to run Firefox; well, since Firefox is a Gtk 
application and is making calls to Gtk functions, Gentoo doesn't have a 
say about it.

Re: [gentoo-user] Re: Shared libraries in Gentoo

[Al]

>> Now I was woundering, which way would Gentoo choose or if that is not
>> package specific at all. Are you sure dlopen() is used as a general
>> approach on Gentoo?
>
> Gentoo doesn't choose anything; it's up to the programs to decide how they
> want to load libraries at runtime.  It's like asking whether Gentoo chooses
> to use Qt or Gtk to run Firefox; well, since Firefox is a Gtk application
> and is making calls to Gtk functions, Gentoo doesn't have a say about it.
>

O.K. that is one importent point I needed to understand. So the choice
of the tool to handle dynamic linking is on the side of the programs
and can't be triggert uniformly from the ebuilds.

Hmm, ist the dlopen() call the same on all plattforms? I assume, else
you would need to adapt the sources.

How can I find that out? Sure grepping the sources.

Al

Re: [gentoo-user] Re: Shared libraries in Gentoo

[Enrico Weigelt]

* Al <oss.elmar@googlemail.com> wrote:

> But I was woundering if the /etc/ld.so.conf was only historical stuff.
> O.K. is not it's up-to-date. Good to know this.

Note that this only applies to certain platforms (mostly GNU/glibc
based ones). There might be completely different approaches.
It all depends on how your platform (in glibc-/gcc-world it is the
ld-stub) handles shared library loading. Some platforms might do
it directly in the process loader (I guess on native Windows it's
done by the kernel, not userland).
 
> But it also writes that dlopen() is specific for Linux and Solaris.
> There would be alternatives:

Depends on the libc you're using (btw: on GNU/Linux it has nothing
to do with the Linux kernel, but all done by glibc) For example,
the Windows API provides methods for loading shared libraries and
retrieving entry points - this can be used for an dlopen() 
implementation (of course, the in-depth semantics, eg. symbol
visibility and linking orders could vary here).

>  1.) The glib library

Glib essentially bridges to the underlying OS API (for GNU platforms,
it calls glibc's dlopen(), on Windows it calls the DLL loader API).
For that case, there's nothing a proper libc could also do. Take a
sane libc and you don't need glib here.

>  2.) libltdl, which is part of GNU libtool

I doubt you really want to go into libtool hell ... ;-o

> Now I was woundering, which way would Gentoo choose or if that is not
> package specific at all. Are you sure dlopen() is used as a general
> approach on Gentoo?

It doesnt have anything to do with Gentoo (or any other distro),
instead it's a purely package specific issue. Gentoo just happens
to be based (mainly) on GNU libc, which provides dlopen().
 
> Also I installed a few libries with Prefix Gentoo on Cygwin. On Cygwin
> there is no /etc/ld.so.conf. Yet the libraries are found somehow. I
> still have to find out how it works in that environment.

Obviously they have a different lookup mechanism. The actually
library loading of course is Windows-specific (I doubt it would
be possible to support GNU-style ELF libraries entirely from 
userland alone).


cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

[gentoo-user] Re: Shared libraries in Gentoo

[walt]

On 09/06/2010 11:28 AM, Al wrote:
> Hello,
>
> I looked into many ebuilds, but didn't come to a final conclusion yet.
> I am rather confused.

Welcome ;)

> How does a program in Gentoo know, where to look for shared libraries?

Try running "ldconfig -p", which relates to Nikos's comment about ld.so.conf.

> Is this compiled into the programes by means of rpath?

Are you coming from a BSD background?  I know NetBSD uses rpath everywhere, and
they don't use the ld.so.conf mechanism at all, but I can't recall if the others
do or don't.

Some gentoo packages use rpath, others don't.  Use readelf -d <file> to list the
runtime needs of <file>.  For example:

#readelf -d /usrlib/evolution/2.30/libevolution-mail-settings.so.0.0.0 | grep Library
  0x0000000e (SONAME)                     Library soname: [libevolution-mail-settings.so.0]
  0x0000000f (RPATH)                      Library rpath: [/usr/lib/evolution/2.30:/usr/lib]
  0x0000001d (RUNPATH)                    Library runpath: [/usr/lib/evolution/2.30:/usr/lib]

On the other hand:

$readelf -d /lib/libm.so.6 | grep Library
  0x0000000e (SONAME)                     Library soname: [libm.so.6]

> Does this rather differ on per package basis?

Apparently yes, but I get the impression that the upstream maintainers make that
decision for gentoo, whereas the NetBSD devs add the needed linker flags for every
package they use: "-Wl,-rpath,'$ORIGIN/../lib"  (man 8 ld.so).

Corrections welcomed.

Re: [gentoo-user] Re: Shared libraries in Gentoo

[Al]

>
> Are you coming from a BSD background?  I know NetBSD uses rpath everywhere,
> and
> they don't use the ld.so.conf mechanism at all, but I can't recall if the
> others
> do or don't.

No, I am comming from a Debian/Ubuntu background where it simply
worked. Now I try to port Gentoo to Cygwin and it doesn't always work.
So I have to dig a little deeper to get it working, asking myself what
makes theese working while others break.

Thank you very much for all extensive hints. Unfortunately Cygwin is
not ELF but PE/COFF so all ELF debugging tools do not help. I even
have to tweak an eclasses where it relies on scanelf.

Al

Re: [gentoo-user] Re: Shared libraries in Gentoo

[Ajai Khattri]

On Tue, 7 Sep 2010, Al wrote:

> No, I am comming from a Debian/Ubuntu background where it simply
> worked.

Same mechanism there too - Debian/Ubuntu also use /etc/ld.so.conf and/or 
/etc/ld.so.conf.d. You dont see it because you only deal with binary 
packages when updating in Debian/Ubuntu.

> Now I try to port Gentoo to Cygwin and it doesn't always work.

When you say Gentoo, do you mean Portage? Remember Windows has a lot of 
limitations that WILL get in your way so dont be surprised when things 
break.


-- 
A

Re: [gentoo-user] Re: Shared libraries in Gentoo

[Al]

>
> When you say Gentoo, do you mean Portage? Remember Windows has a lot of
> limitations that WILL get in your way so dont be surprised when things
> break.

I am specially interested in Gentoo because it is not another linux
distribution, but an administration tool to build your own sources and
it's scope is wider than linux.

Al

Re: [gentoo-user] Re: Shared libraries in Gentoo

[Ajai Khattri]

On Tue, 7 Sep 2010, Al wrote:

>> When you say Gentoo, do you mean Portage? Remember Windows has a lot of
>> limitations that WILL get in your way so dont be surprised when things
>> break.
>
> I am specially interested in Gentoo because it is not another linux
> distribution, but an administration tool to build your own sources and
> it's scope is wider than linux.

Which doesn't actually answer the question...



-- 
A

Re: [gentoo-user] Re: Shared libraries in Gentoo

[Alan McKinnon]

Apparently, though unproven, at 18:13 on Tuesday 07 September 2010, Ajai 
Khattri did opine thusly:

> On Tue, 7 Sep 2010, Al wrote:
> >> When you say Gentoo, do you mean Portage? Remember Windows has a lot of
> >> limitations that WILL get in your way so dont be surprised when things
> >> break.
> > 
> > I am specially interested in Gentoo because it is not another linux
> > distribution, but an administration tool to build your own sources and
> > it's scope is wider than linux.
> 
> Which doesn't actually answer the question...

Gentoo is an idea, a community, an infrastructure. It is not code or a distro.

To build something, you do not use gentoo, you use portage.

To be accurate though, you use the EAPIs, which portage implements. And 
currently, even after a lot of hard work, the EAPIs are still in large part 
effectively defined as "whatever portage does".

So it really does come down to portage after all. Portage has a hard 
dependency on bash. portage is intimately wrapped up in the linux way of doing 
things.

So unless you are someone who likes pain and/or likes massive porting efforts, 
portage (aka gentoo) has an effective scope that is pretty much linux and not 
much else.

As evidence: the only non-linux port that went anywhere was on FreeBSD, now 
moribund for years.

-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] Re: Shared libraries in Gentoo

[Al]

> So it really does come down to portage after all. Portage has a hard
> dependency on bash. portage is intimately wrapped up in the linux way of doing
> things.
>

Right, we have to say Bash. To be exact Bash is GNU not Linux. I
genarally say Linux not Gnu-Linux. However in this case the difference
matters.

http://www.gnu.org/software/bash/

I run portage more or less sucessfully on Cygwins POSIX layer. Other
people run it on Interix or Solaris.

> As evidence: the only non-linux port that went anywhere was on FreeBSD, now
> moribund for years.

True. But FreeBSD isn't that popular like Windows, Mac or Linux.

I think there is a future for second level managers that can be
installed into multiple OS and yet set up the very same POSIX
invironement. Having that you can build complex software that is
portable. You don't depend on Java. You don't need to run a virtual
server.

Currently there are two canditates. One candidate is Cygwin Ports, the
other one is Gentoo Prefix. Cygwin Ports just added cross-compilation
features into the latest edition. Still Cygwin is limited to Windows.
By this Cygwin Ports has done the first steps to become portable to
Linux and Mac in future and it is already very mature on Windows.

Gentoo Prefix is already able to run on Windows-Interix, Linux and Mac
as second level manager, but it isn't that mature. Still it is not
discovered by a bigger community. The potential is already there.

So you finally can't say FreeBSD was the only port of Portage. But
there is none that went to a major public.

Al

Re: [gentoo-user] Re: Shared libraries in Gentoo

[Alan McKinnon]

Apparently, though unproven, at 20:15 on Tuesday 07 September 2010, Al did 
opine thusly:

> > So it really does come down to portage after all. Portage has a hard
> > dependency on bash. portage is intimately wrapped up in the linux way of
> > doing things.
> 
> Right, we have to say Bash. To be exact Bash is GNU not Linux. I
> genarally say Linux not Gnu-Linux. However in this case the difference
> matters.
> 
> http://www.gnu.org/software/bash/

I fail to see how this is relevant. Portage requires bash, not sh.

It is utterly irrelevant what the licensing and political implications are for 
bash, the technical fact is that portage *requires* Chet's app.

To be fair to portage, it's not portage itself that requires it, it's the 
ebuilds. But ebuilds are the dedicated input format to portage and the two are 
inextricably linked.

> I run portage more or less sucessfully on Cygwins POSIX layer. Other
> people run it on Interix or Solaris.
> 
> > As evidence: the only non-linux port that went anywhere was on FreeBSD,
> > now moribund for years.
> 
> True. But FreeBSD isn't that popular like Windows, Mac or Linux.

So you don't work at a Tier 1 ISP then?

FreeBSD rules that space. I get hugely better performance out of Postfix on 
FreeBSD than on Linux - all other ISPs in this country concur.

> I think there is a future for second level managers that can be
> installed into multiple OS and yet set up the very same POSIX
> invironement. Having that you can build complex software that is
> portable. You don't depend on Java. You don't need to run a virtual
> server.
> 
> Currently there are two canditates. One candidate is Cygwin Ports, the
> other one is Gentoo Prefix. Cygwin Ports just added cross-compilation
> features into the latest edition. Still Cygwin is limited to Windows.
> By this Cygwin Ports has done the first steps to become portable to
> Linux and Mac in future and it is already very mature on Windows.
> 
> Gentoo Prefix is already able to run on Windows-Interix, Linux and Mac
> as second level manager, but it isn't that mature. Still it is not
> discovered by a bigger community. The potential is already there.

The benefits of a source-based distro are many and have been covered 
extensively elsewhere. gentoo.org lists most of them in a accurate fashion.

I never said porting portage could not be done, I said it would be hard work.


> 
> So you finally can't say FreeBSD was the only port of Portage. But
> there is none that went to a major public.

I don't see the point of portage on FreeBSD frankly, considering the general 
use-case where FreeBSD shines. ports is more than adequate for that and I 
don't need the maintenance overhead of portage on machines requiring weekly 
updates. I don't build embedded systems or need highly customized OSes.

In fact, portage is complete overkill and I refuse to allow it to be deployed 
at work. Check my posting history for the rationale behind this.


-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] Re: Shared libraries in Gentoo

[Al]

>> > So it really does come down to portage after all. Portage has a hard
>> > dependency on bash. portage is intimately wrapped up in the linux way of
>> > doing things.
>>
>> Right, we have to say Bash. To be exact Bash is GNU not Linux. I
>> genarally say Linux not Gnu-Linux. However in this case the difference
>> matters.
>>
>> http://www.gnu.org/software/bash/
>
> I fail to see how this is relevant. Portage requires bash, not sh.

Did I say sh? It matters that portage depends on bash not on linux.
Bash is available on Win and Mac.

> So you don't work at a Tier 1 ISP then?

Definitly not. I rather work at the other end, the desktop. Never seen
anything bigger then a PC.

> I never said porting portage could not be done, I said it would be hard work.

It is challenging, but not to difficult so far.

> I don't see the point of portage on FreeBSD frankly, considering the general
> use-case where FreeBSD shines. ports is more than adequate for that and I
> don't need the maintenance overhead of portage on machines requiring weekly
> updates. I don't build embedded systems or need highly customized OSes.
>

I wouldn't port it to BSD either. If I want to reach many people I
still have to start with windows.

> In fact, portage is complete overkill and I refuse to allow it to be deployed
> at work. Check my posting history for the rationale behind this.

What is gentoo if you substract portage? I am courious now.

Al

Re: [gentoo-user] Re: Shared libraries in Gentoo

[Al]

>
> In fact, portage is complete overkill and I refuse to allow it to be deployed
> at work. Check my posting history for the rationale behind this.
>

It is another 2 wrappers to facilitate matters, where there is already
is a huge stack of wrappers:

* gcc covers the linker
* libtool covers gcc and ar
* makefile configures it all
* to unburden from makefile writing, it is generated by configure
* configure is needed to be generated from configure.in
* some other generations here that I skip
* now there comes ebuild as the next wrapper to make building easier
* on top of all there is emerge
* really on top? What is the next level?

How many languages are involved only to generate and install a C
program! Maschine code, C, bash, m4, python, ... what else? How much
knowlege is needed to debug this huge stack!

I daily wonder all this still works so well.

Al

Re: [gentoo-user] Re: Shared libraries in Gentoo

[Enrico Weigelt]

* Al <oss.elmar@googlemail.com> wrote:

> * gcc covers the linker

The 'gcc' command is a wrapper for several toolchain commands,
from the actual compilers and assemblers down to linker.
Yes, it's debatable whether that's really the recommended way (tm),
but obviously it seems to be quite comfortable.

(Note that the several toolchain components normally belong
together quite strictly and have to be built in a way so they
play along fine together. That also includes libc).

> * libtool covers gcc and ar

Not particularily well. It's not really a wrapper, at least no 
abstraction whatsoever, but more a command line filter doing
certain (quite unpredictable) magic things. I'd instead suggest
a real abstraction.
 
> * makefile configures it all

Not perfect, but quite fine.

> * to unburden from makefile writing, it is generated by configure

Actually, completely unncessary. It would be much cleaner to
let it just generate some makefile include for the configuration
stuff and maybe provide a library of generic make rules.

> * configure is needed to be generated from configure.in

When you're going into the autotools hell. Also completely
obsoleted before it even came into existence. A set of well-
designed shell functions could do the job *much* better.

> * now there comes ebuild as the next wrapper to make building easier

Not just "easier", it essentially states where how to get the
sources, which different build configurations are provided and
glues that to the individual package's build scripts.

Yes, there could be more generic approaches, eg. maintaining
the whole (already-patched) source in a canonical repository
scheme (see OSS-QM), having the package provide it's switches
and depencies in a purely declarative way (see Briegel), etc.

> How many languages are involved only to generate and install a C
> program! Maschine code, C, bash, m4, python, ... what else? How much
> knowlege is needed to debug this huge stack!

The stack can be coped layer-by-layer. If you'd somehow manage
to get Gentoo devs (along w/ other distro's maintainers) to 
adopt the oss-qm project, the stack would only start at the
ebuild level (since everything beyond will be provided by
oss-qm and checked/tested eccessively) ;-)


cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [gentoo-user] Re: Shared libraries in Gentoo

[Al]

>> * gcc covers the linker
>
> The 'gcc' command is a wrapper for several toolchain commands,
> from the actual compilers and assemblers down to linker.
> Yes, it's debatable whether that's really the recommended way (tm),
> but obviously it seems to be quite comfortable.
>

Somehow I don't really like the way it is done. The levels of
abstraction are mixed and it results in very cryptic parameters. IMHO
it should be one tool with coherent parameters or two tools. In the
latter case  both would be controlled be the same build script. When
gcc says ld it should also say ar at least.

>> * libtool covers gcc and ar
>
> Not particularily well. It's not really a wrapper, at least no
> abstraction whatsoever, but more a command line filter doing
> certain (quite unpredictable) magic things. I'd instead suggest
> a real abstraction.

Similar gcc. Different tasks, different levels of abstraction are
somewhat queerly mixed. Another parameter style steps in.

>
>> * makefile configures it all
>
> Not perfect, but quite fine.
>
>> * to unburden from makefile writing, it is generated by configure
>
> Actually, completely unncessary. It would be much cleaner to
> let it just generate some makefile include for the configuration
> stuff and maybe provide a library of generic make rules.
>
>> * configure is needed to be generated from configure.in
>
> When you're going into the autotools hell. Also completely
> obsoleted before it even came into existence. A set of well-
> designed shell functions could do the job *much* better.
>

The shell script terminology is a little strange, especially the
conditions. They differ from the style of most modern languages,
differ even from C. I don't like the Ant approach either, because
logic doesn't fit well into XML. I would appriciate it based on a
modern programming language. Then there would be lesser needs for
autogeneration.

>> * now there comes ebuild as the next wrapper to make building easier
>
> Not just "easier", it essentially states where how to get the
> sources, which different build configurations are provided and

I wanted to stress how this "making easier" by another wrapper and
another wrapper leads finnally into kind of a debugging hell, that
makes matters more complicated in the end. Having different levels of
abstraction is fine, but they should do it in clean way without mixing
levels at least.

It's clear that the overall package management is a different layer
from program building. However the question of portablility for
example is mixed into all levels, is it libtool or is it eclasses. It
will be similar for things like internationalization.

Often the way it is simply exists for historical reasons. Well that is
the way of open source. At that point a company like MS can make more
courageous decisions of the overall design. Sometimes they make use of
it.

> glues that to the individual package's build scripts.
>
> Yes, there could be more generic approaches, eg. maintaining
> the whole (already-patched) source in a canonical repository
> scheme (see OSS-QM), having the package provide it's switches
> and depencies in a purely declarative way (see Briegel), etc.
>
>> How many languages are involved only to generate and install a C
>> program! Maschine code, C, bash, m4, python, ... what else? How much
>> knowlege is needed to debug this huge stack!
>
> The stack can be coped layer-by-layer. If you'd somehow manage

Ideally it can. In praxis the stack can't always be coped
layer-by-layer by different teams. In the moment the bug is there, you
have to search them all.

For my target of a port I have to understand them all, up to a certain
degree. I have to apply patches at different layers.

> to get Gentoo devs (along w/ other distro's maintainers) to
> adopt the oss-qm project, the stack would only start at the
> ebuild level (since everything beyond will be provided by
> oss-qm and checked/tested eccessively) ;-)

For my case that would be of help. I currently have to mix patches
from cygwin with patches from Gentoo. Sure that leads to conflicts.

However it is always problematic to depend on another team and have to
wait for bugfixes. You must be able to bugfix yourself. Is that
addressed by the oss-qm? Than it seems a layer between the original
sources and the own project. It adds new complexity. Is that price
worth the gain?

Al

Re: [gentoo-user] Re: Shared libraries in Gentoo

[Enrico Weigelt]

* Al <oss.elmar@googlemail.com> wrote:

> Somehow I don't really like the way it is done. The levels of
> abstraction are mixed and it results in very cryptic parameters. 

Yes. Historically grown.

I've did a little proof-of-concept for developing an generic
abstraction of toolchain operations in the unitool project:

    git://pubgit.metux.de/projects/unitool.git
    
> > When you're going into the autotools hell. Also completely
> > obsoleted before it even came into existence. A set of well-
> > designed shell functions could do the job *much* better.
> >
> 
> The shell script terminology is a little strange, especially the
> conditions. They differ from the style of most modern languages,
> differ even from C. 

Well, but it's quite good for the jobs it was invented for.
I've did some experiments on generalizing typical configure
script stuff into shell functions, which works quite well.
That's the way I'd suggest for the future.

A completely different idea would be completely dropping the 
imperative and process-driven approach in favour of an declarative
model, which just describes the structure of an software component.

    git://pubgit.metux.de/projects/treebuild.git

> I wanted to stress how this "making easier" by another wrapper and
> another wrapper leads finnally into kind of a debugging hell, that
> makes matters more complicated in the end. Having different levels of
> abstraction is fine, but they should do it in clean way without mixing
> levels at least.

Yes. And truely, some ebuilds do things they IMHO shouldnt do.
For example, ebuilds should _never_ change source trees arbitrarily,
instead just have a defined set of patches (independent from useflags)
against the original sourcetree.

But that's a problem of some individual ebuilds, not emerge in general.

> It's clear that the overall package management is a different layer
> from program building. However the question of portablility for
> example is mixed into all levels, is it libtool or is it eclasses. It
> will be similar for things like internationalization.

Most of these issues should _not_ be mixed onto several layers,
that's one of the major problems, upstream's misdesign, tolerated
by downstreams/distros ;-P

> Ideally it can. In praxis the stack can't always be coped
> layer-by-layer by different teams. In the moment the bug is there, you
> have to search them all.

You just need access to all layers and test from buttom up.

> For my target of a port I have to understand them all, up to a certain
> degree. I have to apply patches at different layers.

There should only be exactly _one_ place for applying patches:
a QM layer in the middle between upstream release and distro's
package management. And now we're at the OSS-QM project again ;-p

> For my case that would be of help. I currently have to mix patches
> from cygwin with patches from Gentoo. Sure that leads to conflicts.

The problem here is that there's now common QM layer of Gentoo
*and* Cygwin (and other distros), between upstream and distros.
And here we see a typical problem: individual distros tend to do
only distro-specific patches, which often are not upstream-capable,
instead of _generic_ fixes.

Again: OSS-QM ;-p
 
> However it is always problematic to depend on another team and have to
> wait for bugfixes. You must be able to bugfix yourself. Is that
> addressed by the oss-qm? 

Yes, that's exactly the primary goal of it.


cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [gentoo-user] Re: Shared libraries in Gentoo

[Al]

>
> When you're going into the autotools hell. Also completely
> obsoleted before it even came into existence. A set of well-
> designed shell functions could do the job *much* better.
>

While porting to cygwin I can be happy when they use it. For my first
impression those libraries are more easy to port. They produce
libraries with a *.dll.a suffix like the native libraries of Cygwin.

The other example is libz. AFAIK it has a manually written configure
script. It generates libz.so. bzip2 ends up in error messages until I
build it statically.

I still try to understand the relation of shared libraries and dynamic
libraries. I read that dynamic libraries are linked at runtime. I also
read, that you can dynamically link againgst a shared as well as
against a normal library.

But isn't a normal library also shared when multiple programs link it
at runtime or does shared library mean it is shared in memory (PIC)?

Al

Re: [gentoo-user] Shared libraries in Gentoo

[Andrea Conti]

> I still try to understand the relation of shared libraries and dynamic
> libraries. I read that dynamic libraries are linked at runtime. I also
> read, that you can dynamically link againgst a shared as well as
> against a normal library.

Pardon me, but in my opinion if you're asking this kind of questions,
porting the Gentoo build system (or even any non-trivial application
without upstream support) to a different and basically unsupported
environment is way beyond what you can manage with your current level of
technical expertise.

In other words, this is not the kind of thing you can solve by
iteratively trying to build, look at what breaks and doing a point fix.

I am not saying it can't be done, but porting is hard and requires an
in-depth knowledge of the source and the target environment, plus a lot
of development experience in both. You should begin with that, instead
of diving head-first into what is all but a simple task.

just my €0.02,
andrea

Re: [gentoo-user] Shared libraries in Gentoo

[Al]

Ciao Andrea,

> Pardon me, but in my opinion if you're asking this kind of questions,
> porting the Gentoo build system (or even any non-trivial application
> without upstream support) to a different and basically unsupported
> environment is way beyond what you can manage with your current level of
> technical expertise.

I don't know how far I will get. But if I don't have a lot of
experience in the C environment, it doesn't mean that I am out of
experience or completly dumb.

> In other words, this is not the kind of thing you can solve by
> iteratively trying to build, look at what breaks and doing a point fix.
>

It is not a port between two unsupported environments. The Cygwin
layer is well supported. Gentoo Prefix does already run on Interix. I
only try to reach a wider audience be porting the existing sources to
the Cygwin Layer. The gap to close is rather small.

> I am not saying it can't be done, but porting is hard and requires an
> in-depth knowledge of the source and the target environment, plus a lot
> of development experience in both. You should begin with that, instead
> of diving head-first into what is all but a simple task.

Maybe the learning curve is to steep and I have to give up. I already
reached more than expected and I did learn a lot. It's not in vain. I
have documented the results of my research detailed in the wiki.
Anybody can pick it up at that point and doesn't need to go the way
from the very beginning.

I am not the first one, who tried and sure I am not the last. I also
profit from the documentations of my forerunners.

Al

Re: [gentoo-user] Shared libraries in Gentoo

[Enrico Weigelt]

* Andrea Conti <alyf@alyf.net> wrote:

> Pardon me, but in my opinion if you're asking this kind of questions,
> porting the Gentoo build system (or even any non-trivial application
> without upstream support) to a different and basically unsupported
> environment is way beyond what you can manage with your current level of
> technical expertise.

Let him try - learning by doing :)


cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [gentoo-user] Re: Shared libraries in Gentoo

[Enrico Weigelt]

* Al <oss.elmar@googlemail.com> wrote:

> While porting to cygwin I can be happy when they use it. For my first
> impression those libraries are more easy to port. They produce
> libraries with a *.dll.a suffix like the native libraries of Cygwin.

Just a few years ago, autotools (especially w/ libtool) was totally
unusable for any kind of isolated (not just crosscompiling),
based on completely stupid assumptions and practically undebuggable.

All the functionality could be easily done w/ shell functions
instead of syntactically unstable m4 macros. I've started a little
bit of hacking @ zlib (see oss-qm repo) how that could look like.

> The other example is libz. AFAIK it has a manually written configure
> script. It generates libz.so. bzip2 ends up in error messages until I
> build it statically.

AFAIK zlib properly builds shared library as well as static one
(at least on unixoid targets). Win32 targets are still a bit messy,
but there's work ongoing on that.
 
> I still try to understand the relation of shared libraries and dynamic
> libraries. I read that dynamic libraries are linked at runtime. I also
> read, that you can dynamically link againgst a shared as well as
> against a normal library.
> 
> But isn't a normal library also shared when multiple programs link it
> at runtime or does shared library mean it is shared in memory (PIC)?

Well, static libraries are essentially an archive of plain object
files (maybe with an additional symbol table for faster lookup).
They essentially get linked in at build time just like plain objects.

Shared libraries are different: they really get linked together,
certain sections (especially symbol tables) are merged, local
references resolved. An shared library actually is one big object
file (that's why they're also called shared object). Now these
objects are loaded and linked-in at the process startup phase 
(or later using dlopen()). At this point, "shared" means that
multiple programs can use the very same code from one shared
object file.

Another level of sharing is at runtime, by using shared pages by
the MMU. This is a bit more complex: you need to construct the
binary code in a way that multiple processes can map it into 
separate address offsets (historical systems require the addresses
to be defined at compile time, which obviously is impractical).
At that point -fPIC comes into play: the code is generated in a
way that it's position in process's address space doesnt matter
anymore (at least on page-granularity). So calling processes
can directly map the shared object's text segments into quite
arbitraty offsets without touching the actual code pages, and
the MMU only has to maintain one physical copy of them.

PIC tends to be a little bit larger and slower than non-PIC
(more indirect addressing), but on most today's machines
the impact on real workloads is questionable.


cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [gentoo-user] Re: Shared libraries in Gentoo

[Enrico Weigelt]

* Alan McKinnon <alan.mckinnon@gmail.com> wrote:

> > True. But FreeBSD isn't that popular like Windows, Mac or Linux.
> 
> So you don't work at a Tier 1 ISP then?
> 
> FreeBSD rules that space. I get hugely better performance out of Postfix on 
> FreeBSD than on Linux - all other ISPs in this country concur.

Well, not everybody is a tier-1 isp ... ;-o

BTW: one of my customers, a really big one here in Germany
(who also has several of the major free mail portals) runs
its mail systems on GNU/Linux (well, inhouse mailing is done
via Exchange+ADS, surprisingly it actually works ;-)).

But I'd really like to know what produces the performance hits
on Posfix @ Linux.
 
> In fact, portage is complete overkill and I refuse to allow it
> to be deployed at work. Check my posting history for the
> rationale behind this.

Well, portage could be much thinner if certain things would be
moved explicitly out-of-scope or solved more generic on a 
different layer. (yes, I'm explicitly ignoring the historical
issues right now ;-p).

For example:

* distro-specific and various source retrieval methods would not
  be necessary, if the packaging/distro-build system would simply
  fetch it's sources from an vcs (eg. git ;-p) using canonical
  versioning/namespace scheme [1].

* instead of useflags (the terminology implies we're switching
  things some package *uses*, not provides), model the available
  features, eg. like Briegel [2] does. (that's more a methological
  that a technical issue).

* instead of slotting, assign separate package names when multiple
  version concurrency is required (and maybe pull them together
  via virtuals)

* rely on an pure DAG as dependency graph - per definition.
  when circular dependencies occour, fix them in the source tree,
  for example splitting off certain packages in several smaller ones.


[1] http://www.metux.de/download/oss-qm/normalized_repository.pdf
[2] https://sourceforge.net/p/briegel/


Don't get me wrong, that shall not be understood as ranting against
Gentoo, just showing suitable approaches we'd start afresh on a
"green grassland" (w/o all the historical burdens).


cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [gentoo-user] Re: Shared libraries in Gentoo

[Alan McKinnon]

Apparently, though unproven, at 02:56 on Thursday 09 September 2010, Enrico 
Weigelt did opine thusly:

> * Alan McKinnon <alan.mckinnon@gmail.com> wrote:
> > > True. But FreeBSD isn't that popular like Windows, Mac or Linux.
> > 
> > So you don't work at a Tier 1 ISP then?
> > 
> > FreeBSD rules that space. I get hugely better performance out of Postfix
> > on FreeBSD than on Linux - all other ISPs in this country concur.
> 
> Well, not everybody is a tier-1 isp ... ;-o
> 
> BTW: one of my customers, a really big one here in Germany
> (who also has several of the major free mail portals) runs
> its mail systems on GNU/Linux (well, inhouse mailing is done
> via Exchange+ADS, surprisingly it actually works ;-)).
> 
> But I'd really like to know what produces the performance hits
> on Posfix @ Linux.

It comes down to the IO scheduler. Linux is designed to be general purpose. 
FreeBSD is designed to be much more specific.

Both are very good at what they do, the trick is in realising what those 
things are and playing to their strengths.


> > In fact, portage is complete overkill and I refuse to allow it
> > to be deployed at work. Check my posting history for the
> > rationale behind this.
> 
> Well, portage could be much thinner if certain things would be
> moved explicitly out-of-scope or solved more generic on a
> different layer. (yes, I'm explicitly ignoring the historical
> issues right now ;-p).

My beef with portage in my specific production setup is the amount of work it 
takes my guys to keep everything up to date. We don't have 150 identical 
servers in a farm (I'd love that and would switch to Gentoo immediately if it 
were). I have 130 completely different configs and uses for those servers.

The maintenance admins would have to fully grok all of portage, the 
implications, predict the outcome and understand what they are about to update 
every time they do an update. And they'd have to know it for 130 permutations.

Heck, *I* can't track that, I won't expect someone else to. Centos better 
suits our needs - deploy X, you know what you are going to get. Having said 
that, all my personal stuff and my own dev boxes are Gentoo. Why? Coz I can 
change stuff around for testing on a whim, figure out the right approach then 
document what to do on Centos to achieve that.


> 
> For example:
> 
> * distro-specific and various source retrieval methods would not
>   be necessary, if the packaging/distro-build system would simply
>   fetch it's sources from an vcs (eg. git ;-p) using canonical
>   versioning/namespace scheme [1].
> 
> * instead of useflags (the terminology implies we're switching
>   things some package *uses*, not provides), model the available
>   features, eg. like Briegel [2] does. (that's more a methological
>   that a technical issue).
> 
> * instead of slotting, assign separate package names when multiple
>   version concurrency is required (and maybe pull them together
>   via virtuals)
> 
> * rely on an pure DAG as dependency graph - per definition.
>   when circular dependencies occour, fix them in the source tree,
>   for example splitting off certain packages in several smaller ones.
> 
> 
> [1] http://www.metux.de/download/oss-qm/normalized_repository.pdf
> [2] https://sourceforge.net/p/briegel/
> 
> 
> Don't get me wrong, that shall not be understood as ranting against
> Gentoo, just showing suitable approaches we'd start afresh on a
> "green grassland" (w/o all the historical burdens).
> 
> 
> cu

-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] Re: Shared libraries in Gentoo

[Al]

> My beef with portage in my specific production setup is the amount of work it
> takes my guys to keep everything up to date. We don't have 150 identical
> servers in a farm (I'd love that and would switch to Gentoo immediately if it
> were). I have 130 completely different configs and uses for those servers.
>

How about migrating them one-by-one to CentOs first and in a second
step to Gentoo?

Sure every servers migration is work. But there will be routine once
you have migrated 10.

Having a uniform serverfarm should pay the efforts after a reasonable time.

Al

Re: [gentoo-user] Re: Shared libraries in Gentoo

[Alan McKinnon]

Apparently, though unproven, at 13:50 on Thursday 09 September 2010, Al did 
opine thusly:

> > My beef with portage in my specific production setup is the amount of
> > work it takes my guys to keep everything up to date. We don't have 150
> > identical servers in a farm (I'd love that and would switch to Gentoo
> > immediately if it were). I have 130 completely different configs and
> > uses for those servers.
> 
> How about migrating them one-by-one to CentOs first and in a second
> step to Gentoo?

yum install is do once, forget, predict what's gonna happen with high 
certainty.

gentoo is run emerge, study output, understand all of it, consider what 
flameeyes has to say about it, wonder if some screw ball fucked up glibc yet 
again, discuss in upgrade meetings, then proceed with lots of other crap ad 
nauseam.

Ain't gonna happen in this environment.

> Sure every servers migration is work. But there will be routine once
> you have migrated 10.

There is nothing routine about gentoo upgrades. Long hard experience has 
taught me this.

> Having a uniform serverfarm should pay the efforts after a reasonable time.

I don't have a uniform server farm and never will. It's not that kind of 
environment.

It's 130 different configs on 150 machines serving 30 different systems, many 
of them legacy systems. Management once asked what it will take to unlegacy 
all of that. They didn't like my answer:

Triple the salary budget
Two years.



-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] Re: Shared libraries in Gentoo

[Al]

> gentoo is run emerge, study output, understand all of it, consider what
> flameeyes has to say about it, wonder if some screw ball fucked up glibc yet
> again, discuss in upgrade meetings, then proceed with lots of other crap ad
> nauseam.

That's why a was talking of CentOs as fist step. Conservative,
unexperimental approach.

Then you have those guys, that deal with 130 different machines now.
Once uniformly, they could all do Gentoo debugging and patching. The
distro would instantly work as smooth you want it.

> It's 130 different configs on 150 machines serving 30 different systems, many
> of them legacy systems. Management once asked what it will take to unlegacy
> all of that. They didn't like my answer:
>
> Triple the salary budget
> Two years.

That's cheap. Isn't a factor of ten always more to the truth?

Al

Re: [gentoo-user] Re: Shared libraries in Gentoo

[Alan McKinnon]

Apparently, though unproven, at 15:20 on Thursday 09 September 2010, Al did 
opine thusly:

> > gentoo is run emerge, study output, understand all of it, consider what
> > flameeyes has to say about it, wonder if some screw ball fucked up glibc
> > yet again, discuss in upgrade meetings, then proceed with lots of other
> > crap ad nauseam.
> 
> That's why a was talking of CentOs as fist step. Conservative,
> unexperimental approach.
> 
> Then you have those guys, that deal with 130 different machines now.
> Once uniformly, they could all do Gentoo debugging and patching. The
> distro would instantly work as smooth you want it.

You are not familiar with my environment. I am.

We will not use Gentoo here. This is an informed decision and the demonstrable 
fact that can make me change my mind does not exist.


-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] Re: Shared libraries in Gentoo

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 1165 bytes --]

On Thu, 9 Sep 2010 14:25:07 +0200, Alan McKinnon wrote:

> gentoo is run emerge, study output, understand all of it, consider what 
> flameeyes has to say about it, wonder if some screw ball fucked up
> glibc yet again, discuss in upgrade meetings, then proceed with lots of
> other crap ad nauseam.

Part of the problem is that there is no such thing as a stable Gentoo
installation. the portage tree is changing all the time and emerge -u
world wants to continually install new stuff when the old stuff works
fine. It would need some sort of snapshot system where you can pin to a
particular point in the portage tree's history and only bug/security
updates would be applied.

> It's 130 different configs on 150 machines serving 30 different
> systems, many of them legacy systems. Management once asked what it
> will take to unlegacy all of that. They didn't like my answer:
> 
> Triple the salary budget
> Two years.

If you really wanted it, you could quote lower and then slightly revise
upwards in light of "unforseen circumstances"... rinse and repeat.


-- 
Neil Bothwick

A man needs a mistress - just to break the monogamy

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Re: Shared libraries in Gentoo

[Alan McKinnon]

Apparently, though unproven, at 17:12 on Thursday 09 September 2010, Neil 
Bothwick did opine thusly:

> > It's 130 different configs on 150 machines serving 30 different
> > systems, many of them legacy systems. Management once asked what it
> >
> > will take to unlegacy all of that. They didn't like my answer:
> > 
> >
> > Triple the salary budget
> > Two years.
> 
> If you really wanted it, you could quote lower and then slightly revise
> upwards in light of "unforseen circumstances"... rinse and repeat.


I don't want it.

I like Gentoo in -dev
I will not allow Gentoo anywhere near -prod

This is a very considered decision, as a result of already having to fix (many 
times) the monumental fuck ups that happen when it is done in an environment 
such as this.

Why should I do all the work of pinning packages to known good versions when 
the RHEL devs have already done all the heavy lifting for me?



-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] Re: Shared libraries in Gentoo

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 824 bytes --]

On Thu, 9 Sep 2010 17:26:18 +0200, Alan McKinnon wrote:

> I don't want it.  

I know.

> I like Gentoo in -dev
> I will not allow Gentoo anywhere near -prod
> 
> This is a very considered decision, as a result of already having to
> fix (many times) the monumental fuck ups that happen when it is done in
> an environment such as this.
> 
> Why should I do all the work of pinning packages to known good versions
> when the RHEL devs have already done all the heavy lifting for me?  

You shouldn't, that's why I made the point about there not being a
stable, frozen version of the portage tree. If that existed, pinning at a
stable setup would be doable, otherwise it would be a nightmare.


-- 
Neil Bothwick

Vital papers will demonstrate their vitality by moving to where you
can't find them.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Re: Shared libraries in Gentoo

[Graham Murray]

Alan McKinnon <alan.mckinnon@gmail.com> writes:

> Why should I do all the work of pinning packages to known good versions when 
> the RHEL devs have already done all the heavy lifting for me?

The problem with that is when you are starting a new project now, but
the packages were pinned down quite a few years ago and therefore seem
to be rather out of date. 

Re: [gentoo-user] Re: Shared libraries in Gentoo

[Enrico Weigelt]

* Alan McKinnon <alan.mckinnon@gmail.com> wrote:

> > But I'd really like to know what produces the performance hits
> > on Posfix @ Linux.
> 
> It comes down to the IO scheduler. Linux is designed to be general purpose. 
> FreeBSD is designed to be much more specific.

hmm, Linux provides several io schedulers - does choosing another
one help here ?

> > Well, portage could be much thinner if certain things would be
> > moved explicitly out-of-scope or solved more generic on a
> > different layer. (yes, I'm explicitly ignoring the historical
> > issues right now ;-p).
> 
> My beef with portage in my specific production setup is the amount of work it 
> takes my guys to keep everything up to date. 

Yes, that's still a problem. I myself for example have a dozen of 
containers w/ Gentoo, and I'm still trying to figure out how to
control which packages should be updated fully automatically, w/o 
triggering updates of others that I want to approve explicitly.

Once I've ported a buch of more packages to sysroot'ed builds,
I'll migrate to Briegel[1]-built images.


cu

[1] https://sourceforge.net/p/briegel/
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [gentoo-user] Re: Shared libraries in Gentoo

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 627 bytes --]

Am 10.09.2010 01:49, schrieb Enrico Weigelt:
> * Alan McKinnon <alan.mckinnon@gmail.com> wrote:
> 
>>> But I'd really like to know what produces the performance hits
>>> on Posfix @ Linux.
>>
>> It comes down to the IO scheduler. Linux is designed to be general purpose. 
>> FreeBSD is designed to be much more specific.
> 
> hmm, Linux provides several io schedulers - does choosing another
> one help here ?
> 

And if it's so good for certain workloads, why hasn't FreeBSD's
scheduler been ported to Linux? Or is the whole software stack (block
devices etc.) so completely different that it wouldn't work?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [gentoo-user] Re: Shared libraries in Gentoo

[Alan McKinnon]

Apparently, though unproven, at 18:43 on Friday 10 September 2010, Florian 
Philipp did opine thusly:

> Am 10.09.2010 01:49, schrieb Enrico Weigelt:
> > * Alan McKinnon <alan.mckinnon@gmail.com> wrote:
> >>> But I'd really like to know what produces the performance hits
> >>> on Posfix @ Linux.
> >> 
> >> It comes down to the IO scheduler. Linux is designed to be general
> >> purpose. FreeBSD is designed to be much more specific.
> > 
> > hmm, Linux provides several io schedulers - does choosing another
> > one help here ?
> 
> And if it's so good for certain workloads, why hasn't FreeBSD's
> scheduler been ported to Linux? Or is the whole software stack (block
> devices etc.) so completely different that it wouldn't work?

I don't have an answer for that - just some observations that are quite old 
and maybe not even valid anymore.

Either way, I have mail relays that pump 3,000,000 mails each per day and have 
done so flawlessly for years. I'm not about to risk that to see if I can find 
another 0.x% performance gain :-)


-- 
alan dot mckinnon at gmail dot com

[gentoo-user] Re: Shared libraries in Gentoo

[walt]

On 09/10/2010 09:43 AM, Florian Philipp wrote:

> ..why hasn't FreeBSD's
> scheduler been ported to Linux? Or is the whole software stack (block
> devices etc.) so completely different that it wouldn't work?

Well, I can't answer your question, but you just reminded me that Linus
himself once declared that the scheduler "is not an interesting problem".
It certainly seems to be an emotional problem, though, based on more
recent events.

This is really more for linux gossip junkies than for techno-geeks:

http://en.wikipedia.org/wiki/Con_Kolivas

Re: [gentoo-user] Re: Shared libraries in Gentoo

[Enrico Weigelt]

* Al <oss.elmar@googlemail.com> wrote:

> I think there is a future for second level managers that can be
> installed into multiple OS and yet set up the very same POSIX
> invironement. Having that you can build complex software that is
> portable. 

IMHO the most work intensive stuff (on per-package basis) is all
the QM. Parts of it can be done generically, assisted by automatic
massbuilds and various auto-test mathods (some of them already
built into portage), but still leaving much target or even install
specific stuff (eg. will some package foo work properly with 
certain cflags and libc-version X ?).

> You don't depend on Java. You don't need to run a virtual server.

You never really needed Java, but proper build systems. What we 
could gain here is saving a lot of distro-specific extra works if
some distro like Gentoo directly works on different targets
environments.

The interesting point in Java is that it is an (well, was) an 
very cleanly defined virtual machine (even virtual processor)
environment which can be emulated on virtually any known machine
(assuming it has enough resources). That could also work with
plain C, if the basic APIs are stricly and cleanly defined.
(see Plan9 vs. plan9port).

> Currently there are two canditates. One candidate is Cygwin Ports, the
> other one is Gentoo Prefix. Cygwin Ports just added cross-compilation
> features into the latest edition. Still Cygwin is limited to Windows.

Let me add a third candidate: Briegel. It's based on crosscompiling
from ground up. It's not really a distro, but more a generic build
system, as a basic building block for distro generation. But beware
that it's based on very different concepts than traditional distro
build systems.


cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [gentoo-user] Re: Shared libraries in Gentoo

[Al]

Hello Enrico,

I did read with interest the informations about Briegel, and the
oss-qm idea. I also looked into the PDF.

I like the argumentation and the overall idea of such a repository.
Maybe it will work.

I fear there are some drawbacks. In my estimation you can compare
patches to closed sources to some degree. Even if they are pro forma
open source they are mainly usefull for the very distribution. In this
sense they are the propriatary work and the capital of each distro. So
I guess it will be difficult to convince them to store their patches
into a common QM repository. Forks of a distro would become much more
easy and like more successful.

However it's worth I try. OpenSource works and nobody would have
predicted that 40 years ago.

> You never really needed Java, but proper build systems. What we
> could gain here is saving a lot of distro-specific extra works if
> some distro like Gentoo directly works on different targets
> environments.

That's true as far as we speak of unixoid environments. To port
programs to other environments requires more efforts than just a
proper build system. The original idea of Java was, to bring a
standard layer into any environment, that would remove the need of
ports.

> The interesting point in Java is that it is an (well, was) an
> very cleanly defined virtual machine (even virtual processor)

One interesting point is, that it is was very successfull, but on
completly different fields than origninally targeted. Most Java
doesn't run platform independent programs, but runs servers, that
don't need to be platform independent at all.

> environment which can be emulated on virtually any known machine
> (assuming it has enough resources). That could also work with
> plain C, if the basic APIs are stricly and cleanly defined.
> (see Plan9 vs. plan9port).

Java was also designed to facilitate programming comparing C. Still
Java itself isn't the last clue. Take the horrible way to copy a
simple array. The fewer programming in C facilitates matters. It still
makes sense in the Unix environment, but rather for historical
dependencies and performance than for simplicity.

>
> Let me add a third candidate: Briegel. It's based on crosscompiling
> from ground up. It's not really a distro, but more a generic build
> system, as a basic building block for distro generation. But beware
> that it's based on very different concepts than traditional distro
> build systems.

That is very interesing. Briegel is indeed related to what I currently
do. I have a special usecase in mind. I only want to do the basics as
far as necessary, because a portable Posix environment is still
missing yet, then turn back to my original idea.

Briegel is strongly focused on the technolgical basics, without
talking of all possible usecases. At least mine was not addressed. :-)

Still Briegel looks like a one-man-show while there are already
communities behind the Cygwin and the Gentoo candidate.

What is your strategia to build up a community?

Al

Re: [gentoo-user] Re: Shared libraries in Gentoo

[Enrico Weigelt]

* Al <oss.elmar@googlemail.com> wrote:

> Even if they are pro forma open source they are mainly usefull
> for the very distribution. 

No, many patches are quite generic or could be easily fixed
to be that. OSS-QM makes sharing and automatic notification
on new patches easier.

> In this sense they are the propriatary work and the capital of 
> each distro. So I guess it will be difficult to convince them
> to store their patches into a common QM repository. 

That's why I'm working on automatic imports.

For Debian packages (at least those complying the recent
guidelines) it's almost done (for now just for a few packages
as proof-of-concept, but can be made more generic).

Gentoo is a bit trickier, since patching is called explicitly
from each ebuild (sometimes even useflag-dependent). The Gentoo
devs could help a great deal if they would set a policy like:

* all patches have to be normalized (eg. *always* apply w/ -p1)
* a easily machine-readable (eg. via grep+sed ;-o) list of patches
  to apply on a specific version (eg. in the ebuild or a separate
  file)
* make the patches independent from useflags
* do sourcetree changes _exclusively_ via patches (no file copy-in's
  or directly changing files w/ sed etc)

> That's true as far as we speak of unixoid environments. To port
> programs to other environments requires more efforts than just a
> proper build system.

Yes. The individual package has to sit ontop of properly defined
interfaces (eg. POSIX), and require the target to provide that
(if it doesn't, fix the target's system libs, toolchain, etc).

> The original idea of Java was, to bring a standard layer into
> any environment, that would remove the need of ports.

Yes, but that's a matter of the basic class libraries, not
the language. The same can also be done in plain C.

> > The interesting point in Java is that it is an (well, was) an
> > very cleanly defined virtual machine (even virtual processor)
> 
> One interesting point is, that it is was very successfull, but on
> completly different fields than origninally targeted. Most Java
> doesn't run platform independent programs, but runs servers, that
> don't need to be platform independent at all.

Well, that's a quite strange effect. IMHO that doesnt have anything
to do w/ the platform agnosticism, but the language concepts which
might be better suited for those applications.

> Java was also designed to facilitate programming comparing C. Still
> Java itself isn't the last clue. Take the horrible way to copy a
> simple array. 

Yes, there're also other concepts I miss in Java, eg. native 
associative arrays, language constructs for parallelisms, etc.

> Briegel is strongly focused on the technolgical basics, without
> talking of all possible usecases. At least mine was not addressed. :-)

Yes, it's focused on the basic job of building packges to virtually
any target platform (from embedded to clusters ;-p). A bit similar
philosophy as git ;-)
 
> Still Briegel looks like a one-man-show while there are already
> communities behind the Cygwin and the Gentoo candidate.

True, right now I'm the only active developer here. Historically
it had been an proprietary product as building block for very
customer-specific setups, which I published as oss a while ago.

> What is your strategia to build up a community?

Actually, I don't really have any. All I can do is offering it
as OSS and do a little bit advocacy here and there - I don't
have the resources to build up real community structures all
alone. Of course, anybody's welcomed to join in.


cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [gentoo-user] Re: Shared libraries in Gentoo

[Al]

>> What is your strategia to build up a community?
>
> Actually, I don't really have any. All I can do is offering it
> as OSS and do a little bit advocacy here and there - I don't
> have the resources to build up real community structures all
> alone. Of course, anybody's welcomed to join in.
>

This is still a weak point. What are the options to run an OS project
sucessfully?

1.) The field is small and can be covered by a single person. ->
cdrecord from Jörg Schilling

2.) The field is wide and there is a strong community behind. -> Gentoo

3.) The field ist wide and there is a working commercial model -> Redhat, Cygwin

x.) There are some others.

My very subjective estimations:

Briegel looks like type 3. Needs a working company structure, then
there could be money in it. Are you prepared for this? QM Repository,
not standalone, but as prerequest for Briegel and nourished by it.

Maybe it would be possibel to run Briegel as type 2. But a community
has special prerequests to get it started, communiction, server
management, events and meetings. A lot of non-programming work also
for this.

Al

Re: [gentoo-user] Re: Shared libraries in Gentoo

[Enrico Weigelt]

* Al <oss.elmar@googlemail.com> wrote:
> >> What is your strategia to build up a community?
> >
> > Actually, I don't really have any. All I can do is offering it
> > as OSS and do a little bit advocacy here and there - I don't
> > have the resources to build up real community structures all
> > alone. Of course, anybody's welcomed to join in.
> >
> 
> This is still a weak point.

Just let the project grow. It does not require anyone jumping into
it to stick with it for long time. All I'm proposing right now is
adopt the model, which makes collaboration w/ other distros easier.
It's something like an aggreement as FHS.

BTW: meanwhile I've build some automatic import mechanisms
(unfortunately, this approach won't work well for Gentoo, as I
don't have the fundamental data for it available, especially some
deterministic mapping between upstream tarball and the patches
to apply ... that probably would require some kind of fake-emerge)

> Briegel looks like type 3. 

Hey, wait a minute ... we were talking about OSS-QM, thats completely
different project. Briegel is a build system (somewhat similar to
portage, but yet quite different concepts behind), which just happens
to work smoothly w/ oss-qm.


cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [gentoo-user] Re: Shared libraries in Gentoo

[Al]

>
> Just let the project grow. It does not require anyone jumping into
> it to stick with it for long time. All I'm proposing right now is
> adopt the model, which makes collaboration w/ other distros easier.
> It's something like an aggreement as FHS.

Hope you have the power to go that way far enough on your own until it
becomes interesting for more people.

>
> BTW: meanwhile I've build some automatic import mechanisms
> (unfortunately, this approach won't work well for Gentoo, as I
> don't have the fundamental data for it available, especially some
> deterministic mapping between upstream tarball and the patches
> to apply ... that probably would require some kind of fake-emerge)
>

I did good progress with Gentoo on Cygwin this WE. 3 big packages are
directly before me now: Python, Portage and GCC.

> Hey, wait a minute ... we were talking about OSS-QM, thats completely
> different project. Briegel is a build system (somewhat similar to
> portage, but yet quite different concepts behind), which just happens
> to work smoothly w/ oss-qm.

A build system needs sources and patches to build. So I think they are
related. If your patch import system works, it can become very
interesting. You can run something similar to Gentoo without the need
of a big community at the beginning. So you have all freedom to
customize it just the way you want it.

Al

Re: [gentoo-user] Re: Shared libraries in Gentoo

[Enrico Weigelt]

* Al <oss.elmar@googlemail.com> wrote:

> Hope you have the power to go that way far enough on your own until it
> becomes interesting for more people.

Well, let's see where it goes. I'll continue my work and it
seems that soon a few others might jump in.

> I did good progress with Gentoo on Cygwin this WE. 3 big packages are
> directly before me now: Python, Portage and GCC.

That'll be the most troublesome ones of all ;-)

> > Hey, wait a minute ... we were talking about OSS-QM, thats completely
> > different project. Briegel is a build system (somewhat similar to
> > portage, but yet quite different concepts behind), which just happens
> > to work smoothly w/ oss-qm.
> 
> A build system needs sources and patches to build. So I think they are
> related. 

I've designed the two projects so that they're independent from
each other, but just play very well with together.

OSS-QM repositories [1] are layed out in a way that it's trivial 
to fetch specific version of the packages, and also allows separate
namespaces (prefixes) for individual distros (so they can push their
work directly to oss-qm w/o stepping onto other's feet).

> If your patch import system works, it can become very interesting. 

Well, for Debian it already works quite well, Gentoo still is
unsolved, as I'm lacking information to construct the history.
(a little help from Gentoo devs would really be appreciated).

> You can run something similar to Gentoo without the need
> of a big community at the beginning. So you have all freedom to
> customize it just the way you want it.

Actually, I already *do* have something similar - by combination
of OSS-QM and Briegel. But yet only a few packages for certian
embedded targets.


cu

[1] http://www.metux.de/download/oss-qm/normalized_repository.pdf
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [gentoo-user] Re: Shared libraries in Gentoo

[Al]

2010/9/7 Ajai Khattri <ajai@bway.net>:
> On Tue, 7 Sep 2010, Al wrote:
>
>>> When you say Gentoo, do you mean Portage? Remember Windows has a lot of
>>> limitations that WILL get in your way so dont be surprised when things
>>> break.
>>
>> I am specially interested in Gentoo because it is not another linux
>> distribution, but an administration tool to build your own sources and
>> it's scope is wider than linux.
>
> Which doesn't actually answer the question...

Yes, I mean Portage, with the Gentoo Prefix overlay and hence without
the Linux kernel to be most precise. Gentoo Prefix lives on the second
floor. However with that it leaves the scope of gentoo-user.

For now I was interested to find out how it works in the standard
case. From there I have to conquer the exotic worlds mostly on my own.

Al

Re: [gentoo-user] Re: Shared libraries in Gentoo

[Al]

>> How does a program in Gentoo know, where to look for shared libraries?
>
> Try running "ldconfig -p", which relates to Nikos's comment about
> ld.so.conf.
>
As you both indicated me in into that direction I played with libtool.
During usage it gives a comprehensive answer to our question:

<quote libtool>
If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the `-LLIBDIR'
flag during linking and do at least one of the following:
   - add LIBDIR to the `LD_LIBRARY_PATH' environment variable
     during execution
   - add LIBDIR to the `LD_RUN_PATH' environment variable
     during linking
   - use the `-Wl,-rpath -Wl,LIBDIR' linker flag
   - have your system administrator add LIBDIR to `/etc/ld.so.conf'

See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.
</quote>

Al