[gentoo-dev] [RFC] RESTRICT=parallel for builds that can't be executed in parallel

[gentoo-dev] [RFC] RESTRICT=parallel for builds that can't be executed in parallel

[Zac Medico]

Hi everyone,

Should we add a RESTRICT=parallel value for ebuilds that can't be
built at the same time as other ebuilds? Brian says we need it for
things like xorg-server which calls eselect opengl.

If we truly need this, is RESTRICT=parallel a good name? We could
make it a PROPERTIES value instead, but then we'd probably end up
with something like PROPERTIES=no-parallel, so maybe RESTRICT is better.
-- 
Thanks,
Zac

Re: [gentoo-dev] [RFC] RESTRICT=parallel for builds that can't be executed in parallel

[Michał Górny]

On Tue, 13 Apr 2010 19:12:08 -0700
Zac Medico <zmedico@gentoo.org> wrote:

> Should we add a RESTRICT=parallel value for ebuilds that can't be
> built at the same time as other ebuilds? Brian says we need it for
> things like xorg-server which calls eselect opengl.

I don't think that's the right solution. In most cases, xorg-server can
be built in parallel with stuff which doesn't require/set specific
opengl subsystem set.

Well, in fact is there _really_ any package which won't work with
switched opengl? I guess it's more of a runtime problem that buildtime,
and I don't think we really should print out loudly 'libGL has been
switched, please do not start OpenGL apps right now'.

In fact, the best solution in this particular case would be to patch
the buildsystem to use Gentoo location for particular OpenGL headers.

Disabling parallel emerge would be more of a workaround for the issue,
and will influence much more packages than it needs to. And it won't
help if user is running multiple emerge calls at the same time.

Another possible workaround is to enable some kind of 'eselect opengl'
locking so that another package requiring access to it will wait until
our build finishes. But this, of course, would require a quite good
solution for maintaining the lock and dropping it whenever build
process is aborted/killed.

-- 
Best regards,
Michał Górny

<http://mgorny.alt.pl>
<xmpp:mgorny@jabber.ru>

Re: [gentoo-dev] [RFC] RESTRICT=parallel for builds that can't be executed in parallel

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 3588 bytes --]

On Wed, Apr 14, 2010 at 07:45:20AM +0200, Michaaa GGGrny wrote:
> On Tue, 13 Apr 2010 19:12:08 -0700
> Zac Medico <zmedico@gentoo.org> wrote:
> 
> > Should we add a RESTRICT=parallel value for ebuilds that can't be
> > built at the same time as other ebuilds? Brian says we need it for
> > things like xorg-server which calls eselect opengl.
> 
> I don't think that's the right solution. In most cases, xorg-server can
> be built in parallel with stuff which doesn't require/set specific
> opengl subsystem set.

RESTRICT=parallel is basically a big lock that forces building to go 
down to one specific build/merge job- it's not at all fine grained.  
That said, I'm not convinced it's worth actually *trying* to be fine 
grained.

Stuff that needs this 'lock', if you look at it from the purely 
academic angle is broken.  The pkgs in question should be buildable 
without modifying the livefs.

From the pragmatic angle, fixing some of those packages is a pretty 
huge endeavour hence this lock existing.  I see no reason to encourage 
the usage of this lock by making it more fine grained, either.



> Well, in fact is there _really_ any package which won't work with
> switched opengl? I guess it's more of a runtime problem that buildtime,
> and I don't think we really should print out loudly 'libGL has been
> switched, please do not start OpenGL apps right now'.

Runtime and buildtime actually- consider a pkg that is mesa sensitive 
having those headers/libs switched out mid build.  That build likely 
is going to boom in a rather interesting way.

Runtime itself, swapping out the gl resource that is used (going from 
ati to x11 for building xorg-server) isn't going to make new apps that 
start happy either.


> In fact, the best solution in this particular case would be to patch
> the buildsystem to use Gentoo location for particular OpenGL headers.

Academically, you're right, it's the proper solution.  That takes a 
fair amount of time however.  More importantly, this issue *will* pop 
up again elsewhere, meaning we'll have to delay the pkg in question 
from being in the tree till it meets this higher QA standard.

Or we add this functionality, level the restrict, then go and fix the 
ebuild- pragmatic solution.


> Disabling parallel emerge would be more of a workaround for the issue,
> and will influence much more packages than it needs to. And it won't
> help if user is running multiple emerge calls at the same time.

Running multiple emerges in parallel is already a bad idea.  The 
solution for that case is for the new/second emerge to feed the 
request into the original emerge (or a daemon).

Keep in mind if support for multiple emerge invocations was 
implemented it would still need some RESTRICT=parallel functionality 
for screwed up pkgs like xorg-server.  Fixing multiple emerge 
invocations still requires fixing RESTRICT=parallel.


> Another possible workaround is to enable some kind of 'eselect opengl'
> locking so that another package requiring access to it will wait until
> our build finishes.  But this, of course, would require a quite good
> solution for maintaining the lock and dropping it whenever build
> process is aborted/killed.

The other thing to recall is that by the time eselect is called, the 
ebuilds environment may already be localized to settings that eselect 
controls (LDPATH, that 'currently selected implementation' GL var).  
My first thought is any locking scheme of that sort is going to be a 
bigger can of worms.

~brian

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] [RFC] RESTRICT=parallel for builds that can't be executed in parallel

[Justin]

[-- Attachment #1: Type: text/plain, Size: 532 bytes --]

On 14/04/10 04:12, Zac Medico wrote:
> Hi everyone,
> 
> Should we add a RESTRICT=parallel value for ebuilds that can't be
> built at the same time as other ebuilds? Brian says we need it for
> things like xorg-server which calls eselect opengl.

There is at least one other example which benefits from singlular build,
atlas libs. They run a benchmark suite to create platform specific
headers, which is heavily influenced by the system load. So having
RESTRICT=parallel would make the emerge more reliable.

justin


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-dev] Re: [RFC] RESTRICT=parallel for builds that can't be executed in parallel

[Duncan]

Brian Harring posted on Tue, 13 Apr 2010 23:10:16 -0700 as excerpted:

> RESTRICT=parallel is basically a big lock that forces building to go
> down to one specific build/merge job- it's not at all fine grained. That
> said, I'm not convinced it's worth actually *trying* to be fine grained.
> 
> Stuff that needs this 'lock', if you look at it from the purely academic
> angle is broken.  The pkgs in question should be buildable without
> modifying the livefs.
> 
> From the pragmatic angle, fixing some of those packages is a pretty huge
> endeavour hence this lock existing.  I see no reason to encourage the
> usage of this lock by making it more fine grained, either.

What examples of the problem do we have, other than xorg-server due to 
eselect opengl?

For just xorg-server, killing parallel seems like a rather frustrating and 
indeed broken solution (especially for folks who prefer to run freedomware 
and thus have only the X11/mesa opengl version on their system anyway, so 
forcing non-parallel is an exercise in uselessness).  If it's the only 
one, at /least/ only forcing non-parallel if the eselect opengl actually 
changes the selected opengl would seem reasonable.

But if there's other non-contrived examples around, getting a couple of 
them on the table should I think clarify our potential usage constraints 
somewhat.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: [RFC] RESTRICT=parallel for builds that can't be executed in parallel

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 3581 bytes --]

On Wed, Apr 14, 2010 at 07:06:56AM +0000, Duncan wrote:
> Brian Harring posted on Tue, 13 Apr 2010 23:10:16 -0700 as excerpted:
> 
> > RESTRICT=parallel is basically a big lock that forces building to go
> > down to one specific build/merge job- it's not at all fine grained. That
> > said, I'm not convinced it's worth actually *trying* to be fine grained.
> > 
> > Stuff that needs this 'lock', if you look at it from the purely academic
> > angle is broken.  The pkgs in question should be buildable without
> > modifying the livefs.
> > 
> > From the pragmatic angle, fixing some of those packages is a pretty huge
> > endeavour hence this lock existing.  I see no reason to encourage the
> > usage of this lock by making it more fine grained, either.
> 
> What examples of the problem do we have, other than xorg-server due to 
> eselect opengl?

Honestly it's the only hard class of examples I've got- ati-drivers 
(presumably nvidia also although I've not checked) also would require 
this.  That said, 7 years of pkging crap, I know it's not going to be 
the *only* example.

It mostly comes down the perfect world view vs pragmatic- a lot of 
what we're doing in terms of ebuild pkging is a balance between best 
practice and having the thing usable.

Sometimes you've got to do something ugly to make it work- especially 
when the effort required to do it correctly is fairly large.  I'm not 
arguing against doing it correctly, I'm just arguing we should be 
realistic.


> For just xorg-server, killing parallel seems like a rather frustrating and 
> indeed broken solution (especially for folks who prefer to run freedomware 
> and thus have only the X11/mesa opengl version on their system anyway, so 
> forcing non-parallel is an exercise in uselessness).  If it's the only 
> one, at /least/ only forcing non-parallel if the eselect opengl actually 
> changes the selected opengl would seem reasonable.

The thing people need to keep in mind for stuff like this is that 
metadata is *constant*- it's quite a huge amount of work to write a 
framework that is able to ask "hey xorg, are you going to go screwing 
w/ something that means we can't build something in parallel to you 
building?"- it's not horrible, but it's a lot of effort for 
questionable gain.  If you consider users running ati-drivers, that 
check would state "yes, lock it" for building both ati-drivers and 
xorg-server; the only instance where it wouldn't trigger the lock is 
when the user is running *just* xorg-server.

Also, just to be clear, RESTRICT=parallel does not means that building 
xorg-server is going to be make -j1; it can still be make -jN, it just 
keeps the scheduler from building any other jobs in parallel while 
xorg-server is doing it's thing.


> But if there's other non-contrived examples around, getting a couple of 
> them on the table should I think clarify our potential usage constraints 
> somewhat.

As mentioned, I've got nothing else concrete- mostly because I've not 
had the time to look (I do know they're going to be there however).  
One thing to keep in mind also is that when situations of this sort 
pop up we need the functionality *before* it occurs.  Adding it after 
the fact is a PITA.

What I'd like to see w/ RESTRICT=parallel is that it basically is 
unused in the tree- I expect a few screwed up pkgs to need it however.  
It's a bit like RESTRICT=sandbox; it should only be used when there is 
an extremely good reason and the alternatives don't easily exist.
~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] [RFC] RESTRICT=parallel for builds that can't be executed in parallel

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1469 bytes --]

On Wed, 14 Apr 2010 08:46:15 +0200 Justin <jlec@gentoo.org> wrote:

> There is at least one other example which benefits from singlular
> build, atlas libs. They run a benchmark suite to create platform
> specific headers, which is heavily influenced by the system load. So
> having RESTRICT=parallel would make the emerge more reliable.

And that's another example of working around broken ideas. There is
an uncountable number of possibilities of many different factors
affecting the system load, and thus the results of the benchmark.
Removing one of them would indeed help in many cases but in many other
it wouldn't make any difference, additionally slowing down the whole
system update process.

Please notice that in order to implement that correctly, emerge would
have to wait until all previously running emerges are done, and avoid
running further ones while the build process is running. While
in the case of xorg-server, the 'lock' is needed throughout the whole
build process, in your case it is needed only for a short amount
of time when the benchmark is being performed --- and the 'real' part
of building would unnecessarily block emerge.

If this is ever to be implemented, it totally needs to be
user-overridable. Else, we'll end up someday like Windows, forcing user
to reboot the system and perform the merge on a dedicated runlevel.

-- 
Best regards,
Michał Górny

<http://mgorny.alt.pl>
<xmpp:mgorny@jabber.ru>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-dev] Re: [RFC] RESTRICT=parallel for builds that can't be executed in parallel

[Duncan]

Justin posted on Wed, 14 Apr 2010 08:46:15 +0200 as excerpted:

> On 14/04/10 04:12, Zac Medico wrote:
>> Hi everyone,
>> 
>> Should we add a RESTRICT=parallel value for ebuilds that can't be built
>> at the same time as other ebuilds? Brian says we need it for things
>> like xorg-server which calls eselect opengl.
> 
> There is at least one other example which benefits from singlular build,
> atlas libs. They run a benchmark suite to create platform specific
> headers, which is heavily influenced by the system load. So having
> RESTRICT=parallel would make the emerge more reliable.

sci-libs/blas-atlas (and perhaps lapack-atlas, etc, too), right?  
"Automatically tuned..."

Wow!  Yeah, that sounds like a reasonable example.  Sort of like the 
kernel does for md/RAID-5 and 6 at boot, I'd guess, choosing the fastest 
algorithm on the platform, only they're doing it during system runtime 
when who /knows/ what else is running!  Having a second highly 
parallelizable (MAKEOPTS version) build go from config to build and its 
load go from say .8 to 8. in the middle of those benchmarks /could/ screw 
things up "just a little!"

Thanks.  That's just the sort of additional practical example I was asking 
for to try to get my mind around this.  Excellent example as, unlike the 
various xorg/mesa/drivers thing, it's pretty hard to argue "just code 
around it", for this one.  The only technical way out of it here would 
seem to be to change the build-and-benchmark strategy itself, which would 
rather defeat the "automatically tuned" bit entirely.

BTW, gcc seems to do some stage output comparing in its bootstrap 
process.  Is that all absolute code correctness, or is there some 
performance benchmarking there that could benefit from this as well?

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Multiple emerges in parallel (was: [RFC] RESTRICT=parallel for builds that can't be executed in parallel)

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 3187 bytes --]

On Tue, 13 Apr 2010 23:10:16 -0700
Brian Harring <ferringb@gmail.com> wrote:

> Running multiple emerges in parallel is already a bad idea.  The 
> solution for that case is for the new/second emerge to feed the 
> request into the original emerge (or a daemon).

Although such solution will be useful in many cases indeed, there are
still many advantages of having few separate emerge calls running
in parallel.

First of all, the 'emerge daemon' would have to be able to
transparently inject the additional packages onto the dependency tree.
And just appending missing dependencies and the real package at the end
doesn't seem sufficient.

For example, what if such operation would result in a conflict? If
that's only that simple as 'A and B can not be installed at the same
time', the second emerge call could reject appending the packages,
explaining the reason for that.

But what if the conflict could be resolved through pulling in some
other version of one of the packages? We could try to replace
the package in the dependency tree silently if it didn't yet start
being merged; but what if it did? Should we reject adding
the packages, encouraging user to 'try again later'? Or maybe abort
the running builds to merge another version of the depend? Same goes
for virtual dependencies which can be fulfilled by many different
packages.

Of course, that's just implementation-wide issue and running separate
emerge instances could result in even worse results of this problem.
But apart from that, there are side uses of multiple emerges which
would be hard to implement with your idea.

For example, I often use different PORTAGE_TMPDIR to build large
packages like xulrunner. Although it doesn't seem that hard
implementing the support for running merges in different temporary
directories at the same time, should we really waste time implementing
that? What about other possible configuration changes? What about
'USE' overrides within the environment? These ones should certainly be
respected.

So, now our daemon not only has to be able to efficiently mangle
dependency trees at runtime but also to support partial runtime
configuration changes.

The next thing is aborting merges. When running multiple emerges,
aborting one of them is as simple as pressing ^c. With daemon, we would
have to implement an ability of aborting/removing packages in runtime
-- and that would be another example of dependency tree mangling.

But in fact, implementing (at least limited) the support of such
mangling would be useful in some cases. With '--keep-going', it would
allow to drop the packages and continue merging the remaining ones
without waiting for current builds to complete.

And looking from the other side, the support for removing single
packages from the running emerge dependency tree would be useful too
with current emerge concept. Even if it just worked like
'--keep-going', it would be still better than the current workaround --
waiting for particular emerge to start and removing its ${WORKDIR} to
force failure.

-- 
Best regards,
Michał Górny

<http://mgorny.alt.pl>
<xmpp:mgorny@jabber.ru>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Multiple emerges in parallel (was: [RFC] RESTRICT=parallel for builds that can't be executed in parallel)

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 1862 bytes --]

On Wed, Apr 14, 2010 at 04:20:18PM +0200, Michaaa GGGrny wrote:
> On Tue, 13 Apr 2010 23:10:16 -0700
> Brian Harring <ferringb@gmail.com> wrote:
> 
> > Running multiple emerges in parallel is already a bad idea.  The 
> > solution for that case is for the new/second emerge to feed the 
> > request into the original emerge (or a daemon).
> 
> Although such solution will be useful in many cases indeed, there are
> still many advantages of having few separate emerge calls running
> in parallel.

The examples you give are fine and dandy, but if done via parallel 
emerge you can run into situations where PM 1 just added pkg A as a 
dep for PKG B, and PM 2 is removing pkg A due to a blocker for pkg C.

Running multiple emerge's in parallel is unsafe due to the fact 
they've got two potentially very different plans as to what is being 
done, and that there is no possibility to ensure that pkg D that PM-2 
is building isn't affected by PM-1 building something (upgrading a 
dependency of pkg D for example).

Yes you can get away with it occasionally, that doesn't mean it's 
safe however.

> The next thing is aborting merges. When running multiple emerges,
> aborting one of them is as simple as pressing ^c. With daemon, we would
> have to implement an ability of aborting/removing packages in runtime
> -- and that would be another example of dependency tree mangling.

Aborting merges is a very, very bad idea.  Consider a pkg that has 
dlopen'd plugins, and just went through an ABI change for that 
interface.  If you interupt that merge it's entirely possible you'll 
get just the lib merged (meaning a segfault on loadup of the plugins), 
or vice versa (old lib is still in place, but new plugins are there).

Don't abort merges- that really should be effectively an atomic OP, 
not interuptible.

~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Multiple emerges in parallel (was: [RFC] RESTRICT=parallel for builds that can't be executed in parallel)

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1469 bytes --]

On Wed, 14 Apr 2010 11:10:29 -0700
Brian Harring <ferringb@gmail.com> wrote:

> Running multiple emerge's in parallel is unsafe due to the fact 
> they've got two potentially very different plans as to what is being 
> done, and that there is no possibility to ensure that pkg D that PM-2 
> is building isn't affected by PM-1 building something (upgrading a 
> dependency of pkg D for example).
> 
> Yes you can get away with it occasionally, that doesn't mean it's 
> safe however.

I agree with you -- such operations should be performed with
appropriate caution, on user's own responsibility. But this doesn't
mean we should prevent user from being able to do that.

> Aborting merges is a very, very bad idea.  Consider a pkg that has 
> dlopen'd plugins, and just went through an ABI change for that 
> interface.  If you interupt that merge it's entirely possible you'll 
> get just the lib merged (meaning a segfault on loadup of the
> plugins), or vice versa (old lib is still in place, but new plugins
> are there).
> 
> Don't abort merges- that really should be effectively an atomic OP, 
> not interuptible.

Sorry, by 'aborting merges' I meant rather aborting the build process
before the 'merge' phase -- or even aborting it before it is started
(i.e. before it unpacks a load of files into ${PORTAGE_TMPDIR}
without sufficient space).

-- 
Best regards,
Michał Górny

<http://mgorny.alt.pl>
<xmpp:mgorny@jabber.ru>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-dev] Re: Multiple emerges in parallel (was: [RFC] RESTRICT=parallel for builds that can't be executed in parallel)

[Duncan]

Brian Harring posted on Wed, 14 Apr 2010 11:10:29 -0700 as excerpted:

>> The next thing is aborting merges. When running multiple emerges,
>> aborting one of them is as simple as pressing ^c. With daemon, we would
>> have to implement an ability of aborting/removing packages in runtime
>> -- and that would be another example of dependency tree mangling.
> 
> Aborting merges is a very, very bad idea.  Consider a pkg that has
> dlopen'd plugins, and just went through an ABI change for that
> interface. If you interupt that merge it's entirely possible you'll get
> just the lib merged (meaning a segfault on loadup of the plugins), or
> vice versa (old lib is still in place, but new plugins are there).
> 
> Don't abort merges- that really should be effectively an atomic OP, not
> interuptible.

Umm... I think you two are using the same words for different things.

Definitely, aborting qmerge (transfer to the live filesystem) isn't a good 
idea, but in context, it's plain that MG's talking about the entire merge 
process, from setup, unpack and prepare thru qmerge and postinst (which is 
how the terms are used in the ebuild qmerge vs. ebuild merge context as 
well).  Clearly the entire merge doesn't need to be atomic, or we'd not be 
talking about parallel merges in the first place, nor would we have 
available all the individual ebuild <command> substeps.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

[gentoo-dev] Re: [RFC] RESTRICT=parallel for builds that can't be executed in parallel

[Ryan Hill]

[-- Attachment #1: Type: text/plain, Size: 553 bytes --]

On Wed, 14 Apr 2010 13:03:10 +0000 (UTC)
Duncan <1i5t5.duncan@cox.net> wrote:

> BTW, gcc seems to do some stage output comparing in its bootstrap 
> process.  Is that all absolute code correctness, or is there some 
> performance benchmarking there that could benefit from this as well?

It's all code correctness.


-- 
fonts,                                            by design, by neglect
gcc-porting,                              for a fact or just for effect
wxwidgets @ gentoo     EFFD 380E 047A 4B51 D2BD C64F 8AA8 8346 F9A4 0662

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] [RFC] RESTRICT=parallel for builds that can't be executed in parallel

[Luca Barbato]

On 04/14/2010 08:46 AM, Justin wrote:
> There is at least one other example which benefits from singlular build,
> atlas libs. They run a benchmark suite to create platform specific
> headers, which is heavily influenced by the system load. So having
> RESTRICT=parallel would make the emerge more reliable.

I wonder if that case shouldn't be handled better with an huge ewarn so
people concerned would really run it in a benchmark environment, alone.

lu

-- 

Luca Barbato
Gentoo/linux
http://dev.gentoo.org/~lu_zero

Re: [gentoo-dev] [RFC] RESTRICT=parallel for builds that can't be executed in parallel

[Luca Barbato]

On 04/14/2010 04:12 AM, Zac Medico wrote:
> Hi everyone,
> 
> Should we add a RESTRICT=parallel value for ebuilds that can't be
> built at the same time as other ebuilds? Brian says we need it for
> things like xorg-server which calls eselect opengl.

Maybe we could live w/out calling eselect opengl that way first.

The discussion about how to handle two parallel merge in a safer way
seems interesting though.

lu

-- 

Luca Barbato
Gentoo/linux
http://dev.gentoo.org/~lu_zero

Re: [gentoo-dev] [RFC] RESTRICT=parallel for builds that can't be executed in parallel

[Enrico Weigelt]

* Luca Barbato <lu_zero@gentoo.org> schrieb:

> I wonder if that case shouldn't be handled better with an huge ewarn so
> people concerned would really run it in a benchmark environment, alone.

ewarns should be circumvented (make it work w/o them), IMHO.

I can imagine I'm not the only person who doenst want to keep
an close eye on each single build/merge.

In this concrete example, the package's build system is misdesigned.
They shouldn't rely on runtime benchmarks for build time decisions
in the first place - this is unpredictable. But this is not an
issues distros should have to cope with.


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------