[gentoo-dev] Re: Any official position from Gentoo about systemd, mdev and udev-static ?

[gentoo-dev] Re: Any official position from Gentoo about systemd, mdev and udev-static ?

[Duncan]

Ben de Groot posted on Wed, 29 Aug 2012 13:06:29 +0800 as excerpted:

> For now udev is still usable without systemd, even tho upstream is
> making it difficult to build udev separately (and avoid unnecessary
> build-time dependencies). Upstream is also unwilling to work with us to
> make this easier.
> 
> Despite upstream promises that udev will remain usable stand-alone,
> there is a lot of skepticism towards that. So we may find ourselves
> faced with the need to use a fork or a replacement for udev.
> 
> Right now the situation is in flux, so we are basically waiting to see
> how things will develop.

Yours and Luca's summaries LGTM.  Two comments/observations and one 
question, tho.  Question first, at the top:

Question:

For those who have actually built it, what *IS* the full systemd relative 
merge-time, and how does that compare to that of pre-merge udev?

I doubt that most people consider udev's stand-alone build-time a big 
issue.  It's not like a gcc update, or firefox, or the classic example, 
OOo/LO.  Building/installing udev has always been relatively trivial.

Just how much worse are we talking if we DO end up having to build nearly 
all of systemd, only to throw out most of it, just keeping udev?  If it's 
just 50% more than udev by itself, or even double, while having to build 
all of systemd just to throw out most of it is irksum in principle, in 
practice, for most, it wouldn't be a big deal.  (Yes, there's the 
underpowered minor archs where it would be, but even there, 
relatively...)  Ten times the build/install time, however, and it's 
rather more of an issue.  20 or 50 times, and it's a much *BIGGGER* issue.

So in practice, just what are the sorts of times, relative to stand-alone-
build udev, we're talking about?  In all this discussion, what, hundreds 
of posts by now?, I've not seen ANYONE actually ask, let alone answer, 
THAT.  But it would seem to be a rather important question...

Comments/observations (mostly rehash for those who read the other thread, 
feel free to skip to the next post):  

1) On gentoo, "usable standalone" by definition means reasonably 
buildable, standalone, ideally without going to extreme lengths via 
complex and exotic ebuild/eclasses to do it.  That's rather different 
than upstream's definition of "usable standalone", which they did 
promise, but which to them simply means the already built binary (as 
found in systemd-based distros for use in their initr*s before switching 
to real-root, where systemd is found, that being the specific example 
they gave) will continue to function on its own -- they have in fact 
specifically stated that they have absolutely NO interest in keeping it /
buildable/ stand-alone, or in cooperation with distros like gentoo who 
find that IS in their interest.

Given that difference in definition for "usable standalone", it's with 
good reason that gentooers generally view that upstream claim with 
extreme skepticism.

2) But my primary point in an earlier thread on the topic still stands.  
Gentoo's current udev/openrc default simply can't and won't be changed on 
a dime.  Even if gentoo chose to do it today and the project was given 
extreme priority, we're looking at at LEAST a year, more like two.  And 
by 2-3 years out, if Linux/FLOSS history is any guide, the whole 
ecosystem will look different, and we'll have a whole list of new changes 
and challenges to worry about, so somewhere between 3-5 years out, the 
picture simply gets way too fuzzy to predict any more, and throwing the 
dice has about as good a chance.  So with no plans to immediately change 
gentoo's position, it's a pretty safe bet that any changes of that size 
that WOULD occur are safely out beyond that three-year horizon where 
things start getting fuzzy, and beyond that, it's anyone's guess.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: Any official position from Gentoo about systemd, mdev and udev-static ?

[Rich Freeman]

On Wed, Aug 29, 2012 at 6:35 AM, Duncan <1i5t5.duncan@cox.net> wrote:
> And
> by 2-3 years out, if Linux/FLOSS history is any guide, the whole
> ecosystem will look different, and we'll have a whole list of new changes
> and challenges to worry about,

Agreed.  I suspect the status quo will remain in place for a while.
If somebody forks udev and a bunch of distros switch to the fork, that
could shake things up, although that could get really messy if the two
projects start to diverge since we'd need compatibility layers for
packages that use udev since either might be installed (only to the
extent that they diverge).

What I see as the most likely thing to lead to change is if/when
GnomeOS actually starts to exist.  When you can't run Gnome without
systemd I'd expect to see a lot more Gentoo users running it.  Then
again, if Gnome jumps the shark, maybe not.

In any case, seems best to just take the hurdles as they come.

Rich

Re: [gentoo-dev] Re: Any official position from Gentoo about systemd, mdev and udev-static ?

[Tobias Klausmann]

Hi! 

On Wed, 29 Aug 2012, Duncan wrote:
> So in practice, just what are the sorts of times, relative to stand-alone-
> build udev, we're talking about?  In all this discussion, what, hundreds 
> of posts by now?, I've not seen ANYONE actually ask, let alone answer, 
> THAT.  But it would seem to be a rather important question...

As a first crude datapoint, I compared the build times
(configure+make) of udev-171-r6 and -188 on our dev Alpha. This
is a machine that's on the speedier side of off-mainstream
architecures, but as a datapoint, it should be enough.

Tests were run just after ebuild [foo] prepare, i.e. patching was
_not_ measured. Since the machine has enough RAM, everything was
in the page cache.

ver          (1)    (2)   (1+2) 
udev-171-r6: 15s /  71s /  86s;  1m26s
udev-188   : 28s / 636s / 664s; 11m04s

1: "./configure" time
2: "make" time

Other observations:
 - The machine in question did not have dbus, or libcap, which
   some would argue need to be factored into "build cost".
 - I expected more difference for the configure phase, since it
   is what I usually perceive as the slow part of many builds
   when archtesting. Probably some packages suffer more from this
   than others. Also, configure does not run in parallel, make
   does.
 - Test suite run times were not checked. Though it looks like
   the -188 ebuild builds the test suite binaries regardless.
 - This was run as make -j1 even though the machine has 4 CPUs.
   One reason was to make the compile time longer for better
   measurement granularity, the other was that most slower
   machines (as far as we are concerned) are single-cpu.

HTH,
Tobias

[gentoo-dev] Re: Any official position from Gentoo about systemd, mdev and udev-static ?

[Vaeth]

> I doubt that most people consider udev's stand-alone build-time a big
> issue.

The real issue is not the build-time but the dependencies needed
at build-time (and in future versions perhaps also at run-time):
Currently, these are essentially libcap and dbus.

Now that some projects (e.g. hardened and overlayfs) plan to use
extended attributes systematically, there is probably no other
way for most users than to enable them on for file systems.
In this case, it is a security issue (KISS principle) to have
libcap installed.

dbus alone is not a security issue yet if you do not have any *kit
listening to it, but judging by the previous behavior of upstream,
I would not be surprised if in the not-so-far future they will
invent a reason to make *kit or even essential gnome libraries a
build-time (or even run-time) dependency - of course, making
this dependency non-optional like they do now with libcap and dbus.

Currently, Gentoo is the only distribution which can provide
the user a possibility to run a decent system without bloat
by means of useflags.
If upstream wants to undermine this freedom and if there is
a possibility to give the choice back to the user by an udev virtual
supporting the fork, I think that Gentoo should take this route:

Gentoo now can show its real strength, because it is one of the
few distributions which have technically the ability to leave
the choice between gnome os and traditional linux to the user.

If later on the divergence between the udev implementations is
too high and nobody works on a compatibility layer, some
dependencies on the non-virtual udev might be introduced.

This is not worse than the situation which we have currently where
also several packages cannot be used if e.g. *kit should be
avoided.

Re: [gentoo-dev] Re: Any official position from Gentoo about systemd, mdev and udev-static ?

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 450 bytes --]

On Wed, Aug 29, 2012 at 01:57:48PM +0200, Tobias Klausmann wrote:
> As a first crude datapoint, I compared the build times
> (configure+make) of udev-171-r6 and -188 on our dev Alpha. This
> is a machine that's on the speedier side of off-mainstream
> architecures, but as a datapoint, it should be enough.
 
 No, not exactly, because udev-188 doesn't build systemd. We use
 specific targets in the Makefile to avoid doing that.

William


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Re: Any official position from Gentoo about systemd, mdev and udev-static ?

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 443 bytes --]

On Wed, Aug 29, 2012 at 05:12:19PM +0200, Vaeth wrote:
> > I doubt that most people consider udev's stand-alone build-time a big
> > issue.
> 
> The real issue is not the build-time but the dependencies needed
> at build-time (and in future versions perhaps also at run-time):
> Currently, these are essentially libcap and dbus.

Because of the way the udev-189 ebuild is written, it doesn't require
those dependencies.

William


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Re: Any official position from Gentoo about systemd, mdev and udev-static ?

[Tobias Klausmann]

Hi! 

On Wed, 29 Aug 2012, William Hubbs wrote:
> On Wed, Aug 29, 2012 at 01:57:48PM +0200, Tobias Klausmann wrote:
> > As a first crude datapoint, I compared the build times
> > (configure+make) of udev-171-r6 and -188 on our dev Alpha. This
> > is a machine that's on the speedier side of off-mainstream
> > architecures, but as a datapoint, it should be enough.
>  
>  No, not exactly, because udev-188 doesn't build systemd. We use
>  specific targets in the Makefile to avoid doing that.

Amending my earlier post then, here are the numbers for the
171 build and the target-specific build of 188:

ver          (1)    (2)   (1+2)
udev-171-r6: 15s /  71s /  86s;  1m26s
udev-188   : 28s /  78s / 116s;  1m56s

Much less difference. Still, I figure _really_ slow
machines/arches might be in more pain. I'll run my test with a
Geode-based x86 later this week.

Regards,
Tobias

-- 
Sent from aboard the Culture ship
	GCU You'll Thank Me Later

Re: [gentoo-dev] Re: Any official position from Gentoo about systemd, mdev and udev-static ?

[Walter Dnes]

On Wed, Aug 29, 2012 at 07:37:49AM -0400, Rich Freeman wrote

> What I see as the most likely thing to lead to change is if/when
> GnomeOS actually starts to exist.  When you can't run Gnome without
> systemd I'd expect to see a lot more Gentoo users running it.  Then
> again, if Gnome jumps the shark, maybe not.

  Even GNOME 2 seems to depend on udev nowadays, specifically the gvfs
and evdev code.

  And Google's Chromium pulls in udev... and dbus, and elf-utils, and
libusb!!!  This is not the Chrome-OS on their "Chrome-books", but the
linux web-browser build.  We all know how well the browser-as-an-OS idea
worked for AOL/Netscape... !NOT.

  Note that a fork will have to be be "bug-compatable" to Redhat's
version, just like DR-DOS had to be bug-compatable to MS-DOS, way back
when.  And what happens when that "compatability" requires not just
systemd and dbus but pulseaudio and binary syslogs and whatever else the
Redhat developers decree?

-- 
Walter Dnes <waltdnes@waltdnes.org>
I don't run "desktop environments"; I run useful applications

Re: [gentoo-dev] Re: Any official position from Gentoo about systemd, mdev and udev-static ?

[Mart Raudsepp]

On N, 1970-01-01 at 00:00 +0000, Tobias Klausmann wrote:
> Hi! 
> 
> On Wed, 29 Aug 2012, William Hubbs wrote:
> > On Wed, Aug 29, 2012 at 01:57:48PM +0200, Tobias Klausmann wrote:
> > > As a first crude datapoint, I compared the build times
> > > (configure+make) of udev-171-r6 and -188 on our dev Alpha. This
> > > is a machine that's on the speedier side of off-mainstream
> > > architecures, but as a datapoint, it should be enough.
> >  
> >  No, not exactly, because udev-188 doesn't build systemd. We use
> >  specific targets in the Makefile to avoid doing that.
> 
> Amending my earlier post then, here are the numbers for the
> 171 build and the target-specific build of 188:
> 
> ver          (1)    (2)   (1+2)
> udev-171-r6: 15s /  71s /  86s;  1m26s
> udev-188   : 28s /  78s / 116s;  1m56s
> 
> Much less difference. Still, I figure _really_ slow
> machines/arches might be in more pain. I'll run my test with a
> Geode-based x86 later this week.

Geode LX700 (433MHz) with 256MB RAM
MAKEOPTS=-j2 (single core system)
gcc (Gentoo 4.5.2 p1.1, pie-0.4.5) 4.5.2

ebuild prepare done before as well.

1. time ebuild foo configure — real time value
2. time ebuild foo compile — real time value

ver           (1)      (2)    (1+2)
udev-171-r6: 47.2s / 4m36.4 / 5m23.6
udev-188   : 69.6s / 6m27.2 / 7m36.8

Personally of course don't care the slightest of this time.


Mart

[gentoo-dev] Re: Any official position from Gentoo about systemd, mdev and udev-static ?

[Duncan]

Walter Dnes posted on Wed, 29 Aug 2012 21:19:13 -0400 as excerpted:

>  Note that a fork will have to be be "bug-compatable" to Redhat's
> version, just like DR-DOS had to be bug-compatable to MS-DOS, way back
> when.  And what happens when that "compatability" requires not just
> systemd and dbus but pulseaudio and binary syslogs and whatever else the
> Redhat developers decree?

FWIW... in my web wanderings I came across a discussion (on G+ I think) 
where, I believe it was Ted T'so mentioned talking to one of the RHEL 
product engineers... and they're not entirely happy with all the desktop 
stuff, binary logs, etc the combined udev/systemd is pulling in and doing 
these days, either!  After all, they're supporting a product for 
something like 10 years, that is often used in an enterprise server 
environment where the full desktop may not actually be installed -- or at 
least that's the way it *WAS*.  They could ignore pulse-audio in that 
regard, because such servers often didn't have/need sound anyway.  But 
they're not too happy about this whole dragging in the whole desktop 
thing, NOR about potentially having to support "the latest hot fad" 
someone dreamed up (even if that someone's actually a RH employee) for a 
decade!

Remember hal?  They're still dealing with that!

Anyway, it's generally accounted to Red Hat's credit that they don't 
interfere too much with the development policies of the projects they 
sponsor people to work on, but that doesn't necessarily mean even Red 
Hat's particularly happy with said policies!

Based on that discussion, I'd say that while Red Hat will hopefully 
continue its in general non-interference with projects it sponsors 
policies, there's at least SOME possibility that they'll either work 
around the problem in their coming releases with patches then available 
to the community, or interfere albeit in an arguably "least harm" 
fashion, by eventually mandating at least branches (from which they'd 
pull), that at least make some of these sorts of dependencies optional.

Or maybe they'll actually sponsor a fork too, if they have to.  But I 
doubt it'll come to that.

Regardless, coming RHEL releases may not end up being as drastically 
integrated in this regard as the current trend, which is after all now at 
the Fedora level not RHEL level, would indicate.  It DOES remain to be 
seen, but that thread gave me at least SOME hope that the ENTIRE
(Gnu/)Linux world (other than Gentoo and Debian, maybe) hasn't gone mad, 
as well as an explicit awareness that Red Hat is now a large enough 
company (just hitting a billion a year, right?) that like many large 
companies, the right hand and the left hand, separated into their own 
departments, may be working toward not entirely compatible ends.

It'll be interesting to see how it all plays out, when the big dollars 
feeding red hat come in conflict with some of the policies of some of 
their sponsored projects and employees, corporate hands-off policy or no 
corporate hands-off policy.

But one thing's for sure, there's money there, and where there's that 
sort of money involved, one way or another, the code usually "appears".  
So all is not YET lost, regarding "insane" dependencies at the base udev 
level.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

[gentoo-dev] Re: Any official position from Gentoo about systemd, mdev and udev-static ?

[Duncan]

Mart Raudsepp posted on Thu, 30 Aug 2012 07:27:48 +0300 as excerpted:

> Geode LX700 (433MHz) with 256MB RAM MAKEOPTS=-j2 (single core system)
> gcc (Gentoo 4.5.2 p1.1, pie-0.4.5) 4.5.2
> 
> ebuild prepare done before as well.
> 
> 1. time ebuild foo configure — real time value 
> 2. time ebuild foo compile — real time value
> 
> ver           (1)      (2)    (1+2)
> udev-171-r6: 47.2s / 4m36.4 / 5m23.6
> udev-188   : 69.6s / 6m27.2 / 7m36.8


Thanks.  Those are the data-points of an older udev and a current "semi-
integrated" udev that gentoo has "deintegrated" to a point.

Now, for worst-case comparison, on the same machine, what's the 
respective times for a full systemd build?  (I'm not saying actually 
merge it, just configure/compile, plus see the next paragraph.)

Also, note that ebuild foo install only does the install to the fake 
location in PORTAGE_TMPDIR.  It's the ebuild qmerge step that actually 
merges to the live filesystem.  So the install phase should be safe to 
try without actually merging as well, just don't qmerge.  And on some 
packages the install phase actually does enough additional work to be 
worth timing as well.  I don't know if that's the case here or not, but 
it's probably worth testing, just in case.

So far, the results are encouraging.  Higher times, but not THAT much 
higher, with the new version.  But if the worst happens and we really DO 
end up building all of systemd and then throwing most of it away, what's 
the relative times for THAT?  That's actually what I was asking earlier, 
and this gets us part way to the answers, but not all the way.

Regardless, thanks, this is considerably more solid information than was 
available earlier, and not everyone has a geode to actually do the test 
on, so these numbers are a real contribution to the available 
information, indeed! =:^)

(Now I'm wondering just how low I could clock this bulldozer, and whether 
setting uniprocessor kernel command-line-option and downclocking as far 
as possible, plus memory-limiting to say 256MB for my own tests is worth 
the trouble.  FWIW I also have an original atom netbook that's slow 
enough to be interesting, but I do my builds for it in a 32-bit chroot on 
the main machine then rsync them over, and I've not actually updated the 
netbook in over a year, so I'd have to do a seriously major update, then 
figure out how to make the portage tree available on the netbook, to test 
that, and that's DEFINITELY more work than I'm likely to put into such a 
test, near-term anyway, so deliberately crippling the bulldozer is about 
the best I'll get.  But that'd make for an interesting comparison against 
normal performance on this hardware, as well, so I just might do that one 
of these days if I get the urge to do some fiddling.)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: Any official position from Gentoo about systemd, mdev and udev-static ?

[Tobias Klausmann]

Hi! 

On Thu, 30 Aug 2012, Duncan wrote:
> Now, for worst-case comparison, on the same machine, what's the 
> respective times for a full systemd build?  (I'm not saying actually 
> merge it, just configure/compile, plus see the next paragraph.)

I think my first set of numbers illustrates that: just running
"make" should be a full build, AIUI. For that scenario (and the
machine in question, the factor was somewhere between 9 and 10
times slower for a full build.

> So far, the results are encouraging.  Higher times, but not THAT much 
> higher, with the new version.  But if the worst happens and we really DO 
> end up building all of systemd and then throwing most of it away, what's 
> the relative times for THAT?  That's actually what I was asking earlier, 
> and this gets us part way to the answers, but not all the way.
> 
> Regardless, thanks, this is considerably more solid information than was 
> available earlier, and not everyone has a geode to actually do the test 
> on, so these numbers are a real contribution to the available 
> information, indeed! =:^)

I'll try the whole thing again sometime later this week (in my
copious free time!).

Regards,
Tobias

[gentoo-dev] Re: Any official position from Gentoo about systemd, mdev and udev-static ?

[Duncan]

Tobias Klausmann posted on Thu, 30 Aug 2012 09:03:59 +0200 as excerpted:

> On Thu, 30 Aug 2012, Duncan wrote:
>> Now, for worst-case comparison, on the same machine, what's the
>> respective times for a full systemd build?  (I'm not saying actually
>> merge it, just configure/compile, plus see the next paragraph.)
> 
> I think my first set of numbers illustrates that: just running "make"
> should be a full build, AIUI. For that scenario (and the machine in
> question, the factor was somewhere between 9 and 10 times slower for a
> full build.

You're correct.  I was misinterpreting those first numbers as running 
what the ebuild ran, but manually, not as a full "make".  So naturally I 
came to the wrong conclusion about them and figured the full systemd 
build would be even worse!

Thanks for patiently pointing out (again) what they /actually/ covered, 
thereby setting me straight.  I will try to read a bit more carefully, 
next time. =:^\

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman