From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) by finch.gentoo.org (Postfix) with ESMTP id 4CBA21381F3 for ; Tue, 2 Jul 2013 21:48:34 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 58074E0A5F; Tue, 2 Jul 2013 21:48:29 +0000 (UTC) Received: from mail-wg0-f52.google.com (mail-wg0-f52.google.com [74.125.82.52]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 01BBEE09F7 for ; Tue, 2 Jul 2013 21:48:27 +0000 (UTC) Received: by mail-wg0-f52.google.com with SMTP id b12so5223387wgh.7 for ; Tue, 02 Jul 2013 14:48:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=0EcaVvSzUE6eyvX1p/BxzexhTtNY5U/kXhtR/+f20Jg=; b=LRHnIeaSZiI9+QhVzMGH44svAc+MTT1F2ff11m8v0RACULjL8lZadmC0P2emwuP58k lIoE1TD/zMwFnY0yFcQa8XIqnriMBddpHUh7rhie1o+rsbi6R+ED7++8g0vmUIqZ88Qz oRlGTvp3IuKxc9BzuwKMUDJNDvaKw8JIqgzwdyB/4//vFMFOzYgcv9mI5xGU26IQ3liL w5b7Mxqh0FK1CamEOt5wMgkEitFoXylCI7DDM9j5UDYcmBu3t/JEv/AO0F0Q2cYi0oik plsIGVJj/xTG10EM6hcM+64icNSCCSDMD1H2sSEhec2jgcaFkPfYaTiUeDshKNCSUx7K c3BA== Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org MIME-Version: 1.0 X-Received: by 10.194.8.163 with SMTP id s3mr25023210wja.41.1372801706458; Tue, 02 Jul 2013 14:48:26 -0700 (PDT) Received: by 10.180.165.197 with HTTP; Tue, 2 Jul 2013 14:48:25 -0700 (PDT) In-Reply-To: <20130701164149.131490f8@TOMWIJ-GENTOO> References: <20130701164149.131490f8@TOMWIJ-GENTOO> Date: Tue, 2 Jul 2013 23:48:25 +0200 Message-ID: Subject: Re: [gentoo-dev] Proper distribution integration of kernel *-sources, patches and configuration. From: =?UTF-8?B?VG9tw6HFoSBQcnXFvmluYQ==?= To: gentoo-dev@lists.gentoo.org Content-Type: multipart/alternative; boundary=047d7b5d253e0f82bc04e08e4e74 X-Archives-Salt: 259956fc-b7b0-4610-aa0a-2ad4dea8033a X-Archives-Hash: d7f0290fb8ae76f0a11ca22814a196cf --047d7b5d253e0f82bc04e08e4e74 Content-Type: text/plain; charset=UTF-8 This reminds me of: http://imgs.xkcd.com/comics/standards.png It sure sounds nicely, however I would not want to be the guy who maintains the whole mess of (often) incompatible patchsets. Given the fact that some patches lag 1-3 stable versions behind Linuses tree (grsec used by hardened for example), this might be better idea on paper than in reality. If you feel up for this, go for it. Be ready to meet resistance though. // I personally maintain my own tree, based on stable from gregkh and few patches On Mon, Jul 1, 2013 at 4:41 PM, Tom Wijsman wrote: > Hello > > > Please reply to gentoo-dev in case ML daemon changes Reply-To. > > > ### TL; DR ### > > By introducing feature patches which menu options are disabled by > default to genpatches, we can deduplicate *-sources maintainers as well > as large groups of users work. By introducing a distribution section > in the menuconfig, we can provide options that enable minimal Gentoo > requirements by default (DEVTMPFS, making Gentoo systems bootable since > an udev release a long time ago) and other distribution stuff. > > > ### Proper distribution integration of kernel *-sources, patches ... ### > > Gentoo is a distribution; but what is a distribution that doesn't > properly integrate what it provides, but instead expects its users to > do so, needlessly duplicating work amongst its maintainers and users. > > Let's say I want genpatches, aufs and TuxOnIce; closest candidates: > > - sys-kernel/aufs-sources: genpatches, aufs > - sys-kernel/pf-sources: genpatches, CK, BFQ, BFQ, TuxOnIce, UKSM > - sys-kernel/tuxonice-sources: genpatches, TuxOnIce > > What do I do? Take one (eg. aufs) and apply the other (eg. TuxOnIce)? > What if I want to add another common patchset to it? Hardened perhaps? > > You can see, some of these sources (like pf-sources) already attempt to > do so; with pf-sources in mind, why do we even have ck-sources, > tuxonice-sources in the Portage tree? Just to duplicate work? > > I think we should trim down on the amount of *-sources and combine > multiple patches into genpatches. Take an example of geek-sources > which does all this without a problem; I don't really like the approach > with USE flags used there, as the menuconfig can just cover that. > > https://github.com/init6/init_6/wiki/geek-sources > > What does a patch introducing new features really do? Or rather, what > should it do when we add it? Let me summarize: > > 1) The features should be disabled by default. > 2) These feature should depend on a non-vanilla / experimental option. > 3) The patch should not affect the build by default. > 4) The user can optionally enable the feature. > > So, in genpatches, since 3.9.7, BFQ was added to try this out (except > 2.). Ensured it to be disabled by default, did not affect the build for > anyone that does not enable it and the users have been enabling and > using it on their own. So, does it differentiate more from vanilla? No. > > This helps users as well as maintainers to not have to apply BFQ on > their own, they simply have to tick a config option and they are set. > If all patches that introduce new features are added in this fashion, > it spares large groups of users from having to do this on their own; we > can also deduplicate the work in the Portage tree this way. > > > ### ... and configuration. ### > > This problem is not only visible for patches, but also in the config. > > Meet CONFIG_DEVTMPFS; forget to enable it, greet a failing boot. We're > telling users to enable it in some places, in the handbook it's a single > line you must read, on the Wiki it's kind of missing unless you are > luckily on the right page, on the Quick Install book it is missing too. > > So, we are currently providing a configuration that expects anyone to > enable CONFIG_DEVTMPFS. But you have to remember that it need to make > sure you read about it or enable it from the udev upgrade a while ago > if you decide to start from a fresh config or are installing without > that handbook you kind of have memorized. > > Searching for CONFIG_DEVTMPFS in the forums and #gentoo logs shows that > this is quite often suggested as a fix and quite often actually fixes > an user's boot. Why duplicate telling users to do that if we can do it? > > There are a small set of other variables in this nature, mostly *TMPFS*. > > This is why I think it would be handy to add a Gentoo section to the > kernel, along the lines as described by Linus. > > https://lkml.org/lkml/2012/7/13/369 > > The same goes for asking the user to enable configuration options in the > kernel, why can't we just tell him to enable all option that toggles > support for the user. For example; in the Gentoo section, there would be > a "Init System Support" under which you can toggle an option to support > the minimal requirements for some init system. > > Feedback is very welcome. > > P.S.: Not everything in this mail has been acked by the kernel lead; > only some thoughts, I was suggested to take this to ML for discussion. > The usage of the word 'we' in this mail is therefore hypothetical. > > > ### F.A.Q. ### > > Q: I don't want feature X? Please don't add the patch! > > A: It's optional, don't enable it in your menu config. > > Q: What about my stable server? I really don't want to run this stuff! > > A: These options would depend on !CONFIG_VANILLA or CONFIG_EXPERIMENTAL > which would be disabled by default, therefore if you keep this option > the way it is on your stable server; it won't affect you. > > In other words, genpatches stay as stable as before; unless you > explicitly toggle options that intentionally make it unstable. > > Q: Genpatches used to be minimal, would it gain weight? > > A: If you don't enable those options, it would be as minimal as before; > this gives the user the choice between minimal and fat. > > Q: What about patches that fail to build / run? > > A: For testing kernels (~), we would do our best to make them work but > there could be occasions where we will have to cut them; this is no > different from how the kernel herd has been handling this before, we > have already dropped fbcondecor in the past when it was broken and > the current new branch 3.10 does not support deblob yet. > > For stable kernels, which are near the EOL of a branch; if the > feature isn't still there yet then it means that its author is > simply no longer working on it, this is no different from when the > patch would be in another *-sources or manually applied by the user. > > If you are unaware of how upstream releases work, just note that > the major patch between 3.9.8 and 3.10.0 introduces a lot more > changes are applied than the minor patch between 3.9 and 3.9.8; > therefore, we prefer to stabilize 3.9.8 or a later 3.9 kernel than > stabilizing 3.10.0. You can see this pattern in history; the > previous three have been 3.6.11, 3.7.10 and 3.8.13. > > Q: Can't you do something about those build and runtime failures? > I like to run testing, but I don't want breaks half of the time. > > A: To cover this, live ebuilds help, 3.10.9999 and 3.9999 and 9999; > the earlier ones tracking the stable queues, whereas the last one > tracking the upstream rc kernel. This way we can notice what fails > in advance and make it working by the time it reaches testing (~). > > That being said, it is to be expected that things are not always > fine very early in a branch; it's why those kernels are never stable. > > Q: What about kernel bugs, how would you know the user enabled them? > > A: We expect the .config to be provided, which we can run a sanity > check on; it's much more handy if we are aware of these patches than > that we have to figure out the user applies it on their own. > > Q: Are documentation / infrastructure / eclass changes necessary? > > A: The kernel project documentation has to be updated to reflect this; > no infrastructure or eclass changes are needed, since this is all > done through patches in genpatches. (Sub directories supported afaik) > > Q: Does this affect kernel stabilization or QA? > > A: No, experimental features (which would include these optional > features) have historically never been looked at by the arch teams; > if you can remember, there used to be an CONFIG_EXPERIMENTAL option > in the kernel to cover this. For example, you can find yourself > running some pretty unstable code if you use 3.8.13 and enable > Nouveau reclocking and power management; there are some options in > the menuconfig that follow this nature. > > As there would be less *-sources as a consequence; less has to be > taken into consideration for stabilization, this is why most > *-sources are currently unstable, because we don't have the > resources to be stabilizing them all. > > For QA, I don't really see a problem. > > Q: So, would this make vanilla-sources the new gentoo-sources? > > A: No, vanilla-sources undergoes some changes to more closely reflect > the upstream kernels; as a consequence, it would no longer have > stable kernels and older versions will drop more often. You can read > this request and the resulting discussion at the following link. > > http://thread.gmane.org/gmane.linux.gentoo.kernel/697 > > The kernel lead summarized a part of this. > > http://thread.gmane.org/gmane.linux.gentoo.kernel/697/focus=730 > > Q: Is this a one man army comedy show? > > A: One man army comedy shows work, see geek-sources; if you want to > look at some examples of what a one man army can do, the following > link is a good read to go through. > > > http://www.bennorthrop.com/Essays/2013/pair-programming-my-personal-nightmare.php > > But joking aside; no, we are not. > > We are at least with two on the kernel herd and a third developer is > likely to join in the near future; on top of that, other people are > welcome and I think it would be nice to see maintainers from > *-sources jump in to help along. genpatches isn't hard to maintain. > > Since we're no longer a one man army, we can do more. > > -- > With kind regards, > > Tom Wijsman (TomWij) > Gentoo Developer > > E-mail address : TomWij@gentoo.org > GPG Public Key : 6D34E57D > GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D > --047d7b5d253e0f82bc04e08e4e74 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable This reminds me of: h= ttp://imgs.xkcd.com/comics/standards.png
It sure sounds nicely, howe= ver I would not want to be the guy who maintains the whole mess of (often) = incompatible patchsets.
Given the fact that some patches lag 1-3 stable versions behind Linuses tre= e (grsec used by hardened for example),
this might be better idea on pa= per than in reality.

If you feel up for this, go for it. Be ready to= meet resistance though.

// I personally maintain my own tree, based on stable from gregkh and f= ew patches

On Mon, Jul 1, 2013 at 4:41 PM= , Tom Wijsman <TomWij@gentoo.org> wrote:
Hello


Please reply to gentoo-dev in case ML daemon changes Reply-To.


### TL; DR ###

By introducing feature patches which menu options are disabled by
default to genpatches, we can deduplicate *-sources maintainers as well
as large groups of users work. By introducing a distribution section
in the menuconfig, we can provide options that enable minimal Gentoo
requirements by default (DEVTMPFS, making Gentoo systems bootable since
an udev release a long time ago) and other distribution stuff.


### Proper distribution integration of kernel *-sources, patches ... ###
Gentoo is a distribution; but what is a distribution that doesn't
properly integrate what it provides, but instead expects its users to
do so, needlessly duplicating work amongst its maintainers and users.

Let's say I want genpatches, aufs and TuxOnIce; closest candidates:

=C2=A0- sys-kernel/aufs-sources: genpatches, aufs
=C2=A0- sys-kernel/pf-sources: genpatches, CK, BFQ, BFQ, TuxOnIce, UKSM
=C2=A0- sys-kernel/tuxonice-sources: genpatches, TuxOnIce

What do I do? Take one (eg. aufs) and apply the other (eg. TuxOnIce)?
What if I want to add another common patchset to it? Hardened perhaps?

You can see, some of these sources (like pf-sources) already attempt to
do so; with pf-sources in mind, why do we even have ck-sources,
tuxonice-sources in the Portage tree? Just to duplicate work?

I think we should trim down on the amount of *-sources and combine
multiple patches into genpatches. Take an example of geek-sources
which does all this without a problem; I don't really like the approach=
with USE flags used there, as the menuconfig can just cover that.

https://github.com/init6/init_6/wiki/geek-sources

What does a patch introducing new features really do? Or rather, what
should it do when we add it? Let me summarize:

=C2=A01) The features should be disabled by default.
=C2=A02) These feature should depend on a non-vanilla / experimental option= .
=C2=A03) The patch should not affect the build by default.
=C2=A04) The user can optionally enable the feature.

So, in genpatches, since 3.9.7, BFQ was added to try this out (except
2.). Ensured it to be disabled by default, did not affect the build for
anyone that does not enable it and the users have been enabling and
using it on their own. So, does it differentiate more from vanilla? No.

This helps users as well as maintainers to not have to apply BFQ on
their own, they simply have to tick a config option and they are set.
If all patches that introduce new features are added in this fashion,
it spares large groups of users from having to do this on their own; we
can also deduplicate the work in the Portage tree this way.


### ... and configuration. ###

This problem is not only visible for patches, but also in the config.

Meet CONFIG_DEVTMPFS; forget to enable it, greet a failing boot. We're<= br> telling users to enable it in some places, in the handbook it's a singl= e
line you must read, on the Wiki it's kind of missing unless you are
luckily on the right page, on the Quick Install book it is missing too.

So, we are currently providing a configuration that expects anyone to
enable CONFIG_DEVTMPFS. But you have to remember that it need to make
sure you read about it or enable it from the udev upgrade a while ago
if you decide to start from a fresh config or are installing without
that handbook you kind of have memorized.

Searching for CONFIG_DEVTMPFS in the forums and #gentoo logs shows that
this is quite often suggested as a fix and quite often actually fixes
an user's boot. Why duplicate telling users to do that if we can do it?=

There are a small set of other variables in this nature, mostly *TMPFS*.
This is why I think it would be handy to add a Gentoo section to the
kernel, along the lines as described by Linus.

https://l= kml.org/lkml/2012/7/13/369

The same goes for asking the user to enable configuration options in the kernel, why can't we just tell him to enable all option that toggles support for the user. For example; in the Gentoo section, there would be a "Init System Support" under which you can toggle an option to s= upport
the minimal requirements for some init system.

Feedback is very welcome.

P.S.: Not everything in this mail has been acked by the kernel lead;
only some thoughts, I was suggested to take this to ML for discussion.
The usage of the word 'we' in this mail is therefore hypothetical.<= br>

### F.A.Q. ###

Q: I don't want feature X? Please don't add the patch!

A: It's optional, don't enable it in your menu config.

Q: What about my stable server? I really don't want to run this stuff!<= br>
A: These options would depend on !CONFIG_VANILLA or CONFIG_EXPERIMENTAL
=C2=A0 =C2=A0which would be disabled by default, therefore if you keep this= option
=C2=A0 =C2=A0the way it is on your stable server; it won't affect you.<= br>
=C2=A0 =C2=A0In other words, genpatches stay as stable as before; unless yo= u
=C2=A0 =C2=A0explicitly toggle options that intentionally make it unstable.=

Q: Genpatches used to be minimal, would it gain weight?

A: If you don't enable those options, it would be as minimal as before;=
=C2=A0 =C2=A0this gives the user the choice between minimal and fat.

Q: What about patches that fail to build / run?

A: For testing kernels (~), we would do our best to make them work but
=C2=A0 =C2=A0there could be occasions where we will have to cut them; this = is no
=C2=A0 =C2=A0different from how the kernel herd has been handling this befo= re, we
=C2=A0 =C2=A0have already dropped fbcondecor in the past when it was broken= and
=C2=A0 =C2=A0the current new branch 3.10 does not support deblob yet.

=C2=A0 =C2=A0For stable kernels, which are near the EOL of a branch; if the=
=C2=A0 =C2=A0feature isn't still there yet then it means that its autho= r is
=C2=A0 =C2=A0simply no longer working on it, this is no different from when= the
=C2=A0 =C2=A0patch would be in another *-sources or manually applied by the= user.

=C2=A0 =C2=A0If you are unaware of how upstream releases work, just note th= at
=C2=A0 =C2=A0the major patch between 3.9.8 and 3.10.0 introduces a lot more=
=C2=A0 =C2=A0changes are applied than the minor patch between 3.9 and 3.9.8= ;
=C2=A0 =C2=A0therefore, we prefer to stabilize 3.9.8 or a later 3.9 kernel = than
=C2=A0 =C2=A0stabilizing 3.10.0. You can see this pattern in history; the =C2=A0 =C2=A0previous three have been 3.6.11, 3.7.10 and 3.8.13.

Q: Can't you do something about those build and runtime failures?
=C2=A0 =C2=A0I like to run testing, but I don't want breaks half of the= time.

A: To cover this, live ebuilds help, 3.10.9999 and 3.9999 and 9999;
=C2=A0 =C2=A0the earlier ones tracking the stable queues, whereas the last = one
=C2=A0 =C2=A0tracking the upstream rc kernel. This way we can notice what f= ails
=C2=A0 =C2=A0in advance and make it working by the time it reaches testing = (~).

=C2=A0 =C2=A0That being said, it is to be expected that things are not alwa= ys
=C2=A0 =C2=A0fine very early in a branch; it's why those kernels are ne= ver stable.

Q: What about kernel bugs, how would you know the user enabled them?

A: We expect the .config to be provided, which we can run a sanity
=C2=A0 =C2=A0check on; it's much more handy if we are aware of these pa= tches than
=C2=A0 =C2=A0that we have to figure out the user applies it on their own.
Q: Are documentation / infrastructure / eclass changes necessary?

A: The kernel project documentation has to be updated to reflect this;
=C2=A0 =C2=A0no infrastructure or eclass changes are needed, since this is = all
=C2=A0 =C2=A0done through patches in genpatches. (Sub directories supported= afaik)

Q: Does this affect kernel stabilization or QA?

A: No, experimental features (which would include these optional
=C2=A0 =C2=A0features) have historically never been looked at by the arch t= eams;
=C2=A0 =C2=A0if you can remember, there used to be an CONFIG_EXPERIMENTAL o= ption
=C2=A0 =C2=A0in the kernel to cover this. For example, you can find yoursel= f
=C2=A0 =C2=A0running some pretty unstable code if you use 3.8.13 and enable=
=C2=A0 =C2=A0Nouveau reclocking and power management; there are some option= s in
=C2=A0 =C2=A0the menuconfig that follow this nature.

=C2=A0 =C2=A0As there would be less *-sources as a consequence; less has to= be
=C2=A0 =C2=A0taken into consideration for stabilization, this is why most =C2=A0 =C2=A0*-sources are currently unstable, because we don't have th= e
=C2=A0 =C2=A0resources to be stabilizing them all.

=C2=A0 =C2=A0For QA, I don't really see a problem.

Q: So, would this make vanilla-sources the new gentoo-sources?

A: No, vanilla-sources undergoes some changes to more closely reflect
=C2=A0 =C2=A0the upstream kernels; as a consequence, it would no longer hav= e
=C2=A0 =C2=A0stable kernels and older versions will drop more often. You ca= n read
=C2=A0 =C2=A0this request and the resulting discussion at the following lin= k.

=C2=A0 =C2=A0http://thread.gmane.org/gmane.linux.gentoo.kernel/697=

=C2=A0 =C2=A0The kernel lead summarized a part of this.

=C2=A0 =C2=A0http://thread.gmane.org/gmane.linux.gento= o.kernel/697/focus=3D730

Q: Is this a one man army comedy show?

A: One man army comedy shows work, see geek-sources; if you want to
=C2=A0 =C2=A0look at some examples of what a one man army can do, the follo= wing
=C2=A0 =C2=A0link is a good read to go through.

=C2=A0 =C2=A0http://www.bennorthrop.co= m/Essays/2013/pair-programming-my-personal-nightmare.php

=C2=A0 =C2=A0But joking aside; no, we are not.

=C2=A0 =C2=A0We are at least with two on the kernel herd and a third develo= per is
=C2=A0 =C2=A0likely to join in the near future; on top of that, other peopl= e are
=C2=A0 =C2=A0welcome and I think it would be nice to see maintainers from =C2=A0 =C2=A0*-sources jump in to help along. genpatches isn't hard to = maintain.

=C2=A0 =C2=A0Since we're no longer a one man army, we can do more.

--
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address =C2=A0: TomWij@gentoo.o= rg
GPG Public Key =C2=A0: 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2 =C2=A0ABF0 95B2 1FCD 6D34 E57D

--047d7b5d253e0f82bc04e08e4e74--