[gentoo-dev] Proper distribution integration of kernel *-sources, patches and configuration.

[gentoo-dev] Proper distribution integration of kernel *-sources, patches and configuration.

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 9663 bytes --]

Hello


Please reply to gentoo-dev in case ML daemon changes Reply-To.


### TL; DR ###

By introducing feature patches which menu options are disabled by
default to genpatches, we can deduplicate *-sources maintainers as well
as large groups of users work. By introducing a distribution section
in the menuconfig, we can provide options that enable minimal Gentoo
requirements by default (DEVTMPFS, making Gentoo systems bootable since
an udev release a long time ago) and other distribution stuff.


### Proper distribution integration of kernel *-sources, patches ... ###

Gentoo is a distribution; but what is a distribution that doesn't
properly integrate what it provides, but instead expects its users to
do so, needlessly duplicating work amongst its maintainers and users.

Let's say I want genpatches, aufs and TuxOnIce; closest candidates:

 - sys-kernel/aufs-sources: genpatches, aufs
 - sys-kernel/pf-sources: genpatches, CK, BFQ, BFQ, TuxOnIce, UKSM
 - sys-kernel/tuxonice-sources: genpatches, TuxOnIce

What do I do? Take one (eg. aufs) and apply the other (eg. TuxOnIce)?
What if I want to add another common patchset to it? Hardened perhaps?

You can see, some of these sources (like pf-sources) already attempt to
do so; with pf-sources in mind, why do we even have ck-sources,
tuxonice-sources in the Portage tree? Just to duplicate work?

I think we should trim down on the amount of *-sources and combine
multiple patches into genpatches. Take an example of geek-sources
which does all this without a problem; I don't really like the approach
with USE flags used there, as the menuconfig can just cover that.

https://github.com/init6/init_6/wiki/geek-sources

What does a patch introducing new features really do? Or rather, what
should it do when we add it? Let me summarize:

 1) The features should be disabled by default.
 2) These feature should depend on a non-vanilla / experimental option.
 3) The patch should not affect the build by default.
 4) The user can optionally enable the feature.

So, in genpatches, since 3.9.7, BFQ was added to try this out (except
2.). Ensured it to be disabled by default, did not affect the build for
anyone that does not enable it and the users have been enabling and
using it on their own. So, does it differentiate more from vanilla? No.

This helps users as well as maintainers to not have to apply BFQ on
their own, they simply have to tick a config option and they are set.
If all patches that introduce new features are added in this fashion,
it spares large groups of users from having to do this on their own; we
can also deduplicate the work in the Portage tree this way.


### ... and configuration. ###

This problem is not only visible for patches, but also in the config.

Meet CONFIG_DEVTMPFS; forget to enable it, greet a failing boot. We're
telling users to enable it in some places, in the handbook it's a single
line you must read, on the Wiki it's kind of missing unless you are
luckily on the right page, on the Quick Install book it is missing too.

So, we are currently providing a configuration that expects anyone to
enable CONFIG_DEVTMPFS. But you have to remember that it need to make
sure you read about it or enable it from the udev upgrade a while ago
if you decide to start from a fresh config or are installing without
that handbook you kind of have memorized.

Searching for CONFIG_DEVTMPFS in the forums and #gentoo logs shows that
this is quite often suggested as a fix and quite often actually fixes
an user's boot. Why duplicate telling users to do that if we can do it?

There are a small set of other variables in this nature, mostly *TMPFS*.

This is why I think it would be handy to add a Gentoo section to the
kernel, along the lines as described by Linus.

https://lkml.org/lkml/2012/7/13/369

The same goes for asking the user to enable configuration options in the
kernel, why can't we just tell him to enable all option that toggles
support for the user. For example; in the Gentoo section, there would be
a "Init System Support" under which you can toggle an option to support
the minimal requirements for some init system.

Feedback is very welcome.

P.S.: Not everything in this mail has been acked by the kernel lead;
only some thoughts, I was suggested to take this to ML for discussion.
The usage of the word 'we' in this mail is therefore hypothetical.


### F.A.Q. ###

Q: I don't want feature X? Please don't add the patch!

A: It's optional, don't enable it in your menu config.

Q: What about my stable server? I really don't want to run this stuff!

A: These options would depend on !CONFIG_VANILLA or CONFIG_EXPERIMENTAL
   which would be disabled by default, therefore if you keep this option
   the way it is on your stable server; it won't affect you.

   In other words, genpatches stay as stable as before; unless you
   explicitly toggle options that intentionally make it unstable.

Q: Genpatches used to be minimal, would it gain weight?

A: If you don't enable those options, it would be as minimal as before;
   this gives the user the choice between minimal and fat.

Q: What about patches that fail to build / run?

A: For testing kernels (~), we would do our best to make them work but
   there could be occasions where we will have to cut them; this is no
   different from how the kernel herd has been handling this before, we
   have already dropped fbcondecor in the past when it was broken and
   the current new branch 3.10 does not support deblob yet.

   For stable kernels, which are near the EOL of a branch; if the
   feature isn't still there yet then it means that its author is
   simply no longer working on it, this is no different from when the
   patch would be in another *-sources or manually applied by the user.

   If you are unaware of how upstream releases work, just note that
   the major patch between 3.9.8 and 3.10.0 introduces a lot more
   changes are applied than the minor patch between 3.9 and 3.9.8;
   therefore, we prefer to stabilize 3.9.8 or a later 3.9 kernel than
   stabilizing 3.10.0. You can see this pattern in history; the
   previous three have been 3.6.11, 3.7.10 and 3.8.13.

Q: Can't you do something about those build and runtime failures?
   I like to run testing, but I don't want breaks half of the time.

A: To cover this, live ebuilds help, 3.10.9999 and 3.9999 and 9999;
   the earlier ones tracking the stable queues, whereas the last one
   tracking the upstream rc kernel. This way we can notice what fails
   in advance and make it working by the time it reaches testing (~).

   That being said, it is to be expected that things are not always
   fine very early in a branch; it's why those kernels are never stable.

Q: What about kernel bugs, how would you know the user enabled them?

A: We expect the .config to be provided, which we can run a sanity
   check on; it's much more handy if we are aware of these patches than
   that we have to figure out the user applies it on their own.

Q: Are documentation / infrastructure / eclass changes necessary?

A: The kernel project documentation has to be updated to reflect this;
   no infrastructure or eclass changes are needed, since this is all
   done through patches in genpatches. (Sub directories supported afaik)

Q: Does this affect kernel stabilization or QA?

A: No, experimental features (which would include these optional
   features) have historically never been looked at by the arch teams;
   if you can remember, there used to be an CONFIG_EXPERIMENTAL option
   in the kernel to cover this. For example, you can find yourself
   running some pretty unstable code if you use 3.8.13 and enable
   Nouveau reclocking and power management; there are some options in
   the menuconfig that follow this nature.

   As there would be less *-sources as a consequence; less has to be
   taken into consideration for stabilization, this is why most
   *-sources are currently unstable, because we don't have the
   resources to be stabilizing them all.

   For QA, I don't really see a problem.

Q: So, would this make vanilla-sources the new gentoo-sources?

A: No, vanilla-sources undergoes some changes to more closely reflect
   the upstream kernels; as a consequence, it would no longer have
   stable kernels and older versions will drop more often. You can read
   this request and the resulting discussion at the following link.

   http://thread.gmane.org/gmane.linux.gentoo.kernel/697

   The kernel lead summarized a part of this.

   http://thread.gmane.org/gmane.linux.gentoo.kernel/697/focus=730

Q: Is this a one man army comedy show?

A: One man army comedy shows work, see geek-sources; if you want to
   look at some examples of what a one man army can do, the following
   link is a good read to go through.

   http://www.bennorthrop.com/Essays/2013/pair-programming-my-personal-nightmare.php

   But joking aside; no, we are not.

   We are at least with two on the kernel herd and a third developer is
   likely to join in the near future; on top of that, other people are
   welcome and I think it would be nice to see maintainers from
   *-sources jump in to help along. genpatches isn't hard to maintain.

   Since we're no longer a one man army, we can do more.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Proper distribution integration of kernel *-sources, patches and configuration.

[Ben de Groot]

On 1 July 2013 22:41, Tom Wijsman <TomWij@gentoo.org> wrote:
>
> ### TL; DR ###
>
> By introducing feature patches which menu options are disabled by
> default to genpatches, we can deduplicate *-sources maintainers as well
> as large groups of users work. By introducing a distribution section
> in the menuconfig, we can provide options that enable minimal Gentoo
> requirements by default (DEVTMPFS, making Gentoo systems bootable since
> an udev release a long time ago) and other distribution stuff.
>

Sounds like a great idea to me!

--
Cheers,

Ben | yngwin
Gentoo developer

[gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Jeff Horelick]

On 1 July 2013 10:41, Tom Wijsman <TomWij@gentoo.org> wrote:
> Hello
>
>
> Please reply to gentoo-dev in case ML daemon changes Reply-To.
>
>
> ### TL; DR ###
>
> By introducing feature patches which menu options are disabled by
> default to genpatches, we can deduplicate *-sources maintainers as well
> as large groups of users work. By introducing a distribution section
> in the menuconfig, we can provide options that enable minimal Gentoo
> requirements by default (DEVTMPFS, making Gentoo systems bootable since
> an udev release a long time ago) and other distribution stuff.


I really like this idea. geek-sources has appealed to me massively the
past few months, but i didn't want to risk stability and go with a
kernel from a unofficial overlay. I can not wait to see this in
gentoo-sources and I fully support this idea.

Re: [gentoo-dev] Proper distribution integration of kernel *-sources, patches and configuration.

[Rick "Zero_Chaos" Farina]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 07/01/2013 10:41 AM, Tom Wijsman wrote:
> Hello
> 
> 
> Please reply to gentoo-dev in case ML daemon changes Reply-To.
> 
> 
> ### TL; DR ###
> 
> By introducing feature patches which menu options are disabled by
> default to genpatches, we can deduplicate *-sources maintainers as well
> as large groups of users work. By introducing a distribution section
> in the menuconfig, we can provide options that enable minimal Gentoo
> requirements by default (DEVTMPFS, making Gentoo systems bootable since
> an udev release a long time ago) and other distribution stuff.
> 
> 
> ### Proper distribution integration of kernel *-sources, patches ... ###
> 
> Gentoo is a distribution; but what is a distribution that doesn't
> properly integrate what it provides, but instead expects its users to
> do so, needlessly duplicating work amongst its maintainers and users.
> 
> Let's say I want genpatches, aufs and TuxOnIce; closest candidates:
> 
>  - sys-kernel/aufs-sources: genpatches, aufs
>  - sys-kernel/pf-sources: genpatches, CK, BFQ, BFQ, TuxOnIce, UKSM
>  - sys-kernel/tuxonice-sources: genpatches, TuxOnIce
> 
> What do I do? Take one (eg. aufs) and apply the other (eg. TuxOnIce)?
> What if I want to add another common patchset to it? Hardened perhaps?

Some patches are reasonably easy to combine, such as genpatches and
aufs.  Some patches are difficult to combine, such as hardened and *.
When you combine hardened patches and aufs (for example) you need extra
patches.  I would be THRILLED to see the number of sources cut down, but
hardened-sources must be it's own thing (that said, I'll personally
maintain the aufs patches for hardened if they wanted to add a USE=aufs
flag).
> 
> You can see, some of these sources (like pf-sources) already attempt to
> do so; with pf-sources in mind, why do we even have ck-sources,
> tuxonice-sources in the Portage tree? Just to duplicate work?
> 
> I think we should trim down on the amount of *-sources and combine
> multiple patches into genpatches. Take an example of geek-sources
> which does all this without a problem; I don't really like the approach
> with USE flags used there, as the menuconfig can just cover that.
> 
> https://github.com/init6/init_6/wiki/geek-sources
> 
> What does a patch introducing new features really do? Or rather, what
> should it do when we add it? Let me summarize:
> 
>  1) The features should be disabled by default.
>  2) These feature should depend on a non-vanilla / experimental option.
If users want a vanilla kernel, they want vanilla-sources.  Nothing
about that should change.  I don't feel that it would be honest to add a
vanilla use flag to gentoo-sources as in no reality are those vanilla.

Thanks,
Zero

>  3) The patch should not affect the build by default.
>  4) The user can optionally enable the feature.
> 
> So, in genpatches, since 3.9.7, BFQ was added to try this out (except
> 2.). Ensured it to be disabled by default, did not affect the build for
> anyone that does not enable it and the users have been enabling and
> using it on their own. So, does it differentiate more from vanilla? No.
> 
> This helps users as well as maintainers to not have to apply BFQ on
> their own, they simply have to tick a config option and they are set.
> If all patches that introduce new features are added in this fashion,
> it spares large groups of users from having to do this on their own; we
> can also deduplicate the work in the Portage tree this way.
> 
> 
> ### ... and configuration. ###
> 
> This problem is not only visible for patches, but also in the config.
> 
> Meet CONFIG_DEVTMPFS; forget to enable it, greet a failing boot. We're
> telling users to enable it in some places, in the handbook it's a single
> line you must read, on the Wiki it's kind of missing unless you are
> luckily on the right page, on the Quick Install book it is missing too.
> 
> So, we are currently providing a configuration that expects anyone to
> enable CONFIG_DEVTMPFS. But you have to remember that it need to make
> sure you read about it or enable it from the udev upgrade a while ago
> if you decide to start from a fresh config or are installing without
> that handbook you kind of have memorized.
> 
> Searching for CONFIG_DEVTMPFS in the forums and #gentoo logs shows that
> this is quite often suggested as a fix and quite often actually fixes
> an user's boot. Why duplicate telling users to do that if we can do it?
> 
> There are a small set of other variables in this nature, mostly *TMPFS*.
> 
> This is why I think it would be handy to add a Gentoo section to the
> kernel, along the lines as described by Linus.
> 
> https://lkml.org/lkml/2012/7/13/369
> 
> The same goes for asking the user to enable configuration options in the
> kernel, why can't we just tell him to enable all option that toggles
> support for the user. For example; in the Gentoo section, there would be
> a "Init System Support" under which you can toggle an option to support
> the minimal requirements for some init system.
> 
> Feedback is very welcome.
> 
> P.S.: Not everything in this mail has been acked by the kernel lead;
> only some thoughts, I was suggested to take this to ML for discussion.
> The usage of the word 'we' in this mail is therefore hypothetical.
> 
> 
> ### F.A.Q. ###
> 
> Q: I don't want feature X? Please don't add the patch!
> 
> A: It's optional, don't enable it in your menu config.
> 
> Q: What about my stable server? I really don't want to run this stuff!
> 
> A: These options would depend on !CONFIG_VANILLA or CONFIG_EXPERIMENTAL
>    which would be disabled by default, therefore if you keep this option
>    the way it is on your stable server; it won't affect you.
> 
>    In other words, genpatches stay as stable as before; unless you
>    explicitly toggle options that intentionally make it unstable.
> 
> Q: Genpatches used to be minimal, would it gain weight?
> 
> A: If you don't enable those options, it would be as minimal as before;
>    this gives the user the choice between minimal and fat.
> 
> Q: What about patches that fail to build / run?
> 
> A: For testing kernels (~), we would do our best to make them work but
>    there could be occasions where we will have to cut them; this is no
>    different from how the kernel herd has been handling this before, we
>    have already dropped fbcondecor in the past when it was broken and
>    the current new branch 3.10 does not support deblob yet.
> 
>    For stable kernels, which are near the EOL of a branch; if the
>    feature isn't still there yet then it means that its author is
>    simply no longer working on it, this is no different from when the
>    patch would be in another *-sources or manually applied by the user.
> 
>    If you are unaware of how upstream releases work, just note that
>    the major patch between 3.9.8 and 3.10.0 introduces a lot more
>    changes are applied than the minor patch between 3.9 and 3.9.8;
>    therefore, we prefer to stabilize 3.9.8 or a later 3.9 kernel than
>    stabilizing 3.10.0. You can see this pattern in history; the
>    previous three have been 3.6.11, 3.7.10 and 3.8.13.
> 
> Q: Can't you do something about those build and runtime failures?
>    I like to run testing, but I don't want breaks half of the time.
> 
> A: To cover this, live ebuilds help, 3.10.9999 and 3.9999 and 9999;
>    the earlier ones tracking the stable queues, whereas the last one
>    tracking the upstream rc kernel. This way we can notice what fails
>    in advance and make it working by the time it reaches testing (~).
> 
>    That being said, it is to be expected that things are not always
>    fine very early in a branch; it's why those kernels are never stable.
> 
> Q: What about kernel bugs, how would you know the user enabled them?
> 
> A: We expect the .config to be provided, which we can run a sanity
>    check on; it's much more handy if we are aware of these patches than
>    that we have to figure out the user applies it on their own.
> 
> Q: Are documentation / infrastructure / eclass changes necessary?
> 
> A: The kernel project documentation has to be updated to reflect this;
>    no infrastructure or eclass changes are needed, since this is all
>    done through patches in genpatches. (Sub directories supported afaik)
> 
> Q: Does this affect kernel stabilization or QA?
> 
> A: No, experimental features (which would include these optional
>    features) have historically never been looked at by the arch teams;
>    if you can remember, there used to be an CONFIG_EXPERIMENTAL option
>    in the kernel to cover this. For example, you can find yourself
>    running some pretty unstable code if you use 3.8.13 and enable
>    Nouveau reclocking and power management; there are some options in
>    the menuconfig that follow this nature.
> 
>    As there would be less *-sources as a consequence; less has to be
>    taken into consideration for stabilization, this is why most
>    *-sources are currently unstable, because we don't have the
>    resources to be stabilizing them all.
> 
>    For QA, I don't really see a problem.
> 
> Q: So, would this make vanilla-sources the new gentoo-sources?
> 
> A: No, vanilla-sources undergoes some changes to more closely reflect
>    the upstream kernels; as a consequence, it would no longer have
>    stable kernels and older versions will drop more often. You can read
>    this request and the resulting discussion at the following link.
> 
>    http://thread.gmane.org/gmane.linux.gentoo.kernel/697
> 
>    The kernel lead summarized a part of this.
> 
>    http://thread.gmane.org/gmane.linux.gentoo.kernel/697/focus=730
> 
> Q: Is this a one man army comedy show?
> 
> A: One man army comedy shows work, see geek-sources; if you want to
>    look at some examples of what a one man army can do, the following
>    link is a good read to go through.
> 
>    http://www.bennorthrop.com/Essays/2013/pair-programming-my-personal-nightmare.php
> 
>    But joking aside; no, we are not.
> 
>    We are at least with two on the kernel herd and a third developer is
>    likely to join in the near future; on top of that, other people are
>    welcome and I think it would be nice to see maintainers from
>    *-sources jump in to help along. genpatches isn't hard to maintain.
> 
>    Since we're no longer a one man army, we can do more.
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJR0aw5AAoJEKXdFCfdEflKc+AQAKGa7v+beI5x6oBFoMCri4xM
pVX3jNdjtqidNg9Kyg5yMtoUl2WW6jJp8nIgd+gLeJ+8aNg1G/L94Jc4iRIHw8TH
Rv1oTafAa7Xc76ZxOvNHJMQNjTV+DtTsGuAKD0KAeEOJ6ld+wwS55D26NGXYAe0i
J1wPR7ndDHscBQQCGS+S4qAgHvfLaVq6wu7oQaGi9CTBTT1AtlmhkbBTH2qHTBJX
ZV98+hqiLBto6A42FmmZbqEooFWMGtcoJzlZNYCZduQ6uODPy30vt55dhh2OKsZW
5FCmUNdiekY8KRFWlYfcVFPAsrKVCutba/PEZunm9WwZ1Z9AoVz8WzBBLQU6b30P
vAzkUeTfET+y/NnKcuEO7j7CRXXm3+IjbNRPJ8qh5KiEiq1QfyxiOvZBQe08PhdP
i2Hxau6RhGeHVedyv9quImxBwNTdBgdJjuLlEP6cPBRMUb9HNqa0YNdsZ8CYu8I3
6rbL3pbUJmzhUUsbgV5nFdIZOlKWHOICzOuwKnR8T9OklWhI12gZUEePfYvk2Rl1
v5pYsse25O2FduNu24bwDI8qnUvtfEsqLpr9y5z0e94BX0FUR1wITb8Zp4KsKzo3
+R9nzQFqe9cF544Pt1Bp1ljpXEfXQl05aYEWEBFttkLSxHxMNtv2lHFh4C3Q9NNm
s4P30hArFvTeCo4OIfOd
=+mZ8
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Proper distribution integration of kernel *-sources, patches and configuration.

[hasufell]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 07/01/2013 06:20 PM, Rick "Zero_Chaos" Farina wrote:
> On 07/01/2013 10:41 AM, Tom Wijsman wrote:
> 
>> What does a patch introducing new features really do? Or rather,
>> what should it do when we add it? Let me summarize:
> 
>> 1) The features should be disabled by default. 2) These feature
>> should depend on a non-vanilla / experimental option.
> If users want a vanilla kernel, they want vanilla-sources.
> Nothing about that should change.  I don't feel that it would be
> honest to add a vanilla use flag to gentoo-sources as in no reality
> are those vanilla.
> 

Agreed, some of this sounds interesting (although I don't care as a
user), but whatever you do... don't mess with vanilla-sources.

I stopped using gentoo-sources, because some random hacks broke some
network drivers once and I don't want a hackjob on vanilla-sources...
optional or not.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.20 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJR0a5AAAoJEFpvPKfnPDWzf4wIAIhfHvIJmqU3jfIVvGlNL5QR
Z8SM4jyw95ehtTMWDdpnEdLUuW7yu8pK9K9N9cXSchvQ3GJfjxJTI/v7RSI+WPBN
NrTgfKYF1KGg6jNQXjuiyG5QwV9faE7zEZ8unDRyxX0EhWyiWACjuSzBdpzS6Nhm
QcDCEzzuXNtPR44pqfDQ3DqqRU+aUAE7juM+Yd146x3CFDE8vvVuvuGYnXhVczQ8
vkfdqLNLMamIROWapV4HG8p2NOzDbPjbUcMB7uBsP8DPm/HjOQRNxyY0yCD38hq3
4RJFZz7RPTjWQ2p8PacHKZ4Rye6EY7W/x6JTVqmh3zbc+6tm6q7kKtMyn1L03Bw=
=ds3y
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Proper distribution integration of kernel *-sources, patches and configuration.

[Tom Wijsman]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Mon, 01 Jul 2013 12:20:09 -0400
"Rick \"Zero_Chaos\" Farina" <zerochaos@gentoo.org> wrote:

> Some patches are reasonably easy to combine, such as genpatches and
> aufs.  Some patches are difficult to combine, such as hardened and *.
> When you combine hardened patches and aufs (for example) you need
> extra patches.  I would be THRILLED to see the number of sources cut
> down, but hardened-sources must be it's own thing (that said, I'll
> personally maintain the aufs patches for hardened if they wanted to
> add a USE=aufs flag).

Yes, gave it as an quick example but I indeed remember from going
through the sources ebuilds that hardened ebuilds do quite some things.
I think the downside from extending genpatches is that hardened-sources
can no longer rely on it, but we'll have to see that as we go forward.

I don't think that apart from hardened the optional patches on their own
are hard to combine; they each have their own separate goal, I don't
see them conflict on anything. If it happens once in a while, we can
still maintain them to work together.

Also note that I do not plan to introduce any USE flags, since that
would duplicate the options to be listed in the kernel menuconfig.

> If users want a vanilla kernel, they want vanilla-sources.  Nothing
> about that should change.  I don't feel that it would be honest to
> add a vanilla use flag to gentoo-sources as in no reality are those
> vanilla.

Apart from the changes discussed on the gentoo-kernel ML, nothing else
there will change. You can read the thread as well as the summary; if
you disagree, you're welcome to join the discussion there.

http://thread.gmane.org/gmane.linux.gentoo.kernel/697

http://thread.gmane.org/gmane.linux.gentoo.kernel/697/focus=730

But yes, apart from that, vanilla-sources will give you vanilla kernel.

- -- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.20 (GNU/Linux)

iQEcBAEBAgAGBQJR0b3UAAoJEJWyH81tNOV9UAMIAMSQy/EPWr9l4JCVMaB3x7bb
ezv8vrmSDxFUB30mX9cPPAvfpaeWSFvRCpvPfEsKkHAcTJomSfP7PxgEVKLA4jT2
TwoovBQsADEowAIgUHFr9lqIrE9HQ0nik89bNz1HxkbUl4uQijnnfcLTu/6WpdGJ
aMAxImTXQc1Py7w+RNcy9fR0pXh6zXx23CWNVMvn2hb+wGkQDbzy4gOeHFok2i9F
3B3ZjLNUdY2agEBpo0AqF0hwSPHUDbrteFVwJKjauYTyNA8Ofc5OhRnpGHEbq5/4
mB4J2R/MnsukXNeNJUgLzjqSykKChdNbImLl6OXXXl1PyinjuAdBeC7SoUV3DDg=
=5OIS
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Proper distribution integration of kernel *-sources, patches and configuration.

[Rick "Zero_Chaos" Farina]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 07/01/2013 01:35 PM, Tom Wijsman wrote:
> On Mon, 01 Jul 2013 12:20:09 -0400
> "Rick \"Zero_Chaos\" Farina" <zerochaos@gentoo.org> wrote:
> 
>> Some patches are reasonably easy to combine, such as genpatches and
>> aufs.  Some patches are difficult to combine, such as hardened and *.
>> When you combine hardened patches and aufs (for example) you need
>> extra patches.  I would be THRILLED to see the number of sources cut
>> down, but hardened-sources must be it's own thing (that said, I'll
>> personally maintain the aufs patches for hardened if they wanted to
>> add a USE=aufs flag).
> 
> Yes, gave it as an quick example but I indeed remember from going
> through the sources ebuilds that hardened ebuilds do quite some things.
> I think the downside from extending genpatches is that hardened-sources
> can no longer rely on it, but we'll have to see that as we go forward.
> 
> I don't think that apart from hardened the optional patches on their own
> are hard to combine; they each have their own separate goal, I don't
> see them conflict on anything. If it happens once in a while, we can
> still maintain them to work together.

Hardened has K_WANT_GENPATCHES="base" which means it already doesn't
take the extra patches.  We could either introduce a new flag for your
patches like K_WANT_GENPATCHES="base extra geek" or more likely make
each one with their own name so that hardened et al can take what they
like and leave the rest.
> 
> Also note that I do not plan to introduce any USE flags, since that
> would duplicate the options to be listed in the kernel menuconfig.

Good point.

Thanks,
Zero
> 
>> If users want a vanilla kernel, they want vanilla-sources.  Nothing
>> about that should change.  I don't feel that it would be honest to
>> add a vanilla use flag to gentoo-sources as in no reality are those
>> vanilla.
> 
> Apart from the changes discussed on the gentoo-kernel ML, nothing else
> there will change. You can read the thread as well as the summary; if
> you disagree, you're welcome to join the discussion there.
> 
> http://thread.gmane.org/gmane.linux.gentoo.kernel/697
> 
> http://thread.gmane.org/gmane.linux.gentoo.kernel/697/focus=730
> 
> But yes, apart from that, vanilla-sources will give you vanilla kernel.
> 
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJR0cHKAAoJEKXdFCfdEflKwOYQAIArPK7SsJ6M9BFCuhf/bFOg
yZfVpSRR8UJgl7as6FsVg4q48NFuFh3lm8PQK792IraccXkYjk/kBGY6IyDt3rUY
yXHberr53cBZKPC0PPVzDo7ER73GWkobrd+vDrtRHeTJEaPQknvOEmQFBnjY181o
8uTqQi2VtTnWPP2PeIkqdobkasUcf5lDDXdrvX+ipN+oOSSZ0VJK+XfdlstgrRnH
o7sESjKm7RbvxQFizzGuE7gOh9GFtIY92zpQVUO4L4P/L5NrGz0yqyor0WYKuUhN
dZ3k84FQ5SDyKCdCMq/JPKS8jj47gFIkZwfArwKNsRsxkBtcsFHQuj2VXSx0MEcp
aKv5FfZCj4iSUAg2uwKVfVonyn5qt73Fm+XquxjfbEaT4oTq0FFCL5zjCuf1Zzpt
3/VOer5N5xu5gY0y6Yt5w7ionHLAFWqXgJF7s/sC7L9eJNHT3XiQtZSPxLGeAkG8
beM9PNt7fdYzQAudbi6NZiJ35ZwZQrQfKEAc16hbcH0qDd7ndTWqg0nELX3ulL2z
FbeCMM0J+tmZ8lEgRriUL7Ki/een1DJX4eCQhYbKIMYKdxDeEMkbZ8N/T3dAWtxm
IivIs4tTK9EWBF6+8kmUINszCQBsLI6P50TzocqHV+Tj/RyCmeJnOia7DLTD0HGg
53QUb/jcDjkTF34HbUdG
=gsg9
-----END PGP SIGNATURE-----

[gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Greg KH]

On Mon, Jul 01, 2013 at 04:41:49PM +0200, Tom Wijsman wrote:
> This problem is not only visible for patches, but also in the config.
> 
> Meet CONFIG_DEVTMPFS; forget to enable it, greet a failing boot. We're
> telling users to enable it in some places, in the handbook it's a single
> line you must read, on the Wiki it's kind of missing unless you are
> luckily on the right page, on the Quick Install book it is missing too.

This is not the only build option that users must enable to get a
booting system by far.  So why single this one out?

> Q: I don't want feature X? Please don't add the patch!
> 
> A: It's optional, don't enable it in your menu config.
> 
> Q: What about my stable server? I really don't want to run this stuff!
> 
> A: These options would depend on !CONFIG_VANILLA or CONFIG_EXPERIMENTAL

What is CONFIG_VANILLA?  I don't see that in the upstream kernel tree at
all.

CONFIG_EXPERIMENTAL is now gone from upstream, so you are going to have
a problem with this.

>    which would be disabled by default, therefore if you keep this option
>    the way it is on your stable server; it won't affect you.

Not always true.  Look at aufs as an example.  It patches the core
kernel code in ways that are _not_ accepted upstream yet.  Now you all
are running that modified code, even if you don't want aufs.

Note, I'm just using aufs as an example here, I'm not commenting on the
quality of the code, or why it is modifying the core kernel at all, I
happen to run it on some of my own servers, but your feelings might
differ.

>    In other words, genpatches stay as stable as before; unless you
>    explicitly toggle options that intentionally make it unstable.

As pointed out above, this is not always true.

Also, as others stated, the "hardened" patches also don't always only
touch areas covered by non-config-selected portions of the kernel.

Mix and match your external kernel patches at your own risk, what you
are suggesting does make it "easy" for users, but I bet it will be a
huge support issue for the already-overworked gentoo kernel developers,
the combinations just are too big to test and guarantee working.

good luck,

greg "stick to the vanilla-sources" k-h

Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Anthony G. Basile]

On 07/01/2013 11:20 AM, Jeff Horelick wrote:
> On 1 July 2013 10:41, Tom Wijsman <TomWij@gentoo.org> wrote:
>> Hello
>>
>>
>> Please reply to gentoo-dev in case ML daemon changes Reply-To.
>>
>>
>> ### TL; DR ###
>>
>> By introducing feature patches which menu options are disabled by
>> default to genpatches, we can deduplicate *-sources maintainers as well
>> as large groups of users work. By introducing a distribution section
>> in the menuconfig, we can provide options that enable minimal Gentoo
>> requirements by default (DEVTMPFS, making Gentoo systems bootable since
>> an udev release a long time ago) and other distribution stuff.
>
> I really like this idea. geek-sources has appealed to me massively the
> past few months, but i didn't want to risk stability and go with a
> kernel from a unofficial overlay. I can not wait to see this in
> gentoo-sources and I fully support this idea.
>
Tom, you already know my opinion because we discussed it.  I'm all for 
it.  Just a reminder: there's always problems somewhere in the kernel 
which can be triggered by various options.  The kernel is not one big 
take it or leave it chunk of code, but many chunks selectable by Kconfig 
with the exception of course of the core.  The best we can do wrt to BFQ 
and other "risky" patches is mark these options as EXPERIMENTAL.  I was 
going to say depend on CONFIG_EXPERIMENTAL in Kconfig, but this is 
deprecated.  See scripts/checkpatch.pl

     "Use of CONFIG_EXPERIMENTAL is deprecated. For alternatives, see 
https://lkml.org/lkml/2012/10/23/580"

--Tony

-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA

Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Markos Chandras]

On 1 July 2013 19:17, Greg KH <gregkh@gentoo.org> wrote:
>
> greg "stick to the vanilla-sources" k-h
>

Before these changes were introduced, the gentoo-sources and
vanilla-sources were quite similar in the sense that genpatches used
to contain
patches already in Linus' tree. However, given that gentoo-sources
will become heavily patched by external 3rd-party patches, it might
makes more
sense to recommend vanilla-sources to our users (handbook, etc).
I certainly don't feel safe anymore running non-upstream code in
production boxes.

--
Regards,
Markos Chandras - Gentoo Linux Developer
http://dev.gentoo.org/~hwoarang

Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 3161 bytes --]

On Mon, 1 Jul 2013 11:17:49 -0700
Greg KH <gregkh@gentoo.org> wrote:

> > Meet CONFIG_DEVTMPFS; ...
> 
> This is not the only build option that users must enable to get a
> booting system by far.  So why single this one out?

Because it is an example. Later on I explicitly mention "There are a
small set of other variables in this nature, ..."; I'm talking about
configuration options not present in the defconfig, specifically those
that everyone who runs a Gentoo based system (@system set + stage3)
wants to have enabled.

> > Q: What about my stable server? I really don't want to run this
> > stuff!
> > 
> > A: These options would depend on !CONFIG_VANILLA or
> > CONFIG_EXPERIMENTAL
> 
> What is CONFIG_VANILLA?  I don't see that in the upstream kernel tree
> at all.
> 
> CONFIG_EXPERIMENTAL is now gone from upstream, so you are going to
> have a problem with this.

Earlier I mentioned "2) These feature should depend on a non-vanilla /
experimental option." which is an option we would introduce under the
Gentoo distribution menu section.
 
> >    which would be disabled by default, therefore if you keep this
> > option the way it is on your stable server; it won't affect you.
> 
> Not always true.  Look at aufs as an example.  It patches the core
> kernel code in ways that are _not_ accepted upstream yet.  Now you all
> are running that modified code, even if you don't want aufs.

Earlier I mentioned "3) The patch should not affect the build by
default."; if it does, we have to adjust it to not do that, this is
something that can be easily scripted. It's just a matter of embedding
each + block in the diff with a config check and updating the counts.

> >    In other words, genpatches stay as stable as before; unless you
> >    explicitly toggle options that intentionally make it unstable.
> 
> As pointed out above, this is not always true.

Under the above condition (3), it is always true.

> Also, as others stated, the "hardened" patches also don't always only
> touch areas covered by non-config-selected portions of the kernel.

Yes, it seems wise to keep hardened-sources separated.

> Mix and match your external kernel patches at your own risk, what you
> are suggesting does make it "easy" for users, but I bet it will be a
> huge support issue for the already-overworked gentoo kernel
> developers, the combinations just are too big to test and guarantee
> working.

I'm waiting for you to push new kernels to kernel.org.

Joking aside; users are doing this anyway, it is better to know about
it. Who knows some of the bugs we have is the result of our unawareness.

Agreed, we have to watch out how far we push this though; which is why
we should start with just those patches that appear the most in other
sources, carefully meeting our goal over time. Not mimic geek-sources
all at once, I believe it took him some time to reach that point...

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 796 bytes --]

On Mon, 1 Jul 2013 19:38:48 +0100
Markos Chandras <hwoarang@gentoo.org> wrote:

> I certainly don't feel safe anymore running non-upstream code in
> production boxes.

You don't run it unless you explicitly tick on that you want
experimental functionality _as well as_ the optional features in
question; as I said earlier on chat, I don't understand your point here.

If you don't enable them, genpatches is just like it is before; I'm
not sure why the recommendations should change here, especially with
vanilla-sources taking a further step away from Gentoo Security and QA.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 827 bytes --]

On Mon, 01 Jul 2013 14:30:51 -0400
"Anthony G. Basile" <blueness@gentoo.org> wrote:

> I was going to say depend on CONFIG_EXPERIMENTAL in
> Kconfig, but this is deprecated.  See scripts/checkpatch.pl

Yes, I think it wasn't clear from my first post; but the intention was
to introduce such option under the Gentoo distribution menu section. I
think I forgot to mention that near the end. I'm well aware, which is
why we should proceed slowly; since I will have an away period soon, I
don't plan to implement this this month and set up the organization
before starting anything to ensure we do this properly.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Matthew Summers]

On Mon, Jul 1, 2013 at 1:56 PM, Tom Wijsman <TomWij@gentoo.org> wrote:
> On Mon, 1 Jul 2013 19:38:48 +0100
> Markos Chandras <hwoarang@gentoo.org> wrote:
>
>> I certainly don't feel safe anymore running non-upstream code in
>> production boxes.
>
> You don't run it unless you explicitly tick on that you want
> experimental functionality _as well as_ the optional features in
> question; as I said earlier on chat, I don't understand your point here.
>
> If you don't enable them, genpatches is just like it is before; I'm
> not sure why the recommendations should change here, especially with
> vanilla-sources taking a further step away from Gentoo Security and QA.
>

Tom,

I think the point was well-made by grehkh. If the patchset patches the
kernel's core, it doesn't matter what CONFIG_* option is set the core
kernel code _has_now_been_changed_. This is the crux of the argument,
I believe. AUFS simply being one example of this. I'm sure there are
others.

-- 
Matthew W. Summers
Gentoo Foundation Inc.
GPG: 111B C438 35FA EDB5 B5D3 736F 45EE 5DC0 0878 9D46

Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Greg KH]

On Mon, Jul 01, 2013 at 08:45:16PM +0200, Tom Wijsman wrote:
> > > Q: What about my stable server? I really don't want to run this
> > > stuff!
> > > 
> > > A: These options would depend on !CONFIG_VANILLA or
> > > CONFIG_EXPERIMENTAL
> > 
> > What is CONFIG_VANILLA?  I don't see that in the upstream kernel tree
> > at all.
> > 
> > CONFIG_EXPERIMENTAL is now gone from upstream, so you are going to
> > have a problem with this.
> 
> Earlier I mentioned "2) These feature should depend on a non-vanilla /
> experimental option." which is an option we would introduce under the
> Gentoo distribution menu section.

Distro-specific config options, great :(

> > >    which would be disabled by default, therefore if you keep this
> > > option the way it is on your stable server; it won't affect you.
> > 
> > Not always true.  Look at aufs as an example.  It patches the core
> > kernel code in ways that are _not_ accepted upstream yet.  Now you all
> > are running that modified code, even if you don't want aufs.
> 
> Earlier I mentioned "3) The patch should not affect the build by
> default."; if it does, we have to adjust it to not do that, this is
> something that can be easily scripted. It's just a matter of embedding
> each + block in the diff with a config check and updating the counts.

Look at aufs as a specific example of why you can't do that, otherwise,
don't you think that the aufs developer(s) wouldn't have done so?

The goal of "don't touch any other kernel code" is a very good one, but
not always true for these huge out-of-tree kernel patches.  Usually that
is the main reason why these patches aren't merged upstream, because
those changes are not acceptable.

So be very careful here, you are messing with things that are rejected
by upstream.

greg k-h

Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Greg KH]

On Mon, Jul 01, 2013 at 02:30:51PM -0400, Anthony G. Basile wrote:
> Tom, you already know my opinion because we discussed it.  I'm all
> for it.  Just a reminder: there's always problems somewhere in the
> kernel which can be triggered by various options.  The kernel is not
> one big take it or leave it chunk of code, but many chunks
> selectable by Kconfig with the exception of course of the core.  The
> best we can do wrt to BFQ and other "risky" patches is mark these
> options as EXPERIMENTAL.  I was going to say depend on
> CONFIG_EXPERIMENTAL in Kconfig, but this is deprecated.  See
> scripts/checkpatch.pl
> 
>     "Use of CONFIG_EXPERIMENTAL is deprecated. For alternatives, see
> https://lkml.org/lkml/2012/10/23/580"

It's flat out gone now in the 3.10 kernel release, so if you use it,
your code just will never be enabled.

greg k-h

Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 1022 bytes --]

On Mon, 1 Jul 2013 14:09:57 -0500
Matthew Summers <quantumsummers@gentoo.org> wrote:

> I think the point was well-made by grehkh.

You missed my response to that point.

> If the patchset patches the kernel's core, it doesn't matter what
> CONFIG_* option is set the core kernel code _has_now_been_changed_.
> This is the crux of the argument, I believe. AUFS simply being one
> example of this. I'm sure there are others.

As per my response to that point, this statement is no longer true.

Let me re-iterate it here:

Earlier I mentioned "3) The patch should not affect the build by
default."; if it does, we have to adjust it to not do that, this is
something that can be easily scripted. It's just a matter of embedding
each + block in the diff with a config check and updating the counts.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Greg KH]

On Mon, Jul 01, 2013 at 09:25:42PM +0200, Tom Wijsman wrote:
> On Mon, 1 Jul 2013 14:09:57 -0500
> Matthew Summers <quantumsummers@gentoo.org> wrote:
> > If the patchset patches the kernel's core, it doesn't matter what
> > CONFIG_* option is set the core kernel code _has_now_been_changed_.
> > This is the crux of the argument, I believe. AUFS simply being one
> > example of this. I'm sure there are others.
> 
> As per my response to that point, this statement is no longer true.
> 
> Let me re-iterate it here:
> 
> Earlier I mentioned "3) The patch should not affect the build by
> default."; if it does, we have to adjust it to not do that, this is
> something that can be easily scripted. It's just a matter of embedding
> each + block in the diff with a config check and updating the counts.

Wonderful, now you are maintaining a patch that looks nothing like the
one created by the original developers, nor tested by anyone else other
than gentoo developers.

There's a reason that no other distro does this.

Playing fast-and-loose with kernel patches is a fun thing to do, but
really, why?  Users love doing this type of thing, but the interactions
of different kernel patches with core subsystems is almost always a
non-trivial thing.

I'm not saying not to do this, but consider this a friendly warning that
this is going to be a MAJOR pain to maintain and debug over the
long-run.

But hey, what do I know?  It's not like I've ever done this before and
had the experience of the resulting fall-out that took years to recover
from on user's production systems, causing a number of enterprise Linux
companies to swear that they would never do this type of thing again...

Personally, I wish you luck, it will push the sane users to the
vanilla-sources tree, which I strongly encourage :)

greg "kids, get off my lawn!" k-h

Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 1556 bytes --]

On Mon, 1 Jul 2013 12:23:24 -0700
Greg KH <gregkh@gentoo.org> wrote:

> > Earlier I mentioned "3) The patch should not affect the build by
> > default."; if it does, we have to adjust it to not do that, this is
> > something that can be easily scripted. It's just a matter of
> > embedding each + block in the diff with a config check and updating
> > the counts.
> 
> Look at aufs as a specific example of why you can't do that,
> otherwise, don't you think that the aufs developer(s) wouldn't have
> done so?
> 
> The goal of "don't touch any other kernel code" is a very good one,
> but not always true for these huge out-of-tree kernel patches.
> Usually that is the main reason why these patches aren't merged
> upstream, because those changes are not acceptable.
> 
> So be very careful here, you are messing with things that are rejected
> by upstream.

It sounds to me like some of those developers don't care about
providing an option yet; because they would introduce it as an
afterthought, based on its need and not just because they can.

I don't see why this would be problematic, embedding it in config
checks as well as checking whether the patch introduces non-conditional
code are trivial to implement; why do you think this is not always true?

Thank you for making us aware, feedback on this is nice.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 1355 bytes --]

On Mon, 1 Jul 2013 12:24:36 -0700
Greg KH <gregkh@gentoo.org> wrote:

> On Mon, Jul 01, 2013 at 02:30:51PM -0400, Anthony G. Basile wrote:
> > Tom, you already know my opinion because we discussed it.  I'm all
> > for it.  Just a reminder: there's always problems somewhere in the
> > kernel which can be triggered by various options.  The kernel is not
> > one big take it or leave it chunk of code, but many chunks
> > selectable by Kconfig with the exception of course of the core.  The
> > best we can do wrt to BFQ and other "risky" patches is mark these
> > options as EXPERIMENTAL.  I was going to say depend on
> > CONFIG_EXPERIMENTAL in Kconfig, but this is deprecated.  See
> > scripts/checkpatch.pl
> > 
> >     "Use of CONFIG_EXPERIMENTAL is deprecated. For alternatives, see
> > https://lkml.org/lkml/2012/10/23/580"
> 
> It's flat out gone now in the 3.10 kernel release, so if you use it,
> your code just will never be enabled.

As I replied earlier and tried to make clear in my first post, this
would be our own config variable; naming it CONFIG_GENTOO_EXPERIMENTAL
for that matter as to not confuse people.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 3182 bytes --]

On Mon, 1 Jul 2013 12:33:30 -0700
Greg KH <gregkh@gentoo.org> wrote:

> On Mon, Jul 01, 2013 at 09:25:42PM +0200, Tom Wijsman wrote:
> > On Mon, 1 Jul 2013 14:09:57 -0500
> > Matthew Summers <quantumsummers@gentoo.org> wrote:
> > > If the patchset patches the kernel's core, it doesn't matter what
> > > CONFIG_* option is set the core kernel code
> > > _has_now_been_changed_. This is the crux of the argument, I
> > > believe. AUFS simply being one example of this. I'm sure there
> > > are others.
> > 
> > As per my response to that point, this statement is no longer true.
> > 
> > Let me re-iterate it here:
> > 
> > Earlier I mentioned "3) The patch should not affect the build by
> > default."; if it does, we have to adjust it to not do that, this is
> > something that can be easily scripted. It's just a matter of
> > embedding each + block in the diff with a config check and updating
> > the counts.
> 
> Wonderful, now you are maintaining a patch that looks nothing like the
> one created by the original developers, nor tested by anyone else
> other than gentoo developers.

I can convert the original developer's patch whenever he updates it; or
on top of that, write a script to generate the original patch back.

> Playing fast-and-loose with kernel patches is a fun thing to do, but
> really, why?  Users love doing this type of thing, but the
> interactions of different kernel patches with core subsystems is
> almost always a non-trivial thing.

Our main intent is to provide them and deduplicate work; if users have
a problem with one of the patches and it isn't clear to us, we can
forward them to the authors. Nothing in this proposal inherits that we
must support the patches; especially not, since they are "experimental".

> I'm not saying not to do this, but consider this a friendly warning
> that this is going to be a MAJOR pain to maintain and debug over the
> long-run.

Maybe, maybe not; plan is to do a slow start, congestion avoidance. :)

> But hey, what do I know?  It's not like I've ever done this before and
> had the experience of the resulting fall-out that took years to
> recover from on user's production systems, causing a number of
> enterprise Linux companies to swear that they would never do this
> type of thing again...

I covered this in the conditions as well as the F.A.Q.; feel free to
read about it again, this does not affect them unless they explicitly
want them to enable CONFIG_GENTOO_EXPERIMENTAL which help includes a
warning; thinking it through, we might even suffix -exp to version.

> Personally, I wish you luck, it will push the sane users to the
> vanilla-sources tree, which I strongly encourage :)

I will push them to keep CONFIG_GENTOO_EXPERIMENTAL disabled. :)

> greg "kids, get off my lawn!" k-h

Gentoo (Penguin) and Larry do not necessarily like grass; they might
like ice, fire, water, earth, ... whatever their appetite craves.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Fabio Erculiani]

I believe that end users would benefit more from kernel binary ebuilds
(ebuilds building an actual kernel with an official config), rather
than all this.

-- 
Fabio Erculiani

Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Pacho Ramos]

El lun, 01-07-2013 a las 21:55 +0200, Fabio Erculiani escribió:
> I believe that end users would benefit more from kernel binary ebuilds
> (ebuilds building an actual kernel with an official config), rather
> than all this.
> 

I don't see them exclusionary, look different issues to me :/ (with
completely different solutions I think)

Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Fabio Erculiani]

On Mon, Jul 1, 2013 at 9:59 PM, Pacho Ramos <pacho@gentoo.org> wrote:
> El lun, 01-07-2013 a las 21:55 +0200, Fabio Erculiani escribió:
>> I believe that end users would benefit more from kernel binary ebuilds
>> (ebuilds building an actual kernel with an official config), rather
>> than all this.
>>
>
> I don't see them exclusionary, look different issues to me :/ (with
> completely different solutions I think)

Of course I'm not saying that they're not orthogonal. OTOH I believe
that the complexity of supporting something like this (the original
proposal) is not linear. However, I don't see any problem with people
implementing it btw... it's their time.

>
>



-- 
Fabio Erculiani

Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 1802 bytes --]

On Mon, 1 Jul 2013 21:55:23 +0200
Fabio Erculiani <lxnay@gentoo.org> wrote:

> I believe that end users would benefit more from kernel binary ebuilds
> (ebuilds building an actual kernel with an official config), rather
> than all this.
> 

There's been a start on this, but we're looking at you now; feel free
to communicate with Zero_Chaos about this to clarify why and how you do
certain matters, make sure (just ping me on IRC after a conversation 
with him or so) I can follow along so I can more easily bring this into
the Portage tree; as at least someone on the kernel herd needs to
understand what is going on with all of that

There is a bug filed for this with some work from him.

https://bugs.gentoo.org/show_bug.cgi?id=472906

Separate from this whole thread; we definitely have some thoughts about
this in mind, and I'll give you one cool thought: Having the kernel
build the sources can allow the kernel to automatically rebuild out of
kernel modules; eg. `emerge sys-kernel/gentoo-kernel` and it will
rebuild nvidia-drivers, virtualbox, ... Of course, for this to work we
would need to propagate sub slot rebuilds through virtuals.

We thinked through some possible problems and solutions that appear;
but I would like to keep them out of this ML thread; as to not mix a
thought out idea with one that does some early circles in our minds.

On a side note; I'm currently testing genpatches using an ebuild I
hacked together myself, allows me to test the way eclass does the
patches as well as include some neat local QA checks and testing magic.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Markos Chandras]

On 1 July 2013 20:09, Matthew Summers <quantumsummers@gentoo.org> wrote:
> On Mon, Jul 1, 2013 at 1:56 PM, Tom Wijsman <TomWij@gentoo.org> wrote:
>> On Mon, 1 Jul 2013 19:38:48 +0100
>> Markos Chandras <hwoarang@gentoo.org> wrote:
>>
>>> I certainly don't feel safe anymore running non-upstream code in
>>> production boxes.
>>
>> You don't run it unless you explicitly tick on that you want
>> experimental functionality _as well as_ the optional features in
>> question; as I said earlier on chat, I don't understand your point here.
>>
>> If you don't enable them, genpatches is just like it is before; I'm
>> not sure why the recommendations should change here, especially with
>> vanilla-sources taking a further step away from Gentoo Security and QA.
>>
>
> Tom,
>
> I think the point was well-made by grehkh. If the patchset patches the
> kernel's core, it doesn't matter what CONFIG_* option is set the core
> kernel code _has_now_been_changed_. This is the crux of the argument,
> I believe. AUFS simply being one example of this. I'm sure there are
> others.
>
> --
> Matthew W. Summers
> Gentoo Foundation Inc.
> GPG: 111B C438 35FA EDB5 B5D3 736F 45EE 5DC0 0878 9D46
>

And besides that, I am sure that 98% of our users out there do not
know they run a (heavily?) modified upstream kernel when they emerge
the official/supported gentoo-sources. The transition between the
minimal genpatches to the "new-shiny-feature-full" was made behind the
scenes.
This should have been communicated earlier in time.
If you ask me, I would prefer if you apply all the 3rd-party patches
conditionally (use flag?, maybe a new gentoo-sources-ng ebuild?)
It's really scary to have the BFQ in a stable gentoo-sources ebuild.

--
Regards,
Markos Chandras - Gentoo Linux Developer
http://dev.gentoo.org/~hwoarang

Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Christoph Junghans]

2013/7/1 Fabio Erculiani <lxnay@gentoo.org>:
> I believe that end users would benefit more from kernel binary ebuilds
> (ebuilds building an actual kernel with an official config), rather
> than all this.
+1

The "binary" use flag for sys-kernel/*-sources in Funtoo implements
exactly that.

>
> --
> Fabio Erculiani
>



--
Christoph Junghans
http://dev.gentoo.org/~ottxor/

Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Rick "Zero_Chaos" Farina]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 07/01/2013 03:55 PM, Fabio Erculiani wrote:
> I believe that end users would benefit more from kernel binary ebuilds
> (ebuilds building an actual kernel with an official config), rather
> than all this.
> 
++

- -ZC
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJR0eW1AAoJEKXdFCfdEflKK4EP/jHyQsECDwVc27eiCglQQ5vV
i+FP/cLYkzmNkH8AQwHFq37TVD00XWRiW4vTLu8FrEyu2EIUOze3IwDS9z988rhm
TAOcCkoZPF0rA3NoRmapzccSwXMC6W8wOChXSXvRZx+FqYc2kOJ+Yjc7w552ISg3
DQbJlyS/HvspMfAa/zHQYUb2xDX/1HzH1qIdWCBVHo1RpfkmRoblkQ41e7lKnyfA
8oot1XGNc/8jTL5s1rRtpwt0R8Xw2gIEMjSK93B3QLSAzHVkpxmAQ4uELi5m5R5O
l+07jit+vqw/cS3/2mSP5nJg8SfOIXNSHLexErUGYg92JsebrlqwmhZjYIHF+ldC
h5CbRO30CrcF7kjAjP77gfp0r9tw0AAzOxBqB8cP7/YB62Ee7yAJjXOST7ip7KA5
Bvua4PkgUtyL2fs0orm/Vrkg5ckOVdP92E5oYKFh4gM7UeuDDN2ZJ7fuEKNL9pRC
KyCIf3MkQKw18RxO+ecgJlxVN3aj8Qkt4sdJUCZWpiSpry77gVrnciORpHLguKVh
yVqhDVkd9b3u2jsnlWs6SeqBQtsChx+xI3OK1lA8kEzNArF6yQxhFxrMeyLUu2Iq
LWgBWTuilVIxPAiwDpiWOrXSMjWtjvYGH0vxmfX8wOlvx/L7PA+KEYv+jQeHo59m
B1M0btSM6Ut5RJzeS7K8
=cRF9
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Fabio Erculiani]

On Mon, Jul 1, 2013 at 10:14 PM, Markos Chandras <hwoarang@gentoo.org> wrote:
>

[...]

> It's really scary to have the BFQ in a stable gentoo-sources ebuild.

BFQ is not that scary, it's "just" an iosched (and it's quite easy to
write an iosched), what could possibly go wrong?
Jokes apart, I've been using it in production for almost 2 years now
(can't remember exactly).

>
> --
> Regards,
> Markos Chandras - Gentoo Linux Developer
> http://dev.gentoo.org/~hwoarang
>



-- 
Fabio Erculiani

Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Fabio Erculiani]

On Mon, Jul 1, 2013 at 10:24 PM, Christoph Junghans <ottxor@gentoo.org> wrote:
> 2013/7/1 Fabio Erculiani <lxnay@gentoo.org>:
>> I believe that end users would benefit more from kernel binary ebuilds
>> (ebuilds building an actual kernel with an official config), rather
>> than all this.
> +1
>
> The "binary" use flag for sys-kernel/*-sources in Funtoo implements
> exactly that.

[OT]
And why should you throw kernel sources at people as well? Separate
pkgs is much better. I don't want to have kernel sources on my servers
(for the same reason I don't want to have a compiler, and I've been
able to sort this out as well).
[/OT]

Sorry for the OT.

>
>>
>> --
>> Fabio Erculiani
>>
>
>
>
> --
> Christoph Junghans
> http://dev.gentoo.org/~ottxor/
>



-- 
Fabio Erculiani

Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 1963 bytes --]

On Mon, 1 Jul 2013 21:14:45 +0100
Markos Chandras <hwoarang@gentoo.org> wrote:

> And besides that, I am sure that 98% of our users out there do not
> know they run a (heavily?) modified upstream kernel when they emerge
> the official/supported gentoo-sources.

This point I do understand.

> The transition between the minimal genpatches to the
> "new-shiny-feature-full" was made behind the scenes. This should have
> been communicated earlier in time.

Apart from the BFQ test in ~3 versions, there did not really happen a
transition yet; unless you consider fbcondecor to be part of that, ...

> If you ask me, I would prefer if you apply all the 3rd-party patches
> conditionally (use flag?, maybe a new gentoo-sources-ng ebuild?)
> It's really scary to have the BFQ in a stable gentoo-sources ebuild.

An USE flag sounds like something that would make sense; similarly,
someone suggested in a mail to cover this as well under a new set in
genpatches; currently we do "base" (Linux patches + fixes) and "extra",
we could extend this with "experimental" or something like that.

An "experimental" USE flag definitely makes sense; and documents where
users can find the introduced CONFIG_GENTOO_EXPERIMENTAL kernel option
in its USE flag description, as well as a page documenting the patches
present in the kernel as well as where they can be found in menuconfig.

In the long end, this documentation could be generated from the patches.

So, we introduce USE=-experimental in the eclass which users would have
to enable to get these patches to even apply; that way users can still
see and use a completely untampered kernel, in case they want to want to
apply patches that fail against all the experimental patches.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Anthony G. Basile]

On 07/01/2013 03:23 PM, Greg KH wrote:
> On Mon, Jul 01, 2013 at 08:45:16PM +0200, Tom Wijsman wrote:
>>>> Q: What about my stable server? I really don't want to run this
>>>> stuff!
>>>>
>>>> A: These options would depend on !CONFIG_VANILLA or
>>>> CONFIG_EXPERIMENTAL
>>> What is CONFIG_VANILLA?  I don't see that in the upstream kernel tree
>>> at all.
>>>
>>> CONFIG_EXPERIMENTAL is now gone from upstream, so you are going to
>>> have a problem with this.
>> Earlier I mentioned "2) These feature should depend on a non-vanilla /
>> experimental option." which is an option we would introduce under the
>> Gentoo distribution menu section.
> Distro-specific config options, great :(

I'm not sure what you mean by "distro-specific", but suppose people want 
BFQ? Why can't we have it in gentoo-sources.  It is totally disabled by 
not selecting CONFIG_BFQ.  Selecting it is no different than emerging 
pf-sources with the same other options ported over. By your logic, we 
should not distribut pf-sources either.  The truth of the matter is, 
there are forks of the vanilla kernel out there. Are you suggesting we 
distribute none of them?

NOTE: hardened-sources is its own world.  There is not level of turning 
on/off options that get you back to a vanilla kernel.

>
>>>>     which would be disabled by default, therefore if you keep this
>>>> option the way it is on your stable server; it won't affect you.
>>> Not always true.  Look at aufs as an example.  It patches the core
>>> kernel code in ways that are _not_ accepted upstream yet.  Now you all
>>> are running that modified code, even if you don't want aufs.
>> Earlier I mentioned "3) The patch should not affect the build by
>> default."; if it does, we have to adjust it to not do that, this is
>> something that can be easily scripted. It's just a matter of embedding
>> each + block in the diff with a config check and updating the counts.
> Look at aufs as a specific example of why you can't do that, otherwise,
> don't you think that the aufs developer(s) wouldn't have done so?
>
> The goal of "don't touch any other kernel code" is a very good one, but
> not always true for these huge out-of-tree kernel patches.  Usually that
> is the main reason why these patches aren't merged upstream, because
> those changes are not acceptable.
>
> So be very careful here, you are messing with things that are rejected
> by upstream.
>
> greg k-h
>


-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA

Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Anthony G. Basile]

On 07/01/2013 03:24 PM, Greg KH wrote:
> On Mon, Jul 01, 2013 at 02:30:51PM -0400, Anthony G. Basile wrote:
>> Tom, you already know my opinion because we discussed it.  I'm all
>> for it.  Just a reminder: there's always problems somewhere in the
>> kernel which can be triggered by various options.  The kernel is not
>> one big take it or leave it chunk of code, but many chunks
>> selectable by Kconfig with the exception of course of the core.  The
>> best we can do wrt to BFQ and other "risky" patches is mark these
>> options as EXPERIMENTAL.  I was going to say depend on
>> CONFIG_EXPERIMENTAL in Kconfig, but this is deprecated.  See
>> scripts/checkpatch.pl
>>
>>      "Use of CONFIG_EXPERIMENTAL is deprecated. For alternatives, see
>> https://lkml.org/lkml/2012/10/23/580"
> It's flat out gone now in the 3.10 kernel release, so if you use it,
> your code just will never be enabled.
>
> greg k-h
>
Yeah i noticed that right after sending it.

-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA

Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Greg KH]

On Mon, Jul 01, 2013 at 05:17:07PM -0400, Anthony G. Basile wrote:
> On 07/01/2013 03:23 PM, Greg KH wrote:
> >On Mon, Jul 01, 2013 at 08:45:16PM +0200, Tom Wijsman wrote:
> >>>>Q: What about my stable server? I really don't want to run this
> >>>>stuff!
> >>>>
> >>>>A: These options would depend on !CONFIG_VANILLA or
> >>>>CONFIG_EXPERIMENTAL
> >>>What is CONFIG_VANILLA?  I don't see that in the upstream kernel tree
> >>>at all.
> >>>
> >>>CONFIG_EXPERIMENTAL is now gone from upstream, so you are going to
> >>>have a problem with this.
> >>Earlier I mentioned "2) These feature should depend on a non-vanilla /
> >>experimental option." which is an option we would introduce under the
> >>Gentoo distribution menu section.
> >Distro-specific config options, great :(
> 
> I'm not sure what you mean by "distro-specific",

See later mention of CONFIG_GENTOO_EXPERIMENTAL, that is what I was
referring to.

> but suppose people
> want BFQ? Why can't we have it in gentoo-sources.  It is totally
> disabled by not selecting CONFIG_BFQ.  Selecting it is no different
> than emerging pf-sources with the same other options ported over.

Until you run into a patch that modifies code outside of it's CONFIG_
option, like the aufs example I pointed out.

> By your logic, we should not distribut pf-sources either.  The truth
> of the matter is, there are forks of the vanilla kernel out there. Are
> you suggesting we distribute none of them?

That's a total false argument, the discussion here is about our "main"
gentoo-kernel tree, not one of our many domain-specific kernel versions
that are maintained separately.

> NOTE: hardened-sources is its own world.  There is not level of
> turning on/off options that get you back to a vanilla kernel.

Agreed, which keeps that from being merged into this tree, hopefully :)

thanks,

greg k-h

Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Anthony G. Basile]

On 07/01/2013 04:25 PM, Fabio Erculiani wrote:
> On Mon, Jul 1, 2013 at 10:14 PM, Markos Chandras <hwoarang@gentoo.org> wrote:
> [...]
>
>> It's really scary to have the BFQ in a stable gentoo-sources ebuild.
> BFQ is not that scary, it's "just" an iosched (and it's quite easy to
> write an iosched), what could possibly go wrong?
> Jokes apart, I've been using it in production for almost 2 years now
> (can't remember exactly).

I'm pretty sure I hit a genuine deadlock with it.  I've been trying to 
reproduce with debugging on but nothing yet.

But, having said that:

BFQ [Experimtental]

This introduced an experimental io scheduler.  Have fun with it, but 
don't dare use it in production else we will laugh.

>
>> --
>> Regards,
>> Markos Chandras - Gentoo Linux Developer
>> http://dev.gentoo.org/~hwoarang
>>
>
>


-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA

Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Fabio Erculiani]

On Mon, Jul 1, 2013 at 11:26 PM, Anthony G. Basile <blueness@gentoo.org> wrote:
>
>
> I'm pretty sure I hit a genuine deadlock with it.  I've been trying to
> reproduce with debugging on but nothing yet.
>
> But, having said that:
>
> BFQ [Experimtental]
>
> This introduced an experimental io scheduler.  Have fun with it, but don't
> dare use it in production else we will laugh.

Don't trust outdated documentation.
http://algo.ing.unimo.it/people/paolo/disk_sched/sources.php There is
nothing about it in the latest BFQ patchset.

>
>
>>
>>> --
>>> Regards,
>>> Markos Chandras - Gentoo Linux Developer
>>> http://dev.gentoo.org/~hwoarang
>>>
>>
>>
>
>
> --
> Anthony G. Basile, Ph.D.
> Gentoo Linux Developer [Hardened]
> E-Mail    : blueness@gentoo.org
> GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
> GnuPG ID  : F52D4BBA
>
>



-- 
Fabio Erculiani

Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Anthony G. Basile]

On 07/01/2013 05:24 PM, Greg KH wrote:
> On Mon, Jul 01, 2013 at 05:17:07PM -0400, Anthony G. Basile wrote:
>> On 07/01/2013 03:23 PM, Greg KH wrote:
>>> On Mon, Jul 01, 2013 at 08:45:16PM +0200, Tom Wijsman wrote:
>>>>>> Q: What about my stable server? I really don't want to run this
>>>>>> stuff!
>>>>>>
>>>>>> A: These options would depend on !CONFIG_VANILLA or
>>>>>> CONFIG_EXPERIMENTAL
>>>>> What is CONFIG_VANILLA?  I don't see that in the upstream kernel tree
>>>>> at all.
>>>>>
>>>>> CONFIG_EXPERIMENTAL is now gone from upstream, so you are going to
>>>>> have a problem with this.
>>>> Earlier I mentioned "2) These feature should depend on a non-vanilla /
>>>> experimental option." which is an option we would introduce under the
>>>> Gentoo distribution menu section.
>>> Distro-specific config options, great :(
>> I'm not sure what you mean by "distro-specific",
> See later mention of CONFIG_GENTOO_EXPERIMENTAL, that is what I was
> referring to.
>
>> but suppose people
>> want BFQ? Why can't we have it in gentoo-sources.  It is totally
>> disabled by not selecting CONFIG_BFQ.  Selecting it is no different
>> than emerging pf-sources with the same other options ported over.
> Until you run into a patch that modifies code outside of it's CONFIG_
> option, like the aufs example I pointed out.

Yeah, that's the situation with hardened-sources and then we are in 
agreement.  If its orthogonal to the rest of the kernel, I maintain that 
it can safely be included with the appropriate warnings.

>
>> By your logic, we should not distribut pf-sources either.  The truth
>> of the matter is, there are forks of the vanilla kernel out there. Are
>> you suggesting we distribute none of them?
> That's a total false argument, the discussion here is about our "main"
> gentoo-kernel tree, not one of our many domain-specific kernel versions
> that are maintained separately.

Now I'm confused because gentoo-sources is gentoo specific.  It contains 
stuff that we need in gentoo but other distros do not need, like our 
end-to-end support for certain xattr namespaces.  If you remove these 
then we must either 1) maintain a userland which is not in line with 
other distros or 2) give up on critical features we want in gentoo, like 
markings on elf object in user.pax.flags and certain caps, as well as in 
the future preserving selinux labels through emerge.  Upstream will not 
accept them because of "who needs that crap" and we can't give them up 
without loosing core functionality.  Feel free to review those patches 
but don't ask us to drop them from gentoo-sources because their not in 
upstream.

Only vanilla-sources should be exactly that.  upstream vanilla with 
nothing else.  period.


>
>> NOTE: hardened-sources is its own world.  There is not level of
>> turning on/off options that get you back to a vanilla kernel.
> Agreed, which keeps that from being merged into this tree, hopefully :)

Yeah I think everyone is in agreement with that.  But it also fits my 
point about orthogonality above.


-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA

Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 1289 bytes --]

On Mon, 1 Jul 2013 14:24:54 -0700
Greg KH <gregkh@gentoo.org> wrote:

> > but suppose people
> > want BFQ? Why can't we have it in gentoo-sources.  It is totally
> > disabled by not selecting CONFIG_BFQ.  Selecting it is no different
> > than emerging pf-sources with the same other options ported over.
> 
> Until you run into a patch that modifies code outside of it's CONFIG_
> option, like the aufs example I pointed out.

It would be policy to not add such patches, unless wrapped with config
checks by a script; further more, I discussed USE=-experimental with
mpagano and he found this separation a good idea, we can split this into
a third experimental tarball to not surprise non-Gentoo users as well.

mpagano as well as I stand completely behind that gentoo-sources must
remain usable for production servers; which this USE flag fulfills, as
well as separate from all of this to use live ebuilds in our testing to
avoid surprises that even our non-experimental genpatches could bring.

(For those in #gentoo-kernel, that conversation happened there)

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Anthony G. Basile]

On 07/01/2013 05:30 PM, Fabio Erculiani wrote:
> On Mon, Jul 1, 2013 at 11:26 PM, Anthony G. Basile <blueness@gentoo.org> wrote:
>>
>> I'm pretty sure I hit a genuine deadlock with it.  I've been trying to
>> reproduce with debugging on but nothing yet.
>>
>> But, having said that:
>>
>> BFQ [Experimtental]
>>
>> This introduced an experimental io scheduler.  Have fun with it, but don't
>> dare use it in production else we will laugh.
> Don't trust outdated documentation.
> http://algo.ing.unimo.it/people/paolo/disk_sched/sources.php There is
> nothing about it in the latest BFQ patchset.

No, I hit it.  Not read about it.  Hit it.  I can't be 100% sure, but 
next time it happens, I will be ready with the smoking gun.


-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA

Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Richard Yao]

[-- Attachment #1: Type: text/plain, Size: 6690 bytes --]

On 07/01/2013 03:23 PM, Greg KH wrote:
> On Mon, Jul 01, 2013 at 08:45:16PM +0200, Tom Wijsman wrote:
>>>> Q: What about my stable server? I really don't want to run this
>>>> stuff!
>>>>
>>>> A: These options would depend on !CONFIG_VANILLA or
>>>> CONFIG_EXPERIMENTAL
>>>
>>> What is CONFIG_VANILLA?  I don't see that in the upstream kernel tree
>>> at all.
>>>
>>> CONFIG_EXPERIMENTAL is now gone from upstream, so you are going to
>>> have a problem with this.
>>
>> Earlier I mentioned "2) These feature should depend on a non-vanilla /
>> experimental option." which is an option we would introduce under the
>> Gentoo distribution menu section.
> 
> Distro-specific config options, great :(
> 
>>>>    which would be disabled by default, therefore if you keep this
>>>> option the way it is on your stable server; it won't affect you.
>>>
>>> Not always true.  Look at aufs as an example.  It patches the core
>>> kernel code in ways that are _not_ accepted upstream yet.  Now you all
>>> are running that modified code, even if you don't want aufs.
>>
>> Earlier I mentioned "3) The patch should not affect the build by
>> default."; if it does, we have to adjust it to not do that, this is
>> something that can be easily scripted. It's just a matter of embedding
>> each + block in the diff with a config check and updating the counts.
> 
> Look at aufs as a specific example of why you can't do that, otherwise,
> don't you think that the aufs developer(s) wouldn't have done so?

I am accquainted with the developer of a stackable filesystem developer.
According to what he has told me in person offline, the developers on
the LKML cannot decide on how a stackable filesystem should be
implemented. I was told three different variations on the design that
some people liked and others didn't, which ultimately kept the upstream
kernel from adopting anything. I specifically recall two variations,
which were doing it as part of the VFS and doing it as part of ext4. If
you want to criticize stackable filesystems, would you lay out a
groundwork for getting one implemented upon which people will agree?

> The goal of "don't touch any other kernel code" is a very good one, but
> not always true for these huge out-of-tree kernel patches.  Usually that
> is the main reason why these patches aren't merged upstream, because
> those changes are not acceptable.

I was under the impression that there were several reasons for patches
not being merged upstream:

1. Lack of signed-off
2. Code drop that no one will maintain
3. Subsystem maintainers saying no simply because they do not like
<insert non-technical reason here>.
4. Risk of patent trolls
5. Actual technical reasons

> So be very careful here, you are messing with things that are rejected
> by upstream.
> 
> greg k-h
> 

Only some of the patches were rejected. Others were never submitted. The
PaX/GrSecurity developers prefer their code to stay out-of-tree. As one
of the people hacking on ZFSOnLinux, I prefer that the code be
out-of-tree. That is because fixes for other filesystems are either held
back by a lack of system kernel updates or held hostage by regressions
in newer kernels on certain hardware.

With that said, being in Linus' tree does not make code fall under some
golden standard for quality. There are many significant issues in code
committed to Linus' the kernel, some of which have been problems for
years. Just to name a few:

1. Doing `rm -r /dir` on a directory tree containing millions of inodes
(e.g. ccache) on an ext4 filesystem mounted with discard with the CFQ IO
elevator will cause a system to hang for hours on pre-SATA 3.1 hardware.
This is because TRIM is a non-queued command and is being interleaved
with writes for "fairness". Incidentally, using noop turns a multiple
hour hang into a laggy experience of a few minutes.

2. aio_sync() is unimplemented, which means that there is no sane way
for userland software like QEMU and TGT to be both fast and guarantee
data integrity. A single crash and your guest is corrupted. It would
have been better had AIO never been implemented.

3. dm-crypt will reorder write requests across flushes. That is because
upon seeing a write, it sends it to a work queue to be processed
asynchronously and upon seeing a flush, it immediately processes it. A
single kernel panic or sudden power loss can damage filesystems stored
on it.

4. Under low memory conditions with hundreds of concurrent threads (e.g.
package builds), every thread will enter direct reclaim and there will
be a remarkable drop in system throughput, assuming that the system does
not lockup. There is a fairly substantial amount of time wasted after
one thread finishes direct reclaim in other threads because they will
still be performing direct reclaim afterward.

5. The Linux 3.7 nouveau rewrite broke kexec support. The graphics
hardware will not reinitialize properly.

6. A throttle mechanism introduced for memory cgroups can cause the
system to deadlock whenever it is holding a lock needed for swap and
enters direct reclaim with a significant number of dirty pages.

7. Code has been accepted on multiple occasions that does not compile
and the build failures persist for weeks if not months after Linus' tag.
I sent a patch to fix one failure. It was rejected because I had fixed
code to compile with -Werror, people thought that -Werror should be
removed (and therefore was no reason to fix the warnings) and we went 2
months until someone wrote a patch that people liked to fix it. For a
current example of accepted code failing to build, look here:

https://bugzilla.kernel.org/show_bug.cgi?id=38052

Note that I have not checked Linus' tree to see if that bug is still
current, but the bug itself appears to be open as of this writing.

There are plenty more technical issues, but these are just my pet
peeves. If you want more examples, you could look at the patches people
send you each day and ask yourself how many are things that could have
been caught had people been more careful during review. For instance,
look at the barrier patches that were done around Linux 2.6.30. What
prevented those from being caught by review years earlier?

Being outside Linus' tree is not synonymous with being bad and being bad
is not synonymous with being rejected. It is perfectly reasonable to
think that there are examples of good code outside Linus' tree.
Furthermore, should the kernel kernel choose to engage that out-of-tree
code, my expectation is that its quality will improve as they do testing
and write patches.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 901 bytes --]

Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Richard Yao]

[-- Attachment #1: Type: text/plain, Size: 7237 bytes --]

On 07/01/2013 09:36 PM, Richard Yao wrote:
> On 07/01/2013 03:23 PM, Greg KH wrote:
>> On Mon, Jul 01, 2013 at 08:45:16PM +0200, Tom Wijsman wrote:
>>>>> Q: What about my stable server? I really don't want to run this
>>>>> stuff!
>>>>>
>>>>> A: These options would depend on !CONFIG_VANILLA or
>>>>> CONFIG_EXPERIMENTAL
>>>>
>>>> What is CONFIG_VANILLA?  I don't see that in the upstream kernel tree
>>>> at all.
>>>>
>>>> CONFIG_EXPERIMENTAL is now gone from upstream, so you are going to
>>>> have a problem with this.
>>>
>>> Earlier I mentioned "2) These feature should depend on a non-vanilla /
>>> experimental option." which is an option we would introduce under the
>>> Gentoo distribution menu section.
>>
>> Distro-specific config options, great :(
>>
>>>>>    which would be disabled by default, therefore if you keep this
>>>>> option the way it is on your stable server; it won't affect you.
>>>>
>>>> Not always true.  Look at aufs as an example.  It patches the core
>>>> kernel code in ways that are _not_ accepted upstream yet.  Now you all
>>>> are running that modified code, even if you don't want aufs.
>>>
>>> Earlier I mentioned "3) The patch should not affect the build by
>>> default."; if it does, we have to adjust it to not do that, this is
>>> something that can be easily scripted. It's just a matter of embedding
>>> each + block in the diff with a config check and updating the counts.
>>
>> Look at aufs as a specific example of why you can't do that, otherwise,
>> don't you think that the aufs developer(s) wouldn't have done so?
> 
> I am accquainted with the developer of a stackable filesystem developer.

I should probably proofread multiple times before I send emails. Anyway,
that should have been:

> I am acquainted with the developer of a stackable filesystem.

> According to what he has told me in person offline, the developers on
> the LKML cannot decide on how a stackable filesystem should be
> implemented. I was told three different variations on the design that
> some people liked and others didn't, which ultimately kept the upstream
> kernel from adopting anything. I specifically recall two variations,
> which were doing it as part of the VFS and doing it as part of ext4. If
> you want to criticize stackable filesystems, would you lay out a
> groundwork for getting one implemented upon which people will agree?
> 
>> The goal of "don't touch any other kernel code" is a very good one, but
>> not always true for these huge out-of-tree kernel patches.  Usually that
>> is the main reason why these patches aren't merged upstream, because
>> those changes are not acceptable.
> 
> I was under the impression that there were several reasons for patches
> not being merged upstream:
> 
> 1. Lack of signed-off
> 2. Code drop that no one will maintain
> 3. Subsystem maintainers saying no simply because they do not like
> <insert non-technical reason here>.
> 4. Risk of patent trolls
> 5. Actual technical reasons
> 
>> So be very careful here, you are messing with things that are rejected
>> by upstream.
>>
>> greg k-h
>>
> 
> Only some of the patches were rejected. Others were never submitted. The
> PaX/GrSecurity developers prefer their code to stay out-of-tree. As one
> of the people hacking on ZFSOnLinux, I prefer that the code be
> out-of-tree. That is because fixes for other filesystems are either held
> back by a lack of system kernel updates or held hostage by regressions
> in newer kernels on certain hardware.
> 
> With that said, being in Linus' tree does not make code fall under some
> golden standard for quality. There are many significant issues in code
> committed to Linus' the kernel, some of which have been problems for
> years. Just to name a few:
> 
> 1. Doing `rm -r /dir` on a directory tree containing millions of inodes
> (e.g. ccache) on an ext4 filesystem mounted with discard with the CFQ IO
> elevator will cause a system to hang for hours on pre-SATA 3.1 hardware.
> This is because TRIM is a non-queued command and is being interleaved
> with writes for "fairness". Incidentally, using noop turns a multiple
> hour hang into a laggy experience of a few minutes.
> 
> 2. aio_sync() is unimplemented, which means that there is no sane way
> for userland software like QEMU and TGT to be both fast and guarantee
> data integrity. A single crash and your guest is corrupted. It would
> have been better had AIO never been implemented.
> 
> 3. dm-crypt will reorder write requests across flushes. That is because
> upon seeing a write, it sends it to a work queue to be processed
> asynchronously and upon seeing a flush, it immediately processes it. A
> single kernel panic or sudden power loss can damage filesystems stored
> on it.
> 
> 4. Under low memory conditions with hundreds of concurrent threads (e.g.
> package builds), every thread will enter direct reclaim and there will
> be a remarkable drop in system throughput, assuming that the system does
> not lockup. There is a fairly substantial amount of time wasted after
> one thread finishes direct reclaim in other threads because they will
> still be performing direct reclaim afterward.
> 
> 5. The Linux 3.7 nouveau rewrite broke kexec support. The graphics
> hardware will not reinitialize properly.
> 
> 6. A throttle mechanism introduced for memory cgroups can cause the
> system to deadlock whenever it is holding a lock needed for swap and
> enters direct reclaim with a significant number of dirty pages.
> 
> 7. Code has been accepted on multiple occasions that does not compile
> and the build failures persist for weeks if not months after Linus' tag.
> I sent a patch to fix one failure. It was rejected because I had fixed
> code to compile with -Werror, people thought that -Werror should be
> removed (and therefore was no reason to fix the warnings) and we went 2
> months until someone wrote a patch that people liked to fix it. For a
> current example of accepted code failing to build, look here:
> 
> https://bugzilla.kernel.org/show_bug.cgi?id=38052
> 
> Note that I have not checked Linus' tree to see if that bug is still
> current, but the bug itself appears to be open as of this writing.
> 
> There are plenty more technical issues, but these are just my pet
> peeves. If you want more examples, you could look at the patches people
> send you each day and ask yourself how many are things that could have
> been caught had people been more careful during review. For instance,
> look at the barrier patches that were done around Linux 2.6.30. What
> prevented those from being caught by review years earlier?
> 
> Being outside Linus' tree is not synonymous with being bad and being bad
> is not synonymous with being rejected. It is perfectly reasonable to
> think that there are examples of good code outside Linus' tree.
> Furthermore, should the kernel kernel choose to engage that out-of-tree

That should have been:

> Furthermore, should the kernel team choose to engage that out-of-tree

> code, my expectation is that its quality will improve as they do testing
> and write patches.
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 901 bytes --]

Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Greg KH]

On Mon, Jul 01, 2013 at 09:36:21PM -0400, Richard Yao wrote:
> On 07/01/2013 03:23 PM, Greg KH wrote:
> > On Mon, Jul 01, 2013 at 08:45:16PM +0200, Tom Wijsman wrote:
> >>>> Q: What about my stable server? I really don't want to run this
> >>>> stuff!
> >>>>
> >>>> A: These options would depend on !CONFIG_VANILLA or
> >>>> CONFIG_EXPERIMENTAL
> >>>
> >>> What is CONFIG_VANILLA?  I don't see that in the upstream kernel tree
> >>> at all.
> >>>
> >>> CONFIG_EXPERIMENTAL is now gone from upstream, so you are going to
> >>> have a problem with this.
> >>
> >> Earlier I mentioned "2) These feature should depend on a non-vanilla /
> >> experimental option." which is an option we would introduce under the
> >> Gentoo distribution menu section.
> > 
> > Distro-specific config options, great :(
> > 
> >>>>    which would be disabled by default, therefore if you keep this
> >>>> option the way it is on your stable server; it won't affect you.
> >>>
> >>> Not always true.  Look at aufs as an example.  It patches the core
> >>> kernel code in ways that are _not_ accepted upstream yet.  Now you all
> >>> are running that modified code, even if you don't want aufs.
> >>
> >> Earlier I mentioned "3) The patch should not affect the build by
> >> default."; if it does, we have to adjust it to not do that, this is
> >> something that can be easily scripted. It's just a matter of embedding
> >> each + block in the diff with a config check and updating the counts.
> > 
> > Look at aufs as a specific example of why you can't do that, otherwise,
> > don't you think that the aufs developer(s) wouldn't have done so?
> 
> I am accquainted with the developer of a stackable filesystem developer.
> According to what he has told me in person offline, the developers on
> the LKML cannot decide on how a stackable filesystem should be
> implemented. I was told three different variations on the design that
> some people liked and others didn't, which ultimately kept the upstream
> kernel from adopting anything. I specifically recall two variations,
> which were doing it as part of the VFS and doing it as part of ext4. If
> you want to criticize stackable filesystems, would you lay out a
> groundwork for getting one implemented upon which people will agree?

I was not criticising stackable filesystems at all, nor the aufs code,
which I personally run on some of my own systems.

I agree that the upstream kernel development of stackable filesystems
has been all over the place, with seemingly not much guidance on how to
get things merged properly.  Being that I am not part of the subset of
the kernel development community, I am in no position to lay any
groundwork out at all.

And note, this is totally off-topic from this thread.

My main point is that if Gentoo wants to start maintaining out-of-tree
kernel patches, and modifying them, the maintenance nightmare is going
to be huge.  Been there, done that, have way too many t-shirts
commemorating it, and never want to do it again.

> > The goal of "don't touch any other kernel code" is a very good one, but
> > not always true for these huge out-of-tree kernel patches.  Usually that
> > is the main reason why these patches aren't merged upstream, because
> > those changes are not acceptable.
> 
> I was under the impression that there were several reasons for patches
> not being merged upstream:
> 
> 1. Lack of signed-off

No distro will accept code that does not have a signed-off-by line,
otherwise they might get into trouble, as that implies the patch is not
properly licensed, right?

> 2. Code drop that no one will maintain

Agreed.

> 3. Subsystem maintainers saying no simply because they do not like
> <insert non-technical reason here>.

That is very rare and usually the subsystem maintainer can be poked to
resolve this.

> 4. Risk of patent trolls

I know of no kernel patches outside of the tree because of this, do you?

> 5. Actual technical reasons

That's the majority of the reasons patches aren't accepted.

> Only some of the patches were rejected. Others were never submitted. The
> PaX/GrSecurity developers prefer their code to stay out-of-tree. As one
> of the people hacking on ZFSOnLinux, I prefer that the code be
> out-of-tree.

There's also a small legal issue with the zfs code that might prevent it
from being merged :)

> That is because fixes for other filesystems are either held back by a
> lack of system kernel updates or held hostage by regressions in newer
> kernels on certain hardware.

I have never heard this being a reason for keeping code upstream, what
do you mean by it?

> With that said, being in Linus' tree does not make code fall under some
> golden standard for quality. There are many significant issues in code
> committed to Linus' the kernel, some of which have been problems for
> years. Just to name a few:

<bug reports snipped>

The fact that bugs happen in 16 million lines of kernel code, moving at
a rate that is orders of magnitude faster than any other software
development project, is not news to anyone, it's a fact of life.

The fact that we fix them when they are found is the important thing,
don't you agree?

And of course, any recommendations on how we can find these bugs before
they are merged is _always_ accepted.

> Being outside Linus' tree is not synonymous with being bad and being bad
> is not synonymous with being rejected. It is perfectly reasonable to
> think that there are examples of good code outside Linus' tree.

Being outside of Linus's tree puts the maintenance and support burden on
you, and you are working against a body of code that is going faster
this week than it did last week, so your work is constantly increasing.
That is why it should be merged, to save yourself work.

> Furthermore, should the kernel kernel choose to engage that out-of-tree
> code, my expectation is that its quality will improve as they do testing
> and write patches.

What do you mean by this?

thanks,

greg k-h

Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Richard Yao]

[-- Attachment #1: Type: text/plain, Size: 8460 bytes --]

On 07/01/2013 09:56 PM, Greg KH wrote:> On Mon, Jul 01, 2013 at
09:36:21PM -0400, Richard Yao wrote:
>> On 07/01/2013 03:23 PM, Greg KH wrote:
>>> On Mon, Jul 01, 2013 at 08:45:16PM +0200, Tom Wijsman wrote:
>>>>>> Q: What about my stable server? I really don't want to run this
>>>>>> stuff!
>>>>>>
>>>>>> A: These options would depend on !CONFIG_VANILLA or
>>>>>> CONFIG_EXPERIMENTAL
>>>>>
>>>>> What is CONFIG_VANILLA?  I don't see that in the upstream kernel tree
>>>>> at all.
>>>>>
>>>>> CONFIG_EXPERIMENTAL is now gone from upstream, so you are going to
>>>>> have a problem with this.
>>>>
>>>> Earlier I mentioned "2) These feature should depend on a non-vanilla /
>>>> experimental option." which is an option we would introduce under the
>>>> Gentoo distribution menu section.
>>>
>>> Distro-specific config options, great :(
>>>
>>>>>>    which would be disabled by default, therefore if you keep this
>>>>>> option the way it is on your stable server; it won't affect you.
>>>>>
>>>>> Not always true.  Look at aufs as an example.  It patches the core
>>>>> kernel code in ways that are _not_ accepted upstream yet.  Now you all
>>>>> are running that modified code, even if you don't want aufs.
>>>>
>>>> Earlier I mentioned "3) The patch should not affect the build by
>>>> default."; if it does, we have to adjust it to not do that, this is
>>>> something that can be easily scripted. It's just a matter of embedding
>>>> each + block in the diff with a config check and updating the counts.
>>>
>>> Look at aufs as a specific example of why you can't do that, otherwise,
>>> don't you think that the aufs developer(s) wouldn't have done so?
>>
>> I am accquainted with the developer of a stackable filesystem developer.
>> According to what he has told me in person offline, the developers on
>> the LKML cannot decide on how a stackable filesystem should be
>> implemented. I was told three different variations on the design that
>> some people liked and others didn't, which ultimately kept the upstream
>> kernel from adopting anything. I specifically recall two variations,
>> which were doing it as part of the VFS and doing it as part of ext4. If
>> you want to criticize stackable filesystems, would you lay out a
>> groundwork for getting one implemented upon which people will agree?
>
> I was not criticising stackable filesystems at all, nor the aufs code,
> which I personally run on some of my own systems.
>
> I agree that the upstream kernel development of stackable filesystems
> has been all over the place, with seemingly not much guidance on how to
> get things merged properly.  Being that I am not part of the subset of
> the kernel development community, I am in no position to lay any
> groundwork out at all.
>
> And note, this is totally off-topic from this thread.
>
> My main point is that if Gentoo wants to start maintaining out-of-tree
> kernel patches, and modifying them, the maintenance nightmare is going
> to be huge.  Been there, done that, have way too many t-shirts
> commemorating it, and never want to do it again.
>
>>> The goal of "don't touch any other kernel code" is a very good one, but
>>> not always true for these huge out-of-tree kernel patches.  Usually that
>>> is the main reason why these patches aren't merged upstream, because
>>> those changes are not acceptable.
>>
>> I was under the impression that there were several reasons for patches
>> not being merged upstream:
>>
>> 1. Lack of signed-off
>
> No distro will accept code that does not have a signed-off-by line,
> otherwise they might get into trouble, as that implies the patch is not
> properly licensed, right?

That is wrong. Signed-off is an affirmation that the code is under the
license, but the absence of signed-off does not imply that the code is
not under the license. For example, I doubt you will receive signed-off
for PaX/GrSecurity, but is there any doubt that code is under the GPL?

To give another example, I doubt that you will receive signed off for
code from BSD UNIX, but is there any doubt that code is under the
3-clause BSD license?

>> 2. Code drop that no one will maintain
>
> Agreed.
>
>> 3. Subsystem maintainers saying no simply because they do not like
>> <insert non-technical reason here>.
>
> That is very rare and usually the subsystem maintainer can be poked to
> resolve this.

Past conversations with you have made me reluctant to try, but the next
time I see something like this, I will send you an email.

>> 4. Risk of patent trolls
>
> I know of no kernel patches outside of the tree because of this, do you?

There is compatibility code for DOS long filenames in fat that is no
longer in-tree because of this.

>> 5. Actual technical reasons
>
> That's the majority of the reasons patches aren't accepted.

Not all patches that aren't accepted are submitted to be rejected.

>> Only some of the patches were rejected. Others were never submitted. The
>> PaX/GrSecurity developers prefer their code to stay out-of-tree. As one
>> of the people hacking on ZFSOnLinux, I prefer that the code be
>> out-of-tree.
>
> There's also a small legal issue with the zfs code that might prevent it
> from being merged :)

To summarize an email from Linus, the reason people who want ZFS in the
main tree cannot have that is his signed-off policy. At least one of the
copyright holders has no interest in providing it.

>> That is because fixes for other filesystems are either held back by a
>> lack of system kernel updates or held hostage by regressions in newer
>> kernels on certain hardware.
>
> I have never heard this being a reason for keeping code upstream, what
> do you mean by it?

This is an issue that end users tend to encounter where changes in a
newer kernel break stuff that they need. One example is nouveau kexec
support. Another is that the nouveau in the first two RCs of Linux 3.7
(if I recall) broke my display completely.

>> With that said, being in Linus' tree does not make code fall under some
>> golden standard for quality. There are many significant issues in code
>> committed to Linus' the kernel, some of which have been problems for
>> years. Just to name a few:
>
> <bug reports snipped>
>
> The fact that bugs happen in 16 million lines of kernel code, moving at
> a rate that is orders of magnitude faster than any other software
> development project, is not news to anyone, it's a fact of life.
>
> The fact that we fix them when they are found is the important thing,
> don't you agree?
>
> And of course, any recommendations on how we can find these bugs before
> they are merged is _always_ accepted.

My point is that code in Linus' tree is not free from error. It has bugs
too and is by no means always better than out-of-tree code.

>> Being outside Linus' tree is not synonymous with being bad and being bad
>> is not synonymous with being rejected. It is perfectly reasonable to
>> think that there are examples of good code outside Linus' tree.
>
> Being outside of Linus's tree puts the maintenance and support burden on
> you, and you are working against a body of code that is going faster
> this week than it did last week, so your work is constantly increasing.
> That is why it should be merged, to save yourself work.

Then you get to be a kernel subsystem maintainer and you get to check
which version of the code was included in the kernel for each bug
report, rather than just having one codebase.

>> Furthermore, should the kernel kernel choose to engage that out-of-tree
>> code, my expectation is that its quality will improve as they do testing
>> and write patches.
>
> What do you mean by this?
>
> thanks,
>
> greg k-h
>

I started using ZFSOnLinux last year. Since then, I have written several
dozen patches. I wrote support for multiple new kernels, wrote support
for PaX/GrSecurity, diagnosed why swap on zvols was broken and wrote the
initial patches to fix it, collaborated on kernel preemption support,
fixed several kernel panics, etcetera. Nearly all of that work has been
merged by ZFSOnLinux upstream, has been rejected under mutual agreement
or is pending under review. If Gentoo developers begin maintaining other
out-of-linus'-tree kernel code in Gentoo, I imagine that similar things
would happen with that code as well.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 901 bytes --]

Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Richard Yao]

[-- Attachment #1: Type: text/plain, Size: 485 bytes --]

>> Furthermore, should the kernel kernel choose to engage that out-of-tree
>> code, my expectation is that its quality will improve as they do testing
>> and write patches.
> 
> What do you mean by this?

I probably should have clarified that there was a typo in that. I meant
"the kernel team", not "the kernel kernel". I was referring to the
Gentoo kernel team. I sent an email to the list with the correction, but
I neglected to include you on the CC list by mistake.



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 901 bytes --]

Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Richard Yao]

[-- Attachment #1: Type: text/plain, Size: 1015 bytes --]

On 07/01/2013 11:29 PM, Richard Yao wrote:
> On 07/01/2013 09:56 PM, Greg KH wrote:> On Mon, Jul 01, 2013 at
> 09:36:21PM -0400, Richard Yao wrote:
>>> That is because fixes for other filesystems are either held back by a
>>> lack of system kernel updates or held hostage by regressions in newer
>>> kernels on certain hardware.
>>
>> I have never heard this being a reason for keeping code upstream, what
>> do you mean by it?
> 
> This is an issue that end users tend to encounter where changes in a
> newer kernel break stuff that they need. One example is nouveau kexec
> support. Another is that the nouveau in the first two RCs of Linux 3.7
> (if I recall) broke my display completely.

I probably should clarify that this issue keeps some users from
obtaining bug fixes in key components (e.g. their filesystem). That is
why I prefer the situation where code lives out-of-tree and works on a
variety of kernels over the situation where it is bundled with the
kernel for in-tree builds.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 901 bytes --]

Re: [gentoo-dev] Proper distribution integration of kernel *-sources, patches and configuration.

[Sergei Trofimovich]

[-- Attachment #1: Type: text/plain, Size: 810 bytes --]

[ sorry, a lot to quote ]

What upstream do you plan to report bugs when user possibly mixes
all of it in one mess? Each of those is known to be unstable. The mix
is just a disaster.

Is gentoo's kernel team able to resolve user's OOpsen?

> ### ... and configuration. ###
> 
> This problem is not only visible for patches, but also in the config.

Insane :]

> Meet CONFIG_DEVTMPFS; forget to enable it, greet a failing boot. We're
> telling users to enable it in some places, in the handbook it's a single
> line you must read, on the Wiki it's kind of missing unless you are
> luckily on the right page, on the Quick Install book it is missing too.

Forbid users install udev to ROOT=/ if running kernel does not support devtmpfs
(easy to check by /proc/filesystems)

-- 

  Sergei

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Proper distribution integration of kernel *-sources, patches and configuration.

[Fabio Erculiani]

On Tue, Jul 2, 2013 at 9:36 AM, Sergei Trofimovich <slyfox@gentoo.org> wrote:
> [ sorry, a lot to quote ]
>
> What upstream do you plan to report bugs when user possibly mixes
> all of it in one mess? Each of those is known to be unstable. The mix
> is just a disaster.
>
> Is gentoo's kernel team able to resolve user's OOpsen?
>
>> ### ... and configuration. ###
>>
>> This problem is not only visible for patches, but also in the config.
>
> Insane :]
>
>> Meet CONFIG_DEVTMPFS; forget to enable it, greet a failing boot. We're
>> telling users to enable it in some places, in the handbook it's a single
>> line you must read, on the Wiki it's kind of missing unless you are
>> luckily on the right page, on the Quick Install book it is missing too.
>
> Forbid users install udev to ROOT=/ if running kernel does not support devtmpfs
> (easy to check by /proc/filesystems)

No. As explained multiple times, this check is not reliable and
doesn't work (chroot, binpkgs, containers without kernel, and so
on...).
Making sure that the user doesn't build an unbootable kernel is the way to go.

>
> --
>
>   Sergei



-- 
Fabio Erculiani

gentoo-checkconf script Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Michael Weber]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 07/01/2013 11:53 PM, Anthony G. Basile wrote:
> Now I'm confused because gentoo-sources is gentoo specific.  It
> contains stuff that we need in gentoo but other distros do not
> need, like our end-to-end support for certain xattr namespaces.  If
> you remove these then we must either 1) maintain a userland which
> is not in line with other distros or 2) give up on critical
> features we want in gentoo, like markings on elf object in
> user.pax.flags and certain caps, as well as in the future
> preserving selinux labels through emerge.  Upstream will not accept
> them because of "who needs that crap" and we can't give them up 
> without loosing core functionality.  Feel free to review those
> patches but don't ask us to drop them from gentoo-sources because
> their not in upstream.

What about a check-kernel-config-for-gentoo-compliance script for
starterts?

I manage a handfull of kernel configs over some years (laptop vs.
server, graphics, firewalling capabilities) and was always tempted to
write an script to check if the config meets a certain set of
requirements. I think of "xattr", "selinux", "gentoo-boot" and so on,
that can be expanded by users demand, like, "CONFIG_CMDLINE should
include" and "CONFIG_DEFAULT_HOSTNAME=x" and "all iptables target on".

A additional make target in gentoo-sources could the warn about any
missing feature, and ask for "yes" or wait some seconds.
(I remember reaging some funny note about my kernel supporting x32 but
by userland not, like that kernel build would run on that userland)

==> Merging a certain source does always imply to run it on that system.

(diff-ing configs is really nasty since sub*module=N drops lines from
the config)

(and i got lazy on reading all the added features in subsystems [1])


   Michael "I can live with a lot of things, as long as I can
configure/compile/update my kernel and the out-of-tree drivers when i
want" Weber

[1]
http://3.bp.blogspot.com/_rtOXMZlMTkg/RZWVjP3f49I/AAAAAAAAADs/YpHlSwXpiUg/s400/drinking_bird.jpg


- -- 
Michael Weber
Gentoo Developer
web: https://xmw.de/
mailto: Michael Weber <xmw@gentoo.org>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.20 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iF4EAREIAAYFAlHSj+IACgkQknrdDGLu8JD65AD+NHyGeFNQw4GceLp0g9ypik5j
NzoEwKYztMCOwKcjbO4A/A1e/KQv4DabFoZA41kdPBH8DMOITWL7Jb3OHqewwpPL
=OOdc
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Proper distribution integration of kernel *-sources, patches and configuration.

[Michael Weber]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 07/02/2013 10:21 AM, Fabio Erculiani wrote:
> On Tue, Jul 2, 2013 at 9:36 AM, Sergei Trofimovich
> <slyfox@gentoo.org> wrote:
>> Forbid users install udev to ROOT=/ if running kernel does not
>> support devtmpfs (easy to check by /proc/filesystems)
> 
> No. As explained multiple times, this check is not reliable and 
> doesn't work (chroot, binpkgs, containers without kernel, and so 
> on...). Making sure that the user doesn't build an unbootable
> kernel is the way to go.
Nah, as I just wrote on another sub-thread of this, this assumption is
as-bad-as the other one.

Imagine users to cross-compile kernels for other
userlands/hardwares/whatever, like compile a gentoo-sources on a fast
x86 for a ARM RaspberryPi with a Debian userland.

Two thoughts, out package manager does not handle reboots (and I like
it), and kernel builds as well as installs/bootloader are a bit out of
this scope.

- -- 
Michael Weber
Gentoo Developer
web: https://xmw.de/
mailto: Michael Weber <xmw@gentoo.org>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.20 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iF4EAREIAAYFAlHSkVgACgkQknrdDGLu8JDn5AD/bgSoPzVvyajlh8K2vDGWiwhR
Nbjxr8rwPvTl/RW6LE8A/j7QUy1QbSkqOXSmkuXbUNbplSRhVamwDQ+XFMK1jgMI
=HjGE
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Proper distribution integration of kernel *-sources, patches and configuration.

[Michael Weber]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Sorry for that GPG-traffic and me needing 3 mails to formulate my
thoughts.

I am not against providing a gentoo-sources binary kernel package with
a sound setup to suite gentoo-needs and `make allmodconfig`
to give the less enthusiastic `make nconfig` button pushers something
to boot their fresh installs with -- and never move away from it.

As long as it's slot-ed and proper CONFIG_LOCALVERSION'ed* aka having
more than one version installed to do downgrades and tests.

(*) Meaning, the modules dir should change with every release to avoid
loading outdated out of tree modules, and the kernel files in /boot
should not clash.

Suggesting to run `grub2-mkconfig -o /boot/grub/grub.cfg` would be
nice, too.

*enough*

- -- 
Michael Weber
Gentoo Developer
web: https://xmw.de/
mailto: Michael Weber <xmw@gentoo.org>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.20 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iF4EAREIAAYFAlHSlOcACgkQknrdDGLu8JCRhAEAhpl7vsTVmo9CPARie4kWznTd
cam1MmdcRejlpkiOeiUA/0Nn0Q7cERzPyXkiwFO5bbprB9OUMFNrXlygp5Hb8KVn
=UjWq
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Proper distribution integration of kernel *-sources, patches and configuration.

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 2224 bytes --]

On Tue, 2 Jul 2013 10:36:34 +0300
Sergei Trofimovich <slyfox@gentoo.org> wrote:

> What upstream do you plan to report bugs when user possibly mixes
> all of it in one mess? Each of those is known to be unstable. The mix
> is just a disaster.

They used to do this to us and to kernel upstream before. Why? Because
we are unaware. The users just applied their patches, forget about them
when they experience problems; then they come and file a bug without
us knowing they do use these. Thus we'll ask for a .config and any
separate patches the user applied from now on.

The mix is just a disaster, but that's up to the user whether he wants
to go for such disaster; the patches are guarded using an experimental
USE flag and menu config option, if you enable both, you can't expect
everything to work fine. Well, maybe it does, if you sparingly enable
one or two patches but not all of them.

> Is gentoo's kernel team able to resolve user's OOpsen?

When patches are applied, we can ask them to try to reproduce it without
patches; unless, it is clear from the BUG that an experimental patch is
at cause. If they consistently get a BUG with a patch, we can send these
users to report this at the patch author; basically we act as a filter
before sending these reports upstream. And yes, we do try to resolve
some of these on our own.

> > Meet CONFIG_DEVTMPFS; forget to enable it, greet a failing boot.
> > We're telling users to enable it in some places, in the handbook
> > it's a single line you must read, on the Wiki it's kind of missing
> > unless you are luckily on the right page, on the Quick Install book
> > it is missing too.
> 
> Forbid users install udev to ROOT=/ if running kernel does not
> support devtmpfs (easy to check by /proc/filesystems)

But udev comes as part of stage 3 tarball; so, we would have to change
the whole way of installing a system. I think opting to enable it by
default for Gentoo kernel sources is a better / less invasive approach.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Proper distribution integration of kernel *-sources, patches and configuration.

[Sergei Trofimovich]

[-- Attachment #1: Type: text/plain, Size: 2308 bytes --]

On Tue, 2 Jul 2013 10:21:53 +0200
Fabio Erculiani <lxnay@gentoo.org> wrote:

> On Tue, Jul 2, 2013 at 9:36 AM, Sergei Trofimovich <slyfox@gentoo.org> wrote:
> > [ sorry, a lot to quote ]
> >
> > What upstream do you plan to report bugs when user possibly mixes
> > all of it in one mess? Each of those is known to be unstable. The mix
> > is just a disaster.
> >
> > Is gentoo's kernel team able to resolve user's OOpsen?
> >
> >> ### ... and configuration. ###
> >>
> >> This problem is not only visible for patches, but also in the config.
> >
> > Insane :]
> >
> >> Meet CONFIG_DEVTMPFS; forget to enable it, greet a failing boot. We're
> >> telling users to enable it in some places, in the handbook it's a single
> >> line you must read, on the Wiki it's kind of missing unless you are
> >> luckily on the right page, on the Quick Install book it is missing too.
> >
> > Forbid users install udev to ROOT=/ if running kernel does not support devtmpfs
> > (easy to check by /proc/filesystems)
> 
> No. As explained multiple times, this check is not reliable and
> doesn't work (chroot, binpkgs, containers without kernel, and so
> on...).
> Making sure that the user doesn't build an unbootable kernel is the way to go.

udev's case:
It's _not_ the kernel which upgrade breaks user's system. Why do you try to "fix" it?
If you want to save user's box - make check at pre-install time. Otherwise it will break.

containers, etc. case:
If you build gentoo(and, what really matters _install_) in container
- you can run gentoo in container. Yes, with it's stock broken kernel.

Who cares what you have built if the host is xen-3.0.6 with CONFIG_DEVTMPFS=n?
If you really want to proceed unsafe action you need to do it explicitely.

kernel's case:
CONFIG_DEVTMPFS needs 'default yes', right? One-liner to gentoo-sources.
Why 'hide' it? It's very counterintuitive.

What will you do with all-those-required-to-boot-properly
  - root filesystem
  - disk controller
  - USB keyboard drivers
 ?
Include them all unless CONFIG_DONT_REMOVER_OR_WONT_BOOT
option?

I'm afraid I don't see how it's a solution. 
Suppose, tomorrow's udev will require CONFIG_foo, and glibc will require
CONFIG_bar. How will you save user with your mechanism?

-- 

  Sergei

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Greg KH]

Almost all of this portion of the thread is off-topic for gentoo-dev, so
I'll leave it alone, and will be more than willing to take it up
somewhere else it is on-topic for, like linux-kernel, if you want to.

But, there is one thing I do want to ask/comment on, as it is relevant
to users of Gentoo:

On Mon, Jul 01, 2013 at 11:29:39PM -0400, Richard Yao wrote:
> >> 4. Risk of patent trolls
> >
> > I know of no kernel patches outside of the tree because of this, do you?
> 
> There is compatibility code for DOS long filenames in fat that is no
> longer in-tree because of this.

I remember the conversations that a number of us had a few years ago
about this potential issue, but do not see any in-kernel changes that
were ever made because of that.  Could you point me at the changes that
were made that were accepted into the kernel.org tree?

thanks,

gerg k-h

Re: [gentoo-dev] Proper distribution integration of kernel *-sources, patches and configuration.

[Tomáš Pružina]

[-- Attachment #1: Type: text/plain, Size: 10399 bytes --]

This reminds me of: http://imgs.xkcd.com/comics/standards.png
It sure sounds nicely, however I would not want to be the guy who maintains
the whole mess of (often) incompatible patchsets.
Given the fact that some patches lag 1-3 stable versions behind Linuses
tree (grsec used by hardened for example),
this might be better idea on paper than in reality.

If you feel up for this, go for it. Be ready to meet resistance though.

// I personally maintain my own tree, based on stable from gregkh and few
patches

On Mon, Jul 1, 2013 at 4:41 PM, Tom Wijsman <TomWij@gentoo.org> wrote:

> Hello
>
>
> Please reply to gentoo-dev in case ML daemon changes Reply-To.
>
>
> ### TL; DR ###
>
> By introducing feature patches which menu options are disabled by
> default to genpatches, we can deduplicate *-sources maintainers as well
> as large groups of users work. By introducing a distribution section
> in the menuconfig, we can provide options that enable minimal Gentoo
> requirements by default (DEVTMPFS, making Gentoo systems bootable since
> an udev release a long time ago) and other distribution stuff.
>
>
> ### Proper distribution integration of kernel *-sources, patches ... ###
>
> Gentoo is a distribution; but what is a distribution that doesn't
> properly integrate what it provides, but instead expects its users to
> do so, needlessly duplicating work amongst its maintainers and users.
>
> Let's say I want genpatches, aufs and TuxOnIce; closest candidates:
>
>  - sys-kernel/aufs-sources: genpatches, aufs
>  - sys-kernel/pf-sources: genpatches, CK, BFQ, BFQ, TuxOnIce, UKSM
>  - sys-kernel/tuxonice-sources: genpatches, TuxOnIce
>
> What do I do? Take one (eg. aufs) and apply the other (eg. TuxOnIce)?
> What if I want to add another common patchset to it? Hardened perhaps?
>
> You can see, some of these sources (like pf-sources) already attempt to
> do so; with pf-sources in mind, why do we even have ck-sources,
> tuxonice-sources in the Portage tree? Just to duplicate work?
>
> I think we should trim down on the amount of *-sources and combine
> multiple patches into genpatches. Take an example of geek-sources
> which does all this without a problem; I don't really like the approach
> with USE flags used there, as the menuconfig can just cover that.
>
> https://github.com/init6/init_6/wiki/geek-sources
>
> What does a patch introducing new features really do? Or rather, what
> should it do when we add it? Let me summarize:
>
>  1) The features should be disabled by default.
>  2) These feature should depend on a non-vanilla / experimental option.
>  3) The patch should not affect the build by default.
>  4) The user can optionally enable the feature.
>
> So, in genpatches, since 3.9.7, BFQ was added to try this out (except
> 2.). Ensured it to be disabled by default, did not affect the build for
> anyone that does not enable it and the users have been enabling and
> using it on their own. So, does it differentiate more from vanilla? No.
>
> This helps users as well as maintainers to not have to apply BFQ on
> their own, they simply have to tick a config option and they are set.
> If all patches that introduce new features are added in this fashion,
> it spares large groups of users from having to do this on their own; we
> can also deduplicate the work in the Portage tree this way.
>
>
> ### ... and configuration. ###
>
> This problem is not only visible for patches, but also in the config.
>
> Meet CONFIG_DEVTMPFS; forget to enable it, greet a failing boot. We're
> telling users to enable it in some places, in the handbook it's a single
> line you must read, on the Wiki it's kind of missing unless you are
> luckily on the right page, on the Quick Install book it is missing too.
>
> So, we are currently providing a configuration that expects anyone to
> enable CONFIG_DEVTMPFS. But you have to remember that it need to make
> sure you read about it or enable it from the udev upgrade a while ago
> if you decide to start from a fresh config or are installing without
> that handbook you kind of have memorized.
>
> Searching for CONFIG_DEVTMPFS in the forums and #gentoo logs shows that
> this is quite often suggested as a fix and quite often actually fixes
> an user's boot. Why duplicate telling users to do that if we can do it?
>
> There are a small set of other variables in this nature, mostly *TMPFS*.
>
> This is why I think it would be handy to add a Gentoo section to the
> kernel, along the lines as described by Linus.
>
> https://lkml.org/lkml/2012/7/13/369
>
> The same goes for asking the user to enable configuration options in the
> kernel, why can't we just tell him to enable all option that toggles
> support for the user. For example; in the Gentoo section, there would be
> a "Init System Support" under which you can toggle an option to support
> the minimal requirements for some init system.
>
> Feedback is very welcome.
>
> P.S.: Not everything in this mail has been acked by the kernel lead;
> only some thoughts, I was suggested to take this to ML for discussion.
> The usage of the word 'we' in this mail is therefore hypothetical.
>
>
> ### F.A.Q. ###
>
> Q: I don't want feature X? Please don't add the patch!
>
> A: It's optional, don't enable it in your menu config.
>
> Q: What about my stable server? I really don't want to run this stuff!
>
> A: These options would depend on !CONFIG_VANILLA or CONFIG_EXPERIMENTAL
>    which would be disabled by default, therefore if you keep this option
>    the way it is on your stable server; it won't affect you.
>
>    In other words, genpatches stay as stable as before; unless you
>    explicitly toggle options that intentionally make it unstable.
>
> Q: Genpatches used to be minimal, would it gain weight?
>
> A: If you don't enable those options, it would be as minimal as before;
>    this gives the user the choice between minimal and fat.
>
> Q: What about patches that fail to build / run?
>
> A: For testing kernels (~), we would do our best to make them work but
>    there could be occasions where we will have to cut them; this is no
>    different from how the kernel herd has been handling this before, we
>    have already dropped fbcondecor in the past when it was broken and
>    the current new branch 3.10 does not support deblob yet.
>
>    For stable kernels, which are near the EOL of a branch; if the
>    feature isn't still there yet then it means that its author is
>    simply no longer working on it, this is no different from when the
>    patch would be in another *-sources or manually applied by the user.
>
>    If you are unaware of how upstream releases work, just note that
>    the major patch between 3.9.8 and 3.10.0 introduces a lot more
>    changes are applied than the minor patch between 3.9 and 3.9.8;
>    therefore, we prefer to stabilize 3.9.8 or a later 3.9 kernel than
>    stabilizing 3.10.0. You can see this pattern in history; the
>    previous three have been 3.6.11, 3.7.10 and 3.8.13.
>
> Q: Can't you do something about those build and runtime failures?
>    I like to run testing, but I don't want breaks half of the time.
>
> A: To cover this, live ebuilds help, 3.10.9999 and 3.9999 and 9999;
>    the earlier ones tracking the stable queues, whereas the last one
>    tracking the upstream rc kernel. This way we can notice what fails
>    in advance and make it working by the time it reaches testing (~).
>
>    That being said, it is to be expected that things are not always
>    fine very early in a branch; it's why those kernels are never stable.
>
> Q: What about kernel bugs, how would you know the user enabled them?
>
> A: We expect the .config to be provided, which we can run a sanity
>    check on; it's much more handy if we are aware of these patches than
>    that we have to figure out the user applies it on their own.
>
> Q: Are documentation / infrastructure / eclass changes necessary?
>
> A: The kernel project documentation has to be updated to reflect this;
>    no infrastructure or eclass changes are needed, since this is all
>    done through patches in genpatches. (Sub directories supported afaik)
>
> Q: Does this affect kernel stabilization or QA?
>
> A: No, experimental features (which would include these optional
>    features) have historically never been looked at by the arch teams;
>    if you can remember, there used to be an CONFIG_EXPERIMENTAL option
>    in the kernel to cover this. For example, you can find yourself
>    running some pretty unstable code if you use 3.8.13 and enable
>    Nouveau reclocking and power management; there are some options in
>    the menuconfig that follow this nature.
>
>    As there would be less *-sources as a consequence; less has to be
>    taken into consideration for stabilization, this is why most
>    *-sources are currently unstable, because we don't have the
>    resources to be stabilizing them all.
>
>    For QA, I don't really see a problem.
>
> Q: So, would this make vanilla-sources the new gentoo-sources?
>
> A: No, vanilla-sources undergoes some changes to more closely reflect
>    the upstream kernels; as a consequence, it would no longer have
>    stable kernels and older versions will drop more often. You can read
>    this request and the resulting discussion at the following link.
>
>    http://thread.gmane.org/gmane.linux.gentoo.kernel/697
>
>    The kernel lead summarized a part of this.
>
>    http://thread.gmane.org/gmane.linux.gentoo.kernel/697/focus=730
>
> Q: Is this a one man army comedy show?
>
> A: One man army comedy shows work, see geek-sources; if you want to
>    look at some examples of what a one man army can do, the following
>    link is a good read to go through.
>
>
> http://www.bennorthrop.com/Essays/2013/pair-programming-my-personal-nightmare.php
>
>    But joking aside; no, we are not.
>
>    We are at least with two on the kernel herd and a third developer is
>    likely to join in the near future; on top of that, other people are
>    welcome and I think it would be nice to see maintainers from
>    *-sources jump in to help along. genpatches isn't hard to maintain.
>
>    Since we're no longer a one man army, we can do more.
>
> --
> With kind regards,
>
> Tom Wijsman (TomWij)
> Gentoo Developer
>
> E-mail address  : TomWij@gentoo.org
> GPG Public Key  : 6D34E57D
> GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D
>

[-- Attachment #2: Type: text/html, Size: 12220 bytes --]

[gentoo-dev] Re: Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Steven J. Long]

 Tom Wijsman wrote:
> Greg KH wrote:
> 
> > On Mon, Jul 01, 2013 at 09:25:42PM +0200, Tom Wijsman wrote:
> > > On Mon, 1 Jul 2013 14:09:57 -0500
> > > Matthew Summers <quantumsummers@gentoo.org> wrote:
> > > > If the patchset patches the kernel's core, it doesn't matter what
> > > > CONFIG_* option is set the core kernel code
> > > > _has_now_been_changed_. This is the crux of the argument, I
> > > > believe. AUFS simply being one example of this. I'm sure there
> > > > are others.
> > > 
> > > As per my response to that point, this statement is no longer true.
> > > 
> > > Let me re-iterate it here:
> > > 
> > > Earlier I mentioned "3) The patch should not affect the build by
> > > default."; if it does, we have to adjust it to not do that, this is
> > > something that can be easily scripted.

If it does then it should never be applied, unless the user specifically
asks for it, imo, and the resultant kernel is labelled -exp as you suggest.

> It's just a matter of
> > > embedding each + block in the diff with a config check and updating
> > > the counts.
> > 
> > Wonderful, now you are maintaining a patch that looks nothing like the
> > one created by the original developers, nor tested by anyone else
> > other than gentoo developers.
> 
> I can convert the original developer's patch whenever he updates it; or
> on top of that, write a script to generate the original patch back.

Please, just keep a copy of the original patch as well as the modified
output from the script, somewhere reasonable to you, if you are doing any
editing. Traceability is essential here. Personally I think it ill-advised,
and would prefer simply that the patch were not applied if the above process
in your testing prior to usage, showed that it would affect other parts of
the build inappropriately, even when configured off. Or it's known to do so,
like aufs.

Unless of course the user specifically requests it. This can be a simple
variable with a list of required patches, or whatever.

> > Playing fast-and-loose with kernel patches is a fun thing to do, but
> > really, why?  Users love doing this type of thing, but the
> > interactions of different kernel patches with core subsystems is
> > almost always a non-trivial thing.
> 
> Our main intent is to provide them and deduplicate work; if users have
> a problem with one of the patches and it isn't clear to us, we can
> forward them to the authors. Nothing in this proposal inherits that we
> must support the patches; especially not, since they are "experimental".

I agree that users love doing this sort of thing, and that it makes sense
to provide a standard opt-in mechanism, so that it's immediately obvious
in bug-reports that a custom, non-upstream, kernel patch is in use.

> > I'm not saying not to do this, but consider this a friendly warning
> > that this is going to be a MAJOR pain to maintain and debug over the
> > long-run.
> 
> Maybe, maybe not; plan is to do a slow start, congestion avoidance. :)
> 
> > But hey, what do I know?  It's not like I've ever done this before and
> > had the experience of the resulting fall-out that took years to
> > recover from on user's production systems, causing a number of
> > enterprise Linux companies to swear that they would never do this
> > type of thing again...

As you said, users love doing this sort of thing, so it's happening under
our noses in any case; that's the nature of a source distribution. In such
a case it makes sense to have the info in a bug-report from the beginning.
 
> I covered this in the conditions as well as the F.A.Q.; feel free to
> read about it again, this does not affect them unless they explicitly
> want them to enable CONFIG_GENTOO_EXPERIMENTAL which help includes a
> warning; thinking it through, we might even suffix -exp to version.

This makes a lot of sense. Since the point of this is to bring simplicity
to the process for the user, and more useful bug reports for Gentoo, it
seems like a requirement.
 
> > Personally, I wish you luck, it will push the sane users to the
> > vanilla-sources tree, which I strongly encourage :)
> 
> I will push them to keep CONFIG_GENTOO_EXPERIMENTAL disabled. :)

Yup.

-- 
#friendly-coders -- We're friendly, but we're not /that/ friendly ;-)

[gentoo-dev] Re: gentoo-checkconf script Re: Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Steven J. Long]

Michael Weber wrote:
> Anthony G. Basile wrote:
> > Now I'm confused because gentoo-sources is gentoo specific.  It
> > contains stuff that we need in gentoo but other distros do not
> > need, like our end-to-end support for certain xattr namespaces.  If
> > you remove these then we must either 1) maintain a userland which
> > is not in line with other distros or 2) give up on critical
> > features we want in gentoo, like markings on elf object in
> > user.pax.flags and certain caps, as well as in the future
> > preserving selinux labels through emerge.  Upstream will not accept
> > them because of "who needs that crap" and we can't give them up 
> > without loosing core functionality.  Feel free to review those
> > patches but don't ask us to drop them from gentoo-sources because
> > their not in upstream.

Indeed. Every distro has their own kernel patches, and to all sorts of
software. That's why we use it, since things are integrated, and in
Gentoo we get as close to upstream as possible, while still useful.

> What about a check-kernel-config-for-gentoo-compliance script for
> starterts?
> 
> I manage a handfull of kernel configs over some years (laptop vs.
> server, graphics, firewalling capabilities) and was always tempted to
> write an script to check if the config meets a certain set of
> requirements. I think of "xattr", "selinux", "gentoo-boot" and so on,

That makes sense.

> that can be expanded by users demand, like, "CONFIG_CMDLINE should
> include" and "CONFIG_DEFAULT_HOSTNAME=x" and "all iptables target on".
Something like this?
$KBUILD_SRC=${1:-.}
# default user check function
usr_config_check(){
	:
}
user_script=$KBUILD_SRC/gentoo-user-check

if [ -f $user_script ]; then
	# allow user to override settings
	. "$user_script" || die "unable to source user script: $user_script"
fi
..main script..
usr_config_check "$VERSION" "$PATCHLEVEL" etc.. || die "failed user check"
..cleanup..
exit 0

> An additional make target in gentoo-sources could the warn about any
> missing feature, and ask for "yes" or wait some seconds.
> (I remember reaging some funny note about my kernel supporting x32 but
> by userland not, like that kernel build would run on that userland)

Yeah, if it ran after config, it could do simple checks at least, to begin
with. Looking at linux/Makefile, we could just add:
 $(Q)gentoo-check-config "$KBUILD_SRC"
 after: $(Q)$(MAKE) $(build)=scripts/kconfig $@
..in config. I'm not sure if you'd need it for %config case as well; I
don't see the need in the context of overall sanity checks.

It would be useful though, to check for required things like {DEV,}TMPFS

-- 
#friendly-coders -- We're friendly, but we're not /that/ friendly ;-)

Re: [gentoo-dev] Re: Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 2577 bytes --]

On Wed, 3 Jul 2013 11:45:55 +0100
"Steven J. Long" <slong@rathaus.eclipse.co.uk> wrote:

>  Tom Wijsman wrote:
> > > > Earlier I mentioned "3) The patch should not affect the build by
> > > > default."; if it does, we have to adjust it to not do that,
> > > > this is something that can be easily scripted.
> 
> If it does then it should never be applied, unless the user
> specifically asks for it, imo, and the resultant kernel is labelled
> -exp as you suggest.

Yes, we are going to introduce an experimental USE flag for this.

> > It's just a matter of
> > > > embedding each + block in the diff with a config check and
> > > > updating the counts.
> > > 
> > > Wonderful, now you are maintaining a patch that looks nothing
> > > like the one created by the original developers, nor tested by
> > > anyone else other than gentoo developers.
> > 
> > I can convert the original developer's patch whenever he updates
> > it; or on top of that, write a script to generate the original
> > patch back.
> 
> Please, just keep a copy of the original patch as well as the modified
> output from the script, somewhere reasonable to you, if you are doing
> any editing. Traceability is essential here.

The need to keep duplicates around is a broken design; if you would
need to do this, there is something worse going on. But yes, some git
branches can easily cover the editing part.

> Personally I think it ill-advised, and would prefer simply that the
> patch were not applied if the above process in your testing prior to
> usage, showed that it would affect other parts of the build
> inappropriately, even when configured off. Or it's known to do so,
> like aufs.

Maybe, we can hear whether the patch authors want to do something about
this, so we don't have to convert patches like this; maybe most of them
are willing to do this without a problem. Though, there are going to be
parts that they want to unconditionally apply, which is why we will
need to wrap those parts with a check.

> Unless of course the user specifically requests it. This can be a
> simple variable with a list of required patches, or whatever.

With USE=-experimental (which will be the default) they are excluded by
default, after enabling that the user can exclude patches by setting
UNIPATCH_EXCLUDE through the package.env mechanism.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Proper distribution integration of kernel *-sources, patches and configuration.

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 1777 bytes --]

On Tue, 2 Jul 2013 21:16:07 +0300
Sergei Trofimovich <slyfox@gentoo.org> wrote:

> udev's case:
> It's _not_ the kernel which upgrade breaks user's system. Why do you
> try to "fix" it? If you want to save user's box - make check at
> pre-install time. Otherwise it will break.

Won't work for stage3's case, where this check won't run.

> kernel's case:
> CONFIG_DEVTMPFS needs 'default yes', right? One-liner to
> gentoo-sources.

It's not as simple as a one-liner, because we need to respect people
that want to build a kernel without that; not everyone who uses
genpatches runs a Gentoo system, and note every other OS needs that
variable to enabled.

> Why 'hide' it? It's very counterintuitive.

What is being hidden?

> What will you do with all-those-required-to-boot-properly
>   - root filesystem
>   - disk controller
>   - USB keyboard drivers
>  ?
> Include them all unless CONFIG_DONT_REMOVER_OR_WONT_BOOT
> option?

These don't fall under options that need to be enabled for everyone.

From what I see on chat and forums, people often set all these fine;
yet they not always enable CONFIG_DEVTMPFS and similar variables.

We deal with the absolutely necessary, not spoon feed every option.

> I'm afraid I don't see how it's a solution. 
> Suppose, tomorrow's udev will require CONFIG_foo, and glibc will
> require CONFIG_bar. How will you save user with your mechanism?

I don't see how we don't save them; but well, I'm not entirely sure if
you're talking about the opening post mechanism or another one here.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Proper distribution integration of kernel *-sources, patches and configuration.

[Sergei Trofimovich]

[-- Attachment #1: Type: text/plain, Size: 3382 bytes --]

On Wed, 3 Jul 2013 15:06:07 +0200
Tom Wijsman <TomWij@gentoo.org> wrote:

> On Tue, 2 Jul 2013 21:16:07 +0300
> Sergei Trofimovich <slyfox@gentoo.org> wrote:
> 
> > udev's case:
> > It's _not_ the kernel which upgrade breaks user's system. Why do you
> > try to "fix" it? If you want to save user's box - make check at
> > pre-install time. Otherwise it will break.
> 
> Won't work for stage3's case, where this check won't run.

Non-issue.
- stage3 does not work out of the box. you can't boot or run it
- stage3 does not work at all in containers w/o devtmpfs kernel.
  no matter what you try to force on newly built kernel.

Let's consider real-world examples:

My friend at cloud hosting company decided to try gentoo on
a xen hypervisor box (3.0.6 host w/o devtmpfs). unpacked old
stage3, which worked; merged new stuff there and rebooted.
No kernel change, old xen 3.0.6. Guess what happened when he
upgraded udev afterwards.

Another my friend unpacked stage3 on his handheld with android kernel.
2.6.3x-something w/o accept4() syscall. Original stage3 worked, upgrade
broke system.

> > kernel's case:
> > CONFIG_DEVTMPFS needs 'default yes', right? One-liner to
> > gentoo-sources.
> 
> It's not as simple as a one-liner, because we need to respect people
> that want to build a kernel without that; not everyone who uses
> genpatches runs a Gentoo system, and note every other OS needs that
> variable to enabled.

'default yes' is a Kconfig option. It picks a value if user didn't pick it
explicitely.

linux/drivers/base/Kconfig:
    config DEVTMPFS
            bool "Maintain a devtmpfs filesystem to mount at /dev"
            depends on HOTPLUG
+          default y

One-liner, no?

> > Why 'hide' it? It's very counterintuitive.
> 
> What is being hidden?

You plan to have it visible only when something gentoo-specific needs
to be disabled.
[from origina post]
 > This is why I think it would be handy to add a Gentoo section to the
 > kernel, along the lines as described by Linus.
 > 
 > https://lkml.org/lkml/2012/7/13/369

AFAIU I won't allow deselecting option until you deselect gentoo-specific bit.

If you plan to upstream everything, then ok, no distro-specific hacks.

> > What will you do with all-those-required-to-boot-properly
> >   - root filesystem
> >   - disk controller
> >   - USB keyboard drivers
> >  ?
> > Include them all unless CONFIG_DONT_REMOVER_OR_WONT_BOOT
> > option?
> 
> These don't fall under options that need to be enabled for everyone.
> 
> From what I see on chat and forums, people often set all these fine;
> yet they not always enable CONFIG_DEVTMPFS and similar variables.
> 
> We deal with the absolutely necessary, not spoon feed every option.
> 
> > I'm afraid I don't see how it's a solution. 
> > Suppose, tomorrow's udev will require CONFIG_foo, and glibc will
> > require CONFIG_bar. How will you save user with your mechanism?
> 
> I don't see how we don't save them; but well, I'm not entirely sure if
> you're talking about the opening post mechanism or another one here.

Yeah, the opening post one.
Suppose, tomorrow's udev requires CONFIG_CGROUPS=y.
What are your actions to save user's box?

Issue new gentoo-sources and pray user will install/boot it
before new udev? See my first two examples.

-- 

  Sergei

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Proper distribution integration of kernel *-sources, patches and configuration.

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 4589 bytes --]

On Wed, 3 Jul 2013 16:52:37 +0300
Sergei Trofimovich <slyfox@gentoo.org> wrote:

> On Wed, 3 Jul 2013 15:06:07 +0200
> Tom Wijsman <TomWij@gentoo.org> wrote:
> 
> > On Tue, 2 Jul 2013 21:16:07 +0300
> > Sergei Trofimovich <slyfox@gentoo.org> wrote:
> > 
> > > udev's case:
> > > It's _not_ the kernel which upgrade breaks user's system. Why do
> > > you try to "fix" it? If you want to save user's box - make check
> > > at pre-install time. Otherwise it will break.
> > 
> > Won't work for stage3's case, where this check won't run.
> 
> Non-issue.
> - stage3 does not work out of the box. you can't boot or run it
> - stage3 does not work at all in containers w/o devtmpfs kernel.
>   no matter what you try to force on newly built kernel.

Make check doesn't work there. A distro default config option does.

> Let's consider real-world examples:
> 
> My friend at cloud hosting company decided to try gentoo on
> a xen hypervisor box (3.0.6 host w/o devtmpfs). unpacked old
> stage3, which worked; merged new stuff there and rebooted.
> No kernel change, old xen 3.0.6. Guess what happened when he
> upgraded udev afterwards.
>
> Another my friend unpacked stage3 on his handheld with android kernel.
> 2.6.3x-something w/o accept4() syscall. Original stage3 worked,
> upgrade broke system.

Stage3 contains the upgraded udev these days.

> > > kernel's case:
> > > CONFIG_DEVTMPFS needs 'default yes', right? One-liner to
> > > gentoo-sources.
> > 
> > It's not as simple as a one-liner, because we need to respect people
> > that want to build a kernel without that; not everyone who uses
> > genpatches runs a Gentoo system, and note every other OS needs that
> > variable to enabled.
> 
> 'default yes' is a Kconfig option. It picks a value if user didn't
> pick it explicitely.
>
> linux/drivers/base/Kconfig:
>     config DEVTMPFS
>             bool "Maintain a devtmpfs filesystem to mount at /dev"
>             depends on HOTPLUG
> +          default y
> 
> One-liner, no?

We don't want to change DEVTMPFS itself, because not everyone whom
uses genpatches wants DEVTMPFS itself enabled by default; we instead
plan to provide it as a distribution specific option that enables it.

https://lkml.org/lkml/2012/7/13/369

> > > Why 'hide' it? It's very counterintuitive.
> > 
> > What is being hidden?
> 
> You plan to have it visible only when something gentoo-specific needs
> to be disabled.

No, I did not state such thing.

> [from origina post]
>  > This is why I think it would be handy to add a Gentoo section to
>  > the kernel, along the lines as described by Linus.
>  > 
>  > https://lkml.org/lkml/2012/7/13/369
> 
> AFAIU I won't allow deselecting option until you deselect
> gentoo-specific bit.

Then you disable the bit and deselect it, I don't see a problem here.

> If you plan to upstream everything, then ok, no distro-specific hacks.

I don't see how providing distro default options as described is a hack.

> > > What will you do with all-those-required-to-boot-properly
> > >   - root filesystem
> > >   - disk controller
> > >   - USB keyboard drivers
> > >  ?
> > > Include them all unless CONFIG_DONT_REMOVER_OR_WONT_BOOT
> > > option?
> > 
> > These don't fall under options that need to be enabled for everyone.
> > 
> > From what I see on chat and forums, people often set all these fine;
> > yet they not always enable CONFIG_DEVTMPFS and similar variables.
> > 
> > We deal with the absolutely necessary, not spoon feed every option.
> > 
> > > I'm afraid I don't see how it's a solution. 
> > > Suppose, tomorrow's udev will require CONFIG_foo, and glibc will
> > > require CONFIG_bar. How will you save user with your mechanism?
> > 
> > I don't see how we don't save them; but well, I'm not entirely sure
> > if you're talking about the opening post mechanism or another one
> > here.
> 
> Yeah, the opening post one.
> Suppose, tomorrow's udev requires CONFIG_CGROUPS=y.
> What are your actions to save user's box?

If it's along the lines of what's proposed, add it as well.

> Issue new gentoo-sources and pray user will install/boot it
> before new udev? See my first two examples.

That's what news items are for; we're not trying to deal with this
short term case, but rather with the long term case after that.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Proper distribution integration of kernel *-sources, patches and configuration.

[Sergei Trofimovich]

[-- Attachment #1: Type: text/plain, Size: 83 bytes --]

Oh, so It has nothing to do with upgrade path.

My apologies

-- 

  Sergei

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Re: Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Walter Dnes]

On Wed, Jul 03, 2013 at 02:42:56PM +0200, Tom Wijsman wrote

> With USE=-experimental (which will be the default) they are excluded by
> default, after enabling that the user can exclude patches by setting
> UNIPATCH_EXCLUDE through the package.env mechanism.

  Here's my user, not-a-developer, perspective.  I think that there
should be a mechanism to enable one patch at a time.  Here's the
rationale...

  Assume that there are 50 different patches available.  I may want/need
features provided by 1 of those patches.  I probably do *NOT* want to
enable the other 49 patches.  This is similar in concept to enabling one
~amd64 ebuild, versus globally enabling ~amd64.

  Even if I can come up with the list of the 49 patches to exclude, what
happens when the next developer comes along with patch #51?  Does it get
applied next time I build a kernel (ouch)?

  IANAD (I Am Not A Developer), but if I did want to apply custom
patches, I think the right approach would be to somewhere manually
modify UNIPATCH_LIST.  If that approach won't work, maybe a USE_EXPAND
flag make.conf might be the way to go.  E.g. CUSTOM_PATCH="foo bar"
would resolve to USE="custom_patch_foo custom_patch_bar"

-- 
Walter Dnes <waltdnes@waltdnes.org>
I don't run "desktop environments"; I run useful applications

[gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Steven J. Long]

Tom Wijsman wrote:
> "Steven J. Long" wrote:
> > If it does [affect the build by default] then it should never be
> > applied, unless the user specifically asks for it, imo, and the
> > resultant kernel is labelled -exp as you suggest.
> 
> Yes, we are going to introduce an experimental USE flag for this.

That's for the over-arching set of patches isn't it? Which takes care of
the labelling.
 
> > > It's just a matter of embedding each + block in the diff with a
> > > config check and updating the counts.
..
> > > I can convert the original developer's patch whenever he updates
> > > it; or on top of that, write a script to generate the original
> > > patch back.

> > Please, just keep a copy of the original patch as well as the modified
> > output from the script, somewhere reasonable to you, if you are doing
> > any editing. Traceability is essential here.
> 
> The need to keep duplicates around is a broken design; if you would
> need to do this, there is something worse going on.

There is indeed: there's major changes to other parts of the kernel tree,
even when the patch's configuration options are set to off, as they are
by default (or must be in the Gentoo experimental config patch.) That's
the reason they were not accepted, as has been discussed earlier in the
thread.

> But yes, some git branches can easily cover the editing part.

I think a lot of these things are checks that should happen before patches
are included, and on every upgrade. So we need to separate out what the
developer/ebuild-maintainer has to do to prepare files/, and what needs
to happen in the ebuild itself.

> > Personally I think it ill-advised, and would prefer simply that the
> > patch were not applied if the above process in your testing prior to
> > usage, showed that it would affect other parts of the build
> > inappropriately, even when configured off. Or it's known to do so,
> > like aufs.
> 
> Maybe, we can hear whether the patch authors want to do something about
> this, so we don't have to convert patches like this; maybe most of them
> are willing to do this without a problem.

The conversation is definitely worth having. Especially if Gentoo runs
automated tests on the patchsets to verify previously "safe" patches
are still safe to apply unconditionally, ie that all changes are #ifdef
or #if guarded.

> Though, there are going to be
> parts that they want to unconditionally apply, which is why we will
> need to wrap those parts with a check.

Indeed; upstream patch authors are not going to suddenly change, especially
with the known-troublesome cases, while there is still the need to make
the patches available to users who would otherwise be applying them
without Gentoo -exp labelling.

> > Unless of course the user specifically requests it. This can be a
> > simple variable with a list of required patches, or whatever.
> 
> With USE=-experimental (which will be the default) they are excluded by
> default, after enabling that the user can exclude patches by setting
> UNIPATCH_EXCLUDE through the package.env mechanism.

I'd feel happier if certain patches, the troublesome ones discussed, had
to be explicity enabled, before the configuration options and the patch
to kernel files took place. Perhaps via UNIPATCH_INCLUDE or USE=aufs.

Regards,
steveL.
-- 
#friendly-coders -- We're friendly, but we're not /that/ friendly ;-)

[gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Steven J. Long]

Walter Dnes wrote:
> Tom Wijsman wrote
> > With USE=-experimental (which will be the default) they are excluded by
> > default, after enabling that the user can exclude patches by setting
> > UNIPATCH_EXCLUDE through the package.env mechanism.
> 
>   Assume that there are 50 different patches available.  I may want/need
> features provided by 1 of those patches.  I probably do *NOT* want to
> enable the other 49 patches.  This is similar in concept to enabling one
> ~amd64 ebuild, versus globally enabling ~amd64.

Agreed. But I'd still like to have the option of enabling the setting in
the {,n,menu..}config, when it's safe to have available as such.
 
>   Even if I can come up with the list of the 49 patches to exclude, what
> happens when the next developer comes along with patch #51?  Does it get
> applied next time I build a kernel (ouch)?

But how will I know about it? Seeing the new flag in USE or the expand in
emerge -pv is okay, but not as simple as having it under Gentoo experimental
as a new option, just like every other new kernel config option. That's when
you review the new kernel in any case; with oldconfig, and subsequent nconfig
or w/e.

When it's reasonable to do so, as the patch is verified not to change anything
if its option is unset.

>   IANAD (I Am Not A Developer), but if I did want to apply custom
> patches, I think the right approach would be to somewhere manually
> modify UNIPATCH_LIST.  If that approach won't work, maybe a USE_EXPAND
> flag make.conf might be the way to go.  E.g. CUSTOM_PATCH="foo bar"
> would resolve to USE="custom_patch_foo custom_patch_bar"

Yeah, but I don't want to keep an eye on all that, and mess around with
package.use or expand settings in make.conf. I'd much rather have most
in menuconfig by default with USE=experimental, and then explicitly
include things like aufs.

Just my pov, but I think it fits more with original intent of the idea.

Regards,
steveL
-- 
#friendly-coders -- We're friendly, but we're not /that/ friendly ;-)

Re: [gentoo-dev] Re: Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 2484 bytes --]

On Wed, 3 Jul 2013 22:00:19 -0400
"Walter Dnes" <waltdnes@waltdnes.org> wrote:

>   Here's my user, not-a-developer, perspective.  I think that there
> should be a mechanism to enable one patch at a time.  Here's the
> rationale...

Haven't tried this myself but I believe UNIPATCH_EXCLUDE using
the /etc/portage/package.env mechanism should accomplish this.

>   Assume that there are 50 different patches available.  I may
> want/need features provided by 1 of those patches.  I probably do
> *NOT* want to enable the other 49 patches.  This is similar in
> concept to enabling one ~amd64 ebuild, versus globally enabling
> ~amd64.

We don't plan to add this amount of patches, we wouldn't be able to
maintain that number of patches; just the active / most common / ...

>   Even if I can come up with the list of the 49 patches to exclude,
> what happens when the next developer comes along with patch #51?
> Does it get applied next time I build a kernel (ouch)?

Yes, but why would this bother you? None of its code will be included
in the build unless you explicitly enable it in the menu config.

>   IANAD (I Am Not A Developer), but if I did want to apply custom
> patches, I think the right approach would be to somewhere manually
> modify UNIPATCH_LIST.  If that approach won't work, maybe a USE_EXPAND
> flag make.conf might be the way to go.  E.g. CUSTOM_PATCH="foo bar"
> would resolve to USE="custom_patch_foo custom_patch_bar"

Not sure how you would use UNIPATCH_LIST, can you give an example?

As for USE_EXPAND, we want to avoid that; it adds an extra layer of
configuration for the user as well as maintenance for the maintainer for
no good reason, this whole idea works because patches are made optional
and only affect the build if you enable them in the menu config.

Applying the experimental genpatches tarball through the experimental
USE flag is opt-in, enabling CONFIG_GENTOO_EXPERIMENTAL is opt-in and
applying individual options in the menu config is opt-in.

I don't see a reason to make individual patches opt-in; if the user
experience problems with one of them, UNIPATCH_EXCLUDE should suffice.
We can document the existence of this mechanism by documenting about it
in the experimental USE flag.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 3264 bytes --]

On Thu, 4 Jul 2013 06:27:59 +0100
"Steven J. Long" <slong@rathaus.eclipse.co.uk> wrote:

> Tom Wijsman wrote:
> > "Steven J. Long" wrote:
> > > If it does [affect the build by default] then it should never be
> > > applied, unless the user specifically asks for it, imo, and the
> > > resultant kernel is labelled -exp as you suggest.
> > 
> > Yes, we are going to introduce an experimental USE flag for this.
> 
> That's for the over-arching set of patches isn't it? Which takes care
> of the labelling.

Yes, we currently have "base" and "extras" tarballs for genpatches; it
is trivial to add a "experimental" tarball to this set, all the
optional experimental patches will be in that tarball. This is also
handy for maintainers of other distros whom use genpatches.

> > > > It's just a matter of embedding each + block in the diff with a
> > > > config check and updating the counts.
> ..
> > > > I can convert the original developer's patch whenever he updates
> > > > it; or on top of that, write a script to generate the original
> > > > patch back.
> 
> > > Please, just keep a copy of the original patch as well as the
> > > modified output from the script, somewhere reasonable to you, if
> > > you are doing any editing. Traceability is essential here.
> > 
> > But yes, some git branches can easily cover the editing part.
> 
> I think a lot of these things are checks that should happen before
> patches are included, and on every upgrade. So we need to separate
> out what the developer/ebuild-maintainer has to do to prepare files/,
> and what needs to happen in the ebuild itself.

Please note that this discussion is regarding genpatches; so, there is
no files/ directory and the ebuild change is very minimal, changing one
or two numbers to indicate the genpatches version to use.

Every time I add a patch to genpatches I compile a test kernel using a
test ebuild; I plan to add these checks to our genpatches scripts, then
I can call the checks script from the ebuild in a QA way.

> > > Unless of course the user specifically requests it. This can be a
> > > simple variable with a list of required patches, or whatever.
> > 
> > With USE=-experimental (which will be the default) they are
> > excluded by default, after enabling that the user can exclude
> > patches by setting UNIPATCH_EXCLUDE through the package.env
> > mechanism.
> 
> I'd feel happier if certain patches, the troublesome ones discussed,
> had to be explicity enabled, before the configuration options and the
> patch to kernel files took place. Perhaps via UNIPATCH_INCLUDE or
> USE=aufs.

There shouldn't be a problem here unless the user applies a lot of
patches that could introduce a colliding patch; in this case the user
either has to fix the conflicting patch or exclude ours. We can't know
for ourselves which patches will be troublesome in this light; but
well, I feel like this only happens on a very exceptional basis. If an
user keeps this amount of patches, ours are probably not needed.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Proper distribution integration of kernel *-sources, patches and configuration.

[Mike Pagano]

On Monday, July 01, 2013 01:52:10 PM Rick Zero_Chaos Farina wrote:
> On 07/01/2013 01:35 PM, Tom Wijsman wrote:
> > On Mon, 01 Jul 2013 12:20:09 -0400
> > 
> > "Rick \"Zero_Chaos\" Farina" <zerochaos@gentoo.org> wrote:
> >> Some patches are reasonably easy to combine, such as genpatches and
> >> aufs.  Some patches are difficult to combine, such as hardened and *.
> >> When you combine hardened patches and aufs (for example) you need
> >> extra patches.  I would be THRILLED to see the number of sources cut
> >> down, but hardened-sources must be it's own thing (that said, I'll
> >> personally maintain the aufs patches for hardened if they wanted to
> >> add a USE=aufs flag).
> > 
> > Yes, gave it as an quick example but I indeed remember from going
> > through the sources ebuilds that hardened ebuilds do quite some things.
> > I think the downside from extending genpatches is that hardened-sources
> > can no longer rely on it, but we'll have to see that as we go forward.
> > 
> > I don't think that apart from hardened the optional patches on their own
> > are hard to combine; they each have their own separate goal, I don't
> > see them conflict on anything. If it happens once in a while, we can
> > still maintain them to work together.
> 
> Hardened has K_WANT_GENPATCHES="base" which means it already doesn't
> take the extra patches.  We could either introduce a new flag for your
> patches like K_WANT_GENPATCHES="base extra geek" or more likely make
> each one with their own name so that hardened et al can take what they
> like and leave the rest.

Ok, so I have talked to Tom about this on IRC and it's probably prudent to 
chime in.  I have gotten many complaints in the past that there is not enough 
in g-s, and, of course, I've gotten complaints about there being too much.

I have 'relaxed' a tad about what I think should be in g-s, but maybe it has 
gone a bit farther than I wanted it too.

I would like to see a "-experimental" use flag and base,extras,geek (whatever) 
so that g-s goes back to what it's original goal was with nothing non-upstream 
unless the user does a configuration change themselves.

This will actually help us solve both issues.

1) it will allow us to pull g-s back to it's original goal as  a minimal 
kernel sources with upstream only patches.
2) we can carry some patches from upstreams trees that possibly aren't yet in 
-next, or not yet accepted to mainline but do provide some benefit to a smaller 
group of our users. (Thinking about our thinkpad patches)


-- 
Mike Pagano
Gentoo Developer - Kernel Project
E-Mail     : mpagano@gentoo.org
GnuPG FP   : EEE2 601D 0763 B60F 848C  9E14 3C33 C650 B576 E4E3
Public Key : http://pgp.mit.edu:11371/pks/lookup?search=0xB576E4E3&op=index

[gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Steven J. Long]

Tom Wijsman wrote:
> Steven J. Long wrote:
> > Tom Wijsman wrote:
> > > "Steven J. Long" wrote:
> > > > If it does [affect the build by default] then it should never be
> > > > applied, unless the user specifically asks for it, imo, and the
> > > > resultant kernel is labelled -exp as you suggest.
> > > 
> > > Yes, we are going to introduce an experimental USE flag for this.
> > 
> > That's for the over-arching set of patches isn't it? Which takes care
> > of the labelling.
> 
> Yes, we currently have "base" and "extras" tarballs for genpatches; it
> is trivial to add a "experimental" tarball to this set, all the
> optional experimental patches will be in that tarball. This is also
> handy for maintainers of other distros whom use genpatches.

That's cool. The bit about the user explicitly opting-in to 'fragile'
patches still is of concern, however.

> > I think a lot of these things are checks that should happen before
> > patches are included, and on every upgrade. So we need to separate
> > out what the developer/ebuild-maintainer has to do to prepare files/,
> > and what needs to happen in the ebuild itself.
> 
> Please note that this discussion is regarding genpatches; so, there is
> no files/ directory and the ebuild change is very minimal, changing one
> or two numbers to indicate the genpatches version to use.
> 
> Every time I add a patch to genpatches I compile a test kernel using a
> test ebuild; I plan to add these checks to our genpatches scripts, then
> I can call the checks script from the ebuild in a QA way.

Ah OK, sorry for confusion in that case. Glad to hear you have QA covered.
 
> > > > Unless of course the user specifically requests it. This can be a
> > > > simple variable with a list of required patches, or whatever.
> > > 
> > > With USE=-experimental (which will be the default) they are
> > > excluded by default, after enabling that the user can exclude
> > > patches by setting UNIPATCH_EXCLUDE through the package.env
> > > mechanism.
> > 
> > I'd feel happier if certain patches, the troublesome ones discussed,
> > had to be explicity enabled, before the configuration options and the
> > patch to kernel files took place. Perhaps via UNIPATCH_INCLUDE or
> > USE=aufs.
> 
> There shouldn't be a problem here unless the user applies a lot of
> patches that could introduce a colliding patch; in this case the user
> either has to fix the conflicting patch or exclude ours. We can't know
> for ourselves which patches will be troublesome in this light; but
> well, I feel like this only happens on a very exceptional basis. If an
> user keeps this amount of patches, ours are probably not needed.

No, the problem is as mentioned, with patches for which disabling the
configure option, does not stop the patch from changing anything. Such
as aufs, as has been outlined by Greg K-H earlier in the thread.

This addresses the entirely reasonable concerns of users like Walter and
myself, as well as the warning alarms sounded by an upstream maintainer and
others. Not to address the issue risks derailing the whole effort before
it has begun; and can hardly be called "Proper integration", imo.

After all, these are experimental patches. If they don't play nicely, don't
let them out of the sandbox, unless the user tells us to.

Having a clear "policy" on that makes negotiating with patch authors much
simpler too, and it's far more likely they'll fall into line where possible,
just as upstreams are usually quite happy to apply portability patches: it
means they get more users and feedback.

Regards,
steveL
-- 
#friendly-coders -- We're friendly, but we're not /that/ friendly ;-)

Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 2616 bytes --]

On Fri, 5 Jul 2013 09:38:10 +0100
"Steven J. Long" <slong@rathaus.eclipse.co.uk> wrote:

> Tom Wijsman wrote:
> > 
> > Yes, we currently have "base" and "extras" tarballs for genpatches;
> > it is trivial to add a "experimental" tarball to this set, all the
> > optional experimental patches will be in that tarball. This is also
> > handy for maintainers of other distros whom use genpatches.
> 
> That's cool. The bit about the user explicitly opting-in to 'fragile'
> patches still is of concern, however.

Why is this still of concern? (See the next response before replying)

> > There shouldn't be a problem here unless the user applies a lot of
> > patches that could introduce a colliding patch; in this case the
> > user either has to fix the conflicting patch or exclude ours. We
> > can't know for ourselves which patches will be troublesome in this
> > light; but well, I feel like this only happens on a very
> > exceptional basis. If an user keeps this amount of patches, ours
> > are probably not needed.
> 
> No, the problem is as mentioned, with patches for which disabling the
> configure option, does not stop the patch from changing anything. Such
> as aufs, as has been outlined by Greg K-H earlier in the thread.

My original post mentions "3) The patch should not affect the build by
default.", which I later clarified with "It's just a matter of embedding
each + block in the diff with a config check and updating the counts.";
in detail we QA check whether all blocks contain such a config check.

It does not change anything other than insert some code which won't be
parsed if you don't enable the relevant option in the menu config.

> [... SNIP ...]; and can hardly be called "Proper integration", imo.

Why not?

> After all, these are experimental patches. If they don't play nicely,
> don't let them out of the sandbox, unless the user tells us to.

It's the user that can let each individual patch out of its sandbox.

> Having a clear "policy" on that makes negotiating with patch authors
> much simpler too, and it's far more likely they'll fall into line
> where possible, just as upstreams are usually quite happy to apply
> portability patches: it means they get more users and feedback.

Not sure what is meant by this; but given my above clarifications, I
think you were missing a detail in the approach that is taken.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

[gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.

[Steven J. Long]

Tom Wijsman wrote:
> "Steven J. Long" wrote:
> > The bit about the user explicitly opting-in to 'fragile' patches still
> > is of concern, however.
> 
> Why is this still of concern?
..
> My original post mentions "3) The patch should not affect the build by
> default.", which I later clarified with "It's just a matter of embedding
> each + block in the diff with a config check and updating the counts.";
> in detail we QA check whether all blocks contain such a config check.
> 
> It does not change anything other than insert some code which won't be
> parsed if you don't enable the relevant option in the menu config.

In which case you're either not dealing with patches like aufs, or you're
modifying them, when afaic they're still fragile, and the modification
shouldn't even be done. Just let the user opt-in to patches that the QA
process has picked up, or are known to affect the kernel unconditionally.

> > [... SNIP ...]; and can hardly be called "Proper integration", imo.
> 
> Why not?

Because of this:
> > After all, these are experimental patches. If they don't play nicely,
> > don't let them out of the sandbox, unless the user tells us to.
> 
> It's the user that can let each individual patch out of its sandbox.

Not when you default them all to on, given one USE flag, they're not.
 
> > Having a clear "policy" on that makes negotiating with patch authors
> > much simpler too, and it's far more likely they'll fall into line
> > where possible, just as upstreams are usually quite happy to apply
> > portability patches: it means they get more users and feedback.
> 
> Not sure what is meant by this; but given my above clarifications, I
> think you were missing a detail in the approach that is taken.

If you're saying you want to press ahead with modified patches, for things
like aufs, then I think you should reconsider. As it sounds like a god-awful
hack that will lead to complexity where it's not needed, nor useful, as well
as a maintenance burden.

Keep it simple, and let the user explicitly opt-in to patchsets like aufs, so
the maintenance responsibility is theirs from the beginning, and they feel
ownership of the wider implications of that decision[1]. That's all Walter and
I are after (afaict in his regard, ofc.)

It's also easier to implement, and draws the demarcation that I mentioned above,
such that patchset authors have an incentive to "do the right thing" and again,
the maintenance burden shifts to where it belongs: not on your shoulders.

Regards,
steveL, slightly puzzled as to why you *want* to add make-work.

[1] No, asserting that "it's all under the experimental flag" is not enough.
It's just a cop-out, afaic.
-- 
#friendly-coders -- We're friendly, but we're not /that/ friendly ;-)

Re: [gentoo-dev] Proper distribution integration of kernel *-sources, patches and configuration.

[Donnie Berkholz]

[-- Attachment #1: Type: text/plain, Size: 1318 bytes --]

On 20:19 Thu 04 Jul     , Mike Pagano wrote:
> I have 'relaxed' a tad about what I think should be in g-s, but maybe it has 
> gone a bit farther than I wanted it too.
> 
> I would like to see a "-experimental" use flag and base,extras,geek (whatever) 
> so that g-s goes back to what it's original goal was with nothing non-upstream 
> unless the user does a configuration change themselves.
> 
> This will actually help us solve both issues.
> 
> 1) it will allow us to pull g-s back to it's original goal as  a minimal 
> kernel sources with upstream only patches.

Original? Not true. gentoo-sources has, for ages, carried feature 
patches that were considered useful to Gentoo as a whole or to releng in 
particular.

It's carried whole filesystems like XFS, it's carried EVMS, it's carried 
pretty much the whole ck- patchset (Con Kolivas), grsec, FreeS/WAN and 
OpenS/WAN, the bootsplash stuff, etc.

> 2) we can carry some patches from upstreams trees that possibly aren't yet in 
> -next, or not yet accepted to mainline but do provide some benefit to a smaller 
> group of our users. (Thinking about our thinkpad patches)

-- 
Thanks,
Donnie

Donnie Berkholz
Council Member / Sr. Developer, Gentoo Linux <http://dberkholz.com>
Analyst, RedMonk <http://redmonk.com/dberkholz/>

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]