* [gentoo-dev] [PMS] Add RESTRICT="distcc" capability @ 2008-11-01 20:57 Gordon Malm 2008-11-01 21:00 ` Ciaran McCreesh 2008-11-02 0:11 ` David Leverton 0 siblings, 2 replies; 18+ messages in thread From: Gordon Malm @ 2008-11-01 20:57 UTC (permalink / raw To: gentoo-dev I'd like to get "distcc" added as one of the FEATURES we are able to RESTRICT. It is true that RESTRICT="distcc" is usually not the proper solution to problems. But in the case of out-of-tree kernel modules the idea of distributing compile jobs to other machines is fundamentally flawed IMO. Additionally, there are a few packages in the tree already that will never get fixed (or be fixable) and just have some check for FEATURES=distcc && die "disable distcc" type stuff. Thanks, Gordon Malm (gengor) ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [gentoo-dev] [PMS] Add RESTRICT="distcc" capability 2008-11-01 20:57 [gentoo-dev] [PMS] Add RESTRICT="distcc" capability Gordon Malm @ 2008-11-01 21:00 ` Ciaran McCreesh 2008-11-01 21:21 ` Gordon Malm 2008-11-02 0:11 ` David Leverton 1 sibling, 1 reply; 18+ messages in thread From: Ciaran McCreesh @ 2008-11-01 21:00 UTC (permalink / raw To: gentoo-dev [-- Attachment #1: Type: text/plain, Size: 331 bytes --] On Sat, 1 Nov 2008 13:57:17 -0700 Gordon Malm <gengor@gentoo.org> wrote: > But in the case of out-of-tree kernel modules the idea > of distributing compile jobs to other machines is fundamentally > flawed IMO. Why? And how are out of tree kernel modules in any way special when it comes to distcc? -- Ciaran McCreesh [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 197 bytes --] ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [gentoo-dev] [PMS] Add RESTRICT="distcc" capability 2008-11-01 21:00 ` Ciaran McCreesh @ 2008-11-01 21:21 ` Gordon Malm 2008-11-01 21:28 ` Ciaran McCreesh 0 siblings, 1 reply; 18+ messages in thread From: Gordon Malm @ 2008-11-01 21:21 UTC (permalink / raw To: gentoo-dev If you're compiling an out-of-tree module that requires the kernel be compiled with support for a particular item and the distccd host's kernel does not have that support compiles fail. Reference bug #120001 (though I know that it was properly diagnosed there). Then there's also this doozie. -> #167844. Thanks, Gordon Malm (gengor) On Saturday, November 1, 2008 14:00:17 Ciaran McCreesh wrote: > On Sat, 1 Nov 2008 13:57:17 -0700 > > Gordon Malm <gengor@gentoo.org> wrote: > > But in the case of out-of-tree kernel modules the idea > > of distributing compile jobs to other machines is fundamentally > > flawed IMO. > > Why? And how are out of tree kernel modules in any way special when it > comes to distcc? ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [gentoo-dev] [PMS] Add RESTRICT="distcc" capability 2008-11-01 21:21 ` Gordon Malm @ 2008-11-01 21:28 ` Ciaran McCreesh 2008-11-01 21:58 ` Gordon Malm 0 siblings, 1 reply; 18+ messages in thread From: Ciaran McCreesh @ 2008-11-01 21:28 UTC (permalink / raw To: gentoo-dev [-- Attachment #1: Type: text/plain, Size: 742 bytes --] On Sat, 1 Nov 2008 14:21:43 -0700 Gordon Malm <gengor@gentoo.org> wrote: > If you're compiling an out-of-tree module that requires the kernel be > compiled with support for a particular item and the distccd host's > kernel does not have that support compiles fail. Reference bug > #120001 (though I know that it was properly diagnosed there). > > Then there's also this doozie. -> #167844. So far as I can see, when the correct options are passed to a build system, the only issue is hardened being broken. There doesn't seem to be any fundamental reason why distcc with a sane compiler can't be used with kernel modules. If this is the case, wouldn't it be better to get the hardened compiler fixed? -- Ciaran McCreesh [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 197 bytes --] ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [gentoo-dev] [PMS] Add RESTRICT="distcc" capability 2008-11-01 21:28 ` Ciaran McCreesh @ 2008-11-01 21:58 ` Gordon Malm 2008-11-01 22:11 ` Ciaran McCreesh 0 siblings, 1 reply; 18+ messages in thread From: Gordon Malm @ 2008-11-01 21:58 UTC (permalink / raw To: gentoo-dev On Saturday, November 1, 2008 14:28:06 Ciaran McCreesh wrote: > On Sat, 1 Nov 2008 14:21:43 -0700 > > Gordon Malm <gengor@gentoo.org> wrote: > > If you're compiling an out-of-tree module that requires the kernel be > > compiled with support for a particular item and the distccd host's > > kernel does not have that support compiles fail. Reference bug > > #120001 (though I know that it was properly diagnosed there). > > > > Then there's also this doozie. -> #167844. > > So far as I can see, when the correct options are passed to a build > system, the only issue is hardened being broken. There doesn't seem to > be any fundamental reason why distcc with a sane compiler can't be used > with kernel modules. If this is the case, wouldn't it be better to get > the hardened compiler fixed? I use madwifi-ng extensively and have experienced the same issue with madwifi-ng as stated in that bug. For bug #167844, please read comment #13 and http://code.google.com/p/distcc/issues/detail?id=25. There's nothing to fix, hardened is doing the right thing as is distcc. The proper approach is to not distribute compiles of kernel modules. To not get too stuck on a single scenario, I think this would be a useful feature and desireable vs. the alternative of FEATURES=${FEATURES/distcc/}. This is not the first time its been requested, a simple search reveals bugs #28300, #80894. A quick grep shows these packages at minimum have to do some FEATURES checking and either pass a config option (if provided) or die "disable distcc and try again". dev-python/pyqwt media-gfx/sam2p media-tv/mythtv media-video/avidemux Gordon Malm (gengor) ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [gentoo-dev] [PMS] Add RESTRICT="distcc" capability 2008-11-01 21:58 ` Gordon Malm @ 2008-11-01 22:11 ` Ciaran McCreesh 2008-11-01 22:47 ` Gordon Malm 0 siblings, 1 reply; 18+ messages in thread From: Ciaran McCreesh @ 2008-11-01 22:11 UTC (permalink / raw To: gentoo-dev [-- Attachment #1: Type: text/plain, Size: 1233 bytes --] On Sat, 1 Nov 2008 14:58:39 -0700 Gordon Malm <gengor@gentoo.org> wrote: > I use madwifi-ng extensively and have experienced the same issue with > madwifi-ng as stated in that bug. For bug #167844, please read > comment #13 and http://code.google.com/p/distcc/issues/detail?id=25. > There's nothing to fix, hardened is doing the right thing as is > distcc. The proper approach is to not distribute compiles of kernel > modules. It looks to me like hardened is doing entirely the wrong thing. Thus, the proper fix is to make hardened behave itself. > To not get too stuck on a single scenario, I think this would be a > useful feature and desireable vs. the alternative of > FEATURES=${FEATURES/distcc/}. > > This is not the first time its been requested, a simple search > reveals bugs #28300, #80894. It looks a lot like people are blaming distcc for parallelisation bugs that aren't distcc related but that happen to be easier to reproduce when distcc is enabled. Do you have any examples of problems that are definitely caused by distcc, rather than general parallelisation bugs or user misconfigurations? RESTRICTing distcc doesn't seem to be an actual fix for anything... -- Ciaran McCreesh [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 197 bytes --] ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [gentoo-dev] [PMS] Add RESTRICT="distcc" capability 2008-11-01 22:11 ` Ciaran McCreesh @ 2008-11-01 22:47 ` Gordon Malm 2008-11-01 22:57 ` Ciaran McCreesh 2008-11-01 23:28 ` Jan Kundrát 0 siblings, 2 replies; 18+ messages in thread From: Gordon Malm @ 2008-11-01 22:47 UTC (permalink / raw To: gentoo-dev On Saturday, November 1, 2008 15:11:16 Ciaran McCreesh wrote: > On Sat, 1 Nov 2008 14:58:39 -0700 > > Gordon Malm <gengor@gentoo.org> wrote: > > I use madwifi-ng extensively and have experienced the same issue with > > madwifi-ng as stated in that bug. For bug #167844, please read > > comment #13 and http://code.google.com/p/distcc/issues/detail?id=25. > > There's nothing to fix, hardened is doing the right thing as is > > distcc. The proper approach is to not distribute compiles of kernel > > modules. > > It looks to me like hardened is doing entirely the wrong thing. Thus, > the proper fix is to make hardened behave itself. It looks to me like you've already made up your mind. How is hardened doing the entirely wrong thing? What do you propose can be done to "fix" the hardened compiler? What about madwifi-ng? You are getting increasingly narrow in your points of objection. > > > To not get too stuck on a single scenario, I think this would be a > > useful feature and desireable vs. the alternative of > > FEATURES=${FEATURES/distcc/}. > > > > This is not the first time its been requested, a simple search > > reveals bugs #28300, #80894. > > It looks a lot like people are blaming distcc for parallelisation bugs > that aren't distcc related but that happen to be easier to reproduce > when distcc is enabled. Do you have any examples of problems that are > definitely caused by distcc, rather than general parallelisation bugs > or user misconfigurations? RESTRICTing distcc doesn't seem to be an > actual fix for anything... As I said in my first post: 'It is true that RESTRICT="distcc" is usually not the proper solution to problems.' Parallel building problems can often and should be addressed properly. I don't want the answer to every one that comes along to be to add RESTRICT="distcc". This is something to be addressed through developer documentation that using RESTRICT="distcc" should be a last resort. However, in practice making a package parallel-make safe isn't always trivial. So what happens in these cases is FEATURES=distcc && die check is put in place killing the emerge chain and requiring user intervention. Either that or the bug just lingers and the compile fails somewhere in the middle... I don't know about palaudis but this is like a one line patch to portage. But silly me, I thought the package manager was there to support the distribution. Gordon Malm (gengor) ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [gentoo-dev] [PMS] Add RESTRICT="distcc" capability 2008-11-01 22:47 ` Gordon Malm @ 2008-11-01 22:57 ` Ciaran McCreesh 2008-11-01 23:28 ` Jan Kundrát 1 sibling, 0 replies; 18+ messages in thread From: Ciaran McCreesh @ 2008-11-01 22:57 UTC (permalink / raw To: gentoo-dev [-- Attachment #1: Type: text/plain, Size: 2253 bytes --] On Sat, 1 Nov 2008 15:47:09 -0700 Gordon Malm <gengor@gentoo.org> wrote: > > It looks to me like hardened is doing entirely the wrong thing. > > Thus, the proper fix is to make hardened behave itself. > > It looks to me like you've already made up your mind. How is > hardened doing the entirely wrong thing? What do you propose can be > done to "fix" the hardened compiler? What about madwifi-ng? You are > getting increasingly narrow in your points of objection. I suggest you get the hardened upstream people to stop abusing the -D switch to gcc. The distcc people suggest the same. > Parallel building problems can often and should be addressed > properly. I don't want the answer to every one that comes along to > be to add RESTRICT="distcc". This is something to be addressed > through developer documentation that using RESTRICT="distcc" should > be a last resort. Uh, RESTRICT=distcc won't even fix parallel make problems. It'll just make them harder to reproduce on some systems. > However, in practice making a package parallel-make safe isn't always > trivial. So what happens in these cases is FEATURES=distcc && die > check is put in place killing the emerge chain and requiring user > intervention. Either that or the bug just lingers and the compile > fails somewhere in the middle... ...or you could use -j1, which whilst being horrible will at least work. > I don't know about palaudis but this is like a one line patch to > portage. But silly me, I thought the package manager was there to > support the distribution. You have yet to demonstrate how RESTRICT=distcc will help. In fact, you seem to be demonstrating that all it'll do is make a few people apply a 'fix' that won't reliably fix anything. *If* there's a legitimate use for RESTRICT=distcc then I am entirely in favour of it. But it looks like there isn't, with every issue being either a parallelism issue (which RESTRICT=distcc won't fix), a user configuration issue (which RESTRICT=distcc won't fix) or a hardened toolchain bug (for which RESTRICT=distcc is massive overkill, and thus the wrong solution). You've decided upon a solution before you've worked out what the problem is. -- Ciaran McCreesh [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 197 bytes --] ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [gentoo-dev] [PMS] Add RESTRICT="distcc" capability 2008-11-01 22:47 ` Gordon Malm 2008-11-01 22:57 ` Ciaran McCreesh @ 2008-11-01 23:28 ` Jan Kundrát 2008-11-02 1:29 ` Gordon Malm 1 sibling, 1 reply; 18+ messages in thread From: Jan Kundrát @ 2008-11-01 23:28 UTC (permalink / raw To: gentoo-dev [-- Attachment #1: Type: text/plain, Size: 782 bytes --] Gordon Malm wrote: > It looks to me like you've already made up your mind. How is hardened doing > the entirely wrong thing? From the page [1] you mentioned: "If so, that seems to me like an abuse of the -D option." The abuse is in changing the compiler behavior based on -D options. > What do you propose can be done to "fix" the > hardened compiler? From the same page: "It would be better for you to remove the patch from gcc where it makes -D__KERNEL__ imply -nossp -nopie, and to instead patch the Linux kernel build system (Makefiles, etc.) so that it passes "-D__KERNEL__ -nossp -nopie" rather than "-D__KERNEL__"." [1] http://code.google.com/p/distcc/issues/detail?id=25 Cheers, -jkt -- cd /local/pub && more beer > /dev/mouth [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 260 bytes --] ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [gentoo-dev] [PMS] Add RESTRICT="distcc" capability 2008-11-01 23:28 ` Jan Kundrát @ 2008-11-02 1:29 ` Gordon Malm 2008-11-02 3:16 ` Jorge Manuel B. S. Vicetto 2008-11-02 11:26 ` Ciaran McCreesh 0 siblings, 2 replies; 18+ messages in thread From: Gordon Malm @ 2008-11-02 1:29 UTC (permalink / raw To: gentoo-dev On Saturday, November 1, 2008 15:57:52 Ciaran McCreesh wrote: > > Parallel building problems can often and should be addressed > > properly. I don't want the answer to every one that comes along to > > be to add RESTRICT="distcc". This is something to be addressed > > through developer documentation that using RESTRICT="distcc" should > > be a last resort. > > Uh, RESTRICT=distcc won't even fix parallel make problems. It'll just > make them harder to reproduce on some systems. Why don't you post the whole context? Here, I'll do it for you: On Saturday, November 1, 2008 15:11:16 Ciaran McCreesh wrote: > > It looks a lot like people are blaming distcc for parallelisation bugs > > that aren't distcc related but that happen to be easier to reproduce > > when distcc is enabled. Do you have any examples of problems that are > > definitely caused by distcc, rather than general parallelisation bugs > > or user misconfigurations? RESTRICTing distcc doesn't seem to be an > > actual fix for anything... > And my full response... > As I said in my first post: > > 'It is true that RESTRICT="distcc" is usually not the proper solution to > problems.' > > Parallel building problems can often and should be addressed properly. I > don't want the answer to every one that comes along to be to add > RESTRICT="distcc". This is something to be addressed through developer > documentation that using RESTRICT="distcc" should be a last resort. You're the one assuming the only purpose would be to mask parallel make problems. Apparently it does have a purpose though since avidemux builds fine in parallel but NOT via distcc. Back to your most recent post > *If* there's a legitimate use for RESTRICT=distcc then I am entirely in > favour of it. But it looks like there isn't, with every issue being > either a parallelism issue (which RESTRICT=distcc won't fix), a user > configuration issue (which RESTRICT=distcc won't fix) or a hardened > toolchain bug (for which RESTRICT=distcc is massive overkill, and thus > the wrong solution). You've decided upon a solution before you've > worked out what the problem is. You assumed it is a parallelism issue that people are trying to solve. I haven't pointed to any user configuration issues. Using RESTRICT=distcc on kernel modules is hardly overkill. This isn't openoffice. I know exactly what the problem is, but since you have such a better grasp on it.... On Saturday, November 1, 2008 15:57:52 Ciaran McCreesh wrote: > On Sat, 1 Nov 2008 15:47:09 -0700 > > Gordon Malm <gengor@gentoo.org> wrote: > > > It looks to me like hardened is doing entirely the wrong thing. > > > Thus, the proper fix is to make hardened behave itself. > > > > It looks to me like you've already made up your mind. How is > > hardened doing the entirely wrong thing? What do you propose can be > > done to "fix" the hardened compiler? What about madwifi-ng? You are > > getting increasingly narrow in your points of objection. > > I suggest you get the hardened upstream people to stop abusing the -D > switch to gcc. The distcc people suggest the same. On Saturday, November 1, 2008 16:28:17 Jan Kundrát wrote: > Gordon Malm wrote: > > It looks to me like you've already made up your mind. How is hardened > > doing the entirely wrong thing? > > From the page [1] you mentioned: > > "If so, that seems to me like an abuse of the -D option." > > The abuse is in changing the compiler behavior based on -D options. > > > What do you propose can be done to "fix" the > > hardened compiler? > > From the same page: > > "It would be better for you to remove the patch from gcc where it makes > -D__KERNEL__ imply -nossp -nopie, and to instead patch the Linux kernel > build system (Makefiles, etc.) so that it passes "-D__KERNEL__ -nossp > -nopie" rather than "-D__KERNEL__"." > We have to build using different specs sets for kernel code than userland. If we can't depend on the __KERNEL__ macro for detection, how else do you propose one detect if gcc is building kernel code or not? Patching an out-of-tree module's build system to pass -fno-PIE works on x86, I don't know how this will affect other ARCHes, do either of you? This could potentially get really tricky. If it can't be done nicely in the eclass for every possible kernel release and out-of-tree module, then we get into patching *everything* and having to maintain it forward. So this is just a different workaround, a rather heavy-handed and high-maintenance one at that. What makes it any better than a simple option to RESTRICT distcc? I guess the larger question in all this is why does a third party who has demonstrated his anti-Hardened (and anti-Gentoo) slant on multiple occasions define what goes in our PMS? Gordon Malm (gengor) ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [gentoo-dev] [PMS] Add RESTRICT="distcc" capability 2008-11-02 1:29 ` Gordon Malm @ 2008-11-02 3:16 ` Jorge Manuel B. S. Vicetto 2008-11-02 11:26 ` Ciaran McCreesh 1 sibling, 0 replies; 18+ messages in thread From: Jorge Manuel B. S. Vicetto @ 2008-11-02 3:16 UTC (permalink / raw To: gentoo-dev -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Gordon Malm wrote: <snip a lot of exchanges> All the technical discussion on the above is perfectly fine, but the way the arguments are being presented and the tone used by both sides is getting arguments into a thin line from becoming flames. Please step back before turning this into another flame festival. > I guess the larger question in all this is why does a third party who has > demonstrated his anti-Hardened (and anti-Gentoo) slant on multiple occasions > define what goes in our PMS? It's not up to a 3rd party to define what will be or not in PMS. Any 3rd party is free to contribute to the discussion (within boundaries), but it's up to the PMs folks to reach agreements about PMS and, or if they can't, up to the Gentoo Council. > Gordon Malm (gengor) > - -- Regards, Jorge Vicetto (jmbsvicetto) - jmbsvicetto at gentoo dot org Gentoo- forums / Userrel / Devrel / SPARC / KDE -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkkNG6IACgkQcAWygvVEyAKOrgCghGO8W2839jXMaMq9DN3DpWbM JJMAn0PhLDiKoBayr1juI48tzwa8j8Wc =EXnX -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [gentoo-dev] [PMS] Add RESTRICT="distcc" capability 2008-11-02 1:29 ` Gordon Malm 2008-11-02 3:16 ` Jorge Manuel B. S. Vicetto @ 2008-11-02 11:26 ` Ciaran McCreesh 2008-11-02 19:11 ` Gordon Malm 1 sibling, 1 reply; 18+ messages in thread From: Ciaran McCreesh @ 2008-11-02 11:26 UTC (permalink / raw To: gentoo-dev [-- Attachment #1: Type: text/plain, Size: 2573 bytes --] On Sat, 1 Nov 2008 18:29:03 -0700 Gordon Malm <gengor@gentoo.org> wrote: > You're the one assuming the only purpose would be to mask parallel > make problems. Apparently it does have a purpose though since > avidemux builds fine in parallel but NOT via distcc. Have you conclusively established that they do build fine in parallel? If so, how? And why do they break in parallel only under distcc? Given how distcc works, this strikes me as somewhat implausible... > Back to your most recent post > > *If* there's a legitimate use for RESTRICT=distcc then I am > > entirely in favour of it. But it looks like there isn't, with every > > issue being either a parallelism issue (which RESTRICT=distcc won't > > fix), a user configuration issue (which RESTRICT=distcc won't fix) > > or a hardened toolchain bug (for which RESTRICT=distcc is massive > > overkill, and thus the wrong solution). You've decided upon a > > solution before you've worked out what the problem is. > > You assumed it is a parallelism issue that people are trying to > solve. I haven't pointed to any user configuration issues. Using > RESTRICT=distcc on kernel modules is hardly overkill. This isn't > openoffice. I know exactly what the problem is, but since you have > such a better grasp on it.... RESTRICT=distcc on kernel modules is unnecessary for the large proportion of users who don't use hardened. RESTRICTs can't be set dependent upon whether or not something like hardenened is enabled; other mechanisms are available that can be. So why aren't you considering those other mechanisms? > We have to build using different specs sets for kernel code than > userland. If we can't depend on the __KERNEL__ macro for detection, > how else do you propose one detect if gcc is building kernel code or > not? A macro is just a macro. All it does is affect the preprocessor stage. Hardened is trying to abuse it for something else entirely, which is why you're encountering problems. > What makes it any better than a simple option to RESTRICT distcc? You still appear to be confusing "RESTRICT distcc" and "provide a mechanism for selectively disabling distcc". They are entirely different things. > I guess the larger question in all this is why does a third party who > has demonstrated his anti-Hardened (and anti-Gentoo) slant on > multiple occasions define what goes in our PMS? Uh, that's your answer to technical objections to a proposal? You aren't prepared to defend your proposal on its merits? -- Ciaran McCreesh [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 197 bytes --] ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [gentoo-dev] [PMS] Add RESTRICT="distcc" capability 2008-11-02 11:26 ` Ciaran McCreesh @ 2008-11-02 19:11 ` Gordon Malm 2008-11-02 19:28 ` Ciaran McCreesh 2008-11-03 10:22 ` [gentoo-dev] " Peter Volkov 0 siblings, 2 replies; 18+ messages in thread From: Gordon Malm @ 2008-11-02 19:11 UTC (permalink / raw To: gentoo-dev On Sunday, November 2, 2008 03:26:14 Ciaran McCreesh wrote: > On Sat, 1 Nov 2008 18:29:03 -0700 > > Gordon Malm <gengor@gentoo.org> wrote: > > You're the one assuming the only purpose would be to mask parallel > > make problems. Apparently it does have a purpose though since > > avidemux builds fine in parallel but NOT via distcc. > > Have you conclusively established that they do build fine in parallel? > If so, how? And why do they break in parallel only under distcc? Given > how distcc works, this strikes me as somewhat implausible... Yes it builds in parallel. By compiling it in parallel. I don't know why and don't care to investigate for you. It's not my package, I'm not QA and my team is short staffed enough as it is. We have to take care of our own backyard. Maybe you could investigate the answers to your own questions? How about you at least feign attempt at answering my questions for a change? > > > Back to your most recent post > > > > > *If* there's a legitimate use for RESTRICT=distcc then I am > > > entirely in favour of it. But it looks like there isn't, with every > > > issue being either a parallelism issue (which RESTRICT=distcc won't > > > fix), a user configuration issue (which RESTRICT=distcc won't fix) > > > or a hardened toolchain bug (for which RESTRICT=distcc is massive > > > overkill, and thus the wrong solution). You've decided upon a > > > solution before you've worked out what the problem is. > > > > You assumed it is a parallelism issue that people are trying to > > solve. I haven't pointed to any user configuration issues. Using > > RESTRICT=distcc on kernel modules is hardly overkill. This isn't > > openoffice. I know exactly what the problem is, but since you have > > such a better grasp on it.... > > RESTRICT=distcc on kernel modules is unnecessary for the large > proportion of users who don't use hardened. RESTRICTs can't be set > dependent upon whether or not something like hardenened is enabled; > other mechanisms are available that can be. So why aren't you > considering those other mechanisms? I agree with the first sentence, never said otherwise. Who says I am not considering them? In fact, I have stated that I am. They are just more hackish than adding the ability to RESTRICT distcc, hence my reason for soliciting such a feature. I'm surprised that you'd suggest this after debasing a RESTRICT option on the same grounds. > > > We have to build using different specs sets for kernel code than > > userland. If we can't depend on the __KERNEL__ macro for detection, > > how else do you propose one detect if gcc is building kernel code or > > not? > > A macro is just a macro. All it does is affect the preprocessor stage. > Hardened is trying to abuse it for something else entirely, which is why > you're encountering problems. You can cry "abuse" all you want. You FAIL to offer any alternatives or solutions. I'll ask again, how do you detect that you are compiling code destined to be run in-kernel from within gcc without checking for the __KERNEL__ macro? I think we both know the answer. We can't just hack the eclass to pass -fno-PIE because a particular package might build more than just the kernel module itself. Patching kbuild in our sources doesn't address any modules that don't use in-tree kbuild and is stupid in so many ways anyway. That leaves patching every out-of-tree module in portage manually and maintain those patches forward which would be very time/effort intensive on an ongoing basis. Go ahead and forget about it though. I'm done with this discussion. Because we rely on the __KERNEL__ macro to determine which flags to apply, it only makes sense to patch in-portage distcc in the case of hardened to pass it. And that's exactly what I'll be doing. Problem solved and I like that option better anyway. Sorry to everyone who e-mailed me who thought RESTRICT=distcc would be somewhat useful. > > > I guess the larger question in all this is why does a third party who > > has demonstrated his anti-Hardened (and anti-Gentoo) slant on > > multiple occasions define what goes in our PMS? > > Uh, that's your answer to technical objections to a proposal? You > aren't prepared to defend your proposal on its merits? Why are you assuming this point is *at all* related to my proposal? It's not. It's about Gentoo. But I stand corrected, a bunch of people rushed to tell me you don't actually determine anything. ;) Gordon Malm (gengor) P.S. I've seen enough of these threads from you to know that you can't help but respond with another post that does a lot of objecting but offers no solutions. Go ahead and post again so you can feel like you had the last word, now is your chance, I won't be posting back. ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [gentoo-dev] [PMS] Add RESTRICT="distcc" capability 2008-11-02 19:11 ` Gordon Malm @ 2008-11-02 19:28 ` Ciaran McCreesh 2008-11-03 8:18 ` [gentoo-dev] " Steve Long 2008-11-03 10:22 ` [gentoo-dev] " Peter Volkov 1 sibling, 1 reply; 18+ messages in thread From: Ciaran McCreesh @ 2008-11-02 19:28 UTC (permalink / raw To: gentoo-dev [-- Attachment #1: Type: text/plain, Size: 3896 bytes --] On Sun, 2 Nov 2008 12:11:10 -0700 Gordon Malm <gengor@gentoo.org> wrote: > > Have you conclusively established that they do build fine in > > parallel? If so, how? And why do they break in parallel only under > > distcc? Given how distcc works, this strikes me as somewhat > > implausible... > > Yes it builds in parallel. By compiling it in parallel. I don't > know why and don't care to investigate for you. It's not my package, > I'm not QA and my team is short staffed enough as it is. We have to > take care of our own backyard. Maybe you could investigate the > answers to your own questions? How about you at least feign attempt > at answering my questions for a change? If you think compiling it in parallel confirms that it builds in parallel, you're reaffirming my growing suspicion that you're entirely wrong about distcc being responsible for anything other than hardened brokenness... > > RESTRICT=distcc on kernel modules is unnecessary for the large > > proportion of users who don't use hardened. RESTRICTs can't be set > > dependent upon whether or not something like hardenened is enabled; > > other mechanisms are available that can be. So why aren't you > > considering those other mechanisms? > > I agree with the first sentence, never said otherwise. Who says I am > not considering them? In fact, I have stated that I am. They are > just more hackish than adding the ability to RESTRICT distcc, hence > my reason for soliciting such a feature. I'm surprised that you'd > suggest this after debasing a RESTRICT option on the same grounds. And that brings us to the other thing you don't understand: RESTRICT is a global, system independent, invariant metadata key. Things in RESTRICT are in RESTRICT because they have to be known in situations where those constraints are in effect. Things that do not have to be known at the metadata level shouldn't be in metadata. > > A macro is just a macro. All it does is affect the preprocessor > > stage. Hardened is trying to abuse it for something else entirely, > > which is why you're encountering problems. > > You can cry "abuse" all you want. You FAIL to offer any alternatives > or solutions. I'll ask again, how do you detect that you are > compiling code destined to be run in-kernel from within gcc without > checking for the __KERNEL__ macro? I think we both know the answer. I suggest you talk to hardened upstream and see what they have to say about it. What have they offered as justification for using -D in a way in which no-one else uses it? What are their objections to switching to an approach that doesn't break the preprocessor model? > Sorry to everyone who e-mailed me who thought RESTRICT=distcc would > be somewhat useful. Those people are mistaken. > > Uh, that's your answer to technical objections to a proposal? You > > aren't prepared to defend your proposal on its merits? > > Why are you assuming this point is *at all* related to my proposal? > It's not. It's about Gentoo. But I stand corrected, a bunch of > people rushed to tell me you don't actually determine anything. ;) So you don't care about whether your solution is right? You are proposing to add a metadata invariant option for a condition that isn't metadata invariant and that, by being made metadata invariant, means it's being disabled for everyone rather than by the small number of users who might need it. In addition, you can't demonstrate any cases where this option is genuinely useful, other than as a workaround for a hardened bug that you haven't contacted upstream about, and when used to work around said hardened bug the scope of the change isn't limited to hardened. You still haven't explained why you don't do something like: broken_hardened_compiler && export DISTCC_HOSTS=localhost -- Ciaran McCreesh [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 197 bytes --] ^ permalink raw reply [flat|nested] 18+ messages in thread
* [gentoo-dev] Re: [PMS] Add RESTRICT="distcc" capability 2008-11-02 19:28 ` Ciaran McCreesh @ 2008-11-03 8:18 ` Steve Long 0 siblings, 0 replies; 18+ messages in thread From: Steve Long @ 2008-11-03 8:18 UTC (permalink / raw To: gentoo-dev Ciaran McCreesh wrote: > On Sun, 2 Nov 2008 12:11:10 -0700 > Gordon Malm <gengor@gentoo.org> wrote: >> > Have you conclusively established that they do build fine in >> > parallel? If so, how? >> Yes it builds in parallel. By compiling it in parallel. > If you think compiling it in parallel confirms that it builds in > parallel, you're reaffirming my growing suspicion that you're entirely > wrong about distcc being responsible for anything other than hardened > brokenness... > Well my understanding of parallel make is that it would show some issues but not all. I'd hope you were trying to say: "Build it via distcc with N virtual hosts" or sth along those lines. >> > RESTRICT=distcc on kernel modules is unnecessary for the large >> > proportion of users who don't use hardened. RESTRICTs can't be set >> > dependent upon whether or not something like hardenened is enabled; >> > other mechanisms are available that can be. So why aren't you >> > considering those other mechanisms? >> >> I agree with the first sentence, never said otherwise. Who says I am >> not considering them? In fact, I have stated that I am. More why hasn't he proposed them already. >> They are >> just more hackish than adding the ability to RESTRICT distcc, hence >> my reason for soliciting such a feature. I'm surprised that you'd >> suggest this after debasing a RESTRICT option on the same grounds. > > And that brings us to the other thing you don't understand: RESTRICT is > a global, system independent, invariant metadata key. Things in > RESTRICT are in RESTRICT because they have to be known in situations > where those constraints are in effect. "those constraint are in effect" isn't a good way of explaining that imo. > Things that do not have to be > known at the metadata level shouldn't be in metadata. > Yeah, stuff that always applies, no matter what. >> Sorry to everyone who e-mailed me who thought RESTRICT=distcc would >> be somewhat useful. Well a user could always turn off FEATURES externally, which isn't hard to automate[2]; developers are notoriously bad at defining use-cases. > > Those people are mistaken. > >> > Uh, that's your answer to technical objections to a proposal? You >> > aren't prepared to defend your proposal on its merits? I think those two bits go nicely together. It's not supposed to be a fight, btw. >> >> Why are you assuming this point is *at all* related to my proposal? >> It's not. It's about Gentoo. But I stand corrected, a bunch of >> people rushed to tell me you don't actually determine anything. ;) > > So you don't care about whether your solution is right? > *sigh* > You are proposing to add a metadata invariant option for a condition > that isn't metadata invariant and that, by being made metadata > invariant, means it's being disabled for everyone rather than by the > small number of users who might need it. In addition, you can't > demonstrate any cases where this option is genuinely useful, other than > as a workaround for a hardened bug that you haven't contacted upstream > about, and when used to work around said hardened bug the scope of the > change isn't limited to hardened. > I agree this case isn't the best one, nor am I in favour of this RESTRICT. I'm totally neutral on it as a solution. I can envisage wanting to restrict compilation to the local machine, but I'm not bothered about how it gets done. My instinct would be to err toward giving the control to the ebuild maintainer, with a clear QA policy, perhaps around making it something that had to be reviewed on every bump (QA script to watch ebuild as long it has the RESTRICT, unless it's proprietary.) Much as we might want perfect builds, they don't always happen, nor do we always have time to fix upstream bugs, however minor they turn out to be. > You still haven't explained why you don't do something like: > > broken_hardened_compiler && export DISTCC_HOSTS=localhost > Still it would have been easier on the reader if you'd just mentioned this first. [1] http://hardened.gentooexperimental.org/trac/secure/ [2] http://forums.gentoo.org/viewtopic-t-546828.html ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [gentoo-dev] [PMS] Add RESTRICT="distcc" capability 2008-11-02 19:11 ` Gordon Malm 2008-11-02 19:28 ` Ciaran McCreesh @ 2008-11-03 10:22 ` Peter Volkov 2008-11-04 3:37 ` Gordon Malm 1 sibling, 1 reply; 18+ messages in thread From: Peter Volkov @ 2008-11-03 10:22 UTC (permalink / raw To: gentoo-dev В Вск, 02/11/2008 в 12:11 -0700, Gordon Malm пишет: > You can cry "abuse" all you want. You FAIL to offer any alternatives or > solutions. I'll ask again, how do you detect that you are compiling code > destined to be run in-kernel from within gcc without checking for the > __KERNEL__ macro? I think we both know the answer. Gordon, as far as I see our linux-mod.eclass builds each kernel module separately and it is possible to modify it to pass CFLAGS you wish for kernel modules only and as such avoid using __KERNEL__ macro for this... -- Peter. ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [gentoo-dev] [PMS] Add RESTRICT="distcc" capability 2008-11-03 10:22 ` [gentoo-dev] " Peter Volkov @ 2008-11-04 3:37 ` Gordon Malm 0 siblings, 0 replies; 18+ messages in thread From: Gordon Malm @ 2008-11-04 3:37 UTC (permalink / raw To: gentoo-dev On Monday, November 3, 2008 02:22:12 Peter Volkov wrote: > В Вск, 02/11/2008 в 12:11 -0700, Gordon Malm пишет: > > You can cry "abuse" all you want. You FAIL to offer any alternatives or > > solutions. I'll ask again, how do you detect that you are compiling code > > destined to be run in-kernel from within gcc without checking for the > > __KERNEL__ macro? I think we both know the answer. > > Gordon, as far as I see our linux-mod.eclass builds each kernel module > separately and it is possible to modify it to pass CFLAGS you wish for > kernel modules only and as such avoid using __KERNEL__ macro for this... Peter, with much respect, this doesn't really address the question. Here is the relevant snippet of hardened's gcc specs: %(cc1_cpu) %{profile:-p} %{!D__KERNEL__: %{!static: %{!fno-PIC: %{!fno-pic: %{!shared: %{!nostdlib: %{!nostartfiles: %{!fno-PIE: %{!fno-pie: %{!nopie: %{!fPIC: %{!fpic:-fPIE}}} } } } } } } } } %{!nostdlib: %{!fno-stack-protector: -fstack-protector %{!D_LIBC: %{!D_LIBC_REENTRANT: %{!fno-stack-protector-all: -fstack-protector-all} } } } } } As you can see the __KERNEL__ macro is checked to decide which gcc specs to apply any time hardened's gcc compiles anything. Patching linux-mod.eclass isn't going to do anything to break our reliance on the __KERNEL__ macro. What patching linux-mod.eclass *would* allow us to do is not patch distcc to pass -D__KERNEL__ (which I have already done in the case of USE="hardened", so really this is solved). The limitations of the linux-mod.eclass patching approach is that it wouldn't work for any modules that use a standalone build system (dumb as it that might be) instead of kbuild or any module being compiled outside of portage (kbuild or not). The linux-mod.eclass hacking could get pretty ugly. I haven't looked deeply into it but KBUILD_CFLAGS is not yet defined at the time emake is called that I can tell, so I don't think it will be as simple as adding something like KBUILD_CFLAGS="${KBUILD_CFLAGS} -fmy -fflags -fhere". If I'm wrong please let me know, I'd be grateful. :) There's potentially a lot of conditionals involved as well as far as what flags to add for a diversity of ARCHes, what the available out-of-tree modules want for themselves, etc. Overall, linux-mod.eclass could start to look really nasty and I think it would take quite awhile (for me alone at least) to figure out a workable solution, if even possible. When accounting that this would only address the problem for modules using Kbuild and built within portage - I don't think its worth it for a partial solution. Patching KBUILD_CFLAGS in the kernel sources themselves works, but if we don't do that for every kernel portage we restrict users to using hardened-sources when running a hardened toolchain w/o the hardened kernel is a perfectly valid configuration. They wouldn't be able to use a non-gentoo provided kernel without patching it as well. Plus, we're just *not* going to have every kernel in the tree have a patch conditional on USE="hardened" (ick :). Best regards, Gordon Malm (gengor) ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [gentoo-dev] [PMS] Add RESTRICT="distcc" capability 2008-11-01 20:57 [gentoo-dev] [PMS] Add RESTRICT="distcc" capability Gordon Malm 2008-11-01 21:00 ` Ciaran McCreesh @ 2008-11-02 0:11 ` David Leverton 1 sibling, 0 replies; 18+ messages in thread From: David Leverton @ 2008-11-02 0:11 UTC (permalink / raw To: gentoo-dev On Saturday 01 November 2008 20:57:17 Gordon Malm wrote: > I'd like to get "distcc" added as one of the FEATURES we are able to > RESTRICT. Regardless of whether it's a good idea or not, does it fix all the known issues if the ebuild sets DISTCC_HOSTS="localhost" in the environment? ^ permalink raw reply [flat|nested] 18+ messages in thread
end of thread, other threads:[~2008-11-04 3:37 UTC | newest] Thread overview: 18+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2008-11-01 20:57 [gentoo-dev] [PMS] Add RESTRICT="distcc" capability Gordon Malm 2008-11-01 21:00 ` Ciaran McCreesh 2008-11-01 21:21 ` Gordon Malm 2008-11-01 21:28 ` Ciaran McCreesh 2008-11-01 21:58 ` Gordon Malm 2008-11-01 22:11 ` Ciaran McCreesh 2008-11-01 22:47 ` Gordon Malm 2008-11-01 22:57 ` Ciaran McCreesh 2008-11-01 23:28 ` Jan Kundrát 2008-11-02 1:29 ` Gordon Malm 2008-11-02 3:16 ` Jorge Manuel B. S. Vicetto 2008-11-02 11:26 ` Ciaran McCreesh 2008-11-02 19:11 ` Gordon Malm 2008-11-02 19:28 ` Ciaran McCreesh 2008-11-03 8:18 ` [gentoo-dev] " Steve Long 2008-11-03 10:22 ` [gentoo-dev] " Peter Volkov 2008-11-04 3:37 ` Gordon Malm 2008-11-02 0:11 ` David Leverton
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox