* [gentoo-dev] [RFC] RESTRICT=parallel for builds that can't be executed in parallel @ 2010-04-14 2:12 Zac Medico 2010-04-14 5:45 ` Michał Górny ` (2 more replies) 0 siblings, 3 replies; 16+ messages in thread From: Zac Medico @ 2010-04-14 2:12 UTC (permalink / raw To: Gentoo Dev; +Cc: Brian Harring Hi everyone, Should we add a RESTRICT=parallel value for ebuilds that can't be built at the same time as other ebuilds? Brian says we need it for things like xorg-server which calls eselect opengl. If we truly need this, is RESTRICT=parallel a good name? We could make it a PROPERTIES value instead, but then we'd probably end up with something like PROPERTIES=no-parallel, so maybe RESTRICT is better. -- Thanks, Zac ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [gentoo-dev] [RFC] RESTRICT=parallel for builds that can't be executed in parallel 2010-04-14 2:12 [gentoo-dev] [RFC] RESTRICT=parallel for builds that can't be executed in parallel Zac Medico @ 2010-04-14 5:45 ` Michał Górny 2010-04-14 6:10 ` Brian Harring 2010-04-14 6:46 ` [gentoo-dev] [RFC] RESTRICT=parallel for builds that can't be executed in parallel Justin 2010-04-21 10:38 ` Luca Barbato 2 siblings, 1 reply; 16+ messages in thread From: Michał Górny @ 2010-04-14 5:45 UTC (permalink / raw To: gentoo-dev On Tue, 13 Apr 2010 19:12:08 -0700 Zac Medico <zmedico@gentoo.org> wrote: > Should we add a RESTRICT=parallel value for ebuilds that can't be > built at the same time as other ebuilds? Brian says we need it for > things like xorg-server which calls eselect opengl. I don't think that's the right solution. In most cases, xorg-server can be built in parallel with stuff which doesn't require/set specific opengl subsystem set. Well, in fact is there _really_ any package which won't work with switched opengl? I guess it's more of a runtime problem that buildtime, and I don't think we really should print out loudly 'libGL has been switched, please do not start OpenGL apps right now'. In fact, the best solution in this particular case would be to patch the buildsystem to use Gentoo location for particular OpenGL headers. Disabling parallel emerge would be more of a workaround for the issue, and will influence much more packages than it needs to. And it won't help if user is running multiple emerge calls at the same time. Another possible workaround is to enable some kind of 'eselect opengl' locking so that another package requiring access to it will wait until our build finishes. But this, of course, would require a quite good solution for maintaining the lock and dropping it whenever build process is aborted/killed. -- Best regards, Michał Górny <http://mgorny.alt.pl> <xmpp:mgorny@jabber.ru> ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [gentoo-dev] [RFC] RESTRICT=parallel for builds that can't be executed in parallel 2010-04-14 5:45 ` Michał Górny @ 2010-04-14 6:10 ` Brian Harring 2010-04-14 7:06 ` [gentoo-dev] " Duncan 2010-04-14 14:20 ` [gentoo-dev] Multiple emerges in parallel (was: [RFC] RESTRICT=parallel for builds that can't be executed in parallel) Michał Górny 0 siblings, 2 replies; 16+ messages in thread From: Brian Harring @ 2010-04-14 6:10 UTC (permalink / raw To: gentoo-dev [-- Attachment #1: Type: text/plain, Size: 3588 bytes --] On Wed, Apr 14, 2010 at 07:45:20AM +0200, Michaaa GGGrny wrote: > On Tue, 13 Apr 2010 19:12:08 -0700 > Zac Medico <zmedico@gentoo.org> wrote: > > > Should we add a RESTRICT=parallel value for ebuilds that can't be > > built at the same time as other ebuilds? Brian says we need it for > > things like xorg-server which calls eselect opengl. > > I don't think that's the right solution. In most cases, xorg-server can > be built in parallel with stuff which doesn't require/set specific > opengl subsystem set. RESTRICT=parallel is basically a big lock that forces building to go down to one specific build/merge job- it's not at all fine grained. That said, I'm not convinced it's worth actually *trying* to be fine grained. Stuff that needs this 'lock', if you look at it from the purely academic angle is broken. The pkgs in question should be buildable without modifying the livefs. From the pragmatic angle, fixing some of those packages is a pretty huge endeavour hence this lock existing. I see no reason to encourage the usage of this lock by making it more fine grained, either. > Well, in fact is there _really_ any package which won't work with > switched opengl? I guess it's more of a runtime problem that buildtime, > and I don't think we really should print out loudly 'libGL has been > switched, please do not start OpenGL apps right now'. Runtime and buildtime actually- consider a pkg that is mesa sensitive having those headers/libs switched out mid build. That build likely is going to boom in a rather interesting way. Runtime itself, swapping out the gl resource that is used (going from ati to x11 for building xorg-server) isn't going to make new apps that start happy either. > In fact, the best solution in this particular case would be to patch > the buildsystem to use Gentoo location for particular OpenGL headers. Academically, you're right, it's the proper solution. That takes a fair amount of time however. More importantly, this issue *will* pop up again elsewhere, meaning we'll have to delay the pkg in question from being in the tree till it meets this higher QA standard. Or we add this functionality, level the restrict, then go and fix the ebuild- pragmatic solution. > Disabling parallel emerge would be more of a workaround for the issue, > and will influence much more packages than it needs to. And it won't > help if user is running multiple emerge calls at the same time. Running multiple emerges in parallel is already a bad idea. The solution for that case is for the new/second emerge to feed the request into the original emerge (or a daemon). Keep in mind if support for multiple emerge invocations was implemented it would still need some RESTRICT=parallel functionality for screwed up pkgs like xorg-server. Fixing multiple emerge invocations still requires fixing RESTRICT=parallel. > Another possible workaround is to enable some kind of 'eselect opengl' > locking so that another package requiring access to it will wait until > our build finishes. But this, of course, would require a quite good > solution for maintaining the lock and dropping it whenever build > process is aborted/killed. The other thing to recall is that by the time eselect is called, the ebuilds environment may already be localized to settings that eselect controls (LDPATH, that 'currently selected implementation' GL var). My first thought is any locking scheme of that sort is going to be a bigger can of worms. ~brian [-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* [gentoo-dev] Re: [RFC] RESTRICT=parallel for builds that can't be executed in parallel 2010-04-14 6:10 ` Brian Harring @ 2010-04-14 7:06 ` Duncan 2010-04-14 9:02 ` Brian Harring 2010-04-14 14:20 ` [gentoo-dev] Multiple emerges in parallel (was: [RFC] RESTRICT=parallel for builds that can't be executed in parallel) Michał Górny 1 sibling, 1 reply; 16+ messages in thread From: Duncan @ 2010-04-14 7:06 UTC (permalink / raw To: gentoo-dev Brian Harring posted on Tue, 13 Apr 2010 23:10:16 -0700 as excerpted: > RESTRICT=parallel is basically a big lock that forces building to go > down to one specific build/merge job- it's not at all fine grained. That > said, I'm not convinced it's worth actually *trying* to be fine grained. > > Stuff that needs this 'lock', if you look at it from the purely academic > angle is broken. The pkgs in question should be buildable without > modifying the livefs. > > From the pragmatic angle, fixing some of those packages is a pretty huge > endeavour hence this lock existing. I see no reason to encourage the > usage of this lock by making it more fine grained, either. What examples of the problem do we have, other than xorg-server due to eselect opengl? For just xorg-server, killing parallel seems like a rather frustrating and indeed broken solution (especially for folks who prefer to run freedomware and thus have only the X11/mesa opengl version on their system anyway, so forcing non-parallel is an exercise in uselessness). If it's the only one, at /least/ only forcing non-parallel if the eselect opengl actually changes the selected opengl would seem reasonable. But if there's other non-contrived examples around, getting a couple of them on the table should I think clarify our potential usage constraints somewhat. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [gentoo-dev] Re: [RFC] RESTRICT=parallel for builds that can't be executed in parallel 2010-04-14 7:06 ` [gentoo-dev] " Duncan @ 2010-04-14 9:02 ` Brian Harring 0 siblings, 0 replies; 16+ messages in thread From: Brian Harring @ 2010-04-14 9:02 UTC (permalink / raw To: gentoo-dev [-- Attachment #1: Type: text/plain, Size: 3581 bytes --] On Wed, Apr 14, 2010 at 07:06:56AM +0000, Duncan wrote: > Brian Harring posted on Tue, 13 Apr 2010 23:10:16 -0700 as excerpted: > > > RESTRICT=parallel is basically a big lock that forces building to go > > down to one specific build/merge job- it's not at all fine grained. That > > said, I'm not convinced it's worth actually *trying* to be fine grained. > > > > Stuff that needs this 'lock', if you look at it from the purely academic > > angle is broken. The pkgs in question should be buildable without > > modifying the livefs. > > > > From the pragmatic angle, fixing some of those packages is a pretty huge > > endeavour hence this lock existing. I see no reason to encourage the > > usage of this lock by making it more fine grained, either. > > What examples of the problem do we have, other than xorg-server due to > eselect opengl? Honestly it's the only hard class of examples I've got- ati-drivers (presumably nvidia also although I've not checked) also would require this. That said, 7 years of pkging crap, I know it's not going to be the *only* example. It mostly comes down the perfect world view vs pragmatic- a lot of what we're doing in terms of ebuild pkging is a balance between best practice and having the thing usable. Sometimes you've got to do something ugly to make it work- especially when the effort required to do it correctly is fairly large. I'm not arguing against doing it correctly, I'm just arguing we should be realistic. > For just xorg-server, killing parallel seems like a rather frustrating and > indeed broken solution (especially for folks who prefer to run freedomware > and thus have only the X11/mesa opengl version on their system anyway, so > forcing non-parallel is an exercise in uselessness). If it's the only > one, at /least/ only forcing non-parallel if the eselect opengl actually > changes the selected opengl would seem reasonable. The thing people need to keep in mind for stuff like this is that metadata is *constant*- it's quite a huge amount of work to write a framework that is able to ask "hey xorg, are you going to go screwing w/ something that means we can't build something in parallel to you building?"- it's not horrible, but it's a lot of effort for questionable gain. If you consider users running ati-drivers, that check would state "yes, lock it" for building both ati-drivers and xorg-server; the only instance where it wouldn't trigger the lock is when the user is running *just* xorg-server. Also, just to be clear, RESTRICT=parallel does not means that building xorg-server is going to be make -j1; it can still be make -jN, it just keeps the scheduler from building any other jobs in parallel while xorg-server is doing it's thing. > But if there's other non-contrived examples around, getting a couple of > them on the table should I think clarify our potential usage constraints > somewhat. As mentioned, I've got nothing else concrete- mostly because I've not had the time to look (I do know they're going to be there however). One thing to keep in mind also is that when situations of this sort pop up we need the functionality *before* it occurs. Adding it after the fact is a PITA. What I'd like to see w/ RESTRICT=parallel is that it basically is unused in the tree- I expect a few screwed up pkgs to need it however. It's a bit like RESTRICT=sandbox; it should only be used when there is an extremely good reason and the alternatives don't easily exist. ~harring [-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [gentoo-dev] Multiple emerges in parallel (was: [RFC] RESTRICT=parallel for builds that can't be executed in parallel) 2010-04-14 6:10 ` Brian Harring 2010-04-14 7:06 ` [gentoo-dev] " Duncan @ 2010-04-14 14:20 ` Michał Górny 2010-04-14 18:10 ` Brian Harring 1 sibling, 1 reply; 16+ messages in thread From: Michał Górny @ 2010-04-14 14:20 UTC (permalink / raw To: gentoo-dev [-- Attachment #1: Type: text/plain, Size: 3187 bytes --] On Tue, 13 Apr 2010 23:10:16 -0700 Brian Harring <ferringb@gmail.com> wrote: > Running multiple emerges in parallel is already a bad idea. The > solution for that case is for the new/second emerge to feed the > request into the original emerge (or a daemon). Although such solution will be useful in many cases indeed, there are still many advantages of having few separate emerge calls running in parallel. First of all, the 'emerge daemon' would have to be able to transparently inject the additional packages onto the dependency tree. And just appending missing dependencies and the real package at the end doesn't seem sufficient. For example, what if such operation would result in a conflict? If that's only that simple as 'A and B can not be installed at the same time', the second emerge call could reject appending the packages, explaining the reason for that. But what if the conflict could be resolved through pulling in some other version of one of the packages? We could try to replace the package in the dependency tree silently if it didn't yet start being merged; but what if it did? Should we reject adding the packages, encouraging user to 'try again later'? Or maybe abort the running builds to merge another version of the depend? Same goes for virtual dependencies which can be fulfilled by many different packages. Of course, that's just implementation-wide issue and running separate emerge instances could result in even worse results of this problem. But apart from that, there are side uses of multiple emerges which would be hard to implement with your idea. For example, I often use different PORTAGE_TMPDIR to build large packages like xulrunner. Although it doesn't seem that hard implementing the support for running merges in different temporary directories at the same time, should we really waste time implementing that? What about other possible configuration changes? What about 'USE' overrides within the environment? These ones should certainly be respected. So, now our daemon not only has to be able to efficiently mangle dependency trees at runtime but also to support partial runtime configuration changes. The next thing is aborting merges. When running multiple emerges, aborting one of them is as simple as pressing ^c. With daemon, we would have to implement an ability of aborting/removing packages in runtime -- and that would be another example of dependency tree mangling. But in fact, implementing (at least limited) the support of such mangling would be useful in some cases. With '--keep-going', it would allow to drop the packages and continue merging the remaining ones without waiting for current builds to complete. And looking from the other side, the support for removing single packages from the running emerge dependency tree would be useful too with current emerge concept. Even if it just worked like '--keep-going', it would be still better than the current workaround -- waiting for particular emerge to start and removing its ${WORKDIR} to force failure. -- Best regards, Michał Górny <http://mgorny.alt.pl> <xmpp:mgorny@jabber.ru> [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 198 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [gentoo-dev] Multiple emerges in parallel (was: [RFC] RESTRICT=parallel for builds that can't be executed in parallel) 2010-04-14 14:20 ` [gentoo-dev] Multiple emerges in parallel (was: [RFC] RESTRICT=parallel for builds that can't be executed in parallel) Michał Górny @ 2010-04-14 18:10 ` Brian Harring 2010-04-14 18:35 ` Michał Górny 2010-04-14 18:47 ` [gentoo-dev] " Duncan 0 siblings, 2 replies; 16+ messages in thread From: Brian Harring @ 2010-04-14 18:10 UTC (permalink / raw To: gentoo-dev [-- Attachment #1: Type: text/plain, Size: 1862 bytes --] On Wed, Apr 14, 2010 at 04:20:18PM +0200, Michaaa GGGrny wrote: > On Tue, 13 Apr 2010 23:10:16 -0700 > Brian Harring <ferringb@gmail.com> wrote: > > > Running multiple emerges in parallel is already a bad idea. The > > solution for that case is for the new/second emerge to feed the > > request into the original emerge (or a daemon). > > Although such solution will be useful in many cases indeed, there are > still many advantages of having few separate emerge calls running > in parallel. The examples you give are fine and dandy, but if done via parallel emerge you can run into situations where PM 1 just added pkg A as a dep for PKG B, and PM 2 is removing pkg A due to a blocker for pkg C. Running multiple emerge's in parallel is unsafe due to the fact they've got two potentially very different plans as to what is being done, and that there is no possibility to ensure that pkg D that PM-2 is building isn't affected by PM-1 building something (upgrading a dependency of pkg D for example). Yes you can get away with it occasionally, that doesn't mean it's safe however. > The next thing is aborting merges. When running multiple emerges, > aborting one of them is as simple as pressing ^c. With daemon, we would > have to implement an ability of aborting/removing packages in runtime > -- and that would be another example of dependency tree mangling. Aborting merges is a very, very bad idea. Consider a pkg that has dlopen'd plugins, and just went through an ABI change for that interface. If you interupt that merge it's entirely possible you'll get just the lib merged (meaning a segfault on loadup of the plugins), or vice versa (old lib is still in place, but new plugins are there). Don't abort merges- that really should be effectively an atomic OP, not interuptible. ~harring [-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [gentoo-dev] Multiple emerges in parallel (was: [RFC] RESTRICT=parallel for builds that can't be executed in parallel) 2010-04-14 18:10 ` Brian Harring @ 2010-04-14 18:35 ` Michał Górny 2010-04-14 18:47 ` [gentoo-dev] " Duncan 1 sibling, 0 replies; 16+ messages in thread From: Michał Górny @ 2010-04-14 18:35 UTC (permalink / raw To: gentoo-dev [-- Attachment #1: Type: text/plain, Size: 1469 bytes --] On Wed, 14 Apr 2010 11:10:29 -0700 Brian Harring <ferringb@gmail.com> wrote: > Running multiple emerge's in parallel is unsafe due to the fact > they've got two potentially very different plans as to what is being > done, and that there is no possibility to ensure that pkg D that PM-2 > is building isn't affected by PM-1 building something (upgrading a > dependency of pkg D for example). > > Yes you can get away with it occasionally, that doesn't mean it's > safe however. I agree with you -- such operations should be performed with appropriate caution, on user's own responsibility. But this doesn't mean we should prevent user from being able to do that. > Aborting merges is a very, very bad idea. Consider a pkg that has > dlopen'd plugins, and just went through an ABI change for that > interface. If you interupt that merge it's entirely possible you'll > get just the lib merged (meaning a segfault on loadup of the > plugins), or vice versa (old lib is still in place, but new plugins > are there). > > Don't abort merges- that really should be effectively an atomic OP, > not interuptible. Sorry, by 'aborting merges' I meant rather aborting the build process before the 'merge' phase -- or even aborting it before it is started (i.e. before it unpacks a load of files into ${PORTAGE_TMPDIR} without sufficient space). -- Best regards, Michał Górny <http://mgorny.alt.pl> <xmpp:mgorny@jabber.ru> [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 198 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* [gentoo-dev] Re: Multiple emerges in parallel (was: [RFC] RESTRICT=parallel for builds that can't be executed in parallel) 2010-04-14 18:10 ` Brian Harring 2010-04-14 18:35 ` Michał Górny @ 2010-04-14 18:47 ` Duncan 1 sibling, 0 replies; 16+ messages in thread From: Duncan @ 2010-04-14 18:47 UTC (permalink / raw To: gentoo-dev Brian Harring posted on Wed, 14 Apr 2010 11:10:29 -0700 as excerpted: >> The next thing is aborting merges. When running multiple emerges, >> aborting one of them is as simple as pressing ^c. With daemon, we would >> have to implement an ability of aborting/removing packages in runtime >> -- and that would be another example of dependency tree mangling. > > Aborting merges is a very, very bad idea. Consider a pkg that has > dlopen'd plugins, and just went through an ABI change for that > interface. If you interupt that merge it's entirely possible you'll get > just the lib merged (meaning a segfault on loadup of the plugins), or > vice versa (old lib is still in place, but new plugins are there). > > Don't abort merges- that really should be effectively an atomic OP, not > interuptible. Umm... I think you two are using the same words for different things. Definitely, aborting qmerge (transfer to the live filesystem) isn't a good idea, but in context, it's plain that MG's talking about the entire merge process, from setup, unpack and prepare thru qmerge and postinst (which is how the terms are used in the ebuild qmerge vs. ebuild merge context as well). Clearly the entire merge doesn't need to be atomic, or we'd not be talking about parallel merges in the first place, nor would we have available all the individual ebuild <command> substeps. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [gentoo-dev] [RFC] RESTRICT=parallel for builds that can't be executed in parallel 2010-04-14 2:12 [gentoo-dev] [RFC] RESTRICT=parallel for builds that can't be executed in parallel Zac Medico 2010-04-14 5:45 ` Michał Górny @ 2010-04-14 6:46 ` Justin 2010-04-14 12:58 ` Michał Górny ` (2 more replies) 2010-04-21 10:38 ` Luca Barbato 2 siblings, 3 replies; 16+ messages in thread From: Justin @ 2010-04-14 6:46 UTC (permalink / raw To: gentoo-dev [-- Attachment #1: Type: text/plain, Size: 532 bytes --] On 14/04/10 04:12, Zac Medico wrote: > Hi everyone, > > Should we add a RESTRICT=parallel value for ebuilds that can't be > built at the same time as other ebuilds? Brian says we need it for > things like xorg-server which calls eselect opengl. There is at least one other example which benefits from singlular build, atlas libs. They run a benchmark suite to create platform specific headers, which is heavily influenced by the system load. So having RESTRICT=parallel would make the emerge more reliable. justin [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 198 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [gentoo-dev] [RFC] RESTRICT=parallel for builds that can't be executed in parallel 2010-04-14 6:46 ` [gentoo-dev] [RFC] RESTRICT=parallel for builds that can't be executed in parallel Justin @ 2010-04-14 12:58 ` Michał Górny 2010-04-14 13:03 ` [gentoo-dev] " Duncan 2010-04-21 10:34 ` [gentoo-dev] " Luca Barbato 2 siblings, 0 replies; 16+ messages in thread From: Michał Górny @ 2010-04-14 12:58 UTC (permalink / raw To: gentoo-dev [-- Attachment #1: Type: text/plain, Size: 1469 bytes --] On Wed, 14 Apr 2010 08:46:15 +0200 Justin <jlec@gentoo.org> wrote: > There is at least one other example which benefits from singlular > build, atlas libs. They run a benchmark suite to create platform > specific headers, which is heavily influenced by the system load. So > having RESTRICT=parallel would make the emerge more reliable. And that's another example of working around broken ideas. There is an uncountable number of possibilities of many different factors affecting the system load, and thus the results of the benchmark. Removing one of them would indeed help in many cases but in many other it wouldn't make any difference, additionally slowing down the whole system update process. Please notice that in order to implement that correctly, emerge would have to wait until all previously running emerges are done, and avoid running further ones while the build process is running. While in the case of xorg-server, the 'lock' is needed throughout the whole build process, in your case it is needed only for a short amount of time when the benchmark is being performed --- and the 'real' part of building would unnecessarily block emerge. If this is ever to be implemented, it totally needs to be user-overridable. Else, we'll end up someday like Windows, forcing user to reboot the system and perform the merge on a dedicated runlevel. -- Best regards, Michał Górny <http://mgorny.alt.pl> <xmpp:mgorny@jabber.ru> [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 198 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* [gentoo-dev] Re: [RFC] RESTRICT=parallel for builds that can't be executed in parallel 2010-04-14 6:46 ` [gentoo-dev] [RFC] RESTRICT=parallel for builds that can't be executed in parallel Justin 2010-04-14 12:58 ` Michał Górny @ 2010-04-14 13:03 ` Duncan 2010-04-14 23:36 ` Ryan Hill 2010-04-21 10:34 ` [gentoo-dev] " Luca Barbato 2 siblings, 1 reply; 16+ messages in thread From: Duncan @ 2010-04-14 13:03 UTC (permalink / raw To: gentoo-dev Justin posted on Wed, 14 Apr 2010 08:46:15 +0200 as excerpted: > On 14/04/10 04:12, Zac Medico wrote: >> Hi everyone, >> >> Should we add a RESTRICT=parallel value for ebuilds that can't be built >> at the same time as other ebuilds? Brian says we need it for things >> like xorg-server which calls eselect opengl. > > There is at least one other example which benefits from singlular build, > atlas libs. They run a benchmark suite to create platform specific > headers, which is heavily influenced by the system load. So having > RESTRICT=parallel would make the emerge more reliable. sci-libs/blas-atlas (and perhaps lapack-atlas, etc, too), right? "Automatically tuned..." Wow! Yeah, that sounds like a reasonable example. Sort of like the kernel does for md/RAID-5 and 6 at boot, I'd guess, choosing the fastest algorithm on the platform, only they're doing it during system runtime when who /knows/ what else is running! Having a second highly parallelizable (MAKEOPTS version) build go from config to build and its load go from say .8 to 8. in the middle of those benchmarks /could/ screw things up "just a little!" Thanks. That's just the sort of additional practical example I was asking for to try to get my mind around this. Excellent example as, unlike the various xorg/mesa/drivers thing, it's pretty hard to argue "just code around it", for this one. The only technical way out of it here would seem to be to change the build-and-benchmark strategy itself, which would rather defeat the "automatically tuned" bit entirely. BTW, gcc seems to do some stage output comparing in its bootstrap process. Is that all absolute code correctness, or is there some performance benchmarking there that could benefit from this as well? -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman ^ permalink raw reply [flat|nested] 16+ messages in thread
* [gentoo-dev] Re: [RFC] RESTRICT=parallel for builds that can't be executed in parallel 2010-04-14 13:03 ` [gentoo-dev] " Duncan @ 2010-04-14 23:36 ` Ryan Hill 0 siblings, 0 replies; 16+ messages in thread From: Ryan Hill @ 2010-04-14 23:36 UTC (permalink / raw To: gentoo-dev [-- Attachment #1: Type: text/plain, Size: 553 bytes --] On Wed, 14 Apr 2010 13:03:10 +0000 (UTC) Duncan <1i5t5.duncan@cox.net> wrote: > BTW, gcc seems to do some stage output comparing in its bootstrap > process. Is that all absolute code correctness, or is there some > performance benchmarking there that could benefit from this as well? It's all code correctness. -- fonts, by design, by neglect gcc-porting, for a fact or just for effect wxwidgets @ gentoo EFFD 380E 047A 4B51 D2BD C64F 8AA8 8346 F9A4 0662 [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 198 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [gentoo-dev] [RFC] RESTRICT=parallel for builds that can't be executed in parallel 2010-04-14 6:46 ` [gentoo-dev] [RFC] RESTRICT=parallel for builds that can't be executed in parallel Justin 2010-04-14 12:58 ` Michał Górny 2010-04-14 13:03 ` [gentoo-dev] " Duncan @ 2010-04-21 10:34 ` Luca Barbato 2010-04-30 16:32 ` Enrico Weigelt 2 siblings, 1 reply; 16+ messages in thread From: Luca Barbato @ 2010-04-21 10:34 UTC (permalink / raw To: gentoo-dev; +Cc: Justin On 04/14/2010 08:46 AM, Justin wrote: > There is at least one other example which benefits from singlular build, > atlas libs. They run a benchmark suite to create platform specific > headers, which is heavily influenced by the system load. So having > RESTRICT=parallel would make the emerge more reliable. I wonder if that case shouldn't be handled better with an huge ewarn so people concerned would really run it in a benchmark environment, alone. lu -- Luca Barbato Gentoo/linux http://dev.gentoo.org/~lu_zero ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [gentoo-dev] [RFC] RESTRICT=parallel for builds that can't be executed in parallel 2010-04-21 10:34 ` [gentoo-dev] " Luca Barbato @ 2010-04-30 16:32 ` Enrico Weigelt 0 siblings, 0 replies; 16+ messages in thread From: Enrico Weigelt @ 2010-04-30 16:32 UTC (permalink / raw To: gentoo-dev * Luca Barbato <lu_zero@gentoo.org> schrieb: > I wonder if that case shouldn't be handled better with an huge ewarn so > people concerned would really run it in a benchmark environment, alone. ewarns should be circumvented (make it work w/o them), IMHO. I can imagine I'm not the only person who doenst want to keep an close eye on each single build/merge. In this concrete example, the package's build system is misdesigned. They shouldn't rely on runtime benchmarks for build time decisions in the first place - this is unpredictable. But this is not an issues distros should have to cope with. cu -- --------------------------------------------------------------------- Enrico Weigelt == metux IT service - http://www.metux.de/ --------------------------------------------------------------------- Please visit the OpenSource QM Taskforce: http://wiki.metux.de/public/OpenSource_QM_Taskforce Patches / Fixes for a lot dozens of packages in dozens of versions: http://patches.metux.de/ --------------------------------------------------------------------- ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [gentoo-dev] [RFC] RESTRICT=parallel for builds that can't be executed in parallel 2010-04-14 2:12 [gentoo-dev] [RFC] RESTRICT=parallel for builds that can't be executed in parallel Zac Medico 2010-04-14 5:45 ` Michał Górny 2010-04-14 6:46 ` [gentoo-dev] [RFC] RESTRICT=parallel for builds that can't be executed in parallel Justin @ 2010-04-21 10:38 ` Luca Barbato 2 siblings, 0 replies; 16+ messages in thread From: Luca Barbato @ 2010-04-21 10:38 UTC (permalink / raw To: gentoo-dev; +Cc: Zac Medico, Brian Harring On 04/14/2010 04:12 AM, Zac Medico wrote: > Hi everyone, > > Should we add a RESTRICT=parallel value for ebuilds that can't be > built at the same time as other ebuilds? Brian says we need it for > things like xorg-server which calls eselect opengl. Maybe we could live w/out calling eselect opengl that way first. The discussion about how to handle two parallel merge in a safer way seems interesting though. lu -- Luca Barbato Gentoo/linux http://dev.gentoo.org/~lu_zero ^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2010-04-30 21:01 UTC | newest] Thread overview: 16+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2010-04-14 2:12 [gentoo-dev] [RFC] RESTRICT=parallel for builds that can't be executed in parallel Zac Medico 2010-04-14 5:45 ` Michał Górny 2010-04-14 6:10 ` Brian Harring 2010-04-14 7:06 ` [gentoo-dev] " Duncan 2010-04-14 9:02 ` Brian Harring 2010-04-14 14:20 ` [gentoo-dev] Multiple emerges in parallel (was: [RFC] RESTRICT=parallel for builds that can't be executed in parallel) Michał Górny 2010-04-14 18:10 ` Brian Harring 2010-04-14 18:35 ` Michał Górny 2010-04-14 18:47 ` [gentoo-dev] " Duncan 2010-04-14 6:46 ` [gentoo-dev] [RFC] RESTRICT=parallel for builds that can't be executed in parallel Justin 2010-04-14 12:58 ` Michał Górny 2010-04-14 13:03 ` [gentoo-dev] " Duncan 2010-04-14 23:36 ` Ryan Hill 2010-04-21 10:34 ` [gentoo-dev] " Luca Barbato 2010-04-30 16:32 ` Enrico Weigelt 2010-04-21 10:38 ` Luca Barbato
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox