* [gentoo-dev] has_version etc parallelisability
@ 2007-12-30 22:35 Ciaran McCreesh
2007-12-31 3:03 ` Petteri Räty
2007-12-31 4:11 ` Alec Warner
0 siblings, 2 replies; 10+ messages in thread
From: Ciaran McCreesh @ 2007-12-30 22:35 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 202 bytes --]
Is it legal for ebuilds to call has_version and friends in parallel? Is
it legal for ebuilds to call has_version and friends after the ebuild
process has terminated? Discuss.
--
Ciaran McCreesh
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [gentoo-dev] has_version etc parallelisability
2007-12-30 22:35 [gentoo-dev] has_version etc parallelisability Ciaran McCreesh
@ 2007-12-31 3:03 ` Petteri Räty
2007-12-31 14:25 ` Ciaran McCreesh
2007-12-31 4:11 ` Alec Warner
1 sibling, 1 reply; 10+ messages in thread
From: Petteri Räty @ 2007-12-31 3:03 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 318 bytes --]
Ciaran McCreesh kirjoitti:
> Is it legal for ebuilds to call has_version and friends in parallel? Is
> it legal for ebuilds to call has_version and friends after the ebuild
> process has terminated? Discuss.
>
Do you/anybody know if they are used in parallel in the tree at the moment?
Regards,
Petteri
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [gentoo-dev] has_version etc parallelisability
2007-12-30 22:35 [gentoo-dev] has_version etc parallelisability Ciaran McCreesh
2007-12-31 3:03 ` Petteri Räty
@ 2007-12-31 4:11 ` Alec Warner
2007-12-31 14:28 ` Ciaran McCreesh
1 sibling, 1 reply; 10+ messages in thread
From: Alec Warner @ 2007-12-31 4:11 UTC (permalink / raw
To: gentoo-dev
On 12/30/07, Ciaran McCreesh <ciaran.mccreesh@blueyonder.co.uk> wrote:
> Is it legal for ebuilds to call has_version and friends in parallel? Is
> it legal for ebuilds to call has_version and friends after the ebuild
> process has terminated? Discuss.
If the pm implements read/write locking on the underlying datastore
(which it should probably have regardless of this request) then I
don't see a problem in parallel has_version calls.
I don't get your second example..do you mean the ebuild is running
has_version in the background and then terminating?
>
> --
> Ciaran McCreesh
>
>
--
gentoo-dev@gentoo.org mailing list
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [gentoo-dev] has_version etc parallelisability
2007-12-31 3:03 ` Petteri Räty
@ 2007-12-31 14:25 ` Ciaran McCreesh
0 siblings, 0 replies; 10+ messages in thread
From: Ciaran McCreesh @ 2007-12-31 14:25 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 513 bytes --]
On Mon, 31 Dec 2007 05:03:21 +0200
Petteri Räty <betelgeuse@gentoo.org> wrote:
> Ciaran McCreesh kirjoitti:
> > Is it legal for ebuilds to call has_version and friends in
> > parallel? Is it legal for ebuilds to call has_version and friends
> > after the ebuild process has terminated? Discuss.
> >
>
> Do you/anybody know if they are used in parallel in the tree at the
> moment?
I can't see anything obvious, but that doesn't mean there aren't a few
weird hiding cases.
--
Ciaran McCreesh
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [gentoo-dev] has_version etc parallelisability
2007-12-31 4:11 ` Alec Warner
@ 2007-12-31 14:28 ` Ciaran McCreesh
2008-01-05 2:50 ` Brian Harring
0 siblings, 1 reply; 10+ messages in thread
From: Ciaran McCreesh @ 2007-12-31 14:28 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 1108 bytes --]
On Sun, 30 Dec 2007 20:11:16 -0800
"Alec Warner" <antarus@gentoo.org> wrote:
> On 12/30/07, Ciaran McCreesh <ciaran.mccreesh@blueyonder.co.uk> wrote:
> > Is it legal for ebuilds to call has_version and friends in
> > parallel? Is it legal for ebuilds to call has_version and friends
> > after the ebuild process has terminated? Discuss.
>
> If the pm implements read/write locking on the underlying datastore
> (which it should probably have regardless of this request) then I
> don't see a problem in parallel has_version calls.
Actually, it's the communication channel that's the issue... If, for
example, has_version is implemented in terms of a request on a pipe
rather than execing a new package manager, we get into messy bash
locking territory...
> I don't get your second example..do you mean the ebuild is running
> has_version in the background and then terminating?
Yeah. Again, consider the pipe example. If the package manager closes
off the pipe when it thinks the ebuild's done, calling has_version will
get the backgrounded process SIGPIPEd.
--
Ciaran McCreesh
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [gentoo-dev] has_version etc parallelisability
2007-12-31 14:28 ` Ciaran McCreesh
@ 2008-01-05 2:50 ` Brian Harring
2008-01-05 4:52 ` Ciaran McCreesh
0 siblings, 1 reply; 10+ messages in thread
From: Brian Harring @ 2008-01-05 2:50 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 2172 bytes --]
On Mon, Dec 31, 2007 at 02:28:44PM +0000, Ciaran McCreesh wrote:
> On Sun, 30 Dec 2007 20:11:16 -0800
> "Alec Warner" <antarus@gentoo.org> wrote:
> > On 12/30/07, Ciaran McCreesh <ciaran.mccreesh@blueyonder.co.uk> wrote:
> > > Is it legal for ebuilds to call has_version and friends in
> > > parallel? Is it legal for ebuilds to call has_version and friends
> > > after the ebuild process has terminated? Discuss.
> >
> > If the pm implements read/write locking on the underlying datastore
> > (which it should probably have regardless of this request) then I
> > don't see a problem in parallel has_version calls.
>
> Actually, it's the communication channel that's the issue... If, for
> example, has_version is implemented in terms of a request on a pipe
> rather than execing a new package manager, we get into messy bash
> locking territory...
>
> > I don't get your second example..do you mean the ebuild is running
> > has_version in the background and then terminating?
>
> Yeah. Again, consider the pipe example. If the package manager closes
> off the pipe when it thinks the ebuild's done, calling has_version will
> get the backgrounded process SIGPIPEd.
Depends on the implementation; for pkgcore, if that comm pipe is
dead, the ebuild env *should* be dead, or dieing. Background'ing
processes from that env isn't valid imo, either.
If you're refering to an ebuild that parallelizes itself while
executing, iow, parallelization w/in the ebuild env/phase execution,
I'd look more at being able to batch commands instead of trying to run
them in parallel. Reasoning follows-
1) if doing an exec approach to service the request, this means
reparsing of involved files for each request- inefficient, potentially
horribly so on crappy hardware/setups.
2) screws up the pipe approach, should folks take it for control/env
introspection gains.
Summarizing, executing has_version (and friends)
concurrently has it's own issues performance wise, and implementation
wise; growing batch functionality into portageq however avoids those
issues, and would be faster- thus the route I'd advocate.
~harring
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [gentoo-dev] has_version etc parallelisability
2008-01-05 2:50 ` Brian Harring
@ 2008-01-05 4:52 ` Ciaran McCreesh
2008-01-05 17:29 ` Luca Barbato
0 siblings, 1 reply; 10+ messages in thread
From: Ciaran McCreesh @ 2008-01-05 4:52 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 1472 bytes --]
On Fri, 4 Jan 2008 18:50:56 -0800
Brian Harring <ferringb@gmail.com> wrote:
> Depends on the implementation; for pkgcore, if that comm pipe is
> dead, the ebuild env *should* be dead, or dieing. Background'ing
> processes from that env isn't valid imo, either.
Right. Paludis will give a weird die message but not actually fail if
you do:
src_compile() {
{ sleep 10 ; has_version '>=app-misc/foo-1.23' ; } &
}
> If you're refering to an ebuild that parallelizes itself while
> executing, iow, parallelization w/in the ebuild env/phase execution,
> I'd look more at being able to batch commands instead of trying to
> run them in parallel.
That's its own slippery slope. Because of limited size pipes, the
following causes allllll sorts of trouble:
pkg_setup()
{
portageq match ${ROOT} cat/some-pkg | while read a ; do
if has_version ="${a}" ; then
echo "yes to ${a}"
else
echo "no to ${a}"
fi
done
}
The problem is thus: the has_version and portageq match here can be run
in parallel by bash. The portageq match output can be longer than the
maximum size of a pipe. Thus, if the above is legal, no lock that is
visible to the has_version can be held by portageq match once it starts
producing output.
--
Ciaran McCreesh
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [gentoo-dev] has_version etc parallelisability
2008-01-05 4:52 ` Ciaran McCreesh
@ 2008-01-05 17:29 ` Luca Barbato
2008-01-05 18:55 ` Petteri Räty
2008-01-06 0:31 ` Ciaran McCreesh
0 siblings, 2 replies; 10+ messages in thread
From: Luca Barbato @ 2008-01-05 17:29 UTC (permalink / raw
To: gentoo-dev
Ciaran McCreesh wrote:
> On Fri, 4 Jan 2008 18:50:56 -0800
> Brian Harring <ferringb@gmail.com> wrote:
>> Depends on the implementation; for pkgcore, if that comm pipe is
>> dead, the ebuild env *should* be dead, or dieing. Background'ing
>> processes from that env isn't valid imo, either.
>
> Right. Paludis will give a weird die message but not actually fail if
> you do:
>
> src_compile() {
> { sleep 10 ; has_version '>=app-misc/foo-1.23' ; } &
> }
is & allowed in ebuilds? should?
lu
--
Luca Barbato
Gentoo Council Member
Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero
--
gentoo-dev@gentoo.org mailing list
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [gentoo-dev] has_version etc parallelisability
2008-01-05 17:29 ` Luca Barbato
@ 2008-01-05 18:55 ` Petteri Räty
2008-01-06 0:31 ` Ciaran McCreesh
1 sibling, 0 replies; 10+ messages in thread
From: Petteri Räty @ 2008-01-05 18:55 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 686 bytes --]
Luca Barbato kirjoitti:
> Ciaran McCreesh wrote:
>> On Fri, 4 Jan 2008 18:50:56 -0800
>> Brian Harring <ferringb@gmail.com> wrote:
>>> Depends on the implementation; for pkgcore, if that comm pipe is
>>> dead, the ebuild env *should* be dead, or dieing. Background'ing
>>> processes from that env isn't valid imo, either.
>> Right. Paludis will give a weird die message but not actually fail if
>> you do:
>>
>> src_compile() {
>> { sleep 10 ; has_version '>=app-misc/foo-1.23' ; } &
>> }
>
> is & allowed in ebuilds? should?
>
> lu
>
I would say that nothing started in src_* functions should be running
when the function exits.
Regards,
Petteri
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [gentoo-dev] has_version etc parallelisability
2008-01-05 17:29 ` Luca Barbato
2008-01-05 18:55 ` Petteri Räty
@ 2008-01-06 0:31 ` Ciaran McCreesh
1 sibling, 0 replies; 10+ messages in thread
From: Ciaran McCreesh @ 2008-01-06 0:31 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 392 bytes --]
On Sat, 05 Jan 2008 18:29:51 +0100
Luca Barbato <lu_zero@gentoo.org> wrote:
> Ciaran McCreesh wrote:
> > src_compile() {
> > { sleep 10 ; has_version '>=app-misc/foo-1.23' ; } &
> > }
>
> is & allowed in ebuilds? should?
Banning it entirely is excessive. Banning leaving any attached
processes between phases is hopefully not going to upset anyone...
--
Ciaran McCreesh
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2008-01-06 0:36 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2007-12-30 22:35 [gentoo-dev] has_version etc parallelisability Ciaran McCreesh
2007-12-31 3:03 ` Petteri Räty
2007-12-31 14:25 ` Ciaran McCreesh
2007-12-31 4:11 ` Alec Warner
2007-12-31 14:28 ` Ciaran McCreesh
2008-01-05 2:50 ` Brian Harring
2008-01-05 4:52 ` Ciaran McCreesh
2008-01-05 17:29 ` Luca Barbato
2008-01-05 18:55 ` Petteri Räty
2008-01-06 0:31 ` Ciaran McCreesh
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox