[gentoo-dev] has_version etc parallelisability

[gentoo-dev] has_version etc parallelisability

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 202 bytes --]

Is it legal for ebuilds to call has_version and friends in parallel? Is
it legal for ebuilds to call has_version and friends after the ebuild
process has terminated? Discuss.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] has_version etc parallelisability

[Petteri Räty]

[-- Attachment #1: Type: text/plain, Size: 318 bytes --]

Ciaran McCreesh kirjoitti:
> Is it legal for ebuilds to call has_version and friends in parallel? Is
> it legal for ebuilds to call has_version and friends after the ebuild
> process has terminated? Discuss.
> 

Do you/anybody know if they are used in parallel in the tree at the moment?

Regards,
Petteri


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [gentoo-dev] has_version etc parallelisability

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 513 bytes --]

On Mon, 31 Dec 2007 05:03:21 +0200
Petteri Räty <betelgeuse@gentoo.org> wrote:
> Ciaran McCreesh kirjoitti:
> > Is it legal for ebuilds to call has_version and friends in
> > parallel? Is it legal for ebuilds to call has_version and friends
> > after the ebuild process has terminated? Discuss.
> > 
> 
> Do you/anybody know if they are used in parallel in the tree at the
> moment?

I can't see anything obvious, but that doesn't mean there aren't a few
weird hiding cases.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] has_version etc parallelisability

[Alec Warner]

On 12/30/07, Ciaran McCreesh <ciaran.mccreesh@blueyonder.co.uk> wrote:
> Is it legal for ebuilds to call has_version and friends in parallel? Is
> it legal for ebuilds to call has_version and friends after the ebuild
> process has terminated? Discuss.

If the pm implements read/write locking on the underlying datastore
(which it should probably have regardless of this request) then I
don't see a problem in parallel has_version calls.

I don't get your second example..do you mean the ebuild is running
has_version in the background and then terminating?

>
> --
> Ciaran McCreesh
>
>
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] has_version etc parallelisability

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1108 bytes --]

On Sun, 30 Dec 2007 20:11:16 -0800
"Alec Warner" <antarus@gentoo.org> wrote:
> On 12/30/07, Ciaran McCreesh <ciaran.mccreesh@blueyonder.co.uk> wrote:
> > Is it legal for ebuilds to call has_version and friends in
> > parallel? Is it legal for ebuilds to call has_version and friends
> > after the ebuild process has terminated? Discuss.
> 
> If the pm implements read/write locking on the underlying datastore
> (which it should probably have regardless of this request) then I
> don't see a problem in parallel has_version calls.

Actually, it's the communication channel that's the issue... If, for
example, has_version is implemented in terms of a request on a pipe
rather than execing a new package manager, we get into messy bash
locking territory...

> I don't get your second example..do you mean the ebuild is running
> has_version in the background and then terminating?

Yeah. Again, consider the pipe example. If the package manager closes
off the pipe when it thinks the ebuild's done, calling has_version will
get the backgrounded process SIGPIPEd.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] has_version etc parallelisability

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 2172 bytes --]

On Mon, Dec 31, 2007 at 02:28:44PM +0000, Ciaran McCreesh wrote:
> On Sun, 30 Dec 2007 20:11:16 -0800
> "Alec Warner" <antarus@gentoo.org> wrote:
> > On 12/30/07, Ciaran McCreesh <ciaran.mccreesh@blueyonder.co.uk> wrote:
> > > Is it legal for ebuilds to call has_version and friends in
> > > parallel? Is it legal for ebuilds to call has_version and friends
> > > after the ebuild process has terminated? Discuss.
> > 
> > If the pm implements read/write locking on the underlying datastore
> > (which it should probably have regardless of this request) then I
> > don't see a problem in parallel has_version calls.
> 
> Actually, it's the communication channel that's the issue... If, for
> example, has_version is implemented in terms of a request on a pipe
> rather than execing a new package manager, we get into messy bash
> locking territory...
> 
> > I don't get your second example..do you mean the ebuild is running
> > has_version in the background and then terminating?
> 
> Yeah. Again, consider the pipe example. If the package manager closes
> off the pipe when it thinks the ebuild's done, calling has_version will
> get the backgrounded process SIGPIPEd.

Depends on the implementation; for pkgcore, if that comm pipe is 
dead, the ebuild env *should* be dead, or dieing.  Background'ing 
processes from that env isn't valid imo, either.

If you're refering to an ebuild that parallelizes itself while 
executing, iow, parallelization w/in the ebuild env/phase execution, 
I'd look more at being able to batch commands instead of trying to run 
them in parallel.  Reasoning follows-

1) if doing an exec approach to service the request, this means 
reparsing of involved files for each request- inefficient, potentially 
horribly so on crappy hardware/setups.
2) screws up the pipe approach, should folks take it for control/env 
introspection gains.

Summarizing, executing has_version (and friends) 
concurrently has it's own issues performance wise, and implementation 
wise; growing batch functionality into portageq however avoids those 
issues, and would be faster- thus the route I'd advocate.

~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] has_version etc parallelisability

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1472 bytes --]

On Fri, 4 Jan 2008 18:50:56 -0800
Brian Harring <ferringb@gmail.com> wrote:
> Depends on the implementation; for pkgcore, if that comm pipe is 
> dead, the ebuild env *should* be dead, or dieing.  Background'ing 
> processes from that env isn't valid imo, either.

Right. Paludis will give a weird die message but not actually fail if
you do:

src_compile() {
    { sleep 10 ; has_version '>=app-misc/foo-1.23' ; } &
}

> If you're refering to an ebuild that parallelizes itself while 
> executing, iow, parallelization w/in the ebuild env/phase execution, 
> I'd look more at being able to batch commands instead of trying to
> run them in parallel.

That's its own slippery slope. Because of limited size pipes, the
following causes allllll sorts of trouble:

pkg_setup()
{                                                                   
    portageq match ${ROOT} cat/some-pkg | while read a ; do
        if has_version ="${a}" ; then
            echo "yes to ${a}"
        else
            echo "no to ${a}"
        fi
    done                                                                        
}

The problem is thus: the has_version and portageq match here can be run
in parallel by bash. The portageq match output can be longer than the
maximum size of a pipe. Thus, if the above is legal, no lock that is
visible to the has_version can be held by portageq match once it starts
producing output.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] has_version etc parallelisability

[Luca Barbato]

Ciaran McCreesh wrote:
> On Fri, 4 Jan 2008 18:50:56 -0800
> Brian Harring <ferringb@gmail.com> wrote:
>> Depends on the implementation; for pkgcore, if that comm pipe is 
>> dead, the ebuild env *should* be dead, or dieing.  Background'ing 
>> processes from that env isn't valid imo, either.
> 
> Right. Paludis will give a weird die message but not actually fail if
> you do:
> 
> src_compile() {
>     { sleep 10 ; has_version '>=app-misc/foo-1.23' ; } &
> }

is & allowed in ebuilds? should?

lu

-- 

Luca Barbato
Gentoo Council Member
Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] has_version etc parallelisability

[Petteri Räty]

[-- Attachment #1: Type: text/plain, Size: 686 bytes --]

Luca Barbato kirjoitti:
> Ciaran McCreesh wrote:
>> On Fri, 4 Jan 2008 18:50:56 -0800
>> Brian Harring <ferringb@gmail.com> wrote:
>>> Depends on the implementation; for pkgcore, if that comm pipe is 
>>> dead, the ebuild env *should* be dead, or dieing.  Background'ing 
>>> processes from that env isn't valid imo, either.
>> Right. Paludis will give a weird die message but not actually fail if
>> you do:
>>
>> src_compile() {
>>     { sleep 10 ; has_version '>=app-misc/foo-1.23' ; } &
>> }
> 
> is & allowed in ebuilds? should?
> 
> lu
> 

I would say that nothing started in src_* functions should be running 
when the function exits.

Regards,
Petteri


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [gentoo-dev] has_version etc parallelisability

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 392 bytes --]

On Sat, 05 Jan 2008 18:29:51 +0100
Luca Barbato <lu_zero@gentoo.org> wrote:
> Ciaran McCreesh wrote:
> > src_compile() {
> >     { sleep 10 ; has_version '>=app-misc/foo-1.23' ; } &
> > }
> 
> is & allowed in ebuilds? should?

Banning it entirely is excessive. Banning leaving any attached
processes between phases is hopefully not going to upset anyone...

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]