[gentoo-dev] emake question

[gentoo-dev] emake question

[Aron Griffis]

Hello,

I'm looking at emake, trying to understand what is being accomplished.
It seems useless to me, though I'll be quite happy if told that I'm
wrong.

(1) It claims to "auto-parallel make without the annoying syntax".  But
    it does nothing to parallelize the make process that I can see.

(2) It uses env, which does nothing in this case.

(3) It uses a variable MAKEOPTS to pass options along to sub-makes.
    Isn't that redundant to GNU make's built in MAKEFLAGS (which are set
    and passed to sub-makes automatically)?

Thanks,
Aron

Re: [gentoo-dev] emake question

[Daniel Robbins]

On Fri, Aug 17, 2001 at 03:09:25PM -0400, Aron Griffis wrote:
> Hello,
> 
> I'm looking at emake, trying to understand what is being accomplished.
> It seems useless to me, though I'll be quite happy if told that I'm
> wrong.
> 
> (1) It claims to "auto-parallel make without the annoying syntax".  But
>     it does nothing to parallelize the make process that I can see.
> 
> (2) It uses env, which does nothing in this case.
> 
> (3) It uses a variable MAKEOPTS to pass options along to sub-makes.
>     Isn't that redundant to GNU make's built in MAKEFLAGS (which are set
>     and passed to sub-makes automatically)?

By default, MAKEOPTS is defined at "-j 2", so emake does auto parallelize
makes.  The reason for emake is that some packages build in parallel, and 
some don't.  Those that do build in parallel should use emake in order to
build faster, and those that don't should have a little "doesn't work with
emake" comment and just use plan make instead.

Best Regards,

-- 
Daniel Robbins					<drobbins@gentoo.org>
Chief Architect/President			http://www.gentoo.org 
Gentoo Technologies, Inc.			

Re: [gentoo-dev] emake question

[Aron Griffis]

Hi Daniel,

Daniel Robbins wrote:	[Fri Aug 17 2001, 03:40:22PM EDT]
> On Fri, Aug 17, 2001 at 03:09:25PM -0400, Aron Griffis wrote:
> > Hello,
> > 
> > I'm looking at emake, trying to understand what is being accomplished.
> > It seems useless to me, though I'll be quite happy if told that I'm
> > wrong.
> > 
> > (1) It claims to "auto-parallel make without the annoying syntax".  But
> >     it does nothing to parallelize the make process that I can see.
> > 
> > (2) It uses env, which does nothing in this case.
> > 
> > (3) It uses a variable MAKEOPTS to pass options along to sub-makes.
> >     Isn't that redundant to GNU make's built in MAKEFLAGS (which are set
> >     and passed to sub-makes automatically)?
> 
> By default, MAKEOPTS is defined at "-j 2", so emake does auto parallelize
> makes.  The reason for emake is that some packages build in parallel, and 
> some don't.  Those that do build in parallel should use emake in order to
> build faster, and those that don't should have a little "doesn't work with
> emake" comment and just use plan make instead.

Sorry if I'm being obstinate, but it still doesn't seem useful to me.
The passing of MAKEOPTS to the sub-makes doesn't do anything, since GNU
make passes its flags to the sub-makes via MAKEFLAGS.  This includes
parallelization via -j.

That reduces emake to essentially running "make -j2" instead of "make".
Seems like that part could be left up to the ebuild authors.

Exhibit A:

    # A completely useless Makefile
    all:
	    # Start a sub-make that does a lot of nothing
	    $(MAKE) asdf{0,1,2,3,4,5}{0,1,2,3,4,5,6,7,8,9}
    asdf%:
	    @echo $@ -$(MAKEFLAGS)

Run exhibit A with "make -j" or "make -j10" or whatever a few times and
note that the sub-make is running in parallel...  :-)

Thanks,
Aron (who likes abstractions but fights them when they just muddy the water)

Re: [gentoo-dev] emake question

[Mikael Hallendal]

Den 17 Aug 2001 16:30:16 -0400 skrev Aron Griffis:

> That reduces emake to essentially running "make -j2" instead of "make".
> Seems like that part could be left up to the ebuild authors.

And that is the purpose of it. When creating the ebuild the user can
write the much shorter and cleaner syntax 'emake' instead of 'make
${MAKEOPTS}'. Where MAKEOPTS being defined in /etc/make.{globals,conf}.

Regards,
  Mikael Hallendal

-- 

Mikael Hallendal
Gentoo Linux Developer, Desktop Team Leader
Stockholm, Sweden

Re: [gentoo-dev] emake question

[Daniel Robbins]

On Fri, Aug 17, 2001 at 04:30:16PM -0400, Aron Griffis wrote:

> That reduces emake to essentially running "make -j2" instead of "make".
> Seems like that part could be left up to the ebuild authors.

Yes, this is all emake does.  The reason why we don't leave it up to the
ebuild authors is that -j2 is optimal for uniprocessors, -j3 for dual
processors, and -j5 for quad processors.  So the -j part needs to be 
definable by the user.  It's easier to type:

	emake

than
	make $MAKEOPTS MAKE="make $MAKEOPTS"

Also, it's less prone to have syntax errors, and easier to remember.

-- 
Daniel Robbins					<drobbins@gentoo.org>
Chief Architect/President			http://www.gentoo.org 
Gentoo Technologies, Inc.			

Re: [gentoo-dev] emake question

[Aron Griffis]

Hi Mikael and Daniel,

Mikael Hallendal wrote:	[Fri Aug 17 2001, 04:51:05PM EDT]
> And that is the purpose of it. When creating the ebuild the user can
> write the much shorter and cleaner syntax 'emake' instead of 'make
> ${MAKEOPTS}'. Where MAKEOPTS being defined in /etc/make.{globals,conf}.

Daniel Robbins wrote:	[Fri Aug 17 2001, 05:11:55PM EDT]
> Yes, this is all emake does.  The reason why we don't leave it up to the
> ebuild authors is that -j2 is optimal for uniprocessors, -j3 for dual
> processors, and -j5 for quad processors.  So the -j part needs to be 
> definable by the user.  It's easier to type:
> 
> 	emake
> 
> than
> 	make $MAKEOPTS MAKE="make $MAKEOPTS"
> 
> Also, it's less prone to have syntax errors, and easier to remember.

Okay, what is being accomplished here finally makes sense to me!  Thanks
for being patient enough to get it through my skull.  However there are
a couple things I would like to suggest.

(1) The line in emake is

        exec env make ${MAKEOPTS} MAKE="make ${MAKEOPTS}" "$@"

    however it could be

        exec make ${MAKEOPTS} "$@"

    and this has the same result, since it relies on make to pass the
    flags to the sub-makes using the builtin MAKEFLAGS.  I think it's
    actually better, since it allows GNU make to manage it's own
    subprocesses instead of forcing flags on the sub-makes.

(2) However emake is currently inflexible.  For example, say that we
    wanted to provide more potential options to an ebuild's make?  We'd
    add them to MAKEOPTS.  Great!  Except that packages that can't build
    in parallel would also miss out on the extra flags.

(3) Considering the above, a solution would be to provide a couple of
    variables to the ebuild developers.  For example,

        MAKEJOPT = ideal -j option for the current machine (preferably
                   determined at run-time, not install-time)

        MAKEOPTS = other make flags (currently empty)

    Emake can become

        exec make $MAKEJOPT $MAKEOPTS "$@"

    If parallel building doesn't work, the ebuild author could still do
    the following if they want the non-parallel flags.

        make $MAKEOPTS

Does this make sense?  Would it be worthwhile or not?

Aron

Re: [gentoo-dev] emake question

[Mikael Hallendal]

Den 21 Aug 2001 22:24:37 -0400 skrev Aron Griffis:

> (3) Considering the above, a solution would be to provide a couple of
>     variables to the ebuild developers.  For example,
> 
>         MAKEJOPT = ideal -j option for the current machine (preferably
>                    determined at run-time, not install-time)
> 
>         MAKEOPTS = other make flags (currently empty)
> 
>     Emake can become
> 
>         exec make $MAKEJOPT $MAKEOPTS "$@"
> 
>     If parallel building doesn't work, the ebuild author could still do
>     the following if they want the non-parallel flags.
> 
>         make $MAKEOPTS

Do we have any such makeflags? Otherwise I see no point in doing this
before we do. Adding an extra variable with no gain (at least not now)
is imho stupid. All current ebuilds will work with this change (ie, its
transparent for current ebuilds). 

So this discussion can be postponed until we have this need.

Regards,
  Mikael Hallendal

-- 

Mikael Hallendal
Gentoo Linux Developer, Desktop Team Leader
Stockholm, Sweden

Re: [gentoo-dev] emake question

[Aron Griffis]

Mikael Hallendal wrote:	[Wed Aug 22 2001, 12:39:04AM EDT]
> Do we have any such makeflags? Otherwise I see no point in doing this
> before we do. Adding an extra variable with no gain (at least not now)
> is imho stupid. All current ebuilds will work with this change (ie, its
> transparent for current ebuilds). 
> 
> So this discussion can be postponed until we have this need.

Yup, you're right.  It made sense to me at the time, but now we've come
full circle and it doesn't any more.  :-|

Aron

Re: [gentoo-dev] emake question

[Daniel Robbins]

On Tue, Aug 21, 2001 at 11:08:21PM -0400, Aron Griffis wrote:
> Mikael Hallendal wrote:	[Wed Aug 22 2001, 12:39:04AM EDT]
> > So this discussion can be postponed until we have this need.
> 
> Yup, you're right.  It made sense to me at the time, but now we've come
> full circle and it doesn't any more.  :-|

But your idea to not forcefully pass MAKEOPTS to submakes is a good one
and is something that we should probablly do.  This tends to cause the
Linux kernel to for MAKEOPTS * MAKEOPTS gcc processes, from what I recall.
Maybe removing this as you suggest will fix it.  If you want to modify
emake on cvs (/usr/portage/sys-apps/portage/files/1.5/bin/emake) go ahead
and do so, and let me know when you have.

Best Regards,

-- 
Daniel Robbins					<drobbins@gentoo.org>
Chief Architect/President			http://www.gentoo.org 
Gentoo Technologies, Inc.			

Re: [gentoo-dev] emake question

[Aron Griffis]

Daniel Robbins wrote:	[Wed Aug 22 2001, 02:05:35AM EDT]
> But your idea to not forcefully pass MAKEOPTS to submakes is a good one
> and is something that we should probablly do.  This tends to cause the
> Linux kernel to for MAKEOPTS * MAKEOPTS gcc processes, from what I recall.
> Maybe removing this as you suggest will fix it.  If you want to modify
> emake on cvs (/usr/portage/sys-apps/portage/files/1.5/bin/emake) go ahead
> and do so, and let me know when you have.

Sure, I'll do that.

Aron