[gentoo-dev] Usage of dev-utils/ninja in ebuilds

[gentoo-dev] Usage of dev-utils/ninja in ebuilds

[Mike Gilbert]

[-- Attachment #1: Type: text/plain, Size: 1087 bytes --]

For those unaware, dev-util/ninja is a make-replacement created by one
of the Chromium guys at Google. Its focus is on making incremental
builds of large software faster.

In the latest chromium ebuild (chromium-29.0.1516.3), we are using ninja
instead of make. ninja auto-detects the number of jobs to run by
default; in the chromium ebuild, we override this by passing -j
$(makeopts_jobs) instead.

Another place ninja is used cmake-utils.eclass, where it may be used
optionally in place of make depending on the CMAKE_MAKEFILE_GENERATOR
variable. The ninja_src_make function in cmake-utils.eclass calls ninja
with ${MAKEOPTS} or not depending on the value of CMAKE_VERBOSE. This is
a bit inconsistent and could cause problems because ninja only accepts a
limited subset of GNU make's command line options.

I'm wondering if we should create a more global function for calling
ninja in a consistent way. Maybe we should introduce a NINJAOPTS
variable for users to set in make.conf. Should we create a new eclass
for this, or just stick it in eutils?

Thoughts?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 230 bytes --]

Re: [gentoo-dev] Usage of dev-utils/ninja in ebuilds

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 2549 bytes --]

On Sat, 25 May 2013 14:48:30 -0400
Mike Gilbert <floppym@gentoo.org> wrote:

> For those unaware, dev-util/ninja is a make-replacement created by one
> of the Chromium guys at Google. Its focus is on making incremental
> builds of large software faster.

I've no idea how this would work out, so I'm asking some questions from
different perspective to form a better idea; this may step aside your
discussion but on the other hand could help other unaware developers.

If I missed earlier ML discussions on this by not paying attention or
being too new, feel free to point me to those; no need to repeat. :)

From an user perspective I was wondering if ninja in the Portage tree
makes use of this faster incremental builds feature. Should I expect
faster builds when trying this out?

From a developer perspective, aside from the above feature, are there
any other features that make ninja interesting to use in cmake ebuilds?
When should it be used?

From a maintainer's perspective, shouldn't the user be able to choose
whether ninja or make is used and not the developer? Or does ninja
really work out for the majority in a way nobody would complain?

From an upstream perspective, would it be problematic if people use
make instead of ninja; or wouldn't it translate into problems upstream
has to resolve? (To me, using ninja instead of make sounds like using
llvm instead of gcc; correct me if I'm wrong, I'm fairly new to ninja.)

> I'm wondering if we should create a more global function for calling
> ninja in a consistent way. Maybe we should introduce a NINJAOPTS
> variable for users to set in make.conf. Should we create a new eclass
> for this, or just stick it in eutils?

If we don't allow the user to pick which make system is preferred,
which I tried to suggest above; then I would prefer that we introduce
a NINJAOPTS variable like you suggest to go alongside MAKEOPTS.

In the case the user is able to choose which make system is preferred,
then MAKEOPTS should probably apply to that make system; unless we allow
maintainer's to override the user, then splitting them is back in favor.

But well, this is based on preliminary thoughts without really
properly knowing what the advantages and disadvantages here are.

Hopefully you can shed some light here, thanks in advance.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Usage of dev-utils/ninja in ebuilds

[Mike Gilbert]

[-- Attachment #1: Type: text/plain, Size: 2912 bytes --]

On 05/25/2013 03:17 PM, Tom Wijsman wrote:
> From an user perspective I was wondering if ninja in the Portage tree
> makes use of this faster incremental builds feature. Should I expect
> faster builds when trying this out?
> 

No, you will not see any significant speed increase because we always
start with an empty build directory in an ebuild. Nothing has changed in
that regard.

You may see small (~60 seconds) difference when compiling Chromium due
to faster target dependency resolution.

> From a maintainer's perspective, shouldn't the user be able to choose
> whether ninja or make is used and not the developer? Or does ninja
> really work out for the majority in a way nobody would complain?
> 

As a maintainer, it is nice to know exactly how the software is being
built in case of a bug report. This also helps if we need to report an
issue upstream.

I should clarify: ninja is not a *drop-in* replacement for make; it does
not parse Makefiles. Instead, it uses its own configuration format,
which may be generated using GYP or CMake.

The choice of build system should be transparent to the user. However, I
have no strong personal objection to making it configurable.

> From an upstream perspective, would it be problematic if people use
> make instead of ninja; or wouldn't it translate into problems upstream
> has to resolve? (To me, using ninja instead of make sounds like using
> llvm instead of gcc; correct me if I'm wrong, I'm fairly new to ninja.)
> 

So long as GYP supports emitting makefiles as well as ninja build files,
I don't think this will be a problem.

For Gentoo, I believe phajdan.jr simply chose to switch to ninja to more
closely mimic the normal upstream build process.

>> I'm wondering if we should create a more global function for calling
>> ninja in a consistent way. Maybe we should introduce a NINJAOPTS
>> variable for users to set in make.conf. Should we create a new eclass
>> for this, or just stick it in eutils?
> 
> If we don't allow the user to pick which make system is preferred,
> which I tried to suggest above; then I would prefer that we introduce
> a NINJAOPTS variable like you suggest to go alongside MAKEOPTS.
> 
> In the case the user is able to choose which make system is preferred,
> then MAKEOPTS should probably apply to that make system; unless we allow
> maintainer's to override the user, then splitting them is back in favor.
> 

Another thought would be to mimic scons-utils.eclass, which utilizes a
filtered copy of MAKEOPTS if SCONSOPTS is not defined.

> But well, this is based on preliminary thoughts without really
> properly knowing what the advantages and disadvantages here are.
> 

I believe the only real advantage to using ninja for building chromium
is to follow upstream's default build configuration. It should be pretty
transparent to the end user.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 230 bytes --]

Re: [gentoo-dev] Usage of dev-utils/ninja in ebuilds

[Luca Barbato]

On 5/25/13 9:17 PM, Tom Wijsman wrote:
> On Sat, 25 May 2013 14:48:30 -0400
> Mike Gilbert <floppym@gentoo.org> wrote:
>
>> For those unaware, dev-util/ninja is a make-replacement created by one
>> of the Chromium guys at Google. Its focus is on making incremental
>> builds of large software faster.
>
> I've no idea how this would work out, so I'm asking some questions from
> different perspective to form a better idea; this may step aside your
> discussion but on the other hand could help other unaware developers.
>
> If I missed earlier ML discussions on this by not paying attention or
> being too new, feel free to point me to those; no need to repeat. :)
>

Ninja is simpler than make and strives to be as essential as possible, 
for those that know exotic build system is a close relative to plan9 mk.

Ninja doesn't make any effort to make its build rules human readable, as 
said they have to be generated by something else, not sure how easy/hard 
would be debug the generated file.

Being what it is used by default by upstream, probably it makes sense 
using it for building chromium.

Using it on cmake shouldn't make much, if any, difference speed wise and 
would just make cmake build require yet another building block so I 
wouldn't use it if there aren't clear advantages in doing so (somebody 
could benchmark so we can see how it fares)

lu

[gentoo-dev] Re: Usage of dev-utils/ninja in ebuilds

[Ryan Hill]

[-- Attachment #1: Type: text/plain, Size: 786 bytes --]

On Sat, 25 May 2013 14:48:30 -0400
Mike Gilbert <floppym@gentoo.org> wrote:

> I'm wondering if we should create a more global function for calling
> ninja in a consistent way. Maybe we should introduce a NINJAOPTS
> variable for users to set in make.conf. Should we create a new eclass
> for this, or just stick it in eutils?

Is NINJAOPTS a variable recognized by ninja or something that would be Gentoo
specific?  It'd be nice if you could parse out what you need from MAKEOPTS
instead, but I have no strong feelings.

If there's only two users in the tree right now I don't think an eclass is
warranted.


-- 
Ryan Hill                        psn: dirtyepic_sk
   gcc-porting/toolchain/wxwidgets @ gentoo.org

47C3 6D62 4864 0E49 8E9E  7F92 ED38 BD49 957A 8463

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Re: Usage of dev-utils/ninja in ebuilds

[Mike Gilbert]

On Sat, May 25, 2013 at 9:53 PM, Ryan Hill <dirtyepic@gentoo.org> wrote:
> On Sat, 25 May 2013 14:48:30 -0400
> Mike Gilbert <floppym@gentoo.org> wrote:
>
>> I'm wondering if we should create a more global function for calling
>> ninja in a consistent way. Maybe we should introduce a NINJAOPTS
>> variable for users to set in make.conf. Should we create a new eclass
>> for this, or just stick it in eutils?
>
> Is NINJAOPTS a variable recognized by ninja or something that would be Gentoo
> specific?  It'd be nice if you could parse out what you need from MAKEOPTS
> instead, but I have no strong feelings.
>

I was thinking of something like a "eninja" function, which would call
ninja ${NINJAOPTS}. But parsing MAKEOPTS would probably work just as
well.

> If there's only two users in the tree right now I don't think an eclass is
> warranted.
>

I can't really see ninja use really expanding too much. Maybe I will
wait a while and see if it happens.

Re: [gentoo-dev] Re: Usage of dev-utils/ninja in ebuilds

[Alex Xu]

[-- Attachment #1: Type: text/plain, Size: 219 bytes --]

On 25/05/13 09:53 PM, Ryan Hill wrote:
> Is NINJAOPTS a variable recognized by ninja or something that would be Gentoo
> specific?

MAKEOPTS is Gentoo-specific anyways. MAKEFLAGS is parsed by at least GNU
make.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] Usage of dev-utils/ninja in ebuilds

["Paweł Hajdan, Jr."]

[-- Attachment #1: Type: text/plain, Size: 1704 bytes --]

On 5/25/13 12:45 PM, Mike Gilbert wrote:
> You may see small (~60 seconds) difference when compiling Chromium due
> to faster target dependency resolution.

You may also see faster builds in general because ninja is designed for
very high degree of parallelization as compared to make.

>> From a maintainer's perspective, shouldn't the user be able to choose
>> whether ninja or make is used and not the developer? Or does ninja
>> really work out for the majority in a way nobody would complain?

Why would the user want to choose the build system?

> The choice of build system should be transparent to the user. However, I
> have no strong personal objection to making it configurable.

The build system of choice actually is completely transparent to the user...

>> From an upstream perspective, would it be problematic if people use
>> make instead of ninja; or wouldn't it translate into problems upstream
>> has to resolve? (To me, using ninja instead of make sounds like using
>> llvm instead of gcc; correct me if I'm wrong, I'm fairly new to ninja.)
>>
> So long as GYP supports emitting makefiles as well as ninja build files,
> I don't think this will be a problem.
> 
> For Gentoo, I believe phajdan.jr simply chose to switch to ninja to more
> closely mimic the normal upstream build process.

One part is indeed that the upstream default changed to ninja, but
another one is that makefile support will be going away. The faster we
can switch to ninja and fix any bugs resulting from that, the better.

Let me repeat: Chromium project will be dropping support for generating
makefiles, making ninja the only supported build system on Linux.

Paweł


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

[gentoo-dev] Re: Usage of dev-utils/ninja in ebuilds

[Ryan Hill]

[-- Attachment #1: Type: text/plain, Size: 707 bytes --]

On Sat, 25 May 2013 22:00:36 -0400
Alex Xu <alex_y_xu@yahoo.ca> wrote:

> On 25/05/13 09:53 PM, Ryan Hill wrote:
> > Is NINJAOPTS a variable recognized by ninja or something that would be
> > Gentoo specific?
> 
> MAKEOPTS is Gentoo-specific anyways. MAKEFLAGS is parsed by at least GNU
> make.

Yeah, so I'm saying we should use it if possible before introducing another
variable.  As a user you shouldn't have to know or care what build system a
package uses, or have to set a half dozen variables to get consistent
behaviour.


-- 
Ryan Hill                        psn: dirtyepic_sk
   gcc-porting/toolchain/wxwidgets @ gentoo.org

47C3 6D62 4864 0E49 8E9E  7F92 ED38 BD49 957A 8463

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]