[gentoo-portage-dev] running ebuild in src tree

[gentoo-portage-dev] running ebuild in src tree

[Joakim Tjernlund]

When developing code it would be really nice if one could run your ebuild
on that src tree as is(no fetch, unpack etc.)

Then one can compile, install, build a binary pkg etc. just for test. That way you shorten your 
development/test cycle, get to
partially test your brand new ebuild, don't have write
custom build scripts etc.

Any thoughts on this?

 Jocke 

Re: [gentoo-portage-dev] running ebuild in src tree

[Zac Medico]

On 03/11/2015 09:03 AM, Joakim Tjernlund wrote:
> When developing code it would be really nice if one could run your ebuild
> on that src tree as is(no fetch, unpack etc.)

The existing convention is to create an ebuild with version 9999 and use
one of the live vcs eclasses such as git-r3 to pull the live sources in
the src_unpack function. In a future EAPI, we plan to add some features
related to this [1].

> Then one can compile, install, build a binary pkg etc. just for test. That way you shorten your 
> development/test cycle, get to
> partially test your brand new ebuild, don't have write
> custom build scripts etc.
> 
> Any thoughts on this?
> 
>  Jocke 
> 

[1] https://bugs.gentoo.org/show_bug.cgi?id=182028
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] running ebuild in src tree

[Joakim Tjernlund]

On Wed, 2015-03-11 at 11:34 -0700, Zac Medico wrote:
> On 03/11/2015 09:03 AM, Joakim Tjernlund wrote:
> > When developing code it would be really nice if one could run your ebuild
> > on that src tree as is(no fetch, unpack etc.)
> 
> The existing convention is to create an ebuild with version 9999 and use one of the live vcs eclasses such 
> as git-r3 to pull the live sources in the src_unpack function. In a future EAPI, we plan to add some 
> features related to this [1].

I think you misunderstand, [1] is not what I want to do(I think):

Got my src working copy and made a few modds, not commitet yet.
Now I just want build/test etc. before committing and to do that I just
run mytree/overlay/dev-util/myapp/myapp.ebuild compile and voila, my code is
built which I already have in mytree.

> 
> > Then one can compile, install, build a binary pkg etc. just for test. That way you shorten your 
> > development/test cycle, get to
> > partially test your brand new ebuild, don't have write
> > custom build scripts etc.
> > 
> > Any thoughts on this?
> > 
> >  Jocke 
> > 
> 
> [1] https://bugs.gentoo.org/show_bug.cgi?id=182028

Re: [gentoo-portage-dev] running ebuild in src tree

[Zac Medico]

On 03/11/2015 11:56 AM, Joakim Tjernlund wrote:
> On Wed, 2015-03-11 at 11:34 -0700, Zac Medico wrote:
>> On 03/11/2015 09:03 AM, Joakim Tjernlund wrote:
>>> When developing code it would be really nice if one could run your ebuild
>>> on that src tree as is(no fetch, unpack etc.)
>>
>> The existing convention is to create an ebuild with version 9999 and use one of the live vcs eclasses such 
>> as git-r3 to pull the live sources in the src_unpack function. In a future EAPI, we plan to add some 
>> features related to this [1].
> 
> I think you misunderstand, [1] is not what I want to do(I think):
> 
> Got my src working copy and made a few modds, not commitet yet.
> Now I just want build/test etc. before committing and to do that I just
> run mytree/overlay/dev-util/myapp/myapp.ebuild compile and voila, my code is
> built which I already have in mytree.

Well, you can create a -9999 ebuild that copies your sources from
$directory to $WORKDIR. Maybe use an environment to configure whether it
pulls from a local directory or a vcs repository.
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] running ebuild in src tree

[Joakim Tjernlund]

On Wed, 2015-03-11 at 12:03 -0700, Zac Medico wrote:
> On 03/11/2015 11:56 AM, Joakim Tjernlund wrote:
> > On Wed, 2015-03-11 at 11:34 -0700, Zac Medico wrote:
> > > On 03/11/2015 09:03 AM, Joakim Tjernlund wrote:
> > > > When developing code it would be really nice if one could run your ebuild
> > > > on that src tree as is(no fetch, unpack etc.)
> > > 
> > > The existing convention is to create an ebuild with version 9999 and use one of the live vcs eclasses 
> > > such 
> > > as git-r3 to pull the live sources in the src_unpack function. In a future EAPI, we plan to add some 
> > > features related to this [1].
> > 
> > I think you misunderstand, [1] is not what I want to do(I think):
> > 
> > Got my src working copy and made a few modds, not commitet yet. Now I just want build/test etc. before 
> > committing and to do that I just run mytree/overlay/dev-util/myapp/myapp.ebuild compile and voila, my 
> > code is
> > built which I already have in mytree.
> 
> Well, you can create a -9999 ebuild that copies your sources from $directory to $WORKDIR. Maybe use an 
> environment to configure whether it pulls from a local directory or a vcs repository.

No copy, just use the current sources. I realize this is not supported ATM but I think
it would be a very useful extension. It could just be an extra option or two to ebuild

  Jocke

Re: [gentoo-portage-dev] running ebuild in src tree

[Zac Medico]

On 03/11/2015 12:08 PM, Joakim Tjernlund wrote:
> On Wed, 2015-03-11 at 12:03 -0700, Zac Medico wrote:
>> On 03/11/2015 11:56 AM, Joakim Tjernlund wrote:
>>> On Wed, 2015-03-11 at 11:34 -0700, Zac Medico wrote:
>>>> On 03/11/2015 09:03 AM, Joakim Tjernlund wrote:
>>>>> When developing code it would be really nice if one could run your ebuild
>>>>> on that src tree as is(no fetch, unpack etc.)
>>>>
>>>> The existing convention is to create an ebuild with version 9999 and use one of the live vcs eclasses 
>>>> such 
>>>> as git-r3 to pull the live sources in the src_unpack function. In a future EAPI, we plan to add some 
>>>> features related to this [1].
>>>
>>> I think you misunderstand, [1] is not what I want to do(I think):
>>>
>>> Got my src working copy and made a few modds, not commitet yet. Now I just want build/test etc. before 
>>> committing and to do that I just run mytree/overlay/dev-util/myapp/myapp.ebuild compile and voila, my 
>>> code is
>>> built which I already have in mytree.
>>
>> Well, you can create a -9999 ebuild that copies your sources from $directory to $WORKDIR. Maybe use an 
>> environment to configure whether it pulls from a local directory or a vcs repository.
> 
> No copy, just use the current sources. I realize this is not supported ATM but I think
> it would be a very useful extension. It could just be an extra option or two to ebuild

You can already write the code to do this in the ebuild. Maybe have it
create a symlink to your source directory. You can use environment
variables, so there's no need to special options for this. It's really
none of portage's business where your source code comes from.
-- 
Thanks,
Zac

[gentoo-portage-dev] Re: running ebuild in src tree

[Duncan]

Zac Medico posted on Wed, 11 Mar 2015 12:03:10 -0700 as excerpted:

> On 03/11/2015 11:56 AM, Joakim Tjernlund wrote:
>> On Wed, 2015-03-11 at 11:34 -0700, Zac Medico wrote:
>>> On 03/11/2015 09:03 AM, Joakim Tjernlund wrote:
>>>> When developing code it would be really nice if one could run your
>>>> ebuild on that src tree as is(no fetch, unpack etc.)
>>>
>>> The existing convention is to create an ebuild with version 9999 and
>>> use one of the live vcs eclasses such as git-r3 to pull the live
>>> sources in the src_unpack function. In a future EAPI, we plan to add
>>> some features related to this [1].
>> 
>> I think you misunderstand, [1] is not what I want to do(I think):
>> 
>> Got my src working copy and made a few modds, not commitet yet.
>> Now I just want build/test etc. before committing and to do that I just
>> run mytree/overlay/dev-util/myapp/myapp.ebuild compile and voila, my
>> code is built which I already have in mytree.
> 
> Well, you can create a -9999 ebuild that copies your sources from
> $directory to $WORKDIR. Maybe use an environment to configure whether it
> pulls from a local directory or a vcs repository.

@ Joakim T:

FWIW, a commonly recommended user-level portage optimization is to point 
$PORTAGE_TMPDIR at a tmpfs mount.  As long as you have sufficient memory, 
that lets all building take place in the tmpfs and thus in memory, 
eliminating many read-accesses and most/all write accesses to permanent 
storage during the build and (fake-)install phases.

In addition to speeding up emerge itself, this reduces build-time I/O, 
which often becomes the bottleneck on which other processes may be 
waiting as well, thus allowing other processes more efficient access to 
permanent storage while emerge is ongoing.  Between this and setting 
PORTAGE_NICENESS=20, emerge is /much/ better behaved during builds, 
interrupting other processes much less and thus letting you carry on with 
other things while emerge is doing its thing, with far less interruption. 
=:^)

For instance, here I have /tmp as a tmpfs mount (with /var/tmp being a 
bind-mount of the same tmpfs), and in make.conf, have the line:

PORTAGE_TMPDIR=/tmp

Emerge then creates /tmp/portage, and within it, creates the usual cat/
pkg/ build trees, etc, as it emerges various packages.


Obviously, your sources in permanent storage are going to be cache-copied 
into memory as you do the build anyway, and pointing PORTAGE_TMPDIR at 
tmpfs then becomes a copy to (tmpfs) memory only.  While that doesn't 
technically eliminate the copies (since the read into tmpfs will cache 
and you'll have the tmpfs copy as well), it DOES mean most of the work 
beyond the initial read into memory will be memory-only, so you DO 
eliminate the permanent-storage copies.

Is that sufficiently close to what you were looking to do?  Beyond that, 
as Zac suggests, just have the ebuild grab the sources from wherever you 
put them as your src_unpack, as at that point it'll be a copy to tmpfs, 
and thus take essentially the same time (or even less since it'll avoid 
the build-time writes to permanent storage) as doing the in-place build 
directly.  Plus, creating a tmpfs mount if necessary, and setting 
PORTAGE_TMPDIR, is easy, and you'll dramatically speed-up normal builds 
as well. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-portage-dev] Re: running ebuild in src tree

[Joakim Tjernlund]

On Thu, 2015-03-12 at 01:27 +0000, Duncan wrote:
> Zac Medico posted on Wed, 11 Mar 2015 12:03:10 -0700 as excerpted:
> 
> > On 03/11/2015 11:56 AM, Joakim Tjernlund wrote:
> > > On Wed, 2015-03-11 at 11:34 -0700, Zac Medico wrote:
> > > > On 03/11/2015 09:03 AM, Joakim Tjernlund wrote:
> > > > > When developing code it would be really nice if one could run your ebuild on that src tree as is(no 
> > > > > fetch, unpack etc.)
> > > > 
> > > > The existing convention is to create an ebuild with version 9999 and use one of the live vcs eclasses 
> > > > such as git-r3 to pull the live sources in the src_unpack function. In a future EAPI, we plan to add 
> > > > some features related to this [1].
> > > 
> > > I think you misunderstand, [1] is not what I want to do(I think):
> > > 
> > > Got my src working copy and made a few modds, not commitet yet. Now I just want build/test etc. before 
> > > committing and to do that I just run mytree/overlay/dev-util/myapp/myapp.ebuild compile and voila, my 
> > > code is built which I already have in mytree.
> > 
> > Well, you can create a -9999 ebuild that copies your sources from $directory to $WORKDIR. Maybe use an 
> > environment to configure whether it pulls from a local directory or a vcs repository.
> 
> @ Joakim T:
> 
> FWIW, a commonly recommended user-level portage optimization is to point 
> $PORTAGE_TMPDIR at a tmpfs mount.  As long as you have sufficient memory, 
> that lets all building take place in the tmpfs and thus in memory,  eliminating many read-accesses and 
> most/all write accesses to permanent  storage during the build and (fake-)install phases.
> 
> In addition to speeding up emerge itself, this reduces build-time I/O,  which often becomes the bottleneck 
> on which other processes may be  waiting as well, thus allowing other processes more efficient access to  
> permanent storage while emerge is ongoing.  Between this and setting  PORTAGE_NICENESS=20, emerge is /much/ 
> better behaved during builds,  interrupting other processes much less and thus letting you carry on with 
> other things while emerge is doing its thing, with far less interruption. 
> =:^)
> 
> For instance, here I have /tmp as a tmpfs mount (with /var/tmp being a  bind-mount of the same tmpfs), and 
> in make.conf, have the line:
> 
> PORTAGE_TMPDIR=/tmp
> 
> Emerge then creates /tmp/portage, and within it, creates the usual cat/ pkg/ build trees, etc, as it 
> emerges various packages.
> 
> 
> Obviously, your sources in permanent storage are going to be cache-copied 
> into memory as you do the build anyway, and pointing PORTAGE_TMPDIR at  tmpfs then becomes a copy to 
> (tmpfs) memory only.  While that doesn't  technically eliminate the copies (since the read into tmpfs will 
> cache  and you'll have the tmpfs copy as well), it DOES mean most of the work  beyond the initial read into 
> memory will be memory-only, so you DO  eliminate the permanent-storage copies.
> 
> Is that sufficiently close to what you were looking to do?  Beyond that, 
> as Zac suggests, just have the ebuild grab the sources from wherever you 
> put them as your src_unpack, as at that point it'll be a copy to tmpfs,  and thus take essentially the same 
> time (or even less since it'll avoid  the build-time writes to permanent storage) as doing the in-place 
> build  directly.  Plus, creating a tmpfs mount if necessary, and setting  PORTAGE_TMPDIR, is easy, and 
> you'll dramatically speed-up normal builds  as well. =:^)
> 

No, there can be no copy of sources for what I want. It just gets in the way having to do that.
Hacks like this seems to work:

post_src_compile() {
        # make it compile every time
        rm -f ${PORTAGE_BUILDDIR}/.compiled
}

post_src_install() {
        # make it install every time
        rm -f ${PORTAGE_BUILDDIR}/.installed
}

#hmm, doesn't seem to be an post_package function
#post_package() {
#       rm -f ${PORTAGE_BUILDDIR}/.packaged
#}

src_unpack() {
        #dir need to exist
        mkdir -p ${S} || die
}
src_compile() {
        EBUILDDIR=$(dirname ${FILESDIR})
        MYTRUNK=${EBUILDDIR}/../../..
        # add sandbox exceptions
        addwrite ${MYTRUNK}
        
        cd ${MYTRUNK} || die
        cd "${PN}"
.....
}

[gentoo-portage-dev] Re: running ebuild in src tree

[Duncan]

Joakim Tjernlund posted on Thu, 12 Mar 2015 17:26:59 +0000 as excerpted:

> No, there can be no copy of sources for what I want. It just gets in the
> way having to do that.

??

Copying to tmpfs is copying to memory, and copying to memory in 
/some/ form must occur in ordered to operate on the files at all, that 
is, with any build at all, so I don't see the problem, at least if you're 
working on a machine with say 2+ gigs RAM.  (There might be problems on a 
half-gig RAM machine, but that's not particularly appropriate as a 
developer machine in any case, these days, unless you're talking 
embedded, which you didn't mention.)

But of course, gentoo/portage lets you do it your way too, as 
demonstrated by the hacks you posted. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-portage-dev] Re: running ebuild in src tree

[Joakim Tjernlund]

On Fri, 2015-03-13 at 06:32 +0000, Duncan wrote:
> Joakim Tjernlund posted on Thu, 12 Mar 2015 17:26:59 +0000 as excerpted:
> 
> > No, there can be no copy of sources for what I want. It just gets in the way having to do that.
> 
> ??
> 
> Copying to tmpfs is copying to memory, and copying to memory in  /some/ form must occur in ordered to 
> operate on the files at all, that  is, with any build at all, so I don't see the problem, at least if 
> you're 
> working on a machine with say 2+ gigs RAM.  (There might be problems on a 
> half-gig RAM machine, but that's not particularly appropriate as a  developer machine in any case, these 
> days, unless you're talking  embedded, which you didn't mention.)
> 
> But of course, gentoo/portage lets you do it your way too, as  demonstrated by the hacks you posted. =:^)
> 

When you are developing SW you do the edit/build cycle often and it is really 
annoying to copy a big src tree, rebuild everything (as timestamps, deps changes), move to another
src to do any debugging etc. Try it on your own development by just copying you src tree every time
you want to build, you get very tired of it real soon :)

Don't get me wrong, portage is a wounderful pkg mgr, I just want to extend its use to be more
helpful during development too, without having to do really ugly hacks like I showed you.

   Jocke

[gentoo-portage-dev] Re: running ebuild in src tree

[Duncan]

Joakim Tjernlund posted on Fri, 13 Mar 2015 08:13:01 +0000 as excerpted:

> When you are developing SW you do the edit/build cycle often and it is
> really annoying to copy a big src tree, rebuild everything (as
> timestamps, deps changes), move to another src to do any debugging etc.
> Try it on your own development by just copying you src tree every time
> you want to build, you get very tired of it real soon :)

That's what ccache is for. For some time I was following live-kde (tho 
I'm not ATM as kf5 wasn't working for me yet, last I tried, and kde4 
development is pretty much dead, now) and doing rebuilds several times a 
week, so I know what it's like. =:^)

But I do get your point, now.

Still, unless it's something like firefox, while I used to get irritated 
with the extra build time back when I was running a dual-core, a quad-
core improved that, and with my current 6-core bulldozer-1 (fx6100), 
PORTAGE_TMPDIR on tmpfs, and the system and ccache on ssd, most of the 
time I just let it do the rebuild.  It's really not worth the trouble 
worrying about it any more, particularly when I can be doing other stuff 
while it builds.

But you're right, being able to build right in an existing git clone 
without the hassle of hacks such as you posted, and even configuring live-
ebuilds to do just that with say a variable setting that could be set by 
a package.env pointer, would be nice indeed.  Local hacking on such live-
build packages would be quite a bit easier, that way.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-portage-dev] Re: running ebuild in src tree

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 4948 bytes --]

On Thu, Mar 12, 2015 at 10:26 AM, Joakim Tjernlund <
joakim.tjernlund@transmode.se> wrote:

> On Thu, 2015-03-12 at 01:27 +0000, Duncan wrote:
> > Zac Medico posted on Wed, 11 Mar 2015 12:03:10 -0700 as excerpted:
> >
> > > On 03/11/2015 11:56 AM, Joakim Tjernlund wrote:
> > > > On Wed, 2015-03-11 at 11:34 -0700, Zac Medico wrote:
> > > > > On 03/11/2015 09:03 AM, Joakim Tjernlund wrote:
> > > > > > When developing code it would be really nice if one could run
> your ebuild on that src tree as is(no
> > > > > > fetch, unpack etc.)
> > > > >
> > > > > The existing convention is to create an ebuild with version 9999
> and use one of the live vcs eclasses
> > > > > such as git-r3 to pull the live sources in the src_unpack
> function. In a future EAPI, we plan to add
> > > > > some features related to this [1].
> > > >
> > > > I think you misunderstand, [1] is not what I want to do(I think):
> > > >
> > > > Got my src working copy and made a few modds, not commitet yet. Now
> I just want build/test etc. before
> > > > committing and to do that I just run
> mytree/overlay/dev-util/myapp/myapp.ebuild compile and voila, my
> > > > code is built which I already have in mytree.
> > >
> > > Well, you can create a -9999 ebuild that copies your sources from
> $directory to $WORKDIR. Maybe use an
> > > environment to configure whether it pulls from a local directory or a
> vcs repository.
> >
> > @ Joakim T:
> >
> > FWIW, a commonly recommended user-level portage optimization is to point
> > $PORTAGE_TMPDIR at a tmpfs mount.  As long as you have sufficient memory,
> > that lets all building take place in the tmpfs and thus in memory,
> eliminating many read-accesses and
> > most/all write accesses to permanent  storage during the build and
> (fake-)install phases.
> >
> > In addition to speeding up emerge itself, this reduces build-time I/O,
> which often becomes the bottleneck
> > on which other processes may be  waiting as well, thus allowing other
> processes more efficient access to
> > permanent storage while emerge is ongoing.  Between this and setting
> PORTAGE_NICENESS=20, emerge is /much/
> > better behaved during builds,  interrupting other processes much less
> and thus letting you carry on with
> > other things while emerge is doing its thing, with far less interruption.
> > =:^)
> >
> > For instance, here I have /tmp as a tmpfs mount (with /var/tmp being a
> bind-mount of the same tmpfs), and
> > in make.conf, have the line:
> >
> > PORTAGE_TMPDIR=/tmp
> >
> > Emerge then creates /tmp/portage, and within it, creates the usual cat/
> pkg/ build trees, etc, as it
> > emerges various packages.
> >
> >
> > Obviously, your sources in permanent storage are going to be cache-copied
> > into memory as you do the build anyway, and pointing PORTAGE_TMPDIR at
> tmpfs then becomes a copy to
> > (tmpfs) memory only.  While that doesn't  technically eliminate the
> copies (since the read into tmpfs will
> > cache  and you'll have the tmpfs copy as well), it DOES mean most of the
> work  beyond the initial read into
> > memory will be memory-only, so you DO  eliminate the permanent-storage
> copies.
> >
> > Is that sufficiently close to what you were looking to do?  Beyond that,
> > as Zac suggests, just have the ebuild grab the sources from wherever you
> > put them as your src_unpack, as at that point it'll be a copy to tmpfs,
> and thus take essentially the same
> > time (or even less since it'll avoid  the build-time writes to permanent
> storage) as doing the in-place
> > build  directly.  Plus, creating a tmpfs mount if necessary, and
> setting  PORTAGE_TMPDIR, is easy, and
> > you'll dramatically speed-up normal builds  as well. =:^)
> >
>
> No, there can be no copy of sources for what I want. It just gets in the
> way having to do that.
> Hacks like this seems to work:
>
>
I wouldn't really call it a hack either, but whichever ;)


> post_src_compile() {
>         # make it compile every time
>         rm -f ${PORTAGE_BUILDDIR}/.compiled
> }
>
> post_src_install() {
>         # make it install every time
>         rm -f ${PORTAGE_BUILDDIR}/.installed
> }
>
> #hmm, doesn't seem to be an post_package function
> #post_package() {
> #       rm -f ${PORTAGE_BUILDDIR}/.packaged
> #}
>
> src_unpack() {
>         #dir need to exist
>         mkdir -p ${S} || die
> }
> src_compile() {
>         EBUILDDIR=$(dirname ${FILESDIR})
>         MYTRUNK=${EBUILDDIR}/../../..
>         # add sandbox exceptions
>         addwrite ${MYTRUNK}
>
>         cd ${MYTRUNK} || die
>         cd "${PN}"
> .....
> }
>
> I'm a bit curious what sort of workflow you are trying to achieve here is.
Which build artifacts are you looking for; the actual built binaries or the
built package that is merged into the livefs?

If the artifacts are packages, then I think this approach makes some
sense...but otherwise I'd just go into trunk and run the build process
(what does the ebuild gain you here?)

-A

[-- Attachment #2: Type: text/html, Size: 6222 bytes --]