[gentoo-amd64] Resuming an emerge where it left off

[gentoo-amd64] Resuming an emerge where it left off

[Vladimir G. Ivanovic]

Is there a way of resuming an emerge of a package at exactly the place
where it stopped, like restating a make where it stopped?

For example, it took hours for me to get to the point where the emerge
of openoffice failed because I was using the wrong JDK. It would be nice
not to have to toss away all that work and to be able to restart from
the point where I left off.

I see in /etc/make.conf.example that there is a feature called "noauto"
that seems to do what I want, but every time I've tried using it, it has
restarted from the beginning. I didn't try it with openoffice, but if I
had I would have tried:

        ebuild openoffice-2.0.4.ebuild compile qmerge

Is what I'm after possible?

Thanks.

--- Vladimir

-- 
Vladimir G. Ivanovic <vgivanovic@comcast.net>

-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Resuming an emerge where it left off

[Sebastian Redl]

Vladimir G. Ivanovic wrote:

>Is what I'm after possible?
>  
>
AFAIK not through Portage. You could, however, go to the work directory 
and issue the commands of the ebuild yourself, then create a ".compiled" 
file in the right directory (parent directory of "work", I think), and 
finish with "ebuild install qmerge".

Of course, that voids the warranty ;)

Sebastian Redl
-- 
gentoo-amd64@gentoo.org mailing list

[gentoo-amd64] Re: Resuming an emerge where it left off

[Duncan]

Sebastian Redl <sebastian.redl@getdesigned.at> posted
4537D2A8.9080106@getdesigned.at, excerpted below, on  Thu, 19 Oct 2006
21:31:52 +0200:

> Vladimir G. Ivanovic wrote:
> 
>>Is what I'm after possible?
>>  
>>
> AFAIK not through Portage. You could, however, go to the work directory 
> and issue the commands of the ebuild yourself, then create a ".compiled" 
> file in the right directory (parent directory of "work", I think), and 
> finish with "ebuild install qmerge".
> 
> Of course, that voids the warranty ;)

It also screws things up a bit for portage as there's a couple things it
does beyond the ebuild at the end of the compile stage, that become holes
in portage's records later on if you simply issue the ebuild compile
commands and create the .compiled file manually.  I found this out the
hard way. =8^(

I forgot the details now, but I did have a (now outdated) portage patch
available that allowed a workaround.  After I got the memory stability
issue fixed that was causing my crashes, and I could reliably complete
merges, I didn't find it so necessary to maintain the patch any more.  I
could dust it off and brush up on the forgotten details (updating them as
necessary for newer portage), if it would be useful.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Re: Resuming an emerge where it left off

[Vladimir G. Ivanovic]

On Thu, 2006-10-19 at 20:25 +0000, Duncan wrote:
> > AFAIK not through Portage. You could, however, go to the work directory 
> > and issue the commands of the ebuild yourself, then create a ".compiled" 
> > file in the right directory (parent directory of "work", I think), and 
> > finish with "ebuild install qmerge".
> > 
> > Of course, that voids the warranty ;)
> 
> It also screws things up a bit for portage as there's a couple things it
> does beyond the ebuild at the end of the compile stage, that become holes
> in portage's records later on if you simply issue the ebuild compile
> commands and create the .compiled file manually.  I found this out the
> hard way. =8^(
> 
> I forgot the details now, but I did have a (now outdated) portage patch
> available that allowed a workaround.  After I got the memory stability
> issue fixed that was causing my crashes, and I could reliably complete
> merges, I didn't find it so necessary to maintain the patch any more.  I
> could dust it off and brush up on the forgotten details (updating them as
> necessary for newer portage), if it would be useful.
> 
Duncan,

I think my immediate need is over, and since most packages are not as
big as OpenOffice, I'm OK with re-emerging failed packages in their
entirety. 

That being said, I'd like to see the patch even if it's in an
inoperative state, just to see what's involved.

--- Vladimir

-- 
Vladimir G. Ivanovic <vgivanovic@comcast.net>

-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Resuming an emerge where it left off

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 534 bytes --]

On Thu, 19 Oct 2006 11:34:04 -0700, Vladimir G. Ivanovic wrote:

> Is there a way of resuming an emerge of a package at exactly the place
> where it stopped, like restating a make where it stopped?

ebuild /usr/portage/category/package-version.ebuild merge

Whereas emerge clears the work directory and starts again, running ebuild
directly does not. It's also possible with emerge with
FEATURES="keepwork" but I prefer the ebuild method.


-- 
Neil Bothwick

Phasers don't kill people...Unless you set them too high.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-amd64] Resuming an emerge where it left off

[Sebastian Redl]

Neil Bothwick wrote:
> ebuild /usr/portage/category/package-version.ebuild merge
>
> Whereas emerge clears the work directory and starts again, running ebuild
> directly does not. It's also possible with emerge with
> FEATURES="keepwork" but I prefer the ebuild method.
>   
With the Thunderbird ebuild, at least, directly invoking ebuild starts
compilation over, but it does not re-unpack the files. I'm not quite
sure how it does that, though. I know it re-runs configure, which might
have something to do with it.
It might actually resume properly on other ebuilds.

Sebastian Redl
-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Resuming an emerge where it left off

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 848 bytes --]

On Fri, 20 Oct 2006 01:15:08 +0200, Sebastian Redl wrote:

> > Whereas emerge clears the work directory and starts again, running
> > ebuild directly does not. It's also possible with emerge with
> > FEATURES="keepwork" but I prefer the ebuild method.
> >     
> With the Thunderbird ebuild, at least, directly invoking ebuild starts
> compilation over, but it does not re-unpack the files. I'm not quite
> sure how it does that, though. I know it re-runs configure, which might
> have something to do with it.
> It might actually resume properly on other ebuilds.

ebuild touches files in the work directory so it can see what has been
done. While configure may be run again, make won't recompile files that
don't need to be recompiled, so compilation picks up where it left off.

-- 
Neil Bothwick

SUBLIMINALsendmoneyTAGLINE

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-amd64] Resuming an emerge where it left off

[Vladimir G. Ivanovic]

On Fri, 2006-10-20 at 00:34 +0100, Neil Bothwick wrote:
> On Fri, 20 Oct 2006 01:15:08 +0200, Sebastian Redl wrote:
> 
> > > Whereas emerge clears the work directory and starts again, running
> > > ebuild directly does not. It's also possible with emerge with
> > > FEATURES="keepwork" but I prefer the ebuild method.

Thanks. I'll give "keepwork" and/or the ebuild method a try.

--- Vladimir

-- 
Vladimir G. Ivanovic <vgivanovic@comcast.net>

-- 
gentoo-amd64@gentoo.org mailing list

[gentoo-amd64] Re: Resuming an emerge where it left off

[Duncan]

"Vladimir G. Ivanovic" <vgivanovic@comcast.net> posted
1161293905.8172.15.camel@scarlatti.leonora.org, excerpted below, on  Thu,
19 Oct 2006 14:38:25 -0700:

> That being said, I'd like to see the patch even if it's in an inoperative
> state, just to see what's involved.

OK, here you go.  It was a bit of a hack but it worked for my purposes. 
You can see the dates and portage version number, which gives you an idea
of how stale these are. My initials are JED, so I use that to distinguish
stuff I've changed and avoid namespace collisions. 

What the first part does is add a check for an environmental variable
JEDPortageSkipCompile.  If found, the actual src_compile call (the stuff
in the ebuild) is skipped, but portage does the rest of its compile step,
mainly shuffling metadata around so it there where expected to be used in
the install/package/qmerge steps later, as if the src_compile call had
completed normally, but critically also creating that .compiled file.

The second part, which /may/ not be needed any more if I've read portage
changelogs and dev remarks correctly, simply added the missing check for
.compiled, directly paralleling the similar existing checks for .unpacked
and (IIRC) .installed.  Given the file was created, I think missing the
check was an oversight, but it wasn't there in these old portage versions
anyway, so I had to add it.

The third part is simply a patch to be applied to the portage ebuild,
invoking the patch to portage itself.  Again, the dates and portage
version numbers tell the age.  I'd obviously changed this one later than
the other one.

Usage after applying the patches was as follows.

If the compile step crashed, I could investigate and fix the problem if
necessary, then run whatever compile steps needed to be completed
manually, in the working dir.  Usually this simply meant rerunning the
make command, but sometimes there'd be a bit more to it. One looks at the
src_compile step in the ebuild to figure that out, but the important thing
was not having to rerun .configure thereby often eliminating much of the
compiling that had already been done, unless I had to.

After manually completing the make and/or anything else the ebuild's
src_compile did, I'd then "export JEDPortageSkipCompile=true", and run
ebuild <package> merge or package or whatever. (I always used package as I
created binpackages, then ran emerge -K to merge the binpkg, but those
uninterested in FEATURES=buildpkg could simply use ebuild <package> merge
and be done with it.) 

After completing the package, don't forget to unset JEDPortageSkipCompile
if you will be doing anything else with portage, or you'll end up with a
bunch of empty packages! =8^(

--- portage-2.0.51-r3/bin/ebuild.sh     2004-10-18 21:58:42.000000000 -0700
+++ portage/bin/ebuild.sh       2004-11-07 00:48:24.025367508 -0700
@@ -912,7 +912,8 @@
        #some packages use an alternative to $S to build in, cause
        #our libtool to create problematic .la files
        export PWORKDIR="$WORKDIR"
-       src_compile
+# JED> src_compile
+       [ -z $JEDPortageSkipCompile ] && src_compile
        #|| abort_compile "fail"
        cd "${BUILDDIR}"
        touch .compiled
@@ -1003,6 +1003,14 @@

 dyn_install() {
        trap "abort_install" SIGINT SIGQUIT
+# JED>
+       if [ ${BUILDDIR}/.installed -nt "${WORKDIR}" -a -d ${BUILDDIR}/image ]; then
+               echo ">>> It appears that ${PN} is already installed; skipping."
+               echo ">>> (Delete  ${BUILDDIR}/.installed to force reinstall.)"
+               trap SIGINT SIGQUIT
+               return
+       fi
+# JED<
        rm -rf "${BUILDDIR}/image"
        mkdir "${BUILDDIR}/image"
        if [ -d "${S}" ]; then


--- /p/sys-apps/portage/portage-2.0.51.22-r1.ebuild     2005-05-29 01:35:53.000000000 -0700
+++ /usr/local/p/sys-apps/portage/portage-2.0.51.22-r1.ebuild   2005-05-31 10:06:38.000000000 -0700
@@ -26,8 +26,12 @@
 src_unpack() {
        unpack ${A}
        cd "${S}"
        patch -p1 < ${FILESDIR}/2.0.51.22-fixes.patch
+
+# JED > patches
+       patch -p1 </usr/local/src/patch/portage/jed-portage.patch
+# JED < patches
 }

 src_compile() {
        python -O -c "import compileall; compileall.compile_dir('${S}/pym')"







-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Resuming an emerge where it left off

[Paul de Vrieze]

Vladimir G. Ivanovic wrote:
> Is there a way of resuming an emerge of a package at exactly the place
> where it stopped, like restating a make where it stopped?
> 
> For example, it took hours for me to get to the point where the emerge
> of openoffice failed because I was using the wrong JDK. It would be nice
> not to have to toss away all that work and to be able to restart from
> the point where I left off.
> 
> I see in /etc/make.conf.example that there is a feature called "noauto"
> that seems to do what I want, but every time I've tried using it, it has
> restarted from the beginning. I didn't try it with openoffice, but if I
> had I would have tried:
> 
>         ebuild openoffice-2.0.4.ebuild compile qmerge
> 
> Is what I'm after possible?

As long as you don't mess with the stages (the .compiled file for 
example) it should work more or less. A little work will be redone (like 
running configure again), but make will then skip most of the steps. 
This generally goes wrong if you change useflags or significantly change 
the environment (like a different major java version). The result 
normally is just another miscompile so it might be worth your effort.

Paul
-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Resuming an emerge where it left off

[Paul de Vrieze]

Vladimir G. Ivanovic wrote:
> On Fri, 2006-10-20 at 00:34 +0100, Neil Bothwick wrote:
>> On Fri, 20 Oct 2006 01:15:08 +0200, Sebastian Redl wrote:
>>
>>>> Whereas emerge clears the work directory and starts again, running
>>>> ebuild directly does not. It's also possible with emerge with
>>>> FEATURES="keepwork" but I prefer the ebuild method.
> 
> Thanks. I'll give "keepwork" and/or the ebuild method a try.
> 
> --- Vladimir
> 
A safe alternative that may safe some time, but not as much as make 
skipping files is ccache. It has checks that exactly all things are 
equal, so nothing could go wrong. It's as easy as installing ccache.

Paul
-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Re: Resuming an emerge where it left off

[Vladimir G. Ivanovic]

On Fri, 2006-10-20 at 11:33 +0000, Duncan wrote:
> "Vladimir G. Ivanovic" <vgivanovic@comcast.net> posted
> 1161293905.8172.15.camel@scarlatti.leonora.org, excerpted below, on  Thu,
> 19 Oct 2006 14:38:25 -0700:
> 
> > That being said, I'd like to see the patch even if it's in an inoperative
> > state, just to see what's involved.
> 
> OK, here you go.  

The writeup was way more than I had expected. Thanks.

--- Vladimir

-- 
Vladimir G. Ivanovic <vgivanovic@comcast.net>

-- 
gentoo-amd64@gentoo.org mailing list

[gentoo-amd64] Re: Resuming an emerge where it left off

[Duncan]

"Vladimir G. Ivanovic" <vgivanovic@comcast.net> posted
1161362486.12685.3.camel@scarlatti.leonora.org, excerpted below, on  Fri,
20 Oct 2006 09:41:26 -0700:

> The writeup was way more than I had expected. Thanks.

As regulars here will attest, that seems to be my style. <g>

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Re: Resuming an emerge where it left off

[Simon Stelling]

Duncan wrote:
>> The writeup was way more than I had expected. Thanks.
> 
> As regulars here will attest, that seems to be my style. <g>

You can say that again :D Just a few days ago I stumbled across that old 
mail from yours [1] and decided it would be worth to print out. I'm 
using it as wallpaper now, because the printout has exactly the same 
height as my room:

	http://blubb.ch/temp/duncanmail-printout.jpg

(I hope you don't take this as an insult, because it's not meant as one. 
I just find that picture hilarious, and as you're admitting that your 
style is rather special, I thought I could as well share this funny 
picture with the community.)

[1] http://article.gmane.org/gmane.linux.gentoo.amd64/6542/

-- 
Kind Regards,

Simon Stelling
Gentoo/AMD64 developer
-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Re: Resuming an emerge where it left off

[William Tetrault]

On Saturday 21 October 2006 07:27, Simon Stelling wrote:
> You can say that again :D Just a few days ago I stumbled across that old
> mail from yours [1] and decided it would be worth to print out. I'm
> using it as wallpaper now, because the printout has exactly the same
> height as my room:
>
> 	http://blubb.ch/temp/duncanmail-printout.jpg
>
> (I hope you don't take this as an insult, because it's not meant as one.
> I just find that picture hilarious, and as you're admitting that your
> style is rather special, I thought I could as well share this funny
> picture with the community.)
>
> [1] http://article.gmane.org/gmane.linux.gentoo.amd64/6542/
>
> --
> Kind Regards,
>
> Simon Stelling
> Gentoo/AMD64 developer

That's a hoot!  I've generally appreciated Duncan's long-winded explanations 
because I almost always learn something new from them, but that one <was> 
particularly long.  Now I know <how> long!

Bill Tetrault
Madison, WI
-- 
gentoo-amd64@gentoo.org mailing list

[gentoo-amd64] Re: Resuming an emerge where it left off

[Duncan]

Simon Stelling <blubb@gentoo.org> posted 453A1214.3050009@gentoo.org,
excerpted below, on  Sat, 21 Oct 2006 14:27:00 +0200:

> Duncan wrote:
>>> The writeup was way more than I had expected. Thanks.
>> 
>> As regulars here will attest, that seems to be my style. <g>
> 
> You can say that again :D Just a few days ago I stumbled across that old 
> mail from yours [1] and decided it would be worth to print out. I'm 
> using it as wallpaper now, because the printout has exactly the same 
> height as my room:
> 
> 	http://blubb.ch/temp/duncanmail-printout.jpg
> 
> (I hope you don't take this as an insult, because it's not meant as one. 
> I just find that picture hilarious, and as you're admitting that your 
> style is rather special, I thought I could as well share this funny 
> picture with the community.)
> 
> [1] http://article.gmane.org/gmane.linux.gentoo.amd64/6542/

Not an insult, now anyway.  I was a bit... mystified... as to why you'd
have pdf-ed it in your devspace [1] when someone (CiaranM as it happened)
referenced it as a case in point a few months ago, and would have
appreciated knowing why it was there, from you, before just coming across
a URL to it as I did, but then I realized that if it's posted in a public
forum such as this, I've no reason to be offended, certainly not when the
intent isn't obviously malevolent.  Anyway, that was after your "urgent
request for funds" for a new monitor, after one of the "heated debates" in
-dev, so after a few seconds I was able to appreciate the humor of the
situation from that perspective. =8^)

Shortly thereafter I actually linked to the pdf once or twice,
illustrating my own points, so I was quick to take advantage of it once I
knew it was there, in any case, and am actually sort of proud that my work
is considered unusual enough for such an honor, dubious tho it might be.
=8^)  "Who wants to be a Clone?" in the words of a song from a few years
ago.  Certainly not me, so if my work is unusual enough to stand alone
like that, well, I'll choose to take that as an honor! =8^)

[1] http://dev.gentoo.org/~blubb/duncan.pdf

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

-- 
gentoo-amd64@gentoo.org mailing list