[gentoo-user] how to prevent ebuild from checking for available space

[gentoo-user] how to prevent ebuild from checking for available space

[Jack]

[-- Attachment #1: Type: text/plain, Size: 1439 bytes --]

In this case, the offending package is dev-lang/rust, but this has 
happened to me previously with other packages that require a lot of 
space and time to build.

The build fails for any reason.  Since it has already progressed over 
half way through the build, I would like to continue the build instead 
of starting from scratch.  I fix whatever caused the issue.  In this 
particular case, it failed for out of space, so I added another G to the 
ramdisk mounted at /var/tmp/portageg.  I issue "ebuild 
path/to/package.ebuild compile" but it fails immediately because there 
isn't enough free space.  The problem is that it's checking for enough 
free space to do the entire build, not just to continue from where it 
left off.

I've done some searching, and found one old forum post which suggests to 
just temporarily remove the space check from the ebuild.

In this case, that seems to be line 236 of 
/usr/portage/dev-lang/rust/rust-1.75.0-r1.ebuild, which is 
"CHECKREQS_DISK_BUILD=${M}M check-reqs_pkg_${EBUILD_PHASE}" the last 
line withing pre_build_checks().  However, if I comment out that line, 
and run "ebuild path/to/ebuild manifest" another attempt to compile 
still gives the same error about not enough space.

Am I commenting the wrong line?  Have I missed something about where 
this check is atually done?  Is it actually possible to do what I'm 
trying to do?

Thanks for any pointers or suggestions.

Jack

[-- Attachment #2: Type: text/html, Size: 2006 bytes --]

Re: [gentoo-user] how to prevent ebuild from checking for available space

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 1917 bytes --]

On Fri, 22 Mar 2024 12:57:34 -0400, Jack wrote:

> In this case, the offending package is dev-lang/rust, but this has 
> happened to me previously with other packages that require a lot of 
> space and time to build.
> 
> The build fails for any reason.  Since it has already progressed over 
> half way through the build, I would like to continue the build instead 
> of starting from scratch.  I fix whatever caused the issue.  In this 
> particular case, it failed for out of space, so I added another G to
> the ramdisk mounted at /var/tmp/portageg.  I issue "ebuild 
> path/to/package.ebuild compile" but it fails immediately because there 
> isn't enough free space.  The problem is that it's checking for enough 
> free space to do the entire build, not just to continue from where it 
> left off.

Why not add more to the ramdisk, assuming it is a tmpfs. If it needs more
than your physical memory, it will use swap, but that won't happen
because you only need the extra space.

I generally avoid this problem by having large packages use a
PORTAGE_TMPDIR on a spinning disk. It may be slightly slower, but that
doesn't really matter on a long compile, not nearly as much as having to
restart it!

> 
> I've done some searching, and found one old forum post which suggests
> to just temporarily remove the space check from the ebuild.
> 
> In this case, that seems to be line 236 of 
> /usr/portage/dev-lang/rust/rust-1.75.0-r1.ebuild, which is 
> "CHECKREQS_DISK_BUILD=${M}M check-reqs_pkg_${EBUILD_PHASE}" the last 
> line withing pre_build_checks().  However, if I comment out that line, 
> and run "ebuild path/to/ebuild manifest" another attempt to compile 
> still gives the same error about not enough space.

Have you tried setting M to a smaller value immediately before that line?


-- 
Neil Bothwick

Religious error: (A)tone, (R)epent, (I)mmolate?

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] how to prevent ebuild from checking for available space

[Jack]

On 2024.03.22 16:22, Neil Bothwick wrote:
> On Fri, 22 Mar 2024 12:57:34 -0400, Jack wrote:
> 
> > In this case, the offending package is dev-lang/rust, but this has
> > happened to me previously with other packages that require a lot of
> > space and time to build.
> >
> > The build fails for any reason.  Since it has already progressed  
> over
> > half way through the build, I would like to continue the build  
> instead
> > of starting from scratch.  I fix whatever caused the issue.  In this
> > particular case, it failed for out of space, so I added another G to
> > the ramdisk mounted at /var/tmp/portageg.  I issue "ebuild
> > path/to/package.ebuild compile" but it fails immediately because  
> there
> > isn't enough free space.  The problem is that it's checking for  
> enough
> > free space to do the entire build, not just to continue from where  
> it
> > left off.
> 
> Why not add more to the ramdisk, assuming it is a tmpfs. If it needs  
> more
> than your physical memory, it will use swap, but that won't happen
> because you only need the extra space.
That's actually what I did.  The problem is not how to get enough  
space, it's how to resume the emerge instead of starting over, once I  
have added the space.

It was initially set to 14G (out of 32G RAM) and I added 2G.  I suppose  
I can add another 14G, but that would only leave 4G for the system  
itself.  I'm not sure how well that would work, but I suppose it's  
worth a try.  The worst that happens is I reboot and start the emerge  
from scratch.
> 
> I generally avoid this problem by having large packages use a
> PORTAGE_TMPDIR on a spinning disk. It may be slightly slower, but that
> doesn't really matter on a long compile, not nearly as much as having  
> to
> restart it!
> 
> >
> > I've done some searching, and found one old forum post which  
> suggests
> > to just temporarily remove the space check from the ebuild.
> >
> > In this case, that seems to be line 236 of
> > /usr/portage/dev-lang/rust/rust-1.75.0-r1.ebuild, which is
> > "CHECKREQS_DISK_BUILD=${M}M check-reqs_pkg_${EBUILD_PHASE}" the last
> > line withing pre_build_checks().  However, if I comment out that  
> line,
> > and run "ebuild path/to/ebuild manifest" another attempt to compile
> > still gives the same error about not enough space.
> 
> Have you tried setting M to a smaller value immediately before that  
> line?
I tried that first, and when that didn't work, I tried commenting out  
the line I thought was calling the check.  I'll try again, to be sure  
it was a reasonable value for M.
> 
> 
> --
> Neil Bothwick
> 
> Religious error: (A)tone, (R)epent, (I)mmolate?
> 

Re: [gentoo-user] how to prevent ebuild from checking for available space

[Jack]

On 2024.03.22 16:22, Neil Bothwick wrote:
> On Fri, 22 Mar 2024 12:57:34 -0400, Jack wrote:
> 
> > In this case, the offending package is dev-lang/rust, but this has
> > happened to me previously with other packages that require a lot of
> > space and time to build.
> >
> > The build fails for any reason.  Since it has already progressed  
> over
> > half way through the build, I would like to continue the build  
> instead
> > of starting from scratch.  I fix whatever caused the issue.  In this
> > particular case, it failed for out of space, so I added another G to
> > the ramdisk mounted at /var/tmp/portageg.  I issue "ebuild
> > path/to/package.ebuild compile" but it fails immediately because  
> there
> > isn't enough free space.  The problem is that it's checking for  
> enough
> > free space to do the entire build, not just to continue from where  
> it
> > left off.
> 
> Why not add more to the ramdisk, assuming it is a tmpfs. If it needs  
> more
> than your physical memory, it will use swap, but that won't happen
> because you only need the extra space.
> 
> I generally avoid this problem by having large packages use a
> PORTAGE_TMPDIR on a spinning disk. It may be slightly slower, but that
> doesn't really matter on a long compile, not nearly as much as having  
> to
> restart it!
> 
> >
> > I've done some searching, and found one old forum post which  
> suggests
> > to just temporarily remove the space check from the ebuild.
> >
> > In this case, that seems to be line 236 of
> > /usr/portage/dev-lang/rust/rust-1.75.0-r1.ebuild, which is
> > "CHECKREQS_DISK_BUILD=${M}M check-reqs_pkg_${EBUILD_PHASE}" the last
> > line withing pre_build_checks().  However, if I comment out that  
> line,
> > and run "ebuild path/to/ebuild manifest" another attempt to compile
> > still gives the same error about not enough space.
> 
> Have you tried setting M to a smaller value immediately before that  
> line?
It seems the problem is that the enviroment file in the temp dir of the  
build area is sourced when you run ebuild/emerge.  (It's among the  
first output when you run ebuild.)  Since that file was created based  
on the state of the ebuild when the first emerge/ebuild was run,  
editing the ebuild file doesn't affect it.  I ended up finding a place  
to unset CHECKREQS_FAILED, and the compile is now continuing.

Re: [gentoo-user] how to prevent ebuild from checking for available space

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 1152 bytes --]

On Fri, 22 Mar 2024 16:32:28 -0400, Jack wrote:

> > Why not add more to the ramdisk, assuming it is a tmpfs. If it needs  
> > more
> > than your physical memory, it will use swap, but that won't happen
> > because you only need the extra space.  
> That's actually what I did.  The problem is not how to get enough  
> space, it's how to resume the emerge instead of starting over, once I  
> have added the space.
> 
> It was initially set to 14G (out of 32G RAM) and I added 2G.  I suppose
> I can add another 14G, but that would only leave 4G for the system  
> itself.  I'm not sure how well that would work, but I suppose it's  
> worth a try.

tmpfs only uses the space it needs, so it would appear to the ebuild that
there is plenty of space, but it would only use and extra gig or two of
your RAM.

For me, avoiding tmpfs for big ebuilds is the least hassle, using
package.env.

% cat /etc/portage/package.env/chromium
www-client/chromium disk-tmpdir.conf ...

% cat /etc/portage/env/disk-tmpdir.conf 
PORTAGE_TMPDIR="/mnt/scratch"


-- 
Neil Bothwick

Linux like wigwam. No windows, no gates, Apache inside.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] how to prevent ebuild from checking for available space

[Bryan Gardiner]

On Fri, 22 Mar 2024 20:11:57 -0400
Jack <ostroffjh@users.sourceforge.net> wrote:

> It seems the problem is that the enviroment file in the temp dir of
> the build area is sourced when you run ebuild/emerge.  (It's among
> the first output when you run ebuild.)  Since that file was created
> based on the state of the ebuild when the first emerge/ebuild was
> run, editing the ebuild file doesn't affect it.  I ended up finding a
> place to unset CHECKREQS_FAILED, and the compile is now continuing.

According to check-reqs.eclass, you can also bypass the check by
exporting CHECKREQS_DONOTHING=1.  I assume this would work for a
resumed build too.

Cheers,
Bryan

Re: [gentoo-user] how to prevent ebuild from checking for available space

[Wol]

On 23/03/2024 09:20, Neil Bothwick wrote:
> tmpfs only uses the space it needs, so it would appear to the ebuild that
> there is plenty of space, but it would only use and extra gig or two of
> your RAM.
> 
> For me, avoiding tmpfs for big ebuilds is the least hassle, using
> package.env.

I've yet to set my current system up with a tmpfs, but that's my 
preferred setup.

Iirc, my current mobo takes 4x16GB chips, so "double ram as swap" makes 
128GB - I put that on every disk (I have 3 or 4), so I've probably got 
not far short of 1/2TB swap!

So nearly all of my build spaces, I would typically have a ramdisk of 
maybe 128GB.

As previously mentioned, it only uses what space it needs, so I guess 
with 32GB of real RAM, with most stuff my system wouldn't touch swap 
even with huge ramdisks being declared.

Cheers,
Wol