[gentoo-user] Building packages in different prefix without rebuilding system packages

[gentoo-user] Building packages in different prefix without rebuilding system packages

[François-Xavier Carton]

Hi,

Is there a way of installing packages in a different prefix while still
using system packages? I've tried setting EPREFIX, however doing that
will install all dependencies in the prefix, even if there are already
installed in the system.

I was hoping to install some packages in user directories, but I also
don't want to duplicate the packages installed globally. For example,
most packages eventually depend on gcc, which I definitely don't want to
compile twice. So ideally, only dependencies that are not installed
globally should be pulled in.

I was not able to find a way of doing that, but I feel like it shouldn't
be too hard, because EPREFIX almost does what I want. Does someone know
if it's possible without too much tweaking?

Thanks,
-François-Xavier

Re: [gentoo-user] Building packages in different prefix without rebuilding system packages

[Dale]

[-- Attachment #1: Type: text/plain, Size: 1149 bytes --]

François-Xavier Carton wrote:
> Hi,
>
> Is there a way of installing packages in a different prefix while still
> using system packages? I've tried setting EPREFIX, however doing that
> will install all dependencies in the prefix, even if there are already
> installed in the system.
>
> I was hoping to install some packages in user directories, but I also
> don't want to duplicate the packages installed globally. For example,
> most packages eventually depend on gcc, which I definitely don't want to
> compile twice. So ideally, only dependencies that are not installed
> globally should be pulled in.
>
> I was not able to find a way of doing that, but I feel like it shouldn't
> be too hard, because EPREFIX almost does what I want. Does someone know
> if it's possible without too much tweaking?
>
> Thanks,
> -François-Xavier
>

I'm clueless on EPREFIX but if you want to avoid compiling a package
twice, would the -k option help?  If you have portage set to save the
binaries you compiled before, it would install from that instead of
compiling things twice. 

Just thought I'd mention just in case it would help. 

Dale

:-)  :-) 

[-- Attachment #2: Type: text/html, Size: 1512 bytes --]

Re: [gentoo-user] Building packages in different prefix without rebuilding system packages

[Michael]

[-- Attachment #1: Type: text/plain, Size: 2921 bytes --]

On Thursday, 14 May 2020 06:13:33 BST Dale wrote:
> François-Xavier Carton wrote:
> > Hi,
> > 
> > Is there a way of installing packages in a different prefix while still
> > using system packages? I've tried setting EPREFIX, however doing that
> > will install all dependencies in the prefix, even if there are already
> > installed in the system.
> > 
> > I was hoping to install some packages in user directories, but I also
> > don't want to duplicate the packages installed globally. For example,
> > most packages eventually depend on gcc, which I definitely don't want to
> > compile twice. So ideally, only dependencies that are not installed
> > globally should be pulled in.
> > 
> > I was not able to find a way of doing that, but I feel like it shouldn't
> > be too hard, because EPREFIX almost does what I want. Does someone know
> > if it's possible without too much tweaking?
> > 
> > Thanks,
> > -François-Xavier
> 
> I'm clueless on EPREFIX but if you want to avoid compiling a package
> twice, would the -k option help?  If you have portage set to save the
> binaries you compiled before, it would install from that instead of
> compiling things twice. 
> 
> Just thought I'd mention just in case it would help. 
> 
> Dale
> 
> :-)  :-) 

The whole concept of EPREFIX is predicated on installing a Gentoo system 
within a different path/filesystem than the / of the host installation and 
being able to run it as a non-root user.  As I understand it, using libraries 
from the main system and potentially altering them in the process is not going 
to work without changing Gentoo's eprefix intended design.

It should be possible to change the prefix paths selectively, in particular 
the LD_LIBRARY_PATH to link binaries from within the prefix to libraries in 
the host system, but I'm not sure what privileges are needed to install/run 
such a hybrid linkage and how an update of the host system will break 
installed packages within the EPREFIX and vice versa.  We're talking of a 
Frankenstein build here with the potential of install operations on one system 
would be breaking the other, including portage itself. 

With containerisation of applications there may be easier ways to achieve what  
François-Xavier is looking for.  I am thinking of running sandboxed 
applications in the likes of flatpack, snap, zero-install, appimage and 
whatever else may have been devised lately.  However, with these systems you 
end up using what's already been developed and any static libraries their devs 
considered desirable.  If you want a bespoke installation optimised for your 
hardware and chosen compilation flags, then you are probably looking to 
develop a containerised application for your own use.

Someone more knowledgeable in both Gentoo's EPREFIX project and containerised 
apps should chime in soon to offer more helpful advice.

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] Building packages in different prefix without rebuilding system packages

[François-Xavier Carton]

On Thu, May 14, 2020 at 09:07:43AM +0100, Michael wrote:
> On Thursday, 14 May 2020 06:13:33 BST Dale wrote:
> > François-Xavier Carton wrote:
> > > Hi,
> > > 
> > > Is there a way of installing packages in a different prefix while still
> > > using system packages? I've tried setting EPREFIX, however doing that
> > > will install all dependencies in the prefix, even if there are already
> > > installed in the system.
> > > 
> > > I was hoping to install some packages in user directories, but I also
> > > don't want to duplicate the packages installed globally. For example,
> > > most packages eventually depend on gcc, which I definitely don't want to
> > > compile twice. So ideally, only dependencies that are not installed
> > > globally should be pulled in.
> > > 
> > > I was not able to find a way of doing that, but I feel like it shouldn't
> > > be too hard, because EPREFIX almost does what I want. Does someone know
> > > if it's possible without too much tweaking?
> > > 
> > > Thanks,
> > > -François-Xavier
> > 
> > I'm clueless on EPREFIX but if you want to avoid compiling a package
> > twice, would the -k option help?  If you have portage set to save the
> > binaries you compiled before, it would install from that instead of
> > compiling things twice. 
> > 
> > Just thought I'd mention just in case it would help. 
> > 
> > Dale
> > 
> > :-)  :-) 
> 
> The whole concept of EPREFIX is predicated on installing a Gentoo system 
> within a different path/filesystem than the / of the host installation and 
> being able to run it as a non-root user.  As I understand it, using libraries 
> from the main system and potentially altering them in the process is not going 
> to work without changing Gentoo's eprefix intended design.
> 
> It should be possible to change the prefix paths selectively, in particular 
> the LD_LIBRARY_PATH to link binaries from within the prefix to libraries in 
> the host system, but I'm not sure what privileges are needed to install/run 
> such a hybrid linkage and how an update of the host system will break 
> installed packages within the EPREFIX and vice versa.  We're talking of a 
> Frankenstein build here with the potential of install operations on one system 
> would be breaking the other, including portage itself. 
> 
> With containerisation of applications there may be easier ways to achieve what  
> François-Xavier is looking for.  I am thinking of running sandboxed 
> applications in the likes of flatpack, snap, zero-install, appimage and 
> whatever else may have been devised lately.  However, with these systems you 
> end up using what's already been developed and any static libraries their devs 
> considered desirable.  If you want a bespoke installation optimised for your 
> hardware and chosen compilation flags, then you are probably looking to 
> develop a containerised application for your own use.
> 
> Someone more knowledgeable in both Gentoo's EPREFIX project and containerised 
> apps should chime in soon to offer more helpful advice.

I wouldn't try to install different versions of the same package, just
add new packages to the base system. But I can see your point about
updates; since the base system is not aware of the prefixed one, it
cannot rebuild the packages in the prefix automatically when needed.
It's indeed more complex than what I hoped :(

RE: [gentoo-user] Building packages in different prefix without rebuilding system packages

[Pengcheng Xu]

[-- Attachment #1: Type: text/plain, Size: 2511 bytes --]

François-Xavier's first email didn't make its way to my inbox, so I'm replying to this one instead.  EPREFIX is specifically designed for the Gentoo Prefix project [1].  In a word, Portage installs _everything_ inside the prefix, and uses nothing except the kernel (and some user files under home) outside the prefix.  It is widely used in supercomputing and corporate infrastructure where the user does not have escalated permissions and the existing packages are too old or missing (CentOS 5) for whatever the user is trying to do.

What you're trying to achieve sounded a lot like `./configure --prefix=...` to me.  If you're just dealing a small amount of things, I would suggest modifying the ebuild (in a local overlay) and changes where the program installs to.  That means no special handling on Portage's side, just the app's installed to somewhere else rather than the default prefix (in most cases '/usr').

[1]: https://wiki.gentoo.org/wiki/Project:Prefix

Regards,
-- 
Pengcheng Xu
https://jsteward.moe

> -----Original Message-----
> From: Dale <rdalek1967@gmail.com>
> Sent: Thursday, May 14, 2020 1:14 PM
> To: gentoo-user@lists.gentoo.org
> Subject: Re: [gentoo-user] Building packages in different prefix without
> rebuilding system packages
> 
> François-Xavier Carton wrote:
> 
> 
> 	Hi,
> 
> 	Is there a way of installing packages in a different prefix while still
> 	using system packages? I've tried setting EPREFIX, however doing that
> 	will install all dependencies in the prefix, even if there are already
> 	installed in the system.
> 
> 	I was hoping to install some packages in user directories, but I also
> 	don't want to duplicate the packages installed globally. For example,
> 	most packages eventually depend on gcc, which I definitely don't want
> to
> 	compile twice. So ideally, only dependencies that are not installed
> 	globally should be pulled in.
> 
> 	I was not able to find a way of doing that, but I feel like it shouldn't
> 	be too hard, because EPREFIX almost does what I want. Does someone know
> 	if it's possible without too much tweaking?
> 
> 	Thanks,
> 	-François-Xavier
> 
> 
> 
> I'm clueless on EPREFIX but if you want to avoid compiling a package twice,
> would the -k option help?  If you have portage set to save the binaries you
> compiled before, it would install from that instead of compiling things twice.
> 
> Just thought I'd mention just in case it would help.
> 
> Dale
> 
> :-)  :-)

[-- Attachment #2: openpgp-digital-signature.asc --]
[-- Type: application/pgp-signature, Size: 834 bytes --]

Re: [gentoo-user] Building packages in different prefix without rebuilding system packages

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 464 bytes --]

On Thu, 14 May 2020 16:46:58 +0800, Pengcheng Xu wrote:

> What you're trying to achieve sounded a lot like `./configure
> --prefix=...` to me.  If you're just dealing a small amount of things,
> I would suggest modifying the ebuild (in a local overlay) and changes
> where the program installs to.

You may be able to avoid modifying ebuilds by setting the --prefix option
in $EXTRA_ECONF.


-- 
Neil Bothwick

Top Oxymorons Number 8: Tight slacks

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

RE: [gentoo-user] Building packages in different prefix without rebuilding system packages

[Pengcheng Xu]

[-- Attachment #1: Type: text/plain, Size: 974 bytes --]

That seems interesting.  Do we need to include Portage install prefix (/var/tmp/portage/category/package/..., the image path prefix before actually merging with /)?

Regards,
-- 
Pengcheng Xu
https://jsteward.moe

> -----Original Message-----
> From: Neil Bothwick <neil@digimed.co.uk>
> Sent: Thursday, May 14, 2020 6:07 PM
> To: gentoo-user@lists.gentoo.org
> Subject: Re: [gentoo-user] Building packages in different prefix without
> rebuilding system packages
> 
> On Thu, 14 May 2020 16:46:58 +0800, Pengcheng Xu wrote:
> 
> > What you're trying to achieve sounded a lot like `./configure
> > --prefix=...` to me.  If you're just dealing a small amount of things,
> > I would suggest modifying the ebuild (in a local overlay) and changes
> > where the program installs to.
> 
> You may be able to avoid modifying ebuilds by setting the --prefix option in
> $EXTRA_ECONF.
> 
> 
> --
> Neil Bothwick
> 
> Top Oxymorons Number 8: Tight slacks

[-- Attachment #2: openpgp-digital-signature.asc --]
[-- Type: application/pgp-signature, Size: 834 bytes --]

Re: [gentoo-user] Building packages in different prefix without rebuilding system packages

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 610 bytes --]

On Thu, 14 May 2020 18:17:06 +0800, Pengcheng Xu wrote:

> That seems interesting.  Do we need to include Portage install prefix
> (/var/tmp/portage/category/package/..., the image path prefix before
> actually merging with /)?
> 
> Regards,

No, just the --prefix=/home/blah/ that you want added to the ./configure
invocation.

Please don't top post, and definitely don't quote previous mails after
your signature separator as some mail clients are configured to exclude
sigs from quoted replies.

 
-- 
Neil Bothwick

"Self-explanatory": technospeak for "Incomprehensible & undocumented"

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

RE: [gentoo-user] Building packages in different prefix without rebuilding system packages

[Pengcheng Xu]

[-- Attachment #1: Type: text/plain, Size: 1165 bytes --]

> -----Original Message-----
> From: Neil Bothwick <neil@digimed.co.uk>
> Sent: Thursday, May 14, 2020 7:56 PM
> To: gentoo-user@lists.gentoo.org
> Subject: Re: [gentoo-user] Building packages in different prefix without
> rebuilding system packages
> 
> On Thu, 14 May 2020 18:17:06 +0800, Pengcheng Xu wrote:
> 
> > That seems interesting.  Do we need to include Portage install prefix
> > (/var/tmp/portage/category/package/..., the image path prefix before
> > actually merging with /)?
> >
> > Regards,
> 
> No, just the --prefix=/home/blah/ that you want added to the ./configure
> invocation.

That's pretty neat!

> 
> Please don't top post, and definitely don't quote previous mails after your
> signature separator as some mail clients are configured to exclude sigs from
> quoted replies.
> 

Sorry for that.  It's a shame that Outlook does not provide an option for signature location in replies... (using Windows for daily office stuff, please don't roast me :)

> 
> --
> Neil Bothwick
> 
> "Self-explanatory": technospeak for "Incomprehensible & undocumented"



Regards,
-- 
Pengcheng Xu
https://jsteward.moe

[-- Attachment #2: openpgp-digital-signature.asc --]
[-- Type: application/pgp-signature, Size: 834 bytes --]

Re: [gentoo-user] Building packages in different prefix without rebuilding system packages

[Michael Orlitzky]

On 5/14/20 7:55 AM, Neil Bothwick wrote:
> On Thu, 14 May 2020 18:17:06 +0800, Pengcheng Xu wrote:
> 
>> That seems interesting.  Do we need to include Portage install prefix
>> (/var/tmp/portage/category/package/..., the image path prefix before
>> actually merging with /)?
>>
>> Regards,
> 
> No, just the --prefix=/home/blah/ that you want added to the ./configure
> invocation.
> 

This is a good way to install packages that you've built by hand into
(say) your home directory, but it will cause problems if you try to
trick portage into doing it. The big problem is that no other packages
are going to know where to find the thing you just installed. Everything
else in the Gentoo repository is designed to use standard values of
PATH, LD_LIBRARY_PATH, the compiler's include dir, PKG_CONFIG_PATH, etc.
If you take one program and put it somewhere non-standard, then every
package depending on it is going to break.

If you install an *additional* copy (built by hand) in your home
directory, that's fine -- the system copy will still be in the right
place -- you just don't want to hide the system copy where nobody can
find it.

Re: [gentoo-user] Building packages in different prefix without rebuilding system packages

[François-Xavier Carton]

On Thu, May 14, 2020 at 09:26:10AM -0400, Michael Orlitzky wrote:
> On 5/14/20 7:55 AM, Neil Bothwick wrote:
> > On Thu, 14 May 2020 18:17:06 +0800, Pengcheng Xu wrote:
> > 
> >> That seems interesting.  Do we need to include Portage install prefix
> >> (/var/tmp/portage/category/package/..., the image path prefix before
> >> actually merging with /)?
> >>
> >> Regards,
> > 
> > No, just the --prefix=/home/blah/ that you want added to the ./configure
> > invocation.
> > 
> 
> This is a good way to install packages that you've built by hand into
> (say) your home directory, but it will cause problems if you try to
> trick portage into doing it. The big problem is that no other packages
> are going to know where to find the thing you just installed. Everything
> else in the Gentoo repository is designed to use standard values of
> PATH, LD_LIBRARY_PATH, the compiler's include dir, PKG_CONFIG_PATH, etc.
> If you take one program and put it somewhere non-standard, then every
> package depending on it is going to break.
> 
> If you install an *additional* copy (built by hand) in your home
> directory, that's fine -- the system copy will still be in the right
> place -- you just don't want to hide the system copy where nobody can
> find it.
> 

In my case, this wouldn't be a problem: I don't want the packages to be
accessed by anyone, just one user. I can set PATH, LD_LIBRARY_PATH,
PKG_CONFIG_PATH and MANPATH for that user. I already do that for things
I build manually anyway.

EXTRA_ECONF is nice, I didn't know about it. It looks like MYCMAKEARGS
can be used for cmake ebuilds. For other build systems, it might be
necessary to edit the ebuild, or set different variables.

I still kinda think that being able to install with a prefix (like
EPREFIX) but using the base system would be a nice feature. As discussed
previously, there would be the problem of updates; but it isn't very
different from installing software manually. If I clone something and
build it locally, a world update might break break it, and portage
cannot rebuild it automatically since it's not aware of it. This is why
I think it would be nice if portage supported it; that way, after an
update of the base system, updating the prefix system would solve the
problems. It's probably difficult to implement that in portage, though.

[gentoo-user] Re: Building packages in different prefix without rebuilding system packages

[Michael Haubenwallner]

Hi François-Xavier,

On 5/14/20 7:02 AM, François-Xavier Carton wrote:
> Hi,
> 
> Is there a way of installing packages in a different prefix while still
> using system packages? I've tried setting EPREFIX, however doing that
> will install all dependencies in the prefix, even if there are already
> installed in the system.
> 
> I was hoping to install some packages in user directories, but I also
> don't want to duplicate the packages installed globally. For example,
> most packages eventually depend on gcc, which I definitely don't want to
> compile twice. So ideally, only dependencies that are not installed
> globally should be pulled in.
> 
> I was not able to find a way of doing that, but I feel like it shouldn't
> be too hard, because EPREFIX almost does what I want. Does someone know
> if it's possible without too much tweaking?

What you're after is known as "Prefix/Stack", where you have some "base"
Prefix, which's portage does manage packages in another - stacked - Prefix.

While this does work already with "Prefix/Guest" as base Prefix, there is no
technical reason to not work with "Prefix/Standalone" or even "Gentoo Linux"
as the base Prefix.  The only problem maybe is that "Prefix/Guest" does use
a portage version with additional patches.

But still, you can get an idea about how this works, using "Prefix/Guest":

 $ docker run -t -i gentooprefix/prefix-guest-64bit-fedora -c /bin/bash

At the docker prompt:

Enter the Guest Prefix preinstalled to /tmp/gentoo:
 $ /tmp/gentoo/startprefix

Set up your stacked Prefix:
 $ prefix-stack-setup --eprefix=$HOME/stack --profile=/tmp/gentoo/etc/portage/make.profile
Note that "~/stack" will not work here, bash refuses to resolve '~' after '='.

Leave the Guest Prefix:
 $ exit

Enter your just created stacked Prefix:
 $ ~/stack/startprefix

Emerge your package, for example:
 $ emerge sys-libs/zlib

Have fun!

HTH,
/haubi/

Re: [gentoo-user] Re: Building packages in different prefix without rebuilding system packages

[François-Xavier Carton]

On Fri, May 15, 2020 at 12:53:16PM +0200, Michael Haubenwallner wrote:
> Hi François-Xavier,
> 
> What you're after is known as "Prefix/Stack", where you have some "base"
> Prefix, which's portage does manage packages in another - stacked - Prefix.
> 
> While this does work already with "Prefix/Guest" as base Prefix, there is no
> technical reason to not work with "Prefix/Standalone" or even "Gentoo Linux"
> as the base Prefix.  The only problem maybe is that "Prefix/Guest" does use
> a portage version with additional patches.
> 
> But still, you can get an idea about how this works, using "Prefix/Guest":
> 
>  $ docker run -t -i gentooprefix/prefix-guest-64bit-fedora -c /bin/bash
> 
> At the docker prompt:
> 
> Enter the Guest Prefix preinstalled to /tmp/gentoo:
>  $ /tmp/gentoo/startprefix
> 
> Set up your stacked Prefix:
>  $ prefix-stack-setup --eprefix=$HOME/stack --profile=/tmp/gentoo/etc/portage/make.profile
> Note that "~/stack" will not work here, bash refuses to resolve '~' after '='.
> 
> Leave the Guest Prefix:
>  $ exit
> 
> Enter your just created stacked Prefix:
>  $ ~/stack/startprefix
> 
> Emerge your package, for example:
>  $ emerge sys-libs/zlib
> 
> Have fun!
> 
> HTH,
> /haubi/
> 

Thanks, this looks great! I'll play with it and see how it works :)