[gentoo-user] Dynamically change PORTAGE_TMPDIR via bashrc?

[gentoo-user] Dynamically change PORTAGE_TMPDIR via bashrc?

[Florian Gamböck]

(This is my third E-Mail and I hope this time it gets through. Sorry to the
sysadmins if I caused you trouble with my rejected mails, I'm still trying to
get used to Mutt.)

Greetings,

let me quickly explain my setup: I use BTRFS snapshots and chroot to create
binaries of upgradeable packages with emerge. When I have some spare time, I
just need to install these binaries on my "real" system. While compiling, I
set PORTAGE_TMPDIR to a tmpfs mounted directory. When installing on my real
system afterwards, I set PORTAGE_TMPDIR to a "normal" directory on my hard
drive, since I won't gain anything from extracting a package to RAM, just to
immediately copy the contents back to the hard drive.

Now here is the tricky part: Since I do the compiling on temporary snapshots,
I tend to kill emerge and delete the snapshot, when I'm in a hurry and have to
shutdown my workstation. (PKGDIR of course is bind mounted to outside the
snapshot, so I won't lose the already finished packages.) When I later
"resume" my upgrade, there are several binary packages and several others that
still need to be compiled. With PORTAGE_TMPDIR still set to tmpfs, my RAM gets
cluttered with already compiled files from the binaries that may as well stay
on the hard disk.

Now, instead of simply changing my workflow (like for example not deleting the
snapshot prematurely in the first place ...) I want to have some fun and
wondered if there might be a possibility to change PORTAGE_TMPDIR dynamically,
based on wether I want to install a tbz2 or compile sources.

So far I've tried in package.env: "*/* merge_type.${MERGE_TYPE}.conf". But
this just gives me errors like "merge_type..conf not found", so obviously I
can't use such variables here. I guess, those tricks can just be played in
/etc/portage/bashrc. But when I try to change PORTAGE_TMPDIR in bashrc
(EBUILD_PHASE doesn't matter, problem occurs regardless), I always get errors
like "PORTAGE_TMPDIR is a read-only variable".

Now, before I try some crazy stunts like bind-mounting $D and $ED on "preinst"
and cleaning up in "postinst", I wanted to know if some of you guys did
similar experiments and/or have some advice that you could share with me.

Regards,
--Flo

[gentoo-user] Re: Dynamically change PORTAGE_TMPDIR via bashrc?

[James]

Florian Gamböck <ml <at> floga.de> writes:


> Now, before I try some crazy stunts like bind-mounting $D and $ED on "preinst"
> and cleaning up in "postinst", I wanted to know if some of you guys did
> similar experiments and/or have some advice that you could share with me.

I have been following 'bcache' as an interesting addition to complex
compiling scenarios. I'm not certain how it will help your 'wild ideas',
but it is worth a look, imho [1] 


hth,
James

[1] http://bcache.evilpiepirate.org/

Re: [gentoo-user] Re: Dynamically change PORTAGE_TMPDIR via bashrc?

[Rich Freeman]

On Sat, Sep 26, 2015 at 10:08 AM, James <wireless@tampabay.rr.com> wrote:
> Florian Gamböck <ml <at> floga.de> writes:
>> Now, before I try some crazy stunts like bind-mounting $D and $ED on "preinst"
>> and cleaning up in "postinst", I wanted to know if some of you guys did
>> similar experiments and/or have some advice that you could share with me.
>
> I have been following 'bcache' as an interesting addition to complex
> compiling scenarios. I'm not certain how it will help your 'wild ideas',
> but it is worth a look, imho [1]

Honestly, I'm hard-pressed to think of situations where you're better
off with anything other than either tmpfs or whatever filesystem your
/usr happens to be on.

However, if you do want to build on something like bcachefs and you're
using an ssd you might just want to use a pure log-based filesystem
like f2fs.  You can afford to go far more experimental for temp space.
I haven't studied the design of bcachefs closely, so I'm not sure how
it would compare for this use.  I know it has made a bunch of noise
lately, though honestly I'm not impressed with some of the hype (not
something I blame the authors for - more the groupies).

One thing that both zfs and bcachefs seem to be doing (not that I'm in
an expert in either) is taking a multi-tiered approach to storage.
That is, writes can go into a durable log on a separate device and
then be consolidated onto more long-term storage.  I suspect that this
might help with some of the fragmentation issues that you run into
with COW.  Then again, I don't know how its performance would compare
to just dumping everything right to a single tier like btrfs and then
just periodically having some kind of smart defragmenter that just
cleans up the stuff that really needs it.

-- 
Rich

[gentoo-user] Re: Dynamically change PORTAGE_TMPDIR via bashrc?

[James]

Rich Freeman <rich0 <at> gentoo.org> writes:


> One thing that both zfs and bcachefs seem to be doing (not that I'm in
> an expert in either) is taking a multi-tiered approach to storage.
> That is, writes can go into a durable log on a separate device and
> then be consolidated onto more long-term storage.  I suspect that this
> might help with some of the fragmentation issues that you run into
> with COW.  Then again, I don't know how its performance would compare
> to just dumping everything right to a single tier like btrfs and then
> just periodically having some kind of smart defragmenter that just
> cleans up the stuff that really needs it.


I have done quite a bit of research on many aspects of memory and storage.
It is probably the single biggest issue with  how best to 'cluster' systems
and resources and the optimal solutions seem to be numerous, depending 
on the goals and trade-offs of a specific application. That said, there are
as many ideas as there are project leads; so the space in full of
experimentation. With RDMA and compiler enhancements in the mix, I do not
think issues will narrow any time soon.

Sorry I could not be of more help; I swim in these waters pretty much
every day, looking for conclusions......still looking......


hth,
James

Re: [gentoo-user] Dynamically change PORTAGE_TMPDIR via bashrc?

[Florian Gamböck]

Hi James,

thank you for your reply.

On Sat, Sep 26, 2015 at 02:08:19PM +0000, James wrote:
> I have been following 'bcache' as an interesting addition to complex
> compiling scenarios. I'm not certain how it will help your 'wild ideas',
> but it is worth a look, imho

Yes, I have also heard of bcachefs, but unless I am terribly mistaken, it
doesn't offer me anything to solve my actual problem, which is "setting
portage environment variables according to MERGE_TYPE".

I am aware that my setup might not be ideal and that approaches like bcachefs
might help me creating a better compile-install-workflow. But for now -- just
for the fun of it -- let's assume that the underlying basis is fixed and all
that is left is finding a way to dynamically change variables like
PORTAGE_TMPDIR.

That could also be used for scenarios like "if the package needs to be
compiled, then use every single machine in the local network as a distcc
server; if there is already a tbz2 file for the given package, then just
install it without searching for potential servers".

Portage's bashrc can be a powerful script for several use-cases, but at the
moment I am quite sad that I can't change environment variables in there.

So, any other ideas?

Regards,
--Flo