[gentoo-user] Question about emerge sync and where it all goes.

[gentoo-user] Question about emerge sync and where it all goes.

[Dale]

Howdy,

I've mentioned before that I build my packages in a chroot.  I have a OS
copy on a separate drive.  I do this because of the long compile times
of some packages.  On occasion tho, I catch the tree in a bad place. 
Some conflict or other happens and I need to sync again to get fixes
etc.  Given my process tho, I don't want to sync the chroot without also
syncing my main system because the two will not be in sync and I'll lose
my update since it deletes files as well including any updates I did
emerge successfully.  I need the two to be identical.  So, I'd like to
sync my main system and then copy the new data over to the chroot
without copying the rest of the OS.  I did this with the following
command earlier. 

rsync -av --progress --delete  /var/cache/portage/tree/*
/backup/gentoo-build/var/cache/portage/tree/

As you can see, my tree location is not the default.  It is located at
/var/cache/portage/tree/.  It's the same on the chroot obviously.  Would
the above command copy all the needed files/directories over that it
needs after a emerge sync or is there more to it?  The reason I think it
needs more, when I tried to do the usual emerge -aukDN world on my main
system, it wanted to emerge more packages than the chroot did.  I
suspect there is more to it than just the tree directory. 

Anyone have the details on this?  Anyone know what other
files/directories need to be copied over as well?

Thanks.

Dale

:-)  :-) 

Re: [gentoo-user] Question about emerge sync and where it all goes.

[Michael]

[-- Attachment #1: Type: text/plain, Size: 2804 bytes --]

On Monday, 25 March 2024 02:58:21 GMT Dale wrote:
> Howdy,
> 
> I've mentioned before that I build my packages in a chroot.  I have a OS
> copy on a separate drive.  I do this because of the long compile times
> of some packages.  On occasion tho, I catch the tree in a bad place. 
> Some conflict or other happens and I need to sync again to get fixes
> etc.  Given my process tho, I don't want to sync the chroot without also
> syncing my main system because the two will not be in sync and I'll lose
> my update since it deletes files as well including any updates I did
> emerge successfully.  I need the two to be identical.  So, I'd like to
> sync my main system and then copy the new data over to the chroot
> without copying the rest of the OS.

What you do is duplicating effort and storage space with questionable benefit, 
besides warming up your room.  I think it would be more efficient if you used 
the same /var/cache/distfiles and /var/db/repos/ filesystems on both 
installations.  This way you'd sync once and then download any source files 
once only.

However, an even more optimised solution would be to compile packages once in 
your chrooted fs with the '--buildpkg y' option and then emerge these as 
binary packages from /var/cache/binpkgs/ in your production OS.  An overnight 
emerge in the chrooted OS will take only a few minutes on the production 
system.


> I did this with the following command earlier. 
> 
> rsync -av --progress --delete  /var/cache/portage/tree/*
> /backup/gentoo-build/var/cache/portage/tree/

Your syntax for the source tree will fail to copy a couple files immediately 
under /var/cache/portage/tree/ which start with "."

It would be better if you used:

/var/cache/portage/tree /backup/gentoo-build/var/cache/portage/tree


> As you can see, my tree location is not the default.  It is located at
> /var/cache/portage/tree/.  It's the same on the chroot obviously.  Would
> the above command copy all the needed files/directories over that it
> needs after a emerge sync or is there more to it?  The reason I think it
> needs more, when I tried to do the usual emerge -aukDN world on my main
> system, it wanted to emerge more packages than the chroot did.  I
> suspect there is more to it than just the tree directory. 
> 
> Anyone have the details on this?  Anyone know what other
> files/directories need to be copied over as well?
> 
> Thanks.
> 
> Dale
> 
> :-)  :-) 

Did you diff /var/lib/portage/world between the two systems to see what 
differences may be there?

I can think of a hypothetical case where the chrooted fs may have already 
emerged build time dependencies, e.g. if some emerge failed and then you 
fixed/restarted it a number of times, but the production system would not yet 
have such build time dependencies installed.

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] Question about emerge sync and where it all goes.

[Dale]

Michael wrote:
> On Monday, 25 March 2024 02:58:21 GMT Dale wrote:
>> Howdy,
>>
>> I've mentioned before that I build my packages in a chroot.  I have a OS
>> copy on a separate drive.  I do this because of the long compile times
>> of some packages.  On occasion tho, I catch the tree in a bad place. 
>> Some conflict or other happens and I need to sync again to get fixes
>> etc.  Given my process tho, I don't want to sync the chroot without also
>> syncing my main system because the two will not be in sync and I'll lose
>> my update since it deletes files as well including any updates I did
>> emerge successfully.  I need the two to be identical.  So, I'd like to
>> sync my main system and then copy the new data over to the chroot
>> without copying the rest of the OS.
> What you do is duplicating effort and storage space with questionable benefit, 
> besides warming up your room.  I think it would be more efficient if you used 
> the same /var/cache/distfiles and /var/db/repos/ filesystems on both 
> installations.  This way you'd sync once and then download any source files 
> once only.
>
> However, an even more optimised solution would be to compile packages once in 
> your chrooted fs with the '--buildpkg y' option and then emerge these as 
> binary packages from /var/cache/binpkgs/ in your production OS.  An overnight 
> emerge in the chrooted OS will take only a few minutes on the production 
> system.
>

One reason for doing it this way, I have a backup of my OS as well. 
Both are generally identical.  If my main hard drive were to fail, I
could boot rescue media and copy what I usually chroot into over to a
new drive, reinstall grub and such then boot back up.  Should be back up
and running in likely less than a hour, if that.  I took the idea from a
stage4 thing I found.  I just don't compress it since I use it.  After
all, that 750GB hard drive is only used for hourly backups of my email. 
May as well put the rest of the thing to some use. 

The buildpkg option is kinda what happens except I let it also install
and that gives me a backup OS as well.  The main drive with the OS on it
is a bit old.  It is a drive designed for 24/7 use and years of
service.  Still, it could blow smoke or start making noises at any time,
or run for many more years until a OS won't fit on it anymore.  It's
already getting pretty full.  LOL  Someone on this list mailed it to me
years ago with my previous build I think. 

The updates from binary packages only takes a few minutes unless the
number of packages is huge.  Usually a HUGE KDE update and LOo or
something added on top of that can take a hour or so.  Still, much
faster and more stable than trying to compile on my running OS.  Things
get out of sync and some programs start acting weird or won't open due
to a version mismatch.



>> I did this with the following command earlier. 
>>
>> rsync -av --progress --delete  /var/cache/portage/tree/*
>> /backup/gentoo-build/var/cache/portage/tree/
> Your syntax for the source tree will fail to copy a couple files immediately 
> under /var/cache/portage/tree/ which start with "."
>
> It would be better if you used:
>
> /var/cache/portage/tree /backup/gentoo-build/var/cache/portage/tree
>

I edited that in my little script.  The locations with rsync can be
tricky.  Each way that one can use gives a slightly different execution. 


>> As you can see, my tree location is not the default.  It is located at
>> /var/cache/portage/tree/.  It's the same on the chroot obviously.  Would
>> the above command copy all the needed files/directories over that it
>> needs after a emerge sync or is there more to it?  The reason I think it
>> needs more, when I tried to do the usual emerge -aukDN world on my main
>> system, it wanted to emerge more packages than the chroot did.  I
>> suspect there is more to it than just the tree directory. 
>>
>> Anyone have the details on this?  Anyone know what other
>> files/directories need to be copied over as well?
>>
>> Thanks.
>>
>> Dale
>>
>> :-)  :-) 
> Did you diff /var/lib/portage/world between the two systems to see what 
> differences may be there?
>
> I can think of a hypothetical case where the chrooted fs may have already 
> emerged build time dependencies, e.g. if some emerge failed and then you 
> fixed/restarted it a number of times, but the production system would not yet 
> have such build time dependencies installed.


The world file is also copied over each time.  Sometimes I install
something on the running OS and I also want it on the other.  I keep the
two pretty well synced. 

It could be the path change you listed above will change things.  I'll
know the next time I have to do this.  I was even wondering if after
copying the tree over, I needed to run a command to update something to
make sure emerge would see the changes.  I knew it was doing something
different between the two but only did so after I copied the tree over. 
I figured I missed something, be it a file that didn't copy over or a
command I needed to run. 

I'll know when this happens again.  I may even test this when I do the
next update.  I've got most of the kinks worked out on this but things
do pop up.  ;-) 

Thanks much. 

Dale

:-)  :-) 

Re: [gentoo-user] Question about emerge sync and where it all goes.

[Michael]

[-- Attachment #1: Type: text/plain, Size: 1166 bytes --]

On Tuesday, 26 March 2024 00:37:31 GMT Dale wrote:
> Michael wrote:
> > On Monday, 25 March 2024 02:58:21 GMT Dale wrote:

> >> I did this with the following command earlier.
> >> 
> >> rsync -av --progress --delete  /var/cache/portage/tree/*
> >> /backup/gentoo-build/var/cache/portage/tree/
> > 
> > Your syntax for the source tree will fail to copy a couple files
> > immediately under /var/cache/portage/tree/ which start with "."
> > 
> > It would be better if you used:
> > 
> > /var/cache/portage/tree /backup/gentoo-build/var/cache/portage/tree
> 
> I edited that in my little script.  The locations with rsync can be
> tricky.  Each way that one can use gives a slightly different execution.

Oops!  Scratch that - I was wrong.  Sorry!

This is what will work:

/foo/ /backup/foo

will copy the contents of source /foo into destination /backup/foo, without 
copying the top level directory /foo itself.  This way both /foo at source and 
/backup/foo at destination will have identical content.

If you omit a trailing slash you'll copy the source top level directory itself 
and will get:

/backup/foo/foo/...

which NOT what you're after.

Hope this helps.

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]