[gentoo-user] eix-sync data comparison

[gentoo-user] eix-sync data comparison

[meino.cramer]

Hi,

suppose I would do an eix-sync on two different computers of different
platforms -- say AMR and an ordinary PC.

Are the downloaded data identical?

And -- if so -- is it possible to first eix-sync the PC and the
eix-sync the ARM against the PC instead an official Gentoo-server?


Best regards,
mcc

Re: [gentoo-user] eix-sync data comparison

[Alan McKinnon]

On 11/01/2014 22:04, meino.cramer@gmx.de wrote:
> Hi,
> 
> suppose I would do an eix-sync on two different computers of different
> platforms -- say AMR and an ordinary PC.
> 
> Are the downloaded data identical?

the portage tree is identical everywhere

the overlays that layman uses are identical everywhere, but possibly not
identical between your two hosts. I can easily imagine you have
different overlays between them, and no guarantee they will always be
the same.

I consider your question to be fragile and possibly quite dangerous.
What are you trying to accomplish?


-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] eix-sync data comparison

[meino.cramer]

Alan McKinnon <alan.mckinnon@gmail.com> [14-01-11 21:16]:
> On 11/01/2014 22:04, meino.cramer@gmx.de wrote:
> > Hi,
> > 
> > suppose I would do an eix-sync on two different computers of different
> > platforms -- say AMR and an ordinary PC.
> > 
> > Are the downloaded data identical?
> 
> the portage tree is identical everywhere
> 
> the overlays that layman uses are identical everywhere, but possibly not
> identical between your two hosts. I can easily imagine you have
> different overlays between them, and no guarantee they will always be
> the same.
> 
> I consider your question to be fragile and possibly quite dangerous.
> What are you trying to accomplish?
> 
> 
> -- 
> Alan McKinnon
> alan.mckinnon@gmail.com
> 
> 
Hi Alan,

thanks for your reply.

I want to lower the load on the official gentoo server(s).

When the log of an eix-sync process is printed on the console, it is
said, that it is *suggested* (hrrm), to only sinc once a day. Since
the ARM and the PC both use the same DSL line, both syncs are seen by
the gentoo server(s) as it came from the same IP (the one of my ISP).
In the worst case, one could be blacklisted...which I dont want to be.

Especially with the ARM I need to sync once a day, since I have to
limit the amount of software to be recompiled/updated after each sync
since the ARM is not *that* fast (for example a kernel compilation
take ~8h).  Ah, by the way: I quit crosscompilation, distcc and
such...  Both successfully screwed up my gentoos on the ARM.


t regards,
mcc

Re: [gentoo-user] eix-sync data comparison

[Alan McKinnon]

On 11/01/2014 22:44, meino.cramer@gmx.de wrote:
> Alan McKinnon <alan.mckinnon@gmail.com> [14-01-11 21:16]:
>> On 11/01/2014 22:04, meino.cramer@gmx.de wrote:
>>> Hi,
>>>
>>> suppose I would do an eix-sync on two different computers of different
>>> platforms -- say AMR and an ordinary PC.
>>>
>>> Are the downloaded data identical?
>>
>> the portage tree is identical everywhere
>>
>> the overlays that layman uses are identical everywhere, but possibly not
>> identical between your two hosts. I can easily imagine you have
>> different overlays between them, and no guarantee they will always be
>> the same.
>>
>> I consider your question to be fragile and possibly quite dangerous.
>> What are you trying to accomplish?
>>
>>
>> -- 
>> Alan McKinnon
>> alan.mckinnon@gmail.com
>>
>>
> Hi Alan,
> 
> thanks for your reply.
> 
> I want to lower the load on the official gentoo server(s).
> 
> When the log of an eix-sync process is printed on the console, it is
> said, that it is *suggested* (hrrm), to only sinc once a day. Since
> the ARM and the PC both use the same DSL line, both syncs are seen by
> the gentoo server(s) as it came from the same IP (the one of my ISP).
> In the worst case, one could be blacklisted...which I dont want to be.

I wouldn't worry about that at all.

Gentoo infra is vastly more resilient than that, and there are several
official hosts in some sort of cluster arrangement. I have minimally 4
gentoo hosts here at home and many many times I've run --sync
simultaneously on all of them 5-10 times a day, with not a peep out of
infra. I don't do this deliberately, sometimes I forget just how
powerful clusterssh can be :-)

Plus our ftp server at work is a mirror for all of ZA with no special
permissions, it syncs every hour.

The message in --sync is, I believe, a holdover from long long ago that
hasn't been valid for years. But it's left to discourage abuse. What you
are doing is not abuse, it's normal activity. Many of us here have
multiple gentoo installs.

If you want to reduce the data downloaded, then by all means configure
rsyncd to share the tree on the PC, and point the ARM to that. Don't
worry about overlays, the amount of changed data from them is usually
too small to worry about.





> 
> Especially with the ARM I need to sync once a day, since I have to
> limit the amount of software to be recompiled/updated after each sync
> since the ARM is not *that* fast (for example a kernel compilation
> take ~8h).  Ah, by the way: I quit crosscompilation, distcc and
> such...  Both successfully screwed up my gentoos on the ARM.
> 
> 
> t regards,
> mcc
> 
> 
> 
> 
> 


-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] eix-sync data comparison

[meino.cramer]

Alan McKinnon <alan.mckinnon@gmail.com> [14-01-11 22:16]:
> On 11/01/2014 22:44, meino.cramer@gmx.de wrote:
> > Alan McKinnon <alan.mckinnon@gmail.com> [14-01-11 21:16]:
> >> On 11/01/2014 22:04, meino.cramer@gmx.de wrote:
> >>> Hi,
> >>>
> >>> suppose I would do an eix-sync on two different computers of different
> >>> platforms -- say AMR and an ordinary PC.
> >>>
> >>> Are the downloaded data identical?
> >>
> >> the portage tree is identical everywhere
> >>
> >> the overlays that layman uses are identical everywhere, but possibly not
> >> identical between your two hosts. I can easily imagine you have
> >> different overlays between them, and no guarantee they will always be
> >> the same.
> >>
> >> I consider your question to be fragile and possibly quite dangerous.
> >> What are you trying to accomplish?
> >>
> >>
> >> -- 
> >> Alan McKinnon
> >> alan.mckinnon@gmail.com
> >>
> >>
> > Hi Alan,
> > 
> > thanks for your reply.
> > 
> > I want to lower the load on the official gentoo server(s).
> > 
> > When the log of an eix-sync process is printed on the console, it is
> > said, that it is *suggested* (hrrm), to only sinc once a day. Since
> > the ARM and the PC both use the same DSL line, both syncs are seen by
> > the gentoo server(s) as it came from the same IP (the one of my ISP).
> > In the worst case, one could be blacklisted...which I dont want to be.
> 
> I wouldn't worry about that at all.
> 
> Gentoo infra is vastly more resilient than that, and there are several
> official hosts in some sort of cluster arrangement. I have minimally 4
> gentoo hosts here at home and many many times I've run --sync
> simultaneously on all of them 5-10 times a day, with not a peep out of
> infra. I don't do this deliberately, sometimes I forget just how
> powerful clusterssh can be :-)
> 
> Plus our ftp server at work is a mirror for all of ZA with no special
> permissions, it syncs every hour.
> 
> The message in --sync is, I believe, a holdover from long long ago that
> hasn't been valid for years. But it's left to discourage abuse. What you
> are doing is not abuse, it's normal activity. Many of us here have
> multiple gentoo installs.
> 
> If you want to reduce the data downloaded, then by all means configure
> rsyncd to share the tree on the PC, and point the ARM to that. Don't
> worry about overlays, the amount of changed data from them is usually
> too small to worry about.
> 
> 
> 
> 
> 
> > 
> > Especially with the ARM I need to sync once a day, since I have to
> > limit the amount of software to be recompiled/updated after each sync
> > since the ARM is not *that* fast (for example a kernel compilation
> > take ~8h).  Ah, by the way: I quit crosscompilation, distcc and
> > such...  Both successfully screwed up my gentoos on the ARM.
> > 
> > 
> > t regards,
> > mcc
> > 
> > 
> > 
> > 
> > 
> 
> 
> -- 
> Alan McKinnon
> alan.mckinnon@gmail.com
> 

Hi Alan,

thanks a lot for the good news -- Gentoo is such an nice thing and
I didn't want to play the bad guy.

Best regards,
mcc

Re: [gentoo-user] eix-sync data comparison

[Peter Humphrey]

On Saturday 11 Jan 2014 23:14:59 Alan McKinnon wrote:

> If you want to reduce the data downloaded, then by all means configure
> rsyncd to share the tree on the PC, and point the ARM to that. Don't
> worry about overlays, the amount of changed data from them is usually
> too small to worry about.

You might want to look into http-replicator for that, together with rsyncd. I 
find it handy, and of course I have nothing like Alan's load, though it's often 
nice to play at being a good netizen.

-- 
Regards
Peter

[gentoo-user] Re: eix-sync data comparison

[eroen]

[-- Attachment #1: Type: text/plain, Size: 1865 bytes --]

On Sat, 11 Jan 2014 21:44:19 +0100, meino.cramer@gmx.de wrote:
> > On 11/01/2014 22:04, meino.cramer@gmx.de wrote:

> > > suppose I would do an eix-sync on two different computers of
> > > different platforms -- say AMR and an ordinary PC.
Sharing a portage tree (the files in /usr/portage) is perfectly safe,
as this is identical for basic gentoo systems (prefix and similar
are ...different. Best not go there ;-).

However, the eix database is not portable between different systems.
I do not have an exhaustive list of what settings cause
incompatibility, but at least different ARCH (amd64 or arm) will cause
issues since different packages are available to each.

> Especially with the ARM I need to sync once a day, since I have to
> limit the amount of software to be recompiled/updated after each sync
> since the ARM is not *that* fast (for example a kernel compilation
> take ~8h).
While that is not strictly enforced, I myself run an rsync server
locally, mostly to ensure that different systems have identical trees,
thus preserving my sanity.

Instructions for setting up a local rsync server can be found here:
  https://wiki.gentoo.org/wiki/Project:Infrastructure/Rsync
To make a client (the arm box) sync from the local server, just change
the sync-uri variable in /etc/portage/repos.conf/gentoo.conf and the
SYNC variable (if present) in make.conf .

You could even share a portage tree over nfs or similar, which I
sometimes do for VMs. Depending on your (networking and storage)
hardware, that might even work out favourably performance-wise. You
could also generate a squashfs image on the desktop and transfer that to
and mount it on the arm box (over http or something using a short
shell script), if, say, the arm box is limited on storage space. A
squashfs of the gentoo tree is about 70MB.

-- 
eroen

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-user] eix-sync data comparison

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 780 bytes --]

On Sun, 12 Jan 2014 01:15:45 +0000, Peter Humphrey wrote:

> > If you want to reduce the data downloaded, then by all means configure
> > rsyncd to share the tree on the PC, and point the ARM to that. Don't
> > worry about overlays, the amount of changed data from them is usually
> > too small to worry about.  
> 
> You might want to look into http-replicator for that, together with
> rsyncd. I find it handy, and of course I have nothing like Alan's load,
> though it's often nice to play at being a good netizen.

Setting DISTDIR to a network shared location is a good yet simple way of
reducing bandwidth demands.


-- 
Neil Bothwick

In the 60's people took acid to make the world weird.
Now the world is weird and people take Prozac to make it normal.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]