[gentoo-user] Is distfile partial mirror with failover possible?

[gentoo-user] Is distfile partial mirror with failover possible?

[Walter Dnes]

  I may be misunderstanding, but it seems to me that local mirrors are
all-or-nothing.  In the interests of saving bandwidth, I'd like to have
a client first check the host's /var/cache/distfiles directory for a
source tarball file.  If not found, then fail over to another mirror as
per GENTOO_MIRRORS in /etc/portage/make.conf.  This would require emerge
doing the lookup and potential failover for each file.  Is this possible?

-- 
I've seen things, you people wouldn't believe; Gopher, Netscape with
frames, the first Browser Wars.  Searching for pages with AltaVista,
pop-up windows self-replicating, trying to uninstall RealPlayer.  All
those moments, will be lost in time like tears in rain... time to die.

Re: [gentoo-user] Is distfile partial mirror with failover possible?

[Michael]

[-- Attachment #1: Type: text/plain, Size: 1304 bytes --]

On Monday, 4 September 2023 11:12:51 BST Walter Dnes wrote:
>   I may be misunderstanding, but it seems to me that local mirrors are
> all-or-nothing.  In the interests of saving bandwidth, I'd like to have
> a client first check the host's /var/cache/distfiles directory for a
> source tarball file.  If not found, then fail over to another mirror as
> per GENTOO_MIRRORS in /etc/portage/make.conf.  This would require emerge
> doing the lookup and potential failover for each file.  Is this possible?

Unless I misunderstand what you're asking, isn't this what takes place anyway?

Emerge will look at /var/cache/distfiles and if the source tarball is missing, 
will try to fetch it from the list of GENTOO_MIRRORS you have configured.  For 
example, if you have:

GENTOO_MIRRORS="http://192.168.2.35/my_gentoo_distfiles/ https://
www.mirrorservice.org/sites/distfiles.gentoo.org/"

any host configured as above will first look at its own /var/cache/distfiles, 
then at the first configured mirror which happens to be in the same LAN, then 
whatever other external mirrors you have added.  You could of course export 
your local mirror's LAN ${DISTDIR} via NFS for your hosts to use.

PS. Is http_replicator still available/maintained?  I see the wiki mentions 
apt-cacher-ng for local distfiles cache.

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] Is distfile partial mirror with failover possible?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 1066 bytes --]

On Mon, 04 Sep 2023 14:04:53 +0100, Michael wrote:

> On Monday, 4 September 2023 11:12:51 BST Walter Dnes wrote:
> >   I may be misunderstanding, but it seems to me that local mirrors are
> > all-or-nothing.  In the interests of saving bandwidth, I'd like to
> > have a client first check the host's /var/cache/distfiles directory
> > for a source tarball file.  If not found, then fail over to another
> > mirror as per GENTOO_MIRRORS in /etc/portage/make.conf.  This would
> > require emerge doing the lookup and potential failover for each file.
> >  Is this possible?  
> 
> Unless I misunderstand what you're asking, isn't this what takes place
> anyway?

The reference to a host makes me think Walter wants one machne to hold
the distfiles for all on the network.

> PS. Is http_replicator still available/maintained?  I see the wiki
> mentions apt-cacher-ng for local distfiles cache.

I use apt-cacher-ng for this and it does what you are looking for.


-- 
Neil Bothwick

The facts, although interesting, are usually irrelevant.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] Is distfile partial mirror with failover possible?

[Alan McKinnon]

[-- Attachment #1: Type: text/plain, Size: 1389 bytes --]

Quick n dirty solution:

put all distfiles on a central server
FS mount that remote dir to /var/cache/distfiles on all hosts

Alan

On Mon, Sep 4, 2023 at 7:21 PM Neil Bothwick <neil@digimed.co.uk> wrote:

> On Mon, 04 Sep 2023 14:04:53 +0100, Michael wrote:
>
> > On Monday, 4 September 2023 11:12:51 BST Walter Dnes wrote:
> > >   I may be misunderstanding, but it seems to me that local mirrors are
> > > all-or-nothing.  In the interests of saving bandwidth, I'd like to
> > > have a client first check the host's /var/cache/distfiles directory
> > > for a source tarball file.  If not found, then fail over to another
> > > mirror as per GENTOO_MIRRORS in /etc/portage/make.conf.  This would
> > > require emerge doing the lookup and potential failover for each file.
> > >  Is this possible?
> >
> > Unless I misunderstand what you're asking, isn't this what takes place
> > anyway?
>
> The reference to a host makes me think Walter wants one machne to hold
> the distfiles for all on the network.
>
> > PS. Is http_replicator still available/maintained?  I see the wiki
> > mentions apt-cacher-ng for local distfiles cache.
>
> I use apt-cacher-ng for this and it does what you are looking for.
>
>
> --
> Neil Bothwick
>
> The facts, although interesting, are usually irrelevant.
>


-- 
Alan McKinnon
alan dot mckinnon at gmail dot com

[-- Attachment #2: Type: text/html, Size: 1984 bytes --]

Re: [gentoo-user] Is distfile partial mirror with failover possible?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 1761 bytes --]

On Mon, 4 Sep 2023 19:49:56 +0200, Alan McKinnon wrote:

> Quick n dirty solution:
> 
> put all distfiles on a central server
> FS mount that remote dir to /var/cache/distfiles on all hosts

That's what I used to do, but had problems when downloading the same file
from to clients at once.

BTW Welcome back Alan, but leave your dirty top-posting in Archland :P

> 
> Alan
> 
> On Mon, Sep 4, 2023 at 7:21 PM Neil Bothwick <neil@digimed.co.uk> wrote:
> 
> > On Mon, 04 Sep 2023 14:04:53 +0100, Michael wrote:
> >  
> > > On Monday, 4 September 2023 11:12:51 BST Walter Dnes wrote:  
> > > >   I may be misunderstanding, but it seems to me that local
> > > > mirrors are all-or-nothing.  In the interests of saving
> > > > bandwidth, I'd like to have a client first check the host's
> > > > /var/cache/distfiles directory for a source tarball file.  If not
> > > > found, then fail over to another mirror as per GENTOO_MIRRORS in
> > > > /etc/portage/make.conf.  This would require emerge doing the
> > > > lookup and potential failover for each file. Is this possible?  
> > >
> > > Unless I misunderstand what you're asking, isn't this what takes
> > > place anyway?  
> >
> > The reference to a host makes me think Walter wants one machne to hold
> > the distfiles for all on the network.
> >  
> > > PS. Is http_replicator still available/maintained?  I see the wiki
> > > mentions apt-cacher-ng for local distfiles cache.  
> >
> > I use apt-cacher-ng for this and it does what you are looking for.
> >
> >
> > --
> > Neil Bothwick
> >
> > The facts, although interesting, are usually irrelevant.
> >  
> 
> 




-- 
Neil Bothwick

Not one shred of evidence supports the notion that life is serious.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] Is distfile partial mirror with failover possible?

[Alan McKinnon]

[-- Attachment #1: Type: text/plain, Size: 576 bytes --]

On Mon, Sep 4, 2023 at 8:26 PM Neil Bothwick <neil@digimed.co.uk> wrote:

> On Mon, 4 Sep 2023 19:49:56 +0200, Alan McKinnon wrote:
>
> > Quick n dirty solution:
> >
> > put all distfiles on a central server
> > FS mount that remote dir to /var/cache/distfiles on all hosts
>
> That's what I used to do, but had problems when downloading the same file
> from to clients at once.
>
> BTW Welcome back Alan, but leave your dirty top-posting in Archland :P
>
>
Eh, I use gmail in the browser ..... the blerry thing is built to top post,
like Outlook


Alan

[-- Attachment #2: Type: text/html, Size: 977 bytes --]

Re: [gentoo-user] Is distfile partial mirror with failover possible?

[Mark Knecht]

[-- Attachment #1: Type: text/plain, Size: 1034 bytes --]

On Mon, Sep 4, 2023 at 12:38 PM Alan McKinnon <alan.mckinnon@gmail.com>
wrote:
>
>
>
> On Mon, Sep 4, 2023 at 8:26 PM Neil Bothwick <neil@digimed.co.uk> wrote:
>>
>> On Mon, 4 Sep 2023 19:49:56 +0200, Alan McKinnon wrote:
>>
>> > Quick n dirty solution:
>> >
>> > put all distfiles on a central server
>> > FS mount that remote dir to /var/cache/distfiles on all hosts
>>
>> That's what I used to do, but had problems when downloading the same file
>> from to clients at once.
>>
>> BTW Welcome back Alan, but leave your dirty top-posting in Archland :P
>>
>
> Eh, I use gmail in the browser ..... the blerry thing is built to top
post, like Outlook
>
>
> Alan

>
I'm a GMail user also. Sadly you'll want to not only bottom post, but also
select all text you're responding to, remove formatting (Ctrl-V) and then
type your response or you'll be down voted for responding in HTML.

I hate it also, but this list is easily one of my favorites and I'm no
longer
a Gentoo user.

Best wishes,
Mark

[-- Attachment #2: Type: text/html, Size: 1611 bytes --]

Re: [gentoo-user] Is distfile partial mirror with failover possible?

[Arve Barsnes]

On Mon, 4 Sept 2023 at 22:02, Mark Knecht <markknecht@gmail.com> wrote:
> I'm a GMail user also. Sadly you'll want to not only bottom post, but also
> select all text you're responding to, remove formatting (Ctrl-V) and then
> type your response or you'll be down voted for responding in HTML.
>
> I hate it also, but this list is easily one of my favorites and I'm no longer
> a Gentoo user.

I also use both GMail and Outlook, and you can set up GMail to be
plain text by default. It's not too difficult to click the three dots
and type your reply where it belongs. Not so easy in Outlook which is
actively hating its users.

Regards,
Arve

RE: [gentoo-user] Is distfile partial mirror with failover possible?

[Laurence Perkins]

[-- Attachment #1: Type: text/plain, Size: 677 bytes --]

>
>
> From: Alan McKinnon alan.mckinnon@gmail.com<mailto:alan.mckinnon@gmail.com>
> Sent: Monday, September 4, 2023 12:38 PM
> To: gentoo-user@lists.gentoo.org<mailto:gentoo-user@lists.gentoo.org>
> Subject: Re: [gentoo-user] Is distfile partial mirror with failover possible?
>
> BTW Welcome back Alan, but leave your dirty top-posting in Archland :P
>
> Eh, I use gmail in the browser ..... the blerry thing is built to top post, like Outlook
>
>
> Alan
>

I'm in the same boat, but Notepad++ on Windows and Vim on Linux both support column-mode editing, so a couple quick cut-n-pastes and I can preserve the illusion that I'm using a sensible client.

LMP

[-- Attachment #2: Type: text/html, Size: 2955 bytes --]

Re: [gentoo-user] Is distfile partial mirror with failover possible?

[Walter Dnes]

On Mon, Sep 04, 2023 at 06:21:27PM +0100, Neil Bothwick wrote
> On Mon, 04 Sep 2023 14:04:53 +0100, Michael wrote:
> 
> The reference to a host makes me think Walter wants one machne to hold
> the distfiles for all on the network.

  Correct.

> I use apt-cacher-ng for this and it does what you are looking for.

  https://wiki.gentoo.org/wiki/Local_distfiles_cache#Open_issues says...

> Open issues
>
> *  Apt-cacher-ng installs a cron job to delete unreferenced files
> from the cache. Since the Gentoo cache contains no index files,
> this probably deletes either everything or nothing from it.

  Have you run into ptoblems with this?

  It looks like remote-mounting /var/cache/distfiles might be the
quick-n-dirty solution like Alan suggested.  And I never have a need to
have emerge run simultaneously on more than one machine.

-- 
I've seen things, you people wouldn't believe; Gopher, Netscape with
frames, the first Browser Wars.  Searching for pages with AltaVista,
pop-up windows self-replicating, trying to uninstall RealPlayer.  All
those moments, will be lost in time like tears in rain... time to die.

Re: [gentoo-user] Is distfile partial mirror with failover possible?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 939 bytes --]

On Mon, 4 Sep 2023 22:54:38 -0400, Walter Dnes wrote:

> On Mon, Sep 04, 2023 at 06:21:27PM +0100, Neil Bothwick wrote

> > I use apt-cacher-ng for this and it does what you are looking for.  
> 
>   https://wiki.gentoo.org/wiki/Local_distfiles_cache#Open_issues says...
> 
> > Open issues
> >
> > *  Apt-cacher-ng installs a cron job to delete unreferenced files
> > from the cache. Since the Gentoo cache contains no index files,
> > this probably deletes either everything or nothing from it.  
> 
>   Have you run into ptoblems with this?

Not that I've noticed.

>   It looks like remote-mounting /var/cache/distfiles might be the
> quick-n-dirty solution like Alan suggested.  And I never have a need to
> have emerge run simultaneously on more than one machine.

It seems the simplest suitable solution for your needs.


-- 
Neil Bothwick

A pessimist complains about the noise when opportunity knocks.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] Is distfile partial mirror with failover possible?

[Walter Dnes]

On Tue, Sep 05, 2023 at 08:04:19AM +0100, Neil Bothwick wrote
> On Mon, 4 Sep 2023 22:54:38 -0400, Walter Dnes wrote:
> 
> >   It looks like remote-mounting /var/cache/distfiles might be the
> > quick-n-dirty solution like Alan suggested.  And I never have a need to
> > have emerge run simultaneously on more than one machine.
> 
> It seems the simplest suitable solution for your needs.

  And while we're at it, howsabout "emerge --sync" on one host, and then
remote-mounting /var/db/repos/gentoo ?

-- 
I've seen things, you people wouldn't believe; Gopher, Netscape with
frames, the first Browser Wars.  Searching for pages with AltaVista,
pop-up windows self-replicating, trying to uninstall RealPlayer.  All
those moments, will be lost in time like tears in rain... time to die.

Re: [gentoo-user] Is distfile partial mirror with failover possible?

[Michael]

[-- Attachment #1: Type: text/plain, Size: 652 bytes --]

On Tuesday, 5 September 2023 15:58:05 BST Walter Dnes wrote:
> On Tue, Sep 05, 2023 at 08:04:19AM +0100, Neil Bothwick wrote
> 
> > On Mon, 4 Sep 2023 22:54:38 -0400, Walter Dnes wrote:
> > >   It looks like remote-mounting /var/cache/distfiles might be the
> > > 
> > > quick-n-dirty solution like Alan suggested.  And I never have a need to
> > > have emerge run simultaneously on more than one machine.
> > 
> > It seems the simplest suitable solution for your needs.
> 
>   And while we're at it, howsabout "emerge --sync" on one host, and then
> remote-mounting /var/db/repos/gentoo ?

Take a look here:

https://wiki.gentoo.org/wiki/Local_Mirror

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] Is distfile partial mirror with failover possible?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 652 bytes --]

On Tue, 05 Sep 2023 16:07:16 +0100, Michael wrote:

> >   And while we're at it, howsabout "emerge --sync" on one host, and
> > then remote-mounting /var/db/repos/gentoo ?  
> 
> Take a look here:
> 
> https://wiki.gentoo.org/wiki/Local_Mirror

I used to do that, the server syncing with Gentoo then everything else
syncing with that. However, when I switched to git syncing, it was so
fast that there was no point in running a local mirror - everything syncs
directly from Github now.


-- 
Neil Bothwick

BBS: (n.) a system for connecting computers and exchanging gossip,
     facts, and uninformed speculation under false names.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] Is distfile partial mirror with failover possible?

[Walter Dnes]

On Tue, Sep 05, 2023 at 04:07:16PM +0100, Michael wrote
> On Tuesday, 5 September 2023 15:58:05 BST Walter Dnes wrote:
> > 
> >   And while we're at it, howsabout "emerge --sync" on one host, and then
> > remote-mounting /var/db/repos/gentoo ?
> 
> Take a look here:
> 
> https://wiki.gentoo.org/wiki/Local_Mirror

  That's what I do right now.  I figure that remote-mounting might make
things easier.

-- 
I've seen things, you people wouldn't believe; Gopher, Netscape with
frames, the first Browser Wars.  Searching for pages with AltaVista,
pop-up windows self-replicating, trying to uninstall RealPlayer.  All
those moments, will be lost in time like tears in rain... time to die.

Re: [gentoo-user] Is distfile partial mirror with failover possible?

[Michael]

[-- Attachment #1: Type: text/plain, Size: 664 bytes --]

On Tuesday, 5 September 2023 16:33:43 BST Walter Dnes wrote:
> On Tue, Sep 05, 2023 at 04:07:16PM +0100, Michael wrote
> 
> > On Tuesday, 5 September 2023 15:58:05 BST Walter Dnes wrote:
> > >   And while we're at it, howsabout "emerge --sync" on one host, and then
> > > 
> > > remote-mounting /var/db/repos/gentoo ?
> > 
> > Take a look here:
> > 
> > https://wiki.gentoo.org/wiki/Local_Mirror
> 
>   That's what I do right now.  I figure that remote-mounting might make
> things easier.

There's more work required to set it up than just rsyncd and NFS requires you 
run the local mirror all the time if the clients expect it to be up & running 
when they boot.

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] Is distfile partial mirror with failover possible?

[William Kenworthy]

On 5/9/23 22:58, Walter Dnes wrote:
> On Tue, Sep 05, 2023 at 08:04:19AM +0100, Neil Bothwick wrote
>> On Mon, 4 Sep 2023 22:54:38 -0400, Walter Dnes wrote:
>>
>>>    It looks like remote-mounting /var/cache/distfiles might be the
>>> quick-n-dirty solution like Alan suggested.  And I never have a need to
>>> have emerge run simultaneously on more than one machine.
>> It seems the simplest suitable solution for your needs.
>    And while we're at it, howsabout "emerge --sync" on one host, and then
> remote-mounting /var/db/repos/gentoo ?
>
Similarly - I was mounting a single location on moosefs (network 
filesystem) for ~15 hosts.  One host updates using --sync.  One 
buildhost for each architecture building packages.  Distfiles are 
handled the same way. Then "emerge -k" on the remainder checking for 
packages that needed an actual build.  If none, parallel merge (via 
ansible) or else single builds.  Resulting packages are also stored 
remotely per arch (various amd64, arm32, arm64 etc.)

Working well for a few years, but I am in the process of converting to 
nfs with a similar structure as I have decommissioned the moosefs system 
and many machines that are no longer needed.  A single machine will 
handle built packages, distfiles and repos which are mounted on the 
other systems via nfs. I will look into apt-cacher-ng as I was a fan of 
http-replicator until it was deprecated.

BillK

Re: [gentoo-user] Is distfile partial mirror with failover possible?

[hitachi303]

Am 04.09.23 um 15:04 schrieb Michael:
> PS. Is http_replicator still available/maintained?  I see the wiki mentions
> apt-cacher-ng for local distfiles cache.

IIRC it died when python or perl 2.x was no longer available.

Re: [gentoo-user] Is distfile partial mirror with failover possible?

[Michael]

[-- Attachment #1: Type: text/plain, Size: 1406 bytes --]

On Tuesday, 5 September 2023 10:06:42 BST hitachi303 wrote:
> Am 04.09.23 um 15:04 schrieb Michael:
> > PS. Is http_replicator still available/maintained?  I see the wiki
> > mentions
> > apt-cacher-ng for local distfiles cache.
> 
> IIRC it died when python or perl 2.x was no longer available.

It's worth mentioning another option, not involving a full caching proxy as 
such.  It may suit Walter's workflow so I mention it here for completeness.  A 
lightweight browser can serve files directly from the local mirror's own /var/
cache/distfiles.

Unlike a caching proxy, files not already downloaded/needed by the local 
mirror won't be available locally, so the clients will download these from 
external mirrors.  The local server will need to be updated first, to fetch 
all the latest source file versions.  If there is a large difference in 
package requirements between different clients and the server, then only a 
partial bandwidth efficiency will be achieved this way.  Also if all the 
clients require the same packages not available on the server, then these will 
be downloaded by each client separately from external mirrors, further eroding 
any savings in overall bandwidth.

Setting up a small footprint browser like gatling, or just a couple of lines 
of python http.server, will allow any light on resources gentoo machine to 
become a useful local mirror with minimal configuration.

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]