[gentoo-dev] Proposal: re-organizing our source mirrors into two categories: full and partial

[gentoo-dev] Proposal: re-organizing our source mirrors into two categories: full and partial

[Kurt Lieber]

[-- Attachment #1: Type: text/plain, Size: 1583 bytes --]

Folks --

I'd like to propose the following reorg for our mirrors.  Right now, all
mirrors are required to carry all directories.  I'd like to break that into
two categories: full mirrors and distfile mirrors.

Full mirrors would still be required to carry all directories.  Full means
full, after all. :)

distfile mirrors would be required to carry two directories: /distfiles and
/snapshots.  /snapshots contains snapshots of the portage tree and is used
by emerge-webrsync for folks that are behind restrictive firewalls.
Typically, the /snapshot directory is under 250MB.

This gives mirrors who are short on disk space an option to still support
Gentoo without having to set aside nearly 50GB of space for us.  For
mirrors who do have enough disk space, they can continue to mirror the full
tree.  

We will of course continue to work at reducing the overall amount of data
that we have.  However, over time, our space requirements will undoubtedly
continue to increase.  This proposal is meant as a longer term solution to
give mirror admins an option on how best they can support Gentoo in their
environment.

Also, for the many people who suggested a p2p network as an option, that is
certainly something we will consider in the future.  That will take
planning and coordination, however, as it will likely require changes to
portage.  In the mean time, our system of FTP and HTTP mirrors will
continue to be our primary method of distribution and we still needed a
solution to address these mirrors.

Your comments and ideas are welcome.  

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Proposal: re-organizing our source mirrors into two categories: full and partial

[bdharring]

On Friday, July 25, 2003, at 09:11 AM, Kurt Lieber wrote:

> Folks --
>
> I'd like to propose the following reorg for our mirrors.  Right now, 
> all
> mirrors are required to carry all directories.  I'd like to break that 
> into
> two categories: full mirrors and distfile mirrors.
>
> Full mirrors would still be required to carry all directories.  Full 
> means
> full, after all. :)
>
> distfile mirrors would be required to carry two directories: 
> /distfiles and
> /snapshots.  /snapshots contains snapshots of the portage tree and is 
> used
> by emerge-webrsync for folks that are behind restrictive firewalls.
> Typically, the /snapshot directory is under 250MB.
>
> This gives mirrors who are short on disk space an option to still 
> support
> Gentoo without having to set aside nearly 50GB of space for us.  For
> mirrors who do have enough disk space, they can continue to mirror the 
> full
> tree.
>
> We will of course continue to work at reducing the overall amount of 
> data
> that we have.  However, over time, our space requirements will 
> undoubtedly
> continue to increase.  This proposal is meant as a longer term 
> solution to
> give mirror admins an option on how best they can support Gentoo in 
> their
> environment.
Going a bit longer term, I'd wonder about adding a mirror 
classification that handles just diff's (or whatever format is used).
While distfile diff's aren't currently included/existing, I can't help 
but believe that this feature will be integrated into portage at some 
point, mainly due to the fact people request it and more importantly, 
the fact that people are working on this problem.
Also, are you advocating the distfile mirrors carry a *complete* 
distfile mirror, or a partial?  I'm curious what the space savings 
would be...
~bdh


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Proposal: re-organizing our source mirrors into two categories: full and partial

[Kurt Lieber]

[-- Attachment #1: Type: text/plain, Size: 318 bytes --]

On Fri, Jul 25, 2003 at 09:33:07AM -0500 or thereabouts, bdharring wrote:
> Also, are you advocating the distfile mirrors carry a *complete* 
> distfile mirror, or a partial?  I'm curious what the space savings 
> would be...

Distfile mirrors would be required to carry the full /distfiles directory.

--kurt

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Proposal: re-organizing our source mirrors into two categories: full and partial

[Stuart Herbert]

[-- Attachment #1: signed data --]
[-- Type: text/plain, Size: 722 bytes --]

On Saturday 26 July 2003 1:37 pm, Kurt Lieber wrote:
> Distfile mirrors would be required to carry the full /distfiles directory.
>
> --kurt

Just a thought, as I'm still new around here, but is there any automatic 
auditing to make sure that every single file in /distfiles is still required 
by at least 1 ebuild?

Best regards,
Stu
-- 
Stuart Herbert                                              stuart@gentoo.org
Gentoo Developer                                       http://www.gentoo.org/
Upcoming packages list                http://dev.gentoo.org/~stuart/packages/

GnuGP key id# F9AFC57C available from http://pgp.mit.edu
Key fingerprint = 31FB 50D4 1F88 E227 F319  C549 0C2F 80BA F9AF C57C
--

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Proposal: re-organizing our source mirrors into two categories: full and partial

[Alec Berryman]

[-- Attachment #1: Type: text/plain, Size: 597 bytes --]

On Sat, 2003-07-26 at 07:40, Stuart Herbert wrote:
> Just a thought, as I'm still new around here, but is there any automatic 
> auditing to make sure that every single file in /distfiles is still required 
> by at least 1 ebuild?

This bash one-liner was posted to the forums; I can't find it now, so
apologies to the original author:

bash <(comm -23 <(find /usr/portage/distfiles/ -maxdepth 1 -type f -printf %f\\n | sort) <(find /usr/portage/ /usr/local/portage/ -path '*/*/files/digest-*' -print0 | xargs -0 awk '{print $3}' | sort -u) | sed -e 's|^|rm -i /usr/portage/distfiles/|')

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Proposal: re-organizing our source mirrors into two categories: full and partial

[Jon Portnoy]

On Sat, Jul 26, 2003 at 01:40:29PM +0100, Stuart Herbert wrote:
Content-Description: signed data
> On Saturday 26 July 2003 1:37 pm, Kurt Lieber wrote:
> > Distfile mirrors would be required to carry the full /distfiles directory.
> >
> > --kurt
> 
> Just a thought, as I'm still new around here, but is there any automatic 
> auditing to make sure that every single file in /distfiles is still required 
> by at least 1 ebuild?
> 

Yes.

-- 
Jon Portnoy
avenj/irc.freenode.net

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Proposal: re-organizing our source mirrors into two categories: full and partial

[paul]

bdharring wrote:
> 
> On Friday, July 25, 2003, at 09:11 AM, Kurt Lieber wrote:
> 
>> Folks --
>>
>> I'd like to propose the following reorg for our mirrors.  Right now, all
>> mirrors are required to carry all directories.  I'd like to break that 
>> into
>> two categories: full mirrors and distfile mirrors.
>>
>> Full mirrors would still be required to carry all directories.  Full 
>> means
>> full, after all. :)
>>
>> distfile mirrors would be required to carry two directories: 
>> /distfiles and
>> /snapshots.  /snapshots contains snapshots of the portage tree and is 
>> used
>> by emerge-webrsync for folks that are behind restrictive firewalls.
>> Typically, the /snapshot directory is under 250MB.
>>
>> This gives mirrors who are short on disk space an option to still support
>> Gentoo without having to set aside nearly 50GB of space for us.  For
>> mirrors who do have enough disk space, they can continue to mirror the 
>> full
>> tree.
>>
>> We will of course continue to work at reducing the overall amount of data
>> that we have.  However, over time, our space requirements will 
>> undoubtedly
>> continue to increase.  This proposal is meant as a longer term 
>> solution to
>> give mirror admins an option on how best they can support Gentoo in their
>> environment.
> 
> Going a bit longer term, I'd wonder about adding a mirror classification 
> that handles just diff's (or whatever format is used).
he he, opened my MUA to ask about that this morning ;) I'd love to see 
portage handle updates as patches to distfiles. Is this feature planned 
in the near future?

paul

> While distfile diff's aren't currently included/existing, I can't help 
> but believe that this feature will be integrated into portage at some 
> point, mainly due to the fact people request it and more importantly, 
> the fact that people are working on this problem.
> Also, are you advocating the distfile mirrors carry a *complete* 
> distfile mirror, or a partial?  I'm curious what the space savings would 
> be...
> ~bdh
> 
> 
> -- 
> gentoo-dev@gentoo.org mailing list
> 



--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Proposal: re-organizing our source mirrors into two categories: full and partial

[Chris Gianelloni]

I'm no bash expert, but does anyone know what one would need to do to
make this into a script?

On Sat, 2003-07-26 at 09:42, Alec Berryman wrote:
> This bash one-liner was posted to the forums; I can't find it now, so
> apologies to the original author:
> 
> bash <(comm -23 <(find /usr/portage/distfiles/ -maxdepth 1 -type f -printf %f\\n | sort) <(find /usr/portage/ /usr/local/portage/ -path '*/*/files/digest-*' -print0 | xargs -0 awk '{print $3}' | sort -u) | sed -e 's|^|rm -i /usr/portage/distfiles/|')
-- 
Chris Gianelloni
Developer, Gentoo Linux


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Proposal: re-organizing our source mirrors into two categories: full and partial

[Chris Gianelloni]

Never mind that request.  I figured it out.  It must have #!/bin/bash
not #!/bin/sh at the header.

-- 
Chris Gianelloni
Developer, Gentoo Linux


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Proposal: re-organizing our source mirrors into two categories: full and partial

[bdharring]

On Saturday, July 26, 2003, at 07:37 AM, Kurt Lieber wrote:

> On Fri, Jul 25, 2003 at 09:33:07AM -0500 or thereabouts, bdharring 
> wrote:
>> Also, are you advocating the distfile mirrors carry a *complete*
>> distfile mirror, or a partial?  I'm curious what the space savings
>> would be...
>
> Distfile mirrors would be required to carry the full /distfiles 
> directory.
What space savings are avoided by not carrying the portage tree?  Last 
I had checked, I believe the tree was roughly in the range of 100mbs or 
so.
I guess I'm curious if I'm missing something, since 100 mb's (if I'm 
correct) while is a space saving, it really isn't all that much in 
comparison to the distfile dir....
~bdh
>
> --kurt
> <mime-attachment>


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Proposal: re-organizing our source mirrors into two categories: full and partial

[Kurt Lieber]

[-- Attachment #1: Type: text/plain, Size: 978 bytes --]

On Sun, Jul 27, 2003 at 05:05:15AM -0500 or thereabouts, bdharring wrote:
> >Distfile mirrors would be required to carry the full /distfiles 
> >directory.
> What space savings are avoided by not carrying the portage tree?  Last 
> I had checked, I believe the tree was roughly in the range of 100mbs or 
> so.

The portage tree is carried on a separate system of mirrors.  We call them
our portage mirrors or rsync mirrors.  Source mirrors carry the tarballs
and ISOs necessary to build things found in the Portage tree, as well as
installation CDs, GRP packages, etc.

This discussion only revolves around source mirrors, which current hold the
following directories:

/releases
/experimental
/snapshots
/distfiles

Under the current proposal, "full" mirrors would carry all four directories
while distfile mirrors would only carry the last two directories.  The
delta between the two in terms of hard drive space required is (roughly)
15GB.

--kurt

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]