[gentoo-dev] Proposal: patches.gentoo.org

[gentoo-dev] Proposal: patches.gentoo.org

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 2091 bytes --]

We were talking about this on IRC due to the growing concern over
patches and other files in ${FILESDIR} in the portage tree.  This is
basically the idea that we came up with to resolve the problem.

The proposal is to get infrastructure to provide a temporary location
for developers to upload patches which are necessary to be immediately
available to users.  This space would be writable by all CVS developers
and would be accessible to the world via http.  Developers would use
this space for temporary storage only for patches which are
created/rolled in-house.

For the sake of simplicity, I am going to call this server (though it
could be any number of servers) patches.gentoo.org for the remainder of
this document.

Ramereth suggesting using the existing rsync.gentoo.org infrastructure
servers for this task.  This would simplify replication, since the
replication could be done via rsync and use the same schedule as the
master portage tree replication.  This would ensure that the patches are
available at the same time as the ebuild.

The developer would use this temporary location as follows:

1. developer creates an ebuild which needs a patch
2. developer puts SRC_URI="patches.gentoo.org/${P}-$patchname" in the
ebuild, at the end of the SRC_URI definition
3. developer uploads patch to dev.gentoo.org:/space/patches-local and
dev.gentoo.org:/space/distfiles-local
4. developer commits ebuild(s) to cvs

The patch would replicate to the rsync boxes and would be available via
http immediately.

Remember that portage tries the Gentoo mirrors in ${GENTOO_MIRRORS}
before checking SRC_URI, so once the files hit the community mirrors,
the temporary location in SRC_URI would no longer be used.

A cron job would be added to the machine to clean out files older than 7
days.  This gives the mirrors plenty of time to locally mirror the
patches.  This also reduces the amount of space required on the machine
(s) hosting the files.

-- 
Chris Gianelloni
Release Engineering - Operations/QA Manager
Games - Developer
Gentoo Linux

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Proposal: patches.gentoo.org

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 385 bytes --]

On Tue, 12 Oct 2004 15:09:45 -0400 Chris Gianelloni
<wolf31o2@gentoo.org> wrote:
| 2. developer puts SRC_URI="patches.gentoo.org/${P}-$patchname" in the
| ebuild, at the end of the SRC_URI definition

mirror://patches/ please.

-- 
Ciaran McCreesh : Gentoo Developer (Vim, Fluxbox, Sparc, Mips)
Mail            : ciaranm at gentoo.org
Web             : http://dev.gentoo.org/~ciaranm


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Proposal: patches.gentoo.org

[George Shapovalov]

On Tuesday 12 October 2004 12:12, Ciaran McCreesh wrote:
> On Tue, 12 Oct 2004 15:09:45 -0400 Chris Gianelloni
> <wolf31o2@gentoo.org> wrote:
> | 2. developer puts SRC_URI="patches.gentoo.org/${P}-$patchname" in the
> | ebuild, at the end of the SRC_URI definition
>
> mirror://patches/ please.

Um, will this fly without having to modify ebuild 2nd time, as patches are 
getting cleaned off this server as suggested? :

>A cron job would be added to the machine to clean out files older than 7
>days.  This gives the mirrors plenty of time to locally mirror the
>patches.  This also reduces the amount of space required on the machine
>(s) hosting the files.

George


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Proposal: patches.gentoo.org

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 741 bytes --]

On Tue, 12 Oct 2004 12:32:39 -0700 George Shapovalov <george@gentoo.org>
wrote:

| On Tuesday 12 October 2004 12:12, Ciaran McCreesh wrote:
| > On Tue, 12 Oct 2004 15:09:45 -0400 Chris Gianelloni
| > <wolf31o2@gentoo.org> wrote:
| > | 2. developer puts SRC_URI="patches.gentoo.org/${P}-$patchname" in
| > | the ebuild, at the end of the SRC_URI definition
| >
| > mirror://patches/ please.
| 
| Um, will this fly without having to modify ebuild 2nd time, as patches
| are getting cleaned off this server as suggested? :

If it's the last item in SRC_URI it shouldn't be a problem...

-- 
Ciaran McCreesh : Gentoo Developer (Vim, Fluxbox, Sparc, Mips)
Mail            : ciaranm at gentoo.org
Web             : http://dev.gentoo.org/~ciaranm


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Proposal: patches.gentoo.org

[Kurt Lieber]

[-- Attachment #1: Type: text/plain, Size: 861 bytes --]

On Tue, Oct 12, 2004 at 08:12:58PM +0100 or thereabouts, Ciaran McCreesh wrote:
> On Tue, 12 Oct 2004 15:09:45 -0400 Chris Gianelloni
> <wolf31o2@gentoo.org> wrote:
> | 2. developer puts SRC_URI="patches.gentoo.org/${P}-$patchname" in the
> | ebuild, at the end of the SRC_URI definition
> 
> mirror://patches/ please.

If by this you mean that patches should then exist in thirdpartymirrors
with a list of servers appended to it, that's not going to fly.

We adjust rsync mirrors all the time and we need the flexibility to pull a
mirror from the rotation without any notice whatsoever.  With our low TTLs
(20min) on rsync.g.o, we can do this now.  If we have to worry about what
servers are listed in thirdpartymirrors, it's not going to work.

So, SRC_URI="patches.gentoo.org/ please.  Leave the round robin/load
balancing to us.

--kurt

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Proposal: patches.gentoo.org

[Doug Goldstein]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Kurt Lieber wrote:
| On Tue, Oct 12, 2004 at 08:12:58PM +0100 or thereabouts, Ciaran
McCreesh wrote:
|
|>On Tue, 12 Oct 2004 15:09:45 -0400 Chris Gianelloni
|><wolf31o2@gentoo.org> wrote:
|>| 2. developer puts SRC_URI="patches.gentoo.org/${P}-$patchname" in the
|>| ebuild, at the end of the SRC_URI definition
|>
|>mirror://patches/ please.
|
|
| If by this you mean that patches should then exist in thirdpartymirrors
| with a list of servers appended to it, that's not going to fly.
|
| We adjust rsync mirrors all the time and we need the flexibility to pull a
| mirror from the rotation without any notice whatsoever.  With our low TTLs
| (20min) on rsync.g.o, we can do this now.  If we have to worry about what
| servers are listed in thirdpartymirrors, it's not going to work.
|
| So, SRC_URI="patches.gentoo.org/ please.  Leave the round robin/load
| balancing to us.
|
| --kurt

Or you could just leave it as mirror://patches/ for ease and just
specify patches.gentoo.org in thirdpartymirrors and you'd get your load
balancing.

- --
Doug Goldstein
http://dev.gentoo.org/~cardoe

Public Key: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x179106D0
Key fingerprint = 7001 5FBF BACE 9E66 3A1C  55E0 161C FF5C 1791 06D0

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBbS1nFhz/XBeRBtARAlncAJ0Rpz90jGFQlzztarVSMtC63g+ZzgCdFX5V
1LtXsmDMANm0YPSZzQewnLc=
=R4nm
-----END PGP SIGNATURE-----

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Proposal: patches.gentoo.org

[Marius Mauch]

[-- Attachment #1: Type: text/plain, Size: 1043 bytes --]

On 10/12/04  Chris Gianelloni wrote:

> The developer would use this temporary location as follows:
> 
> 1. developer creates an ebuild which needs a patch
> 2. developer puts SRC_URI="patches.gentoo.org/${P}-$patchname" in the
> ebuild, at the end of the SRC_URI definition

make that 
SRC_URI="mirror://gentoo/patchname http://patches.gentoo.org/patchname"
($GENTOO_MIRRORS can be empty)

> 3. developer uploads patch to dev.gentoo.org:/space/patches-local and
> dev.gentoo.org:/space/distfiles-local

I don't see a reason to separate patches from distfiles.

> 4. developer commits ebuild(s) to cvs
> 
> The patch would replicate to the rsync boxes and would be available
> via http immediately.
> 
> Remember that portage tries the Gentoo mirrors in ${GENTOO_MIRRORS}
> before checking SRC_URI, so once the files hit the community mirrors,
> the temporary location in SRC_URI would no longer be used.

As I said: $GENTOO_MIRRORS can be empty so you also need the
mirror://gentoo part ($GENTOO_MIRRORS will still be checked first).

Marius

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Proposal: patches.gentoo.org

[Beber]

Marius Mauch wrote:

>>dev.gentoo.org:/space/distfiles-local
>>    
>>
> 3. developer uploads patch to dev.gentoo.org:/space/patches-local and
>
>
>I don't see a reason to separate patches from distfiles.
>  
>
I think my english is bad, tell if you don't understand what I want to 
say :/

I think that it's not to separate patch from distfiles (a lot are 
already avaible) but to have a emerge sync faster and less bandwith 
consumer.
many many patch are on sync server, many are not apply due to proc 
architecture or USE flag.
Thoses patch that are on distfiles server aren't necessary download like 
on sync
I think that it can be a way to take for futur.

Like an other thing (for bandwith use) : if an update of a pacquet is 
avaible : why download are the new sources instead of just a patch ?

Beber

-- 
/* Beber : beber (AT) setibzh (DOT) com
 * https://guybrush.ath.cx
 * Using Mozilla Thunderbird on Gentoo Linux
 * Rachel @ friends.paris
 */


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Proposal: patches.gentoo.org

[Marius Mauch]

[-- Attachment #1: Type: text/plain, Size: 1066 bytes --]

On 10/12/04  Beber wrote:

> Marius Mauch wrote:
> 
> >>dev.gentoo.org:/space/distfiles-local
> >>    
> >>
> > 3. developer uploads patch to dev.gentoo.org:/space/patches-local
> > and
> >
> >
> >I don't see a reason to separate patches from distfiles.
> >  
> >
> I think my english is bad, tell if you don't understand what I want to
> 
> say :/
> 
> I think that it's not to separate patch from distfiles (a lot are 
> already avaible) but to have a emerge sync faster and less bandwith 
> consumer.
> many many patch are on sync server, many are not apply due to proc 
> architecture or USE flag.
> Thoses patch that are on distfiles server aren't necessary download
> like on sync
> I think that it can be a way to take for futur.
> 
> Like an other thing (for bandwith use) : if an update of a pacquet is 
> avaible : why download are the new sources instead of just a patch ?

You're talking about something completely different, I'm just talking
about the location on dev.gentoo.org, that has nothing to do wether a
patch or a full tarball is used.

Marius

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Proposal: patches.gentoo.org

[Nicholas Jones]

[-- Attachment #1: Type: text/plain, Size: 805 bytes --]

And rewritten for clarity:

1. developer creates an ebuild which needs a patch
2. SRC_URI="mirror://gentoo/patchname http://patches.gentoo.org/patchname"
3. Uploads files to dev.gentoo.org:/space/distfiles-local
4. developer commits ebuild(s) to cvs

Infra notes:

Immediate availability via the "secondary" host while the primary
host has yet to receive the files. Once the file is present on the
primary mirrors it may be deleted from the secondary.

Ensuring that the patch host is not the primary mirror will be a
concern, but a script can be devised to ensure duplication of
the patchname and that a primary mirror is listed prior to it.

The host for the files should not be accessable for any reason
except direct filename downloads. No listings (to discourage
setting it as a mirroring source).



[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Proposal: patches.gentoo.org

[Jeffrey Forman]

[-- Attachment #1: Type: text/plain, Size: 2377 bytes --]

After a somewhat lengthy discussion among a couple of the infra dev's
and Nick, we came to some conclusions

(1) A single point of distribution is just a bad idea imho. Imagine a
critical patch that MUST GET OUT IMMEDIATELY (differing opinions of
'must get out now' apply). You get thousands, even tens of thousands of
people hitting patches.g.o and you can wave goodbye to that machine. 

(2) The distribution network is there for a reason. Let me be the first
one to admit that it isnt perfect. Once you place something on
/space/distfiles-local, it hits two machines before it hits the world,
taking between an hour and four hours (theoretically) to go public.
Latency is involved in whichever solution we can propose, short of
giving everyone write access to our master mirror. 

(3) Whether we use distfiles mirrors, or the rsync rotation to
distribute patches, this has not been decided upon. The idea was
broached for a couple minutes earlier today but not even a preliminary
solution was conjured. 

Infra has complete control over the rsync.g.o rotation, so coming up
with ways to improve it that way can be worked out. I am sure this will
be a topic during the next dev meeting, if not listed, I will definitely
mention it. 

My 2 infra cents...Get irate, berate, celebrate.
-Jeffrey

On Tue, 2004-10-12 at 16:10, Nicholas Jones wrote:
> And rewritten for clarity:
> 
> 1. developer creates an ebuild which needs a patch
> 2. SRC_URI="mirror://gentoo/patchname http://patches.gentoo.org/patchname"
> 3. Uploads files to dev.gentoo.org:/space/distfiles-local
> 4. developer commits ebuild(s) to cvs
> 
> Infra notes:
> 
> Immediate availability via the "secondary" host while the primary
> host has yet to receive the files. Once the file is present on the
> primary mirrors it may be deleted from the secondary.
> 
> Ensuring that the patch host is not the primary mirror will be a
> concern, but a script can be devised to ensure duplication of
> the patchname and that a primary mirror is listed prior to it.
> 
> The host for the files should not be accessable for any reason
> except direct filename downloads. No listings (to discourage
> setting it as a mirroring source).
> 
-- 


--------------------
Jeffrey Forman
Gentoo Infrastructure
Gentoo Release Engin.
jforman@gentoo.org
--------------------

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Proposal: patches.gentoo.org

[Jeffrey Forman]

[-- Attachment #1: Type: text/plain, Size: 2377 bytes --]

After a somewhat lengthy discussion among a couple of the infra dev's
and Nick, we came to some conclusions

(1) A single point of distribution is just a bad idea imho. Imagine a
critical patch that MUST GET OUT IMMEDIATELY (differing opinions of
'must get out now' apply). You get thousands, even tens of thousands of
people hitting patches.g.o and you can wave goodbye to that machine. 

(2) The distribution network is there for a reason. Let me be the first
one to admit that it isnt perfect. Once you place something on
/space/distfiles-local, it hits two machines before it hits the world,
taking between an hour and four hours (theoretically) to go public.
Latency is involved in whichever solution we can propose, short of
giving everyone write access to our master mirror. 

(3) Whether we use distfiles mirrors, or the rsync rotation to
distribute patches, this has not been decided upon. The idea was
broached for a couple minutes earlier today but not even a preliminary
solution was conjured. 

Infra has complete control over the rsync.g.o rotation, so coming up
with ways to improve it that way can be worked out. I am sure this will
be a topic during the next dev meeting, if not listed, I will definitely
mention it. 

My 2 infra cents...Get irate, berate, celebrate.
-Jeffrey

On Tue, 2004-10-12 at 16:10, Nicholas Jones wrote:
> And rewritten for clarity:
> 
> 1. developer creates an ebuild which needs a patch
> 2. SRC_URI="mirror://gentoo/patchname http://patches.gentoo.org/patchname"
> 3. Uploads files to dev.gentoo.org:/space/distfiles-local
> 4. developer commits ebuild(s) to cvs
> 
> Infra notes:
> 
> Immediate availability via the "secondary" host while the primary
> host has yet to receive the files. Once the file is present on the
> primary mirrors it may be deleted from the secondary.
> 
> Ensuring that the patch host is not the primary mirror will be a
> concern, but a script can be devised to ensure duplication of
> the patchname and that a primary mirror is listed prior to it.
> 
> The host for the files should not be accessable for any reason
> except direct filename downloads. No listings (to discourage
> setting it as a mirroring source).
> 
-- 


--------------------
Jeffrey Forman
Gentoo Infrastructure
Gentoo Release Engin.
jforman@gentoo.org
--------------------

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Proposal: patches.gentoo.org

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 3177 bytes --]

On Tue, 2004-10-12 at 16:57 -0400, Jeffrey Forman wrote:
> (1) A single point of distribution is just a bad idea imho. Imagine a
> critical patch that MUST GET OUT IMMEDIATELY (differing opinions of
> 'must get out now' apply). You get thousands, even tens of thousands of
> people hitting patches.g.o and you can wave goodbye to that machine.

Except for the idea would be to use a few machines as
"patches.gentoo.org" not a single machine.  I think you missed that
point in my posting.

> (2) The distribution network is there for a reason. Let me be the first
> one to admit that it isnt perfect. Once you place something on
> /space/distfiles-local, it hits two machines before it hits the world,
> taking between an hour and four hours (theoretically) to go public.
> Latency is involved in whichever solution we can propose, short of
> giving everyone write access to our master mirror.

Remember that we're talking patch files and only as a temporary store
until the files hit the main mirror infrastructure.

> (3) Whether we use distfiles mirrors, or the rsync rotation to
> distribute patches, this has not been decided upon. The idea was
> broached for a couple minutes earlier today but not even a preliminary
> solution was conjured.

Actually, we had a decent idea, which Kurt then asked someone to write
up, so I did.  The idea would be to use the rsync servers, which was the
reason I originally put information about something like /space/patches-
local as we do *not* want it to mirror distfiles, only the critical
patches that we say *have to get out quickly* and we don't want to wait
for the normal distfiles mirroring to take place.  Anything other than
this pretty much makes the whole system useless and adds a *ton* over
extra overhead that we were never discussing.

> Infra has complete control over the rsync.g.o rotation, so coming up
> with ways to improve it that way can be worked out. I am sure this will
> be a topic during the next dev meeting, if not listed, I will definitely
> mention it.

That's what this was supposed to be... ;]

> On Tue, 2004-10-12 at 16:10, Nicholas Jones wrote:
> > And rewritten for clarity:
> > 
> > 1. developer creates an ebuild which needs a patch
> > 2. SRC_URI="mirror://gentoo/patchname http://patches.gentoo.org/patchname"
> > 3. Uploads files to dev.gentoo.org:/space/distfiles-local
> > 4. developer commits ebuild(s) to cvs
> > 
> > Infra notes:
> > 
> > Immediate availability via the "secondary" host while the primary
> > host has yet to receive the files. Once the file is present on the
> > primary mirrors it may be deleted from the secondary.
> > 
> > Ensuring that the patch host is not the primary mirror will be a
> > concern, but a script can be devised to ensure duplication of
> > the patchname and that a primary mirror is listed prior to it.
> > 
> > The host for the files should not be accessable for any reason
> > except direct filename downloads. No listings (to discourage
> > setting it as a mirroring source).
> > 
-- 
Chris Gianelloni
Release Engineering - Operational/QA Manager
Games - Developer
Gentoo Linux

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Proposal: patches.gentoo.org

[Kurt Lieber]

[-- Attachment #1: Type: text/plain, Size: 1318 bytes --]

On Tue, Oct 12, 2004 at 04:10:24PM -0400 or thereabouts, Nicholas Jones wrote:
> Immediate availability via the "secondary" host while the primary
> host has yet to receive the files. Once the file is present on the
> primary mirrors it may be deleted from the secondary.

Rather than trying to write a semi-intelligent script that will make a
determination on whether or not a file has made it to the main mirrors
before deleting, I'd much rather just say "all files will exist for N days
and then be deleted".  

If a file doesn't make it to all our mirrors within 24 hours, there's a
larger problem.  Thus, I'd suggest making N = 3 and leaving it at that.

> Ensuring that the patch host is not the primary mirror will be a
> concern, but a script can be devised to ensure duplication of
> the patchname and that a primary mirror is listed prior to it.

This logic should be added to repoman if it's added anywhere.  Also, by
keeping N=3, devs will quickly learn NOT to use this location as a primary
mirror once they start getting deluged with bug reports about SRC_URI being
broken.

> The host for the files should not be accessable for any reason
> except direct filename downloads. No listings (to discourage
> setting it as a mirroring source).

Great idea -- agree 100%.

--kurt

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Proposal: patches.gentoo.org

[Paul de Vrieze]

[-- Attachment #1: Type: text/plain, Size: 751 bytes --]

On Tuesday 12 October 2004 21:27, Marius Mauch wrote:
> On 10/12/04  Chris Gianelloni wrote:
> > The developer would use this temporary location as follows:
> >
> > 1. developer creates an ebuild which needs a patch
> > 2. developer puts SRC_URI="patches.gentoo.org/${P}-$patchname" in the
> > ebuild, at the end of the SRC_URI definition
>
> make that
> SRC_URI="mirror://gentoo/patchname http://patches.gentoo.org/patchname"
> ($GENTOO_MIRRORS can be empty)

What is the difference from just putting these files in your own dev 
space? If they just exist on dev.gentoo.org the developers themselves can 
arange removing etc.

Paul

-- 
Paul de Vrieze
Gentoo Developer
Mail: pauldv@gentoo.org
Homepage: http://www.devrieze.net

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Proposal: patches.gentoo.org

[Lance Albertson]

[-- Attachment #1: Type: text/plain, Size: 1287 bytes --]

On Wed, 2004-10-13 at 04:46, Paul de Vrieze wrote:
> On Tuesday 12 October 2004 21:27, Marius Mauch wrote:
> > On 10/12/04  Chris Gianelloni wrote:
> > > The developer would use this temporary location as follows:
> > >
> > > 1. developer creates an ebuild which needs a patch
> > > 2. developer puts SRC_URI="patches.gentoo.org/${P}-$patchname" in the
> > > ebuild, at the end of the SRC_URI definition
> >
> > make that
> > SRC_URI="mirror://gentoo/patchname http://patches.gentoo.org/patchname"
> > ($GENTOO_MIRRORS can be empty)
> 
> What is the difference from just putting these files in your own dev 
> space? If they just exist on dev.gentoo.org the developers themselves can 
> arange removing etc.

The difference is, dev is a single point of failure, plus you're
suggesting that dev become a single mirror for that file. It could
potentially create an issue of ddos. The suggestion we're making is
coming up with a solution that is as simple as putting on your dev
space, but its distributed on several servers we already have.
-- 
Lance Albertson <ramereth@gentoo.org>
Gentoo Infrastructure

---
GPG Public Key:  <http://www.ramereth.net/lance.asc>
Key fingerprint: 0423 92F3 544A 1282 5AB1  4D07 416F A15D 27F4 B742

ramereth/irc.freenode.net

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Proposal: patches.gentoo.org

[Kurt Lieber]

[-- Attachment #1: Type: text/plain, Size: 521 bytes --]

On Wed, Oct 13, 2004 at 11:46:47AM +0200 or thereabouts, Paul de Vrieze wrote:
> What is the difference from just putting these files in your own dev 
> space? If they just exist on dev.gentoo.org the developers themselves can 
> arange removing etc.

Developers shouldn't be doing this, really.  dev.g.o is not set up with any
sort of redundancy, nor is it sized to provide reasonable scalability in
the event a popular patch is placed there.  Basically, it's not designed or
intended to be a mirror.

--kurt

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Proposal: patches.gentoo.org

[Nick Dimiduk]

Chris Gianelloni wrote:

> We were talking about this on IRC due to the growing concern over
> patches and other files in ${FILESDIR} in the portage tree.  This is
> basically the idea that we came up with to resolve the problem.
> 

Why do we need a separate location for patches?  How are they 
fundamentally different from the sources in distfiles?

-Nick Dimiduk

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Proposal: patches.gentoo.org

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 647 bytes --]

On Tue, 12 Oct 2004 15:57:00 -0400 Nick Dimiduk <ndimiduk@gentoo.org>
wrote:
| Chris Gianelloni wrote:
| > We were talking about this on IRC due to the growing concern over
| > patches and other files in ${FILESDIR} in the portage tree.  This is
| > basically the idea that we came up with to resolve the problem.
| > 
| 
| Why do we need a separate location for patches?  How are they 
| fundamentally different from the sources in distfiles?

distfiles take eight hours to get mirrored.

-- 
Ciaran McCreesh : Gentoo Developer (Vim, Fluxbox, Sparc, Mips)
Mail            : ciaranm at gentoo.org
Web             : http://dev.gentoo.org/~ciaranm


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Proposal: patches.gentoo.org

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 1165 bytes --]

On Tue, 2004-10-12 at 15:57 -0400, Nick Dimiduk wrote:
> Chris Gianelloni wrote:
> 
> > We were talking about this on IRC due to the growing concern over
> > patches and other files in ${FILESDIR} in the portage tree.  This is
> > basically the idea that we came up with to resolve the problem.
> > 
> 
> Why do we need a separate location for patches?  How are they 
> fundamentally different from the sources in distfiles?

Write a patch yourself and add it to /space/disfiles-local and see how
long it takes to get to *all* the mirrors.  Now look at how many bugs
you have for missing files (your patch).

The point is that people want to see the patches removed from
${FILESDIR} and moved to ${DISTDIR}, but the distfiles replication is
too slow for this to work properly.  Because of this, we came up with
this proposal for a temporary location to hold patches that are
available on-demand and immediately, before the distfiles sync takes
place.

Currently, many of us are using dev.gentoo.org for this purpose.  This
proposal is to move this to a supported infrastructure with some
redundancy rather than a single point of failure.


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Proposal: patches.gentoo.org

[Mike Frysinger]

On Tuesday 12 October 2004 03:09 pm, Chris Gianelloni wrote:
> 1. developer creates an ebuild which needs a patch

umm, just so everyone knows and doesnt forget, dont post plain text files !  
always compress them so they're in binary form

ive handled bugs in the past where plain text files were posted in SRC_URI and 
the mirrors were so kind as to convert the CRLF stuff while transfering in 
ASCII mode thus breaking digests

so, dont do it !
-mike

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Proposal: patches.gentoo.org

[Kurt Lieber]

[-- Attachment #1: Type: text/plain, Size: 698 bytes --]

On Tue, Oct 12, 2004 at 03:09:45PM -0400 or thereabouts, Chris Gianelloni wrote:
> We were talking about this on IRC due to the growing concern over
> patches and other files in ${FILESDIR} in the portage tree.  This is
> basically the idea that we came up with to resolve the problem.

After reading the other thread, sounds like this problem may not be as
large as we thought it was.  If we're talking about a ~5% space savings in
portage itself, I'm not sure it's worth the extra effort of setting up yet
another mirroring system.

Someone please correct me if I'm wrong, but please use facts to do it.  I'm
willing to set up this mirroring system, but only if it solves a real
problem.

--kurt

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Proposal: patches.gentoo.org

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 653 bytes --]

On Wed, 13 Oct 2004 13:05:27 +0000 Kurt Lieber <klieber@gentoo.org>
wrote:
| Someone please correct me if I'm wrong, but please use facts to do it.
|  I'm willing to set up this mirroring system, but only if it solves a
|  real problem.

It is a fact that a lot of users are complaining. A quick glance back
through the threads on this list (or for that matter on the forums) over
the past few weeks shows this.

Or, by "problem", did you mean an actual technical issue rather than a
PR one? :)

-- 
Ciaran McCreesh : Gentoo Developer (Vim, Fluxbox, Sparc, Mips)
Mail            : ciaranm at gentoo.org
Web             : http://dev.gentoo.org/~ciaranm


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Proposal: patches.gentoo.org

[Kurt Lieber]

[-- Attachment #1: Type: text/plain, Size: 538 bytes --]

On Wed, Oct 13, 2004 at 04:16:53PM +0100 or thereabouts, Ciaran McCreesh wrote:
> It is a fact that a lot of users are complaining. A quick glance back
> through the threads on this list (or for that matter on the forums) over
> the past few weeks shows this.

OK, then it's also a fact that the proposed solution (patches.gentoo.org)
is unlikely to solve these complaints as it won't make much of a difference
in tree size.

I also question whether "a lot" is accurate.  Perhaps, "a few very vocal"
users might be more accurate.

--kurt

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Proposal: patches.gentoo.org

[Xavier Neys]

[-- Attachment #1: Type: text/plain, Size: 1830 bytes --]

Kurt Lieber wrote:
> On Wed, Oct 13, 2004 at 04:16:53PM +0100 or thereabouts, Ciaran McCreesh wrote:
> 
>>It is a fact that a lot of users are complaining. A quick glance back
>>through the threads on this list (or for that matter on the forums) over
>>the past few weeks shows this.
> 
> OK, then it's also a fact that the proposed solution (patches.gentoo.org)
> is unlikely to solve these complaints as it won't make much of a difference
> in tree size.
> 
> I also question whether "a lot" is accurate.  Perhaps, "a few very vocal"
> users might be more accurate.

How many are "a few" and how much vocal is "very" vocal?
The fact is there are more users complaining about this than there were a year 
before.
We can let this trend grow or see if anything can be done about it.

I doubt cleaning up the tree a bit would increase performance significantly.
Cleaning up is neat but it might not be worth the effort unless it can be 
significantly cleaned up, as far as improving the performance of emerge --sync 
is concerned.

I compiled some timings here:
http://dev.gentoo.org/~neysx/tests/emerge--sync.html

It looks like the 'metadata' step is taking up half of the time spent when 
running emerge --sync. It is usually more, sometimes much more.
Maybe something could be improved there.
Don't hit me, I know I don't know the code.
In the best of all cases I have tried, running emerge --sync against an 
up-to-date tree from which all 25,000+ /files/* had been removed, took 130 
seconds, 5 seconds to sync, 125 seconds to find out nothing had to be changed 
in the cache.
Hint: use rsync output to update the cache.

If you feel this is heresy, or mere noise, just ignore my ignorance.


Regards,
-- 
/  Xavier Neys
\_ Gentoo Documentation Project
/  French & Internationalisation Lead
\  http://www.gentoo.org/doc/en
/\

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 256 bytes --]

Re: [gentoo-dev] Proposal: patches.gentoo.org

[Dylan Carlson]

On Tue October 12 2004 15:09, Chris Gianelloni wrote:
> The proposal is to get infrastructure to provide a temporary location
> for developers to upload patches which are necessary to be immediately
> available to users.  This space would be writable by all CVS developers
> and would be accessible to the world via http.  Developers would use
> this space for temporary storage only for patches which are
> created/rolled in-house.

Here here.  

Regardless of what we agree the pathname should be, it's a good idea.  

I'd also like to see a naming convention of patches established.   (crowd 
moans: "not more red tape!")  OK, perhaps more of a guideline/best 
practices than a requirement.   Right now I don't believe there is one... 
is there?  It would be refreshing to look at that patch repository and see 
clarity and consistency in naming, as that will be a large listing of 
files.

Cheers,
Dylan Carlson [absinthe@gentoo.org]
Public Key: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x708E165F


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Proposal: patches.gentoo.org

[Mark Loeser]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Dylan Carlson wrote:
| Here here.
|
| Regardless of what we agree the pathname should be, it's a good idea.
|
| I'd also like to see a naming convention of patches established.   (crowd
| moans: "not more red tape!")  OK, perhaps more of a guideline/best
| practices than a requirement.   Right now I don't believe there is one...
| is there?  It would be refreshing to look at that patch repository and
see
| clarity and consistency in naming, as that will be a large listing of
| files.

I agree, this would be a very good policy to put into place.  Removing
all of the patches from the tree will help to reduce its size, by how
much has already been a topic of debate, but it would help regardless.
If it doesn't make that big of a difference now, it has the potential of
stopping this from becoming a bigger problem in the future.

The tree is growing at a very large rate and rsync times are taking a
lot longer now than they did for me a year or so ago.  Removing all of
the patches from the tree will definitely help this, but I think other
policies should be more strictly inforced as well.  There are some very
large files in the tree currently that aren't patches, like enormous
Changelogs for a few packages, and also a lot of old ebuilds that are
sitting around for some packages.  It'd be nice if the tree was kept as
clean as possible to keep rsync times down as much as possible.

Mark
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBbuoHZPlCVB9SdGIRApTpAJ92YzwcPjge6QMofhdpnmx9Z0BpuACdFMK6
AtTFt0hR3utggvOnar7cBU0=
=85Ee
-----END PGP SIGNATURE-----

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Proposal: patches.gentoo.org

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 2506 bytes --]

On Thu, 2004-10-14 at 17:05 -0400, Mark Loeser wrote:
> I agree, this would be a very good policy to put into place.  Removing
> all of the patches from the tree will help to reduce its size, by how
> much has already been a topic of debate, but it would help regardless.
> If it doesn't make that big of a difference now, it has the potential of
> stopping this from becoming a bigger problem in the future.

I know that I have mentioned this before, but having a set of -release
trees would also alleviate much of this.  There would be less reason for
keeping around so many older ebuilds in the -current tree, as they would
still be available in the -release trees.  The real concern with any
separation of the tree is how do we keep them all up to date with
security patches and such.  This would still need to be addressed and I
am still working on a good solution.

> The tree is growing at a very large rate and rsync times are taking a
> lot longer now than they did for me a year or so ago.  Removing all of
> the patches from the tree will definitely help this, but I think other
> policies should be more strictly inforced as well.  There are some very
> large files in the tree currently that aren't patches, like enormous
> Changelogs for a few packages, and also a lot of old ebuilds that are
> sitting around for some packages.  It'd be nice if the tree was kept as
> clean as possible to keep rsync times down as much as possible.

Perhaps some way of moving the ChangeLog files themselves out of the
tree and into some other location where they could be pulled as-needed.
This would be a good place to start, as the ChangeLog is not used by
portage itself and is only informational for the users and developers.
The only thing that would need to be changed is the -l option to portage
would need to be modified to pull the data from the web (or wherever)
instead of the tree.

The real problem we run into with almost any solution is to get any real
gains we have to make drastic changes to the tree which could break
older releases and people's installs that haven't been keeping up to
date.  As a good example, when we make changes to portage, they have to
be done incrementally and have to be able to co-exist with the way
things are currently, and also the way things have been, otherwise we
risk really causing trouble for many of our users.

-- 
Chris Gianelloni
Release Engineering - Operations/QA Manager
Games - Developer
Gentoo Linux

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[gentoo-dev] Patch filename conventions (WAS: Proposal: patches.gentoo.org)

[Dylan Carlson]

On Thu October 14 2004 16:38, Dylan Carlson wrote:
> I'd also like to see a naming convention of patches established.

I know it's bad form to reply to your own message, but I'm doing it anyway. 
Here's my quick stab on filename convention:

${PN}-${PV}-arch/use-##-short_desc[-gentoo].diff.bz2

where...
##  = the patch number (two digit w/leading zeros) in order added
arch  = "any", or arch it applies to
short_desc = ~20 characters or less, use flag at the beginning (if applic),   
  underscores between words
[-gentoo] = optional, if diff is gentoo-specific and/or originated @ gentoo

examples:
 glibc-2.3.4-any-01-pax_localedef_fix_trampoline.diff.bz2
 glibc-2.3.4-any-02-pax_pt.diff.bz2
 glibc-2.3.4-any-03-pax_i386_got_fix.bz2
 glibc-2.3.4-amd64-01-conf_libdir-gentoo.diff.bz2
 glibc-2.3.4-mips-01-addabi.diff.bz2
 glibc-2.3.4-mips-02-syscall.h.diff.bz2
 glibc-2.3.4-mips-03-sysify.diff.bz2
 glibc-2.3.4-mips-04-multilib_nolib3264.diff.bz2
 glibc-2.3.4-mips-05-pax_dl_execstack_support.diff.bz2

This convention doesn't address files that aren't patches (i.e., conf 
files, makefiles, etc)

-- 
Dylan Carlson [absinthe@gentoo.org]
Public Key: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x708E165F

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Patch filename conventions (WAS: Proposal: patches.gentoo.org)

[Mike Frysinger]

On Thursday 14 October 2004 06:08 pm, Dylan Carlson wrote:
> I know it's bad form to reply to your own message, but I'm doing it anyway.
> Here's my quick stab on filename convention:

if a covention is going to be developed it's going to involve what epatch 
already uses:
##_${ARCH}_foo.patch

if you want to flesh out the 'foo' part, fine, but the prefix involving a # 
and $ARCH is already set in stone
-mike

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Patch filename conventions (WAS: Proposal: patches.gentoo.org)

[Dylan Carlson]

On Thu October 14 2004 18:14, Mike Frysinger wrote:
> if you want to flesh out the 'foo' part, fine, but the prefix involving
> a # and $ARCH is already set in stone

Set in stone?  Surely we're not that inflexible.  If we're going to have 
all the patches in a single, shared directory, we're going to need 
something more thoughtful than that, and that means -- at the very least 
-- starting the filename with the name of the package.

-- 
Dylan Carlson [absinthe@gentoo.org]
Public Key: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x708E165F

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Patch filename conventions (WAS: Proposal: patches.gentoo.org)

[Donnie Berkholz]

[-- Attachment #1: Type: text/plain, Size: 637 bytes --]

On Thu, 2004-10-14 at 22:20 -0400, Dylan Carlson wrote:
> On Thu October 14 2004 18:14, Mike Frysinger wrote:
> > if you want to flesh out the 'foo' part, fine, but the prefix involving
> > a # and $ARCH is already set in stone
> 
> Set in stone?  Surely we're not that inflexible.  If we're going to have 
> all the patches in a single, shared directory, we're going to need 
> something more thoughtful than that, and that means -- at the very least 
> -- starting the filename with the name of the package.

Who ever said the tarball had to have the same name as the patches
inside it?
-- 
Donnie Berkholz
Gentoo Linux

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Patch filename conventions (WAS: Proposal: patches.gentoo.org)

[Dylan Carlson]

On Thu October 14 2004 22:58, Donnie Berkholz wrote:
> Who ever said the tarball had to have the same name as the patches
> inside it?

I suggested putting the package name, version, arch, patch # in the 
filename of every patch.  Because epatch isn't going to work with the 
current scheme (##-arch-foo.patch) if all the patches for every package 
are in a single directory.  If I'm understanding the patches.gentoo.org 
thing correctly, and also how epatch works.

-- 
Dylan Carlson [absinthe@gentoo.org]
Public Key: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x708E165F

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Patch filename conventions (WAS: Proposal: patches.gentoo.org)

[Donnie Berkholz]

[-- Attachment #1: Type: text/plain, Size: 1039 bytes --]

On Fri, 2004-10-15 at 00:24 -0400, Dylan Carlson wrote:
> On Thu October 14 2004 22:58, Donnie Berkholz wrote:
> > Who ever said the tarball had to have the same name as the patches
> > inside it?
> 
> I suggested putting the package name, version, arch, patch # in the 
> filename of every patch.  Because epatch isn't going to work with the 
> current scheme (##-arch-foo.patch) if all the patches for every package 
> are in a single directory.  If I'm understanding the patches.gentoo.org 
> thing correctly, and also how epatch works.

Maybe we're talking past each other here:
There's no point in redundantly including $p in every single patch name
if all the patches are in a tarball containing $P. You'd be looking at a
list of tarballs such as $P-patches.tar.bz2, each containing something
like:

patches/
3123_x86_fix-foo.patch
3124_hppa_fix-bar.patch
etc.

The only way what you're talking about makes any sense to me is if a
tarball isn't used, just a bzip2'd patch.
-- 
Donnie Berkholz
Gentoo Linux

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Patch filename conventions (WAS: Proposal: patches.gentoo.org)

[Dylan Carlson]

On Fri October 15 2004 01:13, Donnie Berkholz wrote:
> The only way what you're talking about makes any sense to me is if a
> tarball isn't used, just a bzip2'd patch.

Sure, I see your point.  Going by what's in the tree right now:  ~5700 
plain ASCII patch files, ~200 binary/compressed (mainly bz2), and only a 
few tarballs.   All would need to be binary for this switch, anyway.  
There's some transition work for maintainers to make patchsets.  Because 
some patch files are used between multiple versions, etc.  Using patchsets 
is more of a hassle IMO, but as long as it's a standard practice, I'm fine 
with it...

Cheers,
Dylan Carlson [absinthe@gentoo.org]
Public Key: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x708E165F

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Patch filename conventions (WAS: Proposal: patches.gentoo.org)

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 1349 bytes --]

On Thu, Oct 14, 2004 at 06:14:30PM -0400, Mike Frysinger wrote:
> On Thursday 14 October 2004 06:08 pm, Dylan Carlson wrote:
> > I know it's bad form to reply to your own message, but I'm doing it anyway.
> > Here's my quick stab on filename convention:
> if a covention is going to be developed it's going to involve what epatch 
> already uses:
> ##_${ARCH}_foo.patch
> 
> if you want to flesh out the 'foo' part, fine, but the prefix involving a # 
> and $ARCH is already set in stone
Thats a very bad road to take.

The primary reason against it is packages that share patches.
Eg:
- php, php-cgi, mod_php all use the same hardenedPHP patch.

One thing that hasn't been proposed, but I feel would probably make a
very good suggestion, would be to increase the update frequency on the
master distfiles mirror that users have access to, assuming it can
handle the load (I assume it has the bandwidth given the proposals for
another patches server near it).
If the patch isn't found in local mirror://gentoo/ servers, they
eventully end up at th master distfiles box, which will have the patch.

-- 
Robin Hugh Johnson
E-Mail     : robbat2@orbis-terrarum.net
Home Page  : http://www.orbis-terrarum.net/?l=people.robbat2
ICQ#       : 30269588 or 41961639
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]

Re: [gentoo-dev] Patch filename conventions (WAS: Proposal: patches.gentoo.org)

[Kurt Lieber]

[-- Attachment #1: Type: text/plain, Size: 784 bytes --]

On Thu, Oct 14, 2004 at 11:04:14PM -0700 or thereabouts, Robin H. Johnson wrote:
> One thing that hasn't been proposed, but I feel would probably make a
> very good suggestion, would be to increase the update frequency on the
> master distfiles mirror that users have access to, assuming it can
> handle the load (I assume it has the bandwidth given the proposals for
> another patches server near it).
> If the patch isn't found in local mirror://gentoo/ servers, they
> eventully end up at th master distfiles box, which will have the patch.

The master distfile mirror already syncs hourly, so I'm not sure how much
extra benefit is gained by moving that to every 30 minutes.  The larger
issue the fact that not all users have that mirror in their GENTOO_MIRRORS
variable.

--kurt

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Proposal: patches.gentoo.org

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 1643 bytes --]

On Thu, 2004-10-14 at 16:38 -0400, Dylan Carlson wrote:
> On Tue October 12 2004 15:09, Chris Gianelloni wrote:
> > The proposal is to get infrastructure to provide a temporary location
> > for developers to upload patches which are necessary to be immediately
> > available to users.  This space would be writable by all CVS developers
> > and would be accessible to the world via http.  Developers would use
> > this space for temporary storage only for patches which are
> > created/rolled in-house.
> 
> Here here.  
> 
> Regardless of what we agree the pathname should be, it's a good idea.  
> 
> I'd also like to see a naming convention of patches established.   (crowd 
> moans: "not more red tape!")  OK, perhaps more of a guideline/best 
> practices than a requirement.   Right now I don't believe there is one... 
> is there?  It would be refreshing to look at that patch repository and see 
> clarity and consistency in naming, as that will be a large listing of 
> files.

I believe the current, albeit unwritten, rule is to name the patch as
${P}-name.patch just to make them easier to locate.  As for the large
listing, that is very doubtful, as this is just temporary holding until
the patches hit the distfiles mirrors.  No patch would survive longer
than 7 days in this "holding pen" before it would be removed in favor of
our distfiles mirrors.  This would also mean making tarballs out of all
patches, as vapier mentioned earlier, to keep there from being any
ASCII/Binary download problems.

-- 
Chris Gianelloni
Release Engineering - Operations/QA Manager
Games - Developer
Gentoo Linux

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]