[gentoo-dev] Portage Subcategory Capabilities?

[gentoo-dev] Portage Subcategory Capabilities?

[Chris White]

[-- Attachment #1: Type: text/plain, Size: 1635 bytes --]

All,

    Has there been some talk as to subcategory implementations in 
portage?  I'm going to assume that it could be done (with speed 
sacrifice of course), but it would be well beneficial in an ever 
expanding tree.  As more and more new ebuilds flow in everyday, sooner 
or later categories are split.  This would be somewhat of a time saving 
effort for the result, but its creation would be a little consuming.  I 
wouldn't mind trying to look over the creation of such an implementation 
(I won't of course be fully active on that until the end of this 
semester feh :( ).  Here's somewhat of an idea:

media-sound: ok, I'm working on organizing this because it needs to be.  
So let's see... wc says!:

chris@secures /usr/local/portage/gentoo-x86/media-sound $ ls -1 | wc
    275     275    2349

That's right... 257 nice packages which might actually be installed if 
users knew what they were :).  Something like this might be a little nicer:

media-sound/editors/audacity
media-sound/players/xmms

now let's say players gets way to huge:

media-sound/players/mp3/
media-sound/players/ogg/

etc.  While this may be somewhat of a strange example.  I think it 
somewhat the organizational need the tree will require at some point.  
This is just an idea of a good way to organize this stuff.  If there's 
something I'm just not seeing, feel free to slap me with a wet trout and 
suggest something better.  This should be looked at though.  Thanks to 
everyone's patience in dealing with me as always :).

-- 
Chris White <chriswhite@gentoo.org>
------------------------
Sound | Video | Security
ChrisWhite @ irc.freenode.net


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 256 bytes --]

Re: [gentoo-dev] Portage Subcategory Capabilities?

[Georgi Georgiev]

[-- Attachment #1: Type: text/plain, Size: 713 bytes --]

maillog: 04/10/2004-03:55:13(+0900): Chris White types
<snip>
> Something like this might be a little nicer:
> 
> media-sound/editors/audacity
> media-sound/players/xmms
> 
> now let's say players gets way to huge:
> 
> media-sound/players/mp3/
> media-sound/players/ogg/

I personally like the idea. What about turning the first dash into a
slash as well? 

media/sound/editors/audacity
media/sound/players/xmms
media/sound/players/mp3/
media/sound/players/ogg/

-- 
 >   Georgi Georgiev    > General Protection Fault! [ Ignore ] [        >
<     chutz@gg3.net    <  Reboot ] [ Install Linux ] -- From a         <
 >  +81(90)6266-1163    > Slashdot.org post                             >

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Portage Subcategory Capabilities?

[Andrew Gaffney]

Georgi Georgiev wrote:
> maillog: 04/10/2004-03:55:13(+0900): Chris White types
> <snip>
> 
>>Something like this might be a little nicer:
>>
>>media-sound/editors/audacity
>>media-sound/players/xmms
>>
>>now let's say players gets way to huge:
>>
>>media-sound/players/mp3/
>>media-sound/players/ogg/
> 
> I personally like the idea. What about turning the first dash into a
> slash as well? 
> 
> media/sound/editors/audacity
> media/sound/players/xmms
> media/sound/players/mp3/
> media/sound/players/ogg/

This has been brought up before in the past and been shot down, but I'll humor 
you. If support for this were to be added into Portage, there would be a few 
things to think about:

1) this will cause a performance hit no matter how it is done
2) how will Portage know the difference between a package and another
    sub-category when it is walking the tree? It could do something like walking
    all the way to the end of the category/sub-category/sub-category tree until
    it finds .ebuild files and then backing up a level or 2 to determine the
    category, but again this would cause an enormous performance hit due to the
    additional required I/O.
3) the many additional directories would cause an 'emerge sync' to take
    longer than it does now.

Basically, you'd be pissing everyone off with little benefit (except maybe to 
developer sanity). Keep in mind that I probably have absolutely no idea what I'm 
talking about and nothing I say should be accepted as fact without verifying it 
elsewhere.

-- 
Andrew Gaffney
Network Administrator
Skyline Aeronautics, LLC.
636-357-1548


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Portage Subcategory Capabilities?

[Matthew Schulkind]

> This has been brought up before in the past and been shot down, but I'll humor
> you. If support for this were to be added into Portage, there would be a few
> things to think about:
> 
> 1) this will cause a performance hit no matter how it is done
> 2) how will Portage know the difference between a package and another
>     sub-category when it is walking the tree? It could do something like walking
>     all the way to the end of the category/sub-category/sub-category tree until
>     it finds .ebuild files and then backing up a level or 2 to determine the
>     category, but again this would cause an enormous performance hit due to the
>     additional required I/O.
> 3) the many additional directories would cause an 'emerge sync' to take
>     longer than it does now.
> 
> Basically, you'd be pissing everyone off with little benefit (except maybe to
> developer sanity). Keep in mind that I probably have absolutely no idea what I'm
> talking about and nothing I say should be accepted as fact without verifying it
> elsewhere.
> 
> --
> Andrew Gaffney
> Network Administrator
> Skyline Aeronautics, LLC.
> 636-357-1548
> 
> --
> gentoo-dev@gentoo.org mailing list
> 
> 

Why does this necessarily have to change the portage directory structure?

I think this is a good idea to, but why not implement a secondary
browsing interface which could use these categories and just map to
ebuilds. With this strategy, even if a package fell into two
categories, it could be put into both. This could also lead to any
number of more pretty features.

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Portage Subcategory Capabilities?

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 1895 bytes --]

On Mon, Oct 04, 2004 at 12:34:50AM -0500, Andrew Gaffney wrote:
> 1) this will cause a performance hit no matter how it is done
A performance hit yes, but a lot less of a performance hit than you
might expect, provided one requirement is met:
- A directory cannot be both a category and a package ever. (This means
  no renaming packages to categories too).

> 2) how will Portage know the difference between a package and another
>    sub-category when it is walking the tree? It could do something
>    like walking all the way to the end of the
>    category/sub-category/sub-category tree until it finds .ebuild
>    files and then backing up a level or 2 to determine the category,
>    but again this would cause an enormous performance hit due to the
>    additional required I/O.
Portage already has an explicit list of all valid categories that it
uses to check paths (see profiles/categories). Simply split the provided
package atom into two components by looking for the very last slash.
That gives you (category,pn), which can then be verified for existence
by comparing the category against the list, and then checking it's
directory for the package name. I believe it works in a very similar
fashion to this already.

> 3) the many additional directories would cause an 'emerge sync' to take
>    longer than it does now.
Rsync would not be the bottleneck, and performance would actually
improve on any system that has trouble coping with very large numbers of
files in a single directory. Go and time 'ls' in your $DISTDIR. None of
our package directories have reached that point yet, but I sincerely
hope they never do.

-- 
Robin Hugh Johnson
E-Mail     : robbat2@orbis-terrarum.net
Home Page  : http://www.orbis-terrarum.net/?l=people.robbat2
ICQ#       : 30269588 or 41961639
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]

Re: [gentoo-dev] Portage Subcategory Capabilities?

[George Shapovalov]

On Sunday 03 October 2004 22:34, Andrew Gaffney wrote:
> This has been brought up before in the past and been shot down, but I'll
> humor you. If support for this were to be added into Portage, there would
No it wasn't. Discussion just felt off without leading to any conclusion as it 
happens quite often. Well, actually such discussions were happening multiple 
times in the past and while in some it may have felt like "shot down" in 
others it even went as far as being added to portage-ng features list ;).

> be a few things to think about:
>
> 1) this will cause a performance hit no matter how it is done
> 2) how will Portage know the difference between a package and another
>     sub-category when it is walking the tree? It could do something like
Grouping both as they are basically the same.
Directory walker is a pretty standard piece of code discussed in many text 
books for various languages. Python has explicit support for some basic 
dirwalker functionality in its standard library. 

As for distinguishing: in prior discussions there were calls to go the 
opposite route and flatten namespace. That is to get rid of categories 
altogether and rely solely on search. Thus one approach would be to make one 
pass accumulating all package names and then work off that (store path to 
ebuild dir in the cache for example). However that may not be optimal as this 
will disallow duplicate package names (which we *have* irrespectively of 
whether this is good or bad). 

A simpler and more efficient approach may be to simply prepend path to package 
name, possibly substituting '/' with some other separator if that's a 
problem. This will give a flat list of effective package names (that include 
all their categorization) which may even simplify further processing. 
Actually having a separate pass may be even more of an advantage as portage 
works off its cache for the most part anyway and this cache is created 
server-side already. Thus this change might even speed things up :) (well, 
not really, but shouldn't cause significant performance degradation either).

Ebuilds will have to specify full category for each dependency with such 
approach, but that's a policy anyway..


> 3) the many additional directories would cause an 'emerge sync' to take
>     longer than it does now.
Khm, not really sure about this one. If it really becomes that bad there were 
ways discussed on improving syncing (cvsup or collating small files comes to 
mind, but that's a separate discussion).

> Basically, you'd be pissing everyone off with little benefit (except maybe
> to developer sanity). Keep in mind that I probably have absolutely no idea
I wouldn't call this a small benefit, considering that its not only 
developers' sanity at stake here, I am pretty sure users would appreciate a 
more sane portage structure as well :).

> what I'm talking about and nothing I say should be accepted as fact without
> verifying it elsewhere.
That's what we are trying to do :).

George


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Portage Subcategory Capabilities?

[Georgi Georgiev]

maillog: 04/10/2004-00:16:10(-0700): George Shapovalov types
<snip>
> As for distinguishing: in prior discussions there were calls to go the 
> opposite route and flatten namespace. That is to get rid of categories 
> altogether and rely solely on search. Thus one approach would be to make one 
> pass accumulating all package names and then work off that (store path to 
> ebuild dir in the cache for example). However that may not be optimal as this 
> will disallow duplicate package names (which we *have* irrespectively of 
> whether this is good or bad). 

I remember that discussion. And there were to be symlinks to the
packages in the flat namespace, similar to what we already have in
$PKGDIR.

<snip>
> A simpler and more efficient approach may be to simply prepend path to package 
> name, possibly substituting '/' with some other separator if that's a 
> problem. This will give a flat list of effective package names (that include 
> all their categorization) which may even simplify further processing. 
> Actually having a separate pass may be even more of an advantage as portage 
> works off its cache for the most part anyway and this cache is created 
> server-side already. Thus this change might even speed things up :) (well, 
> not really, but shouldn't cause significant performance degradation either).

Naah, that would also require users to specify full names, which is
ugly.

<snip>
> > 3) the many additional directories would cause an 'emerge sync' to take
> >     longer than it does now.

Having 100 or so more directories when there are 80,000+ items in $PORTDIR
wouldn't really hurt rsync.

-- 
*>   Georgi Georgiev   *> If you go out of your mind, do it quietly,   *>
<*    chutz@gg3.net    <* so as not to disturb those around you.       <*
*>  +81(90)6266-1163   *>                                              *>

--
gentoo-dev@gentoo.org mailing list

[gentoo-dev] Unique package names

[Georgi Georgiev]

maillog: 04/10/2004-00:16:10(-0700): George Shapovalov types
> However that may not be optimal as this will disallow duplicate
> package names (which we *have* irrespectively of whether this is good
> or bad). 

Isn't it about time to do something about these duplicate package names?
Why not get rid of them, and then think of reorganizing the tree.

Most of the duplicates are emacs/xemacs modules, which could be renamed
to the same name with an emacs- xemacs- prefix.

Here is the current list of duplicates, with suggestions.

app-emacs/apel          app-xemacs/apel		: {,x}emacs-apel (same for other (x)emacs plugins)
app-admin/analog        app-emacs/analog	: {,x}emacs-analog
app-emacs/auctex        app-xemacs/auctex	: {,x}emacs-auctex
app-emacs/bbdb          app-xemacs/bbdb		: {,x}emacs-bbdb
app-emacs/binclock      app-misc/binclock	: emacs-binclock, binclock
app-emacs/dictionary    app-xemacs/dictionary	: {,x}emacs-dictionary
app-emacs/ecb           app-xemacs/ecb		: {,x}emacs-ecb
app-emacs/eieio         app-xemacs/eieio	: {,x}emacs-eieio
app-emacs/elib          app-xemacs/elib		: {,x}emacs-elib
app-emacs/ess           app-xemacs/ess		: {,x}emacs-ess
app-emacs/gnus          app-xemacs/gnus		: {,x}emacs-gnus
app-emacs/haskell-mode  app-xemacs/haskell-mode	: {,x}emacs-haskell-mode
app-emacs/ibuffer       app-xemacs/ibuffer	: {,x}emacs-ibuffer
app-emacs/igrep         app-xemacs/igrep	: {,x}emacs-igrep
app-emacs/ilisp         app-xemacs/ilisp	: {,x}emacs-ilisp
app-emacs/jde           app-xemacs/jde		: {,x}emacs-jde
app-emacs/liece         app-xemacs/liece	: {,x}emacs-liece
app-emacs/lookup        app-xemacs/lookup	: {,x}emacs-lookup
app-emacs/mailcrypt     app-xemacs/mailcrypt	: {,x}emacs-mailcrypt
app-emacs/mew           app-xemacs/mew		: {,x}emacs-mew
app-emacs/mldonkey      net-p2p/mldonkey	: emacs-mldonkey, mldonkey
app-emacs/mmm-mode      app-xemacs/mmm-mode	: {,x}emacs-mmm-mode
app-emacs/mule-ucs      app-xemacs/mule-ucs	: {,x}emacs-mule-ucs
app-emacs/psgml         app-xemacs/psgml	: {,x}emacs-psgml
app-emacs/sawfish       x11-wm/sawfish		: emacs-sawfish, sawfish
app-emacs/semantic      app-xemacs/semantic	: {,x}emacs-semantic
app-emacs/sml-mode      app-xemacs/sml-mode	: {,x}emacs-sml-mode
app-emacs/speedbar      app-xemacs/speedbar	: {,x}emacs-speedbar
app-emacs/tramp         app-xemacs/tramp	: {,x}emacs-tramp
app-emacs/view-process  app-xemacs/view-process	: {,x}emacs-view-process
app-emacs/vm            app-xemacs/vm		: {,x}emacs-vm
app-emacs/w3            app-xemacs/w3		: {,x}emacs-w3
app-emacs/xslide        app-xemacs/xslide	: {,x}emacs-xslide
app-emacs/zenirc        app-xemacs/zenirc	: {,x}emacs-zenirc
app-xemacs/ocaml        dev-lang/ocaml		: xemacs-ocaml, ocaml
app-xemacs/texinfo      sys-apps/texinfo	: xemacs-texinfo, texinfo
app-xemacs/time         sys-apps/time		: xemacs-time, time
app-sci/calc            app-xemacs/calc		: calc, xemacs-calc	

app-misc/calendar  app-vim/calendar app-xemacs/calendar : calendar {vim,emacs}-calendar
app-admin/sudo          app-vim/sudo		: sudo, vim-sudo
app-arch/par            app-text/par		: parchive, par
app-forensics/memdump   sys-apps/memdump	: # these appear to be identical
app-sci/balsa           mail-client/balsa	: apt-balsa? mail-balsa?
app-text/cook           dev-util/cook		: ?
app-vim/ant             dev-java/ant		: vim-ant, ant
dev-java/crimson        games-strategy/crimson	: crimson, crimson-fields
dev-libs/ace            games-board/ace		: libace, ace
dev-libs/aterm          x11-terms/aterm		: libaterm, aterm
dev-libs/atlas          games-util/atlas	: math-atlas, atlas
dev-libs/libxml         dev-ruby/libxml		: libxml, ruby-libxml
dev-libs/localizer      net-zope/localizer	: ?
dev-lisp/lush           x11-themes/lush		: lush, kde-lush?
dev-perl/libnet         net-libs/libnet		: ?
dev-python/generator    games-emulation/generator	: py-generator, generator
dev-python/medusa       gnome-extra/medusa	: py-medusa, medusa ?
dev-python/readline     sys-libs/readline	: py-readline, readline ?
games-mud/crystal       x11-themes/crystal	: crystal, kde-crystal
net-print/omni          sys-devel/omni		: omniprint, omni

-- 
\/   Georgi Georgiev   \/ Cobol programmers are down in the dumps.     \/
/\    chutz@gg3.net    /\                                              /\
\/  +81(90)6266-1163   \/                                              \/

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Unique package names

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1121 bytes --]

On Mon, 4 Oct 2004 21:40:26 +0900 Georgi Georgiev <chutz@gg3.net> wrote:
| maillog: 04/10/2004-00:16:10(-0700): George Shapovalov types
| > However that may not be optimal as this will disallow duplicate
| > package names (which we *have* irrespectively of whether this is
| > good or bad). 
| 
| Isn't it about time to do something about these duplicate package
| names? Why not get rid of them, and then think of reorganizing the
| tree.

Great idea! I propose we rename every single package so that it is
prefixed by the category. So, instead of, say, dev-lang/python, we'd
have dev-lang/dev-lang-python. That way we wouldn't get any duplicates!

(Since at least one person is going to miss the sarcasm, a large part of
why we have categories is so that duplicate names isn't a problem. Since
portage now *tells* you when you need to be more specific, duplicates
really aren't a problem. Sticking prefixies on the front of package
names is just plain stupid.)

-- 
Ciaran McCreesh : Gentoo Developer (Sparc, MIPS, Vim, Fluxbox)
Mail            : ciaranm at gentoo.org
Web             : http://dev.gentoo.org/~ciaranm


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Unique package names

[Mike Frysinger]

On Monday 04 October 2004 08:40 am, Georgi Georgiev wrote:
> Isn't it about time to do something about these duplicate package names?
> Why not get rid of them, and then think of reorganizing the tree.

no
-mike

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Unique package names

[Donnie Berkholz]

[-- Attachment #1: Type: text/plain, Size: 544 bytes --]

On Mon, 2004-10-04 at 05:40, Georgi Georgiev wrote:
> maillog: 04/10/2004-00:16:10(-0700): George Shapovalov types
> > However that may not be optimal as this will disallow duplicate
> > package names (which we *have* irrespectively of whether this is good
> > or bad). 
> 
> Isn't it about time to do something about these duplicate package names?
> Why not get rid of them, and then think of reorganizing the tree.

If you want the names changed, get upstream to change them. We'll follow
suit.
-- 
Donnie Berkholz
Gentoo Linux

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Portage Subcategory Capabilities?

[Luke-Jr]

[-- Attachment #1: Type: text/plain, Size: 2147 bytes --]

On Monday 04 October 2004 5:34 am, Andrew Gaffney wrote:
> Georgi Georgiev wrote:
> > maillog: 04/10/2004-03:55:13(+0900): Chris White types
> > <snip>
> >
> >>Something like this might be a little nicer:
> >>
> >>media-sound/editors/audacity
> >>media-sound/players/xmms
> >>
> >>now let's say players gets way to huge:
> >>
> >>media-sound/players/mp3/
> >>media-sound/players/ogg/
> >
> > I personally like the idea. What about turning the first dash into a
> > slash as well?
> >
> > media/sound/editors/audacity
> > media/sound/players/xmms
> > media/sound/players/mp3/
> > media/sound/players/ogg/
>
> This has been brought up before in the past and been shot down, but I'll
> humor you. If support for this were to be added into Portage, there would
> be a few things to think about:
>
> 1) this will cause a performance hit no matter how it is done

What's this I've heard about Portage's performance not being a real issue? 
(argument for not rewriting it in C/etc)

> 2) how will Portage know the difference between a package and another
>     sub-category when it is walking the tree? It could do something like
> walking all the way to the end of the category/sub-category/sub-category
> tree until it finds .ebuild files and then backing up a level or 2 to
> determine the category, but again this would cause an enormous performance
> hit due to the additional required I/O.

Why not assume all [sub-]categories can also be meta-packages. Why emerge 
kde-base/kde to get all of kde-base/* when you can just emerge kde-base 
itself?

> 3) the many additional directories would cause an 'emerge sync' to take
>     longer than it does now.

Then perhaps rsync isn't the proper program for updating the tree. Or maybe we 
should finally implement fetch-on-demand...

>
> Basically, you'd be pissing everyone off with little benefit (except maybe
> to developer sanity). Keep in mind that I probably have absolutely no idea
> what I'm talking about and nothing I say should be accepted as fact without
> verifying it elsewhere.
-- 
Luke-Jr
Developer, Utopios
http://utopios.org/

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

[gentoo-dev] Subcategores and a C comment

[Nicholas Jones]

[-- Attachment #1: Type: text/plain, Size: 2681 bytes --]

> What's this I've heard about Portage's performance not being a
> real issue? (argument for not rewriting it in C/etc)

(Directed at individuals that persist in making such commentary:)

STOP SUGGESTING THAT until you know why it's rejected. Tree walking
will, at no reasonable level, be faster. DiskIO is a bound and there
is no direct way to make it faster in C, asm, or python. The only
realistic benefit of C would be algorithmic... and there is no problem
there except for searches. Any benefits beyond this incur a MASSIVE
testing and QA requirement because of the lower-level code involved.
You would not see nearly the changes and quickness in fixes if portage
were in C and I highly doubt anyone wants to diagnose segfaults on
a users box. Copy and pasting exceptions reduces debugging time by
leaps and bounds. 'Please run gdb on portage after recompiling your
system with frame pointers and debug information.' </rant>

Short version: C is my prefered language, what I code in at work,
and I think it's completely wrong for portage and wide deployment
of an actively changing system such as portage. It's more important
for portage to work than to be blazingly fast.

Any WELL THOUGHT argument or research should be taken to the
gentoo-portage-dev mailing list.

>> 2) how will Portage know the difference between a package
>> and another sub-category when it is walking the tree?

Categories file is easy enough to employ.

>> 3) the many additional directories would cause an 'emerge sync'
>> to take longer than it does now.

It would, but not as much as the thousands of ebuilds and
supporting files that get added to daily.

> should finally implement fetch-on-demand...

So instead of a sync connection and update, we can trade off a
complete tree for 1-to-N connections, comparisons, and fetches.

I'd recommend passing this through infra to make sure that
causing an extra N-times the connections is reasonable.

Also, you potentially cause the mirror to reject you as there
could be a great deal more competition on any given mirror.
Portage would be forced to add logic to repeatedly poll for a
free connection once it has started. If you get stuck behind
Dialup users, you could very well be there a while. Basically,
it can cause a race condition against servers.

>> Basically, you'd be pissing everyone off with little benefit
>> (except maybe to developer sanity).

It's not that bad really. But portage internals make a lot of
assumptions about the directory structure which is why we have
yet to make any attempt at this yet. It should become feasable
some time after 2.0.51 becomes stable and we branch for the
ripping apart of the internals.

--NJ



[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Subcategores and a C comment

[Luke-Jr]

[-- Attachment #1: Type: text/plain, Size: 1640 bytes --]

On Monday 04 October 2004 3:58 pm, Nicholas Jones wrote:
> > What's this I've heard about Portage's performance not being a
> > real issue? (argument for not rewriting it in C/etc)
>
> Short version: C is my prefered language, what I code in at work,
> and I think it's completely wrong for portage and wide deployment
> of an actively changing system such as portage. It's more important
> for portage to work than to be blazingly fast.
>

I wasn't arguing that Portage should be in C. I was just pointing out that 
since there's no real performance gains from such a conversion, it's not 
likely that changing the directory structure would hurt performance either.

> > should finally implement fetch-on-demand...
>
> So instead of a sync connection and update, we can trade off a
> complete tree for 1-to-N connections, comparisons, and fetches.
>
> I'd recommend passing this through infra to make sure that
> causing an extra N-times the connections is reasonable.
>
> Also, you potentially cause the mirror to reject you as there
> could be a great deal more competition on any given mirror.
> Portage would be forced to add logic to repeatedly poll for a
> free connection once it has started. If you get stuck behind
> Dialup users, you could very well be there a while. Basically,
> it can cause a race condition against servers.

It seems like you're assuming that only one file can be fetched per 
connection. I haven't researched the topic much, but I'm pretty sure HTTP, 
not to mention FTP, supports multiple requests with one connection.
-- 
Luke-Jr
Developer, Utopios
http://utopios.org/

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Subcategores and a C comment

[Nicholas Jones]

[-- Attachment #1: Type: text/plain, Size: 666 bytes --]

> It seems like you're assuming that only one file can be fetched per 
> connection. I haven't researched the topic much, but I'm pretty sure HTTP, 
> not to mention FTP, supports multiple requests with one connection.

How do you handle changing depends? You cannot safely assume that
a single download of all current relevant directories will be
complete. Thus... You have to connect and download multiple times.

HTTP/FTP would add in complexity to portage's update procedure.
Now we'd have to ensure that our information is correct. wget
could do this with timestamps, but then you lose the forcable
checking of the contents by checksum.

--NJ



[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Subcategores and a C comment

[Luke-Jr]

[-- Attachment #1: Type: text/plain, Size: 1100 bytes --]

On Monday 04 October 2004 6:00 pm, Nicholas Jones wrote:
> > It seems like you're assuming that only one file can be fetched per
> > connection. I haven't researched the topic much, but I'm pretty sure
> > HTTP, not to mention FTP, supports multiple requests with one connection.
>
> How do you handle changing depends? You cannot safely assume that
> a single download of all current relevant directories will be
> complete. Thus... You have to connect and download multiple times.

Not neccesarilly... Open connection; fetch pkgiwant; check deps; fetch 
depineed; no more deps: Close connection
Determining dependencies should be fast enough to keep the connection open 
for.

>
> HTTP/FTP would add in complexity to portage's update procedure.
> Now we'd have to ensure that our information is correct. wget
> could do this with timestamps, but then you lose the forcable
> checking of the contents by checksum.

How so? Fetch checksums too.
And the purpose of such an ability would be to make sync-updates unneccesary.
-- 
Luke-Jr
Developer, Utopios
http://utopios.org/

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Subcategores and a C comment

[Nicholas Jones]

[-- Attachment #1: Type: text/plain, Size: 1500 bytes --]

> Not neccesarilly... Open connection; fetch pkgiwant; check deps;
> fetch depineed; no more deps: Close connection
> Determining dependencies should be fast enough to keep the
> connection open for.

Sure... But would only require internal handling of web/ftp
protocols and rewriting emerge and/or doing single-dep parsing
for each entry... AND now that you're removed wget, you have to
maintain the file-states manually. That's a LOT of code with
LOTS of room for bugs. If someone feels up to handling all of
this, they can feel free to try and get it right.

It's a matter of sanity. From a programming and support
aspect, I'm not interested in non-consistent states. I
highly doubt this would be supported by dev-portage in any
reasonably near future as there are more important things
to be concerned with. It might also introduce difficulties
in modifying the structure of the tree later.

> How so? Fetch checksums too. And the purpose of such an
> ability would be to make sync-updates unneccesary.

Data gets corrupted.

Checksums of all data in the tree would be unnecessary
duplication of the existing system's capabilities. Why change
if you're simply going to reimplement everything?

GPG will introduce some of the at-sync checking of digests and
Manifests, but you _really_ don't want to experience the full
cost of verifying every file like this. It's already painful
with just manifest checking. (Hopefully, we'll get that sped
up though. Might have to write a lib for gpg.)

--NJ



[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Subcategores and a C comment

[Paul de Vrieze]

[-- Attachment #1: Type: text/plain, Size: 576 bytes --]

On Monday 04 October 2004 17:58, Nicholas Jones wrote:
>
> It would, but not as much as the thousands of ebuilds and
> supporting files that get added to daily.

To this respect we might one time implement some setup where you get a 
full description that includes an ebuild,a digest and all tree-local 
pathes. The only main thing that would be needed is to register in the 
ebuild which files from the tree it uses. Having such a list makes sense 
anyway

Paul

-- 
Paul de Vrieze
Gentoo Developer
Mail: pauldv@gentoo.org
Homepage: http://www.devrieze.net

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]