[gentoo-dev] rsync speed and space taken

[gentoo-dev] rsync speed and space taken

[Bret Towe]

while waiting for a rsync on one of my computers i noticed
there are alot of patchs, gzip, and bzip2 files in portage 
alot of which i wouldnt think needs to be there
x11-base/xfree/files/ for one example
sys-devel/gcc/files/ for another
im sure there are other places that are just as bad

find /mdhd/gentoo/gentoo-portage/ -name "*.bz2"|wc -l
159

find /mdhd/gentoo/gentoo-portage/ -name "*.gz"|wc -l
62

even with my local lans rsync mirror serving i think 6 computers currently
i dont even use half of those bz2 or gz files and im sure similar goes for
alot of the patchs in the tree

i seem to recall there is a script or somethin, or was, that checked sizes
of files in the tree perhaps it should be expanded to file types
or better yet watch file counts for files that arent needed
so whatever isnt a ebuild, digest, manifest, or changelog
considering how long it takes to scan the tree as it is
i would think this would help alot 
and im sure the dialup users wouldnt mind not downloading
stuff they would never use

Bret

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] rsync speed and space taken

[Roman Gaufman]

Its not only about dialup users either. For example if running gentoo
on a production system, you would normally run glsa-check on an hourly
basis. If your system is volnerable, you'll normally have to emerge
sync first, and with the number of files there, it takes far too long.

There has been a lot of discussion about switching away from rsync
and/or changing ebuild structure for portage-ng -- but thats mostly
gossip and any progress is halted.

I do agree that some files really should be removed from the tree,
look at this list:

find ./ -type f | egrep -v "packages|distfiles" |xargs du | sort -n | tail -n 20
72      ./media-video/nvidia-kernel/files/1.0.4499/NVIDIA_kernel-1.0-4499-2.6-20031014.diff
72      ./net-www/apache/files/httpd-2.0.49-ssl_engine_kernel.patch
72      ./net-www/apache/files/patches/2.0.49-r1/01_ssl_engine_kernel.patch
72      ./sys-devel/binutils/files/2.13/binutils-2.13.90.0.18-ppc64-tls1.patch
76      ./app-accessibility/speech-tools/files/1.2.3-gcc3.4.patch
76      ./sys-devel/gcc/ChangeLog
80      ./media-video/nvidia-kernel/files/1.0.5328/NVIDIA_kernel-1.0-5328-2.6-20031226.diff
80      ./sys-devel/binutils/files/2.14/binutils-2.14.90.0.4-cfi.patch
84      ./gnome-base/gnome-session/files/gentoo-splash.png
84      ./gnome-base/gnome-session/files/gnome-splash.png
84      ./media-video/nvidia-kernel/files/1.0.4363/NVIDIA_kernel-1.0-4363-2.5-20030714.diff
84      ./media-video/nvidia-kernel/files/1.0.4496/NVIDIA_kernel-1.0-4496-2.6-20030905.diff
84      ./media-video/nvidia-kernel/files/1.0.4496/NVIDIA_kernel-1.0-4496-2.6-20031026.diff
84      ./x11-base/xfree/ChangeLog
88      ./mail-filter/amavisd-new/files/amavisd.conf
92      ./app-emulation/wine/files/wine-alsa.patch
92      ./games-emulation/nwwine/files/wine-alsa.patch
100     ./media-video/nvidia-kernel/files/1.0.5328/NVIDIA_kernel-1.0-5328-2.6-20040105.diff
120     ./sys-libs/pam/files/pam-0.75-r7-gentoo.tbz2
168     ./licenses/perforce.pdf


On Sun, 10 Oct 2004 10:38:59 -0700, Bret Towe <magnade@gmail.com> wrote:
> while waiting for a rsync on one of my computers i noticed
> there are alot of patchs, gzip, and bzip2 files in portage
> alot of which i wouldnt think needs to be there
> x11-base/xfree/files/ for one example
> sys-devel/gcc/files/ for another
> im sure there are other places that are just as bad
> 
> find /mdhd/gentoo/gentoo-portage/ -name "*.bz2"|wc -l
> 159
> 
> find /mdhd/gentoo/gentoo-portage/ -name "*.gz"|wc -l
> 62
> 
> even with my local lans rsync mirror serving i think 6 computers currently
> i dont even use half of those bz2 or gz files and im sure similar goes for
> alot of the patchs in the tree
> 
> i seem to recall there is a script or somethin, or was, that checked sizes
> of files in the tree perhaps it should be expanded to file types
> or better yet watch file counts for files that arent needed
> so whatever isnt a ebuild, digest, manifest, or changelog
> considering how long it takes to scan the tree as it is
> i would think this would help alot
> and im sure the dialup users wouldnt mind not downloading
> stuff they would never use
> 
> Bret
> 
> --
> gentoo-dev@gentoo.org mailing list
> 
>

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] rsync speed and space taken

[Marius Mauch]

[-- Attachment #1: Type: text/plain, Size: 640 bytes --]

On 10/10/04  Roman Gaufman wrote:

> Its not only about dialup users either. For example if running gentoo
> on a production system, you would normally run glsa-check on an hourly
> basis. If your system is volnerable, you'll normally have to emerge
> sync first, and with the number of files there, it takes far too long.

That's pointless: glsa-check gets the information from the tree, so you
only have to run it after you've run `emerge --sync`. There are plans to
extend it to use the GLSAs on the security website but that isn't
implemented yet (and, as you have pointed out, would require you to run
`emerge --sync` anyway).

Marius

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] rsync speed and space taken

[Allen Parker]

<calm rant>
why is it that developers can't spend an extra moment or 2 tar/bz2'ing
their patchsets and uploading them to the mirrors instead of including
them where they are and making the portage tree bulge from the seams?
there are patchsets that are currently in <pkgname>/files for packages
that are only useful on a graphical workstation... for people that run
servers, this seems like quite a bit of cruft. there are currently
100,712 files in /usr/portage... does this seem a little ridiculous to
anyone else?
</calm rant>

<angry rant>
why don't we put more stuff in mirror://gentoo/ ?? ${FILESDIR} has too
much SHIT in it... i appreciate that patches, etc have been placed
there... but we REALLY need to find a better place. I don't need my
emerge sync's taking 45 minutes on average on a 5Mbit line in
france... *45 minutes* ...just PLEASE keep the SHIT out of the portage
tree.

SHIT: (noun);
1. useless stuff that has no business being in my portage tree.
2. stuff that i'll never have use for... see #1
3. anything related to X11, kde, gnome, etc on a server.
</angry rant>

that's my 2/100ths of a monetary unit.

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] rsync speed and space taken

[Brian Jackson]

A. CALM DOWN
B. One of the main reasons some devs like to push the limits of what can be
in files/ is because it takes up to 8 hours for things to get to mirrors
sometimes.
C. There are some things that are going to be in the portage tree that are
large that not everyone needs. That's just the way it is. No, we don't like
it. No, there isn't much we can do about it.
D. Most devs know it's a problem. Most devs are also already overworked as
is.
E. If you really want to do some good, write a patch for repoman to kick
people in the "jimmy jammy" when they try to commit big files.
F. CALM DOWN


On 11:49:34 pm 10/10/04 Allen Parker <infowolfe@gmail.com> wrote:
>
> >  <calm rant>
> >  why is it that developers can't spend an extra moment or 2
> >  tar/bz2'ing their patchsets and uploading them to the mirrors
> >  instead of including them where they are and making the portage
> >  tree bulge from the seams? there are patchsets that are currently
> >  in <pkgname>/files for packages that are only useful on a
> >  graphical workstation... for people that run servers, this seems
> >  like quite a bit of cruft. there are currently 100,712 files in
> >  /usr/portage... does this seem a little ridiculous to anyone else?
> >  </calm rant>
> >
> >  <angry rant>
> >  why don't we put more stuff in mirror://gentoo/ ?? ${FILESDIR} has
> >  too much SHIT in it... i appreciate that patches, etc have been
> >  placed there... but we REALLY need to find a better place. I don't
> >  need my emerge sync's taking 45 minutes on average on a 5Mbit line
> >  in france... *45 minutes* ...just PLEASE keep the SHIT out of the
> >  portage tree.
> >
> >  SHIT: (noun);
> >  1. useless stuff that has no business being in my portage tree.
> >  2. stuff that i'll never have use for... see #1
> >  3. anything related to X11, kde, gnome, etc on a server.
> >  </angry rant>
> >
> >  that's my 2/100ths of a monetary unit.
> >
> >  --
> >  gentoo-dev@gentoo.org mailing list
> >
>


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] rsync speed and space taken

[Jason Stubbs]

On Monday 11 October 2004 14:19, Brian Jackson wrote:
> A. CALM DOWN
> B. One of the main reasons some devs like to push the limits of what can be
> in files/ is because it takes up to 8 hours for things to get to mirrors
> sometimes.
> C. There are some things that are going to be in the portage tree that are
> large that not everyone needs. That's just the way it is. No, we don't like
> it. No, there isn't much we can do about it.
> D. Most devs know it's a problem. Most devs are also already overworked as
> is.
> E. If you really want to do some good, write a patch for repoman to kick
> people in the "jimmy jammy" when they try to commit big files.
> F. CALM DOWN

G. RSYNC_EXCLUDEFROM

Regards,
Jason Stubbs

--
gentoo-dev@gentoo.org mailing list

[gentoo-dev] Re: rsync speed and space taken

[Ben XO]

Jason Stubbs <jstubbs <at> gentoo.org> writes:

> G. RSYNC_EXCLUDEFROM
> 

Just a small, possibly stupid suggestion for the future of Portage: would it or
should it be possible to make portage obey USE flags?

for example, putting "-X" in your USE could prevent the synchronisation of
portage/x11-* , portage/kde-* , portage/gnome-* ... 

-- 
Ben XO :: dogsonacid.com




--
gentoo-dev@gentoo.org mailing list

[gentoo-dev] Re: rsync speed and space taken

[Duncan]

Ben XO posted <loom.20041019T023042-535@post.gmane.org>, excerpted below, 
on Tue, 19 Oct 2004 00:34:18 +0000:

> Jason Stubbs <jstubbs <at> gentoo.org> writes:
> 
>> G. RSYNC_EXCLUDEFROM
>> 
>> 
> Just a small, possibly stupid suggestion for the future of Portage: would
> it or should it be possible to make portage obey USE flags?
> 
> for example, putting "-X" in your USE could prevent the synchronisation of
> portage/x11-* , portage/kde-* , portage/gnome-* ...

NO!  Use flags are NOT DEPEND flags.  USE=-X does NOT mean that X won't be
emerged, if a package requiring it is emerged.  What it DOES mean is that
packages that have optional features that can be used with X, can be tied
to that X use flag, and will NOT be enabled, if USE=-X.

Thus, USE=-X means for example that links will be built without its X
support (said support /does/ require X libraries be installed and linkable
at runtime, as I found out the hard way, when I was having problems with
X), because it uses the X flag and -X turns off that support for the links
ebuilds.  However, it does NOT mean that KDE won't be installed if you
require it, or that it won't install X because it requires it.

Thus, preventing X from syncing based on the -X use flag is
counter-purpose to what the use flags are for.

-- 
Duncan - List replies preferred.   No HTML msgs.
"They that can give up essential liberty to obtain a little
temporary safety, deserve neither liberty nor safety." --
Benjamin Franklin



--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] rsync speed and space taken

[Ned Ludd]

[-- Attachment #1: Type: text/plain, Size: 763 bytes --]

On Mon, 2004-10-11 at 01:19, Brian Jackson wrote:
[snip]

> E. If you really want to do some good, write a patch for repoman to kick
> people in the "jimmy jammy" when they try to commit big files.

A few revisions back ferringb added support for this per a request.
repoman --pretend scan  shows file sizes. (check it out in the binutils
dir) It's non fatal and will probably remain that will till >=2.0.52 but
after then no patch >=20k can go in the tree and we wont be able to
commit on a dir unless it's cleaned up.

Perhaps we should also display the size totals for the $PWD on a commit.

[snip]

> gentoo-dev@gentoo.org mailing list
-- 
Ned Ludd <solar@gentoo.org>
Gentoo (hardened,security,infrastructure,embedded,toolchain) Developer

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] rsync speed and space taken

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 938 bytes --]

On Mon, 11 Oct 2004 12:14:07 -0400 Ned Ludd <solar@gentoo.org> wrote:
| > E. If you really want to do some good, write a patch for repoman to
| > kick people in the "jimmy jammy" when they try to commit big files.
| 
| A few revisions back ferringb added support for this per a request.
| repoman --pretend scan  shows file sizes. (check it out in the
| binutils dir) It's non fatal and will probably remain that will till
| >=2.0.52 but
| after then no patch >=20k can go in the tree and we wont be able to
| commit on a dir unless it's cleaned up.

That's going to get *really* painful when we're trying to just keyword a
package without going around and changing other people's ebuilds. This
shouldn't be introduced as an error until after the existing tree is
entirely fixed.

-- 
Ciaran McCreesh : Gentoo Developer (Vim, Fluxbox, Sparc, Mips)
Mail            : ciaranm at gentoo.org
Web             : http://dev.gentoo.org/~ciaranm


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] rsync speed and space taken

[Donnie Berkholz]

[-- Attachment #1: Type: text/plain, Size: 393 bytes --]

On Mon, 2004-10-11 at 17:17 +0100, Ciaran McCreesh wrote:
> That's going to get *really* painful when we're trying to just keyword a
> package without going around and changing other people's ebuilds. This
> shouldn't be introduced as an error until after the existing tree is
> entirely fixed.

I doubt the tree will get fixed until it is an error.
-- 
Donnie Berkholz
Gentoo Linux

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] rsync speed and space taken

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 800 bytes --]

On Mon, 11 Oct 2004 09:25:19 -0700 Donnie Berkholz
<spyderous@gentoo.org> wrote:
| On Mon, 2004-10-11 at 17:17 +0100, Ciaran McCreesh wrote:
| > That's going to get *really* painful when we're trying to just
| > keyword a package without going around and changing other people's
| > ebuilds. This shouldn't be introduced as an error until after the
| > existing tree is entirely fixed.
| 
| I doubt the tree will get fixed until it is an error.

Well, if it *doesn't*, then I guess anyone who touches other people's
ebuilds for, for example, keywording will have to just modify repoman...
I'll post a patch if this turns out to be a problem.

-- 
Ciaran McCreesh : Gentoo Developer (Vim, Fluxbox, Sparc, Mips)
Mail            : ciaranm at gentoo.org
Web             : http://dev.gentoo.org/~ciaranm


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] rsync speed and space taken

[Doug Goldstein]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ciaran McCreesh wrote:
| On Mon, 11 Oct 2004 09:25:19 -0700 Donnie Berkholz
| <spyderous@gentoo.org> wrote:
| | On Mon, 2004-10-11 at 17:17 +0100, Ciaran McCreesh wrote:
| | > That's going to get *really* painful when we're trying to just
| | > keyword a package without going around and changing other people's
| | > ebuilds. This shouldn't be introduced as an error until after the
| | > existing tree is entirely fixed.
| |
| | I doubt the tree will get fixed until it is an error.
|
| Well, if it *doesn't*, then I guess anyone who touches other people's
| ebuilds for, for example, keywording will have to just modify repoman...
| I'll post a patch if this turns out to be a problem.
|
Or we can have 1 person just go through and fix everyone's ebuilds on
this issue. There's only a hundred or so.

- --
Doug Goldstein
http://dev.gentoo.org/~cardoe

Public Key: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x179106D0
Key fingerprint = 7001 5FBF BACE 9E66 3A1C  55E0 161C FF5C 1791 06D0

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBatI1Fhz/XBeRBtARAluwAJ9vpt3klyoA2guItEK5a1bei1H2GACfWXkM
dYlumwwErQcWPGKfzRtVLik=
=LGsp
-----END PGP SIGNATURE-----

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] rsync speed and space taken

[Paul de Vrieze]

[-- Attachment #1: Type: text/plain, Size: 888 bytes --]

On Monday 11 October 2004 18:14, Ned Ludd wrote:
> On Mon, 2004-10-11 at 01:19, Brian Jackson wrote:
> [snip]
>
> > E. If you really want to do some good, write a patch for repoman to kick
> > people in the "jimmy jammy" when they try to commit big files.
>
> A few revisions back ferringb added support for this per a request.
> repoman --pretend scan  shows file sizes. (check it out in the binutils
> dir) It's non fatal and will probably remain that will till >=2.0.52 but
> after then no patch >=20k can go in the tree and we wont be able to
> commit on a dir unless it's cleaned up.

Please also add a check for bzipped gzipped files. CVS doesn't play nice with 
them, and neither does rsync. Those shouldn't be in the tree (except for the 
rescue portage)

Paul

-- 
Paul de Vrieze
Gentoo Developer
Mail: pauldv@gentoo.org
Homepage: http://www.devrieze.net

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] rsync speed and space taken

[Eldad Zack]

[-- Attachment #1: Type: text/plain, Size: 529 bytes --]

On Mon, 2004-10-11 at 19:46, Paul de Vrieze wrote:
> On Monday 11 October 2004 18:14, Ned Ludd wrote:
> > On Mon, 2004-10-11 at 01:19, Brian Jackson wrote:
> Please also add a check for bzipped gzipped files. CVS doesn't play nice with 
> them, and neither does rsync. Those shouldn't be in the tree (except for the 
> rescue portage)

Why should rescue portage be in the tree?
anyway, it's gone. but it might be nice to have it as an ebuild, and
slip it in as a system package.


-- 
Eldad Zack <eldad@gentoo.org>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] rsync speed and space taken

[Marius Mauch]

[-- Attachment #1: Type: text/plain, Size: 704 bytes --]

On 10/12/04  Eldad Zack wrote:

> On Mon, 2004-10-11 at 19:46, Paul de Vrieze wrote:
> > On Monday 11 October 2004 18:14, Ned Ludd wrote:
> > > On Mon, 2004-10-11 at 01:19, Brian Jackson wrote:
> > Please also add a check for bzipped gzipped files. CVS doesn't play
> > nice with them, and neither does rsync. Those shouldn't be in the
> > tree (except for the rescue portage)
> 
> Why should rescue portage be in the tree?
> anyway, it's gone. but it might be nice to have it as an ebuild, and
> slip it in as a system package.

What would be the point of that? The rescue portage is only a
pre-packaged version of portage in cases of a corrupted portage install
where you can't use it anymore.

Marius

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] rsync speed and space taken

[Christian Birchinger]

On Mon, Oct 11, 2004 at 12:14:07PM -0400, Ned Ludd wrote:
> 
> A few revisions back ferringb added support for this per a request.
> repoman --pretend scan  shows file sizes. (check it out in the binutils
> dir) It's non fatal and will probably remain that will till >=2.0.52 but
> after then no patch >=20k can go in the tree and we wont be able to
> commit on a dir unless it's cleaned up.
> 

That would really be a pain for arch maintainers who just want
to keyword the ebuild or something similar.

Ofcourse you can first go and find the dev and harass him, but
it's really annoying if you're in the middle of keywording.

I'd say simply refuse to add new files which are too large.


Christian

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] rsync speed and space taken

[Allen Parker]

For the person asking, yes, the machine is I/O limited... cel 2.4 128M
ram 40G disk and LOTS of blog sites :(

~45 minutes nonetheless is a *long* time to wait for an emerge sync to
complete (or glsa-check for that matter). I second the suggestion of a
seperate portion of the rsync mirror SPECIFICALLY for patches. perhaps
with soft/hard-links from ${PN}/${FILESDIR} to ${PATCHDIR}/{$PN} ?
shouldn't be too much harder to just grab what is needed on-use...
carpaski? any ideas? perhaps a feature request for .52?

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] rsync speed and space taken

[Philippe Trottier]

Allen Parker wrote:

>For the person asking, yes, the machine is I/O limited... cel 2.4 128M
>ram 40G disk and LOTS of blog sites :(
>  
>
Why not just having a directory / index and base the download / rsync
on what is needed ?

ls -1R /usr/portage is about 2.1MB ...

Then just fetch the (to be)installed ebuilds on a needed basis ?

>~45 minutes nonetheless is a *long* time to wait for an emerge sync to
>complete (or glsa-check for that matter). I second the suggestion of a
>seperate portion of the rsync mirror SPECIFICALLY for patches. perhaps
>with soft/hard-links from ${PN}/${FILESDIR} to ${PATCHDIR}/{$PN} ?
>shouldn't be too much harder to just grab what is needed on-use...
>carpaski? any ideas? perhaps a feature request for .52?
>  
>

glsa should have it's own server/client system.

Other thing is, for production, no one NEEDS to upgrade stuff every 
single day, if it works, it works. If you are a developer then you 
probably already know how to use CVS, PORTAGE_OVERLAY etc etc and you 
can put your emerge sync in a cron job at 3:00 in the morning...

For sysadmins, it is anyway a good idea to follow the glsa, I put it on 
my startup page of my browser...

Phil

--
gentoo-dev@gentoo.org mailing list

[gentoo-dev] Re: rsync speed and space taken

[Tim Cera]

Philippe Trottier <tchiwam <at> gentoo.org> writes: 
> Why not just having a directory / index and base the download / rsync 
> on what is needed ? 
 
I submitted a patch (a long time ago) that only rsynced the portage tree for 
the installed ebuilds.  Worked great.  Would have to be extended to fetch new 
ebuilds when USE variables or dependencies change. 
 
http://bugs.gentoo.org/show_bug.cgi?id=44526 
 
A really cool benefit would be that you could collect what people installed 
from the rsync server logs. 
 
take care 
tim 
 


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: rsync speed and space taken

[Andrew Fant]

--On Wednesday, October 20, 2004 01:19:28 +0000 Tim Cera 
<timcera@earthlink.net> wrote:

> Philippe Trottier <tchiwam <at> gentoo.org> writes:
>> Why not just having a directory / index and base the download / rsync
>> on what is needed ?
>
> I submitted a patch (a long time ago) that only rsynced the portage tree
> for  the installed ebuilds.  Worked great.  Would have to be extended to
> fetch new  ebuilds when USE variables or dependencies change.
>
> http://bugs.gentoo.org/show_bug.cgi?id=44526
>
> A really cool benefit would be that you could collect what people
> installed  from the rsync server logs.

I'm not sure that I would call that a cool benefit.  It seems to come close 
to an egregious violation of privacy.   I know that there is no promise of 
confidentiality in the use of the portage rsync servers, but to actively 
and publicly start collecting data about who is using what seems to only 
invite more paranoia.

Andy
(JFMuggs)



--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: rsync speed and space taken

[Luke-Jr]

[-- Attachment #1: Type: text/plain, Size: 614 bytes --]

On Friday 22 October 2004 9:56 pm, Andrew Fant wrote:
> I'm not sure that I would call that a cool benefit.  It seems to come close
> to an egregious violation of privacy.   I know that there is no promise of
> confidentiality in the use of the portage rsync servers, but to actively
> and publicly start collecting data about who is using what seems to only
> invite more paranoia.
Except that it won't be able to reliably collect the "who" part, only the 
"what". Sure, you could log IPs, but many users IPs change more frequently 
than they sync.
-- 
Luke-Jr
Developer, Utopios
http://utopios.org/

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Re: rsync speed and space taken

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1057 bytes --]

On Fri, 22 Oct 2004 22:00:55 +0000 Luke-Jr <luke-jr@utopios.org> wrote:
| On Friday 22 October 2004 9:56 pm, Andrew Fant wrote:
| > I'm not sure that I would call that a cool benefit.  It seems to
| > come close to an egregious violation of privacy.   I know that there
| > is no promise of confidentiality in the use of the portage rsync
| > servers, but to actively and publicly start collecting data about
| > who is using what seems to only invite more paranoia.
| Except that it won't be able to reliably collect the "who" part, only
| the "what". Sure, you could log IPs, but many users IPs change more
| frequently than they sync.

Naah. You've got more than enough data to figure out who's doing what,
even if you don't look at IP at all. You know roughly when someone last
synced and you can track by what packages they previously had installed.
There's a looooot of information in that...

-- 
Ciaran McCreesh : Gentoo Developer (Vim, Fluxbox, Sparc, Mips)
Mail            : ciaranm at gentoo.org
Web             : http://dev.gentoo.org/~ciaranm


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

[gentoo-dev] Re: rsync speed and space taken

[Mike]

On Fri, 22 Oct 2004 22:00:55 +0000, Luke-Jr <luke-jr@utopios.org> wrote:


> On Friday 22 October 2004 9:56 pm, Andrew Fant wrote:
> > I'm not sure that I would call that a cool benefit.  It seems to come close
> > to an egregious violation of privacy.   I know that there is no promise of
> > confidentiality in the use of the portage rsync servers, but to actively
> > and publicly start collecting data about who is using what seems to only
> > invite more paranoia.
> Except that it won't be able to reliably collect the "who" part, only the
> "what". Sure, you could log IPs, but many users IPs change more frequently
> than they sync.

Even the what won't be reliable, as it won't show all of us who use
emerge-webrsync due to firewall restrictions.

Mike

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: rsync speed and space taken

[Luke-Jr]

[-- Attachment #1: Type: text/plain, Size: 1225 bytes --]

On Friday 22 October 2004 10:13 pm, Mike wrote:
> On Fri, 22 Oct 2004 22:00:55 +0000, Luke-Jr <luke-jr@utopios.org> wrote:
> > On Friday 22 October 2004 9:56 pm, Andrew Fant wrote:
> > > I'm not sure that I would call that a cool benefit.  It seems to come
> > > close to an egregious violation of privacy.   I know that there is no
> > > promise of confidentiality in the use of the portage rsync servers, but
> > > to actively and publicly start collecting data about who is using what
> > > seems to only invite more paranoia.
> >
> > Except that it won't be able to reliably collect the "who" part, only the
> > "what". Sure, you could log IPs, but many users IPs change more
> > frequently than they sync.
>
> Even the what won't be reliable, as it won't show all of us who use
> emerge-webrsync due to firewall restrictions.
It would be a decent sample, though. Which is a positive thing, IMO...
I recall when I was a developer I was wondering whether a pkg was ready for 
stable or if simply nobody had tried it... In that case, it was a 
not-so-common server, so I wouldn't be surprised if all the people using it 
stuck w/ stable version...
-- 
Luke-Jr
Developer, Utopios
http://utopios.org/

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Re: rsync speed and space taken

[Jason Cooper]

Luke-Jr (luke-jr@utopios.org) scribbled:
> On Friday 22 October 2004 10:13 pm, Mike wrote:
> > On Fri, 22 Oct 2004 22:00:55 +0000, Luke-Jr <luke-jr@utopios.org> wrote:
> > > On Friday 22 October 2004 9:56 pm, Andrew Fant wrote:
> > > > I'm not sure that I would call that a cool benefit.  It seems to come
> > > > close to an egregious violation of privacy.   I know that there is no
> > > > promise of confidentiality in the use of the portage rsync servers, but
> > > > to actively and publicly start collecting data about who is using what
> > > > seems to only invite more paranoia.
> > >
> > > Except that it won't be able to reliably collect the "who" part, only the
> > > "what". Sure, you could log IPs, but many users IPs change more
> > > frequently than they sync.
> >
> > Even the what won't be reliable, as it won't show all of us who use
> > emerge-webrsync due to firewall restrictions.
> It would be a decent sample, though. Which is a positive thing, IMO...
> I recall when I was a developer I was wondering whether a pkg was ready for 
> stable or if simply nobody had tried it... In that case, it was a 
> not-so-common server, so I wouldn't be surprised if all the people using it 
> stuck w/ stable version...

I'm also wary of the idea of logging what people have installed on their
machines... It's too prone to abuse.  There are other solutions to your
dilemma above; say, asking on gentoo-user ;)

I don't think the potential for abuse is worth the knowledge gain.

Cooper.

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: rsync speed and space taken

[Roman Gaufman]

How about getting only the installed ebuilds + *all* the ebuilds that
may have potential patent violation by default? :P

On Sat, 23 Oct 2004 09:29:00 -0400, Jason Cooper <gentoo@lakedaemon.net> wrote:
> Luke-Jr (luke-jr@utopios.org) scribbled:
> 
> 
> > On Friday 22 October 2004 10:13 pm, Mike wrote:
> > > On Fri, 22 Oct 2004 22:00:55 +0000, Luke-Jr <luke-jr@utopios.org> wrote:
> > > > On Friday 22 October 2004 9:56 pm, Andrew Fant wrote:
> > > > > I'm not sure that I would call that a cool benefit.  It seems to come
> > > > > close to an egregious violation of privacy.   I know that there is no
> > > > > promise of confidentiality in the use of the portage rsync servers, but
> > > > > to actively and publicly start collecting data about who is using what
> > > > > seems to only invite more paranoia.
> > > >
> > > > Except that it won't be able to reliably collect the "who" part, only the
> > > > "what". Sure, you could log IPs, but many users IPs change more
> > > > frequently than they sync.
> > >
> > > Even the what won't be reliable, as it won't show all of us who use
> > > emerge-webrsync due to firewall restrictions.
> > It would be a decent sample, though. Which is a positive thing, IMO...
> > I recall when I was a developer I was wondering whether a pkg was ready for
> > stable or if simply nobody had tried it... In that case, it was a
> > not-so-common server, so I wouldn't be surprised if all the people using it
> > stuck w/ stable version...
> 
> I'm also wary of the idea of logging what people have installed on their
> machines... It's too prone to abuse.  There are other solutions to your
> dilemma above; say, asking on gentoo-user ;)
> 
> I don't think the potential for abuse is worth the knowledge gain.
> 
> Cooper.
> 
> 
> 
> --
> gentoo-dev@gentoo.org mailing list
> 
>

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: rsync speed and space taken

[Marcus D. Hanwell]

Roman Gaufman wrote:

>How about getting only the installed ebuilds + *all* the ebuilds that
>may have potential patent violation by default? :P
>  
>
That is a terrible argument - patent violating ebuilds. The ebuilds are
not the applications, just the instructions on how to download, compile
and install on to a Gentoo system. So even if the ebuilds provide
instructions on how to download them, I really don't think anyone could
argue you infringed any patent until you downloaded, installed and used
the application.


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: rsync speed and space taken

[Roman Gaufman]

>That is a terrible argument - patent violating ebuilds. The ebuilds are
>not the applications, just the instructions on how to download, compile
>and install on to a Gentoo system. So even if the ebuilds provide
>instructions on how to download them, I really don't think anyone could
>argue you infringed any patent until you downloaded, installed and used
>the application.

ebuilds that may have *potential* patent violation. -- Even clearer,
ebuilds that install patent infrinding software.

In anycase, what I mean is ebuilds like those that (optionally) fetch w32codecs
are downloaded by default along side new versions of the installed
ebuilds -- then the collected data cannot be used against anyone but
still be useful for general analysis and takes a load of the rsync
mirrors.

I wasnt serious about the suggestion either way though, hense the
smiley, so relax!

--
gentoo-dev@gentoo.org mailing list

[gentoo-dev] Re: rsync speed and space taken

[Tim Cera]

Andrew Fant <andrew.fant <at> tufts.edu> writes:
> I'm not sure that I would call that a cool benefit.  It seems to come close 
> to an egregious violation of privacy.   I know that there is no promise of 
> confidentiality in the use of the portage rsync servers, but to actively 
> and publicly start collecting data about who is using what seems to only 
> invite more paranoia.

Let's say that limiting the rsync to installed packages has a side-effect. 
Whether it is a benefit or not depends on many factors.

I would say that the collection of installed package statistics is NOT a reason
to rsync on installed packages only.  The reasons to rsync on installed packages
is to reduce the load on the rsync servers and to make the rsync faster (rsync
speed was a definite problem on my old laptop).  When I was using rsync to run
against installed packages, I was bringing down about 15,000 files.  Note that
the total size of the portage tree is irrelevant.

I took for granted that the ONLY statistics that would be collected would be
statistics on the entire community. 
For example:
 x% of gentoo users install metalog
 y% of gentoo users install syslogd
 z% of gentoo users install syslog-ng
...etc.

I also imagined that, like in the patch I submitted, the rsync against installed
packages was an option.  The default would be a full rsync.  Just like
gentoo-stats is an option (in fact gentoo-stats sends a bunch of data, and you
can choose anonymity if you wish).

take care
tim


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: rsync speed and space taken

[Ed Grimm]

On Wed, 20 Oct 2004, Tim Cera wrote:

> I submitted a patch (a long time ago) that only rsynced the portage
> tree for the installed ebuilds.  Worked great.  Would have to be
> extended to fetch new ebuilds when USE variables or dependencies
> change.
>
> http://bugs.gentoo.org/show_bug.cgi?id=44526
>
> A really cool benefit would be that you could collect what people installed
> from the rsync server logs.

I'd also like it to be extended so that it could also sync an additional
watchlist of packages.  The main purpose of this is, "I'm planning on
installing Foo on Saturday.  I want my portage to be up to date before I
start the install, but I don't want to sync twice in 24 hours, nor do I
want to sync every package."  Of course, since one could simply sync the
full set, possibly minus a hand-crafted excludefrom list, this is a much
lesser item.

I'm impressed; it applied cleanly after all this time (offset 334
lines on the second chunk.)

Personally, I don't think that the rsync logs would be that significant.
If I did, I wouldn't contemplate using this option.  Of course, I'm
assuming a watch list, which would throw a certain amount of very needed
sand into those stats.

Ed

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] rsync speed and space taken

[Bret Towe]

On Mon, 11 Oct 2004 05:19:50 +0000, Brian Jackson <iggy@gentoo.org> wrote:
> A. CALM DOWN
> B. One of the main reasons some devs like to push the limits of what can be
> in files/ is because it takes up to 8 hours for things to get to mirrors
> sometimes.
the time it takes could be changed

> C. There are some things that are going to be in the portage tree that are
> large that not everyone needs. That's just the way it is. No, we don't like
> it. No, there isn't much we can do about it.
yes there is you just dont want to

> D. Most devs know it's a problem. Most devs are also already overworked as
> is.
yes i hear that excuse alot

> E. If you really want to do some good, write a patch for repoman to kick
> people in the "jimmy jammy" when they try to commit big files.
> F. CALM DOWN

well here is another idea for you 'overworked' devs
make a new dir on rsync called patchfiles (like distfiles) ignored by
emerge sync
add a mirror:// item support to have emerge grab patchs requested by ebuilds
from that rsync location
this would allow the devs to keep their 30m mirrors
would make rsync in general speed up for users
rsync mirrors wouldnt be out any more space than they already are
also this could allow a setup so patchs dont have to go into cvs
which i think is better if gzip/bzip items are going to be in there

> 
> On 11:49:34 pm 10/10/04 Allen Parker <infowolfe@gmail.com> wrote:
> >
> > >  <calm rant>
> > >  why is it that developers can't spend an extra moment or 2
> > >  tar/bz2'ing their patchsets and uploading them to the mirrors
> > >  instead of including them where they are and making the portage
> > >  tree bulge from the seams? there are patchsets that are currently
> > >  in <pkgname>/files for packages that are only useful on a
> > >  graphical workstation... for people that run servers, this seems
> > >  like quite a bit of cruft. there are currently 100,712 files in
> > >  /usr/portage... does this seem a little ridiculous to anyone else?
> > >  </calm rant>
> > >
> > >  <angry rant>
> > >  why don't we put more stuff in mirror://gentoo/ ?? ${FILESDIR} has
> > >  too much SHIT in it... i appreciate that patches, etc have been
> > >  placed there... but we REALLY need to find a better place. I don't
> > >  need my emerge sync's taking 45 minutes on average on a 5Mbit line
> > >  in france... *45 minutes* ...just PLEASE keep the SHIT out of the
> > >  portage tree.
> > >
> > >  SHIT: (noun);
> > >  1. useless stuff that has no business being in my portage tree.
> > >  2. stuff that i'll never have use for... see #1
> > >  3. anything related to X11, kde, gnome, etc on a server.
> > >  </angry rant>
> > >
> > >  that's my 2/100ths of a monetary unit.
> > >
> > >  --
> > >  gentoo-dev@gentoo.org mailing list
> > >
> >
> 
> 
> 
> 
> --
> gentoo-dev@gentoo.org mailing list
> 
>

--
gentoo-dev@gentoo.org mailing list

[gentoo-dev] Re: rsync speed and space taken

[Duncan]

Allen Parker posted <9f27901604101016497d40b0a@mail.gmail.com>, excerpted
below,  on Sun, 10 Oct 2004 15:49:34 -0800:

> I don't need my emerge sync's taking 45 minutes on average on a 5Mbit
> line in france... *45 minutes*

If your emerge syncs are taking *THAT* long, on a 5Mbit line, somethings
TERRIBLY WRONG!  I have a cable modem here (Phoenix, AZ, US), capped @
4Mbps, and my syncs take perhaps a couple minutes, short enough time that
I usually do it interactively -- type in the command, and watch it work,
then do an emerge -auD world to see what is there to change, after that.

FWIW, I note that much of the time, it's actually not downloading
anything, only checking stuff locally (presumably doing local md5 sums
and matching against the new ones in the initially d/led file to see if
they've changed), in the background.  Only the initial file d/l, and the
two-line entries with the percent complete and etc on the second line, are
actually transferred.  The rest of the time is spent locally, generally
CPU bound (single thread/CPU only) That being the case, I suspect the
reason it takes you 45 minutes and me substantially less than five, may be
due to local CPU processing power. I'm running a dual AMD64 Opteron (1 gig
memory, swap entirely kernel disabled so the kernel doesn't bother with it
at all) here, altho as I mentioned, portage/python doesn't appear to be
multi-threaded, so it only tops off one CPU.  If you are running something
less powerful, that'd probably explain your longer sync times even with a
fatter internet pipe.

..  About 3 and a half minutes.  I just timed it.

-- 
Duncan - List replies preferred.   No HTML msgs.
"They that can give up essential liberty to obtain a little
temporary safety, deserve neither liberty nor safety." --
Benjamin Franklin



--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: rsync speed and space taken

[Travis Tilley]

Duncan wrote:
> ..  About 3 and a half minutes.  I just timed it.

rm -rf /usr/portage and time it again.



--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: rsync speed and space taken

[Paul de Vrieze]

[-- Attachment #1: Type: text/plain, Size: 671 bytes --]

On Monday 11 October 2004 12:37, Travis Tilley wrote:
> Duncan wrote:
> > ..  About 3 and a half minutes.  I just timed it.
>
> rm -rf /usr/portage and time it again.
>

For people that have slow rsync times, there is the alternative to run 
emerge-websync. It is maximally a day behind and works well. 
Unfortunately it needs to download a lot more, but doesn't need to scan 
the whole local and remote trees.

Rsync is for a big part actually IO bound. If you are not using dma you're 
basically fucked. A good cache and enough memory helps too.

Paul

-- 
Paul de Vrieze
Gentoo Developer
Mail: pauldv@gentoo.org
Homepage: http://www.devrieze.net

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Re: rsync speed and space taken

[Xavier Neys]

[-- Attachment #1: Type: text/plain, Size: 1095 bytes --]

Paul de Vrieze wrote:
> On Monday 11 October 2004 12:37, Travis Tilley wrote:
> 
>>Duncan wrote:
>>
>>>..  About 3 and a half minutes.  I just timed it.
>>
>>rm -rf /usr/portage and time it again.
>>
> 
> 
> For people that have slow rsync times, there is the alternative to run 
> emerge-websync. It is maximally a day behind and works well. 
> Unfortunately it needs to download a lot more, but doesn't need to scan 
> the whole local and remote trees.

Actually, it will scan two trees, two *local* ones, on top of some extra I/O
FYI, webrsync does :
1) download a 16M snapshot
2) unpack its 100,000+ files
<rant>latest GWN sees it as a record, let's aim for 250,000 files</rant>
3) run rsync between temp dir and /usr/portage
4) rm 100,000 temp files
5) emerge metadata

emerge-webrsync is not meant to decrease I/O, in fact, it increases local I/O. 
It is meant for people who can't use rsync because it's blocked (or because 
there is no connection at all).


Wkr,
-- 
/  Xavier Neys
\_ Gentoo Documentation Project
/  French & Internationalisation Lead
\  http://www.gentoo.org/doc/en
/\

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 256 bytes --]

Re: [gentoo-dev] Re: rsync speed and space taken

[Francesco Riosa]

Travis Tilley wrote:

> Duncan wrote:
>
>> ..  About 3 and a half minutes.  I just timed it.
>
>
> rm -rf /usr/portage and time it again.
>
wget 
http://www.die.unipd.it/pub/Linux/distributions/gentoo-sources/snapshots/portage-20041010.tar.bz2
cd /usr ; tar -jxvf  portage-20041010.tar.bz2
emerge sync

portage-20041010.tar.bz2 is 16 megs

--
gentoo-dev@gentoo.org mailing list

[gentoo-dev] Re: Re: rsync speed and space taken

[Duncan]

Travis Tilley posted <416A6264.9020805@gentoo.org>, excerpted below,  on
Mon, 11 Oct 2004 06:37:24 -0400:

> Duncan wrote:
>> ..  About 3 and a half minutes.  I just timed it.
> 
> rm -rf /usr/portage and time it again.

Well:

#ll /usr/portage
lrwxrwxrwx  1 root root 6 Sep 26 18:02 /usr/portage -> /mnt/p/

I don't have /usr/portage, except as a symlink, for those apps that can't
follow PORTDIR=/mnt/p in /etc/make.conf, so unless emerge sync is one of
those (and let's /hope/ not), your suggestion above would have little
impact, here.

However, removing $PORTDIR is what I assume you meant, so after umounting
/mnt/p/src ($DISTDIR) and /mnt/p/pkg ($PKGDIR) so as not to lose them (I
learned my lesson when I presumed portage would know enough not to attempt
to rsync its $DISTDIR and $PKGDIR =:^(, but that's been covered in
bugzilla already), I backed up /mnt/p and deleted it, then timed an emerge
sync:

real    3m1.038s
user    0m6.512s
sys     0m25.001s

To get a better direct comparison, I then deleted it again, restored my
backup copy, and (after a few minutes lag time to prevent being called on
the carpet for syncing to quickly) did a standard emerge sync.

real    2m27.357s
user    0m30.395s
sys     0m10.033s

Two and a half minutes, that time, as compared to three minutes total
(from blank) sync, and three and a half(ish) minutes yesterday normal
sync.  Note that while I consider all three timings "reasonable", I /do/
occasionally get stuck with a sync mirror that's feeding the initial
file at speeds turning over the tens and hundreds files counter, rather
than the thousands and ten-thousands files counters, in which case I
usually ^C abort the sync and try again, for a different rsync server.   
The point is, there's enough variability at that end, that no conclusion
can be drawn due to the 30-second-ish differences in timings that I
measured.

In any case, 45 minutes on a 5Mbit connection as claimed by the upline
poster, definitely means something other than raw portage tree size or
local pipe bandwidth is the problem.  Someone mentioned it took them about
that long with a 100MHz Pentium and 128M memory, which is what I suggested
the problem may be earlier, local machine performance, not tree size or
bandwidth limitations.

-- 
Duncan - List replies preferred.   No HTML msgs.
"They that can give up essential liberty to obtain a little
temporary safety, deserve neither liberty nor safety." --
Benjamin Franklin



--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] rsync speed and space taken

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 1168 bytes --]

On Sun, 2004-10-10 at 15:49 -0800, Allen Parker wrote:
> <angry rant>
> why don't we put more stuff in mirror://gentoo/ ?? ${FILESDIR} has too
> much SHIT in it... i appreciate that patches, etc have been placed
> there... but we REALLY need to find a better place. I don't need my
> emerge sync's taking 45 minutes on average on a 5Mbit line in
> france... *45 minutes* ...just PLEASE keep the SHIT out of the portage
> tree.

While I completely agree that things need to be taken out of
${FILESDIR}, I have a question for you.  Could you give the specs of
your machine?  I have several machines and even on my oldest machine
with an ATA33 drive and 128MB of RAM, it takes nowhere near 45 minutes
to rsync the tree.  Are you using your geographic rsync mirror?  Do you
have any other I/O bound processes running at the same time?  I'm not
trying to do anything other than point out that your times seem awfully
long in my experience and there is a distinct possibility that something
else is causing the slowdown to such extremes on your system.

-- 
Chris Gianelloni
Release Engineering - Operations/QA Manager
Games - Developer
Gentoo Linux

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] rsync speed and space taken

[Xavier Neys]

[-- Attachment #1: Type: text/plain, Size: 1946 bytes --]

Chris Gianelloni wrote:
> On Sun, 2004-10-10 at 15:49 -0800, Allen Parker wrote:
> 
>><angry rant>
>>why don't we put more stuff in mirror://gentoo/ ?? ${FILESDIR} has too
>>much SHIT in it... i appreciate that patches, etc have been placed
>>there... but we REALLY need to find a better place. I don't need my
>>emerge sync's taking 45 minutes on average on a 5Mbit line in
>>france... *45 minutes* ...just PLEASE keep the SHIT out of the portage
>>tree.
> 
> 
> While I completely agree that things need to be taken out of
> ${FILESDIR}, I have a question for you.  Could you give the specs of
> your machine?  I have several machines and even on my oldest machine
> with an ATA33 drive and 128MB of RAM, it takes nowhere near 45 minutes
> to rsync the tree.  Are you using your geographic rsync mirror?  Do you
> have any other I/O bound processes running at the same time?  I'm not
> trying to do anything other than point out that your times seem awfully
> long in my experience and there is a distinct possibility that something
> else is causing the slowdown to such extremes on your system.
> 
I also have an old box (P100 w/ 128MB of RAM, 2 ATA33 disks at ~6MB/sec, udma 
not available) and it takes ~45 minutes to run emerge --sync. Bandwidth is not 
the bottleneck because its mirror is sitting next to it. The bottleneck is the 
sheer number of files to check.

FYI: /usr/portage :
16276 - directories
83779 - files including
         L_ 24820 files under /files/
         L_ 19118 files under /metadata/cache/

16314 - Digest
16284 - .ebuild
  7816 - Manifest
  7812 - ChangeLog
  6726 - metadata.xml
  5100 - .patch / .diff
   223 - Archive (gz/bz2/tar/...)
    89 - make.defaults
    78 - virtuals
    67 - parent
    61 - use.mask
    49 - use.defaults
  4042 - Uncategorized



My 2/100 of your currency,
-- 
/  Xavier Neys
\_ Gentoo Documentation Project
/  French & Internationalisation Lead
\  http://www.gentoo.org/doc/en
/\

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 256 bytes --]

Re: [gentoo-dev] rsync speed and space taken

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 901 bytes --]

On Sun, 10 Oct 2004 15:49:34 -0800 Allen Parker <infowolfe@gmail.com>
wrote:
| why is it that developers can't spend an extra moment or 2 tar/bz2'ing
| their patchsets and uploading them to the mirrors instead of including
| them where they are and making the portage tree bulge from the seams?

a) because the mirror system really really sucks for that kind of thing,
so we tend to end up with dozens of duplicate bugs whining about patches
being missing because of the huge upload delay

b) because the mirror system really really sucks if you want to make a
few small changes to a patch that aren't worthy of a revbump

c) most of the stuff that you're complaining about has been there for
ages, and when it was added this wasn't an issue.

-- 
Ciaran McCreesh : Gentoo Developer (Vim, Fluxbox, Sparc, Mips)
Mail            : ciaranm at gentoo.org
Web             : http://dev.gentoo.org/~ciaranm


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] rsync speed and space taken

[Donnie Berkholz]

[-- Attachment #1: Type: text/plain, Size: 281 bytes --]

On Mon, 2004-10-11 at 15:32 +0100, Ciaran McCreesh wrote:
> b) because the mirror system really really sucks if you want to make a
> few small changes to a patch that aren't worthy of a revbump

This is why I also have a patchball version.
-- 
Donnie Berkholz
Gentoo Linux

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] rsync speed and space taken

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 650 bytes --]

On Mon, 11 Oct 2004 08:52:19 -0700 Donnie Berkholz
<spyderous@gentoo.org> wrote:
| On Mon, 2004-10-11 at 15:32 +0100, Ciaran McCreesh wrote:
| > b) because the mirror system really really sucks if you want to make
| > a few small changes to a patch that aren't worthy of a revbump
| 
| This is why I also have a patchball version.

Yeah, then that gets messy if you end up trying to keep the same
patchset between revbumps. End up having to stick in a global variable
for it... Hardly ideal.

-- 
Ciaran McCreesh : Gentoo Developer (Vim, Fluxbox, Sparc, Mips)
Mail            : ciaranm at gentoo.org
Web             : http://dev.gentoo.org/~ciaranm


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] rsync speed and space taken

[Chris L. Mason]

I don't really see the problem with rsync speeds.  I am able to rsync
in a couple minutes.  However, the portage cache update takes over an
hour on ppc-macos!!  Is this normal, or is there some kind of
workaround?  I haven't tried in Linux since my hardware isn't
supported yet (iMac G5.)


Chris

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] rsync speed and space taken

[Pieter Van den Abeele]

Hi,

2004.3/PPC supports the G5 imac (and also features a Rendezvous distcc 
enabled darwin/macos cross compiler) :-)

Updating the cache shouldn't take that long. Could you let me know off 
list what filesystem (UFS or HFS+) and Mac OS X release you're using?
Can you send me the output of emerge info?

Best regards,

Pieter Van den Abeele


On 13 Oct 2004, at 19:18, Chris L. Mason wrote:

> I don't really see the problem with rsync speeds.  I am able to rsync
> in a couple minutes.  However, the portage cache update takes over an
> hour on ppc-macos!!  Is this normal, or is there some kind of
> workaround?  I haven't tried in Linux since my hardware isn't
> supported yet (iMac G5.)
>
>
> Chris
>
> --
> gentoo-dev@gentoo.org mailing list
>


--
gentoo-dev@gentoo.org mailing list