[gentoo-user] CVSup vs Gentoo's Rsync

[gentoo-user] CVSup vs Gentoo's Rsync

[Ow Mun Heng]

Anyone knows?? A google search found interesting details on cvsup.
(there's even the purported 50x increase in speed on a 56K
link)http://www.cvsup.org/cvsup6.html

Then I also found a link in one of GWM (Gentoo Weeky News all the way
back in 2002 -
http://www.gentoo.org/news/en/gwn/20021223-newsletter.xml#doc_chap2_sect4) that states

[snip]
CVSup Under Consideration as Replacement for rsync

There has been some discussion in the Gentoo developer community about
migrating away from Portage's dependency on rsync and instead utilizing
CVSup. Currently used in FreeBSD's ports system, CVSup offers a few
distinct advantages, as well as challenges, over rsync:

      * Local changes will be preserved if you want them to, just like
        anoncvs. Anyone who's ever had their package.mask file
        overwritten by an emerge rsync will appreciate this feature. 
      * CVSup is faster and more efficient than rsync. This means that
        CVSup can improve the efficiency of our Portage mirroring
        system. 
      * CVSup's threaded design allows for file transfer begin almost
        immediately, unlike rsync which must build a complete file list
        first. Surprisingly, CVSup uses the rsync algorithm (which is
        very efficient) internally to synchronize individual files, but
        uses a better approach than rsync when coordinating the updates
        of large numbers of files. 
      * CVSup is written in Modula 3, which means that transitioning to
        CVSup will require some non-trivial steps to ensure that we have
        proper Modula 3 support on all architectures. 
      * CVSup also has the added psychological benefit of making FreeBSD
        users feel more at home. 
[/snip]

Anyone wants to comment??

-- 
Ow Mun Heng
Gentoo/Linux on DELL D600 1.4Ghz 1.5GB RAM
98% Microsoft(tm) Free!! 
Neuromancer 18:07:43 up 4 days, 4:40, 6 users, load average: 1.51, 0.99,
0.76 


-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] CVSup vs Gentoo's Rsync

[Julien Cabillot]

On 2/21/06, Ow Mun Heng <Ow.Mun.Heng@wdc.com> wrote:
> [SNIP]
>
>       * Local changes will be preserved if you want them to, just like
>         anoncvs. Anyone who's ever had their package.mask file
>         overwritten by an emerge rsync will appreciate this feature.
>
> [SNIP]

If you don't want loose your package.mask, use /etc/portage/package.mask.

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] CVSup vs Gentoo's Rsync

[Boyd Stephen Smith Jr.]

On Tuesday 21 February 2006 04:07, Ow Mun Heng <Ow.Mun.Heng@wdc.com> wrote 
about '[gentoo-user] CVSup vs Gentoo's Rsync':
> Anyone wants to comment??

I already think Gentoo base system requirements are a bit heavy just having 
python in there.  (I'm trying to run gentoo on my WRT54G 1.1; storage over 
nfs/nbd)  It would be completely over-the-top to require Modula 3 support 
as well.

That said, it would be nice to do away with the "building file list..." 
delay we have with rsync.

-- 
Boyd Stephen Smith Jr.
bss03@volumehost.com
ICQ: 514984 YM/AIM: DaTwinkDaddy

-- Portage needs to be written in C. --
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] CVSup vs Gentoo's Rsync

[Aniruddha Shankar]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Boyd Stephen Smith Jr. wrote:
> That said, it would be nice to do away with the "building file list..." 
> delay we have with rsync.

That I can live with. It's the updating portage cache - especially the
long wait at 50% that drags.

K


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.1 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFD+7JqhJkrd6A3rSsRAhCjAKCpM2MJOicqgRrS/qcfnPZhrbyXzQCfVPMz
ulCZNtBVarR+QTXE4Xw0+4Q=
=tlOw
-----END PGP SIGNATURE-----
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] CVSup vs Gentoo's Rsync

[Boyd Stephen Smith Jr.]

On Tuesday 21 February 2006 18:38, Aniruddha Shankar <karim@sarai.net> 
wrote about 'Re: [gentoo-user] CVSup vs Gentoo's Rsync':
> That I can live with. It's the updating portage cache - especially the
> long wait at 50% that drags.

Go ~ for portage, I don't experience such a delay.

-- 
Boyd Stephen Smith Jr.
bss03@volumehost.com
ICQ: 514984 YM/AIM: DaTwinkDaddy
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] CVSup vs Gentoo's Rsync

[Ow Mun Heng]

On Tue, 2006-02-21 at 05:24 -0600, Boyd Stephen Smith Jr. wrote:
> On Tuesday 21 February 2006 04:07, Ow Mun Heng <Ow.Mun.Heng@wdc.com> wrote 
> about '[gentoo-user] CVSup vs Gentoo's Rsync':
> > Anyone wants to comment??
> 
> I already think Gentoo base system requirements are a bit heavy just having 
> python in there.  (I'm trying to run gentoo on my WRT54G 1.1; storage over 
> nfs/nbd)  It would be completely over-the-top to require Modula 3 support 
> as well.

Just tried cvsup on a FreeBSD box (in vmware) and it totally sucked.
Took 1hr 20 min to update on a 512KB DSL.

Link wasn't even saturated! Might as well be on 56K connection. DL rate
max I've seen was 8KB/s


-- 
Ow Mun Heng
Gentoo/Linux on DELL D600 1.4Ghz 1.5GB RAM
98% Microsoft(tm) Free!! 
Neuromancer 10:04:41 up 1 day, 11:39, 5 users, load average: 0.38, 0.21,
0.33 


-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] CVSup vs Gentoo's Rsync

[Alexander Kirillov]

>>I already think Gentoo base system requirements are a bit heavy just having 
>>python in there.  (I'm trying to run gentoo on my WRT54G 1.1; storage over 
>>nfs/nbd)  It would be completely over-the-top to require Modula 3 support 
>>as well.
> 
> Just tried cvsup on a FreeBSD box (in vmware) and it totally sucked.
> Took 1hr 20 min to update on a 512KB DSL.
> 
> Link wasn't even saturated! Might as well be on 56K connection. DL rate
> max I've seen was 8KB/s

BSD ports system may not be as complex as Gentoo's
and needs more attention to maintain port dependencies
but this "abuse of make" IS REAL FAST.
I wish I could check all installed files against recorded md5 sums
as easily as I can do this in FreeBSD.
I took me 20 min to sync ports tree I didn't sync since December
and 7 min to sync system sources.
If you sync regularly it's usually within 5 min.
The good thing about cvsup is that it never fails
which is not the case with rsync.

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] CVSup vs Gentoo's Rsync

[Hemmann, Volker Armin]

On Thursday 23 February 2006 11:26, Alexander Kirillov wrote:
> >>I already think Gentoo base system requirements are a bit heavy just
> >> having python in there.  (I'm trying to run gentoo on my WRT54G 1.1;
> >> storage over nfs/nbd)  It would be completely over-the-top to require
> >> Modula 3 support as well.
> >
> > Just tried cvsup on a FreeBSD box (in vmware) and it totally sucked.
> > Took 1hr 20 min to update on a 512KB DSL.
> >
> > Link wasn't even saturated! Might as well be on 56K connection. DL rate
> > max I've seen was 8KB/s
>
> BSD ports system may not be as complex as Gentoo's
> and needs more attention to maintain port dependencies
> but this "abuse of make" IS REAL FAST.

portage is REAL SLOW.
So you should install/use cdb, which makes the whole rsync/update process REAL 
FAST.

> If you sync regularly it's usually within 5 min.

it is, with cdb.
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] CVSup vs Gentoo's Rsync

[Alexander Kirillov]

>>BSD ports system may not be as complex as Gentoo's
>>and needs more attention to maintain port dependencies
>>but this "abuse of make" IS REAL FAST.
>
> portage is REAL SLOW.
> So you should install/use cdb, which makes the whole rsync/update process REAL 
> FAST.
> 
>>If you sync regularly it's usually within 5 min.
> 
> it is, with cdb.

Thanks for the hint.
If I got it right here's the link for those interested:
http://gentoo-wiki.com/TIP_speed_up_portage_with_cdb
Looks very interesting and I wonder why is it not in portage yet?

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] CVSup vs Gentoo's Rsync

[Hemmann, Volker Armin]

On Friday 24 February 2006 00:39, Alexander Kirillov wrote:
> >>BSD ports system may not be as complex as Gentoo's
> >>and needs more attention to maintain port dependencies
> >>but this "abuse of make" IS REAL FAST.
> >
> > portage is REAL SLOW.
> > So you should install/use cdb, which makes the whole rsync/update process
> > REAL FAST.
> >
> >>If you sync regularly it's usually within 5 min.
> >
> > it is, with cdb.
>
> Thanks for the hint.
> If I got it right here's the link for those interested:
> http://gentoo-wiki.com/TIP_speed_up_portage_with_cdb
> Looks very interesting and I wonder why is it not in portage yet?

well ,the chat log  tells you why ;)

but you also look into the forum. A portage update broke it for me, and in the 
forum were updated instructions - I don't know, if the gentoowiki was updated 
too.
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] CVSup vs Gentoo's Rsync

[Alexander Skwar]

Alexander Kirillov wrote:

> If I got it right here's the link for those interested:
> http://gentoo-wiki.com/TIP_speed_up_portage_with_cdb

Does this work with recent portage releases?

Alexander Skwar
-- 
"My pan plays down an unprecedented amount of our national debt."
George W. Bush
February 27, 2001
>From a speech concerning the proposed federal budget.
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] CVSup vs Gentoo's Rsync

[Hemmann, Volker Armin]

On Friday 24 February 2006 07:00, Alexander Skwar wrote:
> Alexander Kirillov wrote:
> > If I got it right here's the link for those interested:
> > http://gentoo-wiki.com/TIP_speed_up_portage_with_cdb
>
> Does this work with recent portage releases?
>
it stopped some weeks ago - but in the gentoo forums was  updated 
instructions.
I am not sure, if the instructions in the wiki got updated but that is easy to 
find out.
Try it. If it does not work, go to the forum ;)
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] CVSup vs Gentoo's Rsync

[Alexander Skwar]

Hemmann, Volker Armin wrote:
> On Friday 24 February 2006 07:00, Alexander Skwar wrote:
>> Alexander Kirillov wrote:
>> > If I got it right here's the link for those interested:
>> > http://gentoo-wiki.com/TIP_speed_up_portage_with_cdb
>>
>> Does this work with recent portage releases?
>>
> it stopped some weeks ago - but in the gentoo forums was  updated 
> instructions.
> I am not sure, if the instructions in the wiki got updated but that is easy to 
> find out.
> Try it. If it does not work, go to the forum ;)

*LOL* - Now, that's what I'd call a "practical" approach :)

I just wondered, because even in the most recent portage
releases, there's an einfo text stating, that the cdb
caching module needs to be rewritten.

But, I'll follow your advice and just try it. Thanks!

Alexander Skwar
-- 
If you can keep your head when all about you are losing theirs, then
you clearly don't understand the situation.
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] CVSup vs Gentoo's Rsync

[Hemmann, Volker Armin]

On Friday 24 February 2006 15:37, Alexander Skwar wrote:
> Hemmann, Volker Armin wrote:
> > On Friday 24 February 2006 07:00, Alexander Skwar wrote:
> >> Alexander Kirillov wrote:
> >> > If I got it right here's the link for those interested:
> >> > http://gentoo-wiki.com/TIP_speed_up_portage_with_cdb
> >>
> >> Does this work with recent portage releases?
> >
> > it stopped some weeks ago - but in the gentoo forums was  updated
> > instructions.
> > I am not sure, if the instructions in the wiki got updated but that is
> > easy to find out.
> > Try it. If it does not work, go to the forum ;)
>
> *LOL* - Now, that's what I'd call a "practical" approach :)
>
> I just wondered, because even in the most recent portage
> releases, there's an einfo text stating, that the cdb
> caching module needs to be rewritten.

http://forums.gentoo.org/viewtopic-t-261580.html

is the thread.

But beware, in /usr/portage/modules you need this:

portdbapi.auxdbmodule = cache.cdb.database
eclass_cache.dbmodule = cache.cdb.database

not that:

portdbapi.auxdbmodule = portage_db_cdb.database
eclass_cache.dbmodule = portage_db_cdb.database


that is the only change I remember at the moment (and which did not went into 
the wiki).

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] CVSup vs Gentoo's Rsync

[Alexander Skwar]

Hemmann, Volker Armin wrote:

> that is the only change I remember at the moment (and which did not went into 
> the wiki).

Not true anymore :) I just added "your" changes to the wiki.

Thanks a lot,

Alexander Skwar
-- 
Yow!  I'm having a quadrophonic sensation of two winos alone in a steel mill!
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] CVSup vs Gentoo's Rsync

[Hemmann, Volker Armin]

On Saturday 25 February 2006 07:37, Alexander Skwar wrote:
> Hemmann, Volker Armin wrote:
> > that is the only change I remember at the moment (and which did not went
> > into the wiki).
>
> Not true anymore :) I just added "your" changes to the wiki.

'your' is correct, because I 'stole' it from the forum's thread ;)

Nice, that you updated the wiki.
-- 
gentoo-user@gentoo.org mailing list

[gentoo-user] cdb with portage 2.1 (was: CVSup vs Gentoo's Rsync)

[Alexander Skwar]

Hi!

Hemmann, Volker Armin <volker.armin.hemmann <at> tu-clausthal.de> writes:

> http://forums.gentoo.org/viewtopic-t-261580.html
> 
> is the thread.
> 
> But beware, in /usr/portage/modules you need this:
> 
> portdbapi.auxdbmodule = cache.cdb.database
> eclass_cache.dbmodule = cache.cdb.database
> 
> not that:
> 
> portdbapi.auxdbmodule = portage_db_cdb.database
> eclass_cache.dbmodule = portage_db_cdb.database
> 
> that is the only change I remember at the moment (and which did not went into 
> the wiki).

I now finally got around to actually TRY what I put in the wiki at 
http://gentoo-wiki.com/TIP_speed_up_portage_with_cdb :)

And I discovered, that this doesn't work for me :(
When I write your lines in /etc/portage/modules and
run "emerge --metadata", I get the following error:

alexander@blatt ~ $ time sudo emerge --verbose --metadata

====================================
Error: Failed to import module 'cache.cdb.database'

  File "/usr/lib/portage/pym/portage.py", line 1290, in load_best_module
    mod = load_mod(best_mod)
  File "/usr/lib/portage/pym/portage.py", line 149, in load_mod
    mod = __import__(modname)
No module named cdb
====================================

But when I put the "old lines" (those with portage_db_cdb.database)
in the modules file, I get:

alexander@blatt ~ $ time sudo emerge --verbose --metadata

====================================
Error: Failed to import module 'portage_db_cdb.database'

  File "/usr/lib/portage/pym/portage.py", line 1290, in load_best_module
    mod = load_mod(best_mod)
  File "/usr/lib/portage/pym/portage.py", line 149, in load_mod
    mod = __import__(modname)
  File "/usr/lib/portage/pym/portage_db_cdb.py", line 17, in ?
    import portage_db_template
No module named portage_db_template
====================================

Well, for me, it seems that the 2nd version (with
portage_db_cdb.database) seems to be a little bit
more "successful", as it actually LOADS the
portage_db_cdb.py file. That's not done with the
version using 'cache.cdb.database' - here it fails
to load a module named cdb.

Is anyone using CDB with Portage 2.1?

Regards,

Alexander Skwar
-- 
"Never give in.  Never give in.  Never. Never. Never."
-- Winston Churchill

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] cdb with portage 2.1 (was: CVSup vs Gentoo's Rsync)

[Hemmann, Volker Armin]

On Friday 17 March 2006 19:24, Alexander Skwar wrote:

>
> Is anyone using CDB with Portage 2.1?

with the latest portage incarnations it does not work anymore at all. But 
portage got a lot speedier too, so the problem is not soo big.
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] cdb with portage 2.1 (was: CVSup vs Gentoo's Rsync)

[Bo Andresen]

On Friday 17 March 2006 19:24, Alexander Skwar wrote:
> Is anyone using CDB with Portage 2.1?

 # grep ewarn /usr/portage/sys-apps/portage/portage-2.1_pre6-r3.ebuild 
        ewarn "This series contains a completely rewritten caching framework."
        ewarn "If you are using any cache modules (such as the CDB cache"
        ewarn "module) portage will not work until they have been disabled."

-- 
Bo Andresen
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] cdb with portage 2.1

[Alexander Skwar]

Bo Andresen wrote:
> On Friday 17 March 2006 19:24, Alexander Skwar wrote:
>> Is anyone using CDB with Portage 2.1?
> 
>  # grep ewarn /usr/portage/sys-apps/portage/portage-2.1_pre6-r3.ebuild 
>         ewarn "This series contains a completely rewritten caching framework."
>         ewarn "If you are using any cache modules (such as the CDB cache"
>         ewarn "module) portage will not work until they have been disabled."

Yes, I know, but please read the thread.

Thanks,

Alexander Skwar
-- 
Luke Skywalker:
	I'm Luke Skywalker, I'm here to rescue you.
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] cdb with portage 2.1

[Bo Andresen]

On Saturday 18 March 2006 07:16, Alexander Skwar wrote:
> Yes, I know, but please read the thread.

Oh sorry. I probably shouldn't write to a mailing list when that tired... ;)

-- 
Bo Andresen
-- 
gentoo-user@gentoo.org mailing list