[gentoo-dev] Problems with the current bzr eclass.

[gentoo-dev] Problems with the current bzr eclass.

[Harley Peters]

The current bzr eclass uses EBZR_FETCH_CMD="bzr checkout --lightweight"
to initially fetch the sources.
The problem with this is though it saves some time and bandwidth on the
initial fetch it actually ends up using a lot more time and bandwidth
on every update.
Ok great  I can set EBZR_FETCH_CMD="bzr checkout".
This solves the initial fetch problems.
Now when I try and update the bzr eclass function bzr_fetch() has this
bit of code in it.

# Run bzr_initial_fetch() only if the branch has not been pulled
# before or if the existing local copy is a full checkout (as did
# an older version of bzr.eclass)
if [[ ! -d ${EBZR_BRANCH_DIR} ]] ; then
	bzr_initial_fetch "${EBZR_REPO_URI}" "${EBZR_BRANCH_DIR}"
elif [[ -d "${EBZR_BRANCH_DIR}"/.bzr/repository/ ]]; then
	einfo "Re-fetching the branch to save space..."
	rm -rf "${EBZR_BRANCH_DIR}"
	bzr_initial_fetch "${EBZR_REPO_URI}" "${EBZR_BRANCH_DIR}"
else
	bzr_update "${EBZR_REPO_URI}" "${EBZR_BRANCH_DIR}"
fi

Since I did a full checkout with the EBZR_FETCH_CMD="bzr checkout"
it now deletes the entire previous checked out branch (to save disk space ?) and proceeds to fetch the entire
source again.
Why would I ever want to do that ? The whole point of bzr is to save bandwidth not disk space.
Is there a way arouund this ?

[gentoo-dev] Re: Problems with the current bzr eclass.

[Christian Faulhammer]

[-- Attachment #1: Type: text/plain, Size: 856 bytes --]

Hi,

Harley Peters <harley@thepetersclan.net>:
> Since I did a full checkout with the EBZR_FETCH_CMD="bzr checkout"
> it now deletes the entire previous checked out branch (to save disk
> space ?) and proceeds to fetch the entire source again.
> Why would I ever want to do that ? The whole point of bzr is to save
> bandwidth not disk space. Is there a way arouund this ?

 You can add a modified bzr.eclass to a local overlay which will shadow
the one from the Portage tree.  This idea was born because initial
checkouts are/were incredibly slow, so give first time users a better
first experience and not let them wait 20 minutes (what happened with
the Emacs repository).

V-Li

-- 
Christian Faulhammer, Gentoo Lisp project
<URL:http://www.gentoo.org/proj/en/lisp/>, #gentoo-lisp on FreeNode

<URL:http://gentoo.faulhammer.org/>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Re: Problems with the current bzr eclass.

[Harley Peters]

On Thu, 9 Jul 2009 20:55:54 +0200
Christian Faulhammer <fauli@gentoo.org> wrote:

> Hi,
> 
> Harley Peters <harley@thepetersclan.net>:
> > Since I did a full checkout with the EBZR_FETCH_CMD="bzr checkout"
> > it now deletes the entire previous checked out branch (to save disk
> > space ?) and proceeds to fetch the entire source again.
> > Why would I ever want to do that ? The whole point of bzr is to save
> > bandwidth not disk space. Is there a way arouund this ?
> 
>  You can add a modified bzr.eclass to a local overlay which will
> shadow the one from the Portage tree.  This idea was born because
> initial checkouts are/were incredibly slow, so give first time users
> a better first experience and not let them wait 20 minutes (what
> happened with the Emacs repository).
> 
> V-Li
> 

Ok that's what I am doing. Was just wondering if there was something
simple I over looked.

[gentoo-dev] Re: Problems with the current bzr eclass.

[Christian Faulhammer]

[-- Attachment #1: Type: text/plain, Size: 1298 bytes --]

Hi,

Harley Peters <harley@thepetersclan.net>:
> Ok that's what I am doing. Was just wondering if there was something
> simple I over looked.

 To give you an idea what we gain here, I compiled some numbers.  GNU
Emacs live ebuild (app-editors/emacs-cvs) will be the biggest consumer
once they switch from CVS.

First we will compare timings

$ time bzr checkout --lightweight
real    7m37.044s
user    1m2.789s
sys     0m2.807s

$ time bzr checkout

real    40m47.371s
user    3m57.881s
sys     0m8.586s

 So the normal initial checkout will be more than five times slower than
the lightweight one.  If this were 1s to 6s it would be neglectable,
but we are talking about half an hour difference just for an emerge.

 Next ist the update function.

 $ time bzr update

real    0m11.457s
user    0m0.544s
sys     0m0.092s

 $ time bzr update

real    0m2.710s
user    0m0.237s
sys     0m0.053s

 Clearly the normal checkout wins by factor five, but we are in the
second range, while the space gain is enormous:

$ du -sch Emacs-lw/ Emacs
106M    Emacs-lw/
427M    Emacs
533M    total

V-Li

-- 
Christian Faulhammer, Gentoo Lisp project
<URL:http://www.gentoo.org/proj/en/lisp/>, #gentoo-lisp on FreeNode

<URL:http://gentoo.faulhammer.org/>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Re: Problems with the current bzr eclass.

[Harley Peters]

On Thu, 9 Jul 2009 22:16:34 +0200
Christian Faulhammer <fauli@gentoo.org> wrote:

> Hi,
> 
> Harley Peters <harley@thepetersclan.net>:
> > Ok that's what I am doing. Was just wondering if there was something
> > simple I over looked.
> 
>  To give you an idea what we gain here, I compiled some numbers.  GNU
> Emacs live ebuild (app-editors/emacs-cvs) will be the biggest consumer
> once they switch from CVS.
> 
> First we will compare timings
> 
> $ time bzr checkout --lightweight
> real    7m37.044s
> user    1m2.789s
> sys     0m2.807s
> 
> $ time bzr checkout
> 
> real    40m47.371s
> user    3m57.881s
> sys     0m8.586s
> 
>  So the normal initial checkout will be more than five times slower
> than the lightweight one.  If this were 1s to 6s it would be
> neglectable, but we are talking about half an hour difference just
> for an emerge.
> 
>  Next ist the update function.
> 
>  $ time bzr update
> 
> real    0m11.457s
> user    0m0.544s
> sys     0m0.092s
> 
>  $ time bzr update
> 
> real    0m2.710s
> user    0m0.237s
> sys     0m0.053s
> 
>  Clearly the normal checkout wins by factor five, but we are in the
> second range, while the space gain is enormous:
> 
> $ du -sch Emacs-lw/ Emacs
> 106M    Emacs-lw/
> 427M    Emacs
> 533M    total
> 
> V-Li
> 

I understand why you did it. I just need a way to prevent it from
deleting a full checkout.
Because the numbers don't look so good for Gnash.
bzr branch http://bzr.savannah.gnu.org/r/gnash/trunk
Having said that it's only a two line change to the bzr.eclass code to
get it to work the way I want.
And I can live with that. :)

[gentoo-dev] Re: Problems with the current bzr eclass.

[Christian Faulhammer]

[-- Attachment #1: Type: text/plain, Size: 641 bytes --]

Hi,

Harley Peters <harley@thepetersclan.net>:
> Because the numbers don't look so good for Gnash.
> bzr branch http://bzr.savannah.gnu.org/r/gnash/trunk

 They look similar on my system to that of Emacs, although I have to
try with new revisions added.

> Having said that it's only a two line change to the bzr.eclass code to
> get it to work the way I want.
> And I can live with that. :)

 But in the end I need to justify the changes not only for you. :) 

V-Li

-- 
Christian Faulhammer, Gentoo Lisp project
<URL:http://www.gentoo.org/proj/en/lisp/>, #gentoo-lisp on FreeNode

<URL:http://gentoo.faulhammer.org/>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Re: Problems with the current bzr eclass.

[Harley Peters]

On Fri, 10 Jul 2009 08:50:26 +0200
Christian Faulhammer <fauli@gentoo.org> wrote:

> Hi,
> 
> Harley Peters <harley@thepetersclan.net>:
> > Because the numbers don't look so good for Gnash.
> > bzr branch http://bzr.savannah.gnu.org/r/gnash/trunk
> 
>  They look similar on my system to that of Emacs, although I have to
> try with new revisions added.
> 
> > Having said that it's only a two line change to the bzr.eclass code
> > to get it to work the way I want.
> > And I can live with that. :)
> 
>  But in the end I need to justify the changes not only for you. :) 
> 
> V-Li
> 

I'm getting good numbers from the command line but in the ebuild it's
much different.
I haven't timed it but it takes about 25 minutes to do a full
checkout and about a third that to do the light weight checkout.
The update takes less than a minute with the full checkout. And it
takes ten's of minutes with a lightweight checkout.

Is there something else in the ebuild I need to specify besides
EBZR_REPO_URI="http://bzr.savannah.gnu.org/r/gnash/trunk" ? 

You can download the ebuild at:
http://overlays.biterror.net/files/gnash-9999.ebuild

Re: [gentoo-dev] Re: Problems with the current bzr eclass.

[René 'Necoro' Neumann]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Harley Peters schrieb:
> I'm getting good numbers from the command line but in the ebuild it's
> much different.
> I haven't timed it but it takes about 25 minutes to do a full
> checkout and about a third that to do the light weight checkout.
> The update takes less than a minute with the full checkout. And it
> takes ten's of minutes with a lightweight checkout.

I guess, the "bzr export" is the problem here ... see
https://bugs.launchpad.net/bzr-gentoo-overlay/+bug/343218

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkpXKGwACgkQ4UOg/zhYFuC5/ACcCbUxkcHW7ZuDm0eDvkIFB20a
cIsAn1N7OUdxyKRbOYme/QtsBDo9E+SG
=l+yv
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Re: Problems with the current bzr eclass.

[Marijn Schouten (hkBst)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Christian Faulhammer wrote:
> Hi,
> 
> Harley Peters <harley@thepetersclan.net>:
>> Since I did a full checkout with the EBZR_FETCH_CMD="bzr checkout"
>> it now deletes the entire previous checked out branch (to save disk
>> space ?) and proceeds to fetch the entire source again.
>> Why would I ever want to do that ? The whole point of bzr is to save
>> bandwidth not disk space. Is there a way arouund this ?
> 
>  You can add a modified bzr.eclass to a local overlay which will shadow
> the one from the Portage tree.  This idea was born because initial
> checkouts are/were incredibly slow, so give first time users a better
> first experience and not let them wait 20 minutes (what happened with
> the Emacs repository).
> 
> V-Li

I understand that the time trade-off favors lightweight checkouts, but if there
is a pre-existing full checkout then why secondguess the user and delete it? It
seems to me that the lines


elif [[ -d "${EBZR_BRANCH_DIR}"/.bzr/repository/ ]]; then
	einfo "Re-fetching the branch to save space..."
	rm -rf "${EBZR_BRANCH_DIR}"
	bzr_initial_fetch "${EBZR_REPO_URI}" "${EBZR_BRANCH_DIR}"

should be removed. If users want to get rid of full checkouts they can easily
delete them themselves.

Marijn

- --
If you cannot read my mind, then listen to what I say.

Marijn Schouten (hkBst), Gentoo Lisp project, Gentoo ML
<http://www.gentoo.org/proj/en/lisp/>, #gentoo-{lisp,ml} on FreeNode
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkpbOMwACgkQp/VmCx0OL2yXBQCfVAGkJGkugj3nOoa2vgOn6cLj
X+YAn2LtVEZ3jsFVdAUArtuuRkJfKrp1
=HmE/
-----END PGP SIGNATURE-----

[gentoo-dev] Re: Problems with the current bzr eclass.

[Christian Faulhammer]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

René 'Necoro' Neumann <lists@necoro.eu>:
> Harley Peters schrieb:
> > I'm getting good numbers from the command line but in the ebuild
> > it's much different.
> > I haven't timed it but it takes about 25 minutes to do a full
> > checkout and about a third that to do the light weight checkout.
> > The update takes less than a minute with the full checkout. And it
> > takes ten's of minutes with a lightweight checkout.
> 
> I guess, the "bzr export" is the problem here ... see
> https://bugs.launchpad.net/bzr-gentoo-overlay/+bug/343218

 Ok, that sounds bad.  Which versions of Bazaar did you try?

V-Li

- -- 
Christian Faulhammer, Gentoo Lisp project
<URL:http://www.gentoo.org/proj/en/lisp/>, #gentoo-lisp on FreeNode

<URL:http://gentoo.faulhammer.org/>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.11 (GNU/Linux)

iEYEARECAAYFAkpcUcsACgkQNQqtfCuFneP+AACfdK0Z8D00GDF9fSIl1fU10LOi
5egAoJn58bGPaNAU54JweAlXSrCbj0ch
=dTfd
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Re: Problems with the current bzr eclass.

[harley]

On Tue, July 14, 2009 4:37 am, Christian Faulhammer wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
> Hi,
>
>
> René 'Necoro' Neumann <lists@necoro.eu>:
>
>> Harley Peters schrieb:
>>
>>> I'm getting good numbers from the command line but in the ebuild
>>> it's much different. I haven't timed it but it takes about 25 minutes
>>> to do a full checkout and about a third that to do the light weight
>>> checkout. The update takes less than a minute with the full checkout.
>>> And it
>>> takes ten's of minutes with a lightweight checkout.
>>
>> I guess, the "bzr export" is the problem here ... see
>> https://bugs.launchpad.net/bzr-gentoo-overlay/+bug/343218
>>
>
> Ok, that sounds bad.  Which versions of Bazaar did you try?
>
>
> V-Li
>
>
> - --
> Christian Faulhammer, Gentoo Lisp project
> <URL:http://www.gentoo.org/proj/en/lisp/>, #gentoo-lisp on FreeNode
>
>
> <URL:http://gentoo.faulhammer.org/>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.11 (GNU/Linux)
>
>
> iEYEARECAAYFAkpcUcsACgkQNQqtfCuFneP+AACfdK0Z8D00GDF9fSIl1fU10LOi
> 5egAoJn58bGPaNAU54JweAlXSrCbj0ch
> =dTfd
> -----END PGP SIGNATURE-----
>
>

Not sure who you are referring to but I was using bzr-1.16.1 and bzr-1.15.
Bzr export is definitely the problem for me.

Harley

[gentoo-dev] Re: Problems with the current bzr eclass.

[Christian Faulhammer]

[-- Attachment #1: Type: text/plain, Size: 1806 bytes --]

Hi,

harley@thepetersclan.net:

> On Tue, July 14, 2009 4:37 am, Christian Faulhammer wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> >
> > Hi,
> >
> >
> > René 'Necoro' Neumann <lists@necoro.eu>:
> >
> >> Harley Peters schrieb:
> >>
> >>> I'm getting good numbers from the command line but in the ebuild
> >>> it's much different. I haven't timed it but it takes about 25
> >>> minutes to do a full checkout and about a third that to do the
> >>> light weight checkout. The update takes less than a minute with
> >>> the full checkout. And it
> >>> takes ten's of minutes with a lightweight checkout.
> >>
> >> I guess, the "bzr export" is the problem here ... see
> >> https://bugs.launchpad.net/bzr-gentoo-overlay/+bug/343218
> >>
> >
> > Ok, that sounds bad.  Which versions of Bazaar did you try?
> >
> >
> > V-Li
> >
> >
> > - --
> > Christian Faulhammer, Gentoo Lisp project
> > <URL:http://www.gentoo.org/proj/en/lisp/>, #gentoo-lisp on FreeNode
> >
> >
> > <URL:http://gentoo.faulhammer.org/>
> > -----BEGIN PGP SIGNATURE-----
> > Version: GnuPG v2.0.11 (GNU/Linux)
> >
> >
> > iEYEARECAAYFAkpcUcsACgkQNQqtfCuFneP+AACfdK0Z8D00GDF9fSIl1fU10LOi
> > 5egAoJn58bGPaNAU54JweAlXSrCbj0ch
> > =dTfd
> > -----END PGP SIGNATURE-----
> >
> >
> 
> Not sure who you are referring to but I was using bzr-1.16.1 and
> bzr-1.15. Bzr export is definitely the problem for me.

 Any luck with bzr 1.17 which will likely enter the tree soon.  An
option would be to run bzr clone from the local repository instead of
export or just do a cp run (cp -R should be sufficient?).

V-Li

-- 
Christian Faulhammer, Gentoo Lisp project
<URL:http://www.gentoo.org/proj/en/lisp/>, #gentoo-lisp on FreeNode

<URL:http://gentoo.faulhammer.org/>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]