[gentoo-user] Portage Storage using SVN

[gentoo-user] Portage Storage using SVN

[Trenton Adams]

Hi guys,

I proposed this awhile back, and got shot down.  At the time, the
arguments for using SVN for portage storage were pretty shallow, and
someone was able to easily shoot them down.  I believe I have come up
with better reasoning for using SVN.  Someone may still shoot them
down, but hey, it's worth a try.

PROBLEM 1
Let's say openldap had a problem. So, we decide to mask the latest
version of openldap, in an effort to roll back to the version that was
working.  Well, we find out that openldap still does not work.  So, we
finally determine that it is library W.  So, now we mask library W, in
an attempt to roll back to the version that was working.  Oh no, now
we find out that library W is used by 20 other packages, that require
the latest version of library W in order to work.  So, now we have to
mask library W, and 20 packages in order to get our openldap system
functional, assuming you cared about the 20 other broken packages,
which may break other packages, which may break yet other packages.

Wouldn't it be nice to just go "emerge --revert-portage", which goes
back to the last exported copy of the portage, that you had from
subversion?  Boy, would that ever be convenient.  It would be simple
enough to store a local history of portage tags that the user was
using in the past.

PROBLEM 2
Have you ever synced the portage, only to find out there's a broken
package simply because you synced in the middle of a mirror being
updated?  Well, subversion operations are entirely atomic, meaning no
need to worry about such silly things.

Also, copies (tags) could be created for every single day, and special
ones for milestones.

PROBLEM 3
Don't sync more than once a day, or you may be temporarily banned?
Well, with SVN being tagged only once a day, there would be no need to
worry about this, seeing that

POTENTIAL ISSUES
Now, I'm not entirely sure of the performance implications of
subversion for this purpose.  So, that would definitely have to either
be tested, or someone would have to talk with the subversion folks to
know if it would be a problem for thousands of users to access
subversion in readonly mode.  It would certainly be annoying for a
developer to go "svn commit", and have to wait for half an hour
because everyone else is updating their local copies.  But, that could
be solved by mirrors only getting updated once every day, at 12
midnight.

How does one get SVN copied over to mirrors in an atomic way, or does
the SVN location just get shut down during the mirror update?

Any thoughts?
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Portage Storage using SVN

[Hans-Werner Hilse]

Hi,

On Sun, 23 Jul 2006 02:42:43 -0600
"Trenton Adams" <trenton.d.adams@gmail.com> wrote:

> I proposed this awhile back, and got shot down.  At the time, the
> arguments for using SVN for portage storage were pretty shallow, and
> someone was able to easily shoot them down.  I believe I have come up
> with better reasoning for using SVN.  Someone may still shoot them
> down, but hey, it's worth a try.

#1:
You're aware that there's a CVS for portage, aren't you? I'm still not
quite sure if you are suggesting using SVN for the portage mirrors and
if you are suggesting that users also have a full SVN history on the
clients, too?

> PROBLEM 1
> [...]
> PROBLEM 2
> [...]
> PROBLEM 3
> [...]

Well, are those really problems at all? I mean, isn't it easy to
overcome them? Is it worth dedicating time and work into that svn thing?

> POTENTIAL ISSUES
> Now, I'm not entirely sure of the performance implications of
> subversion for this purpose.  So, that would definitely have to either
> be tested, or someone would have to talk with the subversion folks to
> know if it would be a problem for thousands of users to access
> subversion in readonly mode.

Well, of course! There's definately a reason to use rsync.

> It would certainly be annoying for a
> developer to go "svn commit", and have to wait for half an hour
> because everyone else is updating their local copies.  But, that could
> be solved by mirrors only getting updated once every day, at 12
> midnight.

Oh, yeah. Your midnight, my midnight? It would definately be annoying
to make a small glitch and have to wait >24hrs until the fix for that
gets promoted. The "problem" you mentioned that at some points there
are slightly errorneous ebuilds in portage or minor inconsistencies can
only be fixed by promoting updates fast.

The solution you propose costs a lot of CPU power, even more storage on
the mirrors and lacks some positive aspects that the current solution
has. Take a look at e.g. the major BSDs ports and package systems. They
certainly have similar problems.

OK, looking at the BSDs, I like the feature that there are branches
with the aim to build a package tree that is as consistent as possible.
That would be a plus. But that would imply a lot of work and a change
in ebuild maintainance. I don't see this coming soon for Gentoo.

-hwh
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Portage Storage using SVN

[Trenton Adams]

On 7/23/06, Hans-Werner Hilse <hilse@web.de> wrote:
> Hi,
>
> On Sun, 23 Jul 2006 02:42:43 -0600
> "Trenton Adams" <trenton.d.adams@gmail.com> wrote:
>
> > I proposed this awhile back, and got shot down.  At the time, the
> > arguments for using SVN for portage storage were pretty shallow, and
> > someone was able to easily shoot them down.  I believe I have come up
> > with better reasoning for using SVN.  Someone may still shoot them
> > down, but hey, it's worth a try.
>
> #1:
> You're aware that there's a CVS for portage, aren't you? I'm still not
> quite sure if you are suggesting using SVN for the portage mirrors and
> if you are suggesting that users also have a full SVN history on the
> clients, too?

No, not a full history, just a tag history.

>
> > PROBLEM 1
> > [...]
> > PROBLEM 2
> > [...]
> > PROBLEM 3
> > [...]
>
> Well, are those really problems at all? I mean, isn't it easy to
> overcome them? Is it worth dedicating time and work into that svn thing?

I'm not sure, is it?  Is there scripts already out there to overcome
the problems suggested?  If there are, I would sure appreciate knowing
about them. :)

It could be a lack of my understanding how the portage downgrade
process works.  But if you downgrade a package, will it downgrade all
the packages depending on that version as well?

>
> > POTENTIAL ISSUES
> > Now, I'm not entirely sure of the performance implications of
> > subversion for this purpose.  So, that would definitely have to either
> > be tested, or someone would have to talk with the subversion folks to
> > know if it would be a problem for thousands of users to access
> > subversion in readonly mode.
>
> Well, of course! There's definately a reason to use rsync.
>
> > It would certainly be annoying for a
> > developer to go "svn commit", and have to wait for half an hour
> > because everyone else is updating their local copies.  But, that could
> > be solved by mirrors only getting updated once every day, at 12
> > midnight.
>
> Oh, yeah. Your midnight, my midnight? It would definately be annoying
> to make a small glitch and have to wait >24hrs until the fix for that
> gets promoted. The "problem" you mentioned that at some points there
> are slightly errorneous ebuilds in portage or minor inconsistencies can
> only be fixed by promoting updates fast.

That's true, and I suppose that's not quite as good as what exists
right now.  But does gentoo really have less than 24 hour bug fix turn
arounds?
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Portage Storage using SVN

[Trenton Adams]

On 7/23/06, Neil Bothwick <neil@digimed.co.uk> wrote:
> On Sun, 23 Jul 2006 02:42:43 -0600, Trenton Adams wrote:
>
> > Let's say openldap had a problem. So, we decide to mask the latest
> > version of openldap, in an effort to roll back to the version that was
> > working.  Well, we find out that openldap still does not work.  So, we
> > finally determine that it is library W.  So, now we mask library W, in
> > an attempt to roll back to the version that was working.  Oh no, now
> > we find out that library W is used by 20 other packages, that require
> > the latest version of library W in order to work.  So, now we have to
> > mask library W, and 20 packages in order to get our openldap system
> > functional, assuming you cared about the 20 other broken packages,
> > which may break other packages, which may break yet other packages.
> >
> > Wouldn't it be nice to just go "emerge --revert-portage", which goes
> > back to the last exported copy of the portage, that you had from
> > subversion?  Boy, would that ever be convenient.  It would be simple
> > enough to store a local history of portage tags that the user was
> > using in the past.
>
> Wouldn't that revert the whole of portage, not just the parts affecting
> ldap, W and the rest? All of a sudden emerge world wants to downgrade
> previously upgraded programs, some of which may have been updated for
> security reasons. This means the user has to go through the Changelogs,
> ebuild and package.mask to find out why they are being downgraded and
> whether to apply the downgrade. this would be far more work that package
> masking a few updates dependent on W, which could probably be scripted.

Yes, it would revert the whole portage, and that would be desired, so
as to be up quickly.  It could potentially take hours to figure out
which packages are being affected by a broken updated package.

But, perhaps it would be better to write a utility that will downgrade
a package, as well as all dependencies.  Or does this already happen
when you downgrade?
>
>
> --
> Neil Bothwick
>
> BASIC: Bill's Attempt to Seize Industry Control
>
>
>
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Portage Storage using SVN

[Hemmann, Volker Armin]

On Sunday 23 July 2006 10:42, Trenton Adams wrote:
> Hi guys,
>
> I proposed this awhile back, and got shot down.  At the time, the
> arguments for using SVN for portage storage were pretty shallow, and
> someone was able to easily shoot them down.  I believe I have come up
> with better reasoning for using SVN.  Someone may still shoot them
> down, but hey, it's worth a try.
>
> PROBLEM 1
> Let's say openldap had a problem. So, we decide to mask the latest
> version of openldap, in an effort to roll back to the version that was
> working.  Well, we find out that openldap still does not work.  So, we
> finally determine that it is library W.  So, now we mask library W, in
> an attempt to roll back to the version that was working.  Oh no, now
> we find out that library W is used by 20 other packages, that require
> the latest version of library W in order to work.  So, now we have to
> mask library W, and 20 packages in order to get our openldap system
> functional, assuming you cared about the 20 other broken packages,
> which may break other packages, which may break yet other packages.

no, you don't. You put a dependency for libraryW version XY-working into the 
openldap related ebuilds.


>
> Wouldn't it be nice to just go "emerge --revert-portage", which goes
> back to the last exported copy of the portage, that you had from
> subversion?  Boy, would that ever be convenient.  It would be simple
> enough to store a local history of portage tags that the user was
> using in the past.

why not do a normal emerge sync&& emerge -au world?
the updated openldap package will pull in the right library W, and if other 
apps need library W+1, well that is where slots are used.

Your problem is none. It is solved every day within the portage tree. No svn 
needed for that.

>
> PROBLEM 3
> Don't sync more than once a day, or you may be temporarily banned?
> Well, with SVN being tagged only once a day, there would be no need to
> worry about this, seeing that

you are banned because of the enormous waste of bandwidth if you sync every 
odd hours. I don't see where svn would resolve that. Even if all mirrors 
would only updated once a day (what a nightmare - a broken package may stay 
for 24h or more in the tree, instead being replaced as soon as the dev 
notices), there would be people syncing more than once in 24h. 

-- 
gentoo-user@gentoo.org mailing list