[gentoo-dev] RFC: git-2.eclass & fetching from multiple repos

[gentoo-dev] RFC: git-2.eclass & fetching from multiple repos

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2070 bytes --]

Hello,

Sometimes it is necessary for a single package to pull from multiple
remote repositories. One case are broken packages like glibc (where
two repositories have to be merged for build-time), another are git
submodules (which don't cooperate well with bare checkouts).

Thus, I'm wondering about adding support for such a feature to
git-2.eclass. However, I don't have any concept for it yet so I'd
appreciate your ideas. What needs to be thought up is not only how to
do it but also what to do, exactly.

The simpler case would be only to provide support for fetching from
multiple repositories in git-2, and use it for submodules internally.
This would already fix glibc-9999 & smart-live-rebuild cooperation but
the glibc ebuild would still need to do a little hackery to checkout
the second repository.

The more complete case would be to completely support multiple
repostories but it would require a larger eclass rewrite, to support
hacking more of its variables.

Another question is how to implement it API-wide. The main problem here
is that we already use multiple values for EGIT_REPO_URI to support
fallback URIs, and I'd prefer supporting that for the additional repos
as well.


1/ Mike suggested something like:

EGIT_REPO_URI=( "repo1 fallback1" "repo2 fallback2" )

but IMO it hurts readability and makes the behavior a little surprising
(not that the fallback syntax isn't surprising already).


2/ Exherbo way would be doing something like:

EGIT_SECONDARY_REPOS='foo'
EGIT_foo_REPO_URI=...

but that would be real PITA for smart-live-rebuild, and I really hate
dynamically named variables.


3/ I'm wondering about something in-between those two, supposedly:

EGIT_SECONDARY_REPO_URIS=( "repo2 fallback2" ... )

Still painful for s-l-r but could be implemented without
eclass-specific hacks as array support. Of course, if we support this
not only for fetching but for checkouts as well, it's likely to get
unreadable as well...

What are your ideas?

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] RFC: git-2.eclass & fetching from multiple repos

[Luca Barbato]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 17/05/12 01:40, Michał Górny wrote:
> 
> 
> 1/ Mike suggested something like:
> 
> EGIT_REPO_URI=( "repo1 fallback1" "repo2 fallback2" )
> 
> but IMO it hurts readability and makes the behavior a little surprising
> (not that the fallback syntax isn't surprising already).

Does not look that bad, do you have an actual example so we can see how
bad could get?

lu

- -- 

Luca Barbato
Gentoo/linux
http://dev.gentoo.org/~lu_zero

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.18 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk+39EIACgkQ6Ex4woTpDjQ7WgCdHXC8qOKxKGTkZg0UdlXr2BA0
VoEAoLmcUx/0TfsrtKf7eiOxrLQMcpBj
=MUjk
-----END PGP SIGNATURE-----

Re: [gentoo-dev] RFC: git-2.eclass & fetching from multiple repos

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 948 bytes --]

On Sat, 19 May 2012 12:28:02 -0700
Luca Barbato <lu_zero@gentoo.org> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 17/05/12 01:40, Michał Górny wrote:
> > 
> > 
> > 1/ Mike suggested something like:
> > 
> > EGIT_REPO_URI=( "repo1 fallback1" "repo2 fallback2" )
> > 
> > but IMO it hurts readability and makes the behavior a little
> > surprising (not that the fallback syntax isn't surprising already).
> 
> Does not look that bad, do you have an actual example so we can see
> how bad could get?

Well, for example if the project was hosted on github, it would look
like:

EGIT_REPO_URI=(
	"git://github.com/foo/foo.git
	http://github.com/foo/foo.git"

	"git://github.com/foo/bar.git
	http://github.com/foo/bar.git"
)

Or something like that. What I am afraid of is that we use nested list
with tricky syntax, and one could miss the specifics of "" use.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] RFC: git-2.eclass & fetching from multiple repos

[Luca Barbato]

On 19/05/12 13:18, Michał Górny wrote:
> EGIT_REPO_URI=(
> 	"git://github.com/foo/foo.git
> 	http://github.com/foo/foo.git"
> 
> 	"git://github.com/foo/bar.git
> 	http://github.com/foo/bar.git"
> )
> 
> Or something like that. What I am afraid of is that we use nested list
> with tricky syntax, and one could miss the specifics of "" use.

it doesn't look that bad, the result would be getting

foo and bar in workdir I guess.

for github, sourceforge, bitbucket, gnome, etc, we might use something
like gitmirror://github/foo/bar

lu

-- 

Luca Barbato
Gentoo/linux
http://dev.gentoo.org/~lu_zero

[gentoo-dev] Re: RFC: git-2.eclass & fetching from multiple repos

[Steven J Long]

Michał Górny wrote:
> Sometimes it is necessary for a single package to pull from multiple
> remote repositories.
<snip> 
> Another question is how to implement it API-wide. The main problem here
> is that we already use multiple values for EGIT_REPO_URI to support
> fallback URIs, and I'd prefer supporting that for the additional repos
> as well.

So currently: EGIT_REPO_URI="uri fallbackA fallbackB fallbackC"
..where A, B and C are all fallbacks for uri.

> EGIT_REPO_URI=( "repo1 fallback1" "repo2 fallback2" )

Would require you to support both the above, and the new syntax if it's an 
array. Since this is new functionality, it seems wiser to prevent confusion 
between:
  EGIT_REPO_URI=(uri fallbackA fallbackB fallbackC)
and:
  EGIT_REPO_URI=(repo1 repo2 repo3 repo4)
..and also make the implementation simpler (ie quicker for metadata), by 
using a new variable.

Semantically as well, it's the git repo for the package: in most cases 
that's all that's required, so keep it simple for the majority usage (ie 
separate.)

> 2/ Exherbo way would be doing something like:
> 
EGIT_SECONDARY_REPOS='foo bar'
EGIT_foo_REPO_URI='repo1 fallback1'
EGIT_bar_REPO_URI='repo2 fallback2a fallback2b'
..
if use baz; then
   EGIT_SECONDARY_REPOS+=' baz'
   EGIT_baz_REPO_URI='repo3 fallback3'
fi

> but that would be real PITA for smart-live-rebuild, and I really hate
> dynamically named variables.
>
It's amateurishly-done as well, but there's no fixing the ugliness, nor a 
requirement for duplication, even if you fix the fail. I'd go into it, but 
technical posts I make, discussing implementation, seem to get taken as 
political, so I'll stop here, given the provenance, and stick to the RFC.

> 3/ I'm wondering about something in-between those two, supposedly:
> 
> EGIT_SECONDARY_REPO_URIS=( "repo2 fallback2" ... )
> 
> Still painful for s-l-r but could be implemented without
> eclass-specific hacks as array support. Of course, if we support this
> not only for fetching but for checkouts as well, it's likely to get
> unreadable as well...
> 
> What are your ideas?
>
Assuming an identifier/ tag as above is useful, since you can then 
clone to a named subdirectory, or fetch from a specific repo, as directed by 
the ebuild, or fetch all except the named ones, which might be fetched into 
a different base-directory when the caller has cd'ed there, etc, extend your 
last one with optional tags:

EGIT_SECONDARY=("foo@ repo1 fallback1" "repo2 fallback2a fallback2b")
..
use baz && EGIT_SECONDARY+=("baz@ repo3 fallback3")

So, fake associative arrays in a less cluttered fashion than 2/, that are 
simple to lex in-script, and don't lead to visual or mental overload.

Since tags are optional, this accepts the same format, and extends it only 
when the ebuild needs a tag, so simple builds stay simple. The @ symbol is 
not valid in url's iirc, leave alone git uris, and draws attention to the 
named items. It preserves ordering, which an associative array won't, and 
you only ever have to worry about one variable name. Nor is there any issue 
with making sure tags and variables are in sync. (DRY.)

The simple tag enables all the stuff I mentioned above, and by giving the 
ebuild author a symbolic reference which can be passed around, it can be 
extended ad-hoc.

-- 
#friendly-coders -- We're friendly, but we're not /that/ friendly ;-)