[gentoo-dev] git-r3: initial draft for review

[gentoo-dev] git-r3: initial draft for review

[Michał Górny]

[-- Attachment #1.1: Type: text/plain, Size: 6033 bytes --]

Hello, all.

After a few days of thinking, discovering and working, here it is.
The first working draft of new git eclass codenamed 'git-r3'.

First of all, the name is not final. I'm open to ideas. I'm open to
naming it 'git-r1' to put it in line with my other -r1 eclasses :).
I'd definitely like to avoid 'git-3' though, since that version-like
naming was a mistake as almost-'python-2' eclass shown.

Secondly, it's not even final that there will be a new eclass. Most
likely I will commit it as a new eclass since that way is easier for us
but if you prefer I may try to get it and git-2 more API-friendly
and work on making it a almost-drop-in replacement. Since, after all,
internals have actually changed much more than the API.


And now for the major changes:

1. The code has been split into clean 'fetch' and 'checkout' pieces.

That is, it is suited for distinct src_fetch() and src_unpack() phases
that we'll hopefully have in EAPI 6. What's important, the checkout
code does not rely on passing *any* environment variables from fetching
code. It is also made with concurrency in mind, so multiple ebuilds
using the same repository at the same time shouldn't be a problem.

2. Public fetch/checkout API.

git-2 has a lot of private functions and just src_unpack(). git-r3 has
git-r3_fetch() and git-r3_checkout() which are public API intended to
used in ebuilds that need more than plain fetch+unpack. While this
isn't exactly what multi-repo support pursuers wanted, it should make
supporting multiple repos in one ebuild much cleaner.

3. Clean submodule support with bare clones.

Since the submodules are very straightforward in design, I have decided
to move their support into the eclass directly. As a result, the new
eclass cleanly supports submodules, treating them as additional
repositories and doing submodule fetch/checkout recursively. There is
no need for non-bare clones anymore (and therefore their support has
been removed to make code simpler), and submodules work fine with
EVCS_OFFLINE=1.

4. 'Best-effort' shallow clones support.

I did my best to support shallow clones in the eclass. The code is
specifically designed to handle them whenever possible. However, since
shallow clones have a few limitations:

a) only branch/tag-based fetches support shallow clones. Fetching by
commit id forces complete clone (this is what submodules do BTW).

b) there's EGIT_NONSHALLOW option for users who prefer to have full
clones, and possibly for ebuilds that fail with shallow clones.

c) if shallow clones cause even more trouble than that, I will simply
remove their support from the eclass :).

[see notes about testing at the end]

5. Safer default EGIT_DIR choice. EGIT_PROJECT removed.

Since submodules are cloned as separate repositories as well, we can't
afford having EGIT_PROJECT to change the clone dir. Instead, the eclass
uses full path from first repo URI (with some preprocessing) to
determine the clone location. This should ensure non-colliding clones
with most likeliness that two ebuilds using the same repo will use
the same clone without any special effort from the maintainer.

6. Safer default checkout dir. EGIT_SOURCEDIR removed.

git-2 used to default EGIT_SOURCEDIR=${S}. This kinda sucked since if
one wanted to use subdirectory of the git repo, he needed to both set
EGIT_SOURCEDIR and S. Now, the checkout is done to ${WORKDIR}/${P}
by default and ebuilds can safely do S=${WORKDIR}/${P}/foo. I may
provide EGIT_SOURCEDIR if someone still finds it useful.


API/variables removed:

1. EGIT_SOURCEDIR:

a) if you need it for multiple repos, use the fetch/checkout functions
instead,

b) otherwise, play with S instead,

c) if you really need it, lemme know and I'll put it back.

2. EGIT_HAS_SUBMODULES -> no longer necessary, we autodetect them
(and we don't need that much special magic like we used to).

3. EGIT_OPTIONS -> interfered too much with eclass internals.

4. EGIT_MASTER -> people misused it all the time, and it caused issues
for projects that used different default branch. Now we just respect
upstream's default branch.

5. EGIT_PROJECT -> should be no longer necessary.

6. EGIT_DIR -> still exported, but no longer respects user setting it.

7. EGIT_REPACK, EGIT_PRUNE -> I will probably reintroduce it, or just
provide the ability to set git auto-cleanup options.

8. EGIT_NONBARE -> only bare clones are supported now.

9. EGIT_NOUNPACK -> git-2 is only eclass calling the default. Does
anyone actually need this? Is adding custom src_unpack() that hard?

10. EGIT_BOOTSTRAP -> this really belongs in *your* src_prepare().


I've tested the eclass on 113 live packages I'm using. Most of them
work just fine (I've replaced git-2 with the new eclass). Some
of them only require removing the old variables, some need having S
changed. However, I noticed the following issues as well:

1. code.google fails with 500 when trying to do a shallow clone
(probably they implemented their own dumb git server),

2. sys-apps/portage wants to play with 'git log' for ChangeLogs. That's
something that definitely is not going to work with shallow clones ;).
Not that I understand why someone would like a few megs of detailed git
backlog.

3. sys-fs/bedup's btrfs-progs submodule says the given commit id is
'not a valid branch point'. Need to investigate what this means.

4. 'git fetch --depth 1' seems to be refetching stuff even when nothing
changed. Need to investigate it. It may be enough to do an additional
'did anything change?' check.


I will try to look into those issues tomorrow. In the meantime, please
review this eclass and give me your thoughts. Especially if someone has
some more insight on shallow clones. Thanks.

And a fun fact: LLVM subversion checkout (in svn-src) has around ~2.4k
files which consume around 220M on btrfs. LLVM git shallow clone takes
17M.

-- 
Best regards,
Michał Górny

[-- Attachment #1.2: git-r3.eclass --]
[-- Type: text/plain, Size: 11013 bytes --]

# Copyright 1999-2013 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $

# @ECLASS: git-r3.eclass
# @MAINTAINER:
# Michał Górny <mgorny@gentoo.org>
# @BLURB: Eclass for fetching and unpacking git repositories.
# @DESCRIPTION:
# Third generation eclass for easing maitenance of live ebuilds using
# git as remote repository. Eclass supports lightweight (shallow)
# clones, local object deduplication and submodules.

case "${EAPI:-0}" in
	0|1|2|3|4|5)
		;;
	*)
		die "Unsupported EAPI=${EAPI} (unknown) for ${ECLASS}"
		;;
esac

if [[ ! ${_GIT_R3} ]]; then

inherit eutils

fi

EXPORT_FUNCTIONS src_unpack

if [[ ! ${_GIT_R3} ]]; then

# @ECLASS-VARIABLE: EGIT_STORE_DIR
# @DESCRIPTION:
# Storage directory for git sources.
#
# EGIT_STORE_DIR=${DISTDIR}/git3-src

# @ECLASS-VARIABLE: EGIT_REPO_URI
# @REQUIRED
# @DESCRIPTION:
# URIs to the repository, e.g. git://foo, https://foo. If multiple URIs
# are provided, the eclass will consider them as fallback URIs to try
# if the first URI does not work.
#
# It can be overriden via env using ${PN}_LIVE_REPO variable.
#
# Example:
# @CODE
# EGIT_REPO_URI="git://a/b.git https://c/d.git"
# @CODE

# @ECLASS-VARIABLE: EVCS_OFFLINE
# @DEFAULT_UNSET
# @DESCRIPTION:
# If non-empty, this variable prevents any online operations.

# @ECLASS-VARIABLE: EGIT_BRANCH
# @DEFAULT_UNSET
# @DESCRIPTION:
# The branch name to check out. If unset, the upstream default (HEAD)
# will be used.
#
# It can be overriden via env using ${PN}_LIVE_BRANCH variable.

# @ECLASS-VARIABLE: EGIT_COMMIT
# @DEFAULT_UNSET
# @DESCRIPTION:
# The tag name or commit identifier to check out. If unset, newest
# commit from the branch will be used. If set, EGIT_BRANCH will
# be ignored.
#
# It can be overriden via env using ${PN}_LIVE_COMMIT variable.

# @ECLASS-VARIABLE: EGIT_NONSHALLOW
# @DEFAULT_UNSET
# @DESCRIPTION:
# Disable performing shallow fetches/clones. Shallow clones have
# a fair number of limitations. Therefore, if you'd like the eclass to
# perform complete clones instead, set this to a non-null value.
#
# This variable is to be set in make.conf. Ebuilds are not allowed
# to set it.

# @FUNCTION: _git-r3_env_setup
# @INTERNAL
# @DESCRIPTION:
# Set the eclass variables as necessary for operation. This can involve
# setting EGIT_* to defaults or ${PN}_LIVE_* variables.
_git-r3_env_setup() {
	debug-print-function ${FUNCNAME} "$@"

	local esc_pn livevar
	esc_pn=${PN//[-+]/_}

	livevar=${esc_pn}_LIVE_REPO
	EGIT_REPO_URI=${!livevar:-${EGIT_REPO_URI}}
	[[ ${EGIT_REPO_URI} ]] \
		|| die "EGIT_REPO_URI must be set to a non-empty value"
	[[ ${!livevar} ]] \
		&& ewarn "Using ${livevar}, no support will be provided"

	livevar=${esc_pn}_LIVE_BRANCH
	EGIT_BRANCH=${!livevar:-${EGIT_BRANCH}}
	[[ ${!livevar} ]] \
		&& ewarn "Using ${livevar}, no support will be provided"

	livevar=${esc_pn}_LIVE_COMMIT
	EGIT_COMMIT=${!livevar:-${EGIT_COMMIT}}
	[[ ${!livevar} ]] \
		&& ewarn "Using ${livevar}, no support will be provided"

	# git-2 unsupported cruft
	local v
	for v in EGIT_{SOURCEDIR,MASTER,HAS_SUBMODULES,PROJECT} \
			EGIT_{NOUNPACK,BOOTSTRAP}
	do
		[[ ${!v} ]] && die "${v} is not supported."
	done
}

# @FUNCTION: _git-r3_set_gitdir
# @USAGE: <repo-uri>
# @INTERNAL
# @DESCRIPTION:
# Obtain the local repository path and set it as GIT_DIR. Creates
# a new repository if necessary.
#
# <repo-uri> may be used to compose the path. It should therefore be
# a canonical URI to the repository.
_git-r3_set_gitdir() {
	debug-print-function ${FUNCNAME} "$@"

	local repo_name=${1#*://*/}

	# strip common prefixes to make paths more likely to match
	# e.g. git://X/Y.git vs https://X/git/Y.git
	# (but just one of the prefixes)
	case "${repo_name}" in
		# cgit can proxy requests to git
		cgit/*) repo_name=${repo_name#cgit/};;
		# pretty common
		git/*) repo_name=${repo_name#git/};;
		# gentoo.org
		gitroot/*) repo_name=${repo_name#gitroot/};;
		# google code, sourceforge
		p/*) repo_name=${repo_name#p/};;
		# kernel.org
		pub/scm/*) repo_name=${repo_name#pub/scm/};;
	esac
	# ensure a .git suffix, same reason
	repo_name=${repo_name%.git}.git
	# now replace all the slashes
	repo_name=${repo_name//\//_}

	local distdir=${PORTAGE_ACTUAL_DISTDIR:-${DISTDIR}}
	: ${EGIT_STORE_DIR:=${distdir}/git3-src}

	GIT_DIR=${EGIT_STORE_DIR}/${repo_name}

	if [[ ! -d ${EGIT_STORE_DIR} ]]; then
		(
			addwrite /
			mkdir -m0755 -p "${EGIT_STORE_DIR}"
		) || die "Unable to create ${EGIT_STORE_DIR}"
	fi

	addwrite "${EGIT_STORE_DIR}"
	if [[ ! -d ${GIT_DIR} ]]; then
		mkdir "${GIT_DIR}" || die
		git init --bare || die
	fi
}

# @FUNCTION: _git-r3_set_submodules
# @USAGE: <file-contents>
# @INTERNAL
# @DESCRIPTION:
# Parse .gitmodules contents passed as <file-contents>
# as in "$(cat .gitmodules)"). Composes a 'submodules' array that
# contains in order (name, URL, path) for each submodule.
_git-r3_set_submodules() {
	debug-print-function ${FUNCNAME} "$@"

	local data=${1}

	# ( name url path ... )
	submodules=()

	local l
	while read l; do
		# submodule.<path>.path=<path>
		# submodule.<path>.url=<url>
		[[ ${l} == submodule.*.url=* ]] || continue

		l=${l#submodule.}
		local subname=${l%%.url=*}

		submodules+=(
			"${subname}"
			"$(echo "${data}" | git config -f /dev/fd/0 \
				submodule."${subname}".url)"
			"$(echo "${data}" | git config -f /dev/fd/0 \
				submodule."${subname}".path)"
		)
	done < <(echo "${data}" | git config -f /dev/fd/0 -l)
}

# @FUNCTION: git-r3_fetch
# @USAGE: <repo-uri> <remote-ref> <local-id>
# @DESCRIPTION:
# Fetch new commits to the local clone of repository. <repo-uri> follows
# the syntax of EGIT_REPO_URI and may list multiple (fallback) URIs.
# <remote-ref> specifies the remote ref to fetch (branch, tag
# or commit). <local-id> specifies an identifier that needs to uniquely
# identify the fetch operation in case multiple parallel merges used
# the git repo. <local-id> usually involves using CATEGORY, PN and SLOT.
#
# The fetch operation will only affect the local storage. It will not
# touch the working copy. If the repository contains submodules, they
# will be fetched recursively as well.
git-r3_fetch() {
	debug-print-function ${FUNCNAME} "$@"

	local repos=( ${1} )
	local remote_ref=${2}
	local local_id=${3}
	local local_ref=refs/heads/${local_id}/__main__

	local -x GIT_DIR
	_git-r3_set_gitdir ${repos[0]}

	# try to fetch from the remote
	local r success
	for r in ${repos[@]}; do
		einfo "Fetching ${remote_ref} from ${r} ..."

		# first, try ls-remote to see if ${remote_ref} is a real ref
		# and not a commit id. if it succeeds, we can pass ${remote_ref}
		# to 'fetch'. otherwise, we will just fetch everything

		# split on whitespace
		local ref=(
			$(git ls-remote "${r}" "${remote_ref}")
		)

		local ref_param=()
		if [[ ${ref[0]} ]]; then
			[[ ${EGIT_NONSHALLOW} ]] || ref_param+=( --depth 1 )
			ref_param+=( "${remote_ref}" )
		fi

		# if ${remote_ref} is branch or tag, ${ref[@]} will contain
		# the respective commit id. otherwise, it will be an empty
		# array, so the following won't evaluate to a parameter.
		if git fetch --no-tags "${r}" "${ref_param[@]}"; then
			if ! git branch -f "${local_id}/__main__" "${ref[0]:-${remote_ref}}"
			then
				die "Creating tag failed (${remote_ref} invalid?)"
			fi
			success=1
			break
		fi
	done
	[[ ${success} ]] || die "Unable to fetch from any of EGIT_REPO_URI"

	# recursively fetch submodules
	if git cat-file -e "${local_ref}":.gitmodules &>/dev/null; then
		local submodules
		_git-r3_set_submodules \
			"$(git cat-file -p "${local_ref}":.gitmodules || die)"

		while [[ ${submodules[@]} ]]; do
			local subname=${submodules[0]}
			local url=${submodules[1]}
			local path=${submodules[2]}
			local commit=$(git rev-parse "${local_ref}:${path}")

			if [[ ! ${commit} ]]; then
				die "Unable to get commit id for submodule ${subname}"
			fi

			git-r3_fetch "${url}" "${commit}" "${local_id}/${subname}"

			submodules=( "${submodules[@]:3}" ) # shift
		done
	fi
}

# @FUNCTION: git-r3_checkout
# @USAGE: <repo-uri> <local-id> <path>
# @DESCRIPTION:
# Check the previously fetched commit out to <path> (usually
# ${WORKDIR}/${P}). <repo-uri> follows the syntax of EGIT_REPO_URI
# and will be used to re-construct the local storage path. <local-id>
# is the unique identifier used for the fetch operation and will
# be used to obtain the proper commit.
#
# If the repository contains submodules, they will be checked out
# recursively as well.
git-r3_checkout() {
	debug-print-function ${FUNCNAME} "$@"

	local repos=( ${1} )
	local local_id=${2}
	local out_dir=${3}

	local -x GIT_DIR GIT_WORK_TREE
	_git-r3_set_gitdir ${repos[0]}
	GIT_WORK_TREE=${out_dir}

	einfo "Checking out ${repos[0]} to ${out_dir} ..."

	mkdir -p "${GIT_WORK_TREE}"
	git checkout -f "${local_id}"/__main__ || die

	# diff against previous revision (if any)
	local new_commit_id=$(git rev-parse --verify "${local_id}"/__main__)
	local old_commit_id=$(
		git rev-parse --verify "${local_id}"/__old__ 2>/dev/null
	)

	if [[ ! ${old_commit_id} ]]; then
		echo "GIT NEW branch -->"
		echo "   repository:               ${repos[0]}"
		echo "   at the commit:            ${new_commit_id}"
	else
		echo "GIT update -->"
		echo "   repository:               ${repos[0]}"
		# write out message based on the revisions
		if [[ "${old_commit_id}" != "${new_commit_id}" ]]; then
			echo "   updating from commit:     ${old_commit_id}"
			echo "   to commit:                ${new_commit_id}"
		else
			echo "   at the commit:            ${new_commit_id}"
		fi
	fi
	git branch -f "${local_id}"/{__old__,__main__} || die

	# recursively checkout submodules
	if [[ -f ${GIT_WORK_TREE}/.gitmodules ]]; then
		local submodules
		_git-r3_set_submodules \
			"$(cat "${GIT_WORK_TREE}"/.gitmodules)"

		while [[ ${submodules[@]} ]]; do
			local subname=${submodules[0]}
			local url=${submodules[1]}
			local path=${submodules[2]}

			git-r3_checkout "${url}" "${local_id}/${subname}" \
				"${GIT_WORK_TREE}/${path}"

			submodules=( "${submodules[@]:3}" ) # shift
		done
	fi

	# keep this *after* submodules
	export EGIT_DIR=${GIT_DIR}
	export EGIT_VERSION=${new_commit_id}
}

git-r3_src_fetch() {
	debug-print-function ${FUNCNAME} "$@"

	[[ ${EVCS_OFFLINE} ]] && return

	_git-r3_env_setup
	local branch=${EGIT_BRANCH:+refs/heads/${EGIT_BRANCH}}
	git-r3_fetch "${EGIT_REPO_URI}" \
		"${EGIT_COMMIT:-${branch:-HEAD}}" \
		${CATEGORY}/${PN}/${SLOT}
}

git-r3_src_unpack() {
	debug-print-function ${FUNCNAME} "$@"

	_git-r3_env_setup
	git-r3_src_fetch
	git-r3_checkout "${EGIT_REPO_URI}" \
		${CATEGORY}/${PN}/${SLOT} \
		"${WORKDIR}/${P}"
}

_GIT_R3=1
fi

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] git-r3: initial draft for review

["C. Bergström"]

Do you have any plans to add support for sparse checkout?

Something like this

|cd <parentdir>
git clone -n <url>
cd <repo_dir>
git remote add –f <name> <url>
git config core.sparsecheckout true
echo /<foldername>/ >> .git/info/sparse-checkout
git checkout <tagname>

(Credit goes to : http://stackoverflow.com/questions/15827117/git-sparse-checkout-for-simple-web-deployment )
|

--------
What about subtrees?
--------
Nice work!

Re: [gentoo-dev] git-r3: initial draft for review

[Markos Chandras]

On 31 August 2013 00:37, Michał Górny <mgorny@gentoo.org> wrote:
> Hello, all.
>
> After a few days of thinking, discovering and working, here it is.
> The first working draft of new git eclass codenamed 'git-r3'.
>
> First of all, the name is not final. I'm open to ideas. I'm open to
> naming it 'git-r1' to put it in line with my other -r1 eclasses :).
> I'd definitely like to avoid 'git-3' though, since that version-like
> naming was a mistake as almost-'python-2' eclass shown.
>
> Secondly, it's not even final that there will be a new eclass. Most
> likely I will commit it as a new eclass since that way is easier for us
> but if you prefer I may try to get it and git-2 more API-friendly
> and work on making it a almost-drop-in replacement. Since, after all,
> internals have actually changed much more than the API.
>

Hi,

I have no capacity to review the eclass right now (although I promise
I will soon)
but I was wondering why the whole eclass is inside the " if [[ !
${_GIT_R3} ]] " block.
Is it so you can avoid inheriting the same eclass twice? I haven't
seen that before
so I was wondering if that's the only reason. If yes, then maybe this
needs to be solved in PM scope instead?
As in, prevent portage from inheriting the same eclass twice instead
of handling this case in the eclass itself.

-- 
Regards,
Markos Chandras - Gentoo Linux Developer
http://dev.gentoo.org/~hwoarang

Re: [gentoo-dev] git-r3: initial draft for review

[Rick "Zero_Chaos" Farina]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 08/30/2013 07:37 PM, Michał Górny wrote:
> Hello, all.
> 
> After a few days of thinking, discovering and working, here it is.
> The first working draft of new git eclass codenamed 'git-r3'.
> 
> First of all, the name is not final. I'm open to ideas. I'm open to
> naming it 'git-r1' to put it in line with my other -r1 eclasses :).
> I'd definitely like to avoid 'git-3' though, since that version-like
> naming was a mistake as almost-'python-2' eclass shown.
> 
Seems to be a lot of schizophrenia in versioning here.  I can honestly
tell you that git-2 is greater than git-r3, so I really caution against
this.

> Secondly, it's not even final that there will be a new eclass. Most
> likely I will commit it as a new eclass since that way is easier for us
> but if you prefer I may try to get it and git-2 more API-friendly
> and work on making it a almost-drop-in replacement. Since, after all,
> internals have actually changed much more than the API.
> 
> 
> And now for the major changes:
> 
> 1. The code has been split into clean 'fetch' and 'checkout' pieces.
> 
> That is, it is suited for distinct src_fetch() and src_unpack() phases
> that we'll hopefully have in EAPI 6. What's important, the checkout
> code does not rely on passing *any* environment variables from fetching
> code. It is also made with concurrency in mind, so multiple ebuilds
> using the same repository at the same time shouldn't be a problem.
> 
> 2. Public fetch/checkout API.
> 
> git-2 has a lot of private functions and just src_unpack(). git-r3 has
> git-r3_fetch() and git-r3_checkout() which are public API intended to
> used in ebuilds that need more than plain fetch+unpack. While this
> isn't exactly what multi-repo support pursuers wanted, it should make
> supporting multiple repos in one ebuild much cleaner.
> 
> 3. Clean submodule support with bare clones.
> 
> Since the submodules are very straightforward in design, I have decided
> to move their support into the eclass directly. As a result, the new
> eclass cleanly supports submodules, treating them as additional
> repositories and doing submodule fetch/checkout recursively. There is
> no need for non-bare clones anymore (and therefore their support has
> been removed to make code simpler), and submodules work fine with
> EVCS_OFFLINE=1.
> 
> 4. 'Best-effort' shallow clones support.
> 
> I did my best to support shallow clones in the eclass. The code is
> specifically designed to handle them whenever possible. However, since
> shallow clones have a few limitations:
> 
> a) only branch/tag-based fetches support shallow clones. Fetching by
> commit id forces complete clone (this is what submodules do BTW).
> 
> b) there's EGIT_NONSHALLOW option for users who prefer to have full
> clones, and possibly for ebuilds that fail with shallow clones.
> 
> c) if shallow clones cause even more trouble than that, I will simply
> remove their support from the eclass :).
> 
> [see notes about testing at the end]
> 
> 5. Safer default EGIT_DIR choice. EGIT_PROJECT removed.
> 
> Since submodules are cloned as separate repositories as well, we can't
> afford having EGIT_PROJECT to change the clone dir. Instead, the eclass
> uses full path from first repo URI (with some preprocessing) to
> determine the clone location. This should ensure non-colliding clones
> with most likeliness that two ebuilds using the same repo will use
> the same clone without any special effort from the maintainer.
> 
> 6. Safer default checkout dir. EGIT_SOURCEDIR removed.
> 
> git-2 used to default EGIT_SOURCEDIR=${S}. This kinda sucked since if
> one wanted to use subdirectory of the git repo, he needed to both set
> EGIT_SOURCEDIR and S. Now, the checkout is done to ${WORKDIR}/${P}
> by default and ebuilds can safely do S=${WORKDIR}/${P}/foo. I may
> provide EGIT_SOURCEDIR if someone still finds it useful.

Thank you so much. I've wanted to do this forever.
> 
> 
> API/variables removed:
> 
> 1. EGIT_SOURCEDIR:
> 
> a) if you need it for multiple repos, use the fetch/checkout functions
> instead,
> 
> b) otherwise, play with S instead,
> 
> c) if you really need it, lemme know and I'll put it back.
> 
> 2. EGIT_HAS_SUBMODULES -> no longer necessary, we autodetect them
> (and we don't need that much special magic like we used to).
> 
> 3. EGIT_OPTIONS -> interfered too much with eclass internals.
> 
> 4. EGIT_MASTER -> people misused it all the time, and it caused issues
> for projects that used different default branch. Now we just respect
> upstream's default branch.
> 
> 5. EGIT_PROJECT -> should be no longer necessary.
> 
How so?

Thanks,
Zero

> 6. EGIT_DIR -> still exported, but no longer respects user setting it.
> 
> 7. EGIT_REPACK, EGIT_PRUNE -> I will probably reintroduce it, or just
> provide the ability to set git auto-cleanup options.
> 
> 8. EGIT_NONBARE -> only bare clones are supported now.
> 
> 9. EGIT_NOUNPACK -> git-2 is only eclass calling the default. Does
> anyone actually need this? Is adding custom src_unpack() that hard?
> 
> 10. EGIT_BOOTSTRAP -> this really belongs in *your* src_prepare().
> 
> 
> I've tested the eclass on 113 live packages I'm using. Most of them
> work just fine (I've replaced git-2 with the new eclass). Some
> of them only require removing the old variables, some need having S
> changed. However, I noticed the following issues as well:
> 
> 1. code.google fails with 500 when trying to do a shallow clone
> (probably they implemented their own dumb git server),
> 
> 2. sys-apps/portage wants to play with 'git log' for ChangeLogs. That's
> something that definitely is not going to work with shallow clones ;).
> Not that I understand why someone would like a few megs of detailed git
> backlog.
> 
> 3. sys-fs/bedup's btrfs-progs submodule says the given commit id is
> 'not a valid branch point'. Need to investigate what this means.
> 
> 4. 'git fetch --depth 1' seems to be refetching stuff even when nothing
> changed. Need to investigate it. It may be enough to do an additional
> 'did anything change?' check.
> 
> 
> I will try to look into those issues tomorrow. In the meantime, please
> review this eclass and give me your thoughts. Especially if someone has
> some more insight on shallow clones. Thanks.
> 
> And a fun fact: LLVM subversion checkout (in svn-src) has around ~2.4k
> files which consume around 220M on btrfs. LLVM git shallow clone takes
> 17M.
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.20 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJSIYNAAAoJEKXdFCfdEflKBdoP/Ru+/12Ypn6lg0h55tI14uiX
xzoev+DN1KK19oMiYZxZL4zkiCp9/YSvTSKEcVpokzYIxQyWU+stJ6uLQYFsxovh
lSP+9pOlMikS0xdjDudFjRcS0GMhZjGytrk/4IYmALxRnuKtAVM7XzEZe6vba4Vm
EJ4CSOCRoA4X0Kh49lSn/LhONUfm9PBrTQVkmOYSNZzPOue9WUP5ppIc7212w0x2
+67lq5hxR3LJItpDgX++sOErlDCF9O+eZe+j2JHklsHMYsKJPdScY3Xad9QBRXGa
rpymloDg5MMe7c079H59nCIRC7O5StXRZmiR6skdZDz/gMHMEZ15saQDYM2i3x/A
obbbjINl7MuW576YeHhfGzQhzYf4+41PuGSIyJaHQ7fuvL3l4jWbhYngcVP2WYBi
xudVKQei0eW7W2m5dGAeLD6IDC35PnxUvoWc69iRgDpn6OADkZ/bx4XD3Dx2Rqb9
6tkU4g9zIrlOG+ywlGzlBpDjKctC9AhBqh5SUJuM3WMGUfPUcTYfx+ENblfvTRAT
iz0gJJu62CumioMAx9aQGe3Dnr0f8m5COsPs79WmP+3g8Pm0LwswMx3agQEM5QWV
QFBh0189DNfwHR6GVt03EfxYUZx8Mm7uz+QyNfMJY/2fFPmJnqN4VctegWUn/FYw
8h0JTHjBC2dIAZwoQCvN
=1ESQ
-----END PGP SIGNATURE-----

Re: [gentoo-dev] git-r3: initial draft for review

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 777 bytes --]

Dnia 2013-08-31, o godz. 06:55:01
"C. Bergström" <cbergstrom@pathscale.com> napisał(a):

> Do you have any plans to add support for sparse checkout?
> 
> Something like this
> 
> |cd <parentdir>
> git clone -n <url>
> cd <repo_dir>
> git remote add –f <name> <url>
> git config core.sparsecheckout true
> echo /<foldername>/ >> .git/info/sparse-checkout
> git checkout <tagname>

Considering our specific work flow, I think it would be enough to do:

  git checkout <branch> [<dir1> <dir2> ...]

Depending on the need for it, I can either add it as optional arguments
to git_checkout or an envvar for it.


> --------
> What about subtrees?

Looking at their specific design, don't they work out-of-the-box?

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] git-r3: initial draft for review

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2001 bytes --]

Dnia 2013-08-31, o godz. 01:17:26
Markos Chandras <hwoarang@gentoo.org> napisał(a):

> On 31 August 2013 00:37, Michał Górny <mgorny@gentoo.org> wrote:
> > Hello, all.
> >
> > After a few days of thinking, discovering and working, here it is.
> > The first working draft of new git eclass codenamed 'git-r3'.
> >
> > First of all, the name is not final. I'm open to ideas. I'm open to
> > naming it 'git-r1' to put it in line with my other -r1 eclasses :).
> > I'd definitely like to avoid 'git-3' though, since that version-like
> > naming was a mistake as almost-'python-2' eclass shown.
> >
> > Secondly, it's not even final that there will be a new eclass. Most
> > likely I will commit it as a new eclass since that way is easier for us
> > but if you prefer I may try to get it and git-2 more API-friendly
> > and work on making it a almost-drop-in replacement. Since, after all,
> > internals have actually changed much more than the API.
> >
> 
> Hi,
> 
> I have no capacity to review the eclass right now (although I promise
> I will soon)
> but I was wondering why the whole eclass is inside the " if [[ !
> ${_GIT_R3} ]] " block.
> Is it so you can avoid inheriting the same eclass twice? I haven't
> seen that before
> so I was wondering if that's the only reason. If yes, then maybe this
> needs to be solved in PM scope instead?
> As in, prevent portage from inheriting the same eclass twice instead
> of handling this case in the eclass itself.

Yep, it's exactly that. All my new eclasses use that, and vapier's use
rather some 'spank' serpentines. Plus mine have 'EXPORT_FUNCTIONS' out
of it, so wrapping them does not change export order.

I suggested that into EAPI [1] but it didn't get in so far. The issue
is that some eclasses may actually need to be sourced twice, e.g. due
to environment changing. Though they are nowhere near 'sane' then.

[1]:https://bugs.gentoo.org/show_bug.cgi?id=422533

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] git-r3: initial draft for review

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 823 bytes --]

Dnia 2013-08-31, o godz. 01:46:40
"Rick \"Zero_Chaos\" Farina" <zerochaos@gentoo.org> napisał(a):

> > 5. Safer default EGIT_DIR choice. EGIT_PROJECT removed.
> > 
> > Since submodules are cloned as separate repositories as well, we can't
> > afford having EGIT_PROJECT to change the clone dir. Instead, the eclass
> > uses full path from first repo URI (with some preprocessing) to
> > determine the clone location. This should ensure non-colliding clones
> > with most likeliness that two ebuilds using the same repo will use
> > the same clone without any special effort from the maintainer.
> > 

[snip]

> > 5. EGIT_PROJECT -> should be no longer necessary.
> > 
> How so?

Point 5. over there :). It is supposed to choose perfect clone location
itself now.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] git-r3: initial draft for review [v2]

[Michał Górny]

[-- Attachment #1.1: Type: text/plain, Size: 798 bytes --]

And time for a small update. 

Dnia 2013-08-31, o godz. 01:37:44
Michał Górny <mgorny@gentoo.org> napisał(a):

> 3. sys-fs/bedup's btrfs-progs submodule says the given commit id is
> 'not a valid branch point'. Need to investigate what this means.
> 
> 4. 'git fetch --depth 1' seems to be refetching stuff even when nothing
> changed. Need to investigate it. It may be enough to do an additional
> 'did anything change?' check.

Those are fixed now. The eclass has been adjusted to work properly with
branches, tags and commit ids. It also avoids shallowing repo that was
unshallowed already (since we expect that one of the packages needs it
non-shallow), and properly unshallows shallow repos ;).

Attaching the new version and a diff.

-- 
Best regards,
Michał Górny

[-- Attachment #1.2: git-r3.eclass --]
[-- Type: text/plain, Size: 12309 bytes --]

# Copyright 1999-2013 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $

# @ECLASS: git-r3.eclass
# @MAINTAINER:
# Michał Górny <mgorny@gentoo.org>
# @BLURB: Eclass for fetching and unpacking git repositories.
# @DESCRIPTION:
# Third generation eclass for easing maitenance of live ebuilds using
# git as remote repository. Eclass supports lightweight (shallow)
# clones, local object deduplication and submodules.

case "${EAPI:-0}" in
	0|1|2|3|4|5)
		;;
	*)
		die "Unsupported EAPI=${EAPI} (unknown) for ${ECLASS}"
		;;
esac

if [[ ! ${_GIT_R3} ]]; then

inherit eutils

fi

EXPORT_FUNCTIONS src_unpack

if [[ ! ${_GIT_R3} ]]; then

# @ECLASS-VARIABLE: EGIT_STORE_DIR
# @DESCRIPTION:
# Storage directory for git sources.
#
# EGIT_STORE_DIR=${DISTDIR}/git3-src

# @ECLASS-VARIABLE: EGIT_REPO_URI
# @REQUIRED
# @DESCRIPTION:
# URIs to the repository, e.g. git://foo, https://foo. If multiple URIs
# are provided, the eclass will consider them as fallback URIs to try
# if the first URI does not work.
#
# It can be overriden via env using ${PN}_LIVE_REPO variable.
#
# Example:
# @CODE
# EGIT_REPO_URI="git://a/b.git https://c/d.git"
# @CODE

# @ECLASS-VARIABLE: EVCS_OFFLINE
# @DEFAULT_UNSET
# @DESCRIPTION:
# If non-empty, this variable prevents any online operations.

# @ECLASS-VARIABLE: EGIT_BRANCH
# @DEFAULT_UNSET
# @DESCRIPTION:
# The branch name to check out. If unset, the upstream default (HEAD)
# will be used.
#
# It can be overriden via env using ${PN}_LIVE_BRANCH variable.

# @ECLASS-VARIABLE: EGIT_COMMIT
# @DEFAULT_UNSET
# @DESCRIPTION:
# The tag name or commit identifier to check out. If unset, newest
# commit from the branch will be used. If set, EGIT_BRANCH will
# be ignored.
#
# It can be overriden via env using ${PN}_LIVE_COMMIT variable.

# @ECLASS-VARIABLE: EGIT_NONSHALLOW
# @DEFAULT_UNSET
# @DESCRIPTION:
# Disable performing shallow fetches/clones. Shallow clones have
# a fair number of limitations. Therefore, if you'd like the eclass to
# perform complete clones instead, set this to a non-null value.
#
# This variable is to be set in make.conf. Ebuilds are not allowed
# to set it.

# @FUNCTION: _git-r3_env_setup
# @INTERNAL
# @DESCRIPTION:
# Set the eclass variables as necessary for operation. This can involve
# setting EGIT_* to defaults or ${PN}_LIVE_* variables.
_git-r3_env_setup() {
	debug-print-function ${FUNCNAME} "$@"

	local esc_pn livevar
	esc_pn=${PN//[-+]/_}

	livevar=${esc_pn}_LIVE_REPO
	EGIT_REPO_URI=${!livevar:-${EGIT_REPO_URI}}
	[[ ${EGIT_REPO_URI} ]] \
		|| die "EGIT_REPO_URI must be set to a non-empty value"
	[[ ${!livevar} ]] \
		&& ewarn "Using ${livevar}, no support will be provided"

	livevar=${esc_pn}_LIVE_BRANCH
	EGIT_BRANCH=${!livevar:-${EGIT_BRANCH}}
	[[ ${!livevar} ]] \
		&& ewarn "Using ${livevar}, no support will be provided"

	livevar=${esc_pn}_LIVE_COMMIT
	EGIT_COMMIT=${!livevar:-${EGIT_COMMIT}}
	[[ ${!livevar} ]] \
		&& ewarn "Using ${livevar}, no support will be provided"

	# git-2 unsupported cruft
	local v
	for v in EGIT_{SOURCEDIR,MASTER,HAS_SUBMODULES,PROJECT} \
			EGIT_{NOUNPACK,BOOTSTRAP}
	do
		[[ ${!v} ]] && die "${v} is not supported."
	done
}

# @FUNCTION: _git-r3_set_gitdir
# @USAGE: <repo-uri>
# @INTERNAL
# @DESCRIPTION:
# Obtain the local repository path and set it as GIT_DIR. Creates
# a new repository if necessary.
#
# <repo-uri> may be used to compose the path. It should therefore be
# a canonical URI to the repository.
_git-r3_set_gitdir() {
	debug-print-function ${FUNCNAME} "$@"

	local repo_name=${1#*://*/}

	# strip common prefixes to make paths more likely to match
	# e.g. git://X/Y.git vs https://X/git/Y.git
	# (but just one of the prefixes)
	case "${repo_name}" in
		# cgit can proxy requests to git
		cgit/*) repo_name=${repo_name#cgit/};;
		# pretty common
		git/*) repo_name=${repo_name#git/};;
		# gentoo.org
		gitroot/*) repo_name=${repo_name#gitroot/};;
		# google code, sourceforge
		p/*) repo_name=${repo_name#p/};;
		# kernel.org
		pub/scm/*) repo_name=${repo_name#pub/scm/};;
	esac
	# ensure a .git suffix, same reason
	repo_name=${repo_name%.git}.git
	# now replace all the slashes
	repo_name=${repo_name//\//_}

	local distdir=${PORTAGE_ACTUAL_DISTDIR:-${DISTDIR}}
	: ${EGIT_STORE_DIR:=${distdir}/git3-src}

	GIT_DIR=${EGIT_STORE_DIR}/${repo_name}

	if [[ ! -d ${EGIT_STORE_DIR} ]]; then
		(
			addwrite /
			mkdir -m0755 -p "${EGIT_STORE_DIR}"
		) || die "Unable to create ${EGIT_STORE_DIR}"
	fi

	addwrite "${EGIT_STORE_DIR}"
	if [[ ! -d ${GIT_DIR} ]]; then
		mkdir "${GIT_DIR}" || die
		git init --bare || die

		# avoid auto-unshallow :)
		touch "${GIT_DIR}"/shallow || die
	fi
}

# @FUNCTION: _git-r3_set_submodules
# @USAGE: <file-contents>
# @INTERNAL
# @DESCRIPTION:
# Parse .gitmodules contents passed as <file-contents>
# as in "$(cat .gitmodules)"). Composes a 'submodules' array that
# contains in order (name, URL, path) for each submodule.
_git-r3_set_submodules() {
	debug-print-function ${FUNCNAME} "$@"

	local data=${1}

	# ( name url path ... )
	submodules=()

	local l
	while read l; do
		# submodule.<path>.path=<path>
		# submodule.<path>.url=<url>
		[[ ${l} == submodule.*.url=* ]] || continue

		l=${l#submodule.}
		local subname=${l%%.url=*}

		submodules+=(
			"${subname}"
			"$(echo "${data}" | git config -f /dev/fd/0 \
				submodule."${subname}".url)"
			"$(echo "${data}" | git config -f /dev/fd/0 \
				submodule."${subname}".path)"
		)
	done < <(echo "${data}" | git config -f /dev/fd/0 -l)
}

# @FUNCTION: git-r3_fetch
# @USAGE: <repo-uri> <remote-ref> <local-id>
# @DESCRIPTION:
# Fetch new commits to the local clone of repository. <repo-uri> follows
# the syntax of EGIT_REPO_URI and may list multiple (fallback) URIs.
# <remote-ref> specifies the remote ref to fetch (branch, tag
# or commit). <local-id> specifies an identifier that needs to uniquely
# identify the fetch operation in case multiple parallel merges used
# the git repo. <local-id> usually involves using CATEGORY, PN and SLOT.
#
# The fetch operation will only affect the local storage. It will not
# touch the working copy. If the repository contains submodules, they
# will be fetched recursively as well.
git-r3_fetch() {
	debug-print-function ${FUNCNAME} "$@"

	local repos=( ${1} )
	local remote_ref=${2}
	local local_id=${3}
	local local_ref=refs/heads/${local_id}/__main__

	local -x GIT_DIR
	_git-r3_set_gitdir ${repos[0]}

	# try to fetch from the remote
	local r success
	for r in ${repos[@]}; do
		einfo "Fetching ${remote_ref} from ${r} ..."

		local is_branch lookup_ref
		if [[ ${remote_ref} == refs/heads/* || ${remote_ref} == HEAD ]]
		then
			is_branch=1
			lookup_ref=${remote_ref}
		else
			# ls-remote by commit is going to fail anyway,
			# so we may as well pass refs/tags/ABCDEF...
			lookup_ref=refs/tags/${remote_ref}
		fi

		# first, try ls-remote to see if ${remote_ref} is a real ref
		# and not a commit id. if it succeeds, we can pass ${remote_ref}
		# to 'fetch'. otherwise, we will just fetch everything

		# split on whitespace
		local ref=(
			$(git ls-remote "${r}" "${lookup_ref}")
		)

		# now, another important thing. we may only fetch a remote
		# branch directly to a local branch. Otherwise, we need to fetch
		# the commit and re-create the branch on top of it.

		local ref_param=()
		if [[ ! ${ref[0]} ]]; then
			local EGIT_NONSHALLOW=1
		fi

		if [[ ! -f ${GIT_DIR}/shallow ]]; then
			# if it's a complete repo, fetch it as-is
			:
		elif [[ ${EGIT_NONSHALLOW} ]]; then
			# if it's a shallow clone but we need complete,
			# unshallow it
			ref_param+=( --unshallow )
		else
			# otherwise, just fetch as shallow
			ref_param+=( --depth 1 )
		fi

		if [[ ${ref[0]} ]]; then
			if [[ ${is_branch} ]]; then
				ref_param+=( -f "${remote_ref}:${local_id}/__main__" )
			else
				ref_param+=( "${remote_ref}" )
			fi
		fi

		# if ${remote_ref} is branch or tag, ${ref[@]} will contain
		# the respective commit id. otherwise, it will be an empty
		# array, so the following won't evaluate to a parameter.
		set -- git fetch --no-tags "${r}" "${ref_param[@]}"
		echo "${@}" >&2
		if "${@}"; then
			if [[ ! ${is_branch} ]]; then
				set -- git branch -f "${local_id}/__main__" \
					"${ref[0]:-${remote_ref}}"
				echo "${@}" >&2
				if ! "${@}"; then
					die "Creating branch for ${remote_ref} failed (wrong ref?)."
				fi
			fi

			success=1
			break
		fi
	done
	[[ ${success} ]] || die "Unable to fetch from any of EGIT_REPO_URI"

	# recursively fetch submodules
	if git cat-file -e "${local_ref}":.gitmodules &>/dev/null; then
		local submodules
		_git-r3_set_submodules \
			"$(git cat-file -p "${local_ref}":.gitmodules || die)"

		while [[ ${submodules[@]} ]]; do
			local subname=${submodules[0]}
			local url=${submodules[1]}
			local path=${submodules[2]}
			local commit=$(git rev-parse "${local_ref}:${path}")

			if [[ ! ${commit} ]]; then
				die "Unable to get commit id for submodule ${subname}"
			fi

			git-r3_fetch "${url}" "${commit}" "${local_id}/${subname}"

			submodules=( "${submodules[@]:3}" ) # shift
		done
	fi
}

# @FUNCTION: git-r3_checkout
# @USAGE: <repo-uri> <local-id> <path>
# @DESCRIPTION:
# Check the previously fetched commit out to <path> (usually
# ${WORKDIR}/${P}). <repo-uri> follows the syntax of EGIT_REPO_URI
# and will be used to re-construct the local storage path. <local-id>
# is the unique identifier used for the fetch operation and will
# be used to obtain the proper commit.
#
# If the repository contains submodules, they will be checked out
# recursively as well.
git-r3_checkout() {
	debug-print-function ${FUNCNAME} "$@"

	local repos=( ${1} )
	local local_id=${2}
	local out_dir=${3}

	local -x GIT_DIR GIT_WORK_TREE
	_git-r3_set_gitdir ${repos[0]}
	GIT_WORK_TREE=${out_dir}

	einfo "Checking out ${repos[0]} to ${out_dir} ..."

	mkdir -p "${GIT_WORK_TREE}"
	set -- git checkout -f "${local_id}"/__main__ .
	echo "${@}" >&2
	"${@}" || die "git checkout ${local_id}/__main__ failed"

	# diff against previous revision (if any)
	local new_commit_id=$(git rev-parse --verify "${local_id}"/__main__)
	local old_commit_id=$(
		git rev-parse --verify "${local_id}"/__old__ 2>/dev/null
	)

	if [[ ! ${old_commit_id} ]]; then
		echo "GIT NEW branch -->"
		echo "   repository:               ${repos[0]}"
		echo "   at the commit:            ${new_commit_id}"
	else
		echo "GIT update -->"
		echo "   repository:               ${repos[0]}"
		# write out message based on the revisions
		if [[ "${old_commit_id}" != "${new_commit_id}" ]]; then
			echo "   updating from commit:     ${old_commit_id}"
			echo "   to commit:                ${new_commit_id}"

			git --no-pager diff --color --stat \
				${old_commit_id}..${new_commit_id}
		else
			echo "   at the commit:            ${new_commit_id}"
		fi
	fi
	git branch -f "${local_id}"/{__old__,__main__} || die

	# recursively checkout submodules
	if [[ -f ${GIT_WORK_TREE}/.gitmodules ]]; then
		local submodules
		_git-r3_set_submodules \
			"$(cat "${GIT_WORK_TREE}"/.gitmodules)"

		while [[ ${submodules[@]} ]]; do
			local subname=${submodules[0]}
			local url=${submodules[1]}
			local path=${submodules[2]}

			git-r3_checkout "${url}" "${local_id}/${subname}" \
				"${GIT_WORK_TREE}/${path}"

			submodules=( "${submodules[@]:3}" ) # shift
		done
	fi

	# keep this *after* submodules
	export EGIT_DIR=${GIT_DIR}
	export EGIT_VERSION=${new_commit_id}
}

git-r3_src_fetch() {
	debug-print-function ${FUNCNAME} "$@"

	[[ ${EVCS_OFFLINE} ]] && return

	_git-r3_env_setup
	local branch=${EGIT_BRANCH:+refs/heads/${EGIT_BRANCH}}
	git-r3_fetch "${EGIT_REPO_URI}" \
		"${EGIT_COMMIT:-${branch:-HEAD}}" \
		${CATEGORY}/${PN}/${SLOT}
}

git-r3_src_unpack() {
	debug-print-function ${FUNCNAME} "$@"

	_git-r3_env_setup
	git-r3_src_fetch
	git-r3_checkout "${EGIT_REPO_URI}" \
		${CATEGORY}/${PN}/${SLOT} \
		"${WORKDIR}/${P}"
}

_GIT_R3=1
fi

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.3: git-r3.eclass.diff --]
[-- Type: text/x-patch, Size: 3630 bytes --]

diff --git a/gx86/eclass/git-r3.eclass b/gx86/eclass/git-r3.eclass
index 36673a8..d56c818 100644
--- a/gx86/eclass/git-r3.eclass
+++ b/gx86/eclass/git-r3.eclass
@@ -168,6 +168,9 @@ _git-r3_set_gitdir() {
 	if [[ ! -d ${GIT_DIR} ]]; then
 		mkdir "${GIT_DIR}" || die
 		git init --bare || die
+
+		# avoid auto-unshallow :)
+		touch "${GIT_DIR}"/shallow || die
 	fi
 }
 
@@ -234,29 +237,70 @@ git-r3_fetch() {
 	for r in ${repos[@]}; do
 		einfo "Fetching ${remote_ref} from ${r} ..."
 
+		local is_branch lookup_ref
+		if [[ ${remote_ref} == refs/heads/* || ${remote_ref} == HEAD ]]
+		then
+			is_branch=1
+			lookup_ref=${remote_ref}
+		else
+			# ls-remote by commit is going to fail anyway,
+			# so we may as well pass refs/tags/ABCDEF...
+			lookup_ref=refs/tags/${remote_ref}
+		fi
+
 		# first, try ls-remote to see if ${remote_ref} is a real ref
 		# and not a commit id. if it succeeds, we can pass ${remote_ref}
 		# to 'fetch'. otherwise, we will just fetch everything
 
 		# split on whitespace
 		local ref=(
-			$(git ls-remote "${r}" "${remote_ref}")
+			$(git ls-remote "${r}" "${lookup_ref}")
 		)
 
+		# now, another important thing. we may only fetch a remote
+		# branch directly to a local branch. Otherwise, we need to fetch
+		# the commit and re-create the branch on top of it.
+
 		local ref_param=()
+		if [[ ! ${ref[0]} ]]; then
+			local EGIT_NONSHALLOW=1
+		fi
+
+		if [[ ! -f ${GIT_DIR}/shallow ]]; then
+			# if it's a complete repo, fetch it as-is
+			:
+		elif [[ ${EGIT_NONSHALLOW} ]]; then
+			# if it's a shallow clone but we need complete,
+			# unshallow it
+			ref_param+=( --unshallow )
+		else
+			# otherwise, just fetch as shallow
+			ref_param+=( --depth 1 )
+		fi
+
 		if [[ ${ref[0]} ]]; then
-			[[ ${EGIT_NONSHALLOW} ]] || ref_param+=( --depth 1 )
-			ref_param+=( "${remote_ref}" )
+			if [[ ${is_branch} ]]; then
+				ref_param+=( -f "${remote_ref}:${local_id}/__main__" )
+			else
+				ref_param+=( "${remote_ref}" )
+			fi
 		fi
 
 		# if ${remote_ref} is branch or tag, ${ref[@]} will contain
 		# the respective commit id. otherwise, it will be an empty
 		# array, so the following won't evaluate to a parameter.
-		if git fetch --no-tags "${r}" "${ref_param[@]}"; then
-			if ! git branch -f "${local_id}/__main__" "${ref[0]:-${remote_ref}}"
-			then
-				die "Creating tag failed (${remote_ref} invalid?)"
+		set -- git fetch --no-tags "${r}" "${ref_param[@]}"
+		echo "${@}" >&2
+		if "${@}"; then
+			if [[ ! ${is_branch} ]]; then
+				set -- git branch -f "${local_id}/__main__" \
+					"${ref[0]:-${remote_ref}}"
+				echo "${@}" >&2
+				if ! "${@}"; then
+					die "Creating branch for ${remote_ref} failed (wrong ref?)."
+				fi
 			fi
+
 			success=1
 			break
 		fi
@@ -311,7 +355,9 @@ git-r3_checkout() {
 	einfo "Checking out ${repos[0]} to ${out_dir} ..."
 
 	mkdir -p "${GIT_WORK_TREE}"
-	git checkout -f "${local_id}"/__main__ || die
+	set -- git checkout -f "${local_id}"/__main__ .
+	echo "${@}" >&2
+	"${@}" || die "git checkout ${local_id}/__main__ failed"
 
 	# diff against previous revision (if any)
 	local new_commit_id=$(git rev-parse --verify "${local_id}"/__main__)
@@ -330,6 +376,9 @@ git-r3_checkout() {
 		if [[ "${old_commit_id}" != "${new_commit_id}" ]]; then
 			echo "   updating from commit:     ${old_commit_id}"
 			echo "   to commit:                ${new_commit_id}"
+
+			git --no-pager diff --color --stat \
+				${old_commit_id}..${new_commit_id}
 		else
 			echo "   at the commit:            ${new_commit_id}"
 		fi

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] git-r3: initial draft for review [v2]

[Ulrich Mueller]

>>>>> On Sat, 31 Aug 2013, Michał Górny wrote:

> And time for a small update. 

In git-r3_checkout:

            git --no-pager diff --color --stat \
                ${old_commit_id}..${new_commit_id}

I'd rather omit the --color option, otherwise log files will contain
escape sequences.

Also it won't harm to make the unpack location configurable, so can we
please have a variable like EGIT_SOURCEDIR or EGIT_UNPACK_DIR (as it
was called in git.eclass)?

Ulrich

Re: [gentoo-dev] git-r3: initial draft for review [v2]

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 836 bytes --]

Dnia 2013-08-31, o godz. 11:26:30
Ulrich Mueller <ulm@gentoo.org> napisał(a):

> >>>>> On Sat, 31 Aug 2013, Michał Górny wrote:
> 
> > And time for a small update. 
> 
> In git-r3_checkout:
> 
>             git --no-pager diff --color --stat \
>                 ${old_commit_id}..${new_commit_id}
> 
> I'd rather omit the --color option, otherwise log files will contain
> escape sequences.

I'd rather leave it. The diff is more for pretty-printing anyway, it
shouldn't really matter in the logs.

> Also it won't harm to make the unpack location configurable, so can we
> please have a variable like EGIT_SOURCEDIR or EGIT_UNPACK_DIR (as it
> was called in git.eclass)?

I've added EGIT_CHECKOUT_DIR :).

Will submit the new version and a diff against v2 in a minute.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] git-r3: initial draft for review [v3]

[Michał Górny]

[-- Attachment #1.1: Type: text/plain, Size: 855 bytes --]

A few more changes.

1. Added EGIT_CHECKOUT_DIR to control the place where stuff is checked
   out.

2. Moved the defaults from src_fetch() and src_unpack()
   into git-r3_fetch() and git-r3_checkout(). This makes using them
   easier, like:

   git-r3_fetch http://repo1
   git-r3_fetch http://repo2
   git-r3_checkout http://repo1
   git-r3_checkout http://repo2 ${S}/foo

3. Added prototype pkg_isuptodate() function [bug 482666 [1]].

   (I will update it when we decide on how the func should be named :))

As an additional testing tool, I attached the version of git-r3
converted to git-2 compatible API. IOW, a drop-in replacement that you
could use to test the new eclass internals with API almost fully
compatible with the old one.

[1]:https://bugs.gentoo.org/show_bug.cgi?id=482666

-- 
Best regards,
Michał Górny

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: git-r3.eclass.diff --]
[-- Type: text/x-patch, Size: 8687 bytes --]

diff --git a/gx86/eclass/git-r3.eclass b/gx86/eclass/git-r3.eclass
index d56c818..84c04c7 100644
--- a/gx86/eclass/git-r3.eclass
+++ b/gx86/eclass/git-r3.eclass
@@ -71,6 +71,12 @@ if [[ ! ${_GIT_R3} ]]; then
 #
 # It can be overriden via env using ${PN}_LIVE_COMMIT variable.
 
+# @ECLASS-VARIABLE: EGIT_CHECKOUT_DIR
+# @DESCRIPTION:
+# The directory to check the git sources out to.
+#
+# EGIT_CHECKOUT_DIR=${WORKDIR}/${P}
+
 # @ECLASS-VARIABLE: EGIT_NONSHALLOW
 # @DEFAULT_UNSET
 # @DESCRIPTION:
@@ -94,8 +100,6 @@ _git-r3_env_setup() {
 
 	livevar=${esc_pn}_LIVE_REPO
 	EGIT_REPO_URI=${!livevar:-${EGIT_REPO_URI}}
-	[[ ${EGIT_REPO_URI} ]] \
-		|| die "EGIT_REPO_URI must be set to a non-empty value"
 	[[ ${!livevar} ]] \
 		&& ewarn "Using ${livevar}, no support will be provided"
 
@@ -209,26 +213,44 @@ _git-r3_set_submodules() {
 }
 
 # @FUNCTION: git-r3_fetch
-# @USAGE: <repo-uri> <remote-ref> <local-id>
+# @USAGE: [<repo-uri> [<remote-ref> [<local-id>]]]
 # @DESCRIPTION:
-# Fetch new commits to the local clone of repository. <repo-uri> follows
-# the syntax of EGIT_REPO_URI and may list multiple (fallback) URIs.
-# <remote-ref> specifies the remote ref to fetch (branch, tag
-# or commit). <local-id> specifies an identifier that needs to uniquely
-# identify the fetch operation in case multiple parallel merges used
-# the git repo. <local-id> usually involves using CATEGORY, PN and SLOT.
+# Fetch new commits to the local clone of repository.
+#
+# <repo-uri> specifies the repository URIs to fetch from, as a space-
+# -separated list. The first URI will be used as repository group
+# identifier and therefore must be used consistently. When not
+# specified, defaults to ${EGIT_REPO_URI}.
 #
-# The fetch operation will only affect the local storage. It will not
-# touch the working copy. If the repository contains submodules, they
-# will be fetched recursively as well.
+# <remote-ref> specifies the remote ref or commit id to fetch.
+# It is preferred to use 'refs/heads/<branch-name>' for branches
+# and 'refs/tags/<tag-name>' for tags. Other options are 'HEAD'
+# for upstream default branch and hexadecimal commit SHA1. Defaults
+# to the first of EGIT_COMMIT, EGIT_BRANCH or literal 'HEAD' that
+# is set to a non-null value.
+#
+# <local-id> specifies the local branch identifier that will be used to
+# locally store the fetch result. It should be unique to multiple
+# fetches within the repository that can be performed at the same time
+# (including parallel merges). It defaults to ${CATEGORY}/${PN}/${SLOT}.
+# This default should be fine unless you are fetching multiple trees
+# from the same repository in the same ebuild.
+#
+# The fetch operation will affect the EGIT_STORE only. It will not touch
+# the working copy, nor export any environment variables.
+# If the repository contains submodules, they will be fetched
+# recursively.
 git-r3_fetch() {
 	debug-print-function ${FUNCNAME} "$@"
 
-	local repos=( ${1} )
-	local remote_ref=${2}
-	local local_id=${3}
+	local repos=( ${1:-${EGIT_REPO_URI}} )
+	local branch=${EGIT_BRANCH:+refs/heads/${EGIT_BRANCH}}
+	local remote_ref=${2:-${EGIT_COMMIT:-${branch:-HEAD}}}
+	local local_id=${3:-${CATEGORY}/${PN}/${SLOT}}
 	local local_ref=refs/heads/${local_id}/__main__
 
+	[[ ${repos[@]} ]] || die "No URI provided and EGIT_REPO_URI unset"
+
 	local -x GIT_DIR
 	_git-r3_set_gitdir ${repos[0]}
 
@@ -282,7 +304,7 @@ git-r3_fetch() {
 			if [[ ${is_branch} ]]; then
 				ref_param+=( -f "${remote_ref}:${local_id}/__main__" )
 			else
-				ref_param+=( "${remote_ref}" )
+				ref_param+=( "refs/tags/${remote_ref}" )
 			fi
 		fi
 
@@ -331,22 +353,31 @@ git-r3_fetch() {
 }
 
 # @FUNCTION: git-r3_checkout
-# @USAGE: <repo-uri> <local-id> <path>
+# @USAGE: [<repo-uri> [<checkout-path> [<local-id>]]]
 # @DESCRIPTION:
-# Check the previously fetched commit out to <path> (usually
-# ${WORKDIR}/${P}). <repo-uri> follows the syntax of EGIT_REPO_URI
-# and will be used to re-construct the local storage path. <local-id>
-# is the unique identifier used for the fetch operation and will
-# be used to obtain the proper commit.
+# Check the previously fetched tree to the working copy.
+#
+# <repo-uri> specifies the repository URIs, as a space-separated list.
+# The first URI will be used as repository group identifier
+# and therefore must be used consistently with git-r3_fetch.
+# The remaining URIs are not used and therefore may be omitted.
+# When not specified, defaults to ${EGIT_REPO_URI}.
+#
+# <checkout-path> specifies the path to place the checkout. It defaults
+# to ${EGIT_CHECKOUT_DIR} if set, otherwise to ${WORKDIR}/${P}.
 #
-# If the repository contains submodules, they will be checked out
-# recursively as well.
+# <local-id> needs to specify the local identifier that was used
+# for respective git-r3_fetch.
+#
+# The checkout operation will write to the working copy, and export
+# the repository state into the environment. If the repository contains
+# submodules, they will be checked out recursively.
 git-r3_checkout() {
 	debug-print-function ${FUNCNAME} "$@"
 
-	local repos=( ${1} )
-	local local_id=${2}
-	local out_dir=${3}
+	local repos=( ${1:-${EGIT_REPO_URI}} )
+	local out_dir=${2:-${EGIT_CHECKOUT_DIR:-${WORKDIR}/${P}}}
+	local local_id=${3:-${CATEGORY}/${PN}/${SLOT}}
 
 	local -x GIT_DIR GIT_WORK_TREE
 	_git-r3_set_gitdir ${repos[0]}
@@ -396,8 +427,8 @@ git-r3_checkout() {
 			local url=${submodules[1]}
 			local path=${submodules[2]}
 
-			git-r3_checkout "${url}" "${local_id}/${subname}" \
-				"${GIT_WORK_TREE}/${path}"
+			git-r3_checkout "${url}" "${GIT_WORK_TREE}/${path}" \
+				"${local_id}/${subname}"
 
 			submodules=( "${submodules[@]:3}" ) # shift
 		done
@@ -408,16 +439,72 @@ git-r3_checkout() {
 	export EGIT_VERSION=${new_commit_id}
 }
 
+# @FUNCTION: git-r3_peek_remote_ref
+# @USAGE: [<repo-uri> [<remote-ref>]]
+# @DESCRIPTION:
+# Peek the reference in the remote repository and print the matching
+# (newest) commit SHA1.
+#
+# <repo-uri> specifies the repository URIs to fetch from, as a space-
+# -separated list. When not specified, defaults to ${EGIT_REPO_URI}.
+#
+# <remote-ref> specifies the remote ref to peek.  It is preferred to use
+# 'refs/heads/<branch-name>' for branches and 'refs/tags/<tag-name>'
+# for tags. Alternatively, 'HEAD' may be used for upstream default
+# branch. Defaults to the first of EGIT_COMMIT, EGIT_BRANCH or literal
+# 'HEAD' that is set to a non-null value.
+#
+# The operation will be done purely on the remote, without using local
+# storage. If commit SHA1 is provided as <remote-ref>, the function will
+# fail due to limitations of git protocol.
+#
+# On success, the function returns 0 and writes hexadecimal commit SHA1
+# to stdout. On failure, the function returns 1.
+git-r3_peek_remote_ref() {
+	debug-print-function ${FUNCNAME} "$@"
+
+	local repos=( ${1:-${EGIT_REPO_URI}} )
+	local branch=${EGIT_BRANCH:+refs/heads/${EGIT_BRANCH}}
+	local remote_ref=${2:-${EGIT_COMMIT:-${branch:-HEAD}}}
+
+	[[ ${repos[@]} ]] || die "No URI provided and EGIT_REPO_URI unset"
+
+	local r success
+	for r in ${repos[@]}; do
+		einfo "Peeking ${remote_ref} on ${r} ..." >&2
+
+		local is_branch lookup_ref
+		if [[ ${remote_ref} == refs/heads/* || ${remote_ref} == HEAD ]]
+		then
+			is_branch=1
+			lookup_ref=${remote_ref}
+		else
+			# ls-remote by commit is going to fail anyway,
+			# so we may as well pass refs/tags/ABCDEF...
+			lookup_ref=refs/tags/${remote_ref}
+		fi
+
+		# split on whitespace
+		local ref=(
+			$(git ls-remote "${r}" "${lookup_ref}")
+		)
+
+		if [[ ${ref[0]} ]]; then
+			echo "${ref[0]}"
+			return 0
+		fi
+	done
+
+	return 1
+}
+
 git-r3_src_fetch() {
 	debug-print-function ${FUNCNAME} "$@"
 
 	[[ ${EVCS_OFFLINE} ]] && return
 
 	_git-r3_env_setup
-	local branch=${EGIT_BRANCH:+refs/heads/${EGIT_BRANCH}}
-	git-r3_fetch "${EGIT_REPO_URI}" \
-		"${EGIT_COMMIT:-${branch:-HEAD}}" \
-		${CATEGORY}/${PN}/${SLOT}
+	git-r3_fetch
 }
 
 git-r3_src_unpack() {
@@ -425,9 +512,16 @@ git-r3_src_unpack() {
 
 	_git-r3_env_setup
 	git-r3_src_fetch
-	git-r3_checkout "${EGIT_REPO_URI}" \
-		${CATEGORY}/${PN}/${SLOT} \
-		"${WORKDIR}/${P}"
+	git-r3_checkout
+}
+
+git-r3_pkg_isuptodate() {
+	debug-print-function ${FUNCNAME} "$@"
+
+	local new_commit_id=$(git-r3_peek_remote_ref)
+	ewarn "old: ${EGIT_VERSION}"
+	ewarn "new: ${new_commit_id}"
+	[[ ! ${new_commit_id} || ${EGIT_VERSION} == ${new_commit_id} ]]
 }
 
 _GIT_R3=1

[-- Attachment #1.3: git-r3.eclass --]
[-- Type: text/plain, Size: 15683 bytes --]

# Copyright 1999-2013 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $

# @ECLASS: git-r3.eclass
# @MAINTAINER:
# Michał Górny <mgorny@gentoo.org>
# @BLURB: Eclass for fetching and unpacking git repositories.
# @DESCRIPTION:
# Third generation eclass for easing maitenance of live ebuilds using
# git as remote repository. Eclass supports lightweight (shallow)
# clones, local object deduplication and submodules.

case "${EAPI:-0}" in
	0|1|2|3|4|5)
		;;
	*)
		die "Unsupported EAPI=${EAPI} (unknown) for ${ECLASS}"
		;;
esac

if [[ ! ${_GIT_R3} ]]; then

inherit eutils

fi

EXPORT_FUNCTIONS src_unpack

if [[ ! ${_GIT_R3} ]]; then

# @ECLASS-VARIABLE: EGIT_STORE_DIR
# @DESCRIPTION:
# Storage directory for git sources.
#
# EGIT_STORE_DIR=${DISTDIR}/git3-src

# @ECLASS-VARIABLE: EGIT_REPO_URI
# @REQUIRED
# @DESCRIPTION:
# URIs to the repository, e.g. git://foo, https://foo. If multiple URIs
# are provided, the eclass will consider them as fallback URIs to try
# if the first URI does not work.
#
# It can be overriden via env using ${PN}_LIVE_REPO variable.
#
# Example:
# @CODE
# EGIT_REPO_URI="git://a/b.git https://c/d.git"
# @CODE

# @ECLASS-VARIABLE: EVCS_OFFLINE
# @DEFAULT_UNSET
# @DESCRIPTION:
# If non-empty, this variable prevents any online operations.

# @ECLASS-VARIABLE: EGIT_BRANCH
# @DEFAULT_UNSET
# @DESCRIPTION:
# The branch name to check out. If unset, the upstream default (HEAD)
# will be used.
#
# It can be overriden via env using ${PN}_LIVE_BRANCH variable.

# @ECLASS-VARIABLE: EGIT_COMMIT
# @DEFAULT_UNSET
# @DESCRIPTION:
# The tag name or commit identifier to check out. If unset, newest
# commit from the branch will be used. If set, EGIT_BRANCH will
# be ignored.
#
# It can be overriden via env using ${PN}_LIVE_COMMIT variable.

# @ECLASS-VARIABLE: EGIT_CHECKOUT_DIR
# @DESCRIPTION:
# The directory to check the git sources out to.
#
# EGIT_CHECKOUT_DIR=${WORKDIR}/${P}

# @ECLASS-VARIABLE: EGIT_NONSHALLOW
# @DEFAULT_UNSET
# @DESCRIPTION:
# Disable performing shallow fetches/clones. Shallow clones have
# a fair number of limitations. Therefore, if you'd like the eclass to
# perform complete clones instead, set this to a non-null value.
#
# This variable is to be set in make.conf. Ebuilds are not allowed
# to set it.

# @FUNCTION: _git-r3_env_setup
# @INTERNAL
# @DESCRIPTION:
# Set the eclass variables as necessary for operation. This can involve
# setting EGIT_* to defaults or ${PN}_LIVE_* variables.
_git-r3_env_setup() {
	debug-print-function ${FUNCNAME} "$@"

	local esc_pn livevar
	esc_pn=${PN//[-+]/_}

	livevar=${esc_pn}_LIVE_REPO
	EGIT_REPO_URI=${!livevar:-${EGIT_REPO_URI}}
	[[ ${!livevar} ]] \
		&& ewarn "Using ${livevar}, no support will be provided"

	livevar=${esc_pn}_LIVE_BRANCH
	EGIT_BRANCH=${!livevar:-${EGIT_BRANCH}}
	[[ ${!livevar} ]] \
		&& ewarn "Using ${livevar}, no support will be provided"

	livevar=${esc_pn}_LIVE_COMMIT
	EGIT_COMMIT=${!livevar:-${EGIT_COMMIT}}
	[[ ${!livevar} ]] \
		&& ewarn "Using ${livevar}, no support will be provided"

	# git-2 unsupported cruft
	local v
	for v in EGIT_{SOURCEDIR,MASTER,HAS_SUBMODULES,PROJECT} \
			EGIT_{NOUNPACK,BOOTSTRAP}
	do
		[[ ${!v} ]] && die "${v} is not supported."
	done
}

# @FUNCTION: _git-r3_set_gitdir
# @USAGE: <repo-uri>
# @INTERNAL
# @DESCRIPTION:
# Obtain the local repository path and set it as GIT_DIR. Creates
# a new repository if necessary.
#
# <repo-uri> may be used to compose the path. It should therefore be
# a canonical URI to the repository.
_git-r3_set_gitdir() {
	debug-print-function ${FUNCNAME} "$@"

	local repo_name=${1#*://*/}

	# strip common prefixes to make paths more likely to match
	# e.g. git://X/Y.git vs https://X/git/Y.git
	# (but just one of the prefixes)
	case "${repo_name}" in
		# cgit can proxy requests to git
		cgit/*) repo_name=${repo_name#cgit/};;
		# pretty common
		git/*) repo_name=${repo_name#git/};;
		# gentoo.org
		gitroot/*) repo_name=${repo_name#gitroot/};;
		# google code, sourceforge
		p/*) repo_name=${repo_name#p/};;
		# kernel.org
		pub/scm/*) repo_name=${repo_name#pub/scm/};;
	esac
	# ensure a .git suffix, same reason
	repo_name=${repo_name%.git}.git
	# now replace all the slashes
	repo_name=${repo_name//\//_}

	local distdir=${PORTAGE_ACTUAL_DISTDIR:-${DISTDIR}}
	: ${EGIT_STORE_DIR:=${distdir}/git3-src}

	GIT_DIR=${EGIT_STORE_DIR}/${repo_name}

	if [[ ! -d ${EGIT_STORE_DIR} ]]; then
		(
			addwrite /
			mkdir -m0755 -p "${EGIT_STORE_DIR}"
		) || die "Unable to create ${EGIT_STORE_DIR}"
	fi

	addwrite "${EGIT_STORE_DIR}"
	if [[ ! -d ${GIT_DIR} ]]; then
		mkdir "${GIT_DIR}" || die
		git init --bare || die

		# avoid auto-unshallow :)
		touch "${GIT_DIR}"/shallow || die
	fi
}

# @FUNCTION: _git-r3_set_submodules
# @USAGE: <file-contents>
# @INTERNAL
# @DESCRIPTION:
# Parse .gitmodules contents passed as <file-contents>
# as in "$(cat .gitmodules)"). Composes a 'submodules' array that
# contains in order (name, URL, path) for each submodule.
_git-r3_set_submodules() {
	debug-print-function ${FUNCNAME} "$@"

	local data=${1}

	# ( name url path ... )
	submodules=()

	local l
	while read l; do
		# submodule.<path>.path=<path>
		# submodule.<path>.url=<url>
		[[ ${l} == submodule.*.url=* ]] || continue

		l=${l#submodule.}
		local subname=${l%%.url=*}

		submodules+=(
			"${subname}"
			"$(echo "${data}" | git config -f /dev/fd/0 \
				submodule."${subname}".url)"
			"$(echo "${data}" | git config -f /dev/fd/0 \
				submodule."${subname}".path)"
		)
	done < <(echo "${data}" | git config -f /dev/fd/0 -l)
}

# @FUNCTION: git-r3_fetch
# @USAGE: [<repo-uri> [<remote-ref> [<local-id>]]]
# @DESCRIPTION:
# Fetch new commits to the local clone of repository.
#
# <repo-uri> specifies the repository URIs to fetch from, as a space-
# -separated list. The first URI will be used as repository group
# identifier and therefore must be used consistently. When not
# specified, defaults to ${EGIT_REPO_URI}.
#
# <remote-ref> specifies the remote ref or commit id to fetch.
# It is preferred to use 'refs/heads/<branch-name>' for branches
# and 'refs/tags/<tag-name>' for tags. Other options are 'HEAD'
# for upstream default branch and hexadecimal commit SHA1. Defaults
# to the first of EGIT_COMMIT, EGIT_BRANCH or literal 'HEAD' that
# is set to a non-null value.
#
# <local-id> specifies the local branch identifier that will be used to
# locally store the fetch result. It should be unique to multiple
# fetches within the repository that can be performed at the same time
# (including parallel merges). It defaults to ${CATEGORY}/${PN}/${SLOT}.
# This default should be fine unless you are fetching multiple trees
# from the same repository in the same ebuild.
#
# The fetch operation will affect the EGIT_STORE only. It will not touch
# the working copy, nor export any environment variables.
# If the repository contains submodules, they will be fetched
# recursively.
git-r3_fetch() {
	debug-print-function ${FUNCNAME} "$@"

	local repos=( ${1:-${EGIT_REPO_URI}} )
	local branch=${EGIT_BRANCH:+refs/heads/${EGIT_BRANCH}}
	local remote_ref=${2:-${EGIT_COMMIT:-${branch:-HEAD}}}
	local local_id=${3:-${CATEGORY}/${PN}/${SLOT}}
	local local_ref=refs/heads/${local_id}/__main__

	[[ ${repos[@]} ]] || die "No URI provided and EGIT_REPO_URI unset"

	local -x GIT_DIR
	_git-r3_set_gitdir ${repos[0]}

	# try to fetch from the remote
	local r success
	for r in ${repos[@]}; do
		einfo "Fetching ${remote_ref} from ${r} ..."

		local is_branch lookup_ref
		if [[ ${remote_ref} == refs/heads/* || ${remote_ref} == HEAD ]]
		then
			is_branch=1
			lookup_ref=${remote_ref}
		else
			# ls-remote by commit is going to fail anyway,
			# so we may as well pass refs/tags/ABCDEF...
			lookup_ref=refs/tags/${remote_ref}
		fi

		# first, try ls-remote to see if ${remote_ref} is a real ref
		# and not a commit id. if it succeeds, we can pass ${remote_ref}
		# to 'fetch'. otherwise, we will just fetch everything

		# split on whitespace
		local ref=(
			$(git ls-remote "${r}" "${lookup_ref}")
		)

		# now, another important thing. we may only fetch a remote
		# branch directly to a local branch. Otherwise, we need to fetch
		# the commit and re-create the branch on top of it.

		local ref_param=()
		if [[ ! ${ref[0]} ]]; then
			local EGIT_NONSHALLOW=1
		fi

		if [[ ! -f ${GIT_DIR}/shallow ]]; then
			# if it's a complete repo, fetch it as-is
			:
		elif [[ ${EGIT_NONSHALLOW} ]]; then
			# if it's a shallow clone but we need complete,
			# unshallow it
			ref_param+=( --unshallow )
		else
			# otherwise, just fetch as shallow
			ref_param+=( --depth 1 )
		fi

		if [[ ${ref[0]} ]]; then
			if [[ ${is_branch} ]]; then
				ref_param+=( -f "${remote_ref}:${local_id}/__main__" )
			else
				ref_param+=( "refs/tags/${remote_ref}" )
			fi
		fi

		# if ${remote_ref} is branch or tag, ${ref[@]} will contain
		# the respective commit id. otherwise, it will be an empty
		# array, so the following won't evaluate to a parameter.
		set -- git fetch --no-tags "${r}" "${ref_param[@]}"
		echo "${@}" >&2
		if "${@}"; then
			if [[ ! ${is_branch} ]]; then
				set -- git branch -f "${local_id}/__main__" \
					"${ref[0]:-${remote_ref}}"
				echo "${@}" >&2
				if ! "${@}"; then
					die "Creating branch for ${remote_ref} failed (wrong ref?)."
				fi
			fi

			success=1
			break
		fi
	done
	[[ ${success} ]] || die "Unable to fetch from any of EGIT_REPO_URI"

	# recursively fetch submodules
	if git cat-file -e "${local_ref}":.gitmodules &>/dev/null; then
		local submodules
		_git-r3_set_submodules \
			"$(git cat-file -p "${local_ref}":.gitmodules || die)"

		while [[ ${submodules[@]} ]]; do
			local subname=${submodules[0]}
			local url=${submodules[1]}
			local path=${submodules[2]}
			local commit=$(git rev-parse "${local_ref}:${path}")

			if [[ ! ${commit} ]]; then
				die "Unable to get commit id for submodule ${subname}"
			fi

			git-r3_fetch "${url}" "${commit}" "${local_id}/${subname}"

			submodules=( "${submodules[@]:3}" ) # shift
		done
	fi
}

# @FUNCTION: git-r3_checkout
# @USAGE: [<repo-uri> [<checkout-path> [<local-id>]]]
# @DESCRIPTION:
# Check the previously fetched tree to the working copy.
#
# <repo-uri> specifies the repository URIs, as a space-separated list.
# The first URI will be used as repository group identifier
# and therefore must be used consistently with git-r3_fetch.
# The remaining URIs are not used and therefore may be omitted.
# When not specified, defaults to ${EGIT_REPO_URI}.
#
# <checkout-path> specifies the path to place the checkout. It defaults
# to ${EGIT_CHECKOUT_DIR} if set, otherwise to ${WORKDIR}/${P}.
#
# <local-id> needs to specify the local identifier that was used
# for respective git-r3_fetch.
#
# The checkout operation will write to the working copy, and export
# the repository state into the environment. If the repository contains
# submodules, they will be checked out recursively.
git-r3_checkout() {
	debug-print-function ${FUNCNAME} "$@"

	local repos=( ${1:-${EGIT_REPO_URI}} )
	local out_dir=${2:-${EGIT_CHECKOUT_DIR:-${WORKDIR}/${P}}}
	local local_id=${3:-${CATEGORY}/${PN}/${SLOT}}

	local -x GIT_DIR GIT_WORK_TREE
	_git-r3_set_gitdir ${repos[0]}
	GIT_WORK_TREE=${out_dir}

	einfo "Checking out ${repos[0]} to ${out_dir} ..."

	mkdir -p "${GIT_WORK_TREE}"
	set -- git checkout -f "${local_id}"/__main__ .
	echo "${@}" >&2
	"${@}" || die "git checkout ${local_id}/__main__ failed"

	# diff against previous revision (if any)
	local new_commit_id=$(git rev-parse --verify "${local_id}"/__main__)
	local old_commit_id=$(
		git rev-parse --verify "${local_id}"/__old__ 2>/dev/null
	)

	if [[ ! ${old_commit_id} ]]; then
		echo "GIT NEW branch -->"
		echo "   repository:               ${repos[0]}"
		echo "   at the commit:            ${new_commit_id}"
	else
		echo "GIT update -->"
		echo "   repository:               ${repos[0]}"
		# write out message based on the revisions
		if [[ "${old_commit_id}" != "${new_commit_id}" ]]; then
			echo "   updating from commit:     ${old_commit_id}"
			echo "   to commit:                ${new_commit_id}"

			git --no-pager diff --color --stat \
				${old_commit_id}..${new_commit_id}
		else
			echo "   at the commit:            ${new_commit_id}"
		fi
	fi
	git branch -f "${local_id}"/{__old__,__main__} || die

	# recursively checkout submodules
	if [[ -f ${GIT_WORK_TREE}/.gitmodules ]]; then
		local submodules
		_git-r3_set_submodules \
			"$(cat "${GIT_WORK_TREE}"/.gitmodules)"

		while [[ ${submodules[@]} ]]; do
			local subname=${submodules[0]}
			local url=${submodules[1]}
			local path=${submodules[2]}

			git-r3_checkout "${url}" "${GIT_WORK_TREE}/${path}" \
				"${local_id}/${subname}"

			submodules=( "${submodules[@]:3}" ) # shift
		done
	fi

	# keep this *after* submodules
	export EGIT_DIR=${GIT_DIR}
	export EGIT_VERSION=${new_commit_id}
}

# @FUNCTION: git-r3_peek_remote_ref
# @USAGE: [<repo-uri> [<remote-ref>]]
# @DESCRIPTION:
# Peek the reference in the remote repository and print the matching
# (newest) commit SHA1.
#
# <repo-uri> specifies the repository URIs to fetch from, as a space-
# -separated list. When not specified, defaults to ${EGIT_REPO_URI}.
#
# <remote-ref> specifies the remote ref to peek.  It is preferred to use
# 'refs/heads/<branch-name>' for branches and 'refs/tags/<tag-name>'
# for tags. Alternatively, 'HEAD' may be used for upstream default
# branch. Defaults to the first of EGIT_COMMIT, EGIT_BRANCH or literal
# 'HEAD' that is set to a non-null value.
#
# The operation will be done purely on the remote, without using local
# storage. If commit SHA1 is provided as <remote-ref>, the function will
# fail due to limitations of git protocol.
#
# On success, the function returns 0 and writes hexadecimal commit SHA1
# to stdout. On failure, the function returns 1.
git-r3_peek_remote_ref() {
	debug-print-function ${FUNCNAME} "$@"

	local repos=( ${1:-${EGIT_REPO_URI}} )
	local branch=${EGIT_BRANCH:+refs/heads/${EGIT_BRANCH}}
	local remote_ref=${2:-${EGIT_COMMIT:-${branch:-HEAD}}}

	[[ ${repos[@]} ]] || die "No URI provided and EGIT_REPO_URI unset"

	local r success
	for r in ${repos[@]}; do
		einfo "Peeking ${remote_ref} on ${r} ..." >&2

		local is_branch lookup_ref
		if [[ ${remote_ref} == refs/heads/* || ${remote_ref} == HEAD ]]
		then
			is_branch=1
			lookup_ref=${remote_ref}
		else
			# ls-remote by commit is going to fail anyway,
			# so we may as well pass refs/tags/ABCDEF...
			lookup_ref=refs/tags/${remote_ref}
		fi

		# split on whitespace
		local ref=(
			$(git ls-remote "${r}" "${lookup_ref}")
		)

		if [[ ${ref[0]} ]]; then
			echo "${ref[0]}"
			return 0
		fi
	done

	return 1
}

git-r3_src_fetch() {
	debug-print-function ${FUNCNAME} "$@"

	[[ ${EVCS_OFFLINE} ]] && return

	_git-r3_env_setup
	git-r3_fetch
}

git-r3_src_unpack() {
	debug-print-function ${FUNCNAME} "$@"

	_git-r3_env_setup
	git-r3_src_fetch
	git-r3_checkout
}

git-r3_pkg_isuptodate() {
	debug-print-function ${FUNCNAME} "$@"

	local new_commit_id=$(git-r3_peek_remote_ref)
	ewarn "old: ${EGIT_VERSION}"
	ewarn "new: ${new_commit_id}"
	[[ ! ${new_commit_id} || ${EGIT_VERSION} == ${new_commit_id} ]]
}

_GIT_R3=1
fi

[-- Attachment #1.4: git-2.eclass --]
[-- Type: text/plain, Size: 13543 bytes --]

# Copyright 1999-2013 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $

# @ECLASS: git-r3.eclass
# @MAINTAINER:
# Michał Górny <mgorny@gentoo.org>
# @BLURB: Eclass for fetching and unpacking git repositories.
# @DESCRIPTION:
# Third generation eclass for easing maitenance of live ebuilds using
# git as remote repository. Eclass supports lightweight (shallow)
# clones, local object deduplication and submodules.

case "${EAPI:-0}" in
	0|1|2|3|4|5)
		;;
	*)
		die "Unsupported EAPI=${EAPI} (unknown) for ${ECLASS}"
		;;
esac

if [[ ! ${_GIT_R3} ]]; then

inherit eutils

fi

EXPORT_FUNCTIONS src_unpack

if [[ ! ${_GIT_R3} ]]; then

# @ECLASS-VARIABLE: EGIT_STORE_DIR
# @DESCRIPTION:
# Storage directory for git sources.
#
# EGIT_STORE_DIR=${DISTDIR}/git3-src

# @ECLASS-VARIABLE: EGIT_REPO_URI
# @REQUIRED
# @DESCRIPTION:
# URIs to the repository, e.g. git://foo, https://foo. If multiple URIs
# are provided, the eclass will consider them as fallback URIs to try
# if the first URI does not work.
#
# It can be overriden via env using ${PN}_LIVE_REPO variable.
#
# Example:
# @CODE
# EGIT_REPO_URI="git://a/b.git https://c/d.git"
# @CODE

# @ECLASS-VARIABLE: EVCS_OFFLINE
# @DEFAULT_UNSET
# @DESCRIPTION:
# If non-empty, this variable prevents any online operations.

# @ECLASS-VARIABLE: EGIT_BRANCH
# @DEFAULT_UNSET
# @DESCRIPTION:
# The branch name to check out. If unset, the upstream default (HEAD)
# will be used.
#
# It can be overriden via env using ${PN}_LIVE_BRANCH variable.

# @ECLASS-VARIABLE: EGIT_COMMIT
# @DEFAULT_UNSET
# @DESCRIPTION:
# The tag name or commit identifier to check out. If unset, newest
# commit from the branch will be used. If set, EGIT_BRANCH will
# be ignored.
#
# It can be overriden via env using ${PN}_LIVE_COMMIT variable.

# @ECLASS-VARIABLE: EGIT_CHECKOUT_DIR
# @DESCRIPTION:
# The directory to check the git sources out to.
#
# EGIT_CHECKOUT_DIR=${WORKDIR}/${P}

# @ECLASS-VARIABLE: EGIT_NONSHALLOW
# @DEFAULT_UNSET
# @DESCRIPTION:
# Disable performing shallow fetches/clones. Shallow clones have
# a fair number of limitations. Therefore, if you'd like the eclass to
# perform complete clones instead, set this to a non-null value.
#
# This variable is to be set in make.conf. Ebuilds are not allowed
# to set it.

# @FUNCTION: _git-r3_env_setup
# @INTERNAL
# @DESCRIPTION:
# Set the eclass variables as necessary for operation. This can involve
# setting EGIT_* to defaults or ${PN}_LIVE_* variables.
_git-r3_env_setup() {
	debug-print-function ${FUNCNAME} "$@"

	local esc_pn livevar
	esc_pn=${PN//[-+]/_}

	livevar=${esc_pn}_LIVE_REPO
	EGIT_REPO_URI=${!livevar:-${EGIT_REPO_URI}}
	[[ ${!livevar} ]] \
		&& ewarn "Using ${livevar}, no support will be provided"

	livevar=${esc_pn}_LIVE_BRANCH
	EGIT_BRANCH=${!livevar:-${EGIT_BRANCH}}
	[[ ${!livevar} ]] \
		&& ewarn "Using ${livevar}, no support will be provided"

	livevar=${esc_pn}_LIVE_COMMIT
	EGIT_COMMIT=${!livevar:-${EGIT_COMMIT}}
	[[ ${!livevar} ]] \
		&& ewarn "Using ${livevar}, no support will be provided"

	# git-2 compat
	: ${EGIT_BRANCH:=${EGIT_MASTER}}
	: ${EGIT_CHECKOUT_DIR:=${EGIT_SOURCEDIR}}

	[[ ${EGIT_BOOTSTRAP} ]] && die "EGIT_BOOTSTRAP not supported."
}

# @FUNCTION: _git-r3_set_gitdir
# @USAGE: <repo-uri>
# @INTERNAL
# @DESCRIPTION:
# Obtain the local repository path and set it as GIT_DIR. Creates
# a new repository if necessary.
#
# <repo-uri> may be used to compose the path. It should therefore be
# a canonical URI to the repository.
_git-r3_set_gitdir() {
	debug-print-function ${FUNCNAME} "$@"

	local repo_name=${1#*://*/}

	# strip common prefixes to make paths more likely to match
	# e.g. git://X/Y.git vs https://X/git/Y.git
	# (but just one of the prefixes)
	case "${repo_name}" in
		# cgit can proxy requests to git
		cgit/*) repo_name=${repo_name#cgit/};;
		# pretty common
		git/*) repo_name=${repo_name#git/};;
		# gentoo.org
		gitroot/*) repo_name=${repo_name#gitroot/};;
		# google code, sourceforge
		p/*) repo_name=${repo_name#p/};;
		# kernel.org
		pub/scm/*) repo_name=${repo_name#pub/scm/};;
	esac
	# ensure a .git suffix, same reason
	repo_name=${repo_name%.git}.git
	# now replace all the slashes
	repo_name=${repo_name//\//_}

	local distdir=${PORTAGE_ACTUAL_DISTDIR:-${DISTDIR}}
	: ${EGIT_STORE_DIR:=${distdir}/git3-src}

	GIT_DIR=${EGIT_STORE_DIR}/${repo_name}

	if [[ ! -d ${EGIT_STORE_DIR} ]]; then
		(
			addwrite /
			mkdir -m0755 -p "${EGIT_STORE_DIR}"
		) || die "Unable to create ${EGIT_STORE_DIR}"
	fi

	addwrite "${EGIT_STORE_DIR}"
	if [[ ! -d ${GIT_DIR} ]]; then
		mkdir "${GIT_DIR}" || die
		git init --bare || die

		# avoid auto-unshallow :)
		touch "${GIT_DIR}"/shallow || die
	fi
}

# @FUNCTION: _git-r3_set_submodules
# @USAGE: <file-contents>
# @INTERNAL
# @DESCRIPTION:
# Parse .gitmodules contents passed as <file-contents>
# as in "$(cat .gitmodules)"). Composes a 'submodules' array that
# contains in order (name, URL, path) for each submodule.
_git-r3_set_submodules() {
	debug-print-function ${FUNCNAME} "$@"

	local data=${1}

	# ( name url path ... )
	submodules=()

	local l
	while read l; do
		# submodule.<path>.path=<path>
		# submodule.<path>.url=<url>
		[[ ${l} == submodule.*.url=* ]] || continue

		l=${l#submodule.}
		local subname=${l%%.url=*}

		submodules+=(
			"${subname}"
			"$(echo "${data}" | git config -f /dev/fd/0 \
				submodule."${subname}".url)"
			"$(echo "${data}" | git config -f /dev/fd/0 \
				submodule."${subname}".path)"
		)
	done < <(echo "${data}" | git config -f /dev/fd/0 -l)
}

# @FUNCTION: git-r3_fetch
# @USAGE: [<repo-uri> [<remote-ref> [<local-id>]]]
# @DESCRIPTION:
# Fetch new commits to the local clone of repository.
#
# <repo-uri> specifies the repository URIs to fetch from, as a space-
# -separated list. The first URI will be used as repository group
# identifier and therefore must be used consistently. When not
# specified, defaults to ${EGIT_REPO_URI}.
#
# <remote-ref> specifies the remote ref or commit id to fetch.
# It is preferred to use 'refs/heads/<branch-name>' for branches
# and 'refs/tags/<tag-name>' for tags. Other options are 'HEAD'
# for upstream default branch and hexadecimal commit SHA1. Defaults
# to the first of EGIT_COMMIT, EGIT_BRANCH or literal 'HEAD' that
# is set to a non-null value.
#
# <local-id> specifies the local branch identifier that will be used to
# locally store the fetch result. It should be unique to multiple
# fetches within the repository that can be performed at the same time
# (including parallel merges). It defaults to ${CATEGORY}/${PN}/${SLOT}.
# This default should be fine unless you are fetching multiple trees
# from the same repository in the same ebuild.
#
# The fetch operation will affect the EGIT_STORE only. It will not touch
# the working copy, nor export any environment variables.
# If the repository contains submodules, they will be fetched
# recursively.
git-r3_fetch() {
	debug-print-function ${FUNCNAME} "$@"

	local repos=( ${1:-${EGIT_REPO_URI}} )
	local branch=${EGIT_BRANCH:+refs/heads/${EGIT_BRANCH}}
	local remote_ref=${2:-${EGIT_COMMIT:-${branch:-HEAD}}}
	local local_id=${3:-${CATEGORY}/${PN}/${SLOT}}
	local local_ref=refs/heads/${local_id}/__main__

	[[ ${repos[@]} ]] || die "No URI provided and EGIT_REPO_URI unset"

	local -x GIT_DIR
	_git-r3_set_gitdir ${repos[0]}

	# try to fetch from the remote
	local r success
	for r in ${repos[@]}; do
		einfo "Fetching ${remote_ref} from ${r} ..."

		local is_branch lookup_ref
		if [[ ${remote_ref} == refs/heads/* || ${remote_ref} == HEAD ]]
		then
			is_branch=1
			lookup_ref=${remote_ref}
		else
			# ls-remote by commit is going to fail anyway,
			# so we may as well pass refs/tags/ABCDEF...
			lookup_ref=refs/tags/${remote_ref}
		fi

		# first, try ls-remote to see if ${remote_ref} is a real ref
		# and not a commit id. if it succeeds, we can pass ${remote_ref}
		# to 'fetch'. otherwise, we will just fetch everything

		# split on whitespace
		local ref=(
			$(git ls-remote "${r}" "${lookup_ref}")
		)

		# now, another important thing. we may only fetch a remote
		# branch directly to a local branch. Otherwise, we need to fetch
		# the commit and re-create the branch on top of it.

		local ref_param=()
		if [[ ! ${ref[0]} ]]; then
			local EGIT_NONSHALLOW=1
		fi

		if [[ ! -f ${GIT_DIR}/shallow ]]; then
			# if it's a complete repo, fetch it as-is
			:
		elif [[ ${EGIT_NONSHALLOW} ]]; then
			# if it's a shallow clone but we need complete,
			# unshallow it
			ref_param+=( --unshallow )
		else
			# otherwise, just fetch as shallow
			ref_param+=( --depth 1 )
		fi

		if [[ ${ref[0]} ]]; then
			if [[ ${is_branch} ]]; then
				ref_param+=( -f "${remote_ref}:${local_id}/__main__" )
			else
				ref_param+=( "refs/tags/${remote_ref}" )
			fi
		fi

		# if ${remote_ref} is branch or tag, ${ref[@]} will contain
		# the respective commit id. otherwise, it will be an empty
		# array, so the following won't evaluate to a parameter.
		set -- git fetch --no-tags "${r}" "${ref_param[@]}"
		echo "${@}" >&2
		if "${@}"; then
			if [[ ! ${is_branch} ]]; then
				set -- git branch -f "${local_id}/__main__" \
					"${ref[0]:-${remote_ref}}"
				echo "${@}" >&2
				if ! "${@}"; then
					die "Creating branch for ${remote_ref} failed (wrong ref?)."
				fi
			fi

			success=1
			break
		fi
	done
	[[ ${success} ]] || die "Unable to fetch from any of EGIT_REPO_URI"

	# recursively fetch submodules
	if git cat-file -e "${local_ref}":.gitmodules &>/dev/null; then
		local submodules
		_git-r3_set_submodules \
			"$(git cat-file -p "${local_ref}":.gitmodules || die)"

		while [[ ${submodules[@]} ]]; do
			local subname=${submodules[0]}
			local url=${submodules[1]}
			local path=${submodules[2]}
			local commit=$(git rev-parse "${local_ref}:${path}")

			if [[ ! ${commit} ]]; then
				die "Unable to get commit id for submodule ${subname}"
			fi

			git-r3_fetch "${url}" "${commit}" "${local_id}/${subname}"

			submodules=( "${submodules[@]:3}" ) # shift
		done
	fi
}

# @FUNCTION: git-r3_checkout
# @USAGE: [<repo-uri> [<checkout-path> [<local-id>]]]
# @DESCRIPTION:
# Check the previously fetched tree to the working copy.
#
# <repo-uri> specifies the repository URIs, as a space-separated list.
# The first URI will be used as repository group identifier
# and therefore must be used consistently with git-r3_fetch.
# The remaining URIs are not used and therefore may be omitted.
# When not specified, defaults to ${EGIT_REPO_URI}.
#
# <checkout-path> specifies the path to place the checkout. It defaults
# to ${EGIT_CHECKOUT_DIR} if set, otherwise to ${WORKDIR}/${P}.
#
# <local-id> needs to specify the local identifier that was used
# for respective git-r3_fetch.
#
# The checkout operation will write to the working copy, and export
# the repository state into the environment. If the repository contains
# submodules, they will be checked out recursively.
git-r3_checkout() {
	debug-print-function ${FUNCNAME} "$@"

	local repos=( ${1:-${EGIT_REPO_URI}} )
	local out_dir=${2:-${EGIT_CHECKOUT_DIR:-${WORKDIR}/${P}}}
	local local_id=${3:-${CATEGORY}/${PN}/${SLOT}}

	local -x GIT_DIR GIT_WORK_TREE
	_git-r3_set_gitdir ${repos[0]}
	GIT_WORK_TREE=${out_dir}

	einfo "Checking out ${repos[0]} to ${out_dir} ..."

	mkdir -p "${GIT_WORK_TREE}"
	set -- git checkout -f "${local_id}"/__main__ .
	echo "${@}" >&2
	"${@}" || die "git checkout ${local_id}/__main__ failed"

	# diff against previous revision (if any)
	local new_commit_id=$(git rev-parse --verify "${local_id}"/__main__)
	local old_commit_id=$(
		git rev-parse --verify "${local_id}"/__old__ 2>/dev/null
	)

	if [[ ! ${old_commit_id} ]]; then
		echo "GIT NEW branch -->"
		echo "   repository:               ${repos[0]}"
		echo "   at the commit:            ${new_commit_id}"
	else
		echo "GIT update -->"
		echo "   repository:               ${repos[0]}"
		# write out message based on the revisions
		if [[ "${old_commit_id}" != "${new_commit_id}" ]]; then
			echo "   updating from commit:     ${old_commit_id}"
			echo "   to commit:                ${new_commit_id}"

			git --no-pager diff --color --stat \
				${old_commit_id}..${new_commit_id}
		else
			echo "   at the commit:            ${new_commit_id}"
		fi
	fi
	git branch -f "${local_id}"/{__old__,__main__} || die

	# recursively checkout submodules
	if [[ -f ${GIT_WORK_TREE}/.gitmodules ]]; then
		local submodules
		_git-r3_set_submodules \
			"$(cat "${GIT_WORK_TREE}"/.gitmodules)"

		while [[ ${submodules[@]} ]]; do
			local subname=${submodules[0]}
			local url=${submodules[1]}
			local path=${submodules[2]}

			git-r3_checkout "${url}" "${GIT_WORK_TREE}/${path}" \
				"${local_id}/${subname}"

			submodules=( "${submodules[@]:3}" ) # shift
		done
	fi

	# keep this *after* submodules
	export EGIT_DIR=${GIT_DIR}
	export EGIT_VERSION=${new_commit_id}
}

git-r3_src_fetch() {
	debug-print-function ${FUNCNAME} "$@"

	[[ ${EVCS_OFFLINE} ]] && return

	_git-r3_env_setup
	git-r3_fetch
}

git-2_src_unpack() {
	debug-print-function ${FUNCNAME} "$@"

	_git-r3_env_setup
	git-r3_src_fetch
	git-r3_checkout
}

_GIT_R3=1
fi

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] git-r3: initial draft for review [v2]

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 695 bytes --]

On Sat, Aug 31, 2013 at 10:48:32PM +0200, Michał Górny wrote:
> Dnia 2013-08-31, o godz. 11:26:30
> Ulrich Mueller <ulm@gentoo.org> napisał(a):
> 
> > >>>>> On Sat, 31 Aug 2013, Michał Górny wrote:
> > 
> > > And time for a small update. 
> > 
> > In git-r3_checkout:
> > 
> >             git --no-pager diff --color --stat \
> >                 ${old_commit_id}..${new_commit_id}
> > 
> > I'd rather omit the --color option, otherwise log files will contain
> > escape sequences.
> 
> I'd rather leave it. The diff is more for pretty-printing anyway, it
> shouldn't really matter in the logs.

Please don't. I also do not want escape sequences in log files.

William

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] git-r3: initial draft for review [v2]

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1072 bytes --]

Dnia 2013-09-01, o godz. 16:49:34
William Hubbs <williamh@gentoo.org> napisał(a):

> On Sat, Aug 31, 2013 at 10:48:32PM +0200, Michał Górny wrote:
> > Dnia 2013-08-31, o godz. 11:26:30
> > Ulrich Mueller <ulm@gentoo.org> napisał(a):
> > 
> > > >>>>> On Sat, 31 Aug 2013, Michał Górny wrote:
> > > 
> > > > And time for a small update. 
> > > 
> > > In git-r3_checkout:
> > > 
> > >             git --no-pager diff --color --stat \
> > >                 ${old_commit_id}..${new_commit_id}
> > > 
> > > I'd rather omit the --color option, otherwise log files will contain
> > > escape sequences.
> > 
> > I'd rather leave it. The diff is more for pretty-printing anyway, it
> > shouldn't really matter in the logs.
> 
> Please don't. I also do not want escape sequences in log files.

Ok, '--color' removed. However, I think we should work something out to
get both parties satisfied. Maybe portage feature or a tool to 'uncruft'
logs from escape sequences since many people actually benefit from them.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

[gentoo-dev] Re: git-r3: initial draft for review [v2]

[Duncan]

Michał Górny posted on Mon, 02 Sep 2013 13:33:19 +0200 as excerpted:

>> Please don't. I also do not want escape sequences in log files.
> 
> Ok, '--color' removed. However, I think we should work something out to
> get both parties satisfied. Maybe portage feature or a tool to 'uncruft'
> logs from escape sequences since many people actually benefit from them.

++

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] git-r3: initial draft for review [v2]

[Walter Dnes]

On Mon, Sep 02, 2013 at 01:33:19PM +0200, Micha?? Górny wrote
> Dnia 2013-09-01, o godz. 16:49:34
> William Hubbs <williamh@gentoo.org> napisa??(a):
> 
> > Please don't. I also do not want escape sequences in log files.
> 
> Ok, '--color' removed. However, I think we should work something out to
> get both parties satisfied. Maybe portage feature or a tool to 'uncruft'
> logs from escape sequences since many people actually benefit from them.

  As per my bug https://bugs.gentoo.org/show_bug.cgi?id=463954 it is a
pain.  Try figuring out the following as viewed in mc (Midnight
Commander)...

> WARN: prepare
> It seems that you need to set USE_PYTHON to make sure that legacy
> packages will be built with respect to PYTHON_TARGETS correctly:
>
>         USE_PYTHON='.[35;1m2.7.[0m'

  Note that...

grep foo bar.txt

...returns colour-highlighted text, while...

grep foo bar.txt > output.txt

...returns plain text.  So it can be done properly for everybody.

-- 
Walter Dnes <waltdnes@waltdnes.org>
I don't run "desktop environments"; I run useful applications

[gentoo-dev] Re: git-r3: initial draft for review [v2]

[Martin Vaeth]

Walter Dnes <waltdnes@waltdnes.org> wrote:
>
>   Note that...
>
> grep foo bar.txt
>
> ...returns colour-highlighted text, while...
>
> grep foo bar.txt > output.txt
>
> ...returns plain text.  So it can be done properly for everybody.

No, it cannot be done properly for everybody:

grep foo bar.txt | tee output.txt

Unfortunately, this corresponds exactly to portage's setup:
Commands like einfo would have to produce different output
on different channels which they cannot do if portage
redirects/copies globally. Portage would need a rather
cumbersome (and presumably slow) catching of such output
for each command separately.
Filtering the logfiles afterwards is probably simpler.

Re: [gentoo-dev] git-r3: initial draft for review

[Peter Stuge]

Markos Chandras wrote:
> the whole eclass is inside the " if [[ ! ${_GIT_R3} ]] " block.

Rather than putting the whole eclass inside a block like that maybe
it's possible to test for that condition and "exit" early?


//Peter

Re: [gentoo-dev] git-r3: initial draft for review

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 832 bytes --]

Dnia 2013-09-10, o godz. 07:04:49
Peter Stuge <peter@stuge.se> napisał(a):

> Markos Chandras wrote:
> > the whole eclass is inside the " if [[ ! ${_GIT_R3} ]] " block.
> 
> Rather than putting the whole eclass inside a block like that maybe
> it's possible to test for that condition and "exit" early?

Could you try your solutions before suggesting them on the list? It's
really something you could try at home in less than 5 minutes, and you
wouldn't have to make noise on the mailing list (not that anyone still
reads it). I think that you testing than would even take less time than
me answering it.

And in case you didn't want to do that: exiting ebuild process in middle
of inheritance chain is *not* a good idea. Portage will even give
an explanatory error for you.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] git-r3: initial draft for review

[Peter Stuge]

[-- Attachment #1: Type: text/plain, Size: 507 bytes --]

Michał Górny wrote:
> > > the whole eclass is inside the " if [[ ! ${_GIT_R3} ]] " block.
> > 
> > Rather than putting the whole eclass inside a block like that maybe
> > it's possible to test for that condition and "exit" early?
> 
> exiting ebuild process in middle of inheritance chain is *not* a good idea

I know, that's why I used quotes around the word exit. The question
is of course if there is a usable way to accomplish an early exit in
eclass context. Sad face if not.


//Peter

[-- Attachment #2: Type: application/pgp-signature, Size: 190 bytes --]