[gentoo-user] Local portage git mirror

[gentoo-user] Local portage git mirror

[Peter Humphrey]

Hello list,

Does anyone have some pointers to setting up a local portage mirror for git 
to serve? I'm looking at http://www.funtoo.org/Portage_Git_Mirror but some 
details are obscure to me. I hope I'm not going to have to roll my sleeves 
up and learn the whole of git; I just want to cut down the time my boxes 
take to sync via git - some of them have their portage trees nfs-mounted in 
chroots on my workstation and take anything up to 20 minutes to sync.

One question comes up straight away: should the git-served repository be the 
host's own /usr/portage tree? And what cron task should I run to refresh the 
mirror?

Any help would be gratefully received.

(I can still read e-mails via web-mail even when KMail is playing up.)

-- 
Regards
Peter

Re: [gentoo-user] Local portage git mirror

[Bill Kenworthy]

On 17/01/17 18:03, Peter Humphrey wrote:
> Hello list,
> 
> Does anyone have some pointers to setting up a local portage mirror for git 
> to serve? I'm looking at http://www.funtoo.org/Portage_Git_Mirror but some 
> details are obscure to me. I hope I'm not going to have to roll my sleeves 
> up and learn the whole of git; I just want to cut down the time my boxes 
> take to sync via git - some of them have their portage trees nfs-mounted in 
> chroots on my workstation and take anything up to 20 minutes to sync.
> 
> One question comes up straight away: should the git-served repository be the 
> host's own /usr/portage tree? And what cron task should I run to refresh the 
> mirror?
> 
> Any help would be gratefully received.
> 
> (I can still read e-mails via web-mail even when KMail is playing up.)
> 

I moved my systems to a local git repos some time ago - it works (fast -
much faster than rsync!)but its not perfect:

It would be nice if someone can share how it really should be done!
Below is the contents of an email to the list in 2015 when I got it working:

* problems:
the files in /usr/share/portage sometimes get overridden and give an
error message
renaming "gentoo" to "olympus" (the server host) made sense at the time
but creates odd errors at times requiring files to be restored.
git doesnt use the portage checksums which besides odd errors at times
required using "thin-manifests = true" in
"/usr/portage/metadata/layout.conf"




++++++++++++++++++
Replace your rsync infrastructure with a git one.  After set up, there
is faster syncing than rsync, both externally to the gentoo
infrastructure (reasonable ADSL link), and really fast compared to rsync
internally.  Not simple, but its a one off hit on just a few machines
for me.  Works for me, no warranty etc.

Thanks to Michael and Hasufell for the hints needed to get this working.
Apologies for the line wrap.



Replace with your own values!
repo name: olympus
repo master host: olympus.lan.localdomain

On the master:

install git !!!

Download https://github.com/hasufell/portage-gentoo-git-config

unzip

cp -v portage-gentoo-git-config-master/etc/portage/repo.postsync.d/*
/etc/portage/repo.postsync.d/

cp portage-gentoo-git-config-master/etc/portage/repos.conf/gentoo.conf
/etc/portage/repos.conf/

if needed: chmod +x /etc/portage/repo.postsync.d/sync_overlay_cache
cd /usr/
mv portage portage.old

emerge --sync


To use as the master for your own mirror system:
Uncomment the lines in /etc/rsyncd.conf to set up your own
gentoo-portage rsync server - still needed for now

rc-update add rsyncd

/etc/init.d/rsyncd start



On the internal clients wanting to use the mirror:
reccomended: set up passwordless ssh between clients and the master

emerge dev-vcs/git

create /etc/portage/repos.conf/olympus.conf as follows
--------------------
[DEFAULT]
main-repo = olympus

[olympus]
location = /usr/portage
sync-type = git
sync-uri = git://olympus.lan.localdomain:/usr/portage
auto-sync = yes
--------------------

create /etc/portage/repo.postsync.d/sync_olympus
--------------------
#!/bin/bash

repository_name="${1}"
repository_path="${3}"

[[ ${repository_name} == "olympus" ]] || exit 0

source /lib/gentoo/functions.sh

# Number of jobs for egencache, default is number or processors.
parallel_jobs="$(nproc)"

# caches
ebegin "Fetching pre-generated metadata cache for ${repository_name}"
	rsync -a
rsync://olympus.lan.localdomain/gentoo-portage/metadata/md5-cache/
"${repository_path}"/metadata/md5-cache/
eend $?

ebegin "Updating metadata cache for ${repository_name}"
	egencache --jobs="${parallel_jobs}" --repo="${repository_name}"
--update --update-use-local-desc
eend $?

#DTD's
DTDDIR="${repository_path}"/metadata/dtd
ebegin "Updating DTDs"
if [[ -e ${DTDDIR} ]]; then
        git -C "${DTDDIR}" pull -q --ff
else
        git clone olympus.lan.localdomain:/usr/portage/metadata/dtd
"${DTDDIR}"
fi
eend "$?"

#GLSA's
GLSADIR="${repository_path}"/metadata/glsa
ebegin "Updating GLSAs"
if [[ -e ${GLSADIR} ]]; then
        git -C "${GLSADIR}" pull -q --ff
else
        git clone olympus.lan.localdomain:/usr/portage/metadata/glsa
"${GLSADIR}"
fi
eend "$?"

# herds
ebegin "Updating herds.xml"
    rsync -a
rsync://olympus.lan.localdomain/gentoo-portage/metadata/herds.xml
"${repository_path}"/metadata/md5-cache/
eend $?

#news
NEWSDIR="${repository_path}"/metadata/news
ebegin "Updating news items"
if [[ -e ${NEWSDIR} ]]; then
        git -C "${NEWSDIR}" pull -q --ff
else
        git clone olympus.lan.localdomain:/usr/portage/metadata/news
"${NEWSDIR}"
fi
eend $? "Try to remove ${NEWSDIR}"
--------------------


mv /usr/portage /usr/portage.old

git clone olympus.lan.localdomain:/usr/portage /usr/portage

echo 'olympus' > /usr/portage/profiles/repo_name

echo 'masters = olympus' > /usr/local/portage/metadata/layout.conf

echo 'thin-manifests = true' >> /usr/local/portage/metadata/layout.conf

cp  /usr/portage/metadata/layout.conf
/usr/local/portage/metadata/layout.conf

cp /etc/portage/repos.conf/olympus.conf /usr/share/portage/config/repos.conf

emerge --sync
emerge --sync

* the first sync will complete the process, the second is to check there are
  no errors.

Re: [gentoo-user] Local portage git mirror

[Peter Humphrey]

On Tuesday, 17 January 2017 20:51:38 GMT Bill Kenworthy wrote:
> On 17/01/17 18:03, Peter Humphrey wrote:
> > Hello list,
> > 
> > Does anyone have some pointers to setting up a local portage mirror for
> > git to serve? I'm looking at http://www.funtoo.org/Portage_Git_Mirror
> > but some details are obscure to me. I hope I'm not going to have to
> > roll my sleeves up and learn the whole of git; I just want to cut down
> > the time my boxes take to sync via git - some of them have their
> > portage trees nfs-mounted in chroots on my workstation and take
> > anything up to 20 minutes to sync.
> > 
> > One question comes up straight away: should the git-served repository be
> > the host's own /usr/portage tree? And what cron task should I run to
> > refresh the mirror?
> > 
> > Any help would be gratefully received.
> > 
> > (I can still read e-mails via web-mail even when KMail is playing up.)
> 
> I moved my systems to a local git repos some time ago - it works (fast -
> much faster than rsync!)but its not perfect:
> 
> It would be nice if someone can share how it really should be done!

It seems to me to be ideal material for a wiki page. Any offers?

> Below is the contents of an email to the list in 2015 when I got it
> working:
> 
> * problems:
> the files in /usr/share/portage sometimes get overridden and give an
> error message
> renaming "gentoo" to "olympus" (the server host) made sense at the time
> but creates odd errors at times requiring files to be restored.
> git doesnt use the portage checksums which besides odd errors at times
> required using "thin-manifests = true" in
> "/usr/portage/metadata/layout.conf"

Many thanks, Bill. That'll give me something to think about!

-- 
Regards
Peter

> ++++++++++++++++++
> Replace your rsync infrastructure with a git one.  After set up, there
> is faster syncing than rsync, both externally to the gentoo
> infrastructure (reasonable ADSL link), and really fast compared to rsync
> internally.  Not simple, but its a one off hit on just a few machines
> for me.  Works for me, no warranty etc.
> 
> Thanks to Michael and Hasufell for the hints needed to get this working.
> Apologies for the line wrap.
> 
> 
> 
> Replace with your own values!
> repo name: olympus
> repo master host: olympus.lan.localdomain
> 
> On the master:
> 
> install git !!!
> 
> Download https://github.com/hasufell/portage-gentoo-git-config
> 
> unzip
> 
> cp -v portage-gentoo-git-config-master/etc/portage/repo.postsync.d/*
> /etc/portage/repo.postsync.d/
> 
> cp portage-gentoo-git-config-master/etc/portage/repos.conf/gentoo.conf
> /etc/portage/repos.conf/
> 
> if needed: chmod +x /etc/portage/repo.postsync.d/sync_overlay_cache
> cd /usr/
> mv portage portage.old
> 
> emerge --sync
> 
> 
> To use as the master for your own mirror system:
> Uncomment the lines in /etc/rsyncd.conf to set up your own
> gentoo-portage rsync server - still needed for now
> 
> rc-update add rsyncd
> 
> /etc/init.d/rsyncd start
> 
> 
> 
> On the internal clients wanting to use the mirror:
> reccomended: set up passwordless ssh between clients and the master
> 
> emerge dev-vcs/git
> 
> create /etc/portage/repos.conf/olympus.conf as follows
> --------------------
> [DEFAULT]
> main-repo = olympus
> 
> [olympus]
> location = /usr/portage
> sync-type = git
> sync-uri = git://olympus.lan.localdomain:/usr/portage
> auto-sync = yes
> --------------------
> 
> create /etc/portage/repo.postsync.d/sync_olympus
> --------------------
> #!/bin/bash
> 
> repository_name="${1}"
> repository_path="${3}"
> 
> [[ ${repository_name} == "olympus" ]] || exit 0
> 
> source /lib/gentoo/functions.sh
> 
> # Number of jobs for egencache, default is number or processors.
> parallel_jobs="$(nproc)"
> 
> # caches
> ebegin "Fetching pre-generated metadata cache for ${repository_name}"
> 	rsync -a
> rsync://olympus.lan.localdomain/gentoo-portage/metadata/md5-cache/
> "${repository_path}"/metadata/md5-cache/
> eend $?
> 
> ebegin "Updating metadata cache for ${repository_name}"
> 	egencache --jobs="${parallel_jobs}" --repo="${repository_name}"
> --update --update-use-local-desc
> eend $?
> 
> #DTD's
> DTDDIR="${repository_path}"/metadata/dtd
> ebegin "Updating DTDs"
> if [[ -e ${DTDDIR} ]]; then
>         git -C "${DTDDIR}" pull -q --ff
> else
>         git clone olympus.lan.localdomain:/usr/portage/metadata/dtd
> "${DTDDIR}"
> fi
> eend "$?"
> 
> #GLSA's
> GLSADIR="${repository_path}"/metadata/glsa
> ebegin "Updating GLSAs"
> if [[ -e ${GLSADIR} ]]; then
>         git -C "${GLSADIR}" pull -q --ff
> else
>         git clone olympus.lan.localdomain:/usr/portage/metadata/glsa
> "${GLSADIR}"
> fi
> eend "$?"
> 
> # herds
> ebegin "Updating herds.xml"
>     rsync -a
> rsync://olympus.lan.localdomain/gentoo-portage/metadata/herds.xml
> "${repository_path}"/metadata/md5-cache/
> eend $?
> 
> #news
> NEWSDIR="${repository_path}"/metadata/news
> ebegin "Updating news items"
> if [[ -e ${NEWSDIR} ]]; then
>         git -C "${NEWSDIR}" pull -q --ff
> else
>         git clone olympus.lan.localdomain:/usr/portage/metadata/news
> "${NEWSDIR}"
> fi
> eend $? "Try to remove ${NEWSDIR}"
> --------------------
> 
> 
> mv /usr/portage /usr/portage.old
> 
> git clone olympus.lan.localdomain:/usr/portage /usr/portage
> 
> echo 'olympus' > /usr/portage/profiles/repo_name
> 
> echo 'masters = olympus' > /usr/local/portage/metadata/layout.conf
> 
> echo 'thin-manifests = true' >> /usr/local/portage/metadata/layout.conf
> 
> cp  /usr/portage/metadata/layout.conf
> /usr/local/portage/metadata/layout.conf
> 
> cp /etc/portage/repos.conf/olympus.conf
> /usr/share/portage/config/repos.conf
> 
> emerge --sync
> emerge --sync
> 
> * the first sync will complete the process, the second is to check there
> are no errors.

[gentoo-user] Local portage git mirror

[Peter Humphrey]

Hello list,

I have a small LAN, on which one box has little to do so I've set it up as a 
local git mirror. I use this box - the one I'm writing on now - as a compile 
server for a coule of other boxes, including the new git server, in which the 
clients export their portage tree over NFS v3 to this box.

On those client boxes I've had to "chown -R portage:portage /usr/portage" to 
get the exporting via git to work; otherwise I get a read permission failure 
on .git/FETCH_HEAD.

This is what I see on the compile host:
$ ls -l /usr/portage/.git
total 15M
-rw-r--r--   1 root root  267 May  7 10:59 config
-rw-r--r--   1 root root   73 May  7 10:49 description
-rw-r--r--   1 root root  104 Jun  4 09:49 FETCH_HEAD
[...]

But on the client boxes I now have this:
$ ls -l /usr/portage/.git
total 15M
-rw-r--r--   1 portage portage  277 May  7 11:33 config
-rw-r--r--   1 portage portage   73 May  7 11:31 description
-rw-r--r--   1 portage portage  104 Jun  4 10:24 FETCH_HEAD
[...]

I did try chowning to root:portage first, but I still got the permission 
error.

Have I done something daft here?

-- 
Regards,
Peter.