From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) by finch.gentoo.org (Postfix) with ESMTP id 83233138247 for ; Thu, 9 Jan 2014 19:15:25 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id F355FE0CA4; Thu, 9 Jan 2014 19:15:22 +0000 (UTC) Received: from mail-wi0-f176.google.com (mail-wi0-f176.google.com [209.85.212.176]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 1F1B8E0CA0 for ; Thu, 9 Jan 2014 19:15:21 +0000 (UTC) Received: by mail-wi0-f176.google.com with SMTP id hq4so7383344wib.3 for ; Thu, 09 Jan 2014 11:15:20 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=3QWkmKI4f25cYZnaOSstgVaRLNUkPR7HwQhPwxyoApQ=; b=iIjWcVgH7F++rC/xcoOQ3Z/9fuapqMMm8Rt1tEhheGkFGR2yc1Obf7ZxW2fr1uNmN4 BdPafmIukhN+ujDXoMvUGXC8wpfxXSfsyWSEsRdbAXLvC9SD8aZJRhSC/VK1VEdiYo+l k3rIloGz31bVp16aG+ikgcUPRR1B1Ps7zM4ANiYMpiHRIfTmQUOlTFIBlt/vQUXIMwBj KuAMpQzjNCsW8mhlYXDAhl+37D9qoWc/U8Pvz3x2SR9NMoxGgBYMcu8+mqwS8XZGwp3O yyRGlcdNDtl8kmv+bXd58xDHdZLP5k/+KwE9Wc1HME9IpTaM0t6aoZ+tYdxaiMXxrD/G i8ow== X-Gm-Message-State: ALoCoQk26MJ2OdahKx1tOd11PU3FhI46YWyHaigfPHAyzoorsyztzE1vxPMY1nYnujZZiH0EpKm/ Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-portage-dev@lists.gentoo.org Reply-to: gentoo-portage-dev@lists.gentoo.org MIME-Version: 1.0 X-Received: by 10.194.76.231 with SMTP id n7mr4668178wjw.16.1389294920723; Thu, 09 Jan 2014 11:15:20 -0800 (PST) Sender: antarus@scriptkitty.com Received: by 10.216.170.129 with HTTP; Thu, 9 Jan 2014 11:15:20 -0800 (PST) X-Originating-IP: [2620:0:1000:2e00:ae16:2dff:fe07:51d9] In-Reply-To: <1389288831.7103.75.camel@big_daddy.dol-sen.ca> References: <1389288831.7103.75.camel@big_daddy.dol-sen.ca> Date: Thu, 9 Jan 2014 11:15:20 -0800 X-Google-Sender-Auth: vhYgcVuP7VCeUKnTHBSZrihr4GA Message-ID: Subject: Re: [gentoo-portage-dev] New proposed modular sync system From: Alec Warner To: gentoo-portage-dev@lists.gentoo.org Content-Type: multipart/alternative; boundary=047d7bf0d2dc3d055104ef8e6eea X-Archives-Salt: 7fa9aaea-a34f-4cbe-80ac-0042e4c697b3 X-Archives-Hash: 9c96d7a16a1b8c21156d4f8d921635d5 --047d7bf0d2dc3d055104ef8e6eea Content-Type: text/plain; charset=UTF-8 On Thu, Jan 9, 2014 at 9:33 AM, Brian Dolbec wrote: > First off, I know many of you think portage needs to do everything on > it's own. NO outside dependencies. > > BUT! Please hear me out. > > While working on many of gentoo's tools gentoolkit's equery, eclean, > enalyze, Layman, mirrorselect, etc.. I have found that there was a lot > of needless code duplication. Both within the apps themselves and > copying/re-inventing the nearly identical code in other apps. > > I want to move many of gentoo's core applications to using some more > shared libraries. But that in itself is not a topic to discuss in this > thread. Please, do not sidetrack this thread. The above is only for > background info leading to this proposed change. This proposal follows > along that thinking. > > Many years ago after re-writing ecleans code I began re-writing > portage's emaint code so that it could be imported and used internally > in eclean to re-generate the Packages binpkg index file. In that code I > found a todo which said "make a plug-in system so we don't have to keep > modifying this code" or something along those lines. So I did. It > took 4 years+, rebasing many times and migrating vcs's, but Zac finally > merged it. It has been used in portage for more than a year. That > plug-in system I designed to be very flexible and basic. It can be used > for many areas of portage to simplify it's code and the maintenance > required to keep current with changes in repository types, and transfer > methods. > > I have started a Proposals sub-page under the Portage project page in > the wiki. It has a link to a diagram I made showing how the plug-in > system is laid out. This thread will be used to discuss the proposal > and the details needed for the module_spec dictionary that is used to > provide the manager with the details of what a module provides, options, > etc.. > > For a working example of the system in action, just try out emaint -h, > and some other actions it provides. Then move one of it's modules out > of the the emaint/modules/ directory and re-try emaint -h. You will see > the options for that module and it's actions are no longer available. > No code editing was required, no errors occurred,... Put it back, > retry... > > The modular sync system I propose would re-use that plug-in manager to > interface the sync manager code with the modules that perform the sync > actions. This sytem is easily extended with new modules for different > transfer methods or VCS specific modules. > > This also makes it possible for other gentoo applications like layman to > install a layman specific module which syncs all layman overlays that it > installs. All this done with out effort on the part of the portage team > either for maintenance or code changes to support it's addition. The > same can be said for Funtoo, if our git module is not suiotable, they > just install their own custom sync module. Done :) > > These add-on sync modules can also be installed via their own ebuild. > > For example: Zac has for years had a custom script that syncs the > portage tree and makes a squashfs version of it which he uses for his > PORTDIR. it is faster than a normal one. He, or someone else could make > a app-portage/squashfs-sync ebuild that installs a module for that > purpose. The user then just needs to edit his repos.conf file and > change the sync-type to squashfs. Then emerge --sync will do it > automatically without user intervention and auxiliary scripts run > manually or cron job. > > This system would also make it possible to modify emerge-s cli to accept > target repos to sync. and not any others also installed. Similarly to > "layman -s some-overlay", "emerge --sync some-overlay" > > I think the opposition to this idea primarily falls on reliability. There have been a number of hacks to portage over the years to keep it functioning during upgrades of itself, even down to the non-standard place where its .py files are stored (/usr/lib/portage/pym?) So the real question is when we move to a plug-in architecture, how do we add, remove, upgrade the plugins without breaking the actual package manager? -A > P.S. sorry for such a long email > -- > Brian Dolbec > --047d7bf0d2dc3d055104ef8e6eea Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
On T= hu, Jan 9, 2014 at 9:33 AM, Brian Dolbec <dolsen@gentoo.org>= wrote:
First off, I know many of you think portage = needs to do everything on
it's own. =C2=A0NO outside dependencies.

BUT! =C2=A0Please hear me out.

While working on many of gentoo's tools gentoolkit's equery, eclean= ,
enalyze, Layman, mirrorselect, etc.. =C2=A0I have found that there was a lo= t
of needless code duplication. =C2=A0Both within the apps themselves and
copying/re-inventing the nearly identical code in other apps.

I want to move many of gentoo's core applications to using some more shared libraries. =C2=A0But that in itself is not a topic to discuss in thi= s
thread. =C2=A0Please, do not sidetrack this thread. =C2=A0The above is only= for
background info leading to this proposed change. =C2=A0This proposal follow= s
along that thinking.

Many years ago after re-writing ecleans code I began re-writing
portage's emaint code so that it could be imported and used internally<= br> in eclean to re-generate the Packages binpkg index file. =C2=A0In that code= I
found a todo which said "make a plug-in system so we don't have to= keep
modifying this code" =C2=A0or something along those lines. =C2=A0So I = did. =C2=A0It
took 4 years+, rebasing many times and migrating vcs's, but Zac finally=
merged it. =C2=A0It has been used in portage for more than a year. That
plug-in system I designed to be very flexible and basic. =C2=A0It can be us= ed
for many areas of portage to simplify it's code and the maintenance
required to keep current with changes in repository types, and transfer
methods.

I have started a Proposals sub-page under the Portage project page in
the wiki. =C2=A0It has a link to a diagram I made showing how the plug-in system is laid out. =C2=A0This thread will be used to discuss the proposal<= br> and the details needed for the module_spec dictionary that is used to
provide the manager with the details of what a module provides, options, etc..

For a working example of the system in action, just try out emaint -h,
and some other actions it provides. =C2=A0Then move one of it's modules= out
of the the emaint/modules/ directory and re-try emaint -h. =C2=A0You will s= ee
the options for that module and it's actions are no longer available. No code editing was required, no errors occurred,... =C2=A0Put it back,
retry...

The modular sync system I propose would re-use that plug-in manager to
interface the sync manager code with the modules that perform the sync
actions. =C2=A0This sytem is easily extended with new modules for different=
transfer methods or VCS specific modules.

This also makes it possible for other gentoo applications like layman to install a layman specific module which syncs all layman overlays that it installs. =C2=A0All this done with out effort on the part of the portage te= am
either for maintenance or code changes to support it's addition. =C2=A0= The
same can be said for Funtoo, if our git module is not suiotable, they
just install their own custom sync module. =C2=A0Done :)

These add-on sync modules can also be installed via their own ebuild.

For example: =C2=A0Zac has for years had a custom script that syncs the
portage tree and makes a squashfs version of it which he uses for his
PORTDIR. it is faster than a normal one. =C2=A0He, or someone else could ma= ke
a app-portage/squashfs-sync ebuild that installs a module for that
purpose. =C2=A0The user then just needs to edit his repos.conf file and
change the sync-type to squashfs. =C2=A0Then emerge --sync will do it
automatically without user intervention and auxiliary scripts run
manually or cron job.

This system would also make it possible to modify emerge-s cli to accept target repos to sync. and not any others also installed. =C2=A0Similarly to=
"layman -s some-overlay", "emerge --sync some-overlay"<= br>

I think the opposition to this idea pr= imarily falls on reliability. There have been a number of hacks to portage = over the years to keep it functioning during upgrades of itself, even down = to the non-standard place where its .py files are stored (/usr/lib/portage/= pym?) So the real question is when we move to a plug-in architecture, how d= o we add, remove, upgrade the plugins without breaking the actual package m= anager?

-A


=C2=A0
=
P.S. sorry for such a long email
--
Brian Dolbec <dolsen@gentoo.org= >

--047d7bf0d2dc3d055104ef8e6eea--