[gentoo-portage-dev] New proposed modular sync system

[gentoo-portage-dev] New proposed modular sync system

[Brian Dolbec]

[-- Attachment #1: Type: text/plain, Size: 3723 bytes --]

First off, I know many of you think portage needs to do everything on
it's own.  NO outside dependencies.

BUT!  Please hear me out.

While working on many of gentoo's tools gentoolkit's equery, eclean,
enalyze, Layman, mirrorselect, etc..  I have found that there was a lot
of needless code duplication.  Both within the apps themselves and
copying/re-inventing the nearly identical code in other apps.

I want to move many of gentoo's core applications to using some more
shared libraries.  But that in itself is not a topic to discuss in this
thread.  Please, do not sidetrack this thread.  The above is only for
background info leading to this proposed change.  This proposal follows
along that thinking.

Many years ago after re-writing ecleans code I began re-writing
portage's emaint code so that it could be imported and used internally
in eclean to re-generate the Packages binpkg index file.  In that code I
found a todo which said "make a plug-in system so we don't have to keep
modifying this code"  or something along those lines.  So I did.  It
took 4 years+, rebasing many times and migrating vcs's, but Zac finally
merged it.  It has been used in portage for more than a year. That
plug-in system I designed to be very flexible and basic.  It can be used
for many areas of portage to simplify it's code and the maintenance
required to keep current with changes in repository types, and transfer
methods.

I have started a Proposals sub-page under the Portage project page in
the wiki.  It has a link to a diagram I made showing how the plug-in
system is laid out.  This thread will be used to discuss the proposal
and the details needed for the module_spec dictionary that is used to
provide the manager with the details of what a module provides, options,
etc..

For a working example of the system in action, just try out emaint -h,
and some other actions it provides.  Then move one of it's modules out
of the the emaint/modules/ directory and re-try emaint -h.  You will see
the options for that module and it's actions are no longer available.
No code editing was required, no errors occurred,...  Put it back,
retry...

The modular sync system I propose would re-use that plug-in manager to
interface the sync manager code with the modules that perform the sync
actions.  This sytem is easily extended with new modules for different
transfer methods or VCS specific modules.

This also makes it possible for other gentoo applications like layman to
install a layman specific module which syncs all layman overlays that it
installs.  All this done with out effort on the part of the portage team
either for maintenance or code changes to support it's addition.  The
same can be said for Funtoo, if our git module is not suiotable, they
just install their own custom sync module.  Done :)

These add-on sync modules can also be installed via their own ebuild.

For example:  Zac has for years had a custom script that syncs the
portage tree and makes a squashfs version of it which he uses for his
PORTDIR. it is faster than a normal one.  He, or someone else could make
a app-portage/squashfs-sync ebuild that installs a module for that
purpose.  The user then just needs to edit his repos.conf file and
change the sync-type to squashfs.  Then emerge --sync will do it
automatically without user intervention and auxiliary scripts run
manually or cron job.

This system would also make it possible to modify emerge-s cli to accept
target repos to sync. and not any others also installed.  Similarly to
"layman -s some-overlay", "emerge --sync some-overlay"
 
P.S. sorry for such a long email 
-- 
Brian Dolbec <dolsen@gentoo.org>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 620 bytes --]

Re: [gentoo-portage-dev] New proposed modular sync system

[Sebastian Luther]

Am 09.01.2014 18:33, schrieb Brian Dolbec:
> <introduction>
> 
> <history of the plugin system>

I fully agree with idea behind the plugin system. That is, to keep
things that are separate, separate. It probably wouldn't have occurred
to me to implement it that way (i.e. with auto-detection) but that's
just fine.

I don't see a problem with reusing that system here.

> I have started a Proposals sub-page under the Portage project page
> in the wiki.  It has a link to a diagram I made showing how the
> plug-in system is laid out.  This thread will be used to discuss
> the proposal and the details needed for the module_spec dictionary
> that is used to provide the manager with the details of what a
> module provides, options, etc..
> 

The layout makes sense. Except the problems I see with where the
modules are installed (see later).

Not sure about module_spec yet.

> [...]
> 
> <stuff about ebuilds installing plugins>
> 
While I see a valid use case here (especially with your squashfs
example), I wonder how that is supposed to work.

1) Where should the ebuild install the plugin?
2) How does portage find those plugins.
3) How does portage's behavior to run with the currently selected
python version, mix with the fact that ebuilds usually install only
for some set of python versions? (especially python 2 vs 3).

> This system would also make it possible to modify emerge-s cli to
> accept target repos to sync. and not any others also installed.
> Similarly to "layman -s some-overlay", "emerge --sync
> some-overlay"

"emerge --sync some-overlay" already works.


> P.S. sorry for such a long email
> 

Re: [gentoo-portage-dev] New proposed modular sync system

[Brian Dolbec]

[-- Attachment #1: Type: text/plain, Size: 579 bytes --]

On Thu, 2014-01-09 at 09:33 -0800, Brian Dolbec wrote:

> I have started a Proposals sub-page under the Portage project page in
> the wiki.  It has a link to a diagram I made showing how the plug-in
> system is laid out.  This thread will be used to discuss the proposal
> and the details needed for the module_spec dictionary that is used to
> provide the manager with the details of what a module provides, options,
> etc..
> 
 Forgot to add the link:

https://wiki.gentoo.org/wiki/Project:Portage/Proposals

It has the link to the diagram I created on that page

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 620 bytes --]

Re: [gentoo-portage-dev] New proposed modular sync system

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 4286 bytes --]

On Thu, Jan 9, 2014 at 9:33 AM, Brian Dolbec <dolsen@gentoo.org> wrote:

> First off, I know many of you think portage needs to do everything on
> it's own.  NO outside dependencies.
>
> BUT!  Please hear me out.
>
> While working on many of gentoo's tools gentoolkit's equery, eclean,
> enalyze, Layman, mirrorselect, etc..  I have found that there was a lot
> of needless code duplication.  Both within the apps themselves and
> copying/re-inventing the nearly identical code in other apps.
>
> I want to move many of gentoo's core applications to using some more
> shared libraries.  But that in itself is not a topic to discuss in this
> thread.  Please, do not sidetrack this thread.  The above is only for
> background info leading to this proposed change.  This proposal follows
> along that thinking.
>
> Many years ago after re-writing ecleans code I began re-writing
> portage's emaint code so that it could be imported and used internally
> in eclean to re-generate the Packages binpkg index file.  In that code I
> found a todo which said "make a plug-in system so we don't have to keep
> modifying this code"  or something along those lines.  So I did.  It
> took 4 years+, rebasing many times and migrating vcs's, but Zac finally
> merged it.  It has been used in portage for more than a year. That
> plug-in system I designed to be very flexible and basic.  It can be used
> for many areas of portage to simplify it's code and the maintenance
> required to keep current with changes in repository types, and transfer
> methods.
>
> I have started a Proposals sub-page under the Portage project page in
> the wiki.  It has a link to a diagram I made showing how the plug-in
> system is laid out.  This thread will be used to discuss the proposal
> and the details needed for the module_spec dictionary that is used to
> provide the manager with the details of what a module provides, options,
> etc..
>
> For a working example of the system in action, just try out emaint -h,
> and some other actions it provides.  Then move one of it's modules out
> of the the emaint/modules/ directory and re-try emaint -h.  You will see
> the options for that module and it's actions are no longer available.
> No code editing was required, no errors occurred,...  Put it back,
> retry...
>
> The modular sync system I propose would re-use that plug-in manager to
> interface the sync manager code with the modules that perform the sync
> actions.  This sytem is easily extended with new modules for different
> transfer methods or VCS specific modules.
>
> This also makes it possible for other gentoo applications like layman to
> install a layman specific module which syncs all layman overlays that it
> installs.  All this done with out effort on the part of the portage team
> either for maintenance or code changes to support it's addition.  The
> same can be said for Funtoo, if our git module is not suiotable, they
> just install their own custom sync module.  Done :)
>
> These add-on sync modules can also be installed via their own ebuild.
>
> For example:  Zac has for years had a custom script that syncs the
> portage tree and makes a squashfs version of it which he uses for his
> PORTDIR. it is faster than a normal one.  He, or someone else could make
> a app-portage/squashfs-sync ebuild that installs a module for that
> purpose.  The user then just needs to edit his repos.conf file and
> change the sync-type to squashfs.  Then emerge --sync will do it
> automatically without user intervention and auxiliary scripts run
> manually or cron job.
>
> This system would also make it possible to modify emerge-s cli to accept
> target repos to sync. and not any others also installed.  Similarly to
> "layman -s some-overlay", "emerge --sync some-overlay"
>
>
I think the opposition to this idea primarily falls on reliability. There
have been a number of hacks to portage over the years to keep it
functioning during upgrades of itself, even down to the non-standard place
where its .py files are stored (/usr/lib/portage/pym?) So the real question
is when we move to a plug-in architecture, how do we add, remove, upgrade
the plugins without breaking the actual package manager?

-A




> P.S. sorry for such a long email
> --
> Brian Dolbec <dolsen@gentoo.org>
>

[-- Attachment #2: Type: text/html, Size: 5154 bytes --]

Re: [gentoo-portage-dev] New proposed modular sync system

[Brian Dolbec]

[-- Attachment #1: Type: text/plain, Size: 2485 bytes --]

On Thu, 2014-01-09 at 19:46 +0100, Sebastian Luther wrote:

> The layout makes sense. Except the problems I see with where the
> modules are installed (see later).
> 
> Not sure about module_spec yet.
> 
> > [...]
> > 

The module_spec is a means to make available to the operational manager
what the module supplies and any options it provides.

The second thing it does is not import the files and modules it
supplies.  This reduces the memory requirements by not loading modules,
files that are not needed.  It also speeds up run time as a result.

> > <stuff about ebuilds installing plugins>
> > 
> While I see a valid use case here (especially with your squashfs
> example), I wonder how that is supposed to work.
> 
> 1) Where should the ebuild install the plugin?
> 2) How does portage find those plugins.

That is to be determined.  It doesn't matter where to the plugin
manager.  It needs a path to the plug-in directory on initialization.
It does not need to be in a subdirectory where it is located.


> 3) How does portage's behavior to run with the currently selected
> python version, mix with the fact that ebuilds usually install only
> for some set of python versions? (especially python 2 vs 3).
> 

The plug-in must be able to work with all supported python versions in
one form or another.

In esearch, it supports python 3, but layman does not (yet), so when you
use the -l switch to sync layman overlays as well as the main tree.  It
first tries to import the layman modules, upon failure it falls back to
calling layman using a subprocess call.  Just like portage currently
calls rsync and git in a subprocess now.

see working code:
https://github.com/fuzzyray/esearch/blob/master/esearch/sync.py
layman_sync()  line # 149


If a proposed module is to be added to the tree, it should be capable of
working in all configurations a user may have withing portage's
capability.  If not, if it isn't shipped by and with portage, then it is
a user issue.  Not ours.  In such a case we (the portage-tools team)
would insist on ewarns in the ebuild stating such constraints, etc..

> > This system would also make it possible to modify emerge-s cli to
> > accept target repos to sync. and not any others also installed.
> > Similarly to "layman -s some-overlay", "emerge --sync
> > some-overlay"
> 
> "emerge --sync some-overlay" already works.
> 
> 
> > P.S. sorry for such a long email
> > 
> 
> 


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 620 bytes --]

Re: [gentoo-portage-dev] New proposed modular sync system

[Brian Dolbec]

[-- Attachment #1: Type: text/plain, Size: 1119 bytes --]

On Thu, 2014-01-09 at 11:15 -0800, Alec Warner wrote:

> 
> 
> I think the opposition to this idea primarily falls on reliability.
> There have been a number of hacks to portage over the years to keep it
> functioning during upgrades of itself, even down to the non-standard
> place where its .py files are stored (/usr/lib/portage/pym?) So the
> real question is when we move to a plug-in architecture, how do we
> add, remove, upgrade the plugins without breaking the actual package
> manager?
> 
> 
> -A
> 

Well, a good point.  But, this is the sync module, not some of the
critical pkg installation code.  Another thing, once a module is loaded,
the source file could be removed and/or replaced without adverse effects
depending what the code does. That is not always the case though.
Portage/emerge does not sync and install ebuilds simultaneously, so that
problem is not a concern.  Also a lock file could be used to prevent
another emerge process form interfering with an ongoing sync run.  Just
like it does now for merging completed builds. 


-- 
Brian Dolbec <dolsen@gentoo.org>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 620 bytes --]

Re: [gentoo-portage-dev] New proposed modular sync system

[Brian Dolbec]

On Thu, 09 Jan 2014 09:33:51 -0800
Brian Dolbec <dolsen@gentoo.org> wrote:

>  
> I have started a Proposals sub-page under the Portage project page in
> the wiki.  It has a link to a diagram I made showing how the plug-in
> system is laid out.  This thread will be used to discuss the proposal
> and the details needed for the module_spec dictionary that is used to
> provide the manager with the details of what a module provides,
> options, etc..
> 
> For a working example of the system in action, just try out emaint -h,
> and some other actions it provides.  Then move one of it's modules out
> of the the emaint/modules/ directory and re-try emaint -h.  You will
> see the options for that module and it's actions are no longer
> available. No code editing was required, no errors occurred,...  Put
> it back, retry...
> 
> The modular sync system I propose would re-use that plug-in manager to
> interface the sync manager code with the modules that perform the sync
> actions.  This sytem is easily extended with new modules for different
> transfer methods or VCS specific modules.
> 

STATUS UPDATE:

I have pushed the plugin-sync branch to the main portage.git repo on
g.o.g.o [1].  It has working and tested code for the rsync and git
modules.  The CVS module I have not yet tested or done any debugging on.

First off, let me state that this is only minimally changed code to
make it function.  It is by no means ready for any code review other
than for general program structure and the data that we need/want to
pass between the controller and the modules.

I have updated the Proposal page [2] with more things to do as well as
some insights I discovered during the initial porting.

Next up for discussion is what additional functionality should a module
supply?

I think it should provide a list or dictionary of the repos.conf
variables it needs or uses.  I also think that the module should
provide a function for validating the conf variables meet some minimum
standard/obvious error checking.

For a more involved example of the types of information that can be
passed through the module_spec see the working emaint logs module file
[3].  The information required/provided does not affect the plugin
system.  It is a standard set up by the controller and the modules
requirements.

So, please look over the code, avoid commenting on the poor code
quality just yet.  Before I work on cleaning up/improving the code I
wanted to establish what we feel the requirements should be.



[1]
http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=shortlog;h=refs/heads/plugin-sync
[2] https://wiki.gentoo.org/wiki/Project:Portage/Proposals
[3]
http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=blob;f=pym/portage/emaint/modules/logs/__init__.py;h=0407efe2bef19f2f815c53790e1ea790366b9967;hb=HEAD


-- 
Brian Dolbec <dolsen>

Re: [gentoo-portage-dev] New proposed modular sync system

[Sebastian Luther]

Am 31.01.2014 06:27, schrieb Brian Dolbec:

> I have updated the Proposal page [2] with more things to do as well as
> some insights I discovered during the initial porting.
> 

> 6. Determine the method to be able to do concurrent repo syncs

Take a look at the stuff in portage.util.{_async,_eventloop}. It's used
in various places in emerge if you need an example.

> 18. Modify emerge-webrsync cli to use the new webrsync module

I'd just remove emerge-webrsync and ask the user to use emerge --sync
instead.

> Next up for discussion is what additional functionality should a module
> supply?
> 
> I think it should provide a list or dictionary of the repos.conf
> variables it needs or uses.  I also think that the module should
> provide a function for validating the conf variables meet some minimum
> standard/obvious error checking.

Why should it provide that list at all, if it has a function to validate
its input? The calling code should just check sync-type, select the
appropriate module and pass everything to it.

Re: [gentoo-portage-dev] New proposed modular sync system

[Brian Dolbec]

On Fri, 31 Jan 2014 08:13:56 +0100
Sebastian Luther <SebastianLuther@gmx.de> wrote:

> Am 31.01.2014 06:27, schrieb Brian Dolbec:
> 
> > I have updated the Proposal page [2] with more things to do as well
> > as some insights I discovered during the initial porting.
> > 
> 
> > 6. Determine the method to be able to do concurrent repo syncs
> 
> Take a look at the stuff in portage.util.{_async,_eventloop}. It's
> used in various places in emerge if you need an example.
> 

Thanks, I will have a look

> > 18. Modify emerge-webrsync cli to use the new webrsync module
> 
> I'd just remove emerge-webrsync and ask the user to use emerge --sync
> instead.
>

Some people are not able to use a normal rsync tree due to firewall
issues and bandwith restrictions.

 
> > Next up for discussion is what additional functionality should a
> > module supply?
> > 
> > I think it should provide a list or dictionary of the repos.conf
> > variables it needs or uses.  I also think that the module should
> > provide a function for validating the conf variables meet some
> > minimum standard/obvious error checking.
> 
> Why should it provide that list at all, if it has a function to
> validate its input? The calling code should just check sync-type,
> select the appropriate module and pass everything to it.
> 
> 

I am thinking more about an emaint module for checking repos.conf
entries and help generation.  The sync module would provide the
validation function and the info the emaint module would use to list
variable requirements.  Yes, the plugin module should provide it's own
man page, but maybe it doesn't or the user just wants a quick list of
the variable requirements.


emaint --list --variables syncs

  rsync:
    ...

  git:
    ...
    gpg-sig-uri

  squashfs:
    sync-uri
    gpg-sig-uri  optional, blah blah,...
    ...



-- 
Brian Dolbec <dolsen>

Re: [gentoo-portage-dev] New proposed modular sync system

[Sebastian Luther]

Am 31.01.2014 08:27, schrieb Brian Dolbec:
> 
>>> 18. Modify emerge-webrsync cli to use the new webrsync module
>>
>> I'd just remove emerge-webrsync and ask the user to use emerge --sync
>> instead.
>>
> 
> Some people are not able to use a normal rsync tree due to firewall
> issues and bandwith restrictions.
> 
>  

I didn't mean to remove the functionality provided by emerge-webrsync.
Since in point 17 you want to move the functionality into a module, the
user should be able to set sync-type=web-rsync in repos.conf and have
emerge --sync sync it.

>>> Next up for discussion is what additional functionality should a
>>> module supply?
>>>
>>> I think it should provide a list or dictionary of the repos.conf
>>> variables it needs or uses.  I also think that the module should
>>> provide a function for validating the conf variables meet some
>>> minimum standard/obvious error checking.
>>
>> Why should it provide that list at all, if it has a function to
>> validate its input? The calling code should just check sync-type,
>> select the appropriate module and pass everything to it.
>>
>>
> 
> I am thinking more about an emaint module for checking repos.conf
> entries and help generation.  The sync module would provide the
> validation function and the info the emaint module would use to list
> variable requirements.  Yes, the plugin module should provide it's own
> man page, but maybe it doesn't or the user just wants a quick list of
> the variable requirements.
> 
> 
> emaint --list --variables syncs
> 
>   rsync:
>     ...
> 
>   git:
>     ...
>     gpg-sig-uri
> 
>   squashfs:
>     sync-uri
>     gpg-sig-uri  optional, blah blah,...
>     ...
> 
> 
> 
I guess I'm just not familiar enough with your approach to see all the
things you want to do with it.