[gentoo-portage-dev] [RFC] Add 'emerge --sync-glsa' action and 'emaint sync-glsa' command

[gentoo-portage-dev] [RFC] Add 'emerge --sync-glsa' action and 'emaint sync-glsa' command

[Zac Medico]

Hi,

For server deployments, it's common for administrators to want to use
glsa-check for security vulnerabilities, without necessarily wanting to
to do a full sync [1].

So, I propose that we add an 'emerge --sync-glsa' action and a
corresponding 'emaint sync-glsa' command that will only sync the
relevant metadata/glsa subdirectory of the relevant repository(s).
Please respond if you agree or disagree with this proposal.

[1] https://bugs.gentoo.org/show_bug.cgi?id=89641
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [RFC] Add 'emerge --sync-glsa' action and 'emaint sync-glsa' command

[Brian Dolbec]

On Wed, 17 Dec 2014 12:30:53 -0800
Zac Medico <zmedico@gentoo.org> wrote:

> Hi,
> 
> For server deployments, it's common for administrators to want to use
> glsa-check for security vulnerabilities, without necessarily wanting
> to to do a full sync [1].
> 
> So, I propose that we add an 'emerge --sync-glsa' action and a
> corresponding 'emaint sync-glsa' command that will only sync the
> relevant metadata/glsa subdirectory of the relevant repository(s).
> Please respond if you agree or disagree with this proposal.
> 
> [1] https://bugs.gentoo.org/show_bug.cgi?id=89641

I agree in principal, but you didn't say enough about your idea to be
able to say yes/no to it yet.

I've read the bug comments.

So, what you are saying for the above is set a very restrictive rsync
to just sync the glsa directory?

So, with the new pluging-sync sytem in place, how do you propose we do
this?

1) Make a new emaint sync option which does the restriction and
appropriate sync module calls?
    emaint sync --glsa  & emerge --sync-glsa

2) emaint sync-glsa sounds like a modified copy of the sync module.
   To me this is a poor idea.  It would needlessly duplicate code.


But what about the different sync types.  How do we handle this in a
git repo instead of an rsync one?  

	3) if it is a separate repo, then the current emaint sync
	module does not need __ANY__ modifications.
     
        emaint sync -r glsa  <== would sync the repo named glsa no
        matter what type of sync method is used.  rsync, git, svn,
        cvs,...

	Another advantage to a separate repo, for your use case that
	they don't want the main repo to be synced, the new sync system
	adds an "auto-sync" variable.  "emerge --sync" will only sync
	repos marked with it set to yes.  So, if all they want is the
	glsa's synced automatically, set only i'ts auto-sync to yes.
	All other repos will have to be synced manually via the "emaint
	sync --repo foo" command which does not look at the auto-sync
	variable.


Bottom line of this comes down to getting infra to create a standalone
repo for the glsa's.  The current rsync tree could merge the gentoo and
glsa repos for backwards compatibility.  For your use case, they set the
gentoo repo's rsync restriction to exclude the glsa's.  And have the
separate glsa repo downloaded as it's own repository.  With the gentoo
tree moving to git, then the git tree would not include the glsa's, so
the user would need to install the glsa tree as well.

Only code changes I see to portage, pkgcore (I know nothing about
paludis) are to look for the glsa's in the 2 possible locations.  The
standalone glsa repo, failing that, backup to the gentoo tree.
-- 
Brian Dolbec <dolsen>

Re: [gentoo-portage-dev] [RFC] Add 'emerge --sync-glsa' action and 'emaint sync-glsa' command

[Michael Orlitzky]

On 12/17/2014 05:32 PM, Brian Dolbec wrote:
> 
> Only code changes I see to portage, pkgcore (I know nothing about
> paludis) are to look for the glsa's in the 2 possible locations.  The
> standalone glsa repo, failing that, backup to the gentoo tree.
> 

Could we ship a GLSA overlay enabled by default, regardless of what we
do with the main repo? Then for simplicity, we update the tools to check
metadata/glsa in overlays as well. No need to special-case a fallback
location.

glsa-check would of course want to sync the GLSA repo before running.

Re: [gentoo-portage-dev] [RFC] Add 'emerge --sync-glsa' action and 'emaint sync-glsa' command

[Brian Dolbec]

On Wed, 17 Dec 2014 17:49:56 -0500
Michael Orlitzky <mjo@gentoo.org> wrote:

> On 12/17/2014 05:32 PM, Brian Dolbec wrote:
> > 
> > Only code changes I see to portage, pkgcore (I know nothing about
> > paludis) are to look for the glsa's in the 2 possible locations.
> > The standalone glsa repo, failing that, backup to the gentoo tree.
> > 
> 
> Could we ship a GLSA overlay enabled by default, regardless of what we
> do with the main repo? Then for simplicity, we update the tools to
> check metadata/glsa in overlays as well. No need to special-case a
> fallback location.
> 
> glsa-check would of course want to sync the GLSA repo before running.
> 
> 

yes, that is a very simple edit to the default repos.conf file adding
the glsa repo. Yes, I forgot that glsa-check needed changes
so it would/could be configured to sync the repo too.

Pkgcore does not have the capability (yet) to read/use extended
repos.conf repo attributes.  So, would need more code changes. I think
pkgcore does not yet handle repos.conf files... I'm not sure.
-- 
Brian Dolbec <dolsen>

Re: [gentoo-portage-dev] [RFC] Add 'emerge --sync-glsa' action and 'emaint sync-glsa' command

[Zac Medico]

On 12/17/2014 02:32 PM, Brian Dolbec wrote:
> On Wed, 17 Dec 2014 12:30:53 -0800
> Zac Medico <zmedico@gentoo.org> wrote:
> 
>> Hi,
>>
>> For server deployments, it's common for administrators to want to use
>> glsa-check for security vulnerabilities, without necessarily wanting
>> to to do a full sync [1].
>>
>> So, I propose that we add an 'emerge --sync-glsa' action and a
>> corresponding 'emaint sync-glsa' command that will only sync the
>> relevant metadata/glsa subdirectory of the relevant repository(s).
>> Please respond if you agree or disagree with this proposal.
>>
>> [1] https://bugs.gentoo.org/show_bug.cgi?id=89641
> 
> I agree in principal, but you didn't say enough about your idea to be
> able to say yes/no to it yet.

Yeah, I need to explain the implementation ideas that were floating
around in my head when I made the proposal.

> I've read the bug comments.
> 
> So, what you are saying for the above is set a very restrictive rsync
> to just sync the glsa directory?
> 
> So, with the new pluging-sync sytem in place, how do you propose we do
> this?
> 
> 1) Make a new emaint sync option which does the restriction and
> appropriate sync module calls?
>     emaint sync --glsa  & emerge --sync-glsa
> 
> 2) emaint sync-glsa sounds like a modified copy of the sync module.
>    To me this is a poor idea.  It would needlessly duplicate code.

I was thinking that we could tag the repos containing glsas using an
attribute in repos.conf, and teach the existing rsync module how to sync
the glsas alone.

> But what about the different sync types.  How do we handle this in a
> git repo instead of an rsync one?  

I was thinking that we would only need to implement this for the rsync
repo, since that's the only one that includes the glsas. If there's some
reason to implement it for other sync types in the future, then we can
do it when the need arises.

> 
> 	3) if it is a separate repo, then the current emaint sync
> 	module does not need __ANY__ modifications.
>      
>         emaint sync -r glsa  <== would sync the repo named glsa no
>         matter what type of sync method is used.  rsync, git, svn,
>         cvs,...
> 
> 	Another advantage to a separate repo, for your use case that
> 	they don't want the main repo to be synced, the new sync system
> 	adds an "auto-sync" variable.  "emerge --sync" will only sync
> 	repos marked with it set to yes.  So, if all they want is the
> 	glsa's synced automatically, set only i'ts auto-sync to yes.
> 	All other repos will have to be synced manually via the "emaint
> 	sync --repo foo" command which does not look at the auto-sync
> 	variable.

Well, it sounds like you're introducing a new "repo" type that's
different from a package repository. That's not really necessary if we
take the approach that I've describe above.

> Bottom line of this comes down to getting infra to create a standalone
> repo for the glsa's.  The current rsync tree could merge the gentoo and
> glsa repos for backwards compatibility.  For your use case, they set the
> gentoo repo's rsync restriction to exclude the glsa's.  And have the
> separate glsa repo downloaded as it's own repository.  With the gentoo
> tree moving to git, then the git tree would not include the glsa's, so
> the user would need to install the glsa tree as well.

Note that we can even re-use the existing rsync tree if we take the
"separate repo" approach, and simply configure the "separate repo" to
sync from rsync://rsync.gentoo.org/gentoo-portage/metadata/glsa instead
of rsync://rsync.gentoo.org/gentoo-portage.

> Only code changes I see to portage, pkgcore (I know nothing about
> paludis) are to look for the glsa's in the 2 possible locations.  The
> standalone glsa repo, failing that, backup to the gentoo tree.

Well, is there anything wrong with the tagging the existing repo in
repos.conf as I've described above?
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [RFC] Add 'emerge --sync-glsa' action and 'emaint sync-glsa' command

[Zac Medico]

On 12/17/2014 03:57 PM, Zac Medico wrote:
> On 12/17/2014 02:32 PM, Brian Dolbec wrote:
>> On Wed, 17 Dec 2014 12:30:53 -0800
> 
>> But what about the different sync types.  How do we handle this in a
>> git repo instead of an rsync one?  
> 
> I was thinking that we would only need to implement this for the rsync
> repo, since that's the only one that includes the glsas. If there's some
> reason to implement it for other sync types in the future, then we can
> do it when the need arises.

Also, at some point we might introduce a notion of submodules, similar
to git submodules, and if necessary each submodule could potentially be
synced using a different protocol. However, this is way beyond the
present needs. For now, simply extending the rsync module to allow
syncing of sub-directories should be sufficient (though I would not
expose the ability to sync arbitrary directories through the user
interface).
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [RFC] Add 'emerge --sync-glsa' action and 'emaint sync-glsa' command

[Zac Medico]

On 12/17/2014 06:20 PM, Zac Medico wrote:
> On 12/17/2014 03:57 PM, Zac Medico wrote:
>> On 12/17/2014 02:32 PM, Brian Dolbec wrote:
>>> On Wed, 17 Dec 2014 12:30:53 -0800
>>
>>> But what about the different sync types.  How do we handle this in a
>>> git repo instead of an rsync one?  
>>
>> I was thinking that we would only need to implement this for the rsync
>> repo, since that's the only one that includes the glsas. If there's some
>> reason to implement it for other sync types in the future, then we can
>> do it when the need arises.
> 
> Also, at some point we might introduce a notion of submodules, similar
> to git submodules, and if necessary each submodule could potentially be
> synced using a different protocol. However, this is way beyond the
> present needs. For now, simply extending the rsync module to allow
> syncing of sub-directories should be sufficient (though I would not
> expose the ability to sync arbitrary directories through the user
> interface).

I did some research, and it appears that we'll be able to do something
similar for the git module if we make the metadata/glsa directory a git
submodule. So, it seems that all of our supported sync protocols support
the possibility of updating individual sub-directories!
-- 
Thanks,
Zac