[gentoo-dev] [RFC] Decoupling stabilization from security bugs

[gentoo-dev] [RFC] Decoupling stabilization from security bugs

[Michał Górny]

Hello, everyone.

TL;DR: I'd like to propose that stabilizations are done via blockers of
security bugs instead of security bugs themselves, i.e. as any other
stabilizations.


Right now we're often performing security-related stabilizations via
security bugs. This has a few problems, that are:

1. Stabilization-related activity causes unnecessary mail to the widely
subscribed security alias. That is, subscribed people get notified of
package list changes, NATTkA results, every arch doing its work.
However, in reality the security team only cares about stabilization
being started, stalled or finished -- and for that, getting the usual
'dependent bug added/closed' mail should be sufficient.

2. NATTkA has no good way of distinguishing irrelevant security bugs
from security bugs where something went wrong (and NATTkA doesn't use
persistent state by design). The most important problem is that --
unlike regular stablereqs -- security bugs aren't supposed to be closed
after stabilization. It can't really distinguish a security bug 'left
open' from a security bug with incorrect package list.

3. Proxied maintainers without editbugs can't actually CC arches on
security bugs since the bugs are assigned to security@.


To resolve these problems going forward and establish consistent
behavior in the future, I'd like to propose to disable 'package list'
fields on security bugs and instead expect regular stabilization bugs to
be used (and made block the security bugs) for stabilizations. While I
understand that filing additional bugs might be cumbersome for some
people, I don't think it's such a herculean effort to outweigh
the problems solved.

In the end, consistency is a good thing and we've introduced a dedicated
stabilization category to reduce the spread of stabilization bugs all
around the place.

WDYT?

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] [RFC] Decoupling stabilization from security bugs

[John Helmert III]

[-- Attachment #1: Type: text/plain, Size: 81 bytes --]

The benefits definitely seem to outweigh the added work here, sounds
good to me!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[gentoo-dev] Re: [RFC] Decoupling stabilization from security bugs

[Aaron Bauman]

[-- Attachment #1: Type: text/plain, Size: 462 bytes --]

On Thu, Aug 12, 2021 at 02:53:33PM +0200, Michał Górny wrote:
> Hello, everyone.
> 
> TL;DR: I'd like to propose that stabilizations are done via blockers of
> security bugs instead of security bugs themselves, i.e. as any other
> stabilizations.
> 
> 
> Right now we're often performing security-related stabilizations via
> security bugs. This has a few problems, that are:
> 

<snip>

> WDYT?

I welcome this change. Let's do it.

-Aaron

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] [RFC] Decoupling stabilization from security bugs

[Agostino Sarubbo]

On giovedì 12 agosto 2021 14:53:33 CEST Michał Górny wrote:
> To resolve these problems going forward and establish consistent
> behavior in the future, I'd like to propose to disable 'package list'
> fields on security bugs and instead expect regular stabilization bugs to
> be used (and made block the security bugs) for stabilizations. While I
> understand that filing additional bugs might be cumbersome for some
> people, I don't think it's such a herculean effort to outweigh
> the problems solved.

I think it is a good idea but the stabilization bug that blocks the security 
bug should at least have something (bugzilla KEYWORD?) to facilitate the 
search of the security stabilization.
Atm we look for bugs with assignee = security@ and cc = arch@


Agostino

Re: [gentoo-dev] [RFC] Decoupling stabilization from security bugs

[Ionen Wolkens]

[-- Attachment #1: Type: text/plain, Size: 487 bytes --]

On Thu, Aug 12, 2021 at 05:17:36PM +0200, Agostino Sarubbo wrote:
> I think it is a good idea but the stabilization bug that blocks the security 
> bug should at least have something (bugzilla KEYWORD?) to facilitate the 
> search of the security stabilization.
> Atm we look for bugs with assignee = security@ and cc = arch@

We already have a mostly unused kw:SECURITY, maybe it could be given
some actual use. But well, these still need people to actually set it.
-- 
ionen

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] [RFC] Decoupling stabilization from security bugs

[Sam James]

[-- Attachment #1: Type: text/plain, Size: 1446 bytes --]



> On 12 Aug 2021, at 16:17, Agostino Sarubbo <ago@gentoo.org> wrote:
> 
> On giovedì 12 agosto 2021 14:53:33 CEST Michał Górny wrote:
>> To resolve these problems going forward and establish consistent
>> behavior in the future, I'd like to propose to disable 'package list'
>> fields on security bugs and instead expect regular stabilization bugs to
>> be used (and made block the security bugs) for stabilizations. While I
>> understand that filing additional bugs might be cumbersome for some
>> people, I don't think it's such a herculean effort to outweigh
>> the problems solved.
> 
> I think it is a good idea but the stabilization bug that blocks the security
> bug should at least have something (bugzilla KEYWORD?) to facilitate the
> search of the security stabilization.
> Atm we look for bugs with assignee = security@ and cc = arch@
> 

This is my primary concern and as long as we use e.g. the SECURITY
keyword, I'm happy. From #gentoo-dev:

[22:34:36] <@sam_> ago: I was wondering if I could just detect by blockers but I think SECURITY blocker is simpler and requires less code/handling overall, so WFM
[22:35:25] <@ago> yeah

I'm a _little_ bit unsure about the extra work of filing new bugs, but I suspect
It's going to be worth it because of less special casing for everybody involved
(and not having to explain why security bugs are different to newbies, proxied-maints,
...).

best,
sam

[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] [RFC] Decoupling stabilization from security bugs

[Matt Turner]

On Thu, Aug 12, 2021 at 5:53 AM Michał Górny <mgorny@gentoo.org> wrote:
>
> Hello, everyone.
>
> TL;DR: I'd like to propose that stabilizations are done via blockers of
> security bugs instead of security bugs themselves, i.e. as any other
> stabilizations.
>
>
> Right now we're often performing security-related stabilizations via
> security bugs. This has a few problems, that are:
>
> 1. Stabilization-related activity causes unnecessary mail to the widely
> subscribed security alias. That is, subscribed people get notified of
> package list changes, NATTkA results, every arch doing its work.
> However, in reality the security team only cares about stabilization
> being started, stalled or finished -- and for that, getting the usual
> 'dependent bug added/closed' mail should be sufficient.
>
> 2. NATTkA has no good way of distinguishing irrelevant security bugs
> from security bugs where something went wrong (and NATTkA doesn't use
> persistent state by design). The most important problem is that --
> unlike regular stablereqs -- security bugs aren't supposed to be closed
> after stabilization. It can't really distinguish a security bug 'left
> open' from a security bug with incorrect package list.
>
> 3. Proxied maintainers without editbugs can't actually CC arches on
> security bugs since the bugs are assigned to security@.
>
>
> To resolve these problems going forward and establish consistent
> behavior in the future, I'd like to propose to disable 'package list'
> fields on security bugs and instead expect regular stabilization bugs to
> be used (and made block the security bugs) for stabilizations. While I
> understand that filing additional bugs might be cumbersome for some
> people, I don't think it's such a herculean effort to outweigh
> the problems solved.
>
> In the end, consistency is a good thing and we've introduced a dedicated
> stabilization category to reduce the spread of stabilization bugs all
> around the place.

I love this.

It sounds (from IRC) like you're on board with the idea of having
nattka add kw:security to appropriate stabilization bugs.

Could nattka also do something to the security@-assigned but itself to
indicate that all security-supported architecture have been handled?
Something like leaving a comment or fiddling with the security
whiteboard.

It would be nice to be able to resolve the security@-assigned but
before non-security-supported architectures are handled.

To do that, I think we'd want to change what's required for the "clean
up" step. Since today the "clean up" step is dropping the vulnerable
package versions from the tree, it is dependent on
non-security-supported architectures completing the stabilization bug.
I think we'd like to break that dependence.

I suggest that we redefine the "clean up" step to be: drop
security-supported architectures' keywords from vulnerable versions.
That would allow the security@-assigned bug to be resolved before
non-security-supported architectures are finished with stabilization.

Re: [gentoo-dev] [RFC] Decoupling stabilization from security bugs

[Michał Górny]

On Thu, 2021-08-12 at 13:17 -0700, Matt Turner wrote:
> On Thu, Aug 12, 2021 at 5:53 AM Michał Górny <mgorny@gentoo.org> wrote:
> > 
> > Hello, everyone.
> > 
> > TL;DR: I'd like to propose that stabilizations are done via blockers of
> > security bugs instead of security bugs themselves, i.e. as any other
> > stabilizations.
> > 
> > 
> > Right now we're often performing security-related stabilizations via
> > security bugs. This has a few problems, that are:
> > 
> > 1. Stabilization-related activity causes unnecessary mail to the widely
> > subscribed security alias. That is, subscribed people get notified of
> > package list changes, NATTkA results, every arch doing its work.
> > However, in reality the security team only cares about stabilization
> > being started, stalled or finished -- and for that, getting the usual
> > 'dependent bug added/closed' mail should be sufficient.
> > 
> > 2. NATTkA has no good way of distinguishing irrelevant security bugs
> > from security bugs where something went wrong (and NATTkA doesn't use
> > persistent state by design). The most important problem is that --
> > unlike regular stablereqs -- security bugs aren't supposed to be closed
> > after stabilization. It can't really distinguish a security bug 'left
> > open' from a security bug with incorrect package list.
> > 
> > 3. Proxied maintainers without editbugs can't actually CC arches on
> > security bugs since the bugs are assigned to security@.
> > 
> > 
> > To resolve these problems going forward and establish consistent
> > behavior in the future, I'd like to propose to disable 'package list'
> > fields on security bugs and instead expect regular stabilization bugs to
> > be used (and made block the security bugs) for stabilizations. While I
> > understand that filing additional bugs might be cumbersome for some
> > people, I don't think it's such a herculean effort to outweigh
> > the problems solved.
> > 
> > In the end, consistency is a good thing and we've introduced a dedicated
> > stabilization category to reduce the spread of stabilization bugs all
> > around the place.
> 
> I love this.
> 
> It sounds (from IRC) like you're on board with the idea of having
> nattka add kw:security to appropriate stabilization bugs.
> 
> Could nattka also do something to the security@-assigned but itself to
> indicate that all security-supported architecture have been handled?
> Something like leaving a comment or fiddling with the security
> whiteboard.

Yeah, this shouldn't be a problem.  However, I would prefer if 'security
supported' architectures were defined somewhere reasonably standard,
rather than hardcoded in nattka.

> 
> It would be nice to be able to resolve the security@-assigned but
> before non-security-supported architectures are handled.
> 
> To do that, I think we'd want to change what's required for the "clean
> up" step. Since today the "clean up" step is dropping the vulnerable
> package versions from the tree, it is dependent on
> non-security-supported architectures completing the stabilization bug.
> I think we'd like to break that dependence.
> 
> I suggest that we redefine the "clean up" step to be: drop
> security-supported architectures' keywords from vulnerable versions.
> That would allow the security@-assigned bug to be resolved before
> non-security-supported architectures are finished with stabilization.
> 

To be honest, this sounds like doubling the effort for little benefit. 
After all, removing the old version of the package doesn't resolve any
problems on the end user systems -- upgrading does, and upgrading
usually happens entirely independently of whether we've cleaned up or
not.

Maybe it'd easier to release GLSAs before cleanup happens?  We can
always go the dekeywording way if arch teams are actually slacking.

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] [RFC] Decoupling stabilization from security bugs

[Jaco Kroon]

Hi,

>> It would be nice to be able to resolve the security@-assigned but
>> before non-security-supported architectures are handled.
>>
>> To do that, I think we'd want to change what's required for the "clean
>> up" step. Since today the "clean up" step is dropping the vulnerable
>> package versions from the tree, it is dependent on
>> non-security-supported architectures completing the stabilization bug.
>> I think we'd like to break that dependence.
>>
>> I suggest that we redefine the "clean up" step to be: drop
>> security-supported architectures' keywords from vulnerable versions.
>> That would allow the security@-assigned bug to be resolved before
>> non-security-supported architectures are finished with stabilization.
>>
> To be honest, this sounds like doubling the effort for little benefit. 
> After all, removing the old version of the package doesn't resolve any
> problems on the end user systems -- upgrading does, and upgrading
> usually happens entirely independently of whether we've cleaned up or
> not.
>
> Maybe it'd easier to release GLSAs before cleanup happens?  We can
> always go the dekeywording way if arch teams are actually slacking.
>
I agree with both of you.  In particular, cleaning old versions should
not be a requirement for releasing the GLSA.  The faster the GLSA can
get out the better.  Removing the keywords is a novel idea, but honestly
not sure it's worth the effort.

Kind Regards,
Jaco

Re: [gentoo-dev] [RFC] Decoupling stabilization from security bugs

[Aaron Bauman]

[-- Attachment #1: Type: text/plain, Size: 1931 bytes --]

On Thu, Aug 12, 2021 at 01:17:32PM -0700, Matt Turner wrote:
> On Thu, Aug 12, 2021 at 5:53 AM Michał Górny <mgorny@gentoo.org> wrote:
> >

<snip>

> To do that, I think we'd want to change what's required for the "clean
> up" step. Since today the "clean up" step is dropping the vulnerable
> package versions from the tree, it is dependent on
> non-security-supported architectures completing the stabilization bug.
> I think we'd like to break that dependence.
>

Yes, please. Thank you for bringing this up. This has been a hotly
debated item in the past with former security leads dictating that
"clean up" is not relevant to the security process, but it remained
codified in documentation that it needs to occur.

It is indeed important, as leaving vulnerable versions is the tree is not
good for anyone and impacts many other areas (e.g. promoting tree
cleanliness).

Further, as mgorny mentioned in a follow-up email to this, we need to
understand what is a "security supported" architecture. This has also
been an issue in the past with council intervention needed to declare
dropping specific arches to exp profiles and allowing security to drop
support and subsequently move bugs forward.

And to continue on my soap box, we have a small blurb on the security
page [1] which states what architectures are considered security supported.
This is less than ideal. We also generally state that stable arches are
supported and must be dealt with during the vulnerability process.

So, all in all, it is highly conflated IMHO and is *not* ideal for
anyone to have to determine that a particular arch is stable but not
security supported. 

As such, I advocate for any stable arch to be security supported by
default. If the arch lags or is dropped then it goes to unstable
(process TBD).


[1]: https://www.gentoo.org/support/security/vulnerability-treatment-policy.html

-Aaron

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] [RFC] Decoupling stabilization from security bugs

[Michał Górny]

On Fri, 2021-08-13 at 12:50 -0400, Aaron Bauman wrote:
> On Thu, Aug 12, 2021 at 01:17:32PM -0700, Matt Turner wrote:
> > On Thu, Aug 12, 2021 at 5:53 AM Michał Górny <mgorny@gentoo.org> wrote:
> > > 
> 
> <snip>
> 
> > To do that, I think we'd want to change what's required for the "clean
> > up" step. Since today the "clean up" step is dropping the vulnerable
> > package versions from the tree, it is dependent on
> > non-security-supported architectures completing the stabilization bug.
> > I think we'd like to break that dependence.
> > 
> 
> Yes, please. Thank you for bringing this up. This has been a hotly
> debated item in the past with former security leads dictating that
> "clean up" is not relevant to the security process, but it remained
> codified in documentation that it needs to occur.
> 
> It is indeed important, as leaving vulnerable versions is the tree is not
> good for anyone and impacts many other areas (e.g. promoting tree
> cleanliness).
> 
> Further, as mgorny mentioned in a follow-up email to this, we need to
> understand what is a "security supported" architecture. This has also
> been an issue in the past with council intervention needed to declare
> dropping specific arches to exp profiles and allowing security to drop
> support and subsequently move bugs forward.
> 
> And to continue on my soap box, we have a small blurb on the security
> page [1] which states what architectures are considered security supported.
> This is less than ideal. We also generally state that stable arches are
> supported and must be dealt with during the vulnerability process.
> 
> So, all in all, it is highly conflated IMHO and is *not* ideal for
> anyone to have to determine that a particular arch is stable but not
> security supported. 
> 
> As such, I advocate for any stable arch to be security supported by
> default. If the arch lags or is dropped then it goes to unstable
> (process TBD).
> 

Yes, this sounds like a good idea.  We should avoid having multiple
classifications for architectures.  If something's good enough to be
stable, it should be security supported.  If it's not, then it shouldn't
be stable.

In the end, I guess the primary problem is manpower.  Given that secbugs
are normally given priority, I don't really see a case for an arch that
would be not good enough to be security supported but at the same time
good enough to cope with the wider range of bugs.

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] [RFC] Decoupling stabilization from security bugs

[Lars Wendler]

[-- Attachment #1: Type: text/plain, Size: 2242 bytes --]

Hi Michał,

On Thu, 12 Aug 2021 14:53:33 +0200 Michał Górny wrote:

>Hello, everyone.
>
>TL;DR: I'd like to propose that stabilizations are done via blockers of
>security bugs instead of security bugs themselves, i.e. as any other
>stabilizations.
>
>
>Right now we're often performing security-related stabilizations via
>security bugs. This has a few problems, that are:
>
>1. Stabilization-related activity causes unnecessary mail to the widely
>subscribed security alias. That is, subscribed people get notified of
>package list changes, NATTkA results, every arch doing its work.
>However, in reality the security team only cares about stabilization
>being started, stalled or finished -- and for that, getting the usual
>'dependent bug added/closed' mail should be sufficient.
>
>2. NATTkA has no good way of distinguishing irrelevant security bugs
>from security bugs where something went wrong (and NATTkA doesn't use
>persistent state by design). The most important problem is that --
>unlike regular stablereqs -- security bugs aren't supposed to be closed
>after stabilization. It can't really distinguish a security bug 'left
>open' from a security bug with incorrect package list.
>
>3. Proxied maintainers without editbugs can't actually CC arches on
>security bugs since the bugs are assigned to security@.
>
>
>To resolve these problems going forward and establish consistent
>behavior in the future, I'd like to propose to disable 'package list'
>fields on security bugs and instead expect regular stabilization bugs
>to be used (and made block the security bugs) for stabilizations.
>While I understand that filing additional bugs might be cumbersome for
>some people, I don't think it's such a herculean effort to outweigh
>the problems solved.

Indeed, filing stablereq bugs is not really that big of a deal.

>In the end, consistency is a good thing and we've introduced a
>dedicated stabilization category to reduce the spread of stabilization
>bugs all around the place.
>
>WDYT?
>

I like this proposal and fully support it. Thanks for bringing it up.

Cheers
-- 
Lars Wendler
Gentoo package maintainer
GPG: 21CC CF02 4586 0A07 ED93  9F68 498F E765 960E 9B39

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]