[gentoo-dev] Keywordreqs and slacking arch teams

[gentoo-dev] Keywordreqs and slacking arch teams

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1175 bytes --]

Hello,

The Python team ends up filing a lot of keywordreqs due to new
dependencies.  Many of them end up open for many months, and start
listing obsolete package versions.  Then an arch team wakes up...
and adds keywords to a version that's supposed to be removed already. 
Or complains that the package list is outdated.

I think it's generally a reasonable assumption that keywordreq should be
applied to the newest version of a package, unless the keywordreq
explicitly says otherwise (in the comment).  It's not helpful that
stable-bot requires us to fill specific versions here.

I don't think it's fair to expect package maintainers to keep package
versions up-to-date in this case.  I can take the blame if the package
list becomes outdated, say, in 1 months.  If the arch team can't keyword
something in 6 months, I blame them, and I believe it should be their
responsibility to update the keywordreq.

Otherwise, we're creating a silly workflow where I keep putting
an effort into keeping the keywordreq up-to-date, hoping that one day
arch teams might actually act upon it.

How can we improve this?

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] Keywordreqs and slacking arch teams

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 500 bytes --]

On Sat, 28 Dec 2019 08:09:33 +0100
Michał Górny <mgorny@gentoo.org> wrote:

> How can we improve this?

Every time this kind of issue comes up, I can't help feeling we need
some sort of tool more advanced than what we currently have.

Something that maintains persistence of keyword demands similar to how
the current bot does, but more ... 

Its the sort of thing I get tempted to implement myself, but get too
horrified about the prospect of working with portage internals to do it.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Keywordreqs and slacking arch teams

[Fabian Groffen]

[-- Attachment #1: Type: text/plain, Size: 955 bytes --]

On 28-12-2019 22:27:02 +1300, Kent Fredric wrote:
> On Sat, 28 Dec 2019 08:09:33 +0100
> Michał Górny <mgorny@gentoo.org> wrote:
> 
> > How can we improve this?
> 
> Every time this kind of issue comes up, I can't help feeling we need
> some sort of tool more advanced than what we currently have.
> 
> Something that maintains persistence of keyword demands similar to how
> the current bot does, but more ... 
> 
> Its the sort of thing I get tempted to implement myself, but get too
> horrified about the prospect of working with portage internals to do it.

Hmmm, interested to hear what kind of things you're thinking about here.

I feel it would help if we would have the ability to at least
compile/test ebuilds automatically.  Not sure how helpful qemu could be
there, but I know things like compiling for things like arm (RPi) works
reasonably well.

Thanks,
Fabian


-- 
Fabian Groffen
Gentoo on a different level

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] Keywordreqs and slacking arch teams

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 1999 bytes --]

On Sat, 28 Dec 2019 10:35:09 +0100
Fabian Groffen <grobian@gentoo.org> wrote:

> Hmmm, interested to hear what kind of things you're thinking about here.

A lot of the "Work" of filing a keyword request is modelling all the
consequential keywordings that have to take place.

If there was say, a web based UI, that:

- Automatically determined which packages are ready for stabilization
  due to all their dependencies already being stable (and maybe with
  automatic cooldown-from-testing detection )

- Automatically determined which packages can be keyworded without
  additional work due to all their dependencies being keyworded

- When requesting keywording/stabilization, automatically determined
  what all the consequences are and what else needs to be keyworded to
  satisfy it

- Allowed a simple "Add keyword(s) <y..> for package <x>" interface,
  that intelligently created an issue and a target list, and then once
  the list was built, constantly ensured the list to be valid, or
  determined automatically when sub-work was completed and reducing the
  published list automatically, and then responded to potential issues
  based on changes in git, ( as opposed to being only triggered when
  the bug was touched )

Most of the "pain" and legwork required by maintainers would go away.

As it is, I feel a lot of us are reproducing a lot of logic that is
rather routine and could be automated.

But the overall idea here is to orient the point of keyword-requests in
some way to focus on the primary objective, where the developer
indicates their intent, and the system's job is to facilitate that
intent coming to fruition, pointing out problems on its own.

( I have somewhat hacked together some perl scripts for myself for some
of these tasks, but the command-line interface is not ideal for this
workflow, and the code is not in a condition I can share it, and I
don't think perl is the right language to address this problem with )


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Keywordreqs and slacking arch teams

[Michael 'veremitz' Everitt]

[-- Attachment #1.1: Type: text/plain, Size: 1031 bytes --]

On 28/12/19 11:05, Kent Fredric wrote:
> On Sat, 28 Dec 2019 10:35:09 +0100
> Fabian Groffen <grobian@gentoo.org> wrote:
>
>> Hmmm, interested to hear what kind of things you're thinking about here.
> A lot of the "Work" of filing a keyword request is modelling all the
> consequential keywordings that have to take place.
>
> If there was say, a web based UI, that:
>
> - Automatically determined which packages are ready for stabilization
>   due to all their dependencies already being stable (and maybe with
>   automatic cooldown-from-testing detection )
>
> - Automatically determined which packages can be keyworded without
>   additional work due to all their dependencies being keyworded
<snip>

I know I'm gonna be shot down in flames, because $heresy, but here is where
a package 'database' would actually work quite well, because you can
trivially create a query that pulls this data out, and sorts it by package
category or maintainer or whatever you like ..

Ok, let the flamewars begin ...


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

Re: [gentoo-dev] Keywordreqs and slacking arch teams

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 1037 bytes --]

On Sat, 28 Dec 2019 11:14:15 +0000
Michael 'veremitz' Everitt <gentoo@veremit.xyz> wrote:

> I know I'm gonna be shot down in flames, because $heresy, but here is where
> a package 'database' would actually work quite well, because you can
> trivially create a query that pulls this data out, and sorts it by package
> category or maintainer or whatever you like ..
> 
> Ok, let the flamewars begin ...

There's no real problem with a package database, however, the real
limitation is in the ebuild source format, which ultimately means any
such database needs a lot of bash-sourcing hell to simply stay
up-to-date ( any time an eclass changes, the interpretation of every
ebuild that uses it also changes, necessitating some pretty fun(1) code )

And that winds you up fighting with portage internals.

So simply, in order for somebody like me to actually implement such a
thing, a precursory step is to rewrite enough portage to do just that.

But I haven't (yet) gotten around to that.


1: Not actually fun.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Keywordreqs and slacking arch teams

[James Le Cuirot]

[-- Attachment #1: Type: text/plain, Size: 1378 bytes --]

On Sun, 29 Dec 2019 00:27:36 +1300
Kent Fredric <kentnl@gentoo.org> wrote:

> On Sat, 28 Dec 2019 11:14:15 +0000
> Michael 'veremitz' Everitt <gentoo@veremit.xyz> wrote:
> 
> > I know I'm gonna be shot down in flames, because $heresy, but here is where
> > a package 'database' would actually work quite well, because you can
> > trivially create a query that pulls this data out, and sorts it by package
> > category or maintainer or whatever you like ..
> > 
> > Ok, let the flamewars begin ...  
> 
> There's no real problem with a package database, however, the real
> limitation is in the ebuild source format, which ultimately means any
> such database needs a lot of bash-sourcing hell to simply stay
> up-to-date ( any time an eclass changes, the interpretation of every
> ebuild that uses it also changes, necessitating some pretty fun(1) code )
> 
> And that winds you up fighting with portage internals.
> 
> So simply, in order for somebody like me to actually implement such a
> thing, a precursory step is to rewrite enough portage to do just that.
> 
> But I haven't (yet) gotten around to that.
> 
> 
> 1: Not actually fun.

Doesn't https://packages.gentoo.org already have such a database?
Unfortunately a3li used Elasticsearch, which no one understands, but
it's a start.

-- 
James Le Cuirot (chewi)
Gentoo Linux Developer

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Keywordreqs and slacking arch teams

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 239 bytes --]

On Sat, 28 Dec 2019 11:40:08 +0000
James Le Cuirot <chewi@gentoo.org> wrote:

> Unfortunately a3li used Elasticsearch, which no one understands, but
> it's a start.

And I've used ES enough to say I would rather never use it again.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Keywordreqs and slacking arch teams

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 945 bytes --]

On Sat, 28 Dec 2019 11:14:15 +0000
Michael 'veremitz' Everitt <gentoo@veremit.xyz> wrote:

> I know I'm gonna be shot down in flames, because $heresy, but here is where
> a package 'database' would actually work quite well, because you can
> trivially create a query that pulls this data out, and sorts it by package
> category or maintainer or whatever you like .

Oh, and once upon a time, there was actually a trick you could do to
make portage keep its metadata caches in an SQLite database, which had
its benefits, and I even had a rough tool I wrote in ruby and shared on
the old (defunct) wiki that helped do quick database queries.

But the trick actually makes portage slower in multiple ways, and it
never really got widespread love.

( Though I would argue how the data was stored was also inadequate for
a lot of other tasks one might want to do with a database, like,
keeping DEPEND stored in its string format )

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Keywordreqs and slacking arch teams

[Michael 'veremitz' Everitt]

[-- Attachment #1.1: Type: text/plain, Size: 1324 bytes --]

On 28/12/19 11:32, Kent Fredric wrote:
> On Sat, 28 Dec 2019 11:14:15 +0000
> Michael 'veremitz' Everitt <gentoo@veremit.xyz> wrote:
>
>> I know I'm gonna be shot down in flames, because $heresy, but here is where
>> a package 'database' would actually work quite well, because you can
>> trivially create a query that pulls this data out, and sorts it by package
>> category or maintainer or whatever you like .
> Oh, and once upon a time, there was actually a trick you could do to
> make portage keep its metadata caches in an SQLite database, which had
> its benefits, and I even had a rough tool I wrote in ruby and shared on
> the old (defunct) wiki that helped do quick database queries.
>
> But the trick actually makes portage slower in multiple ways, and it
> never really got widespread love.
>
> ( Though I would argue how the data was stored was also inadequate for
> a lot of other tasks one might want to do with a database, like,
> keeping DEPEND stored in its string format )
Note:  we're nnot acttually talking about replacing portage here, just
creating a tool ((((thiink php script web tthingy)) that will do some of
the pre-screeninng worrrrk that AT hate (eg.... what kensiington did  with
stable-bbot)

* with apologies for keyboard/remote-access lagggg creating typo hell.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

Re: [gentoo-dev] Keywordreqs and slacking arch teams

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 887 bytes --]

On Sat, 28 Dec 2019 11:35:49 +0000
Michael 'veremitz' Everitt <gentoo@veremit.xyz> wrote:

> Note:  we're nnot acttually talking about replacing portage here, just
> creating a tool ((((thiink php script web tthingy)) that will do some of
> the pre-screeninng worrrrk that AT hate (eg.... what kensiington did  with
> stable-bbot)
> 
> * with apologies for keyboard/remote-access lagggg creating typo hell.

But, doing that requires viewing realised copies of ebuilds, which
requires interpreting eclasses and variable interpolation, which
requires bash sourcing, which requires a mountain of portage hell.

Yes, sharing the stable-bot logic would probably be fine.

But it doesn't use a database AFAIK, it would likely just be making use
of the MD5Cache (either directly, or indirectly via portage APIs)

But I won't be volunteering, because I won't touch python.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Keywordreqs and slacking arch teams

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 1052 bytes --]

On Sat, Dec 28, 2019 at 3:43 AM Kent Fredric <kentnl@gentoo.org> wrote:

> On Sat, 28 Dec 2019 11:35:49 +0000
> Michael 'veremitz' Everitt <gentoo@veremit.xyz> wrote:
>
> > Note:  we're nnot acttually talking about replacing portage here, just
> > creating a tool ((((thiink php script web tthingy)) that will do some of
> > the pre-screeninng worrrrk that AT hate (eg.... what kensiington did
> with
> > stable-bbot)
> >
> > * with apologies for keyboard/remote-access lagggg creating typo hell.
>
> But, doing that requires viewing realised copies of ebuilds, which
> requires interpreting eclasses and variable interpolation, which
> requires bash sourcing, which requires a mountain of portage hell.
>

> Yes, sharing the stable-bot logic would probably be fine.
>
> But it doesn't use a database AFAIK, it would likely just be making use
> of the MD5Cache (either directly, or indirectly via portage APIs)
>
> But I won't be volunteering, because I won't touch python.
>

You could of course work together with someone else to write the tool?

-A

[-- Attachment #2: Type: text/html, Size: 1667 bytes --]

Re: [gentoo-dev] Keywordreqs and slacking arch teams

[Aaron Bauman]

[-- Attachment #1: Type: text/plain, Size: 1419 bytes --]

On Sat, Dec 28, 2019 at 10:05:02AM -0800, Alec Warner wrote:
> On Sat, Dec 28, 2019 at 3:43 AM Kent Fredric <kentnl@gentoo.org> wrote:
> 
> > On Sat, 28 Dec 2019 11:35:49 +0000
> > Michael 'veremitz' Everitt <gentoo@veremit.xyz> wrote:
> >
> > > Note:  we're nnot acttually talking about replacing portage here, just
> > > creating a tool ((((thiink php script web tthingy)) that will do some of
> > > the pre-screeninng worrrrk that AT hate (eg.... what kensiington did
> > with
> > > stable-bbot)
> > >
> > > * with apologies for keyboard/remote-access lagggg creating typo hell.
> >
> > But, doing that requires viewing realised copies of ebuilds, which
> > requires interpreting eclasses and variable interpolation, which
> > requires bash sourcing, which requires a mountain of portage hell.
> >
> 
> > Yes, sharing the stable-bot logic would probably be fine.
> >
> > But it doesn't use a database AFAIK, it would likely just be making use
> > of the MD5Cache (either directly, or indirectly via portage APIs)
> >
> > But I won't be volunteering, because I won't touch python.
> >
> 
> You could of course work together with someone else to write the tool?
> 
> -A

Ah, you found the Achilles heel. It is much easier to postulate on the mailing
list, use big words, and then say you just won't do that thing because
tools/languages such.

Perl though...

-- 
Cheers,
Aaron

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] Keywordreqs and slacking arch teams

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 529 bytes --]

On Sat, 28 Dec 2019 21:19:18 -0500
Aaron Bauman <bman@gentoo.org> wrote:

> Ah, you found the Achilles heel. It is much easier to postulate on the mailing
> list, use big words, and then say you just won't do that thing because
> tools/languages such.
> 
> Perl though...

FTR: Even though I'm good at Perl, I wouldn't use it for this task.

There are a lot of reasons behind this.

But at least one of them would be if I wrote it in Perl, I'd have
nobody else contributing either.

And IME, that's fair enough.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Keywordreqs and slacking arch teams

[A Schenck]

On 12/28/19 3:14 AM, Michael 'veremitz' Everitt wrote:
> On 28/12/19 11:05, Kent Fredric wrote:
>> On Sat, 28 Dec 2019 10:35:09 +0100
>> Fabian Groffen <grobian@gentoo.org> wrote:
>>
>>> Hmmm, interested to hear what kind of things you're thinking about here.
>> A lot of the "Work" of filing a keyword request is modelling all the
>> consequential keywordings that have to take place.
>>
>> If there was say, a web based UI, that:
>>
>> - Automatically determined which packages are ready for stabilization
>>   due to all their dependencies already being stable (and maybe with
>>   automatic cooldown-from-testing detection )
>>
>> - Automatically determined which packages can be keyworded without
>>   additional work due to all their dependencies being keyworded
> <snip>
>
> I know I'm gonna be shot down in flames, because $heresy, but here is where
> a package 'database' would actually work quite well, because you can
> trivially create a query that pulls this data out, and sorts it by package
> category or maintainer or whatever you like ..
app-portage/kuroo manages an SQLite database of packages without any
bash sourcing craziness, but defers to `emerge` to do the actual install
/ uninstall work and parses stdout and stderr from it.  DB schema is
described at
https://sourceforge.net/p/kuroo/code/HEAD/tree/kuroo4/trunk/src/core/portagedb.cpp#l219
and there are queries for packages by category / subcategory, installed
/ updateable status, and search string there.  Code to walk the main
tree from md5-cache and fill in those tables starts about
https://sourceforge.net/p/kuroo/code/HEAD/tree/kuroo4/trunk/src/core/scanportagejob.cpp#l135
.  Back before burnout there was some hope of porting it to a 'unified
portage API' that dol-sen wanted to build for other portage tools to
use, but that never came to fruition.  Most of this code is 15 years old
though, and I've only done the bare minimum to keep it working-ish as
portage and the tree have changed.
>
> Ok, let the flamewars begin ...
>

-A

Re: [gentoo-dev] Keywordreqs and slacking arch teams

[Rolf Eike Beer]

[-- Attachment #1: Type: text/plain, Size: 1926 bytes --]

> - Allowed a simple "Add keyword(s) <y..> for package <x>" interface,
>   that intelligently created an issue and a target list, and then once
>   the list was built, constantly ensured the list to be valid, or
>   determined automatically when sub-work was completed and reducing the
>   published list automatically, and then responded to potential issues
>   based on changes in git, ( as opposed to being only triggered when
>   the bug was touched )

As someone who does both keywordings and stabilizations regularly on hppa and 
sparc I think I must share a bit of my experiences:

-some arches are regularly forgotten to be CC'ed, which happens for the above 
arches quite regularly as they are exp

-if I need to do a bug at a later point when I want to newly stabilize a given 
package for a new arch it is extremely helpful if

  - the package list was not reduced on a later point because parts were 
already handled

  - arch specifications for packages are reduced to the absolute need, i.e. 
especially not given if the arch list would match the initial CC list

I use tatt for my work, and that automatically sorts out all packages that 
have non-matching package list. Sure, there could be improvements for several 
things in tatt, but that is IMHO absolutely right the way it is. So if you 
give all arches and I later decide to do the same bug on an additional arch 
then it will not do a single package.

So if you want my work easier, then
-don't forget to CC exp arches
-don't clean the package list only because packages are already done
-let tatt run on your dev box, or preferably in a new chroot yourself, on your 
package, and fix all the broken dependencies and stuff there yourself. Your 
amd64 laptop is still way faster than my crowded C8000, and doing a roundtrip 
through the bugtracker until you find time to fix it will just make you think 
of "slacking arch teams" next time.

Thanks,

Eike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] Keywordreqs and slacking arch teams

[Mike Pagano]

[-- Attachment #1: Type: text/plain, Size: 963 bytes --]

On Thursday, January 2, 2020 3:32:12 PM EST Rolf Eike Beer wrote:
> > - Allowed a simple "Add keyword(s) <y..> for package <x>" interface,
> > 
> >   that intelligently created an issue and a target list, and then once
> >   the list was built, constantly ensured the list to be valid, or
> >   determined automatically when sub-work was completed and reducing the
> >   published list automatically, and then responded to potential issues
> >   based on changes in git, ( as opposed to being only triggered when
> >   the bug was touched )
> 
> As someone who does both keywordings and stabilizations regularly on hppa
> and sparc I think I must share a bit of my experiences:

<snip>

hppa is making us keep old kernels around [1].  Should the kernel team be 
doing more to get your attention then CC'ing hppa on all of the kernel 
STABLEREQ bugs [2]?

[1] https://packages.gentoo.org/packages/sys-kernel/gentoo-sources
[2] https://bugs.gentoo.org/700416

Mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Keywordreqs and slacking arch teams

[Rolf Eike Beer]

[-- Attachment #1: Type: text/plain, Size: 1397 bytes --]

Am Freitag, 3. Januar 2020, 00:25:06 CET schrieb Mike Pagano:
> On Thursday, January 2, 2020 3:32:12 PM EST Rolf Eike Beer wrote:
> > > - Allowed a simple "Add keyword(s) <y..> for package <x>" interface,
> > > 
> > >   that intelligently created an issue and a target list, and then once
> > >   the list was built, constantly ensured the list to be valid, or
> > >   determined automatically when sub-work was completed and reducing the
> > >   published list automatically, and then responded to potential issues
> > >   based on changes in git, ( as opposed to being only triggered when
> > >   the bug was touched )
> > 
> > As someone who does both keywordings and stabilizations regularly on hppa
> 
> > and sparc I think I must share a bit of my experiences:
> <snip>
> 
> hppa is making us keep old kernels around [1].  Should the kernel team be
> doing more to get your attention then CC'ing hppa on all of the kernel
> STABLEREQ bugs [2]?

I only run vanilla-sources since there are still lot of cache corruption 
problems in hppa kernels, or whatever makes them flaky.

Linux pioneer 5.4.6-parisc64 #1 SMP Fri Dec 27 10:23:09 CET 2019 parisc64 
PA8800 (Mako) 9000/785/C8000 GNU/Linux
Linux voyager 5.4.6-parisc #1 Fri Dec 27 15:46:43 CET 2019 parisc PA8600 (PCX-
W+) 9000/785/C3600 GNU/Linux

So _I_ personally would say just drop old kernels, but that is in no way 
authorative.

Eike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] Keywordreqs and slacking arch teams

[Michael 'veremitz' Everitt]

[-- Attachment #1.1: Type: text/plain, Size: 1569 bytes --]

On 02/01/20 23:35, Rolf Eike Beer wrote:
> Am Freitag, 3. Januar 2020, 00:25:06 CET schrieb Mike Pagano:
>> On Thursday, January 2, 2020 3:32:12 PM EST Rolf Eike Beer wrote:
>>>> - Allowed a simple "Add keyword(s) <y..> for package <x>" interface,
>>>>
>>>>   that intelligently created an issue and a target list, and then once
>>>>   the list was built, constantly ensured the list to be valid, or
>>>>   determined automatically when sub-work was completed and reducing the
>>>>   published list automatically, and then responded to potential issues
>>>>   based on changes in git, ( as opposed to being only triggered when
>>>>   the bug was touched )
>>> As someone who does both keywordings and stabilizations regularly on hppa
>>> and sparc I think I must share a bit of my experiences:
>> <snip>
>>
>> hppa is making us keep old kernels around [1].  Should the kernel team be
>> doing more to get your attention then CC'ing hppa on all of the kernel
>> STABLEREQ bugs [2]?
> I only run vanilla-sources since there are still lot of cache corruption 
> problems in hppa kernels, or whatever makes them flaky.
>
> Linux pioneer 5.4.6-parisc64 #1 SMP Fri Dec 27 10:23:09 CET 2019 parisc64 
> PA8800 (Mako) 9000/785/C8000 GNU/Linux
> Linux voyager 5.4.6-parisc #1 Fri Dec 27 15:46:43 CET 2019 parisc PA8600 (PCX-
> W+) 9000/785/C3600 GNU/Linux
>
> So _I_ personally would say just drop old kernels, but that is in no way 
> authorative.
>
> Eike
Is it viable at all to test gentoo-sources or would it be better simply to
unkeyword?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

Re: [gentoo-dev] Keywordreqs and slacking arch teams

[Aaron Bauman]

On January 2, 2020 6:35:08 PM EST, Rolf Eike Beer <eike@sf-mail.de> wrote:
>Am Freitag, 3. Januar 2020, 00:25:06 CET schrieb Mike Pagano:
>> On Thursday, January 2, 2020 3:32:12 PM EST Rolf Eike Beer wrote:
>> > > - Allowed a simple "Add keyword(s) <y..> for package <x>"
>interface,
>> > > 
>> > >   that intelligently created an issue and a target list, and then
>once
>> > >   the list was built, constantly ensured the list to be valid, or
>> > >   determined automatically when sub-work was completed and
>reducing the
>> > >   published list automatically, and then responded to potential
>issues
>> > >   based on changes in git, ( as opposed to being only triggered
>when
>> > >   the bug was touched )
>> > 
>> > As someone who does both keywordings and stabilizations regularly
>on hppa
>> 
>> > and sparc I think I must share a bit of my experiences:
>> <snip>
>> 
>> hppa is making us keep old kernels around [1].  Should the kernel
>team be
>> doing more to get your attention then CC'ing hppa on all of the
>kernel
>> STABLEREQ bugs [2]?
>
>I only run vanilla-sources since there are still lot of cache
>corruption 
>problems in hppa kernels, or whatever makes them flaky.
>
>Linux pioneer 5.4.6-parisc64 #1 SMP Fri Dec 27 10:23:09 CET 2019
>parisc64 
>PA8800 (Mako) 9000/785/C8000 GNU/Linux
>Linux voyager 5.4.6-parisc #1 Fri Dec 27 15:46:43 CET 2019 parisc
>PA8600 (PCX-
>W+) 9000/785/C3600 GNU/Linux
>
>So _I_ personally would say just drop old kernels, but that is in no
>way 
>authorative.
>
>Eike

Ugh. gentoo-sources is just a patch (trivial) on top of vanilla-kernel sources of each stable and LTS version.

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [gentoo-dev] Keywordreqs and slacking arch teams

[Rolf Eike Beer]

[-- Attachment #1: Type: text/plain, Size: 1624 bytes --]

Am Freitag, 3. Januar 2020, 03:40:35 CET schrieb Aaron Bauman:
> On January 2, 2020 6:35:08 PM EST, Rolf Eike Beer <eike@sf-mail.de> wrote:
> >Am Freitag, 3. Januar 2020, 00:25:06 CET schrieb Mike Pagano:

> >> hppa is making us keep old kernels around [1].  Should the kernel team be
> >> doing more to get your attention then CC'ing hppa on all of the kernel
> >> STABLEREQ bugs [2]?
> >
> >I only run vanilla-sources since there are still lot of cache
> >corruption
> >problems in hppa kernels, or whatever makes them flaky.
> >
> >Linux pioneer 5.4.6-parisc64 #1 SMP Fri Dec 27 10:23:09 CET 2019 parisc64
> >PA8800 (Mako) 9000/785/C8000 GNU/Linux
> >Linux voyager 5.4.6-parisc #1 Fri Dec 27 15:46:43 CET 2019 parisc PA8600
> >(PCX-W+) 9000/785/C3600 GNU/Linux
> >
> >So _I_ personally would say just drop old kernels, but that is in no
> >way
> >authorative.

> Ugh. gentoo-sources is just a patch (trivial) on top of vanilla-kernel
> sources of each stable and LTS version.

If it's just that I could test them, but this still be no LTS version that I 
would look at.

Just some background: these machines run CMake and some other software nightly 
builds, which starts ~3:00 and takes them until the afternoon. I have chroots 
for stabilization and keywording on them, and the tatt scripts will wait if 
any process of my buildbot user (that's just the name, not the software) is 
active. If the nightlies are done, then the tatt scripts will continue and hog 
the machine.

I will do a kernel upgrade usually if the machine crashed anyway, which means 
every few weeks, and then I go to the most recent version.

Eike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] Keywordreqs and slacking arch teams

[Rolf Eike Beer]

[-- Attachment #1: Type: text/plain, Size: 1343 bytes --]

Am Freitag, 3. Januar 2020, 11:00:14 CET schrieb Rolf Eike Beer:
> Am Freitag, 3. Januar 2020, 03:40:35 CET schrieb Aaron Bauman:
> > On January 2, 2020 6:35:08 PM EST, Rolf Eike Beer <eike@sf-mail.de> wrote:
> > >Am Freitag, 3. Januar 2020, 00:25:06 CET schrieb Mike Pagano:
> > >> hppa is making us keep old kernels around [1].  Should the kernel team
> > >> be
> > >> doing more to get your attention then CC'ing hppa on all of the kernel
> > >> STABLEREQ bugs [2]?
> > >
> > >I only run vanilla-sources since there are still lot of cache
> > >corruption
> > >problems in hppa kernels, or whatever makes them flaky.
> > >
> > >Linux pioneer 5.4.6-parisc64 #1 SMP Fri Dec 27 10:23:09 CET 2019 parisc64
> > >PA8800 (Mako) 9000/785/C8000 GNU/Linux
> > >Linux voyager 5.4.6-parisc #1 Fri Dec 27 15:46:43 CET 2019 parisc PA8600
> > >(PCX-W+) 9000/785/C3600 GNU/Linux
> > >
> > >So _I_ personally would say just drop old kernels, but that is in no
> > >way
> > >authorative.
> > 
> > Ugh. gentoo-sources is just a patch (trivial) on top of vanilla-kernel
> > sources of each stable and LTS version.
> 
> If it's just that I could test them, but this still be no LTS version that I
> would look at.

So, do you want me to stable a random gentoo-sources (usually the most recent 
one) every few weeks when I just happen to upgrade my machine?

Eike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] Keywordreqs and slacking arch teams

[Michael 'veremitz' Everitt]

[-- Attachment #1.1: Type: text/plain, Size: 2152 bytes --]

On 04/01/20 11:09, Rolf Eike Beer wrote:
> Am Freitag, 3. Januar 2020, 11:00:14 CET schrieb Rolf Eike Beer:
>> Am Freitag, 3. Januar 2020, 03:40:35 CET schrieb Aaron Bauman:
>>> On January 2, 2020 6:35:08 PM EST, Rolf Eike Beer <eike@sf-mail.de> wrote:
>>>> Am Freitag, 3. Januar 2020, 00:25:06 CET schrieb Mike Pagano:
>>>>> hppa is making us keep old kernels around [1].  Should the kernel team
>>>>> be
>>>>> doing more to get your attention then CC'ing hppa on all of the kernel
>>>>> STABLEREQ bugs [2]?
>>>> I only run vanilla-sources since there are still lot of cache
>>>> corruption
>>>> problems in hppa kernels, or whatever makes them flaky.
>>>>
>>>> Linux pioneer 5.4.6-parisc64 #1 SMP Fri Dec 27 10:23:09 CET 2019 parisc64
>>>> PA8800 (Mako) 9000/785/C8000 GNU/Linux
>>>> Linux voyager 5.4.6-parisc #1 Fri Dec 27 15:46:43 CET 2019 parisc PA8600
>>>> (PCX-W+) 9000/785/C3600 GNU/Linux
>>>>
>>>> So _I_ personally would say just drop old kernels, but that is in no
>>>> way
>>>> authorative.
>>> Ugh. gentoo-sources is just a patch (trivial) on top of vanilla-kernel
>>> sources of each stable and LTS version.
>> If it's just that I could test them, but this still be no LTS version that I
>> would look at.
> So, do you want me to stable a random gentoo-sources (usually the most recent 
> one) every few weeks when I just happen to upgrade my machine?
>
> Eike
I don't think that works very well with kernel/security-team stabilisation
policies, sadly.

Is there any possibility you would be able to do a stabilisation run, and
do a reboot cycle on one LTS branch (of choice, eg. most recent) and then
revert to your preferred kernel afterwards?

I appreciate this is a rather onerous process on older hardware, but just
trying to think of some form of semi-compromise that prevents potential
de-keywording, without also suggesting that something that genuinely has
issues is either (1) working or (2) supported, if so.

I believe Whissi is leading kernel stabilisation requests, on behalf of
security- and kernel- teams, so maybe a chat with him may be fruitful.
Cheers,
veremitz/Michael.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

Re: [gentoo-dev] Keywordreqs and slacking arch teams

[Rolf Eike Beer]

[-- Attachment #1: Type: text/plain, Size: 2034 bytes --]

Am Samstag, 4. Januar 2020, 12:25:07 CET schrieb Michael 'veremitz' Everitt:
> On 04/01/20 11:09, Rolf Eike Beer wrote:
> > Am Freitag, 3. Januar 2020, 11:00:14 CET schrieb Rolf Eike Beer:
> >> Am Freitag, 3. Januar 2020, 03:40:35 CET schrieb Aaron Bauman:
> >>> On January 2, 2020 6:35:08 PM EST, Rolf Eike Beer <eike@sf-mail.de> 
wrote:
> >>>> Am Freitag, 3. Januar 2020, 00:25:06 CET schrieb Mike Pagano:
> >>>>> hppa is making us keep old kernels around [1].  Should the kernel team
> >>>>> be
> >>>>> doing more to get your attention then CC'ing hppa on all of the kernel
> >>>>> STABLEREQ bugs [2]?
> >>>> 
> >>>> I only run vanilla-sources since there are still lot of cache
> >>>> corruption
> >>>> problems in hppa kernels, or whatever makes them flaky.
> >>>> 
> >>>> Linux pioneer 5.4.6-parisc64 #1 SMP Fri Dec 27 10:23:09 CET 2019
> >>>> parisc64
> >>>> PA8800 (Mako) 9000/785/C8000 GNU/Linux
> >>>> Linux voyager 5.4.6-parisc #1 Fri Dec 27 15:46:43 CET 2019 parisc
> >>>> PA8600
> >>>> (PCX-W+) 9000/785/C3600 GNU/Linux
> >>>> 
> >>>> So _I_ personally would say just drop old kernels, but that is in no
> >>>> way
> >>>> authorative.
> >>> 
> >>> Ugh. gentoo-sources is just a patch (trivial) on top of vanilla-kernel
> >>> sources of each stable and LTS version.
> >> 
> >> If it's just that I could test them, but this still be no LTS version
> >> that I would look at.
> > 
> > So, do you want me to stable a random gentoo-sources (usually the most
> > recent one) every few weeks when I just happen to upgrade my machine?

> I don't think that works very well with kernel/security-team stabilisation
> policies, sadly.
> 
> Is there any possibility you would be able to do a stabilisation run, and
> do a reboot cycle on one LTS branch (of choice, eg. most recent) and then
> revert to your preferred kernel afterwards?

This is annoying, because it usually collides with the nightly runs in some 
way. Doing the build on the C3600 took ~1d last time, the C8000 is faster, but 
I still have to time this right.

Eike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] Vanilla sources

[Michael Orlitzky]

On 1/2/20 6:35 PM, Rolf Eike Beer wrote:
> 
> I only run vanilla-sources since there are still lot of cache corruption 
> problems in hppa kernels, or whatever makes them flaky.

The vanilla-sources are unsafe to use on Gentoo. Many services have
stupid-easy root exploits, since we install tmpfiles entries by default
and OpenRC runs them insecurely:

  * https://github.com/OpenRC/opentmpfiles/issues/3
  * https://github.com/OpenRC/opentmpfiles/issues/4

I've fixed similar exploits when I've found them in /etc/init.d and
pkg_postinst[0][1], but they continue to be added to the tree. And there
is no fix for opentmpfiles.

The gentoo-sources aren't 100% safe either, but the exploitable scenario
is less common thanks to fs.protected_{hardlinks,symlinks}=1.


[0]
http://michael.orlitzky.com/articles/end_root_chowning_now_%28make_etc-init.d_great_again%29.xhtml

[1]
http://michael.orlitzky.com/articles/end_root_chowning_now_%28make_pkg_postinst_great_again%29.xhtml

Re: [gentoo-dev] Vanilla sources

[Toralf Förster]

[-- Attachment #1.1: Type: text/plain, Size: 295 bytes --]

On 1/3/20 3:37 PM, Michael Orlitzky wrote:
> The gentoo-sources aren't 100% safe either, but the exploitable scenario
> is less common thanks to fs.protected_{hardlinks,symlinks}=1.

But this can be easily achieved w/o installing gentoo-sources, or?

-- 
Toralf
PGP 23217DA7 9B888F45


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] Vanilla sources

[Michael Orlitzky]

On 1/3/20 9:40 AM, Toralf Förster wrote:
> On 1/3/20 3:37 PM, Michael Orlitzky wrote:
>> The gentoo-sources aren't 100% safe either, but the exploitable scenario
>> is less common thanks to fs.protected_{hardlinks,symlinks}=1.
> 
> But this can be easily achieved w/o installing gentoo-sources, or?
> 

Yes, if you know how to do it. And the hard part: if you know that you
*should* do it.

Re: [gentoo-dev] Vanilla sources

[Rich Freeman]

On Fri, Jan 3, 2020 at 9:41 AM Michael Orlitzky <mjo@gentoo.org> wrote:
>
> On 1/3/20 9:40 AM, Toralf Förster wrote:
> > On 1/3/20 3:37 PM, Michael Orlitzky wrote:
> >> The gentoo-sources aren't 100% safe either, but the exploitable scenario
> >> is less common thanks to fs.protected_{hardlinks,symlinks}=1.
> >
> > But this can be easily achieved w/o installing gentoo-sources, or?
> >
>
> Yes, if you know how to do it. And the hard part: if you know that you
> *should* do it.
>

If OpenRC contains a vulnerability wouldn't it make more sense to set
this as part of OpenRC, then to assume somebody is running a kernel
patch that does it, especially since OpenRC doesn't in any way ensure
that gentoo-sources is actually being used?

Of course, fixing the vulnerability seems like a better option.   At
least on Linux based on your one bug description it sounds like
systemd has a Linux-specific fix already.  Obviously it would be best
to secure this on all kernels but there is no reason not to at least
use that fix on Linux.  You could also try to convince the entire
world not to use tmpfiles.d but since it is only a problem if you
aren't using systemd I suspect you won't get much traction there.

In any case this seems more like an OpenRC issue than a Gentoo issue.

-- 
Rich

Re: [gentoo-dev] Vanilla sources

[Toralf Förster]

[-- Attachment #1.1: Type: text/plain, Size: 1366 bytes --]

On 1/3/20 3:46 PM, Rich Freeman wrote:
> If OpenRC contains a vulnerability wouldn't it make more sense to set
> this as part of OpenRC,
Indeed.

Furthermore there's a nifty page https://kernsec.org/wiki/index.php/Kernel_Self_Protection_Project/Recommended_Settings
which yields for me to this /etc/sysctl.d/local.conf :


#       Restrict potential illegal access via links
# 
fs.protected_hardlinks = 1
fs.protected_symlinks = 1 

#
# https://kernsec.org/wiki/index.php/Kernel_Self_Protection_Project#CONFIGs
#

# Try to keep kernel address exposures out of various /proc files (kallsyms, modules, etc).
kernel.kptr_restrict = 1

# Avoid kernel memory address exposures via dmesg.
kernel.dmesg_restrict = 1

# Block non-uid-0 profiling (needs distro patch, otherwise this is the same as "= 2")
kernel.perf_event_paranoid = 3

# Turn off kexec, even if it's built in.
kernel.kexec_load_disabled = 1

# Avoid non-ancestor ptrace access to running processes and their credentials.
kernel.yama.ptrace_scope = 1

# Disable User Namespaces, as it opens up a large attack surface to unprivileged users.
user.max_user_namespaces = 0

# Turn off unprivileged eBPF access.
kernel.unprivileged_bpf_disabled = 1

# Turn on BPF JIT hardening, if the JIT is enabled.
net.core.bpf_jit_harden = 2


-- 
Toralf
PGP 23217DA7 9B888F45


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] Vanilla sources

[Michael 'veremitz' Everitt]

[-- Attachment #1.1: Type: text/plain, Size: 1645 bytes --]

On 03/01/20 14:48, Toralf Förster wrote:
> On 1/3/20 3:46 PM, Rich Freeman wrote:
>> If OpenRC contains a vulnerability wouldn't it make more sense to set
>> this as part of OpenRC,
> Indeed.
>
> Furthermore there's a nifty page https://kernsec.org/wiki/index.php/Kernel_Self_Protection_Project/Recommended_Settings
> which yields for me to this /etc/sysctl.d/local.conf :
>
>
> #       Restrict potential illegal access via links
> # 
> fs.protected_hardlinks = 1
> fs.protected_symlinks = 1 
>
> #
> # https://kernsec.org/wiki/index.php/Kernel_Self_Protection_Project#CONFIGs
> #
>
> # Try to keep kernel address exposures out of various /proc files (kallsyms, modules, etc).
> kernel.kptr_restrict = 1
>
> # Avoid kernel memory address exposures via dmesg.
> kernel.dmesg_restrict = 1
>
> # Block non-uid-0 profiling (needs distro patch, otherwise this is the same as "= 2")
> kernel.perf_event_paranoid = 3
>
> # Turn off kexec, even if it's built in.
> kernel.kexec_load_disabled = 1
>
> # Avoid non-ancestor ptrace access to running processes and their credentials.
> kernel.yama.ptrace_scope = 1
>
> # Disable User Namespaces, as it opens up a large attack surface to unprivileged users.
> user.max_user_namespaces = 0
>
> # Turn off unprivileged eBPF access.
> kernel.unprivileged_bpf_disabled = 1
>
> # Turn on BPF JIT hardening, if the JIT is enabled.
> net.core.bpf_jit_harden = 2
>
>
FWIW, there is a move to add further hardening options to the
Gentoo-sources kernel - bug 689154, based on the kernsec recommendations.
Further details of proposals, and inspiration, are in the bug.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

Re: [gentoo-dev] Vanilla sources

[Hanno Böck]

[-- Attachment #1: Type: text/plain, Size: 541 bytes --]

On Fri, 3 Jan 2020 15:48:54 +0100
Toralf Förster <toralf@gentoo.org> wrote:

> #       Restrict potential illegal access via links
> # 
> fs.protected_hardlinks = 1
> fs.protected_symlinks = 1 

Given the issues with openrc:
Wouldn't it be a good idea to add these by default to Gentoo's
sysctl.conf in baselayout?

As far as I understand this from the thread by now, these are set by
default by Gentoo Sources. So we shouldn't really expect much breakage
if we set them via sysctl.


-- 
Hanno Böck
https://hboeck.de/

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Vanilla sources

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 501 bytes --]

On Sat, Jan 04, 2020 at 08:38:59AM +0100, Hanno Böck wrote:
> On Fri, 3 Jan 2020 15:48:54 +0100
> Toralf Förster <toralf@gentoo.org> wrote:
> 
> > #       Restrict potential illegal access via links
> > # 
> > fs.protected_hardlinks = 1
> > fs.protected_symlinks = 1 
> 
> Given the issues with openrc:
> Wouldn't it be a good idea to add these by default to Gentoo's
> sysctl.conf in baselayout?

If we want to do this, it is easy for me to do it in baselayout.

Thanks,

William


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] Vanilla sources

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 837 bytes --]

On Sat, 2020-01-04 at 08:38 +0100, Hanno Böck wrote:
> On Fri, 3 Jan 2020 15:48:54 +0100
> Toralf Förster <toralf@gentoo.org> wrote:
> 
> > #       Restrict potential illegal access via links
> > # 
> > fs.protected_hardlinks = 1
> > fs.protected_symlinks = 1 
> 
> Given the issues with openrc:
> Wouldn't it be a good idea to add these by default to Gentoo's
> sysctl.conf in baselayout?

Yes, we should.  This really sounds like some horror where developers
are hacking things around in sources instead of communicating with
people maintaining the component where a proper fix belongs.

> 
> As far as I understand this from the thread by now, these are set by
> default by Gentoo Sources. So we shouldn't really expect much breakage
> if we set them via sysctl.
> 
> 

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] Vanilla sources

[Hanno Böck]

[-- Attachment #1: Type: text/plain, Size: 1003 bytes --]

On Sat, 04 Jan 2020 19:41:21 +0100
Michał Górny <mgorny@gentoo.org> wrote:

> On Sat, 2020-01-04 at 08:38 +0100, Hanno Böck wrote:
> > On Fri, 3 Jan 2020 15:48:54 +0100
> > Toralf Förster <toralf@gentoo.org> wrote:
> >   
> > > #       Restrict potential illegal access via links
> > > # 
> > > fs.protected_hardlinks = 1
> > > fs.protected_symlinks = 1   
> > 
> > Given the issues with openrc:
> > Wouldn't it be a good idea to add these by default to Gentoo's
> > sysctl.conf in baselayout?  
> 
> Yes, we should.  This really sounds like some horror where developers
> are hacking things around in sources instead of communicating with
> people maintaining the component where a proper fix belongs.

I created a bug for this so we can move the discussion there:
https://bugs.gentoo.org/704914

Particularly if anyone thinks this is a bad idea or knows of a
situation where this breaks things please speak up now in the bugreport.

-- 
Hanno Böck
https://hboeck.de/

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Vanilla sources

[Michael Orlitzky]

On 1/3/20 9:46 AM, Rich Freeman wrote:
> 
> ...
> 
> In any case this seems more like an OpenRC issue than a Gentoo issue.
> 

It's a specification issue. There's no way to implement tmpfiles safely
on a POSIX system, and opentmpfiles shouldn't exist if OpenRC wants to
work on anything other than Linux.

But here we are. Do we make OpenRC Linux-only and steal the fix from
systemd? Or pretend to support other operating systems, but leave them
insecure?

Re: [gentoo-dev] Vanilla sources

[Michael Orlitzky]

On 1/3/20 9:52 AM, Michael Orlitzky wrote:
> 
> But here we are. Do we make OpenRC Linux-only and steal the fix from
> systemd? Or pretend to support other operating systems, but leave them
> insecure?
> 

Or the gripping hand: rewrite opentmpfiles in C, so that it's only as
insecure as checkpath.

Every option sucks. I was only trying to point out that vanilla-sources
gets no security support -- security@ has stated this, but it's on a
private bug, so I won't quote it -- and the risk is more than academic.

Re: [gentoo-dev] Vanilla sources

[Aaron Bauman]

On January 3, 2020 9:55:31 AM EST, Michael Orlitzky <mjo@gentoo.org> wrote:
>On 1/3/20 9:52 AM, Michael Orlitzky wrote:
>> 
>> But here we are. Do we make OpenRC Linux-only and steal the fix from
>> systemd? Or pretend to support other operating systems, but leave
>them
>> insecure?
>> 
>
>Or the gripping hand: rewrite opentmpfiles in C, so that it's only as
>insecure as checkpath.
>
>Every option sucks. I was only trying to point out that vanilla-sources
>gets no security support -- security@ has stated this, but it's on a
>private bug, so I won't quote it -- and the risk is more than academic.

This should be known. Security does not support vanilla-sources. This is one reason vanilla-sources are not stabilized. 

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [gentoo-dev] Vanilla sources

[Rich Freeman]

On Fri, Jan 3, 2020 at 11:28 AM Aaron Bauman <bman@gentoo.org> wrote:
> On January 3, 2020 9:55:31 AM EST, Michael Orlitzky <mjo@gentoo.org> wrote:
> >On 1/3/20 9:52 AM, Michael Orlitzky wrote:
> >>
> >> But here we are. Do we make OpenRC Linux-only and steal the fix from
> >> systemd? Or pretend to support other operating systems, but leave
> >them
> >> insecure?
> >>
> >
> >Or the gripping hand: rewrite opentmpfiles in C, so that it's only as
> >insecure as checkpath.
> >
> >Every option sucks. I was only trying to point out that vanilla-sources
> >gets no security support -- security@ has stated this, but it's on a
> >private bug, so I won't quote it -- and the risk is more than academic.
>
> This should be known. Security does not support vanilla-sources. This is one reason vanilla-sources are not stabilized.
>

Packages without security support should be masked.  Really I don't
see the point of even having this in the repo.

I run vanilla sources personally but I just get them from upstream.
Makes way more sense than worrying about whether the version in the
repo is up to date for the longterm kernel I'm following.  People
running vanilla sources are probably using out-of-tree modules (like
me) and as such are going to have particular requirements around how
they're updated.  So, Gentoo is adding fairly little value.

All they do is download sources anyway, which is trivially done from
git more efficiently (or tarballs that are probably easy to obtain
just as efficiently).  I can see more of the point in the new
distribution kernel project which will be turnkey.  I can see some of
the value in gentoo-sources (particularly as the upstream for the
distribution kernels) especially if they're tied to Gentoo-specific
bugs.  For more general bugs that apply to all distros I really don't
see the point in trying to compete with the upstream stable branches
(if they're taking forever to merge a patch, chances are there is a
reason for it, and I'm skeptical that Gentoo users are special in some
way).

Is there some reason that we should keep vanilla sources despite not
getting security handling?

-- 
Rich

Re: [gentoo-dev] Vanilla sources

[Roy Bamford]

[-- Attachment #1: Type: text/plain, Size: 791 bytes --]

On 2020.01.04 11:01, Rich Freeman wrote:
>
> Is there some reason that we should keep vanilla sources despite not
> getting security handling?
> 
> -- 
> Rich
> 

Rich,

Gentoo had this discussion before. The outcome was that 
vanilla-sources is just as Linus intended.
If Gentoo did anything to it, it wouldn't be vanilla any longer.

Yes, it should be kept. We should not force users to learn
git or tar.

I agree git or a tarball of vanilla-sources is faster and more
efficient but that's not a reason to drop it.
By the same argument we could drop linux-firmware too.
There are probably other packages that only install whatever
they fetch. Could they be dropped?

-- 
Regards,

Roy Bamford
(Neddyseagoon) a member of
elections
gentoo-ops
forum-mods
arm64

[-- Attachment #2: Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] Vanilla sources

[Rich Freeman]

On Sat, Jan 4, 2020 at 6:42 AM Roy Bamford <neddyseagoon@gentoo.org> wrote:
>
> On 2020.01.04 11:01, Rich Freeman wrote:
> >
> > Is there some reason that we should keep vanilla sources despite not
> > getting security handling?
> >
>
> Gentoo had this discussion before. The outcome was that
> vanilla-sources is just as Linus intended.
> If Gentoo did anything to it, it wouldn't be vanilla any longer.

Obviously.  I wasn't suggesting that we keep vanilla sources but not
make them vanilla.  That doesn't mean that they couldn't be
security-supported, or that we have to have them in the repo.

> Yes, it should be kept. We should not force users to learn
> git or tar.

Uh, all it does is install kernel sources.  They're useless unless you
build a kernel using them.

Apparently git and tar are too complicated for Gentoo users, but
managing symlinks, using make, managing a bootloader, dealing with the
kernel's configuration system, and so on are just fine?

I completely get the point of the distribution kernel project that was
just announced, as I already said.

> I agree git or a tarball of vanilla-sources is faster and more
> efficient but that's not a reason to drop it.
> By the same argument we could drop linux-firmware too.
> There are probably other packages that only install whatever
> they fetch. Could they be dropped?

So, a few issues with that argument:

1.  Those other packages are security supported.
2.  Those other packages are largely functional once installed, and to
the degree that they require configuration that is generally one-time
and after updates they remain functional.

All that said, it seems like vanilla-sources is pretty up-to-date, so
I'm not sure what we mean by it not being security supported.  I just
took that as a given.  Does that mean that we're not releasing patches
before upstream?  If so, that seems like a pretty minor issue since
upstream generally does security bumps pretty quickly.  4.4.208 isn't
in our repo but was released today - I'm not sure how quickly these
get bumped.  If our repo could be days behind that is definitely
another reason not to host this stuff, as users should be directed
upstream if our packages aren't security supported.

On a further aside, I just noticed how up-to-date gentoo-sources are.
Kudos to whoever is doing that these days - for a while it was tending
to slip a bit but it seems like we're basically current.

-- 
Rich

Re: [gentoo-dev] Vanilla sources

[Roy Bamford]

[-- Attachment #1: Type: text/plain, Size: 977 bytes --]

On 2020.01.04 12:54, Rich Freeman wrote:
> On Sat, Jan 4, 2020 at 6:42 AM Roy Bamford <neddyseagoon@gentoo.org>
> wrote:

[snip]
> 
> Apparently git and tar are too complicated for Gentoo users, but
> managing symlinks, using make, managing a bootloader, dealing with the
> kernel's configuration system, and so on are just fine?

emerge -1 vanilla-sources
eselect kernel ...
genkernel all
...

Yep, lots of users have trouble managing their boot loader. They come to
the forums and IRC every day not running the kernel they think they are.
That's not related to any particular kernel sources though.

[snip]

> 
> On a further aside, I just noticed how up-to-date gentoo-sources are.
> Kudos to whoever is doing that these days - for a while it was tending
> to slip a bit but it seems like we're basically current.

+1

> 
> -- 
> Rich
> 
> 

-- 
Regards,

Roy Bamford
(Neddyseagoon) a member of
elections
gentoo-ops
forum-mods
arm64

[-- Attachment #2: Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] Vanilla sources

[Thomas Deutschmann]

[-- Attachment #1.1: Type: text/plain, Size: 585 bytes --]

On 2020-01-04 14:08, Roy Bamford wrote:
> emerge -1 vanilla-sources
> eselect kernel ...
> genkernel all
> ...

Please tell user to do

genkernel --kernel-config=/proc/config.gz all

by default which will give them a better experience because new kernel
will be build based on kernel configuration from current running kernel.
Without providing a kernel config, user will probably fall back to
generic configuration which isn't intended for daily usage.


-- 
Regards,
Thomas Deutschmann / Gentoo Linux Developer
C4DD 695F A713 8F24 2AA1 5638 5849 7EE5 1D5D 74A5


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] Vanilla sources

[Roy Bamford]

[-- Attachment #1: Type: text/plain, Size: 1135 bytes --]

On 2020.01.04 13:43, Thomas Deutschmann wrote:
> On 2020-01-04 14:08, Roy Bamford wrote:
> > emerge -1 vanilla-sources
> > eselect kernel ...
> > genkernel all
> > ...
> 
> Please tell user to do
> 
> genkernel --kernel-config=/proc/config.gz all
> 
> by default which will give them a better experience because new kernel
> will be build based on kernel configuration from current running
> kernel.
> Without providing a kernel config, user will probably fall back to
> generic configuration which isn't intended for daily usage.
> 
> 
> -- 
> Regards,
> Thomas Deutschmann / Gentoo Linux Developer
> C4DD 695F A713 8F24 2AA1 5638 5849 7EE5 1D5D 74A5
> 
> 

Thomas,

Ten years ago the user did 
genkernel all 
and used the default .config provided with genkernel.

Today, genkernel --kernel-config=/proc/config.gz all does not improve 
that config.

genkernel --kernel-config=/proc/config.gz all 
perpetuates the existing .config which is only useful if its not an 
earlier generic configuration.

-- 
Regards,

Roy Bamford
(Neddyseagoon) a member of
elections
gentoo-ops
forum-mods
arm64

[-- Attachment #2: Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] Vanilla sources

[Christopher Head]

[-- Attachment #1: Type: text/plain, Size: 1408 bytes --]

On January 4, 2020 4:54:07 AM PST, Rich Freeman <rich0@gentoo.org> wrote:
>
>Uh, all it does is install kernel sources.  They're useless unless you
>build a kernel using them.
>
>Apparently git and tar are too complicated for Gentoo users, but
>managing symlinks, using make, managing a bootloader, dealing with the
>kernel's configuration system, and so on are just fine?

I use gentoo-sources myself, but still, I would like to propose one reason for keeping vanilla-sources. For me, git/tar are not too complicated, but having V-S in the Gentoo tree would provide another benefit: reducing the number of things I have to check every weekly update cycle. Every piece of software I get from a source other than the Gentoo tree is another website I have to visit every update day to check whether there’s a newer version available. So from that perspective, the advantage of having packages in tree that just install some files is that emerge tells me when a new version is available, rather than me having to go every week to upstream’s website and check manually (or sign up for countless announcement mailing lists).

Of course this would be a bad argument if V-S were lagging behind upstream significantly, and it’s a much better argument for packages that come with expectations of security team support than those that don’t, but it is something to consider.

-- 
Christopher Head

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 244 bytes --]

Re: [gentoo-dev] Vanilla sources

[Rich Freeman]

On Sat, Jan 4, 2020 at 3:13 PM Christopher Head <chead@chead.ca> wrote:
>
>
> Of course this would be a bad argument if V-S were lagging behind upstream significantly, and it’s a much better argument for packages that come with expectations of security team support than those that don’t, but it is something to consider.
>

This was my main concern when it was mentioned that it wasn't
security-supported.

If it is always up-to-date that definitely helps mitigate things.
Though, there should definitely be some kind of warning on the package
that it isn't security supported.  Even if it is up to date it won't
get GLSAs and GLSA-checker won't work.  Though, that really only makes
a difference insofar as the GLSAs are also timely.

In any case, if the just-announced distribution kernel project takes
off and remains active I could easily see that becoming the most
commonly used kernel option.  I'm not knocking minimal kernels but I
suspect a LOT of users are going to be well-served by a modular kernel
that just works 99% of the time.

-- 
Rich

Re: [gentoo-dev] Vanilla sources

[Thomas Deutschmann]

[-- Attachment #1.1: Type: text/plain, Size: 403 bytes --]

On 2020-01-04 12:01, Rich Freeman wrote:
> Packages without security support should be masked.  Really I don't
> see the point of even having this in the repo.

THIS! +infinite

And arches without security support in general can't have stable keywords.

But this is a dream. :-/


-- 
Regards,
Thomas Deutschmann / Gentoo Security Team
C4DD 695F A713 8F24 2AA1 5638 5849 7EE5 1D5D 74A5


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] Vanilla sources

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 514 bytes --]

On Fri, Jan 03, 2020 at 09:55:31AM -0500, Michael Orlitzky wrote:
> On 1/3/20 9:52 AM, Michael Orlitzky wrote:
> > 
> > But here we are. Do we make OpenRC Linux-only and steal the fix from
> > systemd? Or pretend to support other operating systems, but leave them
> > insecure?
> > 
> 
> Or the gripping hand: rewrite opentmpfiles in C, so that it's only as
> insecure as checkpath.

There is a pr open for opentmpfiles for this, and we are also discussing
writing it in rust.

Thanks,

William


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] Vanilla sources

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 691 bytes --]

On Sat, 2020-01-04 at 12:41 -0600, William Hubbs wrote:
> On Fri, Jan 03, 2020 at 09:55:31AM -0500, Michael Orlitzky wrote:
> > On 1/3/20 9:52 AM, Michael Orlitzky wrote:
> > > But here we are. Do we make OpenRC Linux-only and steal the fix from
> > > systemd? Or pretend to support other operating systems, but leave them
> > > insecure?
> > > 
> > 
> > Or the gripping hand: rewrite opentmpfiles in C, so that it's only as
> > insecure as checkpath.
> 
> There is a pr open for opentmpfiles for this, and we are also discussing
> writing it in rust.
> 

You could also force people to install systemd.  That would be more
merciful.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] Vanilla sources

[Rolf Eike Beer]

[-- Attachment #1: Type: text/plain, Size: 611 bytes --]

Am Samstag, 4. Januar 2020, 19:41:05 CET schrieb William Hubbs:
> On Fri, Jan 03, 2020 at 09:55:31AM -0500, Michael Orlitzky wrote:
> > On 1/3/20 9:52 AM, Michael Orlitzky wrote:
> > > But here we are. Do we make OpenRC Linux-only and steal the fix from
> > > systemd? Or pretend to support other operating systems, but leave them
> > > insecure?
> > 
> > Or the gripping hand: rewrite opentmpfiles in C, so that it's only as
> > insecure as checkpath.
> 
> There is a pr open for opentmpfiles for this, and we are also discussing
> writing it in rust.

Bad idea. If you wonder why: eshowkw dev-lang/rust.

Eike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] Vanilla sources

[Michael Orlitzky]

On 1/4/20 2:13 PM, Rolf Eike Beer wrote:
> 
> Bad idea. If you wonder why: eshowkw dev-lang/rust.
> 

Or consider that every rust package in Gentoo bundles hundreds of
libraries. We'd be fixing one security issue by introducing 10x more.

Not that rewriting it in rust would fix anything; writing it in C is
only a solution insofar as it makes the window to exploit the race
condition is as small as possible.