[gentoo-project] [RFC] Ebuild quizzes for existing developers

[gentoo-project] [RFC] Ebuild quizzes for existing developers

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 3650 bytes --]

Hello, everyone.

I'd like to discuss a proposal aiming to improve the general quality of
ebuilds in the Gentoo repository.


As most of you know (or at least suspect), ebuild quizzes are changing
over time.  New questions are added and existing questions are updated,
to cover for new EAPIs, policies or simply cover cases of common
mistakes.  This is all done so that new developers would be better
prepared to work on Gentoo, and would not repeat the same mistakes
as new developers before them did.  Besides, this extends the purpose of
quizzes into being a quick reference of all the stuff you should be
aware of before committing.

However, new quizzes apply only to new recruits.  Ideally, we assume
that existing developers learn of new EAPIs and new policies as they go,
and they've already made most of the mistakes and learned from them. 
However, this is not always true.  There are less-active developers,
there are people returning after long devaways, and sadly there are
cases when they do not try to learn new things but attempt to apply
obsolete standards and solutions.  Furthermore, some of the recruits
consider it unfair that they have to undergo harder quizzes than people
who joined Gentoo earlier.


What I'd like to propose are additional examinations similar to
the ebuild quiz, for existing developers.  They would be performed
as a joint QA/Recruiters effort, and they would aim to align
the level of examinations for all developers.  This would serve two
purposes.  Firstly, it would ensure that existing developers are fully
aware of the current EAPIs, policies, best practices and pitfalls. 
Secondly, it would align the rules and make sure that developers joining
earlier are not privileged compared to those joining later.

Of course, I'm not talking about requiring all developers to pass their
ebuild quizzes again.  The way I see it, we would be establishing a few
reference versions of quizzes, and comparing them to the current one. 
Then, for each version we would establish a smaller quiz specifically
focused on developers recruited at the time.  Naturally, the developers
recruited earlier would have to answer more questions while developers
recruited more recently would have less.

To avoid overburdening our teams and developers, the whole process would
be spread over time.  Every time the quizzes are prepared for the next
group of developers, they will be given 6 months to complete them
and arrange a review session.  As the examinations progress, quizzes for
next groups will be prepared.

If the developer in question fails to pass the examination in 6 months,
the developer's commit access will be revoked.  After 2 more months
without passing, the developer will be retired.


It is quite likely that before we will be able to finish
the examinations for all developers, the quizzes will change again. 
Naturally, the groups examined past that will have their quizzes aligned
to the newest version already.  The earlier groups will be subject to
further (even smaller) examinations once all groups are done for
the first time.  Eventually, we would be able to align all developers to
the same version of quizzes.

Once that is achieved, I'd like to keep quizzes in sync from now on. 
That is, whenever we add a new question we will pass it through
developers.  This will achieve a double goal.  Firstly, it will improve
ebuild skills of developers.  Secondly, it will provide additional
testing of new quiz questions before recruits end up having to answer
them.


What do you think?

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: [gentoo-project] [RFC] Ebuild quizzes for existing developers

[Rich Freeman]

On Mon, Apr 1, 2019 at 10:43 AM Michał Górny <mgorny@gentoo.org> wrote:
>
> I'd like to discuss a proposal aiming to improve the general quality of
> ebuilds in the Gentoo repository.
>

Is something actually broken?

IMO discretion and ability to comply with policies/etc are probably
far more important than rattling off EAPI/eclass/GLEP trivia.  They're
just hard to measure for new recruits so we use the quizzes as a proxy
for estimating whether somebody will be able to fit in.  If somebody
could pass the quizzes in the past, and have a fairly clean record
inside Gentoo, then I don't really see the point in having them
continue to jump through hoops.

If they're making mistakes in their commits then just focus on the
mistakes they're making and boot devs who don't raise their general
level of quality.  You don't need to guess whether an existing dev
will mess up the repo - they already have commit access.  If they
haven't been breaking things left and right so far, it is unlikely
that they will start to do so in the future.

Now, docs for developers about EAPI changes have been written in the
past and are undoubtedly useful.  I just don't see the need to give
them a certification quiz of some sort.

-- 
Rich

Re: [gentoo-project] [RFC] Ebuild quizzes for existing developers

[Dirkjan Ochtman]

[-- Attachment #1: Type: text/plain, Size: 1046 bytes --]

On Mon, Apr 1, 2019 at 6:43 PM Michał Górny <mgorny@gentoo.org> wrote:

> What do you think?
>

I'm not sure this will be the best way to improve the quality. In my view
(although it's a long time ago I did the eclasses) doing ebuilds and
eclasses right is a pretty deep *and* pretty wide subject, such that it
seems likely that even with regular requizzing and quizz updates, there
would still be a lot of material that is not adequately covered by the
ebuild quizzes. Given the largish burden on developers, couldn't they more
productively spend their limited time on actual ebuild improvements?

As an alternative route, I would suggest investing more heavily in
automated (static or dynamic) analysis of ebuilds to find patterns that are
incorrect. This has the benefit that every contribution can be checked
(instead of the skills of some of our contributors, every once in a while),
and even if it's not free of false positives can act as a way to learn
about patterns that might be problematic.

Regards,

Dirkjan

[-- Attachment #2: Type: text/html, Size: 1400 bytes --]

Re: [gentoo-project] [RFC] Ebuild quizzes for existing developers

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 1446 bytes --]

On Tue, 2 Apr 2019 08:08:30 -0600 Rich Freeman wrote:
> On Mon, Apr 1, 2019 at 10:43 AM Michał Górny <mgorny@gentoo.org> wrote:
> >
> > I'd like to discuss a proposal aiming to improve the general quality of
> > ebuilds in the Gentoo repository.
> >
> 
> Is something actually broken?
> 
> IMO discretion and ability to comply with policies/etc are probably
> far more important than rattling off EAPI/eclass/GLEP trivia.  They're
> just hard to measure for new recruits so we use the quizzes as a proxy
> for estimating whether somebody will be able to fit in.  If somebody
> could pass the quizzes in the past, and have a fairly clean record
> inside Gentoo, then I don't really see the point in having them
> continue to jump through hoops.
> 
> If they're making mistakes in their commits then just focus on the
> mistakes they're making and boot devs who don't raise their general
> level of quality.  You don't need to guess whether an existing dev
> will mess up the repo - they already have commit access.  If they
> haven't been breaking things left and right so far, it is unlikely
> that they will start to do so in the future.
> 
> Now, docs for developers about EAPI changes have been written in the
> past and are undoubtedly useful.  I just don't see the need to give
> them a certification quiz of some sort.

Rich, apparently this was a hapless 1st April's joke.

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-project] [RFC] Ebuild quizzes for existing developers

[Michael Orlitzky]

On 4/1/19 12:43 PM, Michał Górny wrote:
> 
> What do you think?
> 

Reminds me of a something I once spotted in an essay[0] by Doron
Zeilberger, a famous mathematician with some non-mainstream ideas:

  Analogously, my own best ideas, far surpassing anything in my
  "serious" papers, are contained in my annual April Fool's jokes, sent
  to my E-correspondents and posted on my website. This way I can
  express my "off the wall" ideas without being considered a crackpot.


[0] http://sites.math.rutgers.edu/~zeilberg/mamarim/mamarimPDF/real.pdf