[gentoo-project] On eclass APIs

[gentoo-project] On eclass APIs

[Fabian Groffen]

[-- Attachment #1: Type: text/plain, Size: 2926 bytes --]

In the Council meeting at 20111108, the Council decided that a backwards
incompatible API change of an eclass should be announced 30-days in
advance:

   "When removing a function or changing the API of an eclass, make
   sure that it doesn't break any ebuilds in the tree, and post a
   notice to gentoo-dev at least 30 days in advance, preferably with
   a patch included."
   http://www.gentoo.org/proj/en/council/meeting-logs/20111108-summary.txt

In addition to this, a broader discussion on policies for API changes on
eclasses was called for by the Council.  This email tries to open up
that discussion.


The problem of eclass API changes is similar to API/ABI changes found
elsewhere.  Since eclasses are intended to have many consumers, changes
to them can potentially affect many ebuilds or other eclasses, also
outside of Gentoo's main focus (e.g. in overlays).

Currently, the following API changing eclass practices seem to be in use:
- deprecate a function, remove all of its usages, wait, remove the
  function (or entire eclass)
- update a function thereby changing its signature and contract,
  followed by fixing all usages of said function (now has to be
  announced to -dev by Council decision)
- create a new revision of the eclass to hold changed interfaces,
  deprecating older revision(s) of the eclass after a long time
[are there more types of changes here?]

One of the ideas is to use revisions in the way libtool applies its
rules for shared library versioning (translated to the ELF naming
scheme, using three fields major.minor.revision):
- no changes, no version changes
- changes to the code (bug fixes), but no API changes, increment
  revision
- only backward compatible API changes (new funcs), increment minor,
  set revision to 0
- if there are (also) incompatible API changes (removed funcs,
  changed arguments), increment major, minor and revision to 0

Each major increment means an incompatible ABI for the given shared
library, the minor and revision denote additions and fixes.  We can
hence only focus on the major number.

Changes that are made to eclasses, can hence be reflected as revbumps:
- fix functions, or make them smarter without breaking any API (no
  revbump)
- add new functions (no revbump)
- drop functions, or change their interface (revbump)

Doing so will, however, lead to a lot of duplication of functions,
since each incompatible change would have to lead to a new revision,
that holds all the other functions as well -- unless some smart
overloading trick can be applied.

Is this necessary for just one function, or is this only beneficial
after a huge revamp of some eclass?  Is it acceptable to take the
breakage outside Gentoo, given the extra work imposed by creating
revisions and keeping track?

Any opinions, ideas or alternatives?


-- 
Fabian Groffen
Gentoo on a different level

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-project] On eclass APIs

[Brian Harring]

On Tue, Dec 27, 2011 at 08:35:36PM +0100, Fabian Groffen wrote:
> In the Council meeting at 20111108, the Council decided that a backwards
> incompatible API change of an eclass should be announced 30-days in
> advance:
> 
>    "When removing a function or changing the API of an eclass, make
>    sure that it doesn't break any ebuilds in the tree, and post a
>    notice to gentoo-dev at least 30 days in advance, preferably with
>    a patch included."
>    http://www.gentoo.org/proj/en/council/meeting-logs/20111108-summary.txt
> 
> In addition to this, a broader discussion on policies for API changes on
> eclasses was called for by the Council.  This email tries to open up
> that discussion.
> 
> 
> The problem of eclass API changes is similar to API/ABI changes found
> elsewhere.  Since eclasses are intended to have many consumers, changes
> to them can potentially affect many ebuilds or other eclasses, also
> outside of Gentoo's main focus (e.g. in overlays).
> 
> Currently, the following API changing eclass practices seem to be in use:
> - deprecate a function, remove all of its usages, wait, remove the
>   function (or entire eclass)
> - update a function thereby changing its signature and contract,
>   followed by fixing all usages of said function (now has to be
>   announced to -dev by Council decision)

This breaks overlays unless the API modification is backwards 
compatible; by it's nature, this shouldn't be done since it's a hard 
cut over (overlays have to be updated at the same time, same for 
ebuilds).

Even if you try to fix all gentoo repo ebuilds at the same time, it 
still is possible due to cvs lacking repo-wide atomic commits for part 
of the work to go out.

So... don't do it, at least not with cvs nor if you care about overlay 
consumers.

> One of the ideas is to use revisions in the way libtool applies its
> rules for shared library versioning (translated to the ELF naming
> scheme, using three fields major.minor.revision):
> - no changes, no version changes
> - changes to the code (bug fixes), but no API changes, increment
>   revision
> - only backward compatible API changes (new funcs), increment minor,
>   set revision to 0
> - if there are (also) incompatible API changes (removed funcs,
>   changed arguments), increment major, minor and revision to 0
> 
> Each major increment means an incompatible ABI for the given shared
> library, the minor and revision denote additions and fixes.  We can
> hence only focus on the major number.
> 
> Changes that are made to eclasses, can hence be reflected as revbumps:
> - fix functions, or make them smarter without breaking any API (no
>   revbump)
> - add new functions (no revbump)
> - drop functions, or change their interface (revbump)
> 
> Doing so will, however, lead to a lot of duplication of functions,
> since each incompatible change would have to lead to a new revision,
> that holds all the other functions as well -- unless some smart
> overloading trick can be applied.
> 
> Is this necessary for just one function, or is this only beneficial
> after a huge revamp of some eclass?  Is it acceptable to take the
> breakage outside Gentoo, given the extra work imposed by creating
> revisions and keeping track?
> 
> Any opinions, ideas or alternatives?

Personal views: generally speaking you can mangle the API as necessary 
w/ the caveat you only change the API in an incompatible way for 
internal functionality (demarked however you'd like- _func_name for 
example).

If you're rewriting the eclass into something sane (python eclass for 
example), than just purely from a stability standpoint... do it in a 
new eclass, and migrate things over.

That said, I don't agree with arbitrary revbumping; one thing to keep 
in mind is that there is *not* an ABI component to eclasses- we have 
env saving.  There purely is API, and overlays are the only thing we 
can't update at the same time, so having to revbump eclasses isn't 
worth it- nor is it in anyway friendly for people trying to do a cvs 
log on the eclass to figure out wtf has occured to it.

~brian

[gentoo-project] Re: On eclass APIs

[Ryan Hill]

[-- Attachment #1: Type: text/plain, Size: 1337 bytes --]

On Tue, 27 Dec 2011 20:35:36 +0100
Fabian Groffen <grobian@gentoo.org> wrote:

> Is this necessary for just one function, or is this only beneficial
> after a huge revamp of some eclass?  Is it acceptable to take the
> breakage outside Gentoo, given the extra work imposed by creating
> revisions and keeping track?

If there has been some major instance of eclass API breakage wrecking the tree
lately then I missed it (lately being defined as "since 2007 when
debug.eclass was purposely poisoned by QA"). So this versioning scheme is IMO
largely over-engineering a solution to a non-existent problem.

I'd like to say "use common sense" when announcing modifications to public
APIs, but we've proven time and again incapable of doing anything without a
painstakingly written procedure that we can all ignore or find loopholes in.
So 30 days sounds like a good minimum for an email to g-d-announce after all
in-tree uses are converted, unless your eclass has an "s", "y", or "e" in the
name in which case you roll a die and multiply by 30 (ie. current standard
practice).


-- 
fonts, gcc-porting,                  it makes no sense how it makes no sense
toolchain, wxwidgets                           but i'll take it free anytime
@ gentoo.org                EFFD 380E 047A 4B51 D2BD C64F 8AA8 8346 F9A4 0662

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-project] On eclass APIs

[Ralph Sennhauser]

On Tue, 27 Dec 2011 20:35:36 +0100
Fabian Groffen <grobian@gentoo.org> wrote:

> In the Council meeting at 20111108, the Council decided that a
> backwards incompatible API change of an eclass should be announced
> 30-days in advance:
> 
>    "When removing a function or changing the API of an eclass, make
>    sure that it doesn't break any ebuilds in the tree, and post a
>    notice to gentoo-dev at least 30 days in advance, preferably with
>    a patch included."
>    http://www.gentoo.org/proj/en/council/meeting-logs/20111108-summary.txt
> 
> In addition to this, a broader discussion on policies for API changes
> on eclasses was called for by the Council.  This email tries to open
> up that discussion.
> 
> 
> The problem of eclass API changes is similar to API/ABI changes found
> elsewhere.  Since eclasses are intended to have many consumers,
> changes to them can potentially affect many ebuilds or other
> eclasses, also outside of Gentoo's main focus (e.g. in overlays).
> 
> Currently, the following API changing eclass practices seem to be in
> use:
> - deprecate a function, remove all of its usages, wait, remove the
>   function (or entire eclass)
> - update a function thereby changing its signature and contract,
>   followed by fixing all usages of said function (now has to be
>   announced to -dev by Council decision)
> - create a new revision of the eclass to hold changed interfaces,
>   deprecating older revision(s) of the eclass after a long time
> [are there more types of changes here?]
> 
> One of the ideas is to use revisions in the way libtool applies its
> rules for shared library versioning (translated to the ELF naming
> scheme, using three fields major.minor.revision):
> - no changes, no version changes
> - changes to the code (bug fixes), but no API changes, increment
>   revision
> - only backward compatible API changes (new funcs), increment minor,
>   set revision to 0
> - if there are (also) incompatible API changes (removed funcs,
>   changed arguments), increment major, minor and revision to 0
> 
> Each major increment means an incompatible ABI for the given shared
> library, the minor and revision denote additions and fixes.  We can
> hence only focus on the major number.
> 
> Changes that are made to eclasses, can hence be reflected as revbumps:
> - fix functions, or make them smarter without breaking any API (no
>   revbump)
> - add new functions (no revbump)
> - drop functions, or change their interface (revbump)
> 
> Doing so will, however, lead to a lot of duplication of functions,
> since each incompatible change would have to lead to a new revision,
> that holds all the other functions as well -- unless some smart
> overloading trick can be applied.
> 
> Is this necessary for just one function, or is this only beneficial
> after a huge revamp of some eclass?  Is it acceptable to take the
> breakage outside Gentoo, given the extra work imposed by creating
> revisions and keeping track?
> 
> Any opinions, ideas or alternatives?
> 
> 

Some rules of thumb which pop into mind:

- Never change a public eclass function in a not backward compatible
  fashion.
- A function must be deprecated at least 1 (preferably 3) year(s) before
  it can be removed.
- A deprecated function must emit a deprecation warning on use.
- It's ok for a function to have different behaviour / api based on
  the ebuilds EAPI. So should be done only sporadically for obvious
  reasons.
- If a function needs change beyond what's possible without a not
  backward compatible api change the function needs a new name
  (appending version or whatever).
- Deprecating and removing / replacing a whole eclass can be done
  relatively fast. Consumers of the eclass can grab the eclass from
  attic if they need.
- Eclass names can't be reused. A replacement eclass gets a version
  appended / bumped.
- Internal and public functions / variables must be clearly
  distinguishable.
- Fixing abusers of internal functions / variables in main tree is
  sufficient for removing or changing the internal api.

Cheers,
sera