[gentoo-dev] What are eblits?

[gentoo-dev] What are eblits?

[rindeal]

I've noticed that ebuilds for at least dev-lang/perl and
sys-libs/glibc are using some concept of "eblits", which seems like
parts of ebuilds scattered across $FILESDIR with ebuilds containing
some logic (involving eval) which includes and runs them.

I haven't found any documentation related to them so I'm asking here:

1) what are they?
2) why are they used?

Re: [gentoo-dev] What are eblits?

[Kent Fredric]

On 27 May 2016 at 10:28, rindeal <dev.rindeal@gmail.com> wrote:
>
> 1) what are they?
> 2) why are they used?


My best explanation is its a way to re-use very large amounts of code
between 2 ebuilds, without resorting to:

a) Copying the whole ebuild and hoping you find the relevant part in diffs
b) Not needing reams of version-specific conditional code to make
copying ebuilds between versions easier.
c) Not needing confusing eclasses that exist to serve a single
package, loaded with lots of weird conditional logic.

My understanding is you could effectively roll the eblits back into
the ebuild statically, just doing so would make keeping the changes
consistent harder.

Its clearly designed for a system where you have ~10 different
versions of Perl available or ~10 different versions of glibc
available, but you don't want to pay the price of duplicating that
logic wholesale for every minor concurrent revision, and only want to
update essential differences when you need to, not because you have
to.

That said, its a very confusing system to get your head around,
because its *basically* yet another "mixin" system like "inherit", but
done in bash, which itself is a rather strange language to be doing
something as complicated as mixins.

-- 
Kent

KENTNL - https://metacpan.org/author/KENTNL

Re: [gentoo-dev] What are eblits?

[konsolebox]

On Fri, May 27, 2016 at 9:28 AM, Kent Fredric <kentfredric@gmail.com> wrote:
> That said, its a very confusing system to get your head around,
> because its *basically* yet another "mixin" system like "inherit", but
> done in bash, which itself is a rather strange language to be doing
> something as complicated as mixins.

This concept is new to me as well, but after seeing the code with `cat
/usr/portage/sys-libs/glibc/glibc-2.17.ebuild` it didn't even take me
5 or 10 minutes to understand it.  I find the implementation safe
enough if used properly.  The core functions also seem stable, besides
some parts which aren't quoted, although commonly normal in ebuilds.
This is all about loading common functions to your ebuilds, and
there's nothing really dangerous in it.

-- 
konsolebox

Re: [gentoo-dev] What are eblits?

[Joshua Kinard]

On 05/26/2016 21:28, Kent Fredric wrote:
> On 27 May 2016 at 10:28, rindeal <dev.rindeal@gmail.com> wrote:
>>
>> 1) what are they?
>> 2) why are they used?
> 
> 
> My best explanation is its a way to re-use very large amounts of code
> between 2 ebuilds, without resorting to:
> 
> a) Copying the whole ebuild and hoping you find the relevant part in diffs
> b) Not needing reams of version-specific conditional code to make
> copying ebuilds between versions easier.
> c) Not needing confusing eclasses that exist to serve a single
> package, loaded with lots of weird conditional logic.
> 
> My understanding is you could effectively roll the eblits back into
> the ebuild statically, just doing so would make keeping the changes
> consistent harder.
> 
> Its clearly designed for a system where you have ~10 different
> versions of Perl available or ~10 different versions of glibc
> available, but you don't want to pay the price of duplicating that
> logic wholesale for every minor concurrent revision, and only want to
> update essential differences when you need to, not because you have
> to.
> 
> That said, its a very confusing system to get your head around,
> because its *basically* yet another "mixin" system like "inherit", but
> done in bash, which itself is a rather strange language to be doing
> something as complicated as mixins.

An accurate explanation, but probably better to say they're more like "local
eclasses".  If you think of the existing eclasses in ${PORTDIR}/eclass as
"global", that is, they contain common code utilized by multiple unrelated
ebuild packages, then eblits were originally envisioned as localized versions
of eclasses to share package-specific code between multiple ebuilds of
differing versions.

Whether the idea is useful in the present day and age, eh, who knows.  For the
mips-sources ebuilds, eblits let me centralize the per-machine notes and
unpacking logic, which reduced each ebuild's size from ~18KB a few years ago
down to ~4.9KB today.

-- 
Joshua Kinard
Gentoo/MIPS
kumba@gentoo.org
6144R/F5C6C943 2015-04-27
177C 1972 1FB8 F254 BAD0 3E72 5C63 F4E3 F5C6 C943

"The past tempts us, the present confuses us, the future frightens us.  And our
lives slip away, moment by moment, lost in that vast, terrible in-between."

--Emperor Turhan, Centauri Republic

Re: [gentoo-dev] What are eblits?

[Rich Freeman]

On Sat, May 28, 2016 at 9:11 PM, Joshua Kinard <kumba@gentoo.org> wrote:
>
> Whether the idea is useful in the present day and age, eh, who knows.  For the
> mips-sources ebuilds, eblits let me centralize the per-machine notes and
> unpacking logic, which reduced each ebuild's size from ~18KB a few years ago
> down to ~4.9KB today.

It sounds pretty useful to me.  I've seen eclasses used by a single
package or two at most and full of conditional logic to handle all the
times upstream changed their distribution scheme.  Eblits sound a
whole lot better.

Since they're limited in scope there isn't really a need for list
review/etc, and they can be created/deleted/etc at will.

I wouldn't use them just to turn every 50 line ebuild into a 10 line
ebuild, but when you have substantial amounts of code re-used across
ebuilds refactoring it just makes sense.

What I would love to see is this be standardized.  An eclass or a GLEP
seems like the logical approach.

-- 
Rich

Re: [gentoo-dev] What are eblits?

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 1038 bytes --]

>>>>> On Sun, 29 May 2016, Rich Freeman wrote:

> What I would love to see is this be standardized. An eclass or a
> GLEP seems like the logical approach.

I am strongly opposed against this. Ebuilds should not source
executable code from random locations. This is also a huge QA
violation, since PMS neither guarantees FILESDIR to be available in
global scope (so especially, not during metadata generation), nor in
any of the pkg_* phases (like pkg_setup, where mips-sources sources
its eblits).

If there really is a need for such a feature, we should rather follow
an approach like the per-package eclasses previously suggested by mabi
and antarus [1], and support a pkg-inherit function in the next EAPI.
(Though I wouldn't add a new function, but add an option to inherit,
like "inherit -p".)

We could even think about per-category eclasses ("inherit -c"),
although it is not obvious where one would store them.

Ulrich

[1] https://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo/users/antarus/projects/gleps/glep-XX.txt?view=markup

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] What are eblits?

[Rich Freeman]

On Sun, May 29, 2016 at 4:25 AM, Ulrich Mueller <ulm@gentoo.org> wrote:
>>>>>> On Sun, 29 May 2016, Rich Freeman wrote:
>
>> What I would love to see is this be standardized. An eclass or a
>> GLEP seems like the logical approach.
>
> If there really is a need for such a feature, we should rather follow
> an approach like the per-package eclasses previously suggested by mabi
> and antarus [1], and support a pkg-inherit function in the next EAPI.
> (Though I wouldn't add a new function, but add an option to inherit,
> like "inherit -p".)
>

A GLEP, then...

-- 
Rich

Re: [gentoo-dev] What are eblits?

[Joshua Kinard]

On 05/29/2016 04:25, Ulrich Mueller wrote:
>>>>>> On Sun, 29 May 2016, Rich Freeman wrote:
> 
>> What I would love to see is this be standardized. An eclass or a
>> GLEP seems like the logical approach.
> 
> I am strongly opposed against this. Ebuilds should not source
> executable code from random locations. This is also a huge QA
> violation, since PMS neither guarantees FILESDIR to be available in
> global scope (so especially, not during metadata generation), nor in
> any of the pkg_* phases (like pkg_setup, where mips-sources sources
> its eblits).

I believe this was addressed already back in 2010, for mips-sources at least.
Can't speak for the other eblit consumers.  See Bug #300370.

Eblits used to be loaded in global scope, which is what caused the $FILESDIR
issue, so the change was to move them into a function, then call that function
from pkg_setup.  This also makes sure the eblit code gets packed into the tbz2
so it's available when $FILESDIR isn't.

As for sourcing from random locations...point, because the standard path isn't
set in stone in an eclass or GLEP.  $FILESDIR/eblits is the path used by at
least glibc and mips-sources eblits.  I haven't looked at Perl or PHP's.
Totally open to addressing that in a sane manner.


> If there really is a need for such a feature, we should rather follow
> an approach like the per-package eclasses previously suggested by mabi
> and antarus [1], and support a pkg-inherit function in the next EAPI.
> (Though I wouldn't add a new function, but add an option to inherit,
> like "inherit -p".)

That seems like an attempt to revive the elibs idea, which is what eblits were
eventually supposed to morph into at some point.  I have no idea why elibs died
off.  Probably lack of interest or need back in 2005, when Gentoo was far, far
smaller in size and scope.

IMHO, to avoid that happening again, any such solution (regardless of name)
should probably look at implementing the bare minimum needed to replace all
existing consumers of eblits with the least amount of upheaval.  Then new
functionality can be added down the road.  Try to do everything at once, and
it'll just die off again.


> We could even think about per-category eclasses ("inherit -c"),
> although it is not obvious where one would store them.

Easy...under eclass/, in per-category sub-folders :)

(yay for more directory bureaucracy!)


> Ulrich
> 
> [1] https://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo/users/antarus/projects/gleps/glep-XX.txt?view=markup
> 


-- 
Joshua Kinard
Gentoo/MIPS
kumba@gentoo.org
6144R/F5C6C943 2015-04-27
177C 1972 1FB8 F254 BAD0 3E72 5C63 F4E3 F5C6 C943

"The past tempts us, the present confuses us, the future frightens us.  And our
lives slip away, moment by moment, lost in that vast, terrible in-between."

--Emperor Turhan, Centauri Republic

[gentoo-dev] Re: What are eblits?

[Duncan]

rindeal posted on Fri, 27 May 2016 00:28:35 +0200 as excerpted:

> I've noticed that ebuilds for at least dev-lang/perl and sys-libs/glibc
> are using some concept of "eblits", which seems like parts of ebuilds
> scattered across $FILESDIR with ebuilds containing some logic (involving
> eval) which includes and runs them.
> 
> I haven't found any documentation related to them so I'm asking here:
> 
> 1) what are they?
> 2) why are they used?

Consider this answer a "gap" answer, until you get a better answer from 
Frysinger (who I believe is the one who invented the concept, or failing 
that, at least one of the primary users and expanders of the concept for 
the glibc ebuilds) or one of the perl devs that are directly involved, or 
QA...

Based on (obviously my understanding of) the explanations I've seen 
previously, you can think of them sort of like eclasses, reusable 
libraries used by multiple ebuilds, except that they're normally specific 
to the multiple ebuilds for a single package, instead of being code 
shared between many different packages.

Because they're normally only used by the different ebuilds for a single 
package, generally maintained by the same person, they don't have the 
same formality applied to them that eclasses do.  Eclass changes are 
normally posted here on the dev list for review before being committed, 
for instance, while eblits... not.

However, as a result of this informality and the subsequent gray area 
eblits inhabit, they tend to be discouraged for most packages and 
definitely for newer devs.  In general, for in-tree use and for repos 
following the same rules, consider eblits an "if you have to ask, it's 
not an option available to you" deal.  Alternatively put, if you believe 
they'd be useful for code destined for in-tree use, definitely consult 
with other devs and with QA before attempting to introduce your eblits, 
and be prepared for a "no" answer.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] What are eblits?

[Ciaran McCreesh]

On Fri, 27 May 2016 00:28:35 +0200
rindeal <dev.rindeal@gmail.com> wrote:
> 1) what are they?

A horrible QA violation.

> 2) why are they used?

Because some people like to feel special...

-- 
Ciaran McCreesh

[gentoo-dev] Re: What are eblits?

[rindeal]

I've found some more info on this topic from the internet archive.

- "[gentoo-dev] RFC: eblits.eclass"
http://news.gmane.org/find-root.php?message_id=4BC0F659.7000506%40gentoo.org
- https://github.com/transtone/zm-overlay/blob/master/eclass/eblits.eclass
- https://wiki.gentoo.org/wiki/GLEP:33
- "RFC: Reviving GLEP33" http://thread.gmane.org/gmane.linux.gentoo.devel/67451

It seems that "eblits" is a relative to yet another concept called
"elibs", which was proposed back in 2005 as GLEP33, but was never
completed. As opposed to "elibs", "eblits" do not require any special
EAPI or portage support and thus they're living and are tolerated to
these days, as they do not interact with anything outside of their
lawn.

Based on the explanation given by Kent Frederic and Duncan, I'd sum it up as:

"a way to share code between ebuilds of a certain package, living
concurrently in different slots"

Which can be abstracted as "isolated package-specific eclasses".

Inter-ebuild code sharing really seems like a problem for maintainers
of certain packages and eblits seem to provide a quick solution, so
that answers the "why".

This whole concept, however, raises the question (as suggested by
Ciaran McCreesh and Duncan) if it's allowed to split ebuilds to
several bash scripts and what have QA and dev-manual got to say in
this regard?

It's always better to have some official material rather than an oral tradition.

Re: [gentoo-dev] Re: What are eblits?

[Kent Fredric]

On 28 May 2016 at 05:35, rindeal <dev.rindeal@gmail.com> wrote:
> This whole concept, however, raises the question (as suggested by
> Ciaran McCreesh and Duncan) if it's allowed to split ebuilds to
> several bash scripts and what have QA and dev-manual got to say in
> this regard?


Personally I'd say the biggest risk from a QA perspective of this
approach is important changes shared amongst ebuilds might require the
change be done in an eblit, but the change itself may require all
existing users of the ebuilds perform a reinstall.

This is already a problem with eclasses where eclass changes might
necessitate all dependent ebuilds being rebuilt in some way, but we
fence that out of existence with QA policies against such changes. ( A
popular fencing mechanism is via EAPI conditionals and ENV vars which
require the end ebuild to explicitly opt in to the change for it to
take affect, thus, causing the propagation to occur explicitly )

Under eblits, the same sorts of logic can occur, but the temptation to
change the eblit and not the ebuild is substantially greater.

But the necessity to bump the ebuild to cascade the rebuild is still
there ( and greater )

Which means in practice, eblits can make cascades harder, and
encourage devs not to perform ...

Which is a rather bad  combination of pressures.

Hence, eblits as they currently exist are experts-only and a big
danger ground for QA violations *to occur within*, even under the
presumption that they're not inherently a QA violation in themselves.

-- 
Kent

KENTNL - https://metacpan.org/author/KENTNL

[gentoo-dev] Re: What are eblits?

[Duncan]

Kent Fredric posted on Sat, 28 May 2016 17:07:56 +1200 as excerpted:

> Which is a rather bad  combination of pressures.
> 
> Hence, eblits as they currently exist are experts-only and a big
> danger ground for QA violations *to occur within*, even under the
> presumption that they're not inherently a QA violation in themselves.

Thanks.  You well explained the details of /why/ eblits are experts-only, 
that I deliberately fuzzed over in my first thread reply.  Now I 
understand them better myself.  =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] What are eblits?

[Joshua Kinard]

[-- Attachment #1: Type: text/plain, Size: 2892 bytes --]

On 05/26/2016 18:28, rindeal wrote:
> I've noticed that ebuilds for at least dev-lang/perl and
> sys-libs/glibc are using some concept of "eblits", which seems like
> parts of ebuilds scattered across $FILESDIR with ebuilds containing
> some logic (involving eval) which includes and runs them.
> 
> I haven't found any documentation related to them so I'm asking here:
> 
> 1) what are they?
> 2) why are they used?

I'm the other user of eblits in sys-kernel/mips-sources.  There's a lot of
common code in that package that doesn't need to be duplicated between multiple
ebuilds, so years ago, I converted to using versioned eblits.  It's a touch
manual, in that I have to manually check that all consumers of a specific eblit
version are removed from the tree before I remove the deprecated version, but
it avoids the nasty problem of changing an eblit for a newer ebuild that breaks
an older ebuild.

I started on an "eblits.eclass" a few months ago, but sidetracked and didn't
get back to it.  Attached, if anyone wants to play with it.  It includes the
core eblit loading functions used in mips-sources, which might be more
updated/robust than the ones in glibc (last I checked the logic, but vapier can
clarify if true or not).  The last comment block is incomplete.  I sidetracked
in finishing item #2 when pondering how to define a global "check" variable in
the eclass itself to control when the eblit core functions were loaded and all
eblits parsed (see latest mips-sources ebuild, $MIPS_SOURCES_EBLITS_LOADED).

Also handles the case of when installing from binary packages and $FILESDIR
isn't available.  The eblit source for the pkg_* phases are packed onto the end
of the bzip2 tarball so they're available to Portage & friends.

The core eblit loading functions are the only duplicated code in the
mips-sources ebuilds, as well as glibc and other eblit users (perl?).  I don't
remember what the argument for or against eblits might have been back in the
day, but for ebuilds whose only real changes are datestamps and/or version
numbers, eblits are a nice way to encapsulate and share common code that is too
specific for a global eclass.  Centralizing the eblit code will allow
mips-sources and glibc ebuilds to shrink their overall filesize a little-bit
more (~1.5KB per mips-sources ebuild, so 3.4KB per ebuild instead of the
current 4.9KB).

And yes, for anyone wondering, I have new mips-sources ebuilds.  Just took a
month to partially rewrite the SGI Origin/IP27 kernel code and then hunt down a
hardware bug on my SGI Octane, so, been distracted...

-- 
Joshua Kinard
Gentoo/MIPS
kumba@gentoo.org
6144R/F5C6C943 2015-04-27
177C 1972 1FB8 F254 BAD0 3E72 5C63 F4E3 F5C6 C943

"The past tempts us, the present confuses us, the future frightens us.  And our
lives slip away, moment by moment, lost in that vast, terrible in-between."

--Emperor Turhan, Centauri Republic

[-- Attachment #2: eblit.eclass --]
[-- Type: text/plain, Size: 3045 bytes --]

# Copyright 1999-2016 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Id$

# Description: eblit.eclass contains the core functions for creating "eblits".
#              eblits are a place to store common code shared among multiple
#              ebuilds for a specific package.
#
# Original Eblit Author: Mike Frysinger <vapier@gentoo.org>
# Original Eclass Author: Joshua Kinard <kumba@gentoo.org>
# Maintainer: base-system@gentoo.org

# Implementation Note: Since code in an eblit is used by multiple ebuilds,
# PLEASE revbump the eblit files when changes are made.  Then update each
# ebuild to use the new eblit version.  Remove old eblit versions when there
# are no more consumers.  This makes it a lot easier to debug problems with
# the shared code within an eblit, as well as the affected ebuilds.

# To create an eblit:
#  1. Create an "eblits" subdirectory under ${FILESDIR}, if one does not
#     already exist.
#  2. Create a new file name using the following formula:
#     [a-z0-9_]-v[0-9]+.eblit
#  3. Add the shared code, save, and close the file.
#
# Try to keep eblits specific to the functions they implement.  E.g., if a
# number of ebuilds have a large, but common src_unpack() function, and it
# is not already provided by an eclass, then add that code to an eblit named
# "src_unpack-vX.eblit".

# To load and use eblits:
#  1. Inherit the "eblit" eclass (this class).
#  2. Define a new function called "load_eblit_funcs" in the ebuild immediately
#     after the global ebuild variables


# eblit-core
# Usage: <function> [version]
# Main eblit engine
eblit-core() {
	local e v func=$1 ver=$2
	for v in ${ver:+-}${ver} -${PVR} -${PV} "" ; do
		e="${FILESDIR}/eblits/${func}${v}.eblit"
		if [[ -e ${e} ]] ; then
			. "${e}"
			[[ ${func} == pkg_* ]] && eval "${func}() { eblit-run ${func} ${ver} ; }"
			return 0
		fi
	done
	return 1
}

# eblit-include
# Usage: [--skip] <function> [version]
# Includes an "eblit" -- a chunk of common code among ebuilds in a given
# package so that its functions can be sourced and utilized within the
# ebuild.
eblit-include() {
	local skipable=false r=0
	[[ $1 == "--skip" ]] && skipable=true && shift
	[[ $1 == pkg_* ]] && skipable=true

	[[ -z $1 ]] && die "Usage: eblit-include <function> [version]"
	eblit-core $1 $2
	r="$?"
	${skipable} && return 0
	[[ "$r" -gt "0" ]] && die "Could not locate requested eblit '$1' in ${FILESDIR}/eblits/"
}

# eblit-run-maybe
# Usage: <function>
# Runs a function if it is defined in an eblit
eblit-run-maybe() {
	[[ $(type -t "$@") == "function" ]] && "$@"
}

# eblit-run
# Usage: <function> [version]
# Runs a function defined in an eblit
eblit-run() {
	eblit-include --skip common "${*:2}"
	eblit-include "$@"
	eblit-run-maybe eblit-$1-pre
	eblit-${PN}-$1
	eblit-run-maybe eblit-$1-post
}

# eblit-pkg
# Usage: <phase> [version]
# Runs the pkg_* functions AND evals them so they're included in the binpkgs
eblit-pkg() {
	[[ -z $1 ]] && die "Usage: eblit-pkg <phase> [version]"
	eblit-core pkg_$1 $2
}