[gentoo-soc] Week 8 Report for Big Data Infrastructure and H2O ebuilds Project

[gentoo-soc] Week 8 Report for Big Data Infrastructure and H2O ebuilds Project

[Yuan Liao (Leo)]

Hi folks,

The past week was largely a week of stabilization.  First and
foremost, I tried to use the Kotlin eclasses I had worked on about a
week ago to retrofit existing ebuilds for Kotlin packages including
okhttp, okio and reactor-core, all of which were successful.  After
tinkering with those eclasses, I believed they were ready for creation
of more Kotlin packages on Gentoo.  eselect-kotlin was also stabilized
a few days later with an initial v0.1.0 release and a specification
for any Kotlin compiler packages on Gentoo that are intended to be
recognized by the eselect module [1].  Then, I continued to fix broken
ebuilds in the Spark overlay and finally reached a state where all
packages can be installed with the default USE flag settings, and the
changes have been merged into Zongyu's main repository created last
year.  During this process, I also helped resolve a bug in the
dev-java/zstd-jni package (#776910) when it was being pulled as a
dependency of packages in the Spark overlay and bump dev-java/jansi in
::gentoo to 1.13 after fixing an ebuild in the Spark overlay for the
same version of the package.

Before I started to revamp existing ebuild in the Spark overlay, there
were about 60 out of 500 packages that could not be built from source
and thus would require USE="binary" to be explicitly enabled for them
by the users.  This week, I went through each of them one by one in
the hope to make them installable without the 'binary' USE flag.  The
most common reason that they could not be built from source was issues
within dependencies, and one of the following resolution was applied
to each ebuild depending on the actual situation:

- A dependency's version was incorrect, and the correct version was
available in either ::gentoo or the Spark overlay.  If the correct
version was in a different slot, then the slot would be changed in the
problematic ebuild.  Otherwise, a new slot would be created for the
correct version so that the problematic ebuild could use the version
without affecting other ebuilds.

- A dependency's version was incorrect, and the correct version is too
old.  This often happened to packages requiring Guava.  If an ebuild
in the Spark overlay would need Guava, it usually would depend on
Guava 29.0; but, the correct Guava version needed to build some of the
packages could be as old as 18.0.  In this case, the reason the ebuild
could not be compiled from source was that some APIs in Guava had been
removed.  Fortunately, for every removed API, Google would state its
replacement API(s) in Guava's documentation, so this kind of issue
could be resolved by patching the problematic ebuild's source files to
switch to the replacement APIs.

- A package that existed in either ::gentoo or the Spark overlay was
missing in the build-time classpath.  In this case, the package would
be added to the problematic ebuild as a dependency.

- A package that did not exist was required for building, and an
ebuild for the package could be easily created with java-ebuilder or
by hand.  The new ebuild would be added to the Spark overlay, and the
problematic ebuild would depend on it.

- A package that did not exist was required, and it was hard to create
an ebuild for it; or, the build failure was caused by an issue not
related to dependencies and could not be trivially fixed.  To save
time for the remaining final parts of my GSoC project, I just simply
added IUSE="+binary" to the ebuild to enable the USE flag by default
with a comment describing the obstacles to compile the package from
source.

My new blog post for this week is now available [2].  The topic for
this post is the ebuild testing infrastructure I created and have been
using to verify the working status of ebuilds in the Spark overlay.
An interesting fact for the testing framework is that I used it to
test my fix for the zstd-jni bug and the new jansi-1.13 ebuild too, so
it is not only suitable for testing small ebuild repositories but
probably can be used to test a subset of ebuilds in ::gentoo as well.

Thanks for reading,
Leo

[1]: https://github.com/Leo3418/eselect-kotlin/blob/v0.1.0/README.md#gentoo-kotlin-compiler-package-specification
[2]: https://leo3418.github.io/2021/08/01/ebuild-repos-testing-solution.html

Re: [gentoo-soc] Week 8 Report for Big Data Infrastructure and H2O ebuilds Project

[Benda Xu]

Hi Leo,

"Yuan Liao (Leo)" <liaoyuan@gmail.com> writes:

> The past week was largely a week of stabilization.  First and
> foremost, I tried to use the Kotlin eclasses I had worked on about a
> week ago to retrofit existing ebuilds for Kotlin packages including
> okhttp, okio and reactor-core, all of which were successful.  After
> tinkering with those eclasses, I believed they were ready for creation
> of more Kotlin packages on Gentoo.  eselect-kotlin was also stabilized
> a few days later with an initial v0.1.0 release and a specification
> for any Kotlin compiler packages on Gentoo that are intended to be
> recognized by the eselect module [1].  Then, I continued to fix broken
> ebuilds in the Spark overlay and finally reached a state where all
> packages can be installed with the default USE flag settings, and the
> changes have been merged into Zongyu's main repository created last
> year.  During this process, I also helped resolve a bug in the
> dev-java/zstd-jni package (#776910) when it was being pulled as a
> dependency of packages in the Spark overlay and bump dev-java/jansi in
> ::gentoo to 1.13 after fixing an ebuild in the Spark overlay for the
> same version of the package.
>
> Before I started to revamp existing ebuild in the Spark overlay, there
> were about 60 out of 500 packages that could not be built from source
> and thus would require USE="binary" to be explicitly enabled for them
> by the users.  This week, I went through each of them one by one in
> the hope to make them installable without the 'binary' USE flag.  The
> most common reason that they could not be built from source was issues
> within dependencies, and one of the following resolution was applied
> to each ebuild depending on the actual situation:
>
> - A dependency's version was incorrect, and the correct version was
> available in either ::gentoo or the Spark overlay.  If the correct
> version was in a different slot, then the slot would be changed in the
> problematic ebuild.  Otherwise, a new slot would be created for the
> correct version so that the problematic ebuild could use the version
> without affecting other ebuilds.
>
> - A dependency's version was incorrect, and the correct version is too
> old.  This often happened to packages requiring Guava.  If an ebuild
> in the Spark overlay would need Guava, it usually would depend on
> Guava 29.0; but, the correct Guava version needed to build some of the
> packages could be as old as 18.0.  In this case, the reason the ebuild
> could not be compiled from source was that some APIs in Guava had been
> removed.  Fortunately, for every removed API, Google would state its
> replacement API(s) in Guava's documentation, so this kind of issue
> could be resolved by patching the problematic ebuild's source files to
> switch to the replacement APIs.
>
> - A package that existed in either ::gentoo or the Spark overlay was
> missing in the build-time classpath.  In this case, the package would
> be added to the problematic ebuild as a dependency.
>
> - A package that did not exist was required for building, and an
> ebuild for the package could be easily created with java-ebuilder or
> by hand.  The new ebuild would be added to the Spark overlay, and the
> problematic ebuild would depend on it.
>
> - A package that did not exist was required, and it was hard to create
> an ebuild for it; or, the build failure was caused by an issue not
> related to dependencies and could not be trivially fixed.  To save
> time for the remaining final parts of my GSoC project, I just simply
> added IUSE="+binary" to the ebuild to enable the USE flag by default
> with a comment describing the obstacles to compile the package from
> source.

It is good enough for practical cases.  Solving it incrementally and
leave the hard ones for the future is the way to go.  Kudos!

> My new blog post for this week is now available [2].  The topic for
> this post is the ebuild testing infrastructure I created and have been
> using to verify the working status of ebuilds in the Spark overlay.
> An interesting fact for the testing framework is that I used it to
> test my fix for the zstd-jni bug and the new jansi-1.13 ebuild too, so
> it is not only suitable for testing small ebuild repositories but
> probably can be used to test a subset of ebuilds in ::gentoo as well.

Cool blog entry.  Learned.

Cheers,
Benda

> [1]: https://github.com/Leo3418/eselect-kotlin/blob/v0.1.0/README.md#gentoo-kotlin-compiler-package-specification
> [2]: https://leo3418.github.io/2021/08/01/ebuild-repos-testing-solution.html