[gentoo-soc] Weekly Report: Big Data Infrastructure and Maven Overlay in Week 6

[gentoo-soc] Weekly Report: Big Data Infrastructure and Maven Overlay in Week 6

[Zhang Zongyu]

Hello,

This week I have made java-ebuilder accept "--binjar-uri" switch,
which will help us avoid circular deps. Also, It is helpful if we want
to compare the locally compiled jars and Maven Central distributed
jars. [1]
I have also patched java-pkg-simple to skip compiling source files if
USE="binary" is set.

One big progress is that I have managed to install
"org.apache.spark:spark-core_2.12:3.0.0-preview2". [2]
Most of the packages are compiled natively, while a few packages are
Maven Central distributed binary jars.

The reason why I have to install binaries varies. Here are the main reasons:
1. Maven central only distributes binary jars (even those jars are
named with a "sources" suffix). I will fix the problem by changing the
SRC_URI of a proper source code file next week.
2. The source code files are not written in Java. Some Jars contain
Ruby, Scala, Kotlin, or Lombok codes. [3] Maybe we should discuss how
to emerge those packages next week.

There are also other issues:
1. Netty is implemented by org.glassfish and com.sun, but they ended
up using the same artifactId. They now share the same package name. I
will rename them next week.
2. packages of javax.* and jakarta.*: I believe they should be merged
into java-virtuals/.

Next week, I will improve the quality of the overlay and submit some
up-to-date ebuild files.
The most urgent package should be netty-tcnative, because openssl has
changed a lot during these years and the current package cannot be
compiled now.

Regards,
Zhang Zongyu

[1] java-ebuilder
https://github.com/6-6-6/java-ebuilder
[2] the overlay
https://github.com/6-6-6/spark-overlay
[3] an example artifact that contains Kotlin source codes
https://repo1.maven.org/maven2/com/squareup/okio/okio/2.6.0/okio-2.6.0-sources.jar

Re: [gentoo-soc] Weekly Report: Big Data Infrastructure and Maven Overlay in Week 6

[Miroslav Šulc]

hi Zhongyu,

thank you for the report! looks like a nice progress to me :-) is there 
anything i could help you with?

wrt java-virtuals/*, there was a request (from mgorny iirc) to get rid 
of java-virtuals/*. imo the logic of java-virtuals should be preserved, 
but the virtuals should be placed under dev-java/ and use a naming 
template like this:

virtual = dev-java/servlet-api
implementations = dev-java/tomcat-servlet-api, 
dev-java/glassfish-servlet-api etc.

also, is java-pkg-simple.eclass ready for the main tree? and if so, did 
you consider to go through the review process of the eclass? it might 
take some time to get through the process so it might be good to start 
the process soon to not be blocked by the process later.

i also just pushed your commits to the main java-ebuilder repo.

thank you for your work and best regards!

miroslav


Dne 12. 07. 20 v 17:42 Zhang Zongyu napsal(a):
> Hello,
>
> This week I have made java-ebuilder accept "--binjar-uri" switch,
> which will help us avoid circular deps. Also, It is helpful if we want
> to compare the locally compiled jars and Maven Central distributed
> jars. [1]
> I have also patched java-pkg-simple to skip compiling source files if
> USE="binary" is set.
>
> One big progress is that I have managed to install
> "org.apache.spark:spark-core_2.12:3.0.0-preview2". [2]
> Most of the packages are compiled natively, while a few packages are
> Maven Central distributed binary jars.
>
> The reason why I have to install binaries varies. Here are the main reasons:
> 1. Maven central only distributes binary jars (even those jars are
> named with a "sources" suffix). I will fix the problem by changing the
> SRC_URI of a proper source code file next week.
> 2. The source code files are not written in Java. Some Jars contain
> Ruby, Scala, Kotlin, or Lombok codes. [3] Maybe we should discuss how
> to emerge those packages next week.
>
> There are also other issues:
> 1. Netty is implemented by org.glassfish and com.sun, but they ended
> up using the same artifactId. They now share the same package name. I
> will rename them next week.
> 2. packages of javax.* and jakarta.*: I believe they should be merged
> into java-virtuals/.
>
> Next week, I will improve the quality of the overlay and submit some
> up-to-date ebuild files.
> The most urgent package should be netty-tcnative, because openssl has
> changed a lot during these years and the current package cannot be
> compiled now.
>
> Regards,
> Zhang Zongyu
>
> [1] java-ebuilder
> https://github.com/6-6-6/java-ebuilder
> [2] the overlay
> https://github.com/6-6-6/spark-overlay
> [3] an example artifact that contains Kotlin source codes
> https://repo1.maven.org/maven2/com/squareup/okio/okio/2.6.0/okio-2.6.0-sources.jar
>

Re: [gentoo-soc] Weekly Report: Big Data Infrastructure and Maven Overlay in Week 6

[Ján Zahornadský]

On 12/07/2020 16:42, Zhang Zongyu wrote:
> The reason why I have to install binaries varies. Here are the main reasons:
> 1. Maven central only distributes binary jars (even those jars are
> named with a "sources" suffix). I will fix the problem by changing the
> SRC_URI of a proper source code file next week.

Could you show me a sources jar that doesn't contain sources? Just 
curious, I've never ran into any.

In principle, a Maven repository can contain any files, so it's 
possible, but in general, if a project uses Maven sources plugin, the 
jar should contain sources just right. Those sources however aren't 
intended to be compiled on their own, they serve as documentation or are 
used for IDE autocomplete.

You'd need to checkout the project's repository (including the build 
tools) to be able to build the project in general.

> 2. The source code files are not written in Java. Some Jars contain
> Ruby, Scala, Kotlin, or Lombok codes. [3] Maybe we should discuss how
> to emerge those packages next week.

Just a note, Lombok isn't a language, but Java compiler plugin that 
generates bytecode from annotations at compile time.

Scala and Kotlin are a common sight, as they compile into the same 
bytecode that JVM can run.

Re: [gentoo-soc] Weekly Report: Big Data Infrastructure and Maven Overlay in Week 6

[Benda Xu]

Zhang Zongyu <zzy2529420793@gmail.com> writes:

> There are also other issues:
> 1. Netty is implemented by org.glassfish and com.sun, but they ended
> up using the same artifactId. They now share the same package name. I
> will rename them next week.

Are the source codes of the two different from each other?

Benda

Re: [gentoo-soc] Weekly Report: Big Data Infrastructure and Maven Overlay in Week 6

[Zhang Zongyu]

Miroslav Šulc <fordfrog@gentoo.org> 于2020年7月12日周日 下午11:59写道：
>
> hi Zhongyu,
>
> thank you for the report! looks like a nice progress to me :-) is there
> anything i could help you with?

I think we may discuss about how to compare locally compiles jars and
binjars, and how to emerge kotlin, scala and lombok source files.

> wrt java-virtuals/*, there was a request (from mgorny iirc) to get rid
> of java-virtuals/*. imo the logic of java-virtuals should be preserved,
> but the virtuals should be placed under dev-java/ and use a naming
> template like this:
>
> virtual = dev-java/servlet-api
> implementations = dev-java/tomcat-servlet-api,
> dev-java/glassfish-servlet-api etc.
>
> also, is java-pkg-simple.eclass ready for the main tree? and if so, did
> you consider to go through the review process of the eclass? it might
> take some time to get through the process so it might be good to start
> the process soon to not be blocked by the process later.

Currently, there are few differences between the main tree version and
the one in my overlay.
Functions to deal with JAVA_RESOURCES and JAVA_TEST_SRC_DIR do not
appear in java-pkg-simple now.
I am not sure whether I should go through the review now, because
there isn't so much progress for java-pkg-simple.
So maybe I will not go through the review this week.

> i also just pushed your commits to the main java-ebuilder repo.
>
> thank you for your work and best regards!
>
> miroslav
>
>

Regards,
Zhang Zongyu

Re: [gentoo-soc] Weekly Report: Big Data Infrastructure and Maven Overlay in Week 6

[Zhang Zongyu]

>
> On 12/07/2020 16:42, Zhang Zongyu wrote:
> > The reason why I have to install binaries varies. Here are the main reasons:
> > 1. Maven central only distributes binary jars (even those jars are
> > named with a "sources" suffix). I will fix the problem by changing the
> > SRC_URI of a proper source code file next week.
>
> Could you show me a sources jar that doesn't contain sources? Just
> curious, I've never ran into any.

I am so sorry that it was a huge mistake. :(
I forgot to clear the old jars of in my ${DISTDIR}, and filenames of
some pre-built jars and source-code-jars happened to be the same.
I recognized it as *-sources.jars with pre-built class files. I am really
sorry for the misleading, and I will fix the problem this week.

> In principle, a Maven repository can contain any files, so it's
> possible, but in general, if a project uses Maven sources plugin, the
> jar should contain sources just right. Those sources however aren't
> intended to be compiled on their own, they serve as documentation or are
> used for IDE autocomplete.
>
> You'd need to checkout the project's repository (including the build
> tools) to be able to build the project in general.

Thanks for the advice. I am going to do so.

> > 2. The source code files are not written in Java. Some Jars contain
> > Ruby, Scala, Kotlin, or Lombok codes. [3] Maybe we should discuss how
> > to emerge those packages next week.
>
> Just a note, Lombok isn't a language, but Java compiler plugin that
> generates bytecode from annotations at compile time.
>
> Scala and Kotlin are a common sight, as they compile into the same
> bytecode that JVM can run.
>

It seems they can all work with Java, so I am not surprised that they
appear in Maven Central.
I will learn about those languages and may have to extend
java-pkg-simple to compile them in the next few weeks.

Regards,
Zhang Zongyu

Re: [gentoo-soc] Weekly Report: Big Data Infrastructure and Maven Overlay in Week 6

[Zhang Zongyu]

>
> Are the source codes of the two different from each other?
>
> Benda
>

I have to apologize for the confusion of "netty" and "jersey". There
are actually two "jersey" artifacts.
Sun-jersey is organised like "com/sun/jersey", while glassfish-jersey
is organized like "org/glassfish/jersey". [1] [2]
And their code differs in some way, so I cannot just
"sed 's/com.sun.jersey/org.glassfish.jersey/'"
to avoid the problem.

However, it is possible that sun-jersey is migrated to
glassfish-jersey in some time.

I may find whether they are two totally different artifacts or they
are the same artifact developed at different times. (like javax.* and
jakarta.*)
Then it will be clear to decide whether to slot those ebuild files or
to rename those packages.

Regards,
Zhang Zongyu

[1] sun-jersey
https://repo1.maven.org/maven2/com/sun/jersey/jersey-server/1.19.4/
[2] glassfish-jersey
https://repo1.maven.org/maven2/org/glassfish/jersey/core/jersey-server/2.31/

Re: [gentoo-soc] Weekly Report: Big Data Infrastructure and Maven Overlay in Week 6

[Miroslav Šulc]

Dne 13. 07. 20 v 5:20 Zhang Zongyu napsal(a):
> Miroslav Šulc <fordfrog@gentoo.org> 于2020年7月12日周日 下午11:59写道：
>> hi Zhongyu,
>>
>> thank you for the report! looks like a nice progress to me :-) is there
>> anything i could help you with?
> I think we may discuss about how to compare locally compiles jars and
> binjars, and how to emerge kotlin, scala and lombok source files.

there are tools that could help with the comparison: 
https://wiki.gentoo.org/wiki/Java_Developer_Guide#Java_API_Check

imo there are two checks needed:
1) check whether the api is the same both for our binary and upstream 
binary (that verifies no code is missing)
2) check whether all needed resources are included in our binary (that 
verifies that all resources needed for runtime will be available)

there might also be changes in the manifest file but those might not be 
that important in most cases imo

>> wrt java-virtuals/*, there was a request (from mgorny iirc) to get rid
>> of java-virtuals/*. imo the logic of java-virtuals should be preserved,
>> but the virtuals should be placed under dev-java/ and use a naming
>> template like this:
>>
>> virtual = dev-java/servlet-api
>> implementations = dev-java/tomcat-servlet-api,
>> dev-java/glassfish-servlet-api etc.
>>
>> also, is java-pkg-simple.eclass ready for the main tree? and if so, did
>> you consider to go through the review process of the eclass? it might
>> take some time to get through the process so it might be good to start
>> the process soon to not be blocked by the process later.
> Currently, there are few differences between the main tree version and
> the one in my overlay.
> Functions to deal with JAVA_RESOURCES and JAVA_TEST_SRC_DIR do not
> appear in java-pkg-simple now.
> I am not sure whether I should go through the review now, because
> there isn't so much progress for java-pkg-simple.
> So maybe I will not go through the review this week.
>
>> i also just pushed your commits to the main java-ebuilder repo.
>>
>> thank you for your work and best regards!
>>
>> miroslav
>>
>>
> Regards,
> Zhang Zongyu
>