[gentoo-soc] Weekly Report: Big Data Infrastructure and Maven Overlay in Week 5

[gentoo-soc] Weekly Report: Big Data Infrastructure and Maven Overlay in Week 5

[Zhang Zongyu]

[-- Attachment #1: Type: text/plain, Size: 1436 bytes --]

Hello,

This week I have done a lot of repetitive work. That is, to go through
all of the ebuild files in Gentoo Repository and to get them the
equivalent GroupId and ArtifactId in Maven Central. There are more
than 500 packages in dev-java/ and java-virtual/ category, and
sometimes one Gentoo package may correspond to more than one Maven
artifact, so it took me a lot of time to complete them. And here is my
lookup up table which covers all packages in dev-java/ and maps
"category/package:slot" into "groupId:artifactId".[1] As a result,
spark-core can make use of more than 100 Gentoo native packages now.

Besides, I have also added java-ebuilder some features. It can check
the metadata of a package and get the correct SLOT, and it can make
ebuild unpack "Maven Central distributed source files" into proper
${JAVA_SRC_DIR} now.

Next week I am going to test and add jaxb (javax.xml.bind:jaxb-api and
javax.xml.bind:jaxb-impl) and some newer version packages which are
required by spark-core, and I hope the newer version ebuild files
could be accepted by Gentoo-Java team. The prototype of the ebuild
files that are just generated by java-ebuilder are listed in [2].

Regards,
Zhang Zongyu

[1] lookup table containing all packages from dev-java/
https://github.com/6-6-6/JEbuilder/blob/master/app-portage/java-ebuilder/files/scripts/LUT/LUT
[2] a prototype overlay
https://github.com/6-6-6/spark-overlay/tree/master/dev-java

[-- Attachment #2: Type: text/html, Size: 1879 bytes --]

Re: [gentoo-soc] Weekly Report: Big Data Infrastructure and Maven Overlay in Week 5

[EBo]

Out of curiosity, can you point me to any project write up that was 
probably part of the proposal?  One of my last big-data projects started 
out with a 3.5 petabyte repository and filtered on down to cover 10 
million contiguous kilometers.  We did not use Maven, and I would be 
curious to see where the project is going.  Now that I know about this I 
will try to follow it a bit more.

   Best of success!

On Jul 5 2020 10:05 AM, Zhang Zongyu wrote:
> Hello,
>
> This week I have done a lot of repetitive work. That is, to go 
> through
> all of the ebuild files in Gentoo Repository and to get them the
> equivalent GroupId and ArtifactId in Maven Central. There are more
> than 500 packages in dev-java/ and java-virtual/ category, and
> sometimes one Gentoo package may correspond to more than one Maven
> artifact, so it took me a lot of time to complete them. And here is 
> my
> lookup up table which covers all packages in dev-java/ and maps
> "category/package:slot" into "groupId:artifactId".[1] As a result,
> spark-core can make use of more than 100 Gentoo native packages now.
>
> Besides, I have also added java-ebuilder some features. It can check
> the metadata of a package and get the correct SLOT, and it can make
> ebuild unpack "Maven Central distributed source files" into proper
> ${JAVA_SRC_DIR} now.
>
> Next week I am going to test and add jaxb (javax.xml.bind:jaxb-api 
> and
> javax.xml.bind:jaxb-impl) and some newer version packages which are
> required by spark-core, and I hope the newer version ebuild files
> could be accepted by Gentoo-Java team. The prototype of the ebuild
> files that are just generated by java-ebuilder are listed in [2].
>
> Regards,
> Zhang Zongyu
>
> [1] lookup table containing all packages from dev-java/
> 
> https://github.com/6-6-6/JEbuilder/blob/master/app-portage/java-ebuilder/files/scripts/LUT/LUT
> [2] a prototype overlay
> https://github.com/6-6-6/spark-overlay/tree/master/dev-java

Re: [gentoo-soc] Weekly Report: Big Data Infrastructure and Maven Overlay in Week 5

[Zhang Zongyu]

[-- Attachment #1: Type: text/plain, Size: 2973 bytes --]

Hello,

Here is the proposal of this project.[1]
In a nutshell, the goal of this project is to intergrate Portage with
Maven, or to translate Maven pom.xml into ebuild files.
And the most related project should be java-ebuilder, which aims at
processing pom.xml and forming an equivalent ebuild.[2]
The deliverable of this project will be an overlay resolving all the
dependencies of spark, which is widely used for big data processing, so my
mentor named the project "big data infrastructure".

Actually it is not related to big data processing or any "big data".

I am sorry if it confused you, and I am glad that you are interested in
this project.

Regards,
Zhang Zongyu

[1] https://summerofcode.withgoogle.com/projects/#4994566568017920
[2] https://github.com/6-6-6/java-ebuilder

EBo <ebo@sandien.com> 于 2020年7月6日周一 上午12:18写道：

> Out of curiosity, can you point me to any project write up that was
> probably part of the proposal?  One of my last big-data projects started
> out with a 3.5 petabyte repository and filtered on down to cover 10
> million contiguous kilometers.  We did not use Maven, and I would be
> curious to see where the project is going.  Now that I know about this I
> will try to follow it a bit more.
>
>    Best of success!
>
> On Jul 5 2020 10:05 AM, Zhang Zongyu wrote:
> > Hello,
> >
> > This week I have done a lot of repetitive work. That is, to go
> > through
> > all of the ebuild files in Gentoo Repository and to get them the
> > equivalent GroupId and ArtifactId in Maven Central. There are more
> > than 500 packages in dev-java/ and java-virtual/ category, and
> > sometimes one Gentoo package may correspond to more than one Maven
> > artifact, so it took me a lot of time to complete them. And here is
> > my
> > lookup up table which covers all packages in dev-java/ and maps
> > "category/package:slot" into "groupId:artifactId".[1] As a result,
> > spark-core can make use of more than 100 Gentoo native packages now.
> >
> > Besides, I have also added java-ebuilder some features. It can check
> > the metadata of a package and get the correct SLOT, and it can make
> > ebuild unpack "Maven Central distributed source files" into proper
> > ${JAVA_SRC_DIR} now.
> >
> > Next week I am going to test and add jaxb (javax.xml.bind:jaxb-api
> > and
> > javax.xml.bind:jaxb-impl) and some newer version packages which are
> > required by spark-core, and I hope the newer version ebuild files
> > could be accepted by Gentoo-Java team. The prototype of the ebuild
> > files that are just generated by java-ebuilder are listed in [2].
> >
> > Regards,
> > Zhang Zongyu
> >
> > [1] lookup table containing all packages from dev-java/
> >
> >
> https://github.com/6-6-6/JEbuilder/blob/master/app-portage/java-ebuilder/files/scripts/LUT/LUT
> > [2] a prototype overlay
> > https://github.com/6-6-6/spark-overlay/tree/master/dev-java
>
>
>

[-- Attachment #2: Type: text/html, Size: 4374 bytes --]

Re: [gentoo-soc] Weekly Report: Big Data Infrastructure and Maven Overlay in Week 5

[EBo]

Ahhh... thank you for your time and explanation.  This all makes sense 
now.  I can see how having something that would pom.xml <==> 
package.ebuild would be extremely handy.  Best of luck to you you and 
your team.

   Thanks again,

   EBo --

On Jul 5 2020 11:23 AM, Zhang Zongyu wrote:
> Hello,
>
> Here is the proposal of this project.[1]
> In a nutshell, the goal of this project is to intergrate Portage with
> Maven, or to translate Maven pom.xml into ebuild files.
> And the most related project should be java-ebuilder, which aims at
> processing pom.xml and forming an equivalent ebuild.[2]
> The deliverable of this project will be an overlay resolving all the
> dependencies of spark, which is widely used for big data processing, 
> so my
> mentor named the project "big data infrastructure".
>
> Actually it is not related to big data processing or any "big data".
>
> I am sorry if it confused you, and I am glad that you are interested 
> in
> this project.
>
> Regards,
> Zhang Zongyu
>
> [1] https://summerofcode.withgoogle.com/projects/#4994566568017920
> [2] https://github.com/6-6-6/java-ebuilder
>
> EBo <ebo@sandien.com> 于 2020年7月6日周一 上午12:18写道：
>
>> Out of curiosity, can you point me to any project write up that was
>> probably part of the proposal?  One of my last big-data projects 
>> started
>> out with a 3.5 petabyte repository and filtered on down to cover 10
>> million contiguous kilometers.  We did not use Maven, and I would be
>> curious to see where the project is going.  Now that I know about 
>> this I
>> will try to follow it a bit more.
>>
>>    Best of success!
>>
>> On Jul 5 2020 10:05 AM, Zhang Zongyu wrote:
>> > Hello,
>> >
>> > This week I have done a lot of repetitive work. That is, to go
>> > through
>> > all of the ebuild files in Gentoo Repository and to get them the
>> > equivalent GroupId and ArtifactId in Maven Central. There are more
>> > than 500 packages in dev-java/ and java-virtual/ category, and
>> > sometimes one Gentoo package may correspond to more than one Maven
>> > artifact, so it took me a lot of time to complete them. And here 
>> is
>> > my
>> > lookup up table which covers all packages in dev-java/ and maps
>> > "category/package:slot" into "groupId:artifactId".[1] As a result,
>> > spark-core can make use of more than 100 Gentoo native packages 
>> now.
>> >
>> > Besides, I have also added java-ebuilder some features. It can 
>> check
>> > the metadata of a package and get the correct SLOT, and it can 
>> make
>> > ebuild unpack "Maven Central distributed source files" into proper
>> > ${JAVA_SRC_DIR} now.
>> >
>> > Next week I am going to test and add jaxb (javax.xml.bind:jaxb-api
>> > and
>> > javax.xml.bind:jaxb-impl) and some newer version packages which 
>> are
>> > required by spark-core, and I hope the newer version ebuild files
>> > could be accepted by Gentoo-Java team. The prototype of the ebuild
>> > files that are just generated by java-ebuilder are listed in [2].
>> >
>> > Regards,
>> > Zhang Zongyu
>> >
>> > [1] lookup table containing all packages from dev-java/
>> >
>> >
>> 
>> https://github.com/6-6-6/JEbuilder/blob/master/app-portage/java-ebuilder/files/scripts/LUT/LUT
>> > [2] a prototype overlay
>> > https://github.com/6-6-6/spark-overlay/tree/master/dev-java
>>
>>
>>

Re: [gentoo-soc] Weekly Report: Big Data Infrastructure and Maven Overlay in Week 5

[Benda Xu]

Zhang Zongyu <zzy2529420793@gmail.com> writes:

> This week I have done a lot of repetitive work. That is, to go through
> all of the ebuild files in Gentoo Repository and to get them the
> equivalent GroupId and ArtifactId in Maven Central. There are more
> than 500 packages in dev-java/ and java-virtual/ category, and
> sometimes one Gentoo package may correspond to more than one Maven
> artifact, so it took me a lot of time to complete them. And here is my
> lookup up table which covers all packages in dev-java/ and maps
> "category/package:slot" into "groupId:artifactId".[1] As a result,
> spark-core can make use of more than 100 Gentoo native packages now.

My previous approach was to `ebuild xxxx prepare` the packages in
::gentoo and inspect the pom.xml file inside ${WORKDIR}.  Does that work
for you?

  https://github.com/gentoo/java-ebuilder/blob/master/scripts/meta.sh

> Besides, I have also added java-ebuilder some features. It can check
> the metadata of a package and get the correct SLOT, and it can make
> ebuild unpack "Maven Central distributed source files" into proper
> ${JAVA_SRC_DIR} now.

Good!

> Next week I am going to test and add jaxb (javax.xml.bind:jaxb-api and
> javax.xml.bind:jaxb-impl) and some newer version packages which are
> required by spark-core, and 

Good to know we are getting back jaxb.  It is strange to me that without
such a seemingly central package packages in ::gentoo still work.  Is it
now part of icedtea jdk?  Please dig more into the story behind jaxb in
the #gentoo-java IRC and bugzilla.

Maybe fordfrog have a better answer (@fordfrog, not sure if you have
subscribed to gentoo-soc, I am Ccing you).

> I hope the newer version ebuild files could be accepted by Gentoo-Java
> team. The prototype of the ebuild files that are just generated by
> java-ebuilder are listed in [2].

Please start the code review processes of the new java packages by
submitting bug reports and pull requests when you see fit.

Benda

> [1] lookup table containing all packages from dev-java/
> https://github.com/6-6-6/JEbuilder/blob/master/app-portage/java-ebuilder/files/scripts/LUT/LUT
> [2] a prototype overlay
> https://github.com/6-6-6/spark-overlay/tree/master/dev-java

Re: [gentoo-soc] Weekly Report: Big Data Infrastructure and Maven Overlay in Week 5

[Miroslav Šulc]

hi,

i was not on this list (now i am) so thanks for cc'ing me.

Dne 06. 07. 20 v 3:09 Benda Xu napsal(a):
> Zhang Zongyu <zzy2529420793@gmail.com> writes:
>
>> This week I have done a lot of repetitive work. That is, to go through
>> all of the ebuild files in Gentoo Repository and to get them the
>> equivalent GroupId and ArtifactId in Maven Central. There are more
>> than 500 packages in dev-java/ and java-virtual/ category, and
>> sometimes one Gentoo package may correspond to more than one Maven
>> artifact, so it took me a lot of time to complete them. And here is my
>> lookup up table which covers all packages in dev-java/ and maps
>> "category/package:slot" into "groupId:artifactId".[1] As a result,
>> spark-core can make use of more than 100 Gentoo native packages now.
i just would like to bring to your attention the content of 
~/.java-ebuilder/cache. it contains all packages that are based at least 
on one of java eclasses, and those eclasses are also listed at the end 
of each row. might help you to find all packages that are java packages, 
even outside dev-java/ category.
> My previous approach was to `ebuild xxxx prepare` the packages in
> ::gentoo and inspect the pom.xml file inside ${WORKDIR}.  Does that work
> for you?
>
>    https://github.com/gentoo/java-ebuilder/blob/master/scripts/meta.sh
this seems to be a really clever idea :-)
>> Besides, I have also added java-ebuilder some features. It can check
>> the metadata of a package and get the correct SLOT, and it can make
>> ebuild unpack "Maven Central distributed source files" into proper
>> ${JAVA_SRC_DIR} now.
> Good!

i wanted to merge your commits to the main repo but your commits are not 
signed (which is required by gentoo) so the merge failed:

$ git push --signed
Enumerating objects: 160, done.
Counting objects: 100% (160/160), done.
Delta compression using up to 8 threads
Compressing objects: 100% (32/32), done.
Writing objects: 100% (140/140), 10.91 KiB | 10.91 MiB/s, done.
Total 140 (delta 48), reused 140 (delta 48), pack-reused 0
remote: Resolving deltas: 100% (48/48), completed with 13 local objects.
remote: FATAL: VREF/proj-gentoo-06-copyright: helper program exit status 256
remote: dd579db2a1505cec77815479ae6a3d6ef3a0c5ce: missing Signed-off-by 
on commit
remote: 99097c6f99c248ccd30fce4519215c257c0c7a89: missing Signed-off-by 
on commit
remote: 58874c99da6c32a2956b8ce4bb4b53478634c0da: missing Signed-off-by 
on commit
remote: 85f88068c6fe41ce0984f90818715967b9b4d15c: missing Signed-off-by 
on commit
remote: d93842e5ba03c6169cc040bab72b5abfc6626d68: missing Signed-off-by 
on commit
remote: ad49c323e6f023c2ddd4935dd1dee1f4a3ad0b60: missing Signed-off-by 
on commit
remote: 0ef403839d003795bb0afcca122e10b6bd0a8c44: missing Signed-off-by 
on commit
remote: e22c64c1c277b3a882cc30d7664cdf508f7f05eb: missing Signed-off-by 
on commit
remote: 974268aa05144c8f28a7b5872e85b4ef2059fb94: missing Signed-off-by 
on commit
remote: a1843f1c17aad889d5b6614bd14eeb992abf9925: missing Signed-off-by 
on commit
remote: 58ee12330f57eeb9b6772ef44889de8e93af4f1e: missing Signed-off-by 
on commit
remote: 2ff7dd8b48f5bc71636e9cedb9f03a95f234208e: missing Signed-off-by 
on commit
remote: f3a06517706afe5868f15508ead6d94c0c9c2080: missing Signed-off-by 
on commit
remote:
remote: Please make sure to read the copyright policy before adding 
Signed-off-by!
remote:   https://www.gentoo.org/glep/glep-0076.html
remote: error: hook declined to update refs/heads/master
To git+ssh://git.gentoo.org/proj/java-ebuilder.git
  ! [remote rejected] master -> master (hook declined)
error: failed to push some refs to 
'git+ssh://git.gentoo.org/proj/java-ebuilder.git'

generally, using `git commit -s ...` adds the required line on new 
commits. you can also rebase your commit messages ( `git rebase -i 
c3b197f0e69c06f58f982f8d8dd16c88e2d5d518`) and push the commits again 
afaik.

>> Next week I am going to test and add jaxb (javax.xml.bind:jaxb-api and
>> javax.xml.bind:jaxb-impl) and some newer version packages which are
>> required by spark-core, and
> Good to know we are getting back jaxb.  It is strange to me that without
> such a seemingly central package packages in ::gentoo still work.  Is it
> now part of icedtea jdk?  Please dig more into the story behind jaxb in
> the #gentoo-java IRC and bugzilla.
>
> Maybe fordfrog have a better answer (@fordfrog, not sure if you have
> subscribed to gentoo-soc, I am Ccing you).
javax/xml/bind is part of jdk:8 but was dropped later and is not present 
in jdk:11 anymore which causes some packages to fail to compile against 
jdk:11 (you can easily check with something like this: `for jdk in 
icedtea:8 openjdk:8 openjdk:11; do echo; echo $jdk; for jar in $(equery 
files $jdk | grep -E ".*\.jar$"); do echo $jar; unzip -l $jar | grep 
javax/xml/bind; done; done`). you have to have those jdk installed 
before running this command.
>> I hope the newer version ebuild files could be accepted by Gentoo-Java
>> team. The prototype of the ebuild files that are just generated by
>> java-ebuilder are listed in [2].
> Please start the code review processes of the new java packages by
> submitting bug reports and pull requests when you see fit.

do these work with java-pkg-simple.eclass as is in the tree or it needs 
your updated eclass? getting the changes in the eclass into the main 
tree requires review and approval by devs.

also, before getting the ebuilds into the tree, i would really like to 
see an automated test to verify that the content of our jar files 
matches the content of the maven repo jar files. without this, we might 
get issues because of missing resources and maybe even other causes.

> Benda

thank you for the work you do! :-)


miroslav

>> [1] lookup table containing all packages from dev-java/
>> https://github.com/6-6-6/JEbuilder/blob/master/app-portage/java-ebuilder/files/scripts/LUT/LUT
>> [2] a prototype overlay
>> https://github.com/6-6-6/spark-overlay/tree/master/dev-java

Re: [gentoo-soc] Weekly Report: Big Data Infrastructure and Maven Overlay in Week 5

[Miroslav Šulc]

Dne 06. 07. 20 v 10:26 Miroslav Šulc napsal(a):
> hi,
>
> i was not on this list (now i am) so thanks for cc'ing me.
>
> Dne 06. 07. 20 v 3:09 Benda Xu napsal(a):
>> Zhang Zongyu <zzy2529420793@gmail.com> writes:
>> ...
>>> I hope the newer version ebuild files could be accepted by Gentoo-Java
>>> team. The prototype of the ebuild files that are just generated by
>>> java-ebuilder are listed in [2].
>> Please start the code review processes of the new java packages by
>> submitting bug reports and pull requests when you see fit.
>
> do these work with java-pkg-simple.eclass as is in the tree or it 
> needs your updated eclass? getting the changes in the eclass into the 
> main tree requires review and approval by devs.
>
> also, before getting the ebuilds into the tree, i would really like to 
> see an automated test to verify that the content of our jar files 
> matches the content of the maven repo jar files. without this, we 
> might get issues because of missing resources and maybe even other 
> causes.

one more note. i noticed in your overlay you have only the ebuild files 
but the rest of the required files are missing. generally, you need to 
provide metadata.xml file for each package and need to run `repoman 
manifest` for each package. also, you have to run `repoman full` and 
also can run `pkgcheck scan` for each package to catch all issues that 
can be caught this way.

>
>> Benda
>
> thank you for the work you do! :-)
>
>
> miroslav
>
>>> [1] lookup table containing all packages from dev-java/
>>> https://github.com/6-6-6/JEbuilder/blob/master/app-portage/java-ebuilder/files/scripts/LUT/LUT 
>>>
>>> [2] a prototype overlay
>>> https://github.com/6-6-6/spark-overlay/tree/master/dev-java
>

Re: [gentoo-soc] Weekly Report: Big Data Infrastructure and Maven Overlay in Week 5

[Zhang Zongyu]

Hello,

> i just would like to bring to your attention the content of
> ~/.java-ebuilder/cache. it contains all packages that are based at least
> on one of java eclasses, and those eclasses are also listed at the end
> of each row. might help you to find all packages that are java packages,
> even outside dev-java/ category.

Thanks for the notice. Previously I believed that those categories
other than dev-java/ and java-virtuals/* are not likely to exist in
the database of Maven, so I ignored them last week. I will fill in the
blank of non-dev-java packages this week.

>>    https://github.com/gentoo/java-ebuilder/blob/master/scripts/meta.sh
> this seems to be a really clever idea :-)

This helped me a lot. Last week, I started with the preliminary cache
generated by meta.sh.

> generally, using `git commit -s ...` adds the required line on new
> commits. you can also rebase your commit messages ( `git rebase -i
> c3b197f0e69c06f58f982f8d8dd16c88e2d5d518`) and push the commits again
> afaik.

Sorry for the mistake I have made, and I will rebase the repository.

> javax/xml/bind is part of jdk:8 but was dropped later and is not present
> in jdk:11 anymore which causes some packages to fail to compile against
> jdk:11 (you can easily check with something like this: `for jdk in
> icedtea:8 openjdk:8 openjdk:11; do echo; echo $jdk; for jar in $(query
> files $jdk | grep -E ".*\.jar$"); do echo $jar; unzip -l $jar | grep
> javax/xml/bind; done; done`). you have to have those jdk installed
> before running this command.

Thanks for the information. So it seems that I do not need concern
about jaxb, and I can just set the proper classpath to make use of
JAXB stuffs instead. Maybe I will need to add a variable (e.g.
JAVA_NEEDS_RUNTIME) to control it.

> do these work with java-pkg-simple.eclass as is in the tree or it needs
> your updated eclass?

The class is modified by me. It can be found here[1]. The major
differences are the way to process JAVA_*_CLASSPATH variables and a
new variable called "JAVA_NEEDS_TOOLS".

> getting the changes in the eclass into the main tree requires review and approval by devs.

Is it correct that I should trigger the review via bugs.gentoo.org?

> also, before getting the ebuilds into the tree, i would really like to
> see an automated test to verify that the content of our jar files
> matches the content of the maven repo jar files. without this, we might
> get issues because of missing resources and maybe even other causes.

Currently, I do not have a clear way of reaching the goal...
A preliminary thought is to maintain a modified version of java-pkg-simple.
The modified eclass will automatically compare the files in postinst phase.
I will try to work out it this week.

> one more note. i noticed in your overlay you have only the ebuild files
> but the rest of the required files are missing. generally, you need to
> provide metadata.xml file for each package and need to run `repoman
> manifest` for each package. also, you have to run `repoman full` and
> also can run `pkgcheck scan` for each package to catch all issues that
> can be caught this way.

Got you, I will add the missing files.

Regards,
Zhang Zongyu

[1] https://github.com/6-6-6/spark-overlay/blob/master/eclass/java-pkg-simple.eclass

Re: [gentoo-soc] Weekly Report: Big Data Infrastructure and Maven Overlay in Week 5

[Benda Xu]

Zhang Zongyu <zzy2529420793@gmail.com> writes:

>> do these work with java-pkg-simple.eclass as is in the tree or it needs
>> your updated eclass?
>
> The class is modified by me. It can be found here[1]. The major
> differences are the way to process JAVA_*_CLASSPATH variables and a
> new variable called "JAVA_NEEDS_TOOLS".
>
>> getting the changes in the eclass into the main tree requires review and approval by devs.
>
> Is it correct that I should trigger the review via bugs.gentoo.org?

Initiate the eclass patch review by starting a thread in
gentoo-dev@l.g.o.   `git format-patch` and `git send-email` are good
tools for this purpose.

>> also, before getting the ebuilds into the tree, i would really like to
>> see an automated test to verify that the content of our jar files
>> matches the content of the maven repo jar files. without this, we might
>> get issues because of missing resources and maybe even other causes.

> Currently, I do not have a clear way of reaching the goal...
> A preliminary thought is to maintain a modified version of java-pkg-simple.
> The modified eclass will automatically compare the files in postinst phase.
> I will try to work out it this week.

I think src_test() is better than pkg_postinst().

Benda

Re: [gentoo-soc] Weekly Report: Big Data Infrastructure and Maven Overlay in Week 5

[Miroslav Šulc]

Dne 06. 07. 20 v 11:58 Zhang Zongyu napsal(a):
> Hello,
>
>> i just would like to bring to your attention the content of
>> ~/.java-ebuilder/cache. it contains all packages that are based at least
>> on one of java eclasses, and those eclasses are also listed at the end
>> of each row. might help you to find all packages that are java packages,
>> even outside dev-java/ category.
> Thanks for the notice. Previously I believed that those categories
> other than dev-java/ and java-virtuals/* are not likely to exist in
> the database of Maven, so I ignored them last week. I will fill in the
> blank of non-dev-java packages this week.
i suppose dev-java/ should contain mostly libraries, specific tools 
written in java can be found in any topic category, like dev-db/ for 
example.
>>>     https://github.com/gentoo/java-ebuilder/blob/master/scripts/meta.sh
>> this seems to be a really clever idea :-)
> This helped me a lot. Last week, I started with the preliminary cache
> generated by meta.sh.
>
>> generally, using `git commit -s ...` adds the required line on new
>> commits. you can also rebase your commit messages ( `git rebase -i
>> c3b197f0e69c06f58f982f8d8dd16c88e2d5d518`) and push the commits again
>> afaik.
> Sorry for the mistake I have made, and I will rebase the repository.
not a problem at all, this is easily fixable :-)
>> javax/xml/bind is part of jdk:8 but was dropped later and is not present
>> in jdk:11 anymore which causes some packages to fail to compile against
>> jdk:11 (you can easily check with something like this: `for jdk in
>> icedtea:8 openjdk:8 openjdk:11; do echo; echo $jdk; for jar in $(query
>> files $jdk | grep -E ".*\.jar$"); do echo $jar; unzip -l $jar | grep
>> javax/xml/bind; done; done`). you have to have those jdk installed
>> before running this command.
> Thanks for the information. So it seems that I do not need concern
> about jaxb, and I can just set the proper classpath to make use of
> JAXB stuffs instead. Maybe I will need to add a variable (e.g.
> JAVA_NEEDS_RUNTIME) to control it.
well, imo we should behave like there is no support for jaxb in jdk/jre 
at all. for example, you can force a package to be compiled using jdk:8 
but user might want to run it using jdk/jre:11. in that case the package 
will compile fine but it won't run because of missing classes. if we 
depend on jaxb then it will run using jdk/jre:11 and it should also run 
using jdk/jre:8. it's definitely preferred to be able to also compile 
packages using jdk:11. that would also make (one day) possible to unmask 
and stabilize jdk:11.
>> do these work with java-pkg-simple.eclass as is in the tree or it needs
>> your updated eclass?
> The class is modified by me. It can be found here[1]. The major
> differences are the way to process JAVA_*_CLASSPATH variables and a
> new variable called "JAVA_NEEDS_TOOLS".

just make sure you do not break existing packages that use this eclass. 
in other words, your modifications should not affect existing packages 
in a way that would break them, but can bring enhancements etc. you can 
even use if...else...fi to introduce new stuff without breaking old one. 
anyway, this is just a general note, not a statement about your changes 
in the eclass :-)

wrt java-pkg-simple.eclass enhancements, you might find this bug useful 
aswell: https://bugs.gentoo.org/564158

>> getting the changes in the eclass into the main tree requires review and approval by devs.
> Is it correct that I should trigger the review via bugs.gentoo.org?

here you can find the info for eclasses: 
https://devmanual.gentoo.org/eclass-writing/index.html

generally, https://devmanual.gentoo.org/ is a good resource for gentoo 
devs :-)

>> also, before getting the ebuilds into the tree, i would really like to
>> see an automated test to verify that the content of our jar files
>> matches the content of the maven repo jar files. without this, we might
>> get issues because of missing resources and maybe even other causes.
> Currently, I do not have a clear way of reaching the goal...
> A preliminary thought is to maintain a modified version of java-pkg-simple.
> The modified eclass will automatically compare the files in postinst phase.
> I will try to work out it this week.
you might also consider making it a part of the official 
java-pkg-simple.eclass and implement it in the test phase. just make 
sure you apply this test only to maven based packages and not to all 
packages built using java-pkg-simple.eclass.
>> one more note. i noticed in your overlay you have only the ebuild files
>> but the rest of the required files are missing. generally, you need to
>> provide metadata.xml file for each package and need to run `repoman
>> manifest` for each package. also, you have to run `repoman full` and
>> also can run `pkgcheck scan` for each package to catch all issues that
>> can be caught this way.
> Got you, I will add the missing files.
>
> Regards,
> Zhang Zongyu
>
> [1] https://github.com/6-6-6/spark-overlay/blob/master/eclass/java-pkg-simple.eclass
>
miroslav

Re: [gentoo-soc] Weekly Report: Big Data Infrastructure and Maven Overlay in Week 5

[Miroslav Šulc]

Dne 06. 07. 20 v 11:58 Zhang Zongyu napsal(a):
> Hello,
>
> ....
>> generally, using `git commit -s ...` adds the required line on new
>> commits. you can also rebase your commit messages ( `git rebase -i
>> c3b197f0e69c06f58f982f8d8dd16c88e2d5d518`) and push the commits again
>> afaik.
> Sorry for the mistake I have made, and I will rebase the repository.
thanks, merged :-)
> ...
> Regards,
> Zhang Zongyu
>
> [1] https://github.com/6-6-6/spark-overlay/blob/master/eclass/java-pkg-simple.eclass


miroslav