[gentoo-soc] Weekly Report: Big Data Infrastructure and Maven Overlay in Week 4

[gentoo-soc] Weekly Report: Big Data Infrastructure and Maven Overlay in Week 4

[Zhang Zongyu]

Hello,

This week I began to install the overlay generated by java-ebuilder.
But I ran into some problems.

They are mainly related to pom files, the metadata of maven artifacts.
Some pom files do not declare their real dependencies, while some
others form into a circular dependency. The problem is really annoying.
And after some discussion with Benda and Miroslav, I am going to manually
write them a proper ebuild to avoid the problem.

Miroslav also reviewed my submitted code, and shed some light on what
the current code is lacking.

Next week, I am going to bump the version of some packages from dev-java/*
and ensure the overlay will make use of the Gentoo repository.
Besides, to meet the requirements of the project, I will continue to develop
on java-ebuilder and java-pkg-simple.eclass.

Regards
Zhang Zongyu

[1] my fork of java-ebuilder
https://github.com/6-6-6/java-ebuilder

Re: [gentoo-soc] Weekly Report: Big Data Infrastructure and Maven Overlay in Week 4

[Ján Zahornadský]

On 28/06/2020 16:23, Zhang Zongyu wrote:
> This week I began to install the overlay generated by java-ebuilder.
> But I ran into some problems.
> 
> They are mainly related to pom files, the metadata of maven artifacts.
> Some pom files do not declare their real dependencies, while some
> others form into a circular dependency. The problem is really annoying.
> And after some discussion with Benda and Miroslav, I am going to manually
> write them a proper ebuild to avoid the problem.

Are you _sure_ that's the case? These projects are using Maven to build, 
test and upload artifacts to Maven central, how can it not compile?

Did you check out a stable commit? Did you set the proper Maven profile? 
Are you invoking Maven the same way upstream is? Please elaborate.


All the best,
Jan

[gentoo-soc] Re: Weekly Report: Big Data Infrastructure and Maven Overlay in Week 4

[Zhang Zongyu]

[-- Attachment #1: Type: text/plain, Size: 1978 bytes --]

Hello,

> Are you _sure_ that's the case?

I would say Yes. The circular dependency originates from pom files, but
the problem is that I cannot find a portage equivalent of some maven
features
(that is "exclusion" exactly).

> These projects are using Maven to build, test and upload artifacts
> to Maven central, how can it not compile?

This is weird. I encountered net.sf.kxml:kxml2:2.3.0, which does
not declare any dependencies.
But it turns out to depend on xmlpull:xmlpull, I need to manually
add it and make net.sf.kxml:kxml2:2.3.0 able to compile.

> Did you check out a stable commit?

I am sorry that I am not sure what the stable commit stands for.
Does that mean Maven artifacts always have a stable commit?
Could you please tell me where can I find it?

> Did you set the proper Maven profile?

I do not set a Maven profile, and I believe it should be the default one.

> Are you invoking Maven the same way upstream is?

No, I am trying to translate pom into ebuild, which is an important
part of my GSoC.

> Please elaborate.

Here is a set of pom files leading to circular deps:
org.codehaus.groovy:groovy-all:2.5.12 depends on
org.codehaus.groovy:groovy:2.5.12,
while org.codehaus.groovy:groovy:2.5.12 depends on
org.codehaus.gpars:gpars:1.2.1,
while org.codehaus.gpars:gpars:1.2.1 depends
on org.codehaus.groovy:groovy-all:2.5.12.

But in pom of groovy, it excludes groovy-all with the "exclusion" attribute.
It means that the gpars that groovy depends on is not depending on
groovy-all now.
So the circular dep gets broken in Maven.

The problem is that portage cannot deal with things like "exclusion", so we
cannot avoid the circumstance of circular deps.

Do you have any suggestions? And Thank you in advance.

Regards,
Zhang Zongyu

[1] an Overlay of ebuild files from Maven artifacts, if you are interested
in
https://github.com/6-6-6/spark-overlay
[2] java-ebuilder which translates pom files into ebuild files.
https://github.com/6-6-6/java-ebuilder

[-- Attachment #2: Type: text/html, Size: 2823 bytes --]

Re: [gentoo-soc] Re: Weekly Report: Big Data Infrastructure and Maven Overlay in Week 4

[EBo]

On Jun 28 2020 6:56 PM, Zhang Zongyu wrote:
> Hello,
>
>> Are you _sure_ that's the case?
>
> I would say Yes. The circular dependency originates from pom files, 
> but
> the problem is that I cannot find a portage equivalent of some maven
> features
> (that is "exclusion" exactly).
>
>> These projects are using Maven to build, test and upload artifacts
>> to Maven central, how can it not compile?
>
> This is weird. I encountered net.sf.kxml:kxml2:2.3.0, which does
> not declare any dependencies.
> But it turns out to depend on xmlpull:xmlpull, I need to manually
> add it and make net.sf.kxml:kxml2:2.3.0 able to compile.

Sorry if it is inappropriate to interject here, but there are 'use' 
flags and dependencies defined in the profile.  My guess it is in there 
some place.  On my system (with set to 
default/linux/amd64/17.0/desktop/plasma) and it includes a lot of things 
like X.  Maybe this will help.

>> Did you check out a stable commit?
>
> I am sorry that I am not sure what the stable commit stands for.
> Does that mean Maven artifacts always have a stable commit?
> Could you please tell me where can I find it?
>
>> Did you set the proper Maven profile?
>
> I do not set a Maven profile, and I believe it should be the default 
> one.
>
>> Are you invoking Maven the same way upstream is?
>
> No, I am trying to translate pom into ebuild, which is an important
> part of my GSoC.
>
>> Please elaborate.
>
> Here is a set of pom files leading to circular deps:
> org.codehaus.groovy:groovy-all:2.5.12 depends on
> org.codehaus.groovy:groovy:2.5.12,
> while org.codehaus.groovy:groovy:2.5.12 depends on
> org.codehaus.gpars:gpars:1.2.1,
> while org.codehaus.gpars:gpars:1.2.1 depends
> on org.codehaus.groovy:groovy-all:2.5.12.
>
> But in pom of groovy, it excludes groovy-all with the "exclusion" 
> attribute.
> It means that the gpars that groovy depends on is not depending on
> groovy-all now.
> So the circular dep gets broken in Maven.
>
> The problem is that portage cannot deal with things like "exclusion", 
> so we
> cannot avoid the circumstance of circular deps.
>
> Do you have any suggestions? And Thank you in advance.
>
> Regards,
> Zhang Zongyu
>
> [1] an Overlay of ebuild files from Maven artifacts, if you are 
> interested
> in
> https://github.com/6-6-6/spark-overlay
> [2] java-ebuilder which translates pom files into ebuild files.
> https://github.com/6-6-6/java-ebuilder

Re: [gentoo-soc] Weekly Report: Big Data Infrastructure and Maven Overlay in Week 4

[Benda Xu]

Hi Zongyu,

Zhang Zongyu <zzy2529420793@gmail.com> writes:

> This week I began to install the overlay generated by java-ebuilder.
> But I ran into some problems.
>
> They are mainly related to pom files, the metadata of maven artifacts.
> Some pom files do not declare their real dependencies, while some
> others form into a circular dependency. The problem is really
> annoying.  And after some discussion with Benda and Miroslav, I am
> going to manually write them a proper ebuild to avoid the problem.
>
> Miroslav also reviewed my submitted code, and shed some light on what
> the current code is lacking.
>
> Next week, I am going to bump the version of some packages from
> dev-java/* and ensure the overlay will make use of the Gentoo
> repository.  Besides, to meet the requirements of the project, I will
> continue to develop on java-ebuilder and java-pkg-simple.eclass.

Your report shows that you are on track to the end goal of this
project.  Carry on.

Cheers,
Benda