[gentoo-java] ideas for packages which use maven to build

[gentoo-java] ideas for packages which use maven to build

[Joshua Nichols]

I have been thinking recently, and I think I came up with a viable
solution for packaging this that use maven for building.

The first problem is that maven will attempt to download jar
dependencies from a remote repository. This can be avoided by calling
maven with -o, for offline mode.

This leads to the question of where, then, to get the jars from. I had
first thought at build time, we could populate a local repository with
symlinks to jars that we provide from packages. This would work, and
could be automated to some extent, but I think it would be tedious to
maintain a list of jars that each package needs. 

So, today I came up with the idea of maintaining a package, like maven-
repo or something, that basically installs a local repository populated
with every jar from the repository... except they would actually be
symlinks to the jars provided by a package. For example:

/usr/share/maven-repo/commons-validator/jars/commons-validator-1.0.2.jar
--> /usr/share/commons-validator/lib/commons-validator.jar

This package would need a good amount of maintenance though, but I still
think it would be the simplest way to provide a maven repository for
building packages.

The next thing would be an eclass to support maven-built packages, like
java-maven.eclass perhaps. This would add a dependency of the most up-
to-date version of maven-repo (~dev-java/maven-repo), and a wrapper for
maven to force offline mode and to use our local repo
at /usr/share/maven-repo/, ie:

emaven() {
	maven -Dmaven.repo.local=/usr/share/maven-repo -o "$@" || die "emaven
failed"
}

Additionally, it could do some sanity checks, by parsing the project.xml
and:
	make sure that all of its dependencies are in /usr/share/maven-repo
	make sure that the symlinks aren't broken, ie the package they point to
is installed
	make sure the packages used in maven-repo are in DEPEND/RDEPEND



Once we have this framework in place, then we can also begin getting
maven built from source.


Regards,

Josh

-- 
gentoo-java@gentoo.org mailing list

Re: [gentoo-java] ideas for packages which use maven to build

[Karl Trygve Kalleberg]

Joshua Nichols wrote:
> I have been thinking recently, and I think I came up with a viable
> solution for packaging this that use maven for building.
> 
> The first problem is that maven will attempt to download jar
> dependencies from a remote repository. This can be avoided by calling
> maven with -o, for offline mode.
> 
> This leads to the question of where, then, to get the jars from. I had
> first thought at build time, we could populate a local repository with
> symlinks to jars that we provide from packages. This would work, and
> could be automated to some extent, but I think it would be tedious to
> maintain a list of jars that each package needs. 

Even in the face of such tedium, there is one thing this gives us that 
your suggestion does not: the ability to actually check for hidden 
dependencies.

If we create a local, minimal .jar environment for each maven-built 
package, we know exactly which jars (and therefore which ebuilds) it 
depends on.

In the case where maven itself goes into the system and looks around for 
.jars that are there, we may quickly end up with it depending on stuff 
we didn't see.

This happens with configure scripts and C/C++ applications all the time. 
Most of the time, we can do ./configure --enable/disable, but sometimes 
flipping this switch has no effect: it will still automatically 
autodetect stuff.


Before abandoning the idea of ebuild-local maven repos, I think we 
should be very certain that we're not opening up this Pandora's box of 
bad mojo.

That being said, a system-wide maven repo that users can avail 
themselves of would be great! And AFAICT, it would only need minor 
tweaks to the java-*.eclasses.


Cheers,

-- Karl T
-- 
gentoo-java@gentoo.org mailing list

Re: [gentoo-java] ideas for packages which use maven to build

[Greg Tassone]

[-- Attachment #1: Type: text/plain, Size: 1684 bytes --]

On Wed, 2005-12-14 at 19:39 +0100, Karl Trygve Kalleberg wrote:
> Joshua Nichols wrote:
< SNIPPED >
> > This leads to the question of where, then, to get the jars from. I had
> > first thought at build time, we could populate a local repository with
> > symlinks to jars that we provide from packages. This would work, and
> > could be automated to some extent, but I think it would be tedious to
> > maintain a list of jars that each package needs. 
> 
> Even in the face of such tedium, there is one thing this gives us that 
> your suggestion does not: the ability to actually check for hidden 
> dependencies.

Karl,
I'm a little confused by the above statement.  Are you agreeing with
Josh, or disagreeing?  In other words, are you in favor of the tedium
because it gives the ability to check for hidden dependencies?


> If we create a local, minimal .jar environment for each maven-built 
> package, we know exactly which jars (and therefore which ebuilds) it 
> depends on.

This sounds like what Josh was suggesting in one way, but I can't be
sure if you're saying that you like his idea or rather the idea of
making smaller, separate repositories.  Just wondering.


> In the case where maven itself goes into the system and looks around for 
> .jars that are there, we may quickly end up with it depending on stuff 
> we didn't see.

I agree, although this really is an upstream problem I think.  If they
aren't even properly documenting what libraries their application
depends on, I see that as a bigger problem we shouldn't necessarily try
to fix for them.  I understand that this does happen in the real world,
however. :-)

Greg

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-java] ideas for packages which use maven to build

[Petteri Räty]

[-- Attachment #1: Type: text/plain, Size: 1133 bytes --]

Greg Tassone wrote:
> On Wed, 2005-12-14 at 19:39 +0100, Karl Trygve Kalleberg wrote:
> 
> Karl,
> I'm a little confused by the above statement.  Are you agreeing with
> Josh, or disagreeing?  In other words, are you in favor of the tedium
> because it gives the ability to check for hidden dependencies?
> 

As far as I understood it a global repo enables hidden dependencies, but
we can go around this by just populating the repo with broken symlinks.
They don't work unless the package is actually installed and as such we
don't have broken depencies. Cleaning every java package away and
starting from stratch is any way something we developers should do when
committing new packages.

> 
> I agree, although this really is an upstream problem I think.  If they
> aren't even properly documenting what libraries their application
> depends on, I see that as a bigger problem we shouldn't necessarily try
> to fix for them.  I understand that this does happen in the real world,
> however. :-)
> 

If I gave you a penny everytime upstream has good documentation on their
dependencies, you wouldn't get too many.

Regards,
Petteri

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 256 bytes --]

[gentoo-java] Re: ideas for packages which use maven to build

[Jörg Schaible]

Hi Josh,

[snip]

well, the idea of symlinks is nice, but not new:
http://thread.gmane.org/gmane.linux.gentoo.java/43

:)

- Jörg

-- 
gentoo-java@gentoo.org mailing list

Re: [gentoo-java] ideas for packages which use maven to build

[Joshua Nichols]

On Wed, 2005-12-14 at 19:39 +0100, Karl Trygve Kalleberg wrote:
> Joshua Nichols wrote:
> > I have been thinking recently, and I think I came up with a viable
> > solution for packaging this that use maven for building.
> > 
> > The first problem is that maven will attempt to download jar
> > dependencies from a remote repository. This can be avoided by calling
> > maven with -o, for offline mode.
> > 
> > This leads to the question of where, then, to get the jars from. I had
> > first thought at build time, we could populate a local repository with
> > symlinks to jars that we provide from packages. This would work, and
> > could be automated to some extent, but I think it would be tedious to
> > maintain a list of jars that each package needs. 
> 
> Even in the face of such tedium, there is one thing this gives us that 
> your suggestion does not: the ability to actually check for hidden 
> dependencies.
> 
> If we create a local, minimal .jar environment for each maven-built 
> package, we know exactly which jars (and therefore which ebuilds) it 
> depends on.
> 
> In the case where maven itself goes into the system and looks around for 
> .jars that are there, we may quickly end up with it depending on stuff 
> we didn't see.
> 
> This happens with configure scripts and C/C++ applications all the time. 
> Most of the time, we can do ./configure --enable/disable, but sometimes 
> flipping this switch has no effect: it will still automatically 
> autodetect stuff.

> 
> Before abandoning the idea of ebuild-local maven repos, I think we 
> should be very certain that we're not opening up this Pandora's box of 
> bad mojo.
As I mentioned, we could do sanity checks. The project.xml of the
project lists EVERY dependency of the project. So, we could parse this
file and know what jars will be used from the local repo. We can check
if the jar's symlink is valid (the package is installed), and based on
where it points to, you can figure out what package it comes from, and
consequently check that this package is in DEPEND or RDEPEND.

> That being said, a system-wide maven repo that users can avail 
> themselves of would be great! And AFAICT, it would only need minor 
> tweaks to the java-*.eclasses.

I was just thinking, and there wouldn't be much needed to support both
standard java eclasses and the ones from axxo-overlay.

Maven has some properties for specifying which target/source to use, so
if it's on a system using axxo-overlay, it could use
java-pkg_get-{source,target}, and otherwise default to 1.4.


I do have an alternative approach (which I thought I had discussed on
the list at some point...).

Instead of having a local repo, there would be a file that contains
information about how a repo path maps to a jar from a package. For
example,

commons-logging/jars/commons-logging-api-1.0.8=commons-logging-api@commons-logging

>From here, there are a couple of paths... there could either be a
package that provides this file for every jar under the sun, this file
could live somewhere in /usr/portage. Alternatively, there could be a
file for each package/version... but this would add a bit of extra
maintenence.


At this point I think either approach would be sufficient. Right now,
I'm leaning towards the symlink-populated repo package. It has the added
benefit of providing a local repository that a user could use if they
wanted 

- Josh
-- 
gentoo-java@gentoo.org mailing list

Re: [gentoo-java] ideas for packages which use maven to build

[Joshua Nichols]

Jose Gonzalez Gomez wrote:

> 2005/12/15, Joshua Nichols <nichoj@gentoo.org 
> <mailto:nichoj@gentoo.org>>:
>
>     On Wed, 2005-12-14 at 19:39 +0100, Karl Trygve Kalleberg wrote:
>     > Joshua Nichols wrote:
>     > > I have been thinking recently, and I think I came up with a viable
>     > > solution for packaging this that use maven for building.
>     > >
>     > > The first problem is that maven will attempt to download jar
>     > > dependencies from a remote repository. This can be avoided by
>     calling
>     > > maven with -o, for offline mode.
>     > >
>     > > This leads to the question of where, then, to get the jars
>     from. I had
>     > > first thought at build time, we could populate a local
>     repository with
>     > > symlinks to jars that we provide from packages. This would
>     work, and
>     > > could be automated to some extent, but I think it would be
>     tedious to
>     > > maintain a list of jars that each package needs.
>     >
>     > Even in the face of such tedium, there is one thing this gives
>     us that
>     > your suggestion does not: the ability to actually check for hidden
>     > dependencies.
>     >
>     > If we create a local, minimal .jar environment for each maven-built
>     > package, we know exactly which jars (and therefore which
>     ebuilds) it
>     > depends on.
>     >
>     > In the case where maven itself goes into the system and looks
>     around for
>     > .jars that are there, we may quickly end up with it depending on
>     stuff
>     > we didn't see.
>     >
>     > This happens with configure scripts and C/C++ applications all
>     the time.
>     > Most of the time, we can do ./configure --enable/disable, but
>     sometimes
>     > flipping this switch has no effect: it will still automatically
>     > autodetect stuff.
>
>     >
>     > Before abandoning the idea of ebuild-local maven repos, I think we
>     > should be very certain that we're not opening up this Pandora's
>     box of
>     > bad mojo.
>     As I mentioned, we could do sanity checks. The project.xml of the
>     project lists EVERY dependency of the project. So, we could parse this
>     file and know what jars will be used from the local repo. We can check
>     if the jar's symlink is valid (the package is installed), and
>     based on
>     where it points to, you can figure out what package it comes from, and
>     consequently check that this package is in DEPEND or RDEPEND.
>
>
> This is true no more for maven 2.0. They have added transitive 
> dependencies to the project building, so now you only list in the 
> project descriptor ( pom.xml) the direct dependencies and their scope. 
> Maven takes care of reading project descriptors of those other 
> dependencies from the remote repository and builds the whole real list 
> of dependencies. So I think such an approach wouldn't work for Maven2.
>
You are correct. With some work though, I think it could work. We could 
mirror the pom.xml's in the gentoo maven repo. Then we could check the 
dependencies the immediate package has, at the least.

> Have you thought of patching the maven sources to intercept dependency 
> resolution and download and do something like calling emerge? I 
> haven't taken a look at maven sources, so I don't have any idea if 
> this is really feasible, or I'm just babbling.

It has been thought of in the past. Of course, just saying, 'hey, let's 
patch maven to do it!' is a lot easier than actually getting it working.

The place that we'd want to get control, if you would, is the point 
where dependencies get downloaded. Given that the maven infrastructure 
is all about plugins for extending functionality, you'd think it'd just 
be a matter of writing a plugin in. I've spoken with some of the folks 
in #maven on irc.codehaus.org, and apparently it wouldn't be quite that 
simple.


-- Josh
-- 
gentoo-java@gentoo.org mailing list