[gentoo-java] Java ideas for Summer of Code

[gentoo-java] Java ideas for Summer of Code

[Joshua Nichols]

(The following is shamelessly yanked from the blog post I just made)

As has been previously announced, we are officially part of Google's 
Summer of Code. I've decided to step up to be a mentor, and in 
particular, to mentor projects related to Java on Linux.

A few ideas follow. The 1a and 1b are listed already on our SoC page to 
an extent, but the idea is elaborated further here. If you plan on 
proposing one of these ideas, I really would expect you to even further 
elaborate.

*1a)*
/Summary:/

Build most of our Java packages with free (libre) virtual machines and 
free implementations of public APIs.

/Background:/

Currently, we really only support using a proprietary virtual machine 
(ie sun, blackdown, ibm, etc), because packages are likely to fail for 
various reason with the open ones.

For many open apis, such as javamail, java activation framework, etc, we 
have binary packages of Sun's proprietary implementations. In a number 
of cases, there are open implementations. However, our packages compile 
against and run using the proprietary implementations.

For reasons why one would want to be using Free Java, see the article on 
the Java trap <http://www.gnu.org/philosophy/java-trap.html>.

As for a practical reason, use of proprietary packages from Sun and IBM 
can annoying for the end user, because in both cases, it requires 
placing a fetch restriction on the distfiles. To the end user, this 
means that an emerge gets halted until they agree to a license, and 
download the files.

/Goals:/

    * Build/run all/most packages using free virtual machines
    * Build/run all/most packages against free implementations of public
      APIs
    * Might want to target specific big name packages, like eclipse,
      azureus, tomcat.
    * Be able to select between different implementations of the same apis

/Tasks:/

    * Work with upstream of packages that use propertary classes from
      the virtual machine (ie com.sun.*, sun.*)
    * Work with upstream of virtual machines and packages when packages
      don't compile or run with using free java
    * Find and package open implementations of public APIs

/Hurdles:/

    * Lots of Java packages (300+). It is unknown how many will need to
      be patched.
    * Upstream might not care about free java
    * Might not be open implementations of all APIs

*1b)*

/Summary:/
Native compiling with gcj

/Background:/

GNU Compiler for Java (gcj) allows for compiling Java code to native 
machine code, as opposed to Java bytecode. Fedora/Redhat has done this 
to an extent for some time. It'd be great to be able to do this under 
Gentoo as well.

/Tasks:/

    * Create a 'JDK' which uses gcj as a backend. In this case, JDK
      really means providing a javac, javadoc, jar, etc
    * Have a way of keeping track of native bytecode
    * Integrate methods for native compiling into the current build
      system for building Java packages on Gentoo (ie eclasses)
    * Target native compilation of high profile applications, ie azreus,
      eclipse, freemind

*2a)*

/Summary:/

Build maven <http://maven.apache.org> entirely from source.

/Background:/

We currently only provide binary packages of maven, due to the fact that 
it is a bootstrapping nightmare because of the monolithic nature of the 
build process. The build system, at its basic level, does the following:

    * Build the 'core' maven functionailty using ant
    * Build the 'core' maven functionality using maven
    * Build all the plugins using maven

The problem with building all the plugins at once is that there are a 
ton of plugins, with many, many dependencies. Because of these 
dependencies, it would make it impractical to have a monolithic package, 
which would be used to build other packages.

/Goals:/

    * Have modular packages for maven
    * Have one package per plugin
    * Have a 'core' maven package
    * Have a 'minimal' maven package, which is core + minimal packages
      for compiling, javadocs, and jarring
    * Have a 'full' maven package which is all the plugins
    * Would need to work out a way to make the build much more modular

*2b)*

/Summary/
Build packages using maven

/Background/
Many packages, particular Apache projects, have been switching over

The main issue is getting jars from a maven repository. The normal 
behavior of maven is to download dependencies from a maven mirror. First
of all, things at build time should not be using the network. Secondly, 
dependencies should be provided by jars from packages that have 
previously been emerged.

Another issue is that maven 2 repositories contain metadata 
(specifically, pom.xml files) about the packages contained within. This 
is normally fetched from the repository, but since we don't want to be 
pulling from there, the question remains open as to where the metadata 
is gotten from and how its stored on the system.

/Goals:/

    * Have a system to have maven use jars that we have built through
      portage
    * Update packages to use maven when possible. These should have fine
      grain control to the plugins that are available at build time.
      Packages should depend on the minimal install, and whatever extra
      plugins they might need, and only those should be available at
      build time.

/Possible solutions for dealing with maven repositories:/
A) Create a maven-repository like structure, and have it populated with 
jars from the system
B) maven (2.0 at least) has a mechanism for providing alternative ways 
of fetching dependencies. We could then create an alternative way that's 
backed by portage.
C) Find a way to tell maven not to fetch dependencies, and hope that 
there are the appropriate classes available on the classpath
-- 
gentoo-java@gentoo.org mailing list

Re: [gentoo-java] Java ideas for Summer of Code

[Gerry Smith]

[-- Attachment #1: Type: text/plain, Size: 57 bytes --]

Is JBoss 4.0 a high priority ?

Thanks

Gerry Smith

[-- Attachment #2: Type: text/html, Size: 69 bytes --]

Re: [gentoo-java] Java ideas for Summer of Code

[Joshua Nichols]

Gerry Smith wrote:
> Is JBoss 4.0 a high priority ?
>
> Thanks
>
> Gerry Smith
It's not a high priority on my radar at the moment at least. Although, I 
think getting it properly packaged (ie modular packages instead of one 
giant package) would be a Good Thing (tm), and I actually had intended 
to include it in the list, but didn't have the time to flesh it out.

I have a an initial 'working'' version of modular packages in our 
experimental overlay. Unfortunately, it's not complete, I'm not longer 
interested in using JBoss, and it's implementation is less than ideal. 
It works enough for what I use it for, as dependencies for hibernate.

If it's something your interested in, feel free to start thinking of a 
proposal (note: the deadline is Monday). I'm available on IRC if you'd 
like to bounce some ideas around.

Josh
-- 
gentoo-java@gentoo.org mailing list

Re: [gentoo-java] Java ideas for Summer of Code

[Joshua Nichols]

robert burrell donkin wrote:
> On 5/4/06, *Joshua Nichols* <nichoj@gentoo.org 
> <mailto:nichoj@gentoo.org>> wrote:
>
>     (The following is shamelessly yanked from the blog post I just made)
>
>
> <snip>
>
>     *1a)*
>     /Summary:/
>
>     Build most of our Java packages with free (libre) virtual machines and
>     free implementations of public APIs.
>
>     /Background:/
>
>     Currently, we really only support using a proprietary virtual machine
>     (ie sun, blackdown, ibm, etc), because packages are likely to fail for
>     various reason with the open ones.
>
>     For many open apis, such as javamail, java activation framework,
>     etc, we
>     have binary packages of Sun's proprietary implementations. In a
>     number
>     of cases, there are open implementations. However, our packages
>     compile
>     against and run using the proprietary implementations.
>
>     For reasons why one would want to be using Free Java, see the
>     article on
>     the Java trap < http://www.gnu.org/philosophy/java-trap.html>.
>
>     As for a practical reason, use of proprietary packages from Sun
>     and IBM
>     can annoying for the end user, because in both cases, it requires
>     placing a fetch restriction on the distfiles. To the end user, this
>     means that an emerge gets halted until they agree to a license, and
>     download the files.
>
>     /Goals:/
>
>         * Build/run all/most packages using free virtual machines
>         * Build/run all/most packages against free implementations of
>     public
>           APIs
>         * Might want to target specific big name packages, like eclipse,
>           azureus, tomcat.
>         * Be able to select between different implementations of the
>     same apis
>
>     /Tasks:/
>
>         * Work with upstream of packages that use propertary classes from
>           the virtual machine (ie com.sun.*, sun.*)
>         * Work with upstream of virtual machines and packages when
>     packages
>           don't compile or run with using free java
>         * Find and package open implementations of public APIs
>
>  
>
>     /Hurdles:/
>
>         * Lots of Java packages (300+). It is unknown how many will
>     need to
>           be patched.
>         * Upstream might not care about free java
>
>         * Might not be open implementations of all APIs
>
>
> why the emphasis on software libre?
>
And I think you're reading too much into what I said. By libre, I really 
mean open source.
> why exclude ASL'd libraries from the effort?
ASL? Do you mean the Apache license? In either case, I don't recall 
saying anything about excluding anything.
> why not open source java?
>
Uh, open source java is EXACTLY what I'm talking about :)
> the ASF has already made considerable progress in creating open source 
> clean room implementations for the major java specifications (harmony, 
> geronimo, ws). gump and harmony have been working closely with the 
> classpath and kaffe teams for a number of years now.
>
Open source java implementations have actually made tons of progress 
everywhere in the past year or so.
> - robert
>
In the future, be sure to reply on-list.


Josh

-- 
gentoo-java@gentoo.org mailing list

Re: [gentoo-java] Java ideas for Summer of Code

[robert burrell donkin]

[-- Attachment #1: Type: text/plain, Size: 1570 bytes --]

On 5/5/06, Joshua Nichols <nichoj@gentoo.org> wrote:
>
> robert burrell donkin wrote:
> > On 5/4/06, *Joshua Nichols* <nichoj@gentoo.org


<snip>

> why the emphasis on software libre?
> >
> And I think you're reading too much into what I said. By libre, I really
> mean open source.


good :-)

> why exclude ASL'd libraries from the effort?
> ASL? Do you mean the Apache license?


yep

In either case, I don't recall
> saying anything about excluding anything.


ASL2 is still incompatible with GPL2

free java means rebuilding the complete stack including replacements for all
ASL'd applications and libraries. so there's a little bit of politics in the
mix.

> why not open source java?
> >
> Uh, open source java is EXACTLY what I'm talking about :)


good :-)

gump (http://gump.apache.org/) builds and tests 700 projects from source
each day. IIRC the classpath team run gump. might be able to dig out an url
if you're interested in seeing the current level of progress.

in general, finding a way to generate EBUILDs from gump descriptors would
make available a lot of libraries and applications without extensive effort.
a maven EBUILD plugin would be even better.

apache has a lot of API implementations but they are scattered amongst
different projects. it would probably be generally a good thing if they were
more easily accessible. fixing that would also be a big gain for a small
amount of effort.

In the future, be sure to reply on-list.


 yeh sorry (mumbles something about gmail)

- robert

[-- Attachment #2: Type: text/html, Size: 2815 bytes --]

Re: [gentoo-java] Java ideas for Summer of Code

[Joshua Nichols]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

robert burrell donkin wrote:
> On 5/5/06, Joshua Nichols <nichoj@gentoo.org> wrote:
>>
>> robert burrell donkin wrote:
>> > On 5/4/06, *Joshua Nichols* <nichoj@gentoo.org
> 
> 
> <snip>
> 
>> why the emphasis on software libre?
>> >
>> And I think you're reading too much into what I said. By libre, I really
>> mean open source.
> 
> 
> good :-)
> 
>> why exclude ASL'd libraries from the effort?
>> ASL? Do you mean the Apache license?
> 
> 
> yep
> 
> In either case, I don't recall
>> saying anything about excluding anything.
> 
> 
> ASL2 is still incompatible with GPL2
> 
> free java means rebuilding the complete stack including replacements for
> all
> ASL'd applications and libraries. so there's a little bit of politics in
> the
> mix.
> 

So you're saying that 'free java' refers to java licened under the GPL?
If that's what you're stuck up on, I guess I should be saying open
source java (which I could have sworn I've been saying from the start)


>> why not open source java?
>> >
>> Uh, open source java is EXACTLY what I'm talking about :)
> 
> 
> good :-)
> 
> gump (http://gump.apache.org/) builds and tests 700 projects from source
> each day. IIRC the classpath team run gump. might be able to dig out an url
> if you're interested in seeing the current level of progress.
> 
> in general, finding a way to generate EBUILDs from gump descriptors would
> make available a lot of libraries and applications without extensive
> effort.
> a maven EBUILD plugin would be even better.

ebuild isn't an acronym, so you don't need to capitalize it all ;)

But having a way to generate ebuilds from gump descriptor or maven
project files would be intersting and all well and good... but someone
would actually have to do the work to put it together. That person isn't
likely me in the foreseeable future, unless someone has a deal on cloning.

> 
> apache has a lot of API implementations but they are scattered amongst
> different projects. it would probably be generally a good thing if they
> were
> more easily accessible. fixing that would also be a big gain for a small
> amount of effort.

'small amount of effort' for whom? Adding a lot of packages doesn't
particularly sound like a small effort to me. Another hurdle is that
many projects are switching to maven for building, and as I describe in
the project ideas, we aren't quiet ready to handle using it to build.


- - Josh
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFEXfIc8ATTzZyw6sMRAkuSAJ9bpAOu1y+FA2cnWtgLhxn86JcbLQCfWO2m
sFtBxnvg/1o1/BahT839u7w=
=7YRB
-----END PGP SIGNATURE-----
-- 
gentoo-java@gentoo.org mailing list

[gentoo-java] Java ideas for Summer of Code

[robert burrell donkin]

[-- Attachment #1: Type: text/plain, Size: 3058 bytes --]

one day i'll remember about gmail :-<

---------- Forwarded message ----------
From: robert burrell donkin <robertburrelldonkin@gmail.com>
Date: May 8, 2006 11:35 PM
Subject: Re: [gentoo-java] Java ideas for Summer of Code
To: Joshua Nichols <nichoj@gentoo.org>

On 5/7/06, Joshua Nichols <nichoj@gentoo.org> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> robert burrell donkin wrote:
> > On 5/5/06, Joshua Nichols <nichoj@gentoo.org> wrote:
> >>
> >> robert burrell donkin wrote:
> >> > On 5/4/06, *Joshua Nichols* <nichoj@gentoo.org


<snip>

>> why not open source java?
> >> >
> >> Uh, open source java is EXACTLY what I'm talking about :)
> >
> >
> > good :-)
> >
> > gump (http://gump.apache.org/ ) builds and tests 700 projects from
> source
> > each day. IIRC the classpath team run gump. might be able to dig out an
> url
> > if you're interested in seeing the current level of progress.
> >
> > in general, finding a way to generate EBUILDs from gump descriptors
> would
> > make available a lot of libraries and applications without extensive
> > effort.
> > a maven EBUILD plugin would be even better.
>
> ebuild isn't an acronym, so you don't need to capitalize it all ;)


looks better as EBUILD (especially as i don't know where the shift key is ;)


maybe need to fake an acronym for marketing purposes :)

But having a way to generate ebuilds from gump descriptor or maven
> project files would be intersting and all well and good... but someone
> would actually have to do the work to put it together. That person isn't
> likely me in the foreseeable future, unless someone has a deal on cloning.
>


just speculating out load ATM  (on the generation, not the cloning)

> apache has a lot of API implementations but they are scattered amongst
> > different projects. it would probably be generally a good thing if they
> > were
> > more easily accessible. fixing that would also be a big gain for a small
>
> > amount of effort.
>
> 'small amount of effort' for whom?


sorry - ASF hat on there :-/

actually providing an index of all available implementations in one place is
something that the ASF should really do. make things a little easier for
those folks downstream consumers looking for clean room implementations and
a lot easier for users. there's a lot of activity around DOAP ATM and this
fits in well.

Adding a lot of packages doesn't
> particularly sound like a small effort to me.

Another hurdle is that
> many projects are switching to maven for building, and as I describe in
> the project ideas, we aren't quiet ready to handle using it to build.


i'm not sure i understand why mavenization is such a hurdle. nearly anything
built with maven can be built with ant. (maven can generate an ant build
from the POM.) projects using techniques such as generation using custom
maven plugins aren't so easy but that's a definite minority.

(but that's not to say that building maven2 isn't a worthy exercise)

- robert

[-- Attachment #2: Type: text/html, Size: 5366 bytes --]

Re: [gentoo-java] Java ideas for Summer of Code

[Joshua Nichols]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

robert burrell donkin wrote:
> one day i'll remember about gmail :-<
> 
> ---------- Forwarded message ----------
> From: robert burrell donkin <robertburrelldonkin@gmail.com>
> Date: May 8, 2006 11:35 PM
> Subject: Re: [gentoo-java] Java ideas for Summer of Code
> To: Joshua Nichols <nichoj@gentoo.org>
> 
> On 5/7/06, Joshua Nichols <nichoj@gentoo.org> wrote:
> 
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> robert burrell donkin wrote:
>> > On 5/5/06, Joshua Nichols <nichoj@gentoo.org> wrote:
>> >>
>> >> robert burrell donkin wrote:
>> >> > On 5/4/06, *Joshua Nichols* <nichoj@gentoo.org
> 
> 
> <snip>
> 
>>> why not open source java?
>> >> >
>> >> Uh, open source java is EXACTLY what I'm talking about :)
>> >
>> >
>> > good :-)
>> >
>> > gump (http://gump.apache.org/ ) builds and tests 700 projects from
>> source
>> > each day. IIRC the classpath team run gump. might be able to dig out an
>> url
>> > if you're interested in seeing the current level of progress.
>> >
>> > in general, finding a way to generate EBUILDs from gump descriptors
>> would
>> > make available a lot of libraries and applications without extensive
>> > effort.
>> > a maven EBUILD plugin would be even better.
>>
>> ebuild isn't an acronym, so you don't need to capitalize it all ;)
> 
> 
> looks better as EBUILD (especially as i don't know where the shift key
> is ;)
> 
> 
> maybe need to fake an acronym for marketing purposes :)
> 
> But having a way to generate ebuilds from gump descriptor or maven
>> project files would be intersting and all well and good... but someone
>> would actually have to do the work to put it together. That person isn't
>> likely me in the foreseeable future, unless someone has a deal on
>> cloning.
>>
> 
> 
> just speculating out load ATM  (on the generation, not the cloning)
> 
>> apache has a lot of API implementations but they are scattered amongst
>> > different projects. it would probably be generally a good thing if they
>> > were
>> > more easily accessible. fixing that would also be a big gain for a
>> small
>>
>> > amount of effort.
>>
>> 'small amount of effort' for whom?
> 
> 
> sorry - ASF hat on there :-/
> 
> actually providing an index of all available implementations in one
> place is
> something that the ASF should really do. make things a little easier for
> those folks downstream consumers looking for clean room implementations and
> a lot easier for users. there's a lot of activity around DOAP ATM and this
> fits in well.

That's actually a really good idea. You should suggest it to somewhere
that can do something about it :) Maybe their bugzilla?

> 
> Adding a lot of packages doesn't
>> particularly sound like a small effort to me.
> 
> Another hurdle is that
>> many projects are switching to maven for building, and as I describe in
>> the project ideas, we aren't quiet ready to handle using it to build.
> 
> 
> i'm not sure i understand why mavenization is such a hurdle. nearly
> anything
> built with maven can be built with ant. (maven can generate an ant build
> from the POM.) projects using techniques such as generation using custom
> maven plugins aren't so easy but that's a definite minority.

That's mostly true for maven1. As you said, you can generate build.xml
files, and we can package stuff using this build.xml. However, for
sophisticated projects (ie multiprojects), build.xml which perfectly
mirror the maven build isn't always possible.

> 
> (but that's not to say that building maven2 isn't a worthy exercise)

Seriously though, it's quite an epic exercise. I've attempted it several
times, and obviously wasn't successful enough, or the point would be moot.

> - robert
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFEX+Jn8ATTzZyw6sMRAvnmAJ4wohQFKayhSI4HUUee7gwNnl7kkACeIZgg
4LGbGE0i31sqVXwZo2HOTFQ=
=KWWs
-----END PGP SIGNATURE-----
-- 
gentoo-java@gentoo.org mailing list