[gentoo-java] Java 1.5 migration plans

[gentoo-java] Java 1.5 migration plans

[Joshua Nichols]

I've been recently trying to work out a few possible plans for migrating 
axxo-overlay into the main portage tree. To give a quick synopsis, the 
migration of axxo-overlay needs to involve:

* A recent portage. We need features out of portage-2.1_pre1 to for all 
the new features of the overlay to work.
* A new version of java-config. Among other things, the new java-config 
uses symlinks, instead of environment variables, for controlling the 
current VM. This allows us do some magic like merge-time VM switching.
* A new version of javatoolkit. This new version provides tools for 
rewriting ant scripts, so that we can have sane -target/-source 
attributes of javac tasks.
* New eclasses. In addition to java.eclass (for installing Vms), 
java-pkg.eclass (for Java packages), and java-utils.eclass (provides 
some methods for checking what VM is being used), we have added 
java-ant.eclass (for Ant based packages) and java-pkg-opt.eclass (for 
packages which only have optional support for Java). Additionally, most 
functions that currently live in java-pkg have moved to java-utils.

Now, migration to the new overlay certainly has a few obstacles.
* The new eclasses require the new java-config and javatoolkit. So, this 
means that that they would need to be stabilized before we could migrate 
to the new eclasses. However, they are not quite 100% backward 
compatible. So therefore, it's a bit difficult to stabilize java-config 
and javatoolkit if we can't use them with the current system
* Can we migrate ~arch and arch seperately, or do they need to be done 
at the same time?

And so, I have to possible plans, with one plan being a slight variation 
of the other. I've divided them into separate phases. From here on, I'm 
going to be assuming you are familiar with the existing system and 
somewhat familiar with axxo-overlay.

Here's the first plan:
----------------------------------------

Phase 1 (can be concurrent with Phase 2)
Prereqs: everyone in java herd agrees on plan

1) bring over new java.eclass and jdk's and jre's
    - we should be able to just drop the in place of existing ones for 
stable and testing, but would need to do version bumps to make sure 
users get the new style env files.

2) bring over new java-config, javatoolkit
    - This seems to be backward compatible for most purposes.
should check for old style configs and die bitterly if they are around
NOTE: this may break a few packages. This happens because we've 
introduced wrapper scripts at /usr/bin/{java,javac,etc}. The problem is 
that some packages might try to be smart, and try to figure out 
JAVA_HOME from the location of java.

----------------------------------------

Phase 2 (can be concurrent with Phase 1):
Prereqs: everyone in java herd agrees on plan

1) bring over new java-utils.eclass
    - ensure packages which use it will continue work
    - don't bring over (because they need stable java-config)
        get-source/target, or anything that uses them
        new is-vm-version-sufficient and anything that uses them
        dolauncher
        java-pkg_die
        java-pkg_need
        
    
2) add new skeleton implementations of new eclasses: java-ant, java-pkg-opt:
    - for now should both inherit java-pkg and java-utils
    - ant_src_unpack for java-ant, and make all ant-based ebuilds use it

3) Implement skelton ejavac and eant in java-utils
    - should only call ant/javac
    - push back -source/-target stuff till
    - Update all ebuilds to use these instead of javac/ant

4) migrate junit use flag to test use flag, and perform tests in src_test

5) remove jikes use flag
    - should also put java-pkg_filter-compiler on things that are known 
not to
        - filter jikes on things that are known not to like jikes

-------------------------------------------

Phase 3
Prereqs: Phase 1 and 2, java-config and javatoolkit have been marked 
stable everywhere

1) Migrate everything from java-utils that was dropped during Phase 2
    Migrate real implemenation of ejavac, ie add appropriate -source/-target
    Migrate dolauncher support
    Migrate java-pkg_need / virtuals

1) Migrate rewriting build.xml's to java-ant

-------------------------------------------

Phase 4:
Prereqs: Phase 3, a version of portage has been marked stable on all 
platforms that have and use java

1) Migrate merge-time vm switching
2) Migrate java-pkg_die
----------------------------------------




And here's the second plan:
----------------------------------------

Phase 1 (can be concurrent with Phase 2)
Prereqs: everyone in java herd agrees on plan

1) Migrate new java.eclass and jdk's and jre's
    - we should be able to just drop the in place of existing ones for 
stable and testing, but would need to do version bumps to make sure 
users get the new style env files.

2) Introduce new java-config and javatoolkit
    - This seems to be backward compatible for most purposes.
        - should check for old style configs and die bitterly if they 
are around

----------------------------------------

Phase 2 (can be concurrent with Phase 1):
Prereqs: everyone in java herd agrees on plan

1) Add new skeleton implementations of new eclasses: java-ant, java-pkg-opt:
    - for now should both inherit java-pkg and java-utils
    - ant_src_unpack for java-ant, and make all ant-based ebuilds use it

2) Implement skelton ejavac and eant in java-utils
    - should only call ant/javac
    - push back -source/-target stuff till
    - Update all ebuilds to use these instead of javac/ant

3) Migrate junit use flag to test use flag, and perform tests in src_test

4) Remove jikes use flag
    - should also put java-pkg_filter-compiler on things that are known 
not to
        - filter jikes on things that are known not to like jikes

-------------------------------------------

Phase 3
Prereqs: Phase 1 and 2, java-config and javatoolkit have been marked 
stable everywhere

1) Migrate new java-utils.eclass
1) Migrate rewriting build.xml's to java-ant

-------------------------------------------

Phase 4:
Prereqs: Phase 3, a version of portage has been marked stable on all 
platforms that have and use java

1) Migrate merge-time vm switching
2) Migrate java-pkg_die



---------------------------------------------

The main difference between the two is how java-utils gets migrated. In 
the first scenario, we migrate everything which doesn't need the new 
java-config and javatoolkit, whereas in the second, we migrate all of 
java-utils at once after java-config and javatoolkit have been stabilized.

I currently prefer the first one. This allows us to get the new eclass 
functionality that we want, and current ebuilds should (mostly) continue 
to work with the old style functionality. By doing it this way, we can 
our time adopting the new functionality (not that I want this to take 
long, but we wouldn't have to immediately change everything).

I know Thomas had another plan, as well as some concerns with these plans.


Regards,

Josh
-- 
gentoo-java@gentoo.org mailing list

Re: [gentoo-java] Java 1.5 migration plans

[Thomas Matthijs]

> Phase 1 (can be concurrent with Phase 2)
> Prereqs: everyone in java herd agrees on plan
> 
> 1) bring over new java.eclass and jdk's and jre's
>    - we should be able to just drop the in place of existing ones for 
> stable and testing, but would need to do version bumps to make sure 
> users get the new style env files.
> 
> 2) bring over new java-config, javatoolkit
>    - This seems to be backward compatible for most purposes.
> should check for old style configs and die bitterly if they are around
> NOTE: this may break a few packages. This happens because we've 
> introduced wrapper scripts at /usr/bin/{java,javac,etc}. The problem is 
> that some packages might try to be smart, and try to figure out 
> JAVA_HOME from the location of java.

In the current state i believe it will break more.
Since it won't export any of the variables anymore.
So these changes would need to be introduced at the same times as the
ebuild fixes

> ----------------------------------------
> 
> Phase 2 (can be concurrent with Phase 1):
> Prereqs: everyone in java herd agrees on plan
> 
> 1) bring over new java-utils.eclass
>    - ensure packages which use it will continue work
>    - don't bring over (because they need stable java-config)
>        get-source/target, or anything that uses them
>        new is-vm-version-sufficient and anything that uses them
>        dolauncher
>        java-pkg_die
>        java-pkg_need

That doesn't leave alot :)

    
> 2) add new skeleton implementations of new eclasses: java-ant, java-pkg-opt:
>    - for now should both inherit java-pkg and java-utils
>    - ant_src_unpack for java-ant, and make all ant-based ebuilds use it
> 
> 3) Implement skelton ejavac and eant in java-utils
>    - should only call ant/javac
>    - push back -source/-target stuff till
>    - Update all ebuilds to use these instead of javac/ant
> 
> 4) migrate junit use flag to test use flag, and perform tests in src_test
> 
> 5) remove jikes use flag
>    - should also put java-pkg_filter-compiler on things that are known 
> not to
>        - filter jikes on things that are known not to like jikes

I'm still not happy with the filtering, It was discusses somewhere, but
i can't find it anymore. If everyone else things its a good idea, i'll
accept it

> 
> -------------------------------------------
> 
> Phase 3
> Prereqs: Phase 1 and 2, java-config and javatoolkit have been marked 
> stable everywhere
> 
> 1) Migrate everything from java-utils that was dropped during Phase 2
>    Migrate real implemenation of ejavac, ie add appropriate -source/-target
>    Migrate dolauncher support
>    Migrate java-pkg_need / virtuals
> 
> 1) Migrate rewriting build.xml's to java-ant
> 
> -------------------------------------------
> 
> Phase 4:
> Prereqs: Phase 3, a version of portage has been marked stable on all 
> platforms that have and use java
> 
> 1) Migrate merge-time vm switching
> 2) Migrate java-pkg_die
> ----------------------------------------

Its not clear to me when you want to do the ebuilds changes, and to arch
or ~arch. or both?

Since for everything to be merged, all ebuild updated need to be done,
and keyworded stable.

If let this be done over time, it'll be another few years before 1.5 can
unmasked. Which can only be done after phase 4. And all previous non
updated ebuilds removed.

> I know Thomas had another plan, as well as some concerns with these plans.

I don't really have a plan, i was trying to come up with a way that
would keep breakage to a minimum and keep everyone happy.

I wasn't able to come up with one.

I'd prefer to just merge everything in one shot, but some people may
completely block that (possibly rightly so).

- 1.5 Needed things, such as merge time vm switching and,
  -target/-source forcing
- And then the eant/launcher/cleanups
Which are both interlinked

Another option i have been thinking about would be:

Revert the new java-config to behave like the old one, or add some of
the new functionality in the old one (Not something i will do, so we
would need someone else for that). Making it 100% backwards compatible.
Rev bump all the jdk/jre with the environment files.

Update the eclasses in tree to include eant/ajavac/launcher, and the
target/source things.

Then first, update all ~arch ebuilds to use this newness. So it can be
tested. Say 2 weeks max.
Then apply the same to arch ebuilds, because we like consistency.

During this time frame, we also get aproval from the maintainers of
packages we need to touch. We'll need to write a doc explaining them why
the changes are needed. Can probbaly be scraped from our docs in README/*.

Then in a one shot operation, we add the new java-config with /usr/bin/*
symlinks, and no longer export the vars. Then update all the ebuilds to 
support merge time switching. arch and ~arch.
If we only add it to ~arch, then arch ebuilds will be broken for ~arch,
and vice versa.

After this, unmask 1.5. Sit back and get flamed about tomcat 5.5


Not sure if this is a good option, best i can come up and try to keep
everyone happy, may be forgetting something aswel.


-- 
 Thomas Matthijs (axxo, knu)  
-- 
gentoo-java@gentoo.org mailing list

Re: [gentoo-java] Java 1.5 migration plans

[Petteri Räty]

[-- Attachment #1: Type: text/plain, Size: 590 bytes --]

Thomas Matthijs wrote:
>>5) remove jikes use flag
>>   - should also put java-pkg_filter-compiler on things that are known 
>>not to
>>       - filter jikes on things that are known not to like jikes
> 
> 
> I'm still not happy with the filtering, It was discusses somewhere, but
> i can't find it anymore. If everyone else things its a good idea, i'll
> accept it
> 
> 

I think filtering is a useful thing to have. We might also think about
deprecating jikes support altogether as upstream seems dead and gcjx/ecj
will provide better alternatives in the future any way.

Regards,
Petteri

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 256 bytes --]

Re: [gentoo-java] Java 1.5 migration plans

[Martin von Gagern]

[-- Attachment #1: Type: text/plain, Size: 5334 bytes --]

Hi There!

I'm neither dev, nor did I have the time to look at all the affected
sources, so please be indulgent when I write nonsense.

After some late night IRC chat with axxo, I have another plan to
propose. I hope it might provide a way to smoothly change over, without
having to break major functionality in between, and with the possibility
to test things on ~arch first and to have changes on a per ebuild basis.


Phase 1)

The basic idea is to install wrappers in /usr/lib/java-wrappers or
something like that for some time. So no current ebuilds would break.

The location of java is determined by
1. PATH,
2. JAVA_HOME,
3. java-config output or
4. well known location (/usr/bin/).

1. and 2. are environment settings and can be modified by an eclass
(java-migrate.eclass?) so that new ebuilds could start using the new
run-java-tool wrappers.
For 3., those packages could depend on the new java-config.
4. is a problem in any case, so that new ebuilds can be expected to have
this solved or they won't work in any scenario.

I give some details on the compatibility of java-config below.
The idea is to have java-config handle new style symlinks and old style
env setting files at the same time with almost no work to be done.


Phase 2)

Now the wrappers are available, and ebuilds can at merge time switch to
using them instead of the old style VM settings. New ebuilds would use
an eclass to set the environment accordingly and to DEPEND on a recent
java-config. This could be done on a per ebuild basis, first for ~arch
and then for arch.

Packages that need it could DEPEND on javatoolkit as well.
New eclasses can be provided as long as they set correct DEPENDencies.


Phase 3)

All "complicated" ebuilds should be moved to using the new system. Those
are the ones that don't simply call ant but do some more elaborate
stuff. By a new version of java-config, the 20java file could be changed
again to make /usr/lib/java-wrappers the JAVA_HOME. Details for this are
below again. ant should continue to work, so still nothing breaks.
Now the new style is the default, and the old style is adapted to use
the new wrappers.

Phase 4)

java 1.5 can be unmasked for ~arch. Things that still don't work will be
noticed and can be fixed by using the merge time features of the new
system without much trouble.

Phase 5)

All ebuilds should be changed to using the wrappers and stabilized.
Now the wrappers could be symlinked, copied or moved to /usr/bin, the
env setting file 20java could be enoticed for removal.



Details on compatibility of java-config and run-java-tool:

Using the run-java-tool wrappers with java-config would require a recent
version of java-config. I suggest the java-config ebuild makes
/etc/env.d/20java a symlink to /etc/java-config/system-vm/env-settings
or some such, and every jdk and jde contains this settings file (which
is just a copy / symlink to the file listed in /etc/env.d/java if I got
this right). This can be done with the revision bumps that are needed
anyway.

So as the new java-config changes the system-vm symlink, automatically
the contents of 20java changes, and the usual procedure of env-update
and source profile makes this configuration change active, as it was
with java-config so far. The new java-config would without much
additional work reproduce the workings of the old one, in addition to
changing the symlinks in the new style.

The 20java file as it was when installing could be moved to some new dir
(e.g. /etc/java-config/old-vm), and the java tools identified by the
java file could be symlinked to this directory as well, so that this
directory can be used as a new style java vm installation exactly
representing the settings in effect when the java-config ebuild is
installed. When java-config is called to switch vm, the system-vm
symlink changes and the directory just described may be safely removed.

So to sum it up, here is how java-config installation should work:
1. mkdir -p /etc/java-config/old-vm/bin
2. mv /etc/env.d/20java /etc/java-config/old-vm/env-settings
3. ln -s ../java-config/system-vm/env-settings /etc/env.d/20java
4. ln -s old-vm /etc/java-config/system-vm
5. ln -s `java-config -c` /usr/java-config/old-vm/bin
   same for all the other java tools

3. makes sure that vm changes with the new java-config also change the
   environment settings the way the old config did.
2. makes the current settings available for old style env-update.
5. makes the currently selected vm available for the new style wrappers,
   it would have to use the old java-config.
4. sets this constructed configuration as current vm in the new style.

The user would need new VM packages with new environment setting files
only for switching VM, the current setting would continue to work under
the new java-config even without this update.



Details about the flow of indirection in Phase 3)

Before Phase 3 we had for old ebuilds:
1. env-update reads 20java
2. 20java points into system-vm
3. system-vm is set by java-config to point to jdk
4. jdk provides env settings
5. env settings provide access to java, javac and so on

Now we have for the same old ebuilds:
1. env-update reads 20java
2. 20java sets env settings
3. env settings provide access to wrappers
4. wrappers use system-vm
5. system-vm set by java-config


Greetings,
 Martin von Gagern

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 256 bytes --]

Re: [gentoo-java] Java 1.5 migration plans

[Thomas Matthijs]

* Martin von Gagern (Martin.vGagern@gmx.net) wrote:
> Hi There!
> 
> I'm neither dev, nor did I have the time to look at all the affected
> sources, so please be indulgent when I write nonsense.
> 
> After some late night IRC chat with axxo, I have another plan to
> propose. I hope it might provide a way to smoothly change over, without
> having to break major functionality in between, and with the possibility
> to test things on ~arch first and to have changes on a per ebuild basis.


Just wanted to point out, that I think this plan will work, and the best
one we have (so far).

-- 
 Thomas Matthijs (axxo, knu) 

-- 
gentoo-java@gentoo.org mailing list

Re: [gentoo-java] Java 1.5 migration plans

[Martin von Gagern]

[-- Attachment #1: Type: text/plain, Size: 663 bytes --]

Hello!

After some talk with Thomas and Petteri on IRC, I put my plan in SVN:

https://svn.gentooexperimental.org/svn/java/axxo-overlay/README/docs/migration.xml

So far it's basically a formatted version of my last email, with a few
addons. But this might change as things evolve.

Any questions, suggestions, improvements, and so on, please tell me.
I'll try to keep this plan up to date.

Sorry to those that use browsers which don't do xsl transformations on
the fly. Please use guide.xsl to get a html file from the xml. I'd
rather keep it just one file in the repository, so I won't have to worry
about keeping files in sync.

Greetings,
 Martin von Gagern

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 256 bytes --]

Re: [gentoo-java] Java 1.5 migration plans

[Martin von Gagern]

[-- Attachment #1: Type: text/plain, Size: 479 bytes --]

Martin von Gagern wrote:
> https://svn.gentooexperimental.org/svn/java/axxo-overlay/README/docs/migration.xml
> 
> So far it's basically a formatted version of my last email, with a few
> addons. But this might change as things evolve.

I added a list of little steps at the end of the guide. I don't know how
the assignment of devs to tasks usually works, but it would be great if
we could fill the list with assigned persons and estimated times.

Greetings,
 Martin von Gagern

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 256 bytes --]

Re: [gentoo-java] Java 1.5 migration plans

[Joshua Nichols]

Martin von Gagern wrote:
> I added a list of little steps at the end of the guide. I don't know how
> the assignment of devs to tasks usually works, but it would be great if
> we could fill the list with assigned persons and estimated times.
>
> Greetings,
>  Martin von Gagern
>   

The list of steps definitely looks good, and gives us a good plan of 
attack. Overall, the migration plan looks really good, although I'm 
still trying to get a few for a few bits of it, particularly when/how 
we'd migrate to the 'next gen' style of ebuilds.

Before going forward with the plan, though, it would probably be a good 
idea for all the java devs to, you know, agree to go through with this 
plan :) This is particularly important because we would be the ones 
actually committing these to portage. I'd say we're at the point where 
we could/should have a meeting in #gentoo-java to go over this.

As far as taking up particular tasks and whatnot, I'm not particular 
sure myself, being the new guy and all. But given that all work is 
voluntary anyways, I'd imagine people would volunteer for taking care of 
this or that task.

- Josh (nichoj)
-- 
gentoo-java@gentoo.org mailing list