[gentoo-dev] [gentoo-java][PR] ivy, mvn, sbt, gradle builders improvement for ebuild development

[gentoo-dev] [gentoo-java][PR] ivy, mvn, sbt, gradle builders improvement for ebuild development

[Samuel Bernardo]

[-- Attachment #1.1: Type: text/plain, Size: 1277 bytes --]

Dear Java team,

I send this email to mention that it seems to be missing eclasses for
JVM builders such as those I mention in this email subject. Dependencies
and tasks management are hard tasks now that I think to have great scope
for improvement.

Looking into the developments made in go eclasses, there is a very
interesting solution in go-module.eclass with EGO_SUM to avoid the need
of additional tarballs to fulfill the network sandbox requirement. This
way, dependencies could be listed from files, optimizing the current
approach without requiring to distribute blinded dependency tarballs.

I also check java-ant-2.eclass to manage build tasks for ant. Would be
very useful to have eclass like this for the other builders.

Also review java-pkg-2.eclass and java-utils-2.eclass that are more
related to Java packages and ant builder. Seems more difficult than
usual to understand them and how to apply to other projects. Maybe to
turn this more friendly a java-common.eclass could be created for
example, collecting the common variables and functions such as
java-pkg_getjars and java-pkg_dojar, to reuse them with all JVM dialect
related projects.

Are you forging any solutions already to improve this kind of PR?

Thanks,

Samuel



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] [gentoo-java][PR] ivy, mvn, sbt, gradle builders improvement for ebuild development

[Benda Xu]

[-- Attachment #1: Type: text/plain, Size: 485 bytes --]

Hi Samuel,

Samuel Bernardo <samuelbernardo.mail@gmail.com> writes:

> I send this email to mention that it seems to be missing eclasses for
> JVM builders such as those I mention in this email subject. Dependencies
> and tasks management are hard tasks now that I think to have great scope
> for improvement.

Have you read the Java Developer Guide of Gentoo[1]?  If so, let's base
our discussion on what we have.

> [...]

Benda

1. https://wiki.gentoo.org/wiki/Java_Developer_Guide

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: [gentoo-dev] [gentoo-java][PR] ivy, mvn, sbt, gradle builders improvement for ebuild development

[Samuel Bernardo]

[-- Attachment #1.1: Type: text/plain, Size: 953 bytes --]

Hi Benda,

I'm sorry, but maybe I miss something after reading the documentation...

I only check java and ant build system.

My point is about other JVM languages such as scala, groovy, kotlin,
clojure, ... And about builders such as gradle or sbt?

I think that eclasses for those would be very useful to develop ebuilds
and evolve with the right procedures.

Thanks,

Samuel

On 4/11/20 2:06 AM, Benda Xu wrote:
> Hi Samuel,
>
> Samuel Bernardo <samuelbernardo.mail@gmail.com> writes:
>
>> I send this email to mention that it seems to be missing eclasses for
>> JVM builders such as those I mention in this email subject. Dependencies
>> and tasks management are hard tasks now that I think to have great scope
>> for improvement.
> Have you read the Java Developer Guide of Gentoo[1]?  If so, let's base
> our discussion on what we have.
>
>> [...]
> Benda
>
> 1. https://wiki.gentoo.org/wiki/Java_Developer_Guide


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development

[Benda Xu]

[-- Attachment #1: Type: text/plain, Size: 476 bytes --]

Hi Samuel,

Samuel Bernardo <samuelbernardo.mail@gmail.com> writes:

> My point is about other JVM languages such as scala, groovy, kotlin,
> clojure, ... And about builders such as gradle or sbt?
>
> I think that eclasses for those would be very useful to develop
> ebuilds and evolve with the right procedures.

That's a good idea.  What's your plan to realize the eclasses?


BTW, additional reference:
https://wiki.gentoo.org/wiki/Gentoo_Java_Packing_Policy

Yours,
Benda

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

[gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development

[Samuel Bernardo]

[-- Attachment #1.1: Type: text/plain, Size: 1506 bytes --]

Hi Benda,

Thanks for the reference to Java Packing policy since I haven't read it
before.

I also forget to mention maven build system in last email, but for now
I'm only focused on scala and sbt.

On 4/19/20 5:31 AM, Benda Xu wrote:
> That's a good idea.  What's your plan to realize the eclasses?

Taking into account the network sandbox requirement, sbt.eclass needs to
download all dependencies with some approach like EGO_SUM implementation
in go-module.eclass[1].

Looking in more detail to scala ebuild[2] as a reference, as a quick
brainstorm, I think that could be defined:

sbt.eclass

- ESBTV = <sbt version to be set in ebuild>

This would set the DEPEND for sbt and set the necessary steps to
java-pkg-2_pkg_setup, check-reqs_pkg_setup, check-reqs_pkg_pretend,
java-pkg_getjars and do the substitution of SBTV in build.properties if
required

- envset parameters (src_prepare): definition of parameters to set
necessary environment for sbt wrapper that use defined ESBTV

- ESBT_SUM = <list of project dependencies to be loaded within source
unpack>

- esbt_compile, esbt_run, esbt_package functions: to run sbt using
envset wrapper in compile, test and install as necessary

scala.eclass

- ESCALAV = <set scala version>

- ... to be defined as required

Best,

Samuel

[1]
https://devmanual.gentoo.org/eclass-reference/go-module.eclass/index.html

[2]
https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-lang/scala/scala-2.12.10.ebuild



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development

[Michael Orlitzky]

On 4/19/20 10:55 AM, Samuel Bernardo wrote:
> 
> Taking into account the network sandbox requirement, sbt.eclass needs to
> download all dependencies with some approach like EGO_SUM implementation
> in go-module.eclass[1].
> 

EGO_SUM is not a legitimate approach to packaging. You have to create
packages for each package. I know, it sounds crazy when you say it.

Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 480 bytes --]

On Sun, Apr 19, 2020 at 8:37 AM Michael Orlitzky <mjo@gentoo.org> wrote:

> On 4/19/20 10:55 AM, Samuel Bernardo wrote:
> >
> > Taking into account the network sandbox requirement, sbt.eclass needs to
> > download all dependencies with some approach like EGO_SUM implementation
> > in go-module.eclass[1].
> >
>
> EGO_SUM is not a legitimate approach to packaging. You have to create
> packages for each package. I know, it sounds crazy when you say it.
>
>
EGO_SUM is great!

-A

[-- Attachment #2: Type: text/html, Size: 829 bytes --]

Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development

[Samuel Bernardo]

[-- Attachment #1.1: Type: text/plain, Size: 904 bytes --]

On 2020-04-19 16:37, Michael Orlitzky wrote:
> On 4/19/20 10:55 AM, Samuel Bernardo wrote:
>> Taking into account the network sandbox requirement, sbt.eclass needs to
>> download all dependencies with some approach like EGO_SUM implementation
>> in go-module.eclass[1].
>>
> EGO_SUM is not a legitimate approach to packaging. You have to create
> packages for each package. I know, it sounds crazy when you say it.
>
I understand what you mean, but deem this dependencies as internal
project dependencies where those libraries doesn't worth the effort to
package them all. I mean, most of these projects have an immutable
deployment approach that are not feasable to be packaged. The problem
starts from upstream...

Maybe that's the reason why docker is so popular in this cases... But to
install the base development tools and services I prefer to have ebuilds
rather than dockers.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development

[Michael Orlitzky]

On 4/19/20 3:41 PM, Samuel Bernardo wrote:
>>
>> EGO_SUM is not a legitimate approach to packaging. You have to create
>> packages for each package. I know, it sounds crazy when you say it.
>>
> I understand what you mean, but deem this dependencies as internal
> project dependencies where those libraries doesn't worth the effort to
> package them all. I mean, most of these projects have an immutable
> deployment approach that are not feasable to be packaged. The problem
> starts from upstream...

You can do whatever you want in an overlay, but you can't introduce
security vulnerabilities and license issues into thousands of peoples'
homes and businesses through ::gentoo because it makes your life a tiny
bit easier.

The job description of distribution developer is, ultimately, to fix all
the dumb things that upstream does before packaging the result so that
your users get a consistent, usable, reliable, and secure product. But
the first step isn't optional. Re-packaging garbage is easy -- that's a
service nobody needs.

Using ebuilds this way is also simply a waste of your time. If you're
not going to package the dependencies, then you're better off using the
upstream bundling tool. At least then you can update the hundreds of
bundled dependencies afterwards. With an ebuild that's not possible.

I don't mean to discourage you any more than necessary. I'm glad you
want to help. But please quit looking to Go as an example of anything
anyone should be doing.

Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development

[Samuel Bernardo]

[-- Attachment #1.1: Type: text/plain, Size: 2039 bytes --]

Hi Michael,

On 4/19/20 9:09 PM, Michael Orlitzky wrote:
> You can do whatever you want in an overlay, but you can't introduce
> security vulnerabilities and license issues into thousands of peoples'
> homes and businesses through ::gentoo because it makes your life a tiny
> bit easier.
>
> The job description of distribution developer is, ultimately, to fix all
> the dumb things that upstream does before packaging the result so that
> your users get a consistent, usable, reliable, and secure product. But
> the first step isn't optional. Re-packaging garbage is easy -- that's a
> service nobody needs.
>
> Using ebuilds this way is also simply a waste of your time. If you're
> not going to package the dependencies, then you're better off using the
> upstream bundling tool. At least then you can update the hundreds of
> bundled dependencies afterwards. With an ebuild that's not possible.
I agree, I have the same concern and that's why I prefer to have an
ebuild instead of using the docker container or running blindly the
provided upstream bundling tool. Avoiding to package all unnecessary
dependencies and keeping away the garbage can only be possible as you
mentioned using the distribution tools. At least, with a Gentoo overlay,
I can have software better than upstream provided and continue my work.
> I don't mean to discourage you any more than necessary. I'm glad you
> want to help. But please quit looking to Go as an example of anything
> anyone should be doing.

Thanks for your advice, but I believe that we win more with the ebuild
for the development tools and base services, even if it will only be
available in the overlay. Is better to have the eyes of experienced
maintainers, Gentoo QA and sharing information, instead of keeping it
standalone, perpetuating the bad procedures and software usage.

I really feel like scoring when reviewing what I need to do for the
ebuild, also when it makes me spent some more time. The profit comes
afterwards!

Best,
Samuel



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development

[Patrick McLean]

On Sun, 19 Apr 2020 11:37:15 -0400
Michael Orlitzky <mjo@gentoo.org> wrote:

> On 4/19/20 10:55 AM, Samuel Bernardo wrote:
> > 
> > Taking into account the network sandbox requirement, sbt.eclass needs to
> > download all dependencies with some approach like EGO_SUM implementation
> > in go-module.eclass[1].
> >   
> 
> EGO_SUM is not a legitimate approach to packaging. You have to create
> packages for each package. I know, it sounds crazy when you say it.
> 
A lot of upstreams are doing things that are not very friendly to packagers. Sometimes we have to take different approaches to packaging to make it feasible to package these very popular programs.

Having 300+ packages in the tree with a single version that are only used as a dependency by a single package is not the way to a well-maintained tree. Sometimes things like EGO_SUM are the only feasible way to do things given we all have limited time to work on Gentoo, and we want these packages in the tree because many people find them useful.

If you object to the way these are handled, you are free to mask dev-lang/go on your system to endure that you do not get anything written in go installed. Simply having things in ::gentoo does not affect anyone who does not use them.

-- 

Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development

[Michael Orlitzky]

On 4/20/20 1:31 PM, Patrick McLean wrote:
>> Simply having things in ::gentoo does not affect anyone who does not
use them.
> 

You keep saying that, but the fact that dev-go/* is filled with trash
that has your name on it prevents anyone else from doing a better job.

Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development

[Patrick McLean]

On Mon, 20 Apr 2020 14:07:01 -0400
Michael Orlitzky <mjo@gentoo.org> wrote:

> On 4/20/20 1:31 PM, Patrick McLean wrote:
> >> Simply having things in ::gentoo does not affect anyone who does not  
> use them.
> >   
> 
> You keep saying that, but the fact that dev-go/* is filled with trash
> that has your name on it prevents anyone else from doing a better job.
> 
Ad-hominen attacks are unwarranted, please refrain from them. I challenge you to find *anything* in dev-go/* with my name on it.

Are you volunteering to do the work to package go packages? The people doing the work generally get to decide how that work gets done, and which approach they would like to take. The upstream situation makes it very labour-intensive to do the work in a the way you are proposing (many packages would end up with hundreds to thousands of packages in the tree). Separating everything out in to separate packages will just increase the maintenance load exponentially, with no gains as go upstreams version lock all their dependencies.

Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development

[Michael Orlitzky]

On 4/20/20 2:58 PM, Patrick McLean wrote:
>>
>> You keep saying that, but the fact that dev-go/* is filled with trash
>> that has your name on it prevents anyone else from doing a better job.
>>
> Ad-hominen attacks are unwarranted, please refrain from them. I challenge you to find *anything* in dev-go/* with my name on it.

You quoted me one sentence prior saying that it's the Go ebuilds that
are trash and not anyone personally. But OK, I should have said "Sony
Interactive Entertainment" there instead of "you."


> Are you volunteering to do the work to package go packages? The people doing the work generally get to decide how that work gets done, and which approach they would like to take. The upstream situation makes it very labour-intensive to do the work in a the way you are proposing (many packages would end up with hundreds to thousands of packages in the tree). Separating everything out in to separate packages will just increase the maintenance load exponentially, with no gains as go upstreams version lock all their dependencies.
> 

I'm volunteering to work on one or two small Go packages. Can I convert
the eclass to use dynamic linking? Can I start replacing your packages
with my own if I need them as dependencies? I suspect not, and that's
one of many reasons why "having things in ::gentoo does not affect
anyone who does not use them" is bullshit.

Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development

[Rich Freeman]

On Mon, Apr 20, 2020 at 2:07 PM Michael Orlitzky <mjo@gentoo.org> wrote:
>
> On 4/20/20 1:31 PM, Patrick McLean wrote:
> >> Simply having things in ::gentoo does not affect anyone who does not
> use them.
> >
>
> You keep saying that, but the fact that dev-go/* is filled with trash
> that has your name on it prevents anyone else from doing a better job.
>

As far as I'm aware Gentoo policy does not prevent somebody from
re-implementing a package that is already in Gentoo.  Obviously it
could get pretty confusing if that happened.

IMO it isn't really worth worrying about, because right now the main
limitation seems to be a lack of people working on projects, not 25
devs who each want to re-implement go their own way...

-- 
Rich

Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development

[Patrick McLean]

On Mon, 20 Apr 2020 15:23:15 -0400
Michael Orlitzky <mjo@gentoo.org> wrote:

> On 4/20/20 2:58 PM, Patrick McLean wrote:
> >>
> >> You keep saying that, but the fact that dev-go/* is filled with trash
> >> that has your name on it prevents anyone else from doing a better job.
> >>  
> > Ad-hominen attacks are unwarranted, please refrain from them. I challenge you to find *anything* in dev-go/* with my name on it.  
> 
> You quoted me one sentence prior saying that it's the Go ebuilds that
> are trash and not anyone personally. But OK, I should have said "Sony
> Interactive Entertainment" there instead of "you."

My co-workers are not the only ones adding/maintaining go packages in the tree, please do not single out any groups, and let's all work to make Gentoo the best it can be.

> 
> > Are you volunteering to do the work to package go packages? The people doing the work generally get to decide how that work gets done, and which approach they would like to take. The upstream situation makes it very labour-intensive to do the work in a the way you are proposing (many packages would end up with hundreds to thousands of packages in the tree). Separating everything out in to separate packages will just increase the maintenance load exponentially, with no gains as go upstreams version lock all their dependencies.
> >   
> 
> I'm volunteering to work on one or two small Go packages. Can I convert
> the eclass to use dynamic linking? Can I start replacing your packages
> with my own if I need them as dependencies? I suspect not, and that's
> one of many reasons why "having things in ::gentoo does not affect
> anyone who does not use them" is bullshit.
> 
Please do not make backward-incompatible changes to existing eclasses. If you would like to make a new go eclasss (say go-dynamic.eclass) then go ahead, I am certainly not going to stop you. As was said elsewhere in the thread, there is no policy against having a second version of a package in the tree. If the second version is truly better, then the other packages can be moved over to depending on it instead, and the original package can be moved or last-rited.

The current method of packaging go packages tries to strike a balance between maintainability, following upstream, and something that is compatible with ebuilds. If you have a better way to do things, we are not going to block you from trying, but we are also not going to necessarily adopt it if working with it would be orders of magnitude more work. We all have limited time to work on Gentoo stuff.

Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development

[Georg Rudoy]

вс, 19 апр. 2020 г. в 16:09, Michael Orlitzky <mjo@gentoo.org>:
>  But please quit looking to Go as an example of anything
> anyone should be doing.

On a somewhat related note, I'd like to take this chance to ask about
packaging haskell software, which has somewhat similar issues:

1. Two different executables might depend on different versions of a
library. This is, of course, solvable via slots, so this is minor
(but, looking at a random sample of `dev-haskell/`, most of libs there
aren't slotted).
2. The dependency trees are pretty big and wide. A simple project of
mine depends on a hundred or two of dependencies on average, and the
union of all deps of all the projects I do is probably closer to a
thousand. Again, this is minor, just bear in mind the corresponding
ebuilds tree size increase.
3. What is major is the following. Let's say two executables A and B
depend on libfoo:1 and libfoo:2 respectively, and depend on the same
version of libbar. libbar itself depends on libfoo (and supports both
versions), but libbar built with libfoo:1 is incompatible with libbar
built with libfoo:2. Loosely speaking, the slot of a library is
defined by all its dependencies' _versions_.

Of all these, (3) is quite major, as it leads to exponential explosion
of the slots count. Otherwise, one might choose to carefully curate
the library versions, ensuring that no such situation arises, but this
approach is neither feasible nor possible at all for sufficiently
large packages base.

In practice, outside of Gentoo/system PM issues, this does not pose a
problem since each executable is built in an isolated sandbox and
links statically to its dependencies, so different executables can
happily be deployed alongside each other even if they have
incompatible dependencies. But looks like Gentoo's approach to haskell
is to use dynamic linking and expose the dependencies to the package
manager (which I sure agree is the right thing to do), so what should
be right the solution here?

-- 
  Georg Rudoy

Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development

[Michael Orlitzky]

On 4/20/20 4:21 PM, Georg Rudoy wrote:
> вс, 19 апр. 2020 г. в 16:09, Michael Orlitzky <mjo@gentoo.org>:
>>  But please quit looking to Go as an example of anything
>> anyone should be doing.
> 
> On a somewhat related note, I'd like to take this chance to ask about
> packaging haskell software, which has somewhat similar issues:
> 
> ...
> 3. What is major is the following. Let's say two executables A and B
> depend on libfoo:1 and libfoo:2 respectively, and depend on the same
> version of libbar. libbar itself depends on libfoo (and supports both
> versions), but libbar built with libfoo:1 is incompatible with libbar
> built with libfoo:2. Loosely speaking, the slot of a library is
> defined by all its dependencies' _versions_.
> 
> Of all these, (3) is quite major, as it leads to exponential explosion
> of the slots count. Otherwise, one might choose to carefully curate
> the library versions, ensuring that no such situation arises, but this
> approach is neither feasible nor possible at all for sufficiently
> large packages base.

There's nothing Haskell-specific here, is there? Except that Haskell as
a research language tends to inspire more "fire and forget" software.
(And Haskell works better in Gentoo than anywhere else, in my opinion.)

You package each dependency as a separate package. If there's a conflict
between two packages, then... you can't have those two packages
installed together. When that happens, you alert upstream, and try to
convince them to be more lenient with their dependencies (again, nothing
Haskell-specific here). Sometimes that involves code changes, but often
it's just an edit to the cabal file. Many people follow either semver or
the PVP (https://pvp.haskell.org/) which keeps things from getting too
far out of control.

If upstream absolutely insists on minor-version dependencies, then you
either tolerate it conflicting with everything else, or leave it out of
the tree. You probably shouldn't even be packaging a library whose API
is distinguishable across minor releases.

Software engineering is a fractal. Curating APIs and dependencies is as
much a part of it as what sorting algorithms you choose. These are all
old problems with known solutions (sonames, semver, etc.) and software
isn't magically high-quality because it's written in a language that
came our last week.

So aside from the "What's Gentoo?" that's going to be more common among
Haskell upstreams, I don't see anything fundamentally different here
between Haskell (or Go, or...) and C. We should be helping to make
upstream software more good, and then packaging it. (It's OK to hack the
cabal file while the bug report is open.)

Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development

[Samuel Bernardo]

[-- Attachment #1.1: Type: text/plain, Size: 1919 bytes --]

On 4/20/20 8:27 PM, Rich Freeman wrote:
> IMO it isn't really worth worrying about, because right now the main
> limitation seems to be a lack of people working on projects, not 25
> devs who each want to re-implement go their own way...

This is another reason I think is so important the overlays in Gentoo.

::gentoo overlay can be the most stricter one.

It is a fact that is more feasible to deliver software in overlays as
already happen (for example gitlab, science stuff, ...).

But should ::gentoo be confined to GNU, Linux and other OS base stuff?

Lets look into a specific case for a distro where everything should be
packaged and placed in an universal repository (crazy debian):

https://packages.debian.org/buster/snapd

There are very radical package pattern people there and looking into
snapd we can see that upstream just regarded the package integration,
bundling all necessary dependencies into it. So, even there, we can't
find all the required go dependencies packaged... Actually, the
dependencies are only those that undoubtedly could appear on ::gentoo.

IMHO, there are many software that doesn't need to be in a ebuild, but
the base pieces such as go, snapd, docker-ce, docker engine, are some
examples of go tools that we need to have available in ::gentoo. The
same happens for other platforms such as those mentioned in the subject
of this email.

In conclusion, I think that all the trouble about licenses and security
will be always a growing challenge and a distro should adapt their tools
to allow software being available in a better way than upstream
provided. I think that the point here is about:
- how to bring the software to an OS
- how to manage the software in a better way
- share knowledge and magnetize the community so Gentoo can grow and
show how is so useful helping to deliver better software

I'm very pleased for Gentoo!



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development

[Michael Orlitzky]

On 4/20/20 4:19 PM, Patrick McLean wrote:
> 
> My co-workers are not the only ones adding/maintaining go packages in the tree, please do not single out any groups, and let's all work to make Gentoo the best it can be.
> 

Everyone else is just using the eclass that your coworkers defended and
committed the last time we did this. We're not in this together and you
sound like a COVID commercial from a billionaire that wants me to go
back to work. You're in this for you, and everything you do for you
makes things worse for me.


[0] Sony Interactive Entertainment

Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development

[William Hubbs]

On Mon, Apr 20, 2020 at 03:23:15PM -0400, Michael Orlitzky wrote:
> > Are you volunteering to do the work to package go packages? The people doing the work generally get to decide how that work gets done, and which approach they would like to take. The upstream situation makes it very labour-intensive to do the work in a the way you are proposing (many packages would end up with hundreds to thousands of packages in the tree). Separating everything out in to separate packages will just increase the maintenance load exponentially, with no gains as go upstreams version lock all their dependencies.
> > 
> 
> I'm volunteering to work on one or two small Go packages. Can I convert
> the eclass to use dynamic linking? Can I start replacing your packages
> with my own if I need them as dependencies? I suspect not, and that's
> one of many reasons why "having things in ::gentoo does not affect
> anyone who does not use them" is bullshit.

As one of the maintainers of the go-module eclass, I'm all ears, but
keep a couple of things in mind.

First, I have yet to see an upstream go-based build system that uses
dynamic linking for go libraries. No one does it. they all statically
link everything. This alone probably means you'll have to patch every
upstream build system you find.

Another issue is the sheer number of dependencies that a single go
package would pull into the tree. 
For example, dev-vcs/cli-0.6.3, which looks  like a small enough
package, has 65 dependencies. sys-apps/kubernetes-1.18.2, a bigger package for
sure, has 246. So, the worst case you are looking at here is over 300
dependencies, and this is just for two packages. The only way this
number would come down is if these packages happened to share exact
dependencies since you can't really unlock the versions.

and, yes, gccgo is out there, but no one writes for it. I have yet
to see an upstream that supports using it. So, if you go that route,
you are looking at patching build systems.

Your proposal seems to completely go against how the go ecosystem operates,
but if you can come up with a proof-of-concept for how it would work
without forcing a lot of busy work on us that would never get accepted
upstream, I'll take a look.

William

Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development

[Georg Rudoy]

пн, 20 апр. 2020 г. в 16:51, Michael Orlitzky <mjo@gentoo.org>:
> Except that Haskell as a research language tends to inspire more "fire and forget" software.

I'd say this is not the case for the last few years, but that's maybe
just my own impression from interacting with

> (And Haskell works better in Gentoo than anywhere else, in my opinion.)

Totally agreed here. That's one of the main reasons I run Gentoo
(especially before dev-haskell/stack emerged upstream a few years
ago).

> You package each dependency as a separate package. If there's a conflict
> between two packages, then... you can't have those two packages
> installed together. When that happens, you alert upstream, and try to
> convince them to be more lenient with their dependencies (again, nothing
> Haskell-specific here). Sometimes that involves code changes, but often
> it's just an edit to the cabal file. Many people follow either semver or
> the PVP (https://pvp.haskell.org/) which keeps things from getting too
> far out of control.

I'd say the Haskell-specific thing is the presence of stackage LTS
releases that people tend to stick to. For instance, beam-postgres
disappeared in recent LTSes, so some projects of mine are stuck on
older LTSes (adding that to extra-deps was a pain for reasons I don't
remember at the moment), while the others can happily use the new
ones. Libraries incompatibilities are just a matter of time in this
case.

Sure, hacking on .cabal/package.yaml is one way to solve this, but I
wasn't sure it's reasonably robust in the long run.

Anyway, thanks for your feedback. This probably means I should stop
thinking about the general case and maybe start playing around with
packaging stuff, and then seeing where it breaks.

> If upstream absolutely insists on minor-version dependencies, then you
> either tolerate it conflicting with everything else, or leave it out of
> the tree. You probably shouldn't even be packaging a library whose API
> is distinguishable across minor releases.

That's not a matter of just the API per se. Even the library file name
encodes its deps in its name (if I understand correctly, that's the
hash in libHSregex-base-0.94.0.0-541xQVwwNRiBGjsNKmOPoy-ghc8.6.5.so
for example). Personally I just find it hard to reason about this sort
of dependencies management. But, again, I should probably avoid trying
that and just jump head first to packaging.

> I don't see anything fundamentally different here
> between Haskell (or Go, or...)

Dunno much about Go and I don't have a single Go package locally to
check. Do they do static or dynamic linking with the deps, for
instance? What's the language model wrt API and linking?

> and C.

More stable API (and ABI).


-- 
  Georg Rudoy

Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development

[Patrick McLean]

On Mon, 20 Apr 2020 17:03:50 -0400
Michael Orlitzky <mjo@gentoo.org> wrote:

> On 4/20/20 4:19 PM, Patrick McLean wrote:
> > 
> > My co-workers are not the only ones adding/maintaining go packages in the tree, please do not single out any groups, and let's all work to make Gentoo the best it can be.
> >   
> 
> Everyone else is just using the eclass that your coworkers defended and
> committed the last time we did this. We're not in this together and you
> sound like a COVID commercial from a billionaire that wants me to go
> back to work. You're in this for you, and everything you do for you
> makes things worse for me.
> 
We are under no obligation to contribute anything to ::gentoo, we could do all our go packaging work in a private overlay, and never push anything upstream. It is a considerable amount of extra work to push our packages upstream. We do this to contribute to the wider Gentoo community. I resent your statement that we are not doing this to contribute to the community.

Please explain how we are actively making things worse for you? We are contributing useful packages to the tree, we are doing the work and we are doing it in the way that makes the most effective use of our time. We simply do not have time to be trying to convince upstreams to make changes to their build systems to support a single, somewhat niche, distribution. We certainly don't have time to be patching build systems with every version bump.

Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development

[Michael Orlitzky]

On 4/20/20 5:05 PM, Georg Rudoy wrote:
> 
>> If upstream absolutely insists on minor-version dependencies, then you
>> either tolerate it conflicting with everything else, or leave it out of
>> the tree. You probably shouldn't even be packaging a library whose API
>> is distinguishable across minor releases.
> 
> That's not a matter of just the API per se. Even the library file name
> encodes its deps in its name (if I understand correctly, that's the
> hash in libHSregex-base-0.94.0.0-541xQVwwNRiBGjsNKmOPoy-ghc8.6.5.so
> for example). Personally I just find it hard to reason about this sort
> of dependencies management. But, again, I should probably avoid trying
> that and just jump head first to packaging.

Haskell already requires transitive rebuilds of dependencies due to a
(serious) implementation constraint. Subslots help with that, but it
will take a new EAPI to truly tell an ebuild how to do what we need to
do. In any case, all Haskell ebuilds depend on separate packages for
their dependencies, and with haskell-updater, subslots, and
@preserved-rebuild it all more or less works.

I think that's largely separate from the "diamond dependency problem"
you posed. Diamond dependencies manifest mainly in delayed version
bumps, while slyfox does all the work to make sure that the things
already in the tree will work with the new version. This requires lots
of careful updates to neighboring packages, and unfortunately a lot of
cabal file hacking.


> Dunno much about Go and I don't have a single Go package locally to
> check. Do they do static or dynamic linking with the deps, for
> instance? What's the language model wrt API and linking?

FORGET I MENTIONED IT


>> and C.
> 
> More stable API (and ABI).
> 

Definitely. The "Haskell" language changes entirely every few years.
I've learned the hard way that it discourages you from doing all the
things that I just said high-quality software should do.

There's a core of mature Haskell that's pretty easy to develop against.
(I think you just have to wait about five years with any project before
the authors realize that changing everything every month isn't fun.) Out
in the woods you can still get into a lot of trouble though. We now have
the mature core stuff in ::gentoo, and the crazies out in the ::haskell
overlay. That feels like the right mix.

Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development

[Georg Rudoy]

пн, 20 апр. 2020 г. в 17:38, Michael Orlitzky <mjo@gentoo.org>:
> Diamond dependencies manifest mainly in delayed version
> bumps, while slyfox does all the work to make sure that the things
> already in the tree will work with the new version. This requires lots
> of careful updates to neighboring packages, and unfortunately a lot of
> cabal file hacking.

This is precisely what I meant by "doesn't scale well".

Wonder if this can be automated.

> > Dunno much about Go and I don't have a single Go package locally to
> > check. Do they do static or dynamic linking with the deps, for
> > instance? What's the language model wrt API and linking?
>
> FORGET I MENTIONED IT

Lol, happy to!

> >> and C.
> >
> > More stable API (and ABI).
> >
>
> Definitely. The "Haskell" language changes entirely every few years.

Some say the same about C++ (and ABI does change even within a standard), so...

> I've learned the hard way that it discourages you from doing all the
> things that I just said high-quality software should do.

Again, ranging from one-off pseudo-scripts that I had to come back to
after a couple of years, to quite complicated pieces of software
running in prod and actually serving customers I had to update up to
the latest LTS after 3-4 years of inactivity, I tend to disagree. Even
the latter was a surprisingly smooth process boiling down to about an
hour or two of very effortless, mechanical and thus boring code
changes (mostly about Semigroup/Monoid hierarchy, if you wonder).

This is surely offtopic, but I just don't want to silently participate
in spreading things that aren't objectively true (even if they are
subjectively true in your/mine/etc experience).

> There's a core of mature Haskell that's pretty easy to develop against.
> (I think you just have to wait about five years with any project before
> the authors realize that changing everything every month isn't fun.) Out
> in the woods you can still get into a lot of trouble though. We now have
> the mature core stuff in ::gentoo, and the crazies out in the ::haskell
> overlay. That feels like the right mix.

I've seen transifex-client last-rited the other day and was thinking
of throwing up something in haskell (plus I have a couple of other
projects that might be useful). Let's see what it'd take to get that
into ::gentoo once I'm done.

BTW having things like servant in ::haskell as opposed to ::gentoo is
like having boost in some C++ overlay as opposed to ::gentoo.

-- 
  Georg Rudoy

Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development

[Michael Orlitzky]

On 4/20/20 5:25 PM, Patrick McLean wrote:
> 
> Please explain how we are actively making things worse for you? We
> are contributing useful packages to the tree, we are doing the work
> and we are doing it in the way that makes the most effective use of
> our time. We simply do not have time to be trying to convince
> upstreams to make changes to their build systems to support a single,
> somewhat niche, distribution. We certainly don't have time to be
> patching build systems with every version bump.
> 

You're introducing unfixable security vulnerabilities into a
distribution that I recommend to people, that my name implicitly
endorses, and that I use to professionally to handle peoples' sensitive
personal data.

Likewise with the license issues that can't be fixed.

It's a waste of time having to explain to every new contributor that Go
ebuilds in ::gentoo are "special" and aren't something they should be
using as an example.

You're setting an example that our QA and security standards (that we
beat to death in the quizzes) can be ignored whenever you feel like it.
This indirectly creates technical debt as other developers follow your
example.

You're directly creating technical debt that someone else is going to
have to clean up when you move on to the next shiny and the treecleaners
have to waste six months last-riting everything.

And you're blocking a golang implementation that doesn't have these
problems.

Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development

[Michael Orlitzky]

On 4/20/20 5:48 PM, Georg Rudoy wrote:
> 
>> I've learned the hard way that it discourages you from doing all the
>> things that I just said high-quality software should do.
> 
> Again, ranging from one-off pseudo-scripts that I had to come back to
> after a couple of years, to quite complicated pieces of software
> running in prod and actually serving customers I had to update up to
> the latest LTS after 3-4 years of inactivity, I tend to disagree. Even
> the latter was a surprisingly smooth process boiling down to about an
> hour or two of very effortless, mechanical and thus boring code
> changes (mostly about Semigroup/Monoid hierarchy, if you wonder).
> 
> This is surely offtopic, but I just don't want to silently participate
> in spreading things that aren't objectively true (even if they are
> subjectively true in your/mine/etc experience).

Fair enough, I was speaking only from personal experience. But if you do
an experiment I bet you'll agree with me. Updating your code to work
with the latest GHC is pretty easy. But if you wanted to write an ebuild
for that program, it would have a dependency on that specific version of
GHC, leading to these same problems that we're talking about when other
programs require other specific versions.

I can compile my C programs with any GCC going back to the 4.x days --
or even clang. In the past three years, we've collected ghc-8.0 though
ghc-8.10 in ::gentoo. Try to make your code work with all of those GHC
releases at once; if you come back and tell me it was easy, I'll be
impressed (and may start asking you to update my Haskell code).

Working around something like gcc-10's -fno-common breakage, on the
other hand, is relatively easy. We write a patch, send it upstream, add
it to ::gentoo in the meantime, and everything keeps working. But that's
only backwards-compatible because the C standard says how it should
work, and the C standard doesn't change every six months to be "whatever
this version of GCC does."

To refer back to something on-topic: some projects are willing to make
the extra effort, and some aren't. The ones that are, are easy to
package and definitely belong in ::gentoo. The others... should be dealt
with case-by-case. But lowering the standards of the main tree should
not be how we get them over the bar.

Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1883 bytes --]

On Mon, 2020-04-20 at 16:04 -0500, William Hubbs wrote:
> Your proposal seems to completely go against how the go ecosystem operates,
> but if you can come up with a proof-of-concept for how it would work
> without forcing a lot of busy work on us that would never get accepted
> upstream, I'll take a look.
> 

Define 'busy work'.

Is doing things right 'busy work' vs taking shortcuts?  Is it 'busy
work' that I'm putting a lot of effort into fixing Python packages? 
Should I just last rite them all and tell people to use virtualenv?

Is testing packages 'busy work'?  I suppose it's all easier when you
just silently include 240 dependencies without testing a single one of
them and call it a day.  Sure, you run some tests on the final package. 
Leaves 240 untested, some of them likely failing tests and requiring
'busy work' to fix them.

Is security support 'busy work'?  I suppose it's all easier when you
ignore them problem and let security team deal with it (except they
won't).  Sure, you can assume that when vulnerability is discovered, all
upstreams will eventually learn of it, update their bundled dependencies
and *maybe* inform people that their code might have been indirectly
vulnerable (but would they do the 'busy work' of discovering whether it
affected them or not?).

I realize Go is not isolated here.  It's just brought as one major
example.  Rust is no better.  All these shiny 'write and forget'
languages share the same problem.  Pay for some work hours, get
a working product, deploy it and forget unless customers want more
features.

Today these languages are still young and the problems can be considered
largely theoretical.  But some day -- well, unless all the cool kids
manage to move on to next shiny new language before then -- this will be
a major catastrophe.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development

[Samuel Bernardo]

[-- Attachment #1.1: Type: text/plain, Size: 1724 bytes --]

On 4/21/20 6:50 AM, Michał Górny wrote:
> I realize Go is not isolated here.  It's just brought as one major
> example.  Rust is no better.  All these shiny 'write and forget'
> languages share the same problem.  Pay for some work hours, get
> a working product, deploy it and forget unless customers want more
> features.
>
> Today these languages are still young and the problems can be considered
> largely theoretical.  But some day -- well, unless all the cool kids
> manage to move on to next shiny new language before then -- this will be
> a major catastrophe.

Looking into an old language (25 years old): php

So many security problems...

Just a few essential ebuilds exists in ::gentoo and being very well
maintained by Gentoo.

I think the approach to the new languages have the same requirement.
Minimizing the ebuilds to the required ones would allow to minimize the
effort. Then we can use an overlay to deliver all other software that
would be useful.

So the goal in this cases would be to define the right sieve about those
libraries and tools that are required to all and that fits in package
maintenance procedures, so it could be merged into ::gentoo.

But don't forget the tooling available to the overlays development,
where another abstraction layer takes place, since it's focused on a
specific target and not all community in main ::gentoo. That's why I
consider go-module as a very cleaver solution since it gives a hybrid
solution connecting immutable delivery model into the mutable reality.

The PR present in this subject is related to missing eclass for JVM
languages, to define the common procedures to use those languages and
available builders.



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 3775 bytes --]

On Tue, Apr 21, 2020 at 07:50:13AM +0200, Michał Górny wrote:
> On Mon, 2020-04-20 at 16:04 -0500, William Hubbs wrote:
> > Your proposal seems to completely go against how the go ecosystem operates,
> > but if you can come up with a proof-of-concept for how it would work
> > without forcing a lot of busy work on us that would never get accepted
> > upstream, I'll take a look.
> > 
> 
> Define 'busy work'.
 
 Busy work, to me at least, is work that generates a large amount of
overhead compared to the gain it offers the distro and will never be
accepted by upstream.

> Is doing things right 'busy work' vs taking shortcuts?

This depends on what you think "doing things right" or "taking
shortcuts" mean. For example, I have had discussions with maintainers who
are fine with writing patches to fix issues without reporting the issues
upstream. Personally, I disagree with this approach.

> Is it 'busy
> work' that I'm putting a lot of effort into fixing Python packages? 
> Should I just last rite them all and tell people to use virtualenv?
 
This gets into the other issue being discussed on the thread, but I have
heard people say that yes you should. All we really need in portage is
the base python tools -- the language, maybe pip and maybe setuptools
and portage itself, then everything else should be installed via pip.

> Is testing packages 'busy work'?  I suppose it's all easier when you
> just silently include 240 dependencies without testing a single one of
> them and call it a day.  Sure, you run some tests on the final package. 
> Leaves 240 untested, some of them likely failing tests and requiring
> 'busy work' to fix them.
 
In general I would say no, but I have run into tests that do not tollerate
portage's sandbox, tests that take hours or days to run, or tests that
require root privs. We don't force those kinds of tests to be run in src_test.

> Is security support 'busy work'?  I suppose it's all easier when you
> ignore them problem and let security team deal with it (except they
> won't).  Sure, you can assume that when vulnerability is discovered, all
> upstreams will eventually learn of it, update their bundled dependencies
> and *maybe* inform people that their code might have been indirectly
> vulnerable (but would they do the 'busy work' of discovering whether it
> affected them or not?).
 
In my experience, the security team doesn't write patches to fix
vulnerabilities, so what they do wouldn't be any different in this case.

> I realize Go is not isolated here.  It's just brought as one major
> example.  Rust is no better.  All these shiny 'write and forget'
> languages share the same problem.  Pay for some work hours, get
> a working product, deploy it and forget unless customers want more
> features.
> 
> Today these languages are still young and the problems can be considered
> largely theoretical.  But some day -- well, unless all the cool kids
> manage to move on to next shiny new language before then -- this will be
> a major catastrophe.

We can talk about theoretical issues in languages all day, but if we are
going to do that we also have to talk about how terrible and permissive
c is. You can google things like "security in c" and you will find that
since it allows you to do exactly what you want, it is a major
catastrophe in terms of security, but we accept new applications in c
without any questions.

Also, what about the nightmare that is php?

In short, I don't see any reason to bikeshed about theoretical issues in
languages.

The way I see Go and Rust specifically is, more and more things are
being written in these languages, and if we don't accept them, we will
become irrelivent fast.

William


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]