[gentoo-dev] lua upgrade plan

[gentoo-dev] lua upgrade plan

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 1027 bytes --]

All,

lua has been behind in Gentoo for a while, but I have time now to start
working on fixing that this coming week, so consider this a heads-up
that things are going to start happening with regards to dev-lang/lua
and its dependencies.

Upstream does not support liblua as a shared library, and they do not
support installing multiple versions of lua onto a system. After
conferring with the other lua maintainer, the decision has been made to
remove this custom support from our lua package as well. This has been
talked about many times upstream. They do not want it, and using
liblua as a shared library causes performance issues.

Since  that support never came out of p.mask, I will be removing it this
weekend or Monday at the latest and committing the next version of lua
to ~arch.

Once we get the tree fixed, I will address the lua overlay, preferably
we can address it by migrating things there to the tree.

I will be around most of the time next week, so I will be able to follow
up on issues.

Thanks,

William

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] lua upgrade plan

[Azamat Hackimov]

[-- Attachment #1: Type: text/plain, Size: 1681 bytes --]

2017-06-30 22:16 GMT+05:00 William Hubbs <williamh@gentoo.org>:

> All,
>
> Upstream does not support liblua as a shared library, and they do not
> support installing multiple versions of lua onto a system. After
> conferring with the other lua maintainer, the decision has been made to
> remove this custom support from our lua package as well. This has been
> talked about many times upstream.


Lua devs very "hostile" to Linux distributers. I don't see why we should do
as they want to do.
They not have open vcs to simply see what they changes in new release, they
don't accepts
patches for system integration. They didn't even elementary easy-to-use
build system. Just
look to another distributives, they all do versioned and shared libraries
of Lua 5.{1,2,3}. Fedora
devs did custom Autotools-based buildsystem, Debian - provided pkg-config
files. There also
exists excellent LuaDist framework - still outdated, yes, but we can take
from them CMake
buildsystem to provide better integration into Gentoo enviroment. You have
so many options
but you still want to follow unwelcome Lua rules.


> They do not want it, and using liblua as a shared library causes
> performance issues.
>

Why, we live in XXI century, where this argument came from? What about
security, did you
forgot about it? How do you planning to do backward compatibility with old
lua5.1 libraries
and projects? They definitely have breakage since lua 5.2 and 5.3 not
compatible with each
other. Why Lua can't have same eclass as multislotted Python or Ruby? Lua
ecosystem not
so big, about 500 packages so why there no even little efforts to make Lua
support in Gentoo
better?

-- 
From Siberia with Love!

[-- Attachment #2: Type: text/html, Size: 2421 bytes --]

Re: [gentoo-dev] lua upgrade plan

[Vadim A. Misbakh-Soloviov]

> excellent LuaDist
I'd not say it is excellent :( I'd rather say "NIH-syndromed"

>  Why Lua can't have same eclass as multislotted Python or Ruby?
>  Lua ecosystem not so big, about 500 packages
>  so why there no even little efforts to make Lua support in Gentoo better?

Well... Actually, it does. In Lua overlay. But mgorny (?) had a bunch of 
critics about it, so I just keeping it in Lua overlay and not proposing it to 
the gentoo repo anymore (I just have not enough time to rewrite it in the way 
it will pass mgorny's review :).

Re: [gentoo-dev] lua upgrade plan

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 2678 bytes --]

On Sat, Jul 01, 2017 at 01:16:02PM +0500, Azamat Hackimov wrote:
> 2017-06-30 22:16 GMT+05:00 William Hubbs <williamh@gentoo.org>:
> 
> > All,
> >
> > Upstream does not support liblua as a shared library, and they do not
> > support installing multiple versions of lua onto a system. After
> > conferring with the other lua maintainer, the decision has been made to
> > remove this custom support from our lua package as well. This has been
> > talked about many times upstream.
> 
> 
> Lua devs very "hostile" to Linux distributers. I don't see why we should do
> as they want to do.
> They not have open vcs to simply see what they changes in new release, they
> don't accepts
> patches for system integration. They didn't even elementary easy-to-use
> build system. Just
> look to another distributives, they all do versioned and shared libraries
> of Lua 5.{1,2,3}. Fedora
> devs did custom Autotools-based buildsystem, Debian - provided pkg-config
> files. There also
> exists excellent LuaDist framework - still outdated, yes, but we can take
> from them CMake
> buildsystem to provide better integration into Gentoo enviroment. You have
> so many options
> but you still want to follow unwelcome Lua rules.

It is Gentoo's policy to stay as close to upstream as possible. However,
there are a couple of things that I can say about lua from what I've
seen so far.


> > They do not want it, and using liblua as a shared library causes
> > performance issues.
> >
> 
> Why, we live in XXI century, where this argument came from? What about
> security, did you
> forgot about it? How do you planning to do backward compatibility with old
> lua5.1 libraries
> and projects? They definitely have breakage since lua 5.2 and 5.3 not
> compatible with each
> other. Why Lua can't have same eclass as multislotted Python or Ruby? Lua
> ecosystem not
> so big, about 500 packages so why there no even little efforts to make Lua
> support in Gentoo
> better?

Portage has improved handling security issues like the ones with static
libraries a lot from what I understand by making --with-bdeps y the
default setting most of the time.

The lua build system seems to have a way to tell it to support
older things, there is a LUA_COMPAT_ALL compile flag. We'll have to see
what that does when it hits ~arch.

See this article for why using liblua as a shared library is not
recommended.

http://article.gmane.org/gmane.comp.lang.lua.general/18519

Yes, it talks about the interpretor, but it goes further and really
discourages even making a shared library available.

> 
> -- 
> From Siberia with Love!

William


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[gentoo-dev] Re: lua upgrade plan

[Duncan]

William Hubbs posted on Sat, 01 Jul 2017 11:53:59 -0500 as excerpted:

> See this article for why using liblua as a shared library is not
> recommended.
> 
> http://article.gmane.org/gmane.comp.lang.lua.general/18519
> 
> Yes, it talks about the interpretor, but it goes further and really
> discourages even making a shared library available.

PMFJI, but...

That reply is from 2005 and is apparently specific to (32-bit) x86's 
extreme shortage of general purpose registers.  Back then it made sense 
as 32-bit x86 was the dominant arch, both for gentoo, and (apparently) 
for that lua discussion (which was in the debian context).

That x86 general lack of registers was one of the big pressures behind 
the switch to amd64, before system memory sizes increased to 4GB+, and 
applies far less to today's dominant amd64, with x86 now legacy/embedded.

Now it may well be that lua performance remains register sensitive even 
with the increased number of registers available in amd64 and other 
modern archs, but quoting an 11+-year-old email written in the extremely 
register-restricted 32-bit x86 context does little to argue that point.

So... got any equivalent links to posts with more modern figures?


All that said, in FLOSS, he who volunteers, makes the rules, and 
particularly given the upstream opposition meaning more gentoo-level work 
required, if there's nobody willing to support lua in gentoo with dynamic 
linking...

... people unable or unwilling to volunteer to support their preferred 
alternative get to live with one they don't like, whether that be 
accepting what their existing distro provides even if it's not optimal, 
switching to another, or supporting their own patches or builds apart 
from gentoo.

At least gentoo makes the latter case relatively easy compared to some 
distros.  I did it with kde twice, when gentoo/kde wasn't supporting 
builds without semantic-desktop. =:^)

And in this case it appears there's even someone already doing it and 
making their work public via the lua overlay. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: lua upgrade plan

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 2800 bytes --]

On Sun, Jul 02, 2017 at 03:55:54AM +0000, Duncan wrote:
> William Hubbs posted on Sat, 01 Jul 2017 11:53:59 -0500 as excerpted:
> 
> > See this article for why using liblua as a shared library is not
> > recommended.
> > 
> > http://article.gmane.org/gmane.comp.lang.lua.general/18519
> > 
> > Yes, it talks about the interpretor, but it goes further and really
> > discourages even making a shared library available.
> 
> PMFJI, but...
> 
> That reply is from 2005 and is apparently specific to (32-bit) x86's 
> extreme shortage of general purpose registers.  Back then it made sense 
> as 32-bit x86 was the dominant arch, both for gentoo, and (apparently) 
> for that lua discussion (which was in the debian context).
> 
> That x86 general lack of registers was one of the big pressures behind 
> the switch to amd64, before system memory sizes increased to 4GB+, and 
> applies far less to today's dominant amd64, with x86 now legacy/embedded.
> 
> Now it may well be that lua performance remains register sensitive even 
> with the increased number of registers available in amd64 and other 
> modern archs, but quoting an 11+-year-old email written in the extremely 
> register-restricted 32-bit x86 context does little to argue that point.
> 
> So... got any equivalent links to posts with more modern figures?
 
That link actually came from our current lua ebuilds. The shared library
is offered, but the comments in the ebuild basically say the same thing
and cite that link. Also, x86 is still a stable supported arch in
Gentoo.

> All that said, in FLOSS, he who volunteers, makes the rules, and 
> particularly given the upstream opposition meaning more gentoo-level work 
> required, if there's nobody willing to support lua in gentoo with dynamic 
> linking...
 
You are correct. Basically we have to write our own build system and
keep it up to date for every version of lua in order to support this. If
someone can convince upstream to support it, this would be far better for
everyone.

> ... people unable or unwilling to volunteer to support their preferred 
> alternative get to live with one they don't like, whether that be 
> accepting what their existing distro provides even if it's not optimal, 
> switching to another, or supporting their own patches or builds apart 
> from gentoo.
> 
> At least gentoo makes the latter case relatively easy compared to some 
> distros.  I did it with kde twice, when gentoo/kde wasn't supporting 
> builds without semantic-desktop. =:^)
> 
> And in this case it appears there's even someone already doing it and 
> making their work public via the lua overlay. =:^)

The plan is to migrate what can be migrated from there to the
tree once everything is up to date.

William


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[gentoo-dev] Re: lua upgrade plan

[Duncan]

William Hubbs posted on Sun, 02 Jul 2017 10:30:12 -0500 as excerpted:

> On Sun, Jul 02, 2017 at 03:55:54AM +0000, Duncan wrote:
>> William Hubbs posted on Sat, 01 Jul 2017 11:53:59 -0500 as excerpted:
>> 
>> > See this article for why using liblua as a shared library is not
>> > recommended.
>> > 
>> > http://article.gmane.org/gmane.comp.lang.lua.general/18519
>> 
>> PMFJI, but...
>> 
>> That reply is from 2005 and is apparently specific to (32-bit) x86's
>> extreme shortage of general purpose registers.  Back then it made sense
>> as 32-bit x86 was the dominant arch, both for gentoo, and (apparently)
>> for that lua discussion (which was in the debian context). [...]

>> So... got any equivalent links to posts with more modern figures?
>  
> That link actually came from our current lua ebuilds. The shared library
> is offered, but the comments in the ebuild basically say the same thing
> and cite that link.

Thanks. =:^)

> Also, x86 is still a stable supported arch in Gentoo.

Two points:

1) x86 is indeed still a stable supported arch, as are, I think, a few 
others.  But we're not discussing _breaking_ lua on x86, simply accepting 
that default-dynamic-linking lua wouldn't be as efficient on x86 as it 
would be on less register-short archs, including the now dominant amd64.

Seems a reasonable tradeoff to be made, particularly given it's the 
choice other distros as well were choosing even when x86 /was/ the 
majority arch.

Especially when given that on gentoo, those that believe they have reason 
to can choose to statically link in any case.  It's just that gentoo 
normally defaults to dynamic linking if people haven't specifically 
chosen static linking.  (Or would static linking not be a user-side 
option in this case if the dynamic-link route were chosen?)

2) Just to be explicitly in case the (quote I've now omitted) bit below 
that didn't make it clear: I still support killing the shared lib if 
maintainer-desired, even if the only /technical/ reason is that upstream 
doesn't support it (for reasons apparently no longer valid on modern 
archs, but...), due to the extra work then required.

I simply believe it's important to be honest about it, if that is indeed 
the case, and am wondering if it is.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: lua upgrade plan

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 908 bytes --]

Hi,

On Sun, 2 Jul 2017 10:30:12 -0500 William Hubbs wrote:
> > All that said, in FLOSS, he who volunteers, makes the rules, and 
> > particularly given the upstream opposition meaning more gentoo-level work 
> > required, if there's nobody willing to support lua in gentoo with dynamic 
> > linking...
>  
> You are correct. Basically we have to write our own build system and
> keep it up to date for every version of lua in order to support this. 

Why should we? We can borrow from other distributions like Debian
or Fedora. Of course some Gentoo-specific tuning may be required,
but it will be not writing from scratch.

> If someone can convince upstream to support it, this would be far better for
> everyone.

In the ideal world, yes, I agree with this. But we are in real
world with real lua developers unwilling to cooperate from what I
see.

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 833 bytes --]

[gentoo-dev] Re: lua upgrade plan

[Martin Vaeth]

Duncan <1i5t5.duncan@cox.net> wrote:
>>
>> http://article.gmane.org/gmane.comp.lang.lua.general/18519
>
> That reply is from 2005 and is apparently specific to (32-bit) x86's

Even more is true! The only argument there concerns pic.
But most distributions (hopefully also gentoo in a not-so-distant future,
hopefully also on x86) will have PIE as a built-in gcc default,
and programs should better all be built with PIE due to security
considerations.

So the whole argument concerning efficiency is simply invalid, nowadays.

Yes, there might be very exceptional situations where it might be
appropriate to sacrifice security considerations completely
as a tradeoff for slightly more speed (although on amd64 this
is probably already hardly measurable). But it should be exactly this:
very exceptional situations.
Most games should better not be exceptional concerning security,
at least in a "standard" distribution.

Re: [gentoo-dev] lua upgrade plan

[Vadim A. Misbakh-Soloviov]

By the way, it will also brake some proprietary games, that distributes via 
steam, humble, gog and so on.

Some of them depends on shared lua and doesn't bundle it (instead, their 
installer calls apt (since they're doing games for ubuntu), but since we have 
no apt, we (gamerlay/games team) just writing proper DEPENDs, unpacking the 
content, and installing it.

So, if shared lua will be dropped, we will either have a bunch of broken games 
or used to create some kludgy "lua-shared" custom ebuild, which is worse way 
of doing the things.

So, I'm voting for status quo. 

Re: [gentoo-dev] lua upgrade plan

[M. J. Everitt]

[-- Attachment #1.1: Type: text/plain, Size: 743 bytes --]

On 02/07/17 21:12, Vadim A. Misbakh-Soloviov wrote:
> By the way, it will also brake some proprietary games, that distributes via 
> steam, humble, gog and so on.
>
> Some of them depends on shared lua and doesn't bundle it (instead, their 
> installer calls apt (since they're doing games for ubuntu), but since we have 
> no apt, we (gamerlay/games team) just writing proper DEPENDs, unpacking the 
> content, and installing it.
>
> So, if shared lua will be dropped, we will either have a bunch of broken games 
> or used to create some kludgy "lua-shared" custom ebuild, which is worse way 
> of doing the things.
>
> So, I'm voting for status quo. 
>
>
A custom games-lua eclass ... ?!

*runs from mgorny et al ....*


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] lua upgrade plan

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 749 bytes --]

On Mon, Jul 03, 2017 at 03:12:00AM +0700, Vadim A. Misbakh-Soloviov wrote:
> By the way, it will also brake some proprietary games, that distributes via 
> steam, humble, gog and so on.
> 
> Some of them depends on shared lua and doesn't bundle it (instead, their 
> installer calls apt (since they're doing games for ubuntu), but since we have 
> no apt, we (gamerlay/games team) just writing proper DEPENDs, unpacking the 
> content, and installing it.
> 
> So, if shared lua will be dropped, we will either have a bunch of broken games 
> or used to create some kludgy "lua-shared" custom ebuild, which is worse way 
> of doing the things.
 
Are these games in the tree? I don't see anything related to steam or humble.

 William

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]