[gentoo-dev] the state of dev-lang/lua

[gentoo-dev] the state of dev-lang/lua

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 960 bytes --]

Hi all,

Soon I will be working on fixing up the state of dev-lang/lua, and there
are a couple of things I want to mention.

The first thing is liblua as a shared library. If you are using lua
internally in a program, upstream strongly recommends not linking it
this way; it is supposed to be statically linked into the executable.
Because of this, and because of the amount of custom patching we do to
maintain liblua as a shared library, I plan to stop creating the shared
library.

I'm a bit undecided still about slotting lua. I'm sure we
need subslots so we can force rebuilds when new lua releases enter the
tree. However, I'm still unsure whether we need slots. I don't know of
many things in the tree that are locked to a specific version
of lua (there is only one package based on an irc conversation I
had this week).
Does anyone have any thoughts? Are there more packages in the tree that
are locked to a specific version of lua?

Thanks,

William


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] the state of dev-lang/lua

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1588 bytes --]

On Sat, 2019-03-23 at 16:23 -0500, William Hubbs wrote:
> Hi all,
> 
> Soon I will be working on fixing up the state of dev-lang/lua, and there
> are a couple of things I want to mention.
> 
> The first thing is liblua as a shared library. If you are using lua
> internally in a program, upstream strongly recommends not linking it
> this way; it is supposed to be statically linked into the executable.
> Because of this, and because of the amount of custom patching we do to
> maintain liblua as a shared library, I plan to stop creating the shared
> library.

You mean the ~8 lines used to create shared objects and library?  Sure,
it really makes sense to turn Gentoo into security/minor rebuild
nightmare just to please upstream's wannabe/laziness.

I suppose you realize that >50% of that patch is actually about
slotting, right?

> I'm a bit undecided still about slotting lua. I'm sure we
> need subslots so we can force rebuilds when new lua releases enter the
> tree. However, I'm still unsure whether we need slots. I don't know of
> many things in the tree that are locked to a specific version
> of lua (there is only one package based on an irc conversation I
> had this week).
> Does anyone have any thoughts? Are there more packages in the tree that
> are locked to a specific version of lua?
> 

You won't know that, given that there's only a single unmasked version,
so not many people have tested the new versions.  You could create
unslotted, masked 5.3 ebuild and ask Toralf to give it a ride.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: [gentoo-dev] the state of dev-lang/lua

[contact]

[2019-03-23 16:23:27] William Hubbs:
> The first thing is liblua as a shared library. If you are using lua
> internally in a program, upstream strongly recommends not linking it
> this way; it is supposed to be statically linked into the executable.
> Because of this, and because of the amount of custom patching we do to
> maintain liblua as a shared library, I plan to stop creating the shared
> library.

Pretty sure removing the shared version of liblua will cause a large
amount of patching to be done in a lot of software that may not support
static linking out of the box.

> I'm a bit undecided still about slotting lua. I'm sure we
> need subslots so we can force rebuilds when new lua releases enter the
> tree. However, I'm still unsure whether we need slots. I don't know of
> many things in the tree that are locked to a specific version
> of lua (there is only one package based on an irc conversation I
> had this week).
> Does anyone have any thoughts? Are there more packages in the tree that
> are locked to a specific version of lua?

There should be slotting of lua, for example in the few programs that
I know will need it:

- app-editors/vis::{lua,lanodanOverlay}: Requires >=lua-5.2
	does not support luajit (incomplete 5.2 compatibility)

- media-video/mpv: requires lua:5.2 or luajit
	doesn’t seems to plan supporting 5.3[1]

Also the lua overlay, which I would highly recommend using as a base
has a slotted version of lua, instead of extending SLOT="0".

1: https://github.com/mpv-player/mpv/wiki/FAQ#Why_does_mpv_not_support_Lua_53

Re: [gentoo-dev] the state of dev-lang/lua

[Andreas K. Huettel]

[-- Attachment #1: Type: text/plain, Size: 907 bytes --]

Am Samstag, 23. März 2019, 22:23:27 CET schrieb William Hubbs:
> Hi all,
> 
> Soon I will be working on fixing up the state of dev-lang/lua, and there
> are a couple of things I want to mention.
> 
> The first thing is liblua as a shared library. If you are using lua
> internally in a program, upstream strongly recommends not linking it
> this way; it is supposed to be statically linked into the executable.
> Because of this, and because of the amount of custom patching we do to
> maintain liblua as a shared library, I plan to stop creating the shared
> library.
> 

Please dont. Static linking is a security nightmare.

I'd much rather consider removing the static library, and fix programs broken 
by that. (No matter what silly opinions lua upstream has.)

-- 
Andreas K. Hüttel
dilfridge@gentoo.org
Gentoo Linux developer
(council, toolchain, base-system, perl, libreoffice)

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] the state of dev-lang/lua

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 1543 bytes --]

On Sun, Mar 24, 2019 at 02:26:35PM +0100, Andreas K. Huettel wrote:
> Am Samstag, 23. März 2019, 22:23:27 CET schrieb William Hubbs:
> > Hi all,
> > 
> > Soon I will be working on fixing up the state of dev-lang/lua, and there
> > are a couple of things I want to mention.
> > 
> > The first thing is liblua as a shared library. If you are using lua
> > internally in a program, upstream strongly recommends not linking it
> > this way; it is supposed to be statically linked into the executable.
> > Because of this, and because of the amount of custom patching we do to
> > maintain liblua as a shared library, I plan to stop creating the shared
> > library.
> > 
> 
> Please dont. Static linking is a security nightmare.
> 
> I'd much rather consider removing the static library, and fix programs broken 
> by that. (No matter what silly opinions lua upstream has.)

Here is what upstream says, so let me know how you interpret it.

http://www.lua.org/manual/5.3/readme.html

Also, there is this in our ebuilds:

# Using dynamic linked lua is not recommended for performance
# reasons. http://article.gmane.org/gmane.comp.lang.lua.general/18519
# Mainly, this is of concern if your arch is poor with GPRs, like x86
# Note that this only affects the interpreter binary (named lua), not the lua
# compiler (built statically) nor the lua libraries (both shared and static
# are installed)

It looks like the link is dead, so I don't have any idea what
performance issues they are talking about.

William


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] the state of dev-lang/lua

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 2261 bytes --]

On Sun, Mar 24, 2019 at 12:24:26PM +0100, contact@hacktivis.me wrote:
> [2019-03-23 16:23:27] William Hubbs:
> > The first thing is liblua as a shared library. If you are using lua
> > internally in a program, upstream strongly recommends not linking it
> > this way; it is supposed to be statically linked into the executable.
> > Because of this, and because of the amount of custom patching we do to
> > maintain liblua as a shared library, I plan to stop creating the shared
> > library.
> 
> Pretty sure removing the shared version of liblua will cause a large
> amount of patching to be done in a lot of software that may not support
> static linking out of the box.

I will look into this, but I think you are talking about build systems
that don't read pkgconfig files or something similar.

> 
> > I'm a bit undecided still about slotting lua. I'm sure we
> > need subslots so we can force rebuilds when new lua releases enter the
> > tree. However, I'm still unsure whether we need slots. I don't know of
> > many things in the tree that are locked to a specific version
> > of lua (there is only one package based on an irc conversation I
> > had this week).
> > Does anyone have any thoughts? Are there more packages in the tree that
> > are locked to a specific version of lua?
> 
> There should be slotting of lua, for example in the few programs that
> I know will need it:
> 
> - app-editors/vis::{lua,lanodanOverlay}: Requires >=lua-5.2
> 	does not support luajit (incomplete 5.2 compatibility)

This example isn't really a slot, just a lower bound. This is pretty
common.
 
> - media-video/mpv: requires lua:5.2 or luajit
> 	doesn’t seems to plan supporting 5.3[1]

This is more concerning, so I'll take a look at it.

> 
> Also the lua overlay, which I would highly recommend using as a base
> has a slotted version of lua, instead of extending SLOT="0".

There were many concerns about lua.eclass in the overlay when it was
presented here for inclusion in the tree, and it was never presented
again [1]. I also spoke with the previous maintainer of dev-lang/lua and
he was concerned about it.

William

[1] https://archives.gentoo.org/gentoo-dev/message/befce09046d0c80e3cbbee543a378f59

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] the state of dev-lang/lua

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2038 bytes --]

On Sun, 2019-03-24 at 12:44 -0500, William Hubbs wrote:
> On Sun, Mar 24, 2019 at 02:26:35PM +0100, Andreas K. Huettel wrote:
> > Am Samstag, 23. März 2019, 22:23:27 CET schrieb William Hubbs:
> > > Hi all,
> > > 
> > > Soon I will be working on fixing up the state of dev-lang/lua, and there
> > > are a couple of things I want to mention.
> > > 
> > > The first thing is liblua as a shared library. If you are using lua
> > > internally in a program, upstream strongly recommends not linking it
> > > this way; it is supposed to be statically linked into the executable.
> > > Because of this, and because of the amount of custom patching we do to
> > > maintain liblua as a shared library, I plan to stop creating the shared
> > > library.
> > > 
> > 
> > Please dont. Static linking is a security nightmare.
> > 
> > I'd much rather consider removing the static library, and fix programs broken 
> > by that. (No matter what silly opinions lua upstream has.)
> 
> Here is what upstream says, so let me know how you interpret it.
> 
> http://www.lua.org/manual/5.3/readme.html

They don't provide any rationale, and they say to use shared libraries
on Windows.  To me, it looks like they didn't want to figure out how to
build shared libs for various unix flavors, and took the easy way out.

> Also, there is this in our ebuilds:
> 
> # Using dynamic linked lua is not recommended for performance
> # reasons. http://article.gmane.org/gmane.comp.lang.lua.general/18519
> # Mainly, this is of concern if your arch is poor with GPRs, like x86
> # Note that this only affects the interpreter binary (named lua), not the lua
> # compiler (built statically) nor the lua libraries (both shared and static
> # are installed)
> 
> It looks like the link is dead, so I don't have any idea what
> performance issues they are talking about.
> 

Probably the performance loss resulting from use of -fPIC.  Which we're
forcing anyway nowadays, so no loss to us.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: [gentoo-dev] the state of dev-lang/lua

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 5370 bytes --]

On Sat, Mar 23, 2019 at 04:23:27PM -0500, William Hubbs wrote:
> Hi all,
> 
> Soon I will be working on fixing up the state of dev-lang/lua, and there
> are a couple of things I want to mention.
> 
> The first thing is liblua as a shared library. If you are using lua
> internally in a program, upstream strongly recommends not linking it
> this way; it is supposed to be statically linked into the executable.
> Because of this, and because of the amount of custom patching we do to
> maintain liblua as a shared library, I plan to stop creating the shared
> library.
Please don't go back to static libraries. Look at the other major
distros, all of them shipped shared Lua as the primary method.

I'm just going to link to the Ubuntu examples here, but Debian, Fedora &
Arch are similar:
https://packages.ubuntu.com/source/cosmic/lua5.1
https://packages.ubuntu.com/source/cosmic/lua5.2
https://packages.ubuntu.com/source/cosmic/lua5.3
https://packages.ubuntu.com/cosmic/amd64/liblua5.1-0/filelist
https://packages.ubuntu.com/cosmic/amd64/liblua5.2-0/filelist
https://packages.ubuntu.com/cosmic/amd64/liblua5.3-0/filelist

Ubuntu & Debian DO ship the static library .a file, as part of
the liblua5.X-dev binary package.

That said, the pkgconfig files we output for Lua5.2 & Lua5.3 do have
errors in some cases, bringing in the non-slotted include directory, and
trying to use the non-slotted library soname; which need to be fixed
properly.

Upstream's original objections about static libraries mostly revolved
around the lack of registers on i386 with -fPIC, as well as subtle
breakage when the library SONAME was just generically 'liblua', without
much version information, as this broke when functions were dropped.

> I'm a bit undecided still about slotting lua. I'm sure we
> need subslots so we can force rebuilds when new lua releases enter the
> tree. However, I'm still unsure whether we need slots. I don't know of
> many things in the tree that are locked to a specific version
> of lua (there is only one package based on an irc conversation I
> had this week).
> Does anyone have any thoughts? 
Lua needs first class slots, just like Python & Ruby, not just subslots.
Changing between versions can be a major undertaking.

I think the slots to start with should be:
- lua5.1
- lua5.2
- lua5.3
- luajit5.1 (this is basically an alternative implementation of Lua5.1,
much like pypy implements Python2).

Porting between versions is non-trivial, as the API has changed in
incompatible ways.

There are two tricks to make porting easier, but they are not
guaranteed. 
1. USE=deprecated on Lua5.2/5.3 offers most of the prior version APIs;
   mostly from Lua5.3 back to Lua5.2 & Lua5.1 with the LUA_COMPAT_5_2 &
   LUA_COMPAT_5_1 compile-time defines.
2. In the other direction, there is 'lua-compat-5.3', which we don't
   have packaged; It offers parts of Lua-5.3 functionality to
   Lua-5.1/5.2. I'll quote from the docs:
   "This does not make Lua 5.2 (or even Lua 5.1) entirely compatible
   with Lua 5.3, but it brings the API closer to that of Lua 5.3."
   That project maintains a good list of what's not implemented:
   https://github.com/keplerproject/lua-compat-5.3#whats-not-implemented

> Are there more packages in the tree that are locked to a specific version of lua?
Yes, several, and more on that in a second.

I think packages will need one of two variables:
LUA_SINGLE_TARGET
LUA_TARGETS

LUA_TARGETS is probably only going to be used by the dev-lua/ modules.
LUA_SINGLE_TARGET is probably going to be used by MOST packages that
include Lua as an embedded scripting language; of which I'll discuss a
few below.

The pain in Lua thusfar I see comes when one of the packages with
embedded Lua scripting needs to consume a system-wide Lua module.

Some Lua modules provide special-case handling for luajit, to ensure
best performance.

I personally work on net-proxy/haproxy, which supports ONLY Lua 5.3. It
relies on some functionality that cannot be back-ported to earlier
releases (heavy usage of coroutine/yield & uint64)

I'm also about to add back sys-apps/likwid to the tree, for which the
upstream bundles Lua 5.3, but the code explicitly is written to support
a system-wide copy of Lua 5.2 or Lua 5.3. The ebuild now uses the
system-wide copy of Lua 5.3. likwid ships as both a set of Lua scripts
that bring in a special C library, as well as Lua scripting placed back
into that C library.

x11-wm/awesome is a good example of a package with support for multiple
versions of Lua. For a long time upstream only supported Lua 5.1, but in
the recent major versions of Awesome, the upstream dev has Lua 5.2 & Lua
5.3 support with a test environment. I'm running on my own local ebuild
with Awesome linked against Lua 5.3 as I write this.

dev-db/redis is a good example of where Upstream tried to port from 5.1
to newer versions, but did an incomplete job:
- Try to compile it with Lua5.3 and it falls over numeric types.
- Try to compile it with Lua5.2 and it falls over the
  lua_open/lua_newstate.

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail   : robbat2@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1113 bytes --]

Re: [gentoo-dev] the state of dev-lang/lua

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1089 bytes --]

On Mon, 2019-03-25 at 04:23 +0000, Robin H. Johnson wrote:
> > Are there more packages in the tree that are locked to a specific version of lua?
> Yes, several, and more on that in a second.
> 
> I think packages will need one of two variables:
> LUA_SINGLE_TARGET
> LUA_TARGETS
> 
> LUA_TARGETS is probably only going to be used by the dev-lua/ modules.
> LUA_SINGLE_TARGET is probably going to be used by MOST packages that
> include Lua as an embedded scripting language; of which I'll discuss a
> few below.
> 

Please don't copy the PYTHON_* insanity here.  The only reason for
PYTHON_SINGLE_TARGET, as I keep repeating, is that we *needed* a sane
default of py2+py3 for a lot of packages, without causing REQUIRED_USE
violations all over the place.

I don't think Lua situation is similar.  That is, I don't think you can
pinpoint a combination of 2 or more default 'targets' and a single
'target' that would work for the majority of packages.  Just use
a single USE_EXPAND combined with REQUIRED_USE and sane defaults.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: [gentoo-dev] the state of dev-lang/lua

[Patrick McLean]

On Mon, 25 Mar 2019 04:23:08 +0000
"Robin H. Johnson" <robbat2@gentoo.org> wrote:

> On Sat, Mar 23, 2019 at 04:23:27PM -0500, William Hubbs wrote:
> > Hi all,
> > 
> > Soon I will be working on fixing up the state of dev-lang/lua, and
> > there are a couple of things I want to mention.
> > 
> > The first thing is liblua as a shared library. If you are using lua
> > internally in a program, upstream strongly recommends not linking it
> > this way; it is supposed to be statically linked into the
> > executable. Because of this, and because of the amount of custom
> > patching we do to maintain liblua as a shared library, I plan to
> > stop creating the shared library.  
> Please don't go back to static libraries. Look at the other major
> distros, all of them shipped shared Lua as the primary method.

+1

> 
> > I'm a bit undecided still about slotting lua. I'm sure we
> > need subslots so we can force rebuilds when new lua releases enter
> > the tree. However, I'm still unsure whether we need slots. I don't
> > know of many things in the tree that are locked to a specific
> > version of lua (there is only one package based on an irc
> > conversation I had this week).
> > Does anyone have any thoughts?   
> Lua needs first class slots, just like Python & Ruby, not just
> subslots. Changing between versions can be a major undertaking.
> 
> I think the slots to start with should be:
> - lua5.1
> - lua5.2
> - lua5.3
> - luajit5.1 (this is basically an alternative implementation of
> Lua5.1, much like pypy implements Python2).

I think we are going to have to have slots for the "openresty" lua
fork here as well. Several nginx modules require this version to work
properly (I can provide more details if needed).