[gentoo-dev] LFS QA warnings coming soon to a build near you

[gentoo-dev] LFS QA warnings coming soon to a build near you

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 2124 bytes --]

we've got a new QA check that warns whenever a package is built using a 32bit 
filesystem interface.  in practice, this applies to arm/mips/ppc/sh/x86 systems
(not including multilib -- for now).

this topic has come up in Gentoo a few times over the years but we've never 
really amassed the will power to fix it.  instead we fix it in one-offs based 
on user reports (like "can't download 4GiB file with ftp" #101038).  this was 
worked well enough because most users have moved on to 64bit systems and the 
interaction with >2GiB files tends to correlate with a few packages.

however, "recent" winds have started blowing where file systems are utilizing
64bit inodes to handle large file counts.  this means apps that do even basic
things like stat() will now fail.  the number of applications that this affects
is significantly higher, although still relegated to systems that happen to use
a file system with 64bit inodes.

since our QA warnings in portage have historically proven themselves to (1) get
users to generate bug reports and (2) get the warnings fixed (both here and in
upstream), there is now a QA check for this issue.  the plan is to let this bake 
in portage-9999 for a while before it gets kicked into ~arch and the flood gates
are unleashed.

more background details (including recipes for fixing the issue in your package)
can be found in the tracker:
	https://bugs.gentoo.org/471102

an example warning message:
 * QA Notice: The following files were not built with LFS support:
 *   Please file a bug at http://bugs.gentoo.org/ and mark it as a blocker of 471102.
 *   See that tracker bug (https://bugs.gentoo.org/471102) for more details.
 * fopen@@GLIBC_2.1 bin/route
 * fopen@@GLIBC_2.1 bin/ifconfig
 * fopen@@GLIBC_2.1 bin/hostname
 * __fxstat@@GLIBC_2.0,open@@GLIBC_2.0,fopen@@GLIBC_2.1,readdir@@GLIBC_2.0 bin/netstat
 * fopen@@GLIBC_2.1 sbin/nameif
 * fopen@@GLIBC_2.1 sbin/ipmaddr
 * fopen@@GLIBC_2.1 sbin/arp
 * fopen@@GLIBC_2.1 sbin/iptunnel
 * fopen@@GLIBC_2.1 sbin/rarp
 * open@@GLIBC_2.0,creat@@GLIBC_2.0,fopen@@GLIBC_2.1 sbin/slattach
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] LFS QA warnings coming soon to a build near you

[Alexis Ballier]

On Sat, 30 May 2015 14:54:42 -0400
Mike Frysinger <vapier@gentoo.org> wrote:

> an example warning message:
>  * QA Notice: The following files were not built with LFS support:
>  *   Please file a bug at http://bugs.gentoo.org/ and mark it as a
> blocker of 471102.
>  *   See that tracker bug (https://bugs.gentoo.org/471102) for more
> details.
>  * fopen@@GLIBC_2.1 bin/route
>  * fopen@@GLIBC_2.1 bin/ifconfig
>  * fopen@@GLIBC_2.1 bin/hostname
>  *
> __fxstat@@GLIBC_2.0,open@@GLIBC_2.0,fopen@@GLIBC_2.1,readdir@@GLIBC_2.0
> bin/netstat
>  * fopen@@GLIBC_2.1 sbin/nameif
>  * fopen@@GLIBC_2.1 sbin/ipmaddr
>  * fopen@@GLIBC_2.1 sbin/arp
>  * fopen@@GLIBC_2.1 sbin/iptunnel
>  * fopen@@GLIBC_2.1 sbin/rarp
>  * open@@GLIBC_2.0,creat@@GLIBC_2.0,fopen@@GLIBC_2.1 sbin/slattach


nice, but can't we add the lfs flags to our default toolchain flags or
even better patch glibc headers to always redefine these functions to
the 64bits variants?

I don't understand why one should add append-lfs-flags to almost every
single package out there. Freebsd has been doing this since version 7.
And even dropped the 32bits symbols when going to libc.so.7 I think.

Some packages might break with such a change, but they're already
half-broken according to your e-mail, so those are the ones that should
be fixed.


Alexis.

Re: [gentoo-dev] LFS QA warnings coming soon to a build near you

[Diego Elio Pettenò]

[-- Attachment #1: Type: text/plain, Size: 936 bytes --]

On 31 May 2015 at 12:59, Alexis Ballier <aballier@gentoo.org> wrote:

> nice, but can't we add the lfs flags to our default toolchain flags or
> even better patch glibc headers to always redefine these functions to
> the 64bits variants?
>

No, because that can easily break ABI of programs that actually want the
non-LFS one (for instance anything that wraps around function calls,
including but not limited to padsp, aoss, and similar wrappers.) FreeBSD
has removed the symbols on an ABI bump, which hopefully could happen for
glibc in a far and remote future, too. But as long as the symbols are
there, the defines shouldn't be forced.

Mike, thanks for doing this, it has been a pain in my shoe since 2008
<https://blog.flameeyes.eu/2008/11/who-wants-to-support-largefile>. I'm
looking forward to the protests that 64-bit inodes don't exist, though.

Diego Elio Pettenò — Flameeyes
https://blog.flameeyes.eu/

[-- Attachment #2: Type: text/html, Size: 1518 bytes --]

Re: [gentoo-dev] LFS QA warnings coming soon to a build near you

[Philip Webb]

150531 Diego Elio Pettenò wrote:
> Mike, thanks for doing this, it has been a pain in my shoe since 2008
> <https://blog.flameeyes.eu/2008/11/who-wants-to-support-largefile>.

Do users with 64-bit systems have to pay attention to this ?

-- 
========================,,============================================
SUPPORT     ___________//___,   Philip Webb
ELECTRIC   /] [] [] [] [] []|   Cities Centre, University of Toronto
TRANSIT    `-O----------O---'   purslowatchassdotutorontodotca

Re: [gentoo-dev] LFS QA warnings coming soon to a build near you

[Diego Elio Pettenò]

[-- Attachment #1: Type: text/plain, Size: 323 bytes --]

On 31 May 2015 at 14:33, Philip Webb <purslow@ca.inter.net> wrote:

> Do users with 64-bit systems have to pay attention to this ?
>

Only as far as multilib is concerned. The 64-bit ABIs are (AFAIR) all LFS
with no opt-out. x86-64 is definitely LFS.

Diego Elio Pettenò — Flameeyes
https://blog.flameeyes.eu/

[-- Attachment #2: Type: text/html, Size: 876 bytes --]

Re: [gentoo-dev] LFS QA warnings coming soon to a build near you

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 403 bytes --]

On 31 May 2015 12:59, Alexis Ballier wrote:
> I don't understand why one should add append-lfs-flags to almost every
> single package out there.

no one is suggesting that route.  append-lfs-flags is a kludge (albeit an 
effective one) in the face of no upstream support.  if you look at the 
referenced tracker it includes pointers for fixing packages in ways that
can be sent upstream.
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] LFS QA warnings coming soon to a build near you

[Alexis Ballier]

On Sun, 31 May 2015 13:50:49 +0200
Diego Elio Pettenò <flameeyes@flameeyes.eu> wrote:

> On 31 May 2015 at 12:59, Alexis Ballier <aballier@gentoo.org> wrote:
> 
> > nice, but can't we add the lfs flags to our default toolchain flags
> > or even better patch glibc headers to always redefine these
> > functions to the 64bits variants?
> >
> 
> No, because that can easily break ABI of programs that actually want
> the non-LFS one (for instance anything that wraps around function
> calls, including but not limited to padsp, aoss, and similar
> wrappers.)

This seems easily fixed with an opt-out for lfs flags that such
programs can use. They'll need to be touched to disable the QA warning
anyway.

Re: [gentoo-dev] LFS QA warnings coming soon to a build near you

[Alexis Ballier]

On Sun, 31 May 2015 09:46:30 -0400
Mike Frysinger <vapier@gentoo.org> wrote:

> On 31 May 2015 12:59, Alexis Ballier wrote:
> > I don't understand why one should add append-lfs-flags to almost
> > every single package out there.
> 
> no one is suggesting that route.  append-lfs-flags is a kludge
> (albeit an effective one) in the face of no upstream support.  if you
> look at the referenced tracker it includes pointers for fixing
> packages in ways that can be sent upstream.

well, other suggestions are similar, except of course that appending
the flags is done upstream in some way or another

however, reading more in details the links you provided, what happened
to your patch setting _FILE_OFFSET_BITS default to 64 ?
( https://sourceware.org/ml/libc-alpha/2014-03/msg00290.html )
which seems to be a much better solution to what i initially proposed.

Re: [gentoo-dev] LFS QA warnings coming soon to a build near you

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 1228 bytes --]

On 31 May 2015 15:52, Alexis Ballier wrote:
> On Sun, 31 May 2015 13:50:49 +0200 Diego Elio Pettenò wrote:
> > On 31 May 2015 at 12:59, Alexis Ballier wrote:
> > > nice, but can't we add the lfs flags to our default toolchain flags
> > > or even better patch glibc headers to always redefine these
> > > functions to the 64bits variants?
> > 
> > No, because that can easily break ABI of programs that actually want
> > the non-LFS one (for instance anything that wraps around function
> > calls, including but not limited to padsp, aoss, and similar
> > wrappers.)
> 
> This seems easily fixed with an opt-out for lfs flags that such
> programs can use. They'll need to be touched to disable the QA warning
> anyway.

this is a discussion for upstream toolchain packages (largely glibc) and in fact 
i started such a heretical thread over a year ago.  it was not well received due
to the implicit/silent ABI change that new builds would receive.  glibc likes to
be conservative as it is the foundation of everything.

so unless glibc changes, updating our copy of glibc would only somewhat help our
users.  conversely, getting the changes pushed to the respective upstream would 
help everyone.
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] LFS QA warnings coming soon to a build near you

[Alexis Ballier]

On Sun, 31 May 2015 10:17:02 -0400
Mike Frysinger <vapier@gentoo.org> wrote:

> On 31 May 2015 15:52, Alexis Ballier wrote:
> > On Sun, 31 May 2015 13:50:49 +0200 Diego Elio Pettenò wrote:
> > > On 31 May 2015 at 12:59, Alexis Ballier wrote:
> > > > nice, but can't we add the lfs flags to our default toolchain
> > > > flags or even better patch glibc headers to always redefine
> > > > these functions to the 64bits variants?
> > > 
> > > No, because that can easily break ABI of programs that actually
> > > want the non-LFS one (for instance anything that wraps around
> > > function calls, including but not limited to padsp, aoss, and
> > > similar wrappers.)
> > 
> > This seems easily fixed with an opt-out for lfs flags that such
> > programs can use. They'll need to be touched to disable the QA
> > warning anyway.
> 
> this is a discussion for upstream toolchain packages (largely glibc)
> and in fact i started such a heretical thread over a year ago.  it
> was not well received due to the implicit/silent ABI change that new
> builds would receive.  glibc likes to be conservative as it is the
> foundation of everything.
> 
> so unless glibc changes, updating our copy of glibc would only
> somewhat help our users.  conversely, getting the changes pushed to
> the respective upstream would help everyone.


I'm not sure what's best for every one:
1. Push hundreds of patches upstream to add lfs flags;
2. Apply your patch to our glibc ebuilds, fix the corner cases, and go
  back to glibc upstream with these data.

Least to say is that I'm not a big fan of lfs flags as done in glibc and
I certainly prefer 2., but you'd know better.


Alexis.

Re: [gentoo-dev] LFS QA warnings coming soon to a build near you

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 2893 bytes --]

On 31 May 2015 16:33, Alexis Ballier wrote:
> On Sun, 31 May 2015 10:17:02 -0400 Mike Frysinger wrote:
> > On 31 May 2015 15:52, Alexis Ballier wrote:
> > > On Sun, 31 May 2015 13:50:49 +0200 Diego Elio Pettenò wrote:
> > > > On 31 May 2015 at 12:59, Alexis Ballier wrote:
> > > > > nice, but can't we add the lfs flags to our default toolchain
> > > > > flags or even better patch glibc headers to always redefine
> > > > > these functions to the 64bits variants?
> > > > 
> > > > No, because that can easily break ABI of programs that actually
> > > > want the non-LFS one (for instance anything that wraps around
> > > > function calls, including but not limited to padsp, aoss, and
> > > > similar wrappers.)
> > > 
> > > This seems easily fixed with an opt-out for lfs flags that such
> > > programs can use. They'll need to be touched to disable the QA
> > > warning anyway.
> > 
> > this is a discussion for upstream toolchain packages (largely glibc)
> > and in fact i started such a heretical thread over a year ago.  it
> > was not well received due to the implicit/silent ABI change that new
> > builds would receive.  glibc likes to be conservative as it is the
> > foundation of everything.
> > 
> > so unless glibc changes, updating our copy of glibc would only
> > somewhat help our users.  conversely, getting the changes pushed to
> > the respective upstream would help everyone.
> 
> I'm not sure what's best for every one:
> 1. Push hundreds of patches upstream to add lfs flags;
> 2. Apply your patch to our glibc ebuilds, fix the corner cases, and go
>   back to glibc upstream with these data.
> 
> Least to say is that I'm not a big fan of lfs flags as done in glibc and
> I certainly prefer 2., but you'd know better.

well if we're going to do arbitrary lists ;)
(1) your options aren't mutually exclusive
(2) implementing both are desirable
(3) considering the glibc effort has been stalled for over a year, (1) is 
something we can help accomplish and make reasonable progress on
(4) glibc isn't the only one that implements LFS in a transparent backwards 
compatible manner

which leads me to the next part ... my first suggestion in the tracker is for 
autotool based projects to use AC_SYS_LARGEFILE because:
(a) it supports a variety of systems
(b) as new systems come up or bugs are found or whatever, the autoconf macro 
will improve and people eventually get those fixes for free
(c) it does it all transparently for you -- add the macro and you're done
(d) it fixes the package for all users, new & old

the reason i listed only the raw CPPFLAGS and autoconf macros are because those 
are the two i'm familiar with.  i don't know how other build systems (e.g. 
cmake) support this (assuming they do at all).  if people have other recipes on 
hand, then it would be great to collect more there.
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] LFS QA warnings coming soon to a build near you

[Mike Gilbert]

On Sat, May 30, 2015 at 2:54 PM, Mike Frysinger <vapier@gentoo.org> wrote:
> we've got a new QA check that warns whenever a package is built using a 32bit
> filesystem interface.  in practice, this applies to arm/mips/ppc/sh/x86 systems
> (not including multilib -- for now).
>
> this topic has come up in Gentoo a few times over the years but we've never
> really amassed the will power to fix it.  instead we fix it in one-offs based
> on user reports (like "can't download 4GiB file with ftp" #101038).  this was
> worked well enough because most users have moved on to 64bit systems and the
> interaction with >2GiB files tends to correlate with a few packages.
>
> however, "recent" winds have started blowing where file systems are utilizing
> 64bit inodes to handle large file counts.  this means apps that do even basic
> things like stat() will now fail.  the number of applications that this affects
> is significantly higher, although still relegated to systems that happen to use
> a file system with 64bit inodes.

Does this issue affect software that only reads/writes small files and
never calls stat? For example, pkg-config.

It might still be nice to adjust such packages for consistency, but it
might be harder to justify patches to upstream developers.

Re: [gentoo-dev] LFS QA warnings coming soon to a build near you

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 2406 bytes --]

On 31 May 2015 11:58, Mike Gilbert wrote:
> On Sat, May 30, 2015 at 2:54 PM, Mike Frysinger wrote:
> > we've got a new QA check that warns whenever a package is built using a 32bit
> > filesystem interface.  in practice, this applies to arm/mips/ppc/sh/x86 systems
> > (not including multilib -- for now).
> >
> > this topic has come up in Gentoo a few times over the years but we've never
> > really amassed the will power to fix it.  instead we fix it in one-offs based
> > on user reports (like "can't download 4GiB file with ftp" #101038).  this was
> > worked well enough because most users have moved on to 64bit systems and the
> > interaction with >2GiB files tends to correlate with a few packages.
> >
> > however, "recent" winds have started blowing where file systems are utilizing
> > 64bit inodes to handle large file counts.  this means apps that do even basic
> > things like stat() will now fail.  the number of applications that this affects
> > is significantly higher, although still relegated to systems that happen to use
> > a file system with 64bit inodes.
> 
> Does this issue affect software that only reads/writes small files and
> never calls stat? For example, pkg-config.

it is possible to use some file system operations and never run into problems.
the trouble is that it requires diligence and careful auditing, and you have to 
make sure your interactions with other packages are also OK.  considering how 
bad/unaware the wider community is wrt LFS, ABI compatibility, and such critical 
but oft overlooked issues, i suspect most people also won't have the will 
power/capabilities to keep up here.

conversely, flip the autoconf/cppflags magic, and most projects Just Work.

> It might still be nice to adjust such packages for consistency, but it
> might be harder to justify patches to upstream developers.

pkg-config already merged it and it's already been pulled back into Gentoo ;)
	https://bugs.freedesktop.org/show_bug.cgi?id=90078
	https://bugs.gentoo.org/550508

that said, i'm not entirely sure why there would be a lot of push back.
when i gathered data on this years ago (which i no longer have, so i'd have
to recreate), the increased foot print and runtime cost was not significant.
especially when you consider that we're talking about I/O operations which
inherently kill CPU performance due to stalls.
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] LFS QA warnings coming soon to a build near you

[Mike Gilbert]

On Sun, May 31, 2015 at 12:50 PM, Mike Frysinger <vapier@gentoo.org> wrote:
> On 31 May 2015 11:58, Mike Gilbert wrote:
>> It might still be nice to adjust such packages for consistency, but it
>> might be harder to justify patches to upstream developers.
>
> pkg-config already merged it and it's already been pulled back into Gentoo ;)
>         https://bugs.freedesktop.org/show_bug.cgi?id=90078
>         https://bugs.gentoo.org/550508
>
> that said, i'm not entirely sure why there would be a lot of push back.
> when i gathered data on this years ago (which i no longer have, so i'd have
> to recreate), the increased foot print and runtime cost was not significant.
> especially when you consider that we're talking about I/O operations which
> inherently kill CPU performance due to stalls.
> -mike

Sure, I don't really expect much resistance. I just like to be
prepared in the event some developer starts asking questions.

Re: [gentoo-dev] LFS QA warnings coming soon to a build near you

[Alexis Ballier]

On Sun, 31 May 2015 11:17:50 -0400
Mike Frysinger <vapier@gentoo.org> wrote:

> well if we're going to do arbitrary lists ;)
> (1) your options aren't mutually exclusive
> (2) implementing both are desirable

good to know your longterm plan :)
however, even if both can be done, i still don't see the point of going
through patching if we end up changing the default anyway.

> (3) considering the glibc effort has been stalled for over a year,
> (1) is something we can help accomplish and make reasonable progress
> on (4) glibc isn't the only one that implements LFS in a transparent
> backwards compatible manner

maybe the fact that some file operations are broken with glibc's
default settings on a somewhat popular fs would be a good argument to
un-stall it ?

> which leads me to the next part ... my first suggestion in the
> tracker is for autotool based projects to use AC_SYS_LARGEFILE
> because: (a) it supports a variety of systems
> (b) as new systems come up or bugs are found or whatever, the
> autoconf macro will improve and people eventually get those fixes for
> free (c) it does it all transparently for you -- add the macro and
> you're done (d) it fixes the package for all users, new & old
> 
> the reason i listed only the raw CPPFLAGS and autoconf macros are
> because those are the two i'm familiar with.  i don't know how other
> build systems (e.g. cmake) support this (assuming they do at all).
> if people have other recipes on hand, then it would be great to
> collect more there. -mike


yes, but that is not what i am discussing: unless i missed something,
they'll all end up one way or another adding the relevant defines;
whether it is done with an m4 macro, append-lfs-flags, a cmake function
or what else is an implementation detail of little interest to me trying
to understand why we don't simply change the default :)

Alexis.

Re: [gentoo-dev] LFS QA warnings coming soon to a build near you

[Christopher Head]

On May 31, 2015 7:33:28 AM PDT, Alexis Ballier <aballier@gentoo.org> wrote:
>
>I'm not sure what's best for every one:
>1. Push hundreds of patches upstream to add lfs flags;
>2. Apply your patch to our glibc ebuilds, fix the corner cases, and go
>  back to glibc upstream with these data.
>

If the changes are made to glibc, would these be under a new symbol version for ABI compatibility, or just be changes to headers to make _FILE_OFFSET_BITS=64 the default? If not, what about binary software? Not saying you haven’t considered the relevant issues; I just haven’t seen binary software brought up on this list yet.

-- 
Christopher Head

Re: [gentoo-dev] LFS QA warnings coming soon to a build near you

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 3096 bytes --]

On 01 Jun 2015 10:15, Alexis Ballier wrote:
> On Sun, 31 May 2015 11:17:50 -0400 Mike Frysinger wrote:
> > (3) considering the glibc effort has been stalled for over a year,
> > (1) is something we can help accomplish and make reasonable progress
> > on (4) glibc isn't the only one that implements LFS in a transparent
> > backwards compatible manner
> 
> maybe the fact that some file operations are broken with glibc's
> default settings on a somewhat popular fs would be a good argument to
> un-stall it ?

it isn't that black & white.  while for many packages the change would be 
harmless, glibc is more conservative than that.  it can be shown that a not 
uncommon coding style is to use long's everywhere when dealing with the fs.
when sizeof(off_t)==sizeof(long), this works out.  when off_t is suddenly 
twice as large, then you can get truncation at best (which is kind of the
status quo) and random corruption at worse (like an API that takes pointers
to off_t's but the caller passed in a long).  if you want to join the list
upstream and contribute to the discussion, you're more than welcome to.  the
maintainers aren't clueless -- they fully understand the trade-offs we're
discussing here and know that 32bit fs accesses lead to failures (including
stat).

since sitting on our hands and hoping for the best continues to accomplish
nothing, getting the various upstream packages to opt-in to LFS themselves
can make real progress now.

> > which leads me to the next part ... my first suggestion in the
> > tracker is for autotool based projects to use AC_SYS_LARGEFILE
> > because: (a) it supports a variety of systems
> > (b) as new systems come up or bugs are found or whatever, the
> > autoconf macro will improve and people eventually get those fixes for
> > free (c) it does it all transparently for you -- add the macro and
> > you're done (d) it fixes the package for all users, new & old
> > 
> > the reason i listed only the raw CPPFLAGS and autoconf macros are
> > because those are the two i'm familiar with.  i don't know how other
> > build systems (e.g. cmake) support this (assuming they do at all).
> > if people have other recipes on hand, then it would be great to
> > collect more there.
> 
> yes, but that is not what i am discussing: unless i missed something,
> they'll all end up one way or another adding the relevant defines;
> whether it is done with an m4 macro, append-lfs-flags, a cmake function
> or what else is an implementation detail of little interest to me trying
> to understand why we don't simply change the default :)

as i already said, changing the default in *glibc* doesn't help non-glibc 
systems.  using portable build systems (like autotools) lets you easily 
enable features like this and the build system knows the various platform
details.  if you decided to roll your own build system or use one that
isn't that portable, well that's kind of your fault isn't it ?  if you've
opted to only support a subset of systems, then that means adding these
CPPFLAGS yourself.
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] LFS QA warnings coming soon to a build near you

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 924 bytes --]

On 01 Jun 2015 09:51, Christopher Head wrote:
> On May 31, 2015 7:33:28 AM PDT, Alexis Ballier wrote:
> >I'm not sure what's best for every one:
> >1. Push hundreds of patches upstream to add lfs flags;
> >2. Apply your patch to our glibc ebuilds, fix the corner cases, and go
> >  back to glibc upstream with these data.
>
> If the changes are made to glibc, would these be under a new symbol version 
> for ABI compatibility, or just be changes to headers to make 
> _FILE_OFFSET_BITS=64 the default? If not, what about binary software? Not 
> saying you haven’t considered the relevant issues; I just haven’t seen binary 
> software brought up on this list yet.

regardless of what the headers export as a default, glibc's ABI would not 
change.  the CPPFLAGS merely control which set of symbols are used.  any
existing binary packages would continue to operate the same way they always
have.
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] LFS QA warnings coming soon to a build near you

[Alexis Ballier]

On Tue, 2 Jun 2015 10:13:54 -0400
Mike Frysinger <vapier@gentoo.org> wrote:

> On 01 Jun 2015 10:15, Alexis Ballier wrote:
> > On Sun, 31 May 2015 11:17:50 -0400 Mike Frysinger wrote:
> > > (3) considering the glibc effort has been stalled for over a year,
> > > (1) is something we can help accomplish and make reasonable
> > > progress on (4) glibc isn't the only one that implements LFS in a
> > > transparent backwards compatible manner
> > 
> > maybe the fact that some file operations are broken with glibc's
> > default settings on a somewhat popular fs would be a good argument
> > to un-stall it ?
> 
> it isn't that black & white.  while for many packages the change
> would be harmless, glibc is more conservative than that.  it can be
> shown that a not uncommon coding style is to use long's everywhere
> when dealing with the fs. when sizeof(off_t)==sizeof(long), this
> works out.  when off_t is suddenly twice as large, then you can get
> truncation at best (which is kind of the status quo) and random
> corruption at worse (like an API that takes pointers to off_t's but
> the caller passed in a long).  if you want to join the list upstream
> and contribute to the discussion, you're more than welcome to.  the
> maintainers aren't clueless -- they fully understand the trade-offs
> we're discussing here and know that 32bit fs accesses lead to
> failures (including stat).
> 
> since sitting on our hands and hoping for the best continues to
> accomplish nothing, getting the various upstream packages to opt-in
> to LFS themselves can make real progress now.


agreed, thanks for the explanation

i take it as the main goal of these qa warnings is to fix the above
mentioned half broken packages and that pushing lfs flags everywhere is
a slow but safe way to achieve it :)

it'd be interesting to have data about how many packages can break;
my belief is that more than half the c/c++ programs use one way or
another fs ops but maybe less than one in a thousand will break; i can
understand this is still too much for a libc (even though other
'breaking fixes' weren't so controversial in the past), and we'll see
after gathering the data your qa warnings will provide

> > > which leads me to the next part ... my first suggestion in the
> > > tracker is for autotool based projects to use AC_SYS_LARGEFILE
> > > because: (a) it supports a variety of systems
> > > (b) as new systems come up or bugs are found or whatever, the
> > > autoconf macro will improve and people eventually get those fixes
> > > for free (c) it does it all transparently for you -- add the
> > > macro and you're done (d) it fixes the package for all users, new
> > > & old
> > > 
> > > the reason i listed only the raw CPPFLAGS and autoconf macros are
> > > because those are the two i'm familiar with.  i don't know how
> > > other build systems (e.g. cmake) support this (assuming they do
> > > at all). if people have other recipes on hand, then it would be
> > > great to collect more there.
> > 
> > yes, but that is not what i am discussing: unless i missed
> > something, they'll all end up one way or another adding the
> > relevant defines; whether it is done with an m4 macro,
> > append-lfs-flags, a cmake function or what else is an
> > implementation detail of little interest to me trying to understand
> > why we don't simply change the default :)
> 
> as i already said, changing the default in *glibc* doesn't help
> non-glibc systems.

even with ten different libc's, the order of magnitude is still smaller
for changing the default :)


Alexis.

Re: [gentoo-dev] LFS QA warnings coming soon to a build near you

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 3683 bytes --]

On 03 Jun 2015 10:26, Alexis Ballier wrote:
> On Tue, 2 Jun 2015 10:13:54 -0400 Mike Frysinger wrote:
> > On 01 Jun 2015 10:15, Alexis Ballier wrote:
> > > On Sun, 31 May 2015 11:17:50 -0400 Mike Frysinger wrote:
> > > > (3) considering the glibc effort has been stalled for over a year,
> > > > (1) is something we can help accomplish and make reasonable
> > > > progress on (4) glibc isn't the only one that implements LFS in a
> > > > transparent backwards compatible manner
> > > 
> > > maybe the fact that some file operations are broken with glibc's
> > > default settings on a somewhat popular fs would be a good argument
> > > to un-stall it ?
> > 
> > it isn't that black & white.  while for many packages the change
> > would be harmless, glibc is more conservative than that.  it can be
> > shown that a not uncommon coding style is to use long's everywhere
> > when dealing with the fs. when sizeof(off_t)==sizeof(long), this
> > works out.  when off_t is suddenly twice as large, then you can get
> > truncation at best (which is kind of the status quo) and random
> > corruption at worse (like an API that takes pointers to off_t's but
> > the caller passed in a long).  if you want to join the list upstream
> > and contribute to the discussion, you're more than welcome to.  the
> > maintainers aren't clueless -- they fully understand the trade-offs
> > we're discussing here and know that 32bit fs accesses lead to
> > failures (including stat).
> > 
> > since sitting on our hands and hoping for the best continues to
> > accomplish nothing, getting the various upstream packages to opt-in
> > to LFS themselves can make real progress now.
> 
> agreed, thanks for the explanation
> 
> i take it as the main goal of these qa warnings is to fix the above
> mentioned half broken packages and that pushing lfs flags everywhere is
> a slow but safe way to achieve it :)
> 
> it'd be interesting to have data about how many packages can break;
> my belief is that more than half the c/c++ programs use one way or
> another fs ops but maybe less than one in a thousand will break; i can
> understand this is still too much for a libc (even though other
> 'breaking fixes' weren't so controversial in the past), and we'll see
> after gathering the data your qa warnings will provide

let's ignore the open/read/write issue for now on the assumption it's more 
uncommon to incidentally have that break.  instead, let's look at things that
can hit inodes -- stat & dir listing.  a file simply being allocated a 64bit
inode will break an app -- both directly stating and even trying to list the 
dir.  oops! :)

here's the command i'm using for just inode funcs:
	scanelf / -mqRygs \
		'^((__)?[fl]?x?stat(v?fs)?(at)?|alphasort|getdirentries|readdir(_r)?|scandir(at)?|versionsort)$' 
you'd have to ignore glibc and sandbox as they provide both 32bit & 64bit.

i have an x86 chroot for random testing, but it's not as "full" as my desktop.
it has a measily 245 packages installed.  of those:
 - 25 are using 32bit inode functions
 - 34 are using some 32bit fs interface (this is a superset of the above)

if someone had a fuller install of like >1000 pkgs, that'd be interesting.

> > as i already said, changing the default in *glibc* doesn't help
> > non-glibc systems.
> 
> even with ten different libc's, the order of magnitude is still smaller
> for changing the default :)

true.  each fight though will be significantly higher i think, assuming you can 
even find someone to talk to (as not all are open source projects).  and it will 
take longer for the fix to trickle out to deployments.
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]