* [gentoo-dev] LFS QA warnings coming soon to a build near you
@ 2015-05-30 18:54 Mike Frysinger
2015-05-31 10:59 ` Alexis Ballier
2015-05-31 15:58 ` Mike Gilbert
0 siblings, 2 replies; 20+ messages in thread
From: Mike Frysinger @ 2015-05-30 18:54 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 2124 bytes --]
we've got a new QA check that warns whenever a package is built using a 32bit
filesystem interface. in practice, this applies to arm/mips/ppc/sh/x86 systems
(not including multilib -- for now).
this topic has come up in Gentoo a few times over the years but we've never
really amassed the will power to fix it. instead we fix it in one-offs based
on user reports (like "can't download 4GiB file with ftp" #101038). this was
worked well enough because most users have moved on to 64bit systems and the
interaction with >2GiB files tends to correlate with a few packages.
however, "recent" winds have started blowing where file systems are utilizing
64bit inodes to handle large file counts. this means apps that do even basic
things like stat() will now fail. the number of applications that this affects
is significantly higher, although still relegated to systems that happen to use
a file system with 64bit inodes.
since our QA warnings in portage have historically proven themselves to (1) get
users to generate bug reports and (2) get the warnings fixed (both here and in
upstream), there is now a QA check for this issue. the plan is to let this bake
in portage-9999 for a while before it gets kicked into ~arch and the flood gates
are unleashed.
more background details (including recipes for fixing the issue in your package)
can be found in the tracker:
https://bugs.gentoo.org/471102
an example warning message:
* QA Notice: The following files were not built with LFS support:
* Please file a bug at http://bugs.gentoo.org/ and mark it as a blocker of 471102.
* See that tracker bug (https://bugs.gentoo.org/471102) for more details.
* fopen@@GLIBC_2.1 bin/route
* fopen@@GLIBC_2.1 bin/ifconfig
* fopen@@GLIBC_2.1 bin/hostname
* __fxstat@@GLIBC_2.0,open@@GLIBC_2.0,fopen@@GLIBC_2.1,readdir@@GLIBC_2.0 bin/netstat
* fopen@@GLIBC_2.1 sbin/nameif
* fopen@@GLIBC_2.1 sbin/ipmaddr
* fopen@@GLIBC_2.1 sbin/arp
* fopen@@GLIBC_2.1 sbin/iptunnel
* fopen@@GLIBC_2.1 sbin/rarp
* open@@GLIBC_2.0,creat@@GLIBC_2.0,fopen@@GLIBC_2.1 sbin/slattach
-mike
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [gentoo-dev] LFS QA warnings coming soon to a build near you
2015-05-30 18:54 [gentoo-dev] LFS QA warnings coming soon to a build near you Mike Frysinger
@ 2015-05-31 10:59 ` Alexis Ballier
2015-05-31 11:50 ` Diego Elio Pettenò
2015-05-31 13:46 ` Mike Frysinger
2015-05-31 15:58 ` Mike Gilbert
1 sibling, 2 replies; 20+ messages in thread
From: Alexis Ballier @ 2015-05-31 10:59 UTC (permalink / raw
To: Mike Frysinger; +Cc: gentoo-dev
On Sat, 30 May 2015 14:54:42 -0400
Mike Frysinger <vapier@gentoo.org> wrote:
> an example warning message:
> * QA Notice: The following files were not built with LFS support:
> * Please file a bug at http://bugs.gentoo.org/ and mark it as a
> blocker of 471102.
> * See that tracker bug (https://bugs.gentoo.org/471102) for more
> details.
> * fopen@@GLIBC_2.1 bin/route
> * fopen@@GLIBC_2.1 bin/ifconfig
> * fopen@@GLIBC_2.1 bin/hostname
> *
> __fxstat@@GLIBC_2.0,open@@GLIBC_2.0,fopen@@GLIBC_2.1,readdir@@GLIBC_2.0
> bin/netstat
> * fopen@@GLIBC_2.1 sbin/nameif
> * fopen@@GLIBC_2.1 sbin/ipmaddr
> * fopen@@GLIBC_2.1 sbin/arp
> * fopen@@GLIBC_2.1 sbin/iptunnel
> * fopen@@GLIBC_2.1 sbin/rarp
> * open@@GLIBC_2.0,creat@@GLIBC_2.0,fopen@@GLIBC_2.1 sbin/slattach
nice, but can't we add the lfs flags to our default toolchain flags or
even better patch glibc headers to always redefine these functions to
the 64bits variants?
I don't understand why one should add append-lfs-flags to almost every
single package out there. Freebsd has been doing this since version 7.
And even dropped the 32bits symbols when going to libc.so.7 I think.
Some packages might break with such a change, but they're already
half-broken according to your e-mail, so those are the ones that should
be fixed.
Alexis.
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [gentoo-dev] LFS QA warnings coming soon to a build near you
2015-05-31 10:59 ` Alexis Ballier
@ 2015-05-31 11:50 ` Diego Elio Pettenò
2015-05-31 12:33 ` Philip Webb
2015-05-31 13:52 ` Alexis Ballier
2015-05-31 13:46 ` Mike Frysinger
1 sibling, 2 replies; 20+ messages in thread
From: Diego Elio Pettenò @ 2015-05-31 11:50 UTC (permalink / raw
To: gentoo-dev; +Cc: Mike Frysinger
[-- Attachment #1: Type: text/plain, Size: 936 bytes --]
On 31 May 2015 at 12:59, Alexis Ballier <aballier@gentoo.org> wrote:
> nice, but can't we add the lfs flags to our default toolchain flags or
> even better patch glibc headers to always redefine these functions to
> the 64bits variants?
>
No, because that can easily break ABI of programs that actually want the
non-LFS one (for instance anything that wraps around function calls,
including but not limited to padsp, aoss, and similar wrappers.) FreeBSD
has removed the symbols on an ABI bump, which hopefully could happen for
glibc in a far and remote future, too. But as long as the symbols are
there, the defines shouldn't be forced.
Mike, thanks for doing this, it has been a pain in my shoe since 2008
<https://blog.flameeyes.eu/2008/11/who-wants-to-support-largefile>. I'm
looking forward to the protests that 64-bit inodes don't exist, though.
Diego Elio Pettenò — Flameeyes
https://blog.flameeyes.eu/
[-- Attachment #2: Type: text/html, Size: 1518 bytes --]
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [gentoo-dev] LFS QA warnings coming soon to a build near you
2015-05-31 11:50 ` Diego Elio Pettenò
@ 2015-05-31 12:33 ` Philip Webb
2015-05-31 12:48 ` Diego Elio Pettenò
2015-05-31 13:52 ` Alexis Ballier
1 sibling, 1 reply; 20+ messages in thread
From: Philip Webb @ 2015-05-31 12:33 UTC (permalink / raw
To: gentoo-dev
150531 Diego Elio Pettenò wrote:
> Mike, thanks for doing this, it has been a pain in my shoe since 2008
> <https://blog.flameeyes.eu/2008/11/who-wants-to-support-largefile>.
Do users with 64-bit systems have to pay attention to this ?
--
========================,,============================================
SUPPORT ___________//___, Philip Webb
ELECTRIC /] [] [] [] [] []| Cities Centre, University of Toronto
TRANSIT `-O----------O---' purslowatchassdotutorontodotca
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [gentoo-dev] LFS QA warnings coming soon to a build near you
2015-05-31 11:50 ` Diego Elio Pettenò
2015-05-31 12:33 ` Philip Webb
@ 2015-05-31 13:52 ` Alexis Ballier
2015-05-31 14:17 ` Mike Frysinger
1 sibling, 1 reply; 20+ messages in thread
From: Alexis Ballier @ 2015-05-31 13:52 UTC (permalink / raw
To: gentoo-dev
On Sun, 31 May 2015 13:50:49 +0200
Diego Elio Pettenò <flameeyes@flameeyes.eu> wrote:
> On 31 May 2015 at 12:59, Alexis Ballier <aballier@gentoo.org> wrote:
>
> > nice, but can't we add the lfs flags to our default toolchain flags
> > or even better patch glibc headers to always redefine these
> > functions to the 64bits variants?
> >
>
> No, because that can easily break ABI of programs that actually want
> the non-LFS one (for instance anything that wraps around function
> calls, including but not limited to padsp, aoss, and similar
> wrappers.)
This seems easily fixed with an opt-out for lfs flags that such
programs can use. They'll need to be touched to disable the QA warning
anyway.
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [gentoo-dev] LFS QA warnings coming soon to a build near you
2015-05-31 13:52 ` Alexis Ballier
@ 2015-05-31 14:17 ` Mike Frysinger
2015-05-31 14:33 ` Alexis Ballier
0 siblings, 1 reply; 20+ messages in thread
From: Mike Frysinger @ 2015-05-31 14:17 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 1228 bytes --]
On 31 May 2015 15:52, Alexis Ballier wrote:
> On Sun, 31 May 2015 13:50:49 +0200 Diego Elio Pettenò wrote:
> > On 31 May 2015 at 12:59, Alexis Ballier wrote:
> > > nice, but can't we add the lfs flags to our default toolchain flags
> > > or even better patch glibc headers to always redefine these
> > > functions to the 64bits variants?
> >
> > No, because that can easily break ABI of programs that actually want
> > the non-LFS one (for instance anything that wraps around function
> > calls, including but not limited to padsp, aoss, and similar
> > wrappers.)
>
> This seems easily fixed with an opt-out for lfs flags that such
> programs can use. They'll need to be touched to disable the QA warning
> anyway.
this is a discussion for upstream toolchain packages (largely glibc) and in fact
i started such a heretical thread over a year ago. it was not well received due
to the implicit/silent ABI change that new builds would receive. glibc likes to
be conservative as it is the foundation of everything.
so unless glibc changes, updating our copy of glibc would only somewhat help our
users. conversely, getting the changes pushed to the respective upstream would
help everyone.
-mike
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [gentoo-dev] LFS QA warnings coming soon to a build near you
2015-05-31 14:17 ` Mike Frysinger
@ 2015-05-31 14:33 ` Alexis Ballier
2015-05-31 15:17 ` Mike Frysinger
2015-06-01 16:51 ` Christopher Head
0 siblings, 2 replies; 20+ messages in thread
From: Alexis Ballier @ 2015-05-31 14:33 UTC (permalink / raw
To: gentoo-dev
On Sun, 31 May 2015 10:17:02 -0400
Mike Frysinger <vapier@gentoo.org> wrote:
> On 31 May 2015 15:52, Alexis Ballier wrote:
> > On Sun, 31 May 2015 13:50:49 +0200 Diego Elio Pettenò wrote:
> > > On 31 May 2015 at 12:59, Alexis Ballier wrote:
> > > > nice, but can't we add the lfs flags to our default toolchain
> > > > flags or even better patch glibc headers to always redefine
> > > > these functions to the 64bits variants?
> > >
> > > No, because that can easily break ABI of programs that actually
> > > want the non-LFS one (for instance anything that wraps around
> > > function calls, including but not limited to padsp, aoss, and
> > > similar wrappers.)
> >
> > This seems easily fixed with an opt-out for lfs flags that such
> > programs can use. They'll need to be touched to disable the QA
> > warning anyway.
>
> this is a discussion for upstream toolchain packages (largely glibc)
> and in fact i started such a heretical thread over a year ago. it
> was not well received due to the implicit/silent ABI change that new
> builds would receive. glibc likes to be conservative as it is the
> foundation of everything.
>
> so unless glibc changes, updating our copy of glibc would only
> somewhat help our users. conversely, getting the changes pushed to
> the respective upstream would help everyone.
I'm not sure what's best for every one:
1. Push hundreds of patches upstream to add lfs flags;
2. Apply your patch to our glibc ebuilds, fix the corner cases, and go
back to glibc upstream with these data.
Least to say is that I'm not a big fan of lfs flags as done in glibc and
I certainly prefer 2., but you'd know better.
Alexis.
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [gentoo-dev] LFS QA warnings coming soon to a build near you
2015-05-31 14:33 ` Alexis Ballier
@ 2015-05-31 15:17 ` Mike Frysinger
2015-06-01 8:15 ` Alexis Ballier
2015-06-01 16:51 ` Christopher Head
1 sibling, 1 reply; 20+ messages in thread
From: Mike Frysinger @ 2015-05-31 15:17 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 2893 bytes --]
On 31 May 2015 16:33, Alexis Ballier wrote:
> On Sun, 31 May 2015 10:17:02 -0400 Mike Frysinger wrote:
> > On 31 May 2015 15:52, Alexis Ballier wrote:
> > > On Sun, 31 May 2015 13:50:49 +0200 Diego Elio Pettenò wrote:
> > > > On 31 May 2015 at 12:59, Alexis Ballier wrote:
> > > > > nice, but can't we add the lfs flags to our default toolchain
> > > > > flags or even better patch glibc headers to always redefine
> > > > > these functions to the 64bits variants?
> > > >
> > > > No, because that can easily break ABI of programs that actually
> > > > want the non-LFS one (for instance anything that wraps around
> > > > function calls, including but not limited to padsp, aoss, and
> > > > similar wrappers.)
> > >
> > > This seems easily fixed with an opt-out for lfs flags that such
> > > programs can use. They'll need to be touched to disable the QA
> > > warning anyway.
> >
> > this is a discussion for upstream toolchain packages (largely glibc)
> > and in fact i started such a heretical thread over a year ago. it
> > was not well received due to the implicit/silent ABI change that new
> > builds would receive. glibc likes to be conservative as it is the
> > foundation of everything.
> >
> > so unless glibc changes, updating our copy of glibc would only
> > somewhat help our users. conversely, getting the changes pushed to
> > the respective upstream would help everyone.
>
> I'm not sure what's best for every one:
> 1. Push hundreds of patches upstream to add lfs flags;
> 2. Apply your patch to our glibc ebuilds, fix the corner cases, and go
> back to glibc upstream with these data.
>
> Least to say is that I'm not a big fan of lfs flags as done in glibc and
> I certainly prefer 2., but you'd know better.
well if we're going to do arbitrary lists ;)
(1) your options aren't mutually exclusive
(2) implementing both are desirable
(3) considering the glibc effort has been stalled for over a year, (1) is
something we can help accomplish and make reasonable progress on
(4) glibc isn't the only one that implements LFS in a transparent backwards
compatible manner
which leads me to the next part ... my first suggestion in the tracker is for
autotool based projects to use AC_SYS_LARGEFILE because:
(a) it supports a variety of systems
(b) as new systems come up or bugs are found or whatever, the autoconf macro
will improve and people eventually get those fixes for free
(c) it does it all transparently for you -- add the macro and you're done
(d) it fixes the package for all users, new & old
the reason i listed only the raw CPPFLAGS and autoconf macros are because those
are the two i'm familiar with. i don't know how other build systems (e.g.
cmake) support this (assuming they do at all). if people have other recipes on
hand, then it would be great to collect more there.
-mike
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [gentoo-dev] LFS QA warnings coming soon to a build near you
2015-05-31 15:17 ` Mike Frysinger
@ 2015-06-01 8:15 ` Alexis Ballier
2015-06-02 14:13 ` Mike Frysinger
0 siblings, 1 reply; 20+ messages in thread
From: Alexis Ballier @ 2015-06-01 8:15 UTC (permalink / raw
To: gentoo-dev
On Sun, 31 May 2015 11:17:50 -0400
Mike Frysinger <vapier@gentoo.org> wrote:
> well if we're going to do arbitrary lists ;)
> (1) your options aren't mutually exclusive
> (2) implementing both are desirable
good to know your longterm plan :)
however, even if both can be done, i still don't see the point of going
through patching if we end up changing the default anyway.
> (3) considering the glibc effort has been stalled for over a year,
> (1) is something we can help accomplish and make reasonable progress
> on (4) glibc isn't the only one that implements LFS in a transparent
> backwards compatible manner
maybe the fact that some file operations are broken with glibc's
default settings on a somewhat popular fs would be a good argument to
un-stall it ?
> which leads me to the next part ... my first suggestion in the
> tracker is for autotool based projects to use AC_SYS_LARGEFILE
> because: (a) it supports a variety of systems
> (b) as new systems come up or bugs are found or whatever, the
> autoconf macro will improve and people eventually get those fixes for
> free (c) it does it all transparently for you -- add the macro and
> you're done (d) it fixes the package for all users, new & old
>
> the reason i listed only the raw CPPFLAGS and autoconf macros are
> because those are the two i'm familiar with. i don't know how other
> build systems (e.g. cmake) support this (assuming they do at all).
> if people have other recipes on hand, then it would be great to
> collect more there. -mike
yes, but that is not what i am discussing: unless i missed something,
they'll all end up one way or another adding the relevant defines;
whether it is done with an m4 macro, append-lfs-flags, a cmake function
or what else is an implementation detail of little interest to me trying
to understand why we don't simply change the default :)
Alexis.
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [gentoo-dev] LFS QA warnings coming soon to a build near you
2015-06-01 8:15 ` Alexis Ballier
@ 2015-06-02 14:13 ` Mike Frysinger
2015-06-03 8:26 ` Alexis Ballier
0 siblings, 1 reply; 20+ messages in thread
From: Mike Frysinger @ 2015-06-02 14:13 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 3096 bytes --]
On 01 Jun 2015 10:15, Alexis Ballier wrote:
> On Sun, 31 May 2015 11:17:50 -0400 Mike Frysinger wrote:
> > (3) considering the glibc effort has been stalled for over a year,
> > (1) is something we can help accomplish and make reasonable progress
> > on (4) glibc isn't the only one that implements LFS in a transparent
> > backwards compatible manner
>
> maybe the fact that some file operations are broken with glibc's
> default settings on a somewhat popular fs would be a good argument to
> un-stall it ?
it isn't that black & white. while for many packages the change would be
harmless, glibc is more conservative than that. it can be shown that a not
uncommon coding style is to use long's everywhere when dealing with the fs.
when sizeof(off_t)==sizeof(long), this works out. when off_t is suddenly
twice as large, then you can get truncation at best (which is kind of the
status quo) and random corruption at worse (like an API that takes pointers
to off_t's but the caller passed in a long). if you want to join the list
upstream and contribute to the discussion, you're more than welcome to. the
maintainers aren't clueless -- they fully understand the trade-offs we're
discussing here and know that 32bit fs accesses lead to failures (including
stat).
since sitting on our hands and hoping for the best continues to accomplish
nothing, getting the various upstream packages to opt-in to LFS themselves
can make real progress now.
> > which leads me to the next part ... my first suggestion in the
> > tracker is for autotool based projects to use AC_SYS_LARGEFILE
> > because: (a) it supports a variety of systems
> > (b) as new systems come up or bugs are found or whatever, the
> > autoconf macro will improve and people eventually get those fixes for
> > free (c) it does it all transparently for you -- add the macro and
> > you're done (d) it fixes the package for all users, new & old
> >
> > the reason i listed only the raw CPPFLAGS and autoconf macros are
> > because those are the two i'm familiar with. i don't know how other
> > build systems (e.g. cmake) support this (assuming they do at all).
> > if people have other recipes on hand, then it would be great to
> > collect more there.
>
> yes, but that is not what i am discussing: unless i missed something,
> they'll all end up one way or another adding the relevant defines;
> whether it is done with an m4 macro, append-lfs-flags, a cmake function
> or what else is an implementation detail of little interest to me trying
> to understand why we don't simply change the default :)
as i already said, changing the default in *glibc* doesn't help non-glibc
systems. using portable build systems (like autotools) lets you easily
enable features like this and the build system knows the various platform
details. if you decided to roll your own build system or use one that
isn't that portable, well that's kind of your fault isn't it ? if you've
opted to only support a subset of systems, then that means adding these
CPPFLAGS yourself.
-mike
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [gentoo-dev] LFS QA warnings coming soon to a build near you
2015-06-02 14:13 ` Mike Frysinger
@ 2015-06-03 8:26 ` Alexis Ballier
2015-06-03 11:41 ` Mike Frysinger
0 siblings, 1 reply; 20+ messages in thread
From: Alexis Ballier @ 2015-06-03 8:26 UTC (permalink / raw
To: gentoo-dev
On Tue, 2 Jun 2015 10:13:54 -0400
Mike Frysinger <vapier@gentoo.org> wrote:
> On 01 Jun 2015 10:15, Alexis Ballier wrote:
> > On Sun, 31 May 2015 11:17:50 -0400 Mike Frysinger wrote:
> > > (3) considering the glibc effort has been stalled for over a year,
> > > (1) is something we can help accomplish and make reasonable
> > > progress on (4) glibc isn't the only one that implements LFS in a
> > > transparent backwards compatible manner
> >
> > maybe the fact that some file operations are broken with glibc's
> > default settings on a somewhat popular fs would be a good argument
> > to un-stall it ?
>
> it isn't that black & white. while for many packages the change
> would be harmless, glibc is more conservative than that. it can be
> shown that a not uncommon coding style is to use long's everywhere
> when dealing with the fs. when sizeof(off_t)==sizeof(long), this
> works out. when off_t is suddenly twice as large, then you can get
> truncation at best (which is kind of the status quo) and random
> corruption at worse (like an API that takes pointers to off_t's but
> the caller passed in a long). if you want to join the list upstream
> and contribute to the discussion, you're more than welcome to. the
> maintainers aren't clueless -- they fully understand the trade-offs
> we're discussing here and know that 32bit fs accesses lead to
> failures (including stat).
>
> since sitting on our hands and hoping for the best continues to
> accomplish nothing, getting the various upstream packages to opt-in
> to LFS themselves can make real progress now.
agreed, thanks for the explanation
i take it as the main goal of these qa warnings is to fix the above
mentioned half broken packages and that pushing lfs flags everywhere is
a slow but safe way to achieve it :)
it'd be interesting to have data about how many packages can break;
my belief is that more than half the c/c++ programs use one way or
another fs ops but maybe less than one in a thousand will break; i can
understand this is still too much for a libc (even though other
'breaking fixes' weren't so controversial in the past), and we'll see
after gathering the data your qa warnings will provide
> > > which leads me to the next part ... my first suggestion in the
> > > tracker is for autotool based projects to use AC_SYS_LARGEFILE
> > > because: (a) it supports a variety of systems
> > > (b) as new systems come up or bugs are found or whatever, the
> > > autoconf macro will improve and people eventually get those fixes
> > > for free (c) it does it all transparently for you -- add the
> > > macro and you're done (d) it fixes the package for all users, new
> > > & old
> > >
> > > the reason i listed only the raw CPPFLAGS and autoconf macros are
> > > because those are the two i'm familiar with. i don't know how
> > > other build systems (e.g. cmake) support this (assuming they do
> > > at all). if people have other recipes on hand, then it would be
> > > great to collect more there.
> >
> > yes, but that is not what i am discussing: unless i missed
> > something, they'll all end up one way or another adding the
> > relevant defines; whether it is done with an m4 macro,
> > append-lfs-flags, a cmake function or what else is an
> > implementation detail of little interest to me trying to understand
> > why we don't simply change the default :)
>
> as i already said, changing the default in *glibc* doesn't help
> non-glibc systems.
even with ten different libc's, the order of magnitude is still smaller
for changing the default :)
Alexis.
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [gentoo-dev] LFS QA warnings coming soon to a build near you
2015-06-03 8:26 ` Alexis Ballier
@ 2015-06-03 11:41 ` Mike Frysinger
0 siblings, 0 replies; 20+ messages in thread
From: Mike Frysinger @ 2015-06-03 11:41 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 3683 bytes --]
On 03 Jun 2015 10:26, Alexis Ballier wrote:
> On Tue, 2 Jun 2015 10:13:54 -0400 Mike Frysinger wrote:
> > On 01 Jun 2015 10:15, Alexis Ballier wrote:
> > > On Sun, 31 May 2015 11:17:50 -0400 Mike Frysinger wrote:
> > > > (3) considering the glibc effort has been stalled for over a year,
> > > > (1) is something we can help accomplish and make reasonable
> > > > progress on (4) glibc isn't the only one that implements LFS in a
> > > > transparent backwards compatible manner
> > >
> > > maybe the fact that some file operations are broken with glibc's
> > > default settings on a somewhat popular fs would be a good argument
> > > to un-stall it ?
> >
> > it isn't that black & white. while for many packages the change
> > would be harmless, glibc is more conservative than that. it can be
> > shown that a not uncommon coding style is to use long's everywhere
> > when dealing with the fs. when sizeof(off_t)==sizeof(long), this
> > works out. when off_t is suddenly twice as large, then you can get
> > truncation at best (which is kind of the status quo) and random
> > corruption at worse (like an API that takes pointers to off_t's but
> > the caller passed in a long). if you want to join the list upstream
> > and contribute to the discussion, you're more than welcome to. the
> > maintainers aren't clueless -- they fully understand the trade-offs
> > we're discussing here and know that 32bit fs accesses lead to
> > failures (including stat).
> >
> > since sitting on our hands and hoping for the best continues to
> > accomplish nothing, getting the various upstream packages to opt-in
> > to LFS themselves can make real progress now.
>
> agreed, thanks for the explanation
>
> i take it as the main goal of these qa warnings is to fix the above
> mentioned half broken packages and that pushing lfs flags everywhere is
> a slow but safe way to achieve it :)
>
> it'd be interesting to have data about how many packages can break;
> my belief is that more than half the c/c++ programs use one way or
> another fs ops but maybe less than one in a thousand will break; i can
> understand this is still too much for a libc (even though other
> 'breaking fixes' weren't so controversial in the past), and we'll see
> after gathering the data your qa warnings will provide
let's ignore the open/read/write issue for now on the assumption it's more
uncommon to incidentally have that break. instead, let's look at things that
can hit inodes -- stat & dir listing. a file simply being allocated a 64bit
inode will break an app -- both directly stating and even trying to list the
dir. oops! :)
here's the command i'm using for just inode funcs:
scanelf / -mqRygs \
'^((__)?[fl]?x?stat(v?fs)?(at)?|alphasort|getdirentries|readdir(_r)?|scandir(at)?|versionsort)$'
you'd have to ignore glibc and sandbox as they provide both 32bit & 64bit.
i have an x86 chroot for random testing, but it's not as "full" as my desktop.
it has a measily 245 packages installed. of those:
- 25 are using 32bit inode functions
- 34 are using some 32bit fs interface (this is a superset of the above)
if someone had a fuller install of like >1000 pkgs, that'd be interesting.
> > as i already said, changing the default in *glibc* doesn't help
> > non-glibc systems.
>
> even with ten different libc's, the order of magnitude is still smaller
> for changing the default :)
true. each fight though will be significantly higher i think, assuming you can
even find someone to talk to (as not all are open source projects). and it will
take longer for the fix to trickle out to deployments.
-mike
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [gentoo-dev] LFS QA warnings coming soon to a build near you
2015-05-31 14:33 ` Alexis Ballier
2015-05-31 15:17 ` Mike Frysinger
@ 2015-06-01 16:51 ` Christopher Head
2015-06-02 14:15 ` Mike Frysinger
1 sibling, 1 reply; 20+ messages in thread
From: Christopher Head @ 2015-06-01 16:51 UTC (permalink / raw
To: gentoo-dev
On May 31, 2015 7:33:28 AM PDT, Alexis Ballier <aballier@gentoo.org> wrote:
>
>I'm not sure what's best for every one:
>1. Push hundreds of patches upstream to add lfs flags;
>2. Apply your patch to our glibc ebuilds, fix the corner cases, and go
> back to glibc upstream with these data.
>
If the changes are made to glibc, would these be under a new symbol version for ABI compatibility, or just be changes to headers to make _FILE_OFFSET_BITS=64 the default? If not, what about binary software? Not saying you haven’t considered the relevant issues; I just haven’t seen binary software brought up on this list yet.
--
Christopher Head
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [gentoo-dev] LFS QA warnings coming soon to a build near you
2015-06-01 16:51 ` Christopher Head
@ 2015-06-02 14:15 ` Mike Frysinger
0 siblings, 0 replies; 20+ messages in thread
From: Mike Frysinger @ 2015-06-02 14:15 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 924 bytes --]
On 01 Jun 2015 09:51, Christopher Head wrote:
> On May 31, 2015 7:33:28 AM PDT, Alexis Ballier wrote:
> >I'm not sure what's best for every one:
> >1. Push hundreds of patches upstream to add lfs flags;
> >2. Apply your patch to our glibc ebuilds, fix the corner cases, and go
> > back to glibc upstream with these data.
>
> If the changes are made to glibc, would these be under a new symbol version
> for ABI compatibility, or just be changes to headers to make
> _FILE_OFFSET_BITS=64 the default? If not, what about binary software? Not
> saying you haven’t considered the relevant issues; I just haven’t seen binary
> software brought up on this list yet.
regardless of what the headers export as a default, glibc's ABI would not
change. the CPPFLAGS merely control which set of symbols are used. any
existing binary packages would continue to operate the same way they always
have.
-mike
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [gentoo-dev] LFS QA warnings coming soon to a build near you
2015-05-31 10:59 ` Alexis Ballier
2015-05-31 11:50 ` Diego Elio Pettenò
@ 2015-05-31 13:46 ` Mike Frysinger
2015-05-31 14:09 ` Alexis Ballier
1 sibling, 1 reply; 20+ messages in thread
From: Mike Frysinger @ 2015-05-31 13:46 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 403 bytes --]
On 31 May 2015 12:59, Alexis Ballier wrote:
> I don't understand why one should add append-lfs-flags to almost every
> single package out there.
no one is suggesting that route. append-lfs-flags is a kludge (albeit an
effective one) in the face of no upstream support. if you look at the
referenced tracker it includes pointers for fixing packages in ways that
can be sent upstream.
-mike
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [gentoo-dev] LFS QA warnings coming soon to a build near you
2015-05-31 13:46 ` Mike Frysinger
@ 2015-05-31 14:09 ` Alexis Ballier
0 siblings, 0 replies; 20+ messages in thread
From: Alexis Ballier @ 2015-05-31 14:09 UTC (permalink / raw
To: gentoo-dev
On Sun, 31 May 2015 09:46:30 -0400
Mike Frysinger <vapier@gentoo.org> wrote:
> On 31 May 2015 12:59, Alexis Ballier wrote:
> > I don't understand why one should add append-lfs-flags to almost
> > every single package out there.
>
> no one is suggesting that route. append-lfs-flags is a kludge
> (albeit an effective one) in the face of no upstream support. if you
> look at the referenced tracker it includes pointers for fixing
> packages in ways that can be sent upstream.
well, other suggestions are similar, except of course that appending
the flags is done upstream in some way or another
however, reading more in details the links you provided, what happened
to your patch setting _FILE_OFFSET_BITS default to 64 ?
( https://sourceware.org/ml/libc-alpha/2014-03/msg00290.html )
which seems to be a much better solution to what i initially proposed.
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [gentoo-dev] LFS QA warnings coming soon to a build near you
2015-05-30 18:54 [gentoo-dev] LFS QA warnings coming soon to a build near you Mike Frysinger
2015-05-31 10:59 ` Alexis Ballier
@ 2015-05-31 15:58 ` Mike Gilbert
2015-05-31 16:50 ` Mike Frysinger
1 sibling, 1 reply; 20+ messages in thread
From: Mike Gilbert @ 2015-05-31 15:58 UTC (permalink / raw
To: Gentoo Dev
On Sat, May 30, 2015 at 2:54 PM, Mike Frysinger <vapier@gentoo.org> wrote:
> we've got a new QA check that warns whenever a package is built using a 32bit
> filesystem interface. in practice, this applies to arm/mips/ppc/sh/x86 systems
> (not including multilib -- for now).
>
> this topic has come up in Gentoo a few times over the years but we've never
> really amassed the will power to fix it. instead we fix it in one-offs based
> on user reports (like "can't download 4GiB file with ftp" #101038). this was
> worked well enough because most users have moved on to 64bit systems and the
> interaction with >2GiB files tends to correlate with a few packages.
>
> however, "recent" winds have started blowing where file systems are utilizing
> 64bit inodes to handle large file counts. this means apps that do even basic
> things like stat() will now fail. the number of applications that this affects
> is significantly higher, although still relegated to systems that happen to use
> a file system with 64bit inodes.
Does this issue affect software that only reads/writes small files and
never calls stat? For example, pkg-config.
It might still be nice to adjust such packages for consistency, but it
might be harder to justify patches to upstream developers.
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [gentoo-dev] LFS QA warnings coming soon to a build near you
2015-05-31 15:58 ` Mike Gilbert
@ 2015-05-31 16:50 ` Mike Frysinger
2015-05-31 17:26 ` Mike Gilbert
0 siblings, 1 reply; 20+ messages in thread
From: Mike Frysinger @ 2015-05-31 16:50 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 2406 bytes --]
On 31 May 2015 11:58, Mike Gilbert wrote:
> On Sat, May 30, 2015 at 2:54 PM, Mike Frysinger wrote:
> > we've got a new QA check that warns whenever a package is built using a 32bit
> > filesystem interface. in practice, this applies to arm/mips/ppc/sh/x86 systems
> > (not including multilib -- for now).
> >
> > this topic has come up in Gentoo a few times over the years but we've never
> > really amassed the will power to fix it. instead we fix it in one-offs based
> > on user reports (like "can't download 4GiB file with ftp" #101038). this was
> > worked well enough because most users have moved on to 64bit systems and the
> > interaction with >2GiB files tends to correlate with a few packages.
> >
> > however, "recent" winds have started blowing where file systems are utilizing
> > 64bit inodes to handle large file counts. this means apps that do even basic
> > things like stat() will now fail. the number of applications that this affects
> > is significantly higher, although still relegated to systems that happen to use
> > a file system with 64bit inodes.
>
> Does this issue affect software that only reads/writes small files and
> never calls stat? For example, pkg-config.
it is possible to use some file system operations and never run into problems.
the trouble is that it requires diligence and careful auditing, and you have to
make sure your interactions with other packages are also OK. considering how
bad/unaware the wider community is wrt LFS, ABI compatibility, and such critical
but oft overlooked issues, i suspect most people also won't have the will
power/capabilities to keep up here.
conversely, flip the autoconf/cppflags magic, and most projects Just Work.
> It might still be nice to adjust such packages for consistency, but it
> might be harder to justify patches to upstream developers.
pkg-config already merged it and it's already been pulled back into Gentoo ;)
https://bugs.freedesktop.org/show_bug.cgi?id=90078
https://bugs.gentoo.org/550508
that said, i'm not entirely sure why there would be a lot of push back.
when i gathered data on this years ago (which i no longer have, so i'd have
to recreate), the increased foot print and runtime cost was not significant.
especially when you consider that we're talking about I/O operations which
inherently kill CPU performance due to stalls.
-mike
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [gentoo-dev] LFS QA warnings coming soon to a build near you
2015-05-31 16:50 ` Mike Frysinger
@ 2015-05-31 17:26 ` Mike Gilbert
0 siblings, 0 replies; 20+ messages in thread
From: Mike Gilbert @ 2015-05-31 17:26 UTC (permalink / raw
To: Gentoo Dev
On Sun, May 31, 2015 at 12:50 PM, Mike Frysinger <vapier@gentoo.org> wrote:
> On 31 May 2015 11:58, Mike Gilbert wrote:
>> It might still be nice to adjust such packages for consistency, but it
>> might be harder to justify patches to upstream developers.
>
> pkg-config already merged it and it's already been pulled back into Gentoo ;)
> https://bugs.freedesktop.org/show_bug.cgi?id=90078
> https://bugs.gentoo.org/550508
>
> that said, i'm not entirely sure why there would be a lot of push back.
> when i gathered data on this years ago (which i no longer have, so i'd have
> to recreate), the increased foot print and runtime cost was not significant.
> especially when you consider that we're talking about I/O operations which
> inherently kill CPU performance due to stalls.
> -mike
Sure, I don't really expect much resistance. I just like to be
prepared in the event some developer starts asking questions.
^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2015-06-03 11:41 UTC | newest]
Thread overview: 20+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-05-30 18:54 [gentoo-dev] LFS QA warnings coming soon to a build near you Mike Frysinger
2015-05-31 10:59 ` Alexis Ballier
2015-05-31 11:50 ` Diego Elio Pettenò
2015-05-31 12:33 ` Philip Webb
2015-05-31 12:48 ` Diego Elio Pettenò
2015-05-31 13:52 ` Alexis Ballier
2015-05-31 14:17 ` Mike Frysinger
2015-05-31 14:33 ` Alexis Ballier
2015-05-31 15:17 ` Mike Frysinger
2015-06-01 8:15 ` Alexis Ballier
2015-06-02 14:13 ` Mike Frysinger
2015-06-03 8:26 ` Alexis Ballier
2015-06-03 11:41 ` Mike Frysinger
2015-06-01 16:51 ` Christopher Head
2015-06-02 14:15 ` Mike Frysinger
2015-05-31 13:46 ` Mike Frysinger
2015-05-31 14:09 ` Alexis Ballier
2015-05-31 15:58 ` Mike Gilbert
2015-05-31 16:50 ` Mike Frysinger
2015-05-31 17:26 ` Mike Gilbert
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox