[gentoo-portage-dev] [PATCH] install-qa-check.d: issue warnings for 32bit ELFs not using LFS

[gentoo-portage-dev] [PATCH] install-qa-check.d: issue warnings for 32bit ELFs not using LFS

[Mike Frysinger]

From: Mike Frysinger <vapier@chromium.org>

Start issuing QA warnings when ELFs are installed using the old 32bit
file interface.  This programs can fail when working with large files
(like ones more than 4GiB), but even just trying to stat a file that
happens to have a 64bit inode.  It also can lead to silent corruption
when one library/app is using 64bit structures and tries to work with
another one that uses 32bit (or vice versa).  This is because the API
might utilize off_t's, the library was compiled with off_t==off64_t,
and the app was compiled with off_t==off32_t.

X-Gentoo-Bug: 471102
X-Gentoo-Bug-url: https://bugs.gentoo.org/471102
---
 bin/install-qa-check.d/10large-file-support | 158 ++++++++++++++++++++++++++++
 1 file changed, 158 insertions(+)
 create mode 100644 bin/install-qa-check.d/10large-file-support

diff --git a/bin/install-qa-check.d/10large-file-support b/bin/install-qa-check.d/10large-file-support
new file mode 100644
index 0000000..34726d7
--- /dev/null
+++ b/bin/install-qa-check.d/10large-file-support
@@ -0,0 +1,158 @@
+# Detect 32-bit builds that are using legacy 32-bit file interfaces.
+# https://en.wikipedia.org/wiki/Large_file_support
+#
+# We want to make sure everyone is using the 64-bit interfaces.
+# https://bugs.gentoo.org/471102
+
+# Lists gleaned from headers and this doc:
+# http://people.redhat.com/berrange/notes/largefile.html
+# http://opengroup.org/platform/lfs.html
+SYMBOLS=(
+	# aio.h
+	aio_cancel
+	aio_error
+	aio_fsync
+	aio_read
+	aio_return
+	aio_suspend
+	aio_write
+	lio_listio
+
+	# dirent.h
+	alphasort
+	getdirentries
+	readdir
+	readdir_r
+	scandir
+	scandirat
+	versionsort
+
+	# fcntl.h
+	creat
+	fallocate
+	fopen
+	fopenat
+	freopen
+	open
+	openat
+	posix_fadvise
+	posix_fallocate
+	__open
+	__open_2
+	__openat_2
+
+	# ftw.h
+	ftw
+	nftw
+
+	# glob.h
+	glob
+	globfree
+
+	# stdio.h
+	fgetpos
+	fopen
+	freopen
+	fseeko
+	fsetpos
+	ftello
+	tmpfile
+
+	# stdlib.h
+	mkostemp
+	mkostemps
+	mkstemp
+	mkstemps
+
+	# sys/mman.h
+	mmap
+
+	# sys/resource.h
+	getrlimit
+	prlimit
+	setrlimit
+
+	# sys/sendfile.h
+	sendfile
+
+	# sys/stat.h
+	fstat
+	fstatat
+	lstat
+	stat
+	__fxstat
+	__fxstatat
+	__lxstat
+	__xstat
+
+	# sys/statfs.h
+	fstatfs
+
+	# sys/statvfs.h
+	statvfs
+	fstatvfs
+
+	# unistd.h
+	lockf
+	lseek
+	ftruncate
+	pread
+	preadv
+	pwrite
+	pwritev
+	truncate
+	__pread_chk
+)
+SYMBOLS_REGEX=$(printf '%s|' "${SYMBOLS[@]}")
+# The @@ part is to workaround a limitation in pax-utils w/versioned symbols.
+SYMBOLS_REGEX="^(${SYMBOLS_REGEX%|})(@@.*)?$"
+
+check_lfs() {
+	local files=$(scanelf -F '%s %p' -qyRgs "-${SYMBOLS_REGEX}" "$@")
+
+	if [[ -n ${files} ]]; then
+		echo
+		eqawarn "QA Notice: The following files were not built with LFS support:"
+		eqawarn "  Please see https://bugs.gentoo.org/471102 for details."
+		eqawarn "${files}"
+		echo
+	fi
+}
+
+filtered_check_lfs() {
+	if ! type -P scanelf >/dev/null || has binchecks ${RESTRICT}; then
+		return
+	fi
+
+	# Only check glibc & uClibc libraries.  Others are presumed to use LFS by
+	# default (e.g. musl), or it's not relevant (e.g. newlib).
+	case ${CHOST} in
+	*-gnu*|*-uclibc*) ;;
+	*) return ;;
+	esac
+
+	# Only check on 32-bit systems.  Filtering by $ARCH here isn't perfect, but
+	# it should be good enough for our needs.
+	case ${ARCH} in
+	arm|mips|ppc|sh|x86) ;;
+	*) return ;;
+	esac
+
+	# Obviously filter out C libraries themselves :).
+	# The sandbox has to capture all symbols by design.
+	case ${CATEGORY}/${PN} in
+	*/glibc|\
+	*/uclibc|\
+	*/gcc|\
+	sys-apps/sandbox) ;;
+	*) check_lfs "${ED}" ;;
+	esac
+}
+
+# Allow for people to run manually for testing/debugging.
+if [[ $# -ne 0 ]]; then
+	eqawarn() { echo " * $*"; }
+	check_lfs "$@"
+else
+	filtered_check_lfs
+fi
-- 
2.4.1

[gentoo-portage-dev] [PATCH v2] install-qa-check.d: issue warnings for 32bit ELFs not using LFS

[Mike Frysinger]

From: Mike Frysinger <vapier@chromium.org>

Start issuing QA warnings when ELFs are installed using the old 32bit
file interface.  These programs can fail out right:
* working with large files (more than 2GiB) can return EOVERFLOW
* stating files on large filesystems w/64bit inodes can fail too

It also can lead to silent corruption that is hard to debug like when
one library/app is compiled using 64bit structures and tries to work
with another one that uses 32bit (or vice versa).  This is because
the API in the header utilizes off_t's or structs (like stat) that
have off_t's embedded in them.  By default, off_t is defined as an
off32_t, but when the code is compiled with -D_FILE_OFFSET_BITS=64,
it is transparently changed to off64_t.  That means while one side
was compiled (w/out warnings) expecting 32bit structs, the other
was compiled (w/out warnings) expecting 64bit structs.  The ABI is
different, but C does not support type checking, so no one notices.

The only sane way forward is to use LFS everywhere.

X-Gentoo-Bug: 549092
X-Gentoo-Bug-url: https://bugs.gentoo.org/549092
---
v2
	- do some word smithing in the commit message

 bin/install-qa-check.d/10large-file-support | 158 ++++++++++++++++++++++++++++
 1 file changed, 158 insertions(+)
 create mode 100644 bin/install-qa-check.d/10large-file-support

diff --git a/bin/install-qa-check.d/10large-file-support b/bin/install-qa-check.d/10large-file-support
new file mode 100644
index 0000000..34726d7
--- /dev/null
+++ b/bin/install-qa-check.d/10large-file-support
@@ -0,0 +1,158 @@
+# Detect 32-bit builds that are using legacy 32-bit file interfaces.
+# https://en.wikipedia.org/wiki/Large_file_support
+#
+# We want to make sure everyone is using the 64-bit interfaces.
+# https://bugs.gentoo.org/471102
+
+# Lists gleaned from headers and this doc:
+# http://people.redhat.com/berrange/notes/largefile.html
+# http://opengroup.org/platform/lfs.html
+SYMBOLS=(
+	# aio.h
+	aio_cancel
+	aio_error
+	aio_fsync
+	aio_read
+	aio_return
+	aio_suspend
+	aio_write
+	lio_listio
+
+	# dirent.h
+	alphasort
+	getdirentries
+	readdir
+	readdir_r
+	scandir
+	scandirat
+	versionsort
+
+	# fcntl.h
+	creat
+	fallocate
+	fopen
+	fopenat
+	freopen
+	open
+	openat
+	posix_fadvise
+	posix_fallocate
+	__open
+	__open_2
+	__openat_2
+
+	# ftw.h
+	ftw
+	nftw
+
+	# glob.h
+	glob
+	globfree
+
+	# stdio.h
+	fgetpos
+	fopen
+	freopen
+	fseeko
+	fsetpos
+	ftello
+	tmpfile
+
+	# stdlib.h
+	mkostemp
+	mkostemps
+	mkstemp
+	mkstemps
+
+	# sys/mman.h
+	mmap
+
+	# sys/resource.h
+	getrlimit
+	prlimit
+	setrlimit
+
+	# sys/sendfile.h
+	sendfile
+
+	# sys/stat.h
+	fstat
+	fstatat
+	lstat
+	stat
+	__fxstat
+	__fxstatat
+	__lxstat
+	__xstat
+
+	# sys/statfs.h
+	fstatfs
+
+	# sys/statvfs.h
+	statvfs
+	fstatvfs
+
+	# unistd.h
+	lockf
+	lseek
+	ftruncate
+	pread
+	preadv
+	pwrite
+	pwritev
+	truncate
+	__pread_chk
+)
+SYMBOLS_REGEX=$(printf '%s|' "${SYMBOLS[@]}")
+# The @@ part is to workaround a limitation in pax-utils w/versioned symbols.
+SYMBOLS_REGEX="^(${SYMBOLS_REGEX%|})(@@.*)?$"
+
+check_lfs() {
+	local files=$(scanelf -F '%s %p' -qyRgs "-${SYMBOLS_REGEX}" "$@")
+
+	if [[ -n ${files} ]]; then
+		echo
+		eqawarn "QA Notice: The following files were not built with LFS support:"
+		eqawarn "  Please see https://bugs.gentoo.org/471102 for details."
+		eqawarn "${files}"
+		echo
+	fi
+}
+
+filtered_check_lfs() {
+	if ! type -P scanelf >/dev/null || has binchecks ${RESTRICT}; then
+		return
+	fi
+
+	# Only check glibc & uClibc libraries.  Others are presumed to use LFS by
+	# default (e.g. musl), or it's not relevant (e.g. newlib).
+	case ${CHOST} in
+	*-gnu*|*-uclibc*) ;;
+	*) return ;;
+	esac
+
+	# Only check on 32-bit systems.  Filtering by $ARCH here isn't perfect, but
+	# it should be good enough for our needs.
+	case ${ARCH} in
+	arm|mips|ppc|sh|x86) ;;
+	*) return ;;
+	esac
+
+	# Obviously filter out C libraries themselves :).
+	# The sandbox has to capture all symbols by design.
+	case ${CATEGORY}/${PN} in
+	*/glibc|\
+	*/uclibc|\
+	*/gcc|\
+	sys-apps/sandbox) ;;
+	*) check_lfs "${ED}" ;;
+	esac
+}
+
+# Allow for people to run manually for testing/debugging.
+if [[ $# -ne 0 ]]; then
+	eqawarn() { echo " * $*"; }
+	check_lfs "$@"
+else
+	filtered_check_lfs
+fi
-- 
2.4.1

Re: [gentoo-portage-dev] [PATCH v2] install-qa-check.d: issue warnings for 32bit ELFs not using LFS

[Zac Medico]

On 05/26/2015 07:24 AM, Mike Frysinger wrote:
> +	# Only check on 32-bit systems.  Filtering by $ARCH here isn't perfect, but
> +	# it should be good enough for our needs.
> +	case ${ARCH} in
> +	arm|mips|ppc|sh|x86) ;;
> +	*) return ;;
> +	esac

Shouldn't we also enable this for 64-bit archs when multilib is enabled?
I've experienced a problem with multilib pkgconfig and 64-bit inodes,
and if I enable this check for amd64 then I get these warnings for
dev-util/pkgconfig-0.28-r2:

 * QA Notice: The following files were not built with LFS support:
 *   Please see https://bugs.gentoo.org/471102 for details.
 * readdir@@GLIBC_2.2.5,fopen@@GLIBC_2.2.5
usr/bin/x86_64-pc-linux-gnu-pkg-config
 * readdir@@GLIBC_2.2.5,fopen@@GLIBC_2.2.5 usr/bin/pkg-config
 * fopen@@GLIBC_2.1,readdir@@GLIBC_2.0 usr/bin/i686-pc-linux-gnu-pkg-config

-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [PATCH v2] install-qa-check.d: issue warnings for 32bit ELFs not using LFS

[Zac Medico]

On 05/26/2015 08:58 AM, Zac Medico wrote:
> On 05/26/2015 07:24 AM, Mike Frysinger wrote:
>> +	# Only check on 32-bit systems.  Filtering by $ARCH here isn't perfect, but
>> +	# it should be good enough for our needs.
>> +	case ${ARCH} in
>> +	arm|mips|ppc|sh|x86) ;;
>> +	*) return ;;
>> +	esac
> 
> Shouldn't we also enable this for 64-bit archs when multilib is enabled?
> I've experienced a problem with multilib pkgconfig and 64-bit inodes,
> and if I enable this check for amd64 then I get these warnings for
> dev-util/pkgconfig-0.28-r2:
> 
>  * QA Notice: The following files were not built with LFS support:
>  *   Please see https://bugs.gentoo.org/471102 for details.
>  * readdir@@GLIBC_2.2.5,fopen@@GLIBC_2.2.5
> usr/bin/x86_64-pc-linux-gnu-pkg-config
>  * readdir@@GLIBC_2.2.5,fopen@@GLIBC_2.2.5 usr/bin/pkg-config
>  * fopen@@GLIBC_2.1,readdir@@GLIBC_2.0 usr/bin/i686-pc-linux-gnu-pkg-config
> 

I've confirmed that append-lfs-flags from flag-o-matic.eclass solves the
problem that I have experienced, and filed the following bug:

https://bugs.gentoo.org/show_bug.cgi?id=550482
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [PATCH v2] install-qa-check.d: issue warnings for 32bit ELFs not using LFS

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 1449 bytes --]

On 26 May 2015 08:58, Zac Medico wrote:
> On 05/26/2015 07:24 AM, Mike Frysinger wrote:
> > +	# Only check on 32-bit systems.  Filtering by $ARCH here isn't perfect, but
> > +	# it should be good enough for our needs.
> > +	case ${ARCH} in
> > +	arm|mips|ppc|sh|x86) ;;
> > +	*) return ;;
> > +	esac
> 
> Shouldn't we also enable this for 64-bit archs when multilib is enabled?

yes, but i think we should start here first.  getting multilib right is kind of 
a pain.  this should give us enough coverage i think to get people to start 
filing bugs which implicitly covers multilib users.  once the dust has settled, 
we can look at expanding the multilib coverage.  although that would really 
require a python implementation, and the current install hooks logic implicitly 
requires every file to be bash.

to use your pkg-config example, it installs 32bit & 64bit ELFs into /usr/bin.  
we can't scan all 32bit ELFs because it would incorrectly flag ILP32 ABIs like 
x32 & n32.

ideally we'd scan the whole multilib dir, but that too runs into problems.
when /usr/lib is the path for both x86 multilib and non-multilib content (like 
libexec stuff), we can't blindly scan & reject all ELFs in there.  the previous 
note about ILP32 applies here too.

so if we're happy with this implementation, i'll start a thread on gentoo-dev so 
people aren't caught by surprise, and we can merge this for the next release.
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-portage-dev] [PATCH v2] install-qa-check.d: issue warnings for 32bit ELFs not using LFS

[Zac Medico]

On 05/30/2015 07:36 AM, Mike Frysinger wrote:
> On 26 May 2015 08:58, Zac Medico wrote:
>> On 05/26/2015 07:24 AM, Mike Frysinger wrote:
>>> +	# Only check on 32-bit systems.  Filtering by $ARCH here isn't perfect, but
>>> +	# it should be good enough for our needs.
>>> +	case ${ARCH} in
>>> +	arm|mips|ppc|sh|x86) ;;
>>> +	*) return ;;
>>> +	esac
>>
>> Shouldn't we also enable this for 64-bit archs when multilib is enabled?
> 
> yes, but i think we should start here first.  getting multilib right is kind of 
> a pain.  this should give us enough coverage i think to get people to start 
> filing bugs which implicitly covers multilib users.  once the dust has settled, 
> we can look at expanding the multilib coverage.  although that would really 
> require a python implementation, and the current install hooks logic implicitly 
> requires every file to be bash.
>
> to use your pkg-config example, it installs 32bit & 64bit ELFs into /usr/bin.  
> we can't scan all 32bit ELFs because it would incorrectly flag ILP32 ABIs like 
> x32 & n32.

Our compute_multilib_category function has logic that could be used to
separate and filter them:

https://gitweb.gentoo.org/proj/portage.git/tree/pym/portage/dep/soname/multilib_category.py

> ideally we'd scan the whole multilib dir, but that too runs into problems.
> when /usr/lib is the path for both x86 multilib and non-multilib content (like 
> libexec stuff), we can't blindly scan & reject all ELFs in there.  the previous 
> note about ILP32 applies here too.
> 
> so if we're happy with this implementation, i'll start a thread on gentoo-dev so 
> people aren't caught by surprise, and we can merge this for the next release.
> -mike
> 

Okay, fine.
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [PATCH v2] install-qa-check.d: issue warnings for 32bit ELFs not using LFS

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 1463 bytes --]

On 30 May 2015 11:22, Zac Medico wrote:
> On 05/30/2015 07:36 AM, Mike Frysinger wrote:
> > On 26 May 2015 08:58, Zac Medico wrote:
> >> On 05/26/2015 07:24 AM, Mike Frysinger wrote:
> >>> +	# Only check on 32-bit systems.  Filtering by $ARCH here isn't perfect, but
> >>> +	# it should be good enough for our needs.
> >>> +	case ${ARCH} in
> >>> +	arm|mips|ppc|sh|x86) ;;
> >>> +	*) return ;;
> >>> +	esac
> >>
> >> Shouldn't we also enable this for 64-bit archs when multilib is enabled?
> > 
> > yes, but i think we should start here first.  getting multilib right is kind of 
> > a pain.  this should give us enough coverage i think to get people to start 
> > filing bugs which implicitly covers multilib users.  once the dust has settled, 
> > we can look at expanding the multilib coverage.  although that would really 
> > require a python implementation, and the current install hooks logic implicitly 
> > requires every file to be bash.
> >
> > to use your pkg-config example, it installs 32bit & 64bit ELFs into /usr/bin.  
> > we can't scan all 32bit ELFs because it would incorrectly flag ILP32 ABIs like 
> > x32 & n32.
> 
> Our compute_multilib_category function has logic that could be used to
> separate and filter them:
> 
> https://gitweb.gentoo.org/proj/portage.git/tree/pym/portage/dep/soname/multilib_category.py

that'd require a python implementation which hooks are somewhat limited by 
currently ;)
-mik

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-portage-dev] [PATCH v2] install-qa-check.d: issue warnings for 32bit ELFs not using LFS

[Brian Dolbec]

On Sat, 30 May 2015 10:36:05 -0400
Mike Frysinger <vapier@gentoo.org> wrote:

> 
> so if we're happy with this implementation, i'll start a thread on
> gentoo-dev so people aren't caught by surprise, and we can merge this
> for the next release. -mike


So, what's the status of this one?  I don't recall many responses to
the -dev thread.
-- 
Brian Dolbec <dolsen>

Re: [gentoo-portage-dev] [PATCH v2] install-qa-check.d: issue warnings for 32bit ELFs not using LFS

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 492 bytes --]

On 11 Jun 2015 00:21, Brian Dolbec wrote:
> On Sat, 30 May 2015 10:36:05 -0400 Mike Frysinger wrote:
> > so if we're happy with this implementation, i'll start a thread on
> > gentoo-dev so people aren't caught by surprise, and we can merge this
> > for the next release. -mike
> 
> So, what's the status of this one?  I don't recall many responses to
> the -dev thread.

i'll be pushing it at some point.  guess i could do it now since it doesn't 
depend on anything else.
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]