public inbox for gentoo-portage-dev@lists.gentoo.org
 help / color / mirror / Atom feed
* [gentoo-portage-dev] [PATCH] unpack: use chmod-lite helper for bug 554084
@ 2015-10-02  7:22 Zac Medico
  2015-10-02  7:32 ` [gentoo-portage-dev] " Zac Medico
  2015-10-02 18:57 ` [gentoo-portage-dev] [PATCH v2] " Zac Medico
  0 siblings, 2 replies; 6+ messages in thread
From: Zac Medico @ 2015-10-02  7:22 UTC (permalink / raw
  To: gentoo-portage-dev; +Cc: Zac Medico

Use the apply_recursive_permissions function to minimize the number of
chmod calls.

Also, fix an UnboundLocalError triggered in portage.data._get_global
by chmod-lite.

X-Gentoo-Bug: 554084
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=554084
---
 bin/chmod-lite       | 10 ++++++++++
 bin/chmod-lite.py    | 30 ++++++++++++++++++++++++++++++
 bin/phase-helpers.sh |  2 +-
 pym/portage/data.py  |  2 +-
 4 files changed, 42 insertions(+), 2 deletions(-)
 create mode 100755 bin/chmod-lite
 create mode 100755 bin/chmod-lite.py

diff --git a/bin/chmod-lite b/bin/chmod-lite
new file mode 100755
index 0000000..ffa8d4d
--- /dev/null
+++ b/bin/chmod-lite
@@ -0,0 +1,10 @@
+#!/bin/bash
+# Copyright 2015 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+export __PORTAGE_HELPER_CWD=${PWD}
+
+# Use safe cwd, avoiding unsafe import for bug #469338.
+cd "${PORTAGE_PYM_PATH}" || exit 1
+PYTHONPATH=${PORTAGE_PYTHONPATH:-${PORTAGE_PYM_PATH}} \
+	exec "${PORTAGE_PYTHON:-/usr/bin/python}" "$PORTAGE_BIN_PATH/chmod-lite.py" "$@"
diff --git a/bin/chmod-lite.py b/bin/chmod-lite.py
new file mode 100755
index 0000000..697cf77
--- /dev/null
+++ b/bin/chmod-lite.py
@@ -0,0 +1,30 @@
+#!/usr/bin/python -b
+# Copyright 2015 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+import os
+import sys
+
+from portage.util import apply_recursive_permissions
+
+# Change back to original cwd _after_ all imports (bug #469338).
+os.chdir(os.environ["__PORTAGE_HELPER_CWD"])
+
+def main(files):
+
+	if sys.hexversion >= 0x3000000:
+		# We can't trust that the filesystem encoding (locale dependent)
+		# correctly matches the arguments, so use surrogateescape to
+		# pass through the original argv bytes for Python 3.
+		fs_encoding = sys.getfilesystemencoding()
+		files = [x.encode(fs_encoding, 'surrogateescape') for x in files]
+
+	for filename in files:
+		# Emulate 'chmod -fR a+rX,u+w,g-w,o-w' with mininal chmod calls.
+		apply_recursive_permissions(filename, filemode=0o644,
+			filemask=0o002, dirmode=0o755, dirmask=0o002)
+
+	return os.EX_OK
+
+if __name__ == "__main__":
+	sys.exit(main(sys.argv[1:]))
diff --git a/bin/phase-helpers.sh b/bin/phase-helpers.sh
index efd2cfa..0c25ffe 100644
--- a/bin/phase-helpers.sh
+++ b/bin/phase-helpers.sh
@@ -532,7 +532,7 @@ unpack() {
 	# Do not chmod '.' since it's probably ${WORKDIR} and PORTAGE_WORKDIR_MODE
 	# should be preserved.
 	find . -mindepth 1 -maxdepth 1 ! -type l -print0 | \
-		${XARGS} -0 chmod -fR a+rX,u+w,g-w,o-w
+		${XARGS} -0 "${PORTAGE_BIN_PATH}/chmod-lite"
 }
 
 econf() {
diff --git a/pym/portage/data.py b/pym/portage/data.py
index 2fd287d..2c99548 100644
--- a/pym/portage/data.py
+++ b/pym/portage/data.py
@@ -139,7 +139,7 @@ def _get_global(k):
 			v = 2
 		elif unprivileged:
 			v = 2
-		elif portage_gid in os.getgroups():
+		elif _get_global('portage_gid') in os.getgroups():
 			v = 1
 
 	elif k in ('portage_gid', 'portage_uid'):
-- 
2.4.6



^ permalink raw reply related	[flat|nested] 6+ messages in thread

* [gentoo-portage-dev] Re: [PATCH] unpack: use chmod-lite helper for bug 554084
  2015-10-02  7:22 [gentoo-portage-dev] [PATCH] unpack: use chmod-lite helper for bug 554084 Zac Medico
@ 2015-10-02  7:32 ` Zac Medico
  2015-10-02 18:57 ` [gentoo-portage-dev] [PATCH v2] " Zac Medico
  1 sibling, 0 replies; 6+ messages in thread
From: Zac Medico @ 2015-10-02  7:32 UTC (permalink / raw
  To: gentoo-portage-dev; +Cc: Zac Medico

On Fri, Oct 2, 2015 at 12:22 AM, Zac Medico <zmedico@gentoo.org> wrote:
> Use the apply_recursive_permissions function to minimize the number of
> chmod calls.

Actually, apply_recursive_permissions fails with
www-apache/mod_auth_token-1.0.6_beta, so it needs some work.


^ permalink raw reply	[flat|nested] 6+ messages in thread

* [gentoo-portage-dev] [PATCH v2] unpack: use chmod-lite helper for bug 554084
  2015-10-02  7:22 [gentoo-portage-dev] [PATCH] unpack: use chmod-lite helper for bug 554084 Zac Medico
  2015-10-02  7:32 ` [gentoo-portage-dev] " Zac Medico
@ 2015-10-02 18:57 ` Zac Medico
  2015-10-02 19:11   ` [gentoo-portage-dev] [PATCH v3] " Zac Medico
  2015-10-03 16:15   ` [gentoo-portage-dev] [PATCH v4] " Zac Medico
  1 sibling, 2 replies; 6+ messages in thread
From: Zac Medico @ 2015-10-02 18:57 UTC (permalink / raw
  To: gentoo-portage-dev; +Cc: Zac Medico

Use the apply_recursive_permissions function to minimize the number of
chmod calls.

Also, fix an UnboundLocalError triggered in portage.data._get_global
by chmod-lite.

X-Gentoo-Bug: 554084
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=554084
---
[PATCH v2] fixes apply_recursive_permissions to apply permissions to
a directory before that directory is traversed.

 bin/chmod-lite               | 10 ++++++++++
 bin/chmod-lite.py            | 30 ++++++++++++++++++++++++++++++
 bin/phase-helpers.sh         |  2 +-
 pym/portage/data.py          |  2 +-
 pym/portage/util/__init__.py | 30 +++++++++++++++++-------------
 5 files changed, 59 insertions(+), 15 deletions(-)
 create mode 100755 bin/chmod-lite
 create mode 100755 bin/chmod-lite.py

diff --git a/bin/chmod-lite b/bin/chmod-lite
new file mode 100755
index 0000000..ffa8d4d
--- /dev/null
+++ b/bin/chmod-lite
@@ -0,0 +1,10 @@
+#!/bin/bash
+# Copyright 2015 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+export __PORTAGE_HELPER_CWD=${PWD}
+
+# Use safe cwd, avoiding unsafe import for bug #469338.
+cd "${PORTAGE_PYM_PATH}" || exit 1
+PYTHONPATH=${PORTAGE_PYTHONPATH:-${PORTAGE_PYM_PATH}} \
+	exec "${PORTAGE_PYTHON:-/usr/bin/python}" "$PORTAGE_BIN_PATH/chmod-lite.py" "$@"
diff --git a/bin/chmod-lite.py b/bin/chmod-lite.py
new file mode 100755
index 0000000..a552b25
--- /dev/null
+++ b/bin/chmod-lite.py
@@ -0,0 +1,30 @@
+#!/usr/bin/python -b
+# Copyright 2015 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+import os
+import sys
+
+from portage.util import apply_recursive_permissions
+
+# Change back to original cwd _after_ all imports (bug #469338).
+os.chdir(os.environ["__PORTAGE_HELPER_CWD"])
+
+def main(files):
+
+	if sys.hexversion >= 0x3000000:
+		# We can't trust that the filesystem encoding (locale dependent)
+		# correctly matches the arguments, so use surrogateescape to
+		# pass through the original argv bytes for Python 3.
+		fs_encoding = sys.getfilesystemencoding()
+		files = [x.encode(fs_encoding, 'surrogateescape') for x in files]
+
+	for filename in files:
+		# Emulate 'chmod -fR a+rX,u+w,g-w,o-w' with mininal chmod calls.
+		apply_recursive_permissions(filename, filemode=0o644,
+			filemask=0o022, dirmode=0o755, dirmask=0o022)
+
+	return os.EX_OK
+
+if __name__ == "__main__":
+	sys.exit(main(sys.argv[1:]))
diff --git a/bin/phase-helpers.sh b/bin/phase-helpers.sh
index efd2cfa..0c25ffe 100644
--- a/bin/phase-helpers.sh
+++ b/bin/phase-helpers.sh
@@ -532,7 +532,7 @@ unpack() {
 	# Do not chmod '.' since it's probably ${WORKDIR} and PORTAGE_WORKDIR_MODE
 	# should be preserved.
 	find . -mindepth 1 -maxdepth 1 ! -type l -print0 | \
-		${XARGS} -0 chmod -fR a+rX,u+w,g-w,o-w
+		${XARGS} -0 "${PORTAGE_BIN_PATH}/chmod-lite"
 }
 
 econf() {
diff --git a/pym/portage/data.py b/pym/portage/data.py
index 2fd287d..2c99548 100644
--- a/pym/portage/data.py
+++ b/pym/portage/data.py
@@ -139,7 +139,7 @@ def _get_global(k):
 			v = 2
 		elif unprivileged:
 			v = 2
-		elif portage_gid in os.getgroups():
+		elif _get_global('portage_gid') in os.getgroups():
 			v = 1
 
 	elif k in ('portage_gid', 'portage_uid'):
diff --git a/pym/portage/util/__init__.py b/pym/portage/util/__init__.py
index c0b509b..2dcd5b6 100644
--- a/pym/portage/util/__init__.py
+++ b/pym/portage/util/__init__.py
@@ -17,9 +17,9 @@ from copy import deepcopy
 import errno
 import io
 try:
-	from itertools import filterfalse
+	from itertools import chain, filterfalse
 except ImportError:
-	from itertools import ifilterfalse as filterfalse
+	from itertools import chain, ifilterfalse as filterfalse
 import logging
 import re
 import shlex
@@ -1177,22 +1177,26 @@ def apply_recursive_permissions(top, uid=-1, gid=-1,
 			else:
 				raise
 
+	# For bug 554084, always apply permissions to a directory before
+	# that directory is traversed.
 	all_applied = True
-	for dirpath, dirnames, filenames in os.walk(top):
-		try:
-			applied = apply_secpass_permissions(dirpath,
-				uid=uid, gid=gid, mode=dirmode, mask=dirmask,
-				follow_links=follow_links)
-			if not applied:
-				all_applied = False
-		except PortageException as e:
+
+	try:
+		applied = apply_secpass_permissions(top,
+			uid=uid, gid=gid, mode=dirmode, mask=dirmask,
+			follow_links=follow_links)
+		if not applied:
 			all_applied = False
-			onerror(e)
+	except PortageException as e:
+		all_applied = False
+		onerror(e)
 
-		for name in filenames:
+	for dirpath, dirnames, filenames in os.walk(top):
+		for name, mode in chain(((x, filemode) for x in filenames),
+			((x, dirmode) for x in dirnames)):
 			try:
 				applied = apply_secpass_permissions(os.path.join(dirpath, name),
-					uid=uid, gid=gid, mode=filemode, mask=filemask,
+					uid=uid, gid=gid, mode=mode, mask=filemask,
 					follow_links=follow_links)
 				if not applied:
 					all_applied = False
-- 
2.4.6



^ permalink raw reply related	[flat|nested] 6+ messages in thread

* [gentoo-portage-dev] [PATCH v3] unpack: use chmod-lite helper for bug 554084
  2015-10-02 18:57 ` [gentoo-portage-dev] [PATCH v2] " Zac Medico
@ 2015-10-02 19:11   ` Zac Medico
  2015-10-03 16:15   ` [gentoo-portage-dev] [PATCH v4] " Zac Medico
  1 sibling, 0 replies; 6+ messages in thread
From: Zac Medico @ 2015-10-02 19:11 UTC (permalink / raw
  To: gentoo-portage-dev; +Cc: Zac Medico

Use the apply_recursive_permissions function to minimize the number of
chmod calls.

Also, fix an UnboundLocalError triggered in portage.data._get_global
by chmod-lite.

X-Gentoo-Bug: 554084
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=554084
---
[PATCH v3] fixes apply_recursive_permissions to properly separate modes
and masks for files and directories

 bin/chmod-lite               | 10 +++++
 bin/chmod-lite.py            | 30 +++++++++++++++
 bin/phase-helpers.sh         |  2 +-
 pym/portage/data.py          |  2 +-
 pym/portage/util/__init__.py | 88 ++++++++++++++++++++++----------------------
 5 files changed, 87 insertions(+), 45 deletions(-)
 create mode 100755 bin/chmod-lite
 create mode 100755 bin/chmod-lite.py

diff --git a/bin/chmod-lite b/bin/chmod-lite
new file mode 100755
index 0000000..ffa8d4d
--- /dev/null
+++ b/bin/chmod-lite
@@ -0,0 +1,10 @@
+#!/bin/bash
+# Copyright 2015 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+export __PORTAGE_HELPER_CWD=${PWD}
+
+# Use safe cwd, avoiding unsafe import for bug #469338.
+cd "${PORTAGE_PYM_PATH}" || exit 1
+PYTHONPATH=${PORTAGE_PYTHONPATH:-${PORTAGE_PYM_PATH}} \
+	exec "${PORTAGE_PYTHON:-/usr/bin/python}" "$PORTAGE_BIN_PATH/chmod-lite.py" "$@"
diff --git a/bin/chmod-lite.py b/bin/chmod-lite.py
new file mode 100755
index 0000000..a552b25
--- /dev/null
+++ b/bin/chmod-lite.py
@@ -0,0 +1,30 @@
+#!/usr/bin/python -b
+# Copyright 2015 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+import os
+import sys
+
+from portage.util import apply_recursive_permissions
+
+# Change back to original cwd _after_ all imports (bug #469338).
+os.chdir(os.environ["__PORTAGE_HELPER_CWD"])
+
+def main(files):
+
+	if sys.hexversion >= 0x3000000:
+		# We can't trust that the filesystem encoding (locale dependent)
+		# correctly matches the arguments, so use surrogateescape to
+		# pass through the original argv bytes for Python 3.
+		fs_encoding = sys.getfilesystemencoding()
+		files = [x.encode(fs_encoding, 'surrogateescape') for x in files]
+
+	for filename in files:
+		# Emulate 'chmod -fR a+rX,u+w,g-w,o-w' with mininal chmod calls.
+		apply_recursive_permissions(filename, filemode=0o644,
+			filemask=0o022, dirmode=0o755, dirmask=0o022)
+
+	return os.EX_OK
+
+if __name__ == "__main__":
+	sys.exit(main(sys.argv[1:]))
diff --git a/bin/phase-helpers.sh b/bin/phase-helpers.sh
index efd2cfa..0c25ffe 100644
--- a/bin/phase-helpers.sh
+++ b/bin/phase-helpers.sh
@@ -532,7 +532,7 @@ unpack() {
 	# Do not chmod '.' since it's probably ${WORKDIR} and PORTAGE_WORKDIR_MODE
 	# should be preserved.
 	find . -mindepth 1 -maxdepth 1 ! -type l -print0 | \
-		${XARGS} -0 chmod -fR a+rX,u+w,g-w,o-w
+		${XARGS} -0 "${PORTAGE_BIN_PATH}/chmod-lite"
 }
 
 econf() {
diff --git a/pym/portage/data.py b/pym/portage/data.py
index 2fd287d..2c99548 100644
--- a/pym/portage/data.py
+++ b/pym/portage/data.py
@@ -139,7 +139,7 @@ def _get_global(k):
 			v = 2
 		elif unprivileged:
 			v = 2
-		elif portage_gid in os.getgroups():
+		elif _get_global('portage_gid') in os.getgroups():
 			v = 1
 
 	elif k in ('portage_gid', 'portage_uid'):
diff --git a/pym/portage/util/__init__.py b/pym/portage/util/__init__.py
index c0b509b..38780a7 100644
--- a/pym/portage/util/__init__.py
+++ b/pym/portage/util/__init__.py
@@ -17,9 +17,9 @@ from copy import deepcopy
 import errno
 import io
 try:
-	from itertools import filterfalse
+	from itertools import chain, filterfalse
 except ImportError:
-	from itertools import ifilterfalse as filterfalse
+	from itertools import chain, ifilterfalse as filterfalse
 import logging
 import re
 import shlex
@@ -1041,6 +1041,23 @@ def unique_everseen(iterable, key=None):
 				seen_add(k)
 				yield element
 
+def _do_stat(filename, follow_links=True):
+	try:
+		if follow_links:
+			return os.stat(filename)
+		else:
+			return os.lstat(filename)
+	except OSError as oe:
+		func_call = "stat('%s')" % filename
+		if oe.errno == errno.EPERM:
+			raise OperationNotPermitted(func_call)
+		elif oe.errno == errno.EACCES:
+			raise PermissionDenied(func_call)
+		elif oe.errno == errno.ENOENT:
+			raise FileNotFound(filename)
+		else:
+			raise
+
 def apply_permissions(filename, uid=-1, gid=-1, mode=-1, mask=-1,
 	stat_cached=None, follow_links=True):
 	"""Apply user, group, and mode bits to a file if the existing bits do not
@@ -1058,21 +1075,7 @@ def apply_permissions(filename, uid=-1, gid=-1, mode=-1, mask=-1,
 	gid = int(gid)
 
 	if stat_cached is None:
-		try:
-			if follow_links:
-				stat_cached = os.stat(filename)
-			else:
-				stat_cached = os.lstat(filename)
-		except OSError as oe:
-			func_call = "stat('%s')" % filename
-			if oe.errno == errno.EPERM:
-				raise OperationNotPermitted(func_call)
-			elif oe.errno == errno.EACCES:
-				raise PermissionDenied(func_call)
-			elif oe.errno == errno.ENOENT:
-				raise FileNotFound(filename)
-			else:
-				raise
+		stat_cached = _do_stat(filename, follow_links=follow_links)
 
 	if	(uid != -1 and uid != stat_cached.st_uid) or \
 		(gid != -1 and gid != stat_cached.st_gid):
@@ -1177,22 +1180,35 @@ def apply_recursive_permissions(top, uid=-1, gid=-1,
 			else:
 				raise
 
+	# For bug 554084, always apply permissions to a directory before
+	# that directory is traversed.
 	all_applied = True
-	for dirpath, dirnames, filenames in os.walk(top):
-		try:
-			applied = apply_secpass_permissions(dirpath,
-				uid=uid, gid=gid, mode=dirmode, mask=dirmask,
-				follow_links=follow_links)
-			if not applied:
-				all_applied = False
-		except PortageException as e:
+
+	stat_cached = _do_stat(top, follow_links=follow_links)
+	if stat.S_ISDIR(stat_cached.st_mode):
+		mode = dirmode
+		mask = dirmask
+	else:
+		mode = filemode
+		mask = filemask
+
+	try:
+		applied = apply_secpass_permissions(top,
+			uid=uid, gid=gid, mode=mode, mask=mask,
+			stat_cached=stat_cached, follow_links=follow_links)
+		if not applied:
 			all_applied = False
-			onerror(e)
+	except PortageException as e:
+		all_applied = False
+		onerror(e)
 
-		for name in filenames:
+	for dirpath, dirnames, filenames in os.walk(top):
+		for name, mode, mask in chain(
+			((x, filemode, filemask) for x in filenames),
+			((x, dirmode, dirmask) for x in dirnames)):
 			try:
 				applied = apply_secpass_permissions(os.path.join(dirpath, name),
-					uid=uid, gid=gid, mode=filemode, mask=filemask,
+					uid=uid, gid=gid, mode=mode, mask=mask,
 					follow_links=follow_links)
 				if not applied:
 					all_applied = False
@@ -1216,21 +1232,7 @@ def apply_secpass_permissions(filename, uid=-1, gid=-1, mode=-1, mask=-1,
 	unapplied."""
 
 	if stat_cached is None:
-		try:
-			if follow_links:
-				stat_cached = os.stat(filename)
-			else:
-				stat_cached = os.lstat(filename)
-		except OSError as oe:
-			func_call = "stat('%s')" % filename
-			if oe.errno == errno.EPERM:
-				raise OperationNotPermitted(func_call)
-			elif oe.errno == errno.EACCES:
-				raise PermissionDenied(func_call)
-			elif oe.errno == errno.ENOENT:
-				raise FileNotFound(filename)
-			else:
-				raise
+		stat_cached = _do_stat(filename, follow_links=follow_links)
 
 	all_applied = True
 
-- 
2.4.6



^ permalink raw reply related	[flat|nested] 6+ messages in thread

* [gentoo-portage-dev] [PATCH v4] unpack: use chmod-lite helper for bug 554084
  2015-10-02 18:57 ` [gentoo-portage-dev] [PATCH v2] " Zac Medico
  2015-10-02 19:11   ` [gentoo-portage-dev] [PATCH v3] " Zac Medico
@ 2015-10-03 16:15   ` Zac Medico
  2015-10-03 16:24     ` Brian Dolbec
  1 sibling, 1 reply; 6+ messages in thread
From: Zac Medico @ 2015-10-03 16:15 UTC (permalink / raw
  To: gentoo-portage-dev; +Cc: Zac Medico

Use the apply_recursive_permissions function to minimize the number of
chmod calls.

Also, fix an UnboundLocalError triggered in portage.data._get_global
by chmod-lite.

X-Gentoo-Bug: 554084
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=554084
---
[PATCH v4] adds backward compatibility to apply_recursive_permissions
for the case where the given path does not exist

 bin/chmod-lite               | 10 +++++
 bin/chmod-lite.py            | 30 ++++++++++++++
 bin/phase-helpers.sh         |  2 +-
 pym/portage/data.py          |  2 +-
 pym/portage/util/__init__.py | 93 ++++++++++++++++++++++++--------------------
 5 files changed, 92 insertions(+), 45 deletions(-)
 create mode 100755 bin/chmod-lite
 create mode 100755 bin/chmod-lite.py

diff --git a/bin/chmod-lite b/bin/chmod-lite
new file mode 100755
index 0000000..ffa8d4d
--- /dev/null
+++ b/bin/chmod-lite
@@ -0,0 +1,10 @@
+#!/bin/bash
+# Copyright 2015 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+export __PORTAGE_HELPER_CWD=${PWD}
+
+# Use safe cwd, avoiding unsafe import for bug #469338.
+cd "${PORTAGE_PYM_PATH}" || exit 1
+PYTHONPATH=${PORTAGE_PYTHONPATH:-${PORTAGE_PYM_PATH}} \
+	exec "${PORTAGE_PYTHON:-/usr/bin/python}" "$PORTAGE_BIN_PATH/chmod-lite.py" "$@"
diff --git a/bin/chmod-lite.py b/bin/chmod-lite.py
new file mode 100755
index 0000000..177be7e
--- /dev/null
+++ b/bin/chmod-lite.py
@@ -0,0 +1,30 @@
+#!/usr/bin/python -b
+# Copyright 2015 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+import os
+import sys
+
+from portage.util import apply_recursive_permissions
+
+# Change back to original cwd _after_ all imports (bug #469338).
+os.chdir(os.environ["__PORTAGE_HELPER_CWD"])
+
+def main(files):
+
+	if sys.hexversion >= 0x3000000:
+		# We can't trust that the filesystem encoding (locale dependent)
+		# correctly matches the arguments, so use surrogateescape to
+		# pass through the original argv bytes for Python 3.
+		fs_encoding = sys.getfilesystemencoding()
+		files = [x.encode(fs_encoding, 'surrogateescape') for x in files]
+
+	for filename in files:
+		# Emulate 'chmod -fR a+rX,u+w,g-w,o-w' with minimal chmod calls.
+		apply_recursive_permissions(filename, filemode=0o644,
+			filemask=0o022, dirmode=0o755, dirmask=0o022)
+
+	return os.EX_OK
+
+if __name__ == "__main__":
+	sys.exit(main(sys.argv[1:]))
diff --git a/bin/phase-helpers.sh b/bin/phase-helpers.sh
index efd2cfa..0c25ffe 100644
--- a/bin/phase-helpers.sh
+++ b/bin/phase-helpers.sh
@@ -532,7 +532,7 @@ unpack() {
 	# Do not chmod '.' since it's probably ${WORKDIR} and PORTAGE_WORKDIR_MODE
 	# should be preserved.
 	find . -mindepth 1 -maxdepth 1 ! -type l -print0 | \
-		${XARGS} -0 chmod -fR a+rX,u+w,g-w,o-w
+		${XARGS} -0 "${PORTAGE_BIN_PATH}/chmod-lite"
 }
 
 econf() {
diff --git a/pym/portage/data.py b/pym/portage/data.py
index 2fd287d..2c99548 100644
--- a/pym/portage/data.py
+++ b/pym/portage/data.py
@@ -139,7 +139,7 @@ def _get_global(k):
 			v = 2
 		elif unprivileged:
 			v = 2
-		elif portage_gid in os.getgroups():
+		elif _get_global('portage_gid') in os.getgroups():
 			v = 1
 
 	elif k in ('portage_gid', 'portage_uid'):
diff --git a/pym/portage/util/__init__.py b/pym/portage/util/__init__.py
index c0b509b..2b7ff8d 100644
--- a/pym/portage/util/__init__.py
+++ b/pym/portage/util/__init__.py
@@ -17,9 +17,9 @@ from copy import deepcopy
 import errno
 import io
 try:
-	from itertools import filterfalse
+	from itertools import chain, filterfalse
 except ImportError:
-	from itertools import ifilterfalse as filterfalse
+	from itertools import chain, ifilterfalse as filterfalse
 import logging
 import re
 import shlex
@@ -1041,6 +1041,23 @@ def unique_everseen(iterable, key=None):
 				seen_add(k)
 				yield element
 
+def _do_stat(filename, follow_links=True):
+	try:
+		if follow_links:
+			return os.stat(filename)
+		else:
+			return os.lstat(filename)
+	except OSError as oe:
+		func_call = "stat('%s')" % filename
+		if oe.errno == errno.EPERM:
+			raise OperationNotPermitted(func_call)
+		elif oe.errno == errno.EACCES:
+			raise PermissionDenied(func_call)
+		elif oe.errno == errno.ENOENT:
+			raise FileNotFound(filename)
+		else:
+			raise
+
 def apply_permissions(filename, uid=-1, gid=-1, mode=-1, mask=-1,
 	stat_cached=None, follow_links=True):
 	"""Apply user, group, and mode bits to a file if the existing bits do not
@@ -1058,21 +1075,7 @@ def apply_permissions(filename, uid=-1, gid=-1, mode=-1, mask=-1,
 	gid = int(gid)
 
 	if stat_cached is None:
-		try:
-			if follow_links:
-				stat_cached = os.stat(filename)
-			else:
-				stat_cached = os.lstat(filename)
-		except OSError as oe:
-			func_call = "stat('%s')" % filename
-			if oe.errno == errno.EPERM:
-				raise OperationNotPermitted(func_call)
-			elif oe.errno == errno.EACCES:
-				raise PermissionDenied(func_call)
-			elif oe.errno == errno.ENOENT:
-				raise FileNotFound(filename)
-			else:
-				raise
+		stat_cached = _do_stat(filename, follow_links=follow_links)
 
 	if	(uid != -1 and uid != stat_cached.st_uid) or \
 		(gid != -1 and gid != stat_cached.st_gid):
@@ -1177,22 +1180,40 @@ def apply_recursive_permissions(top, uid=-1, gid=-1,
 			else:
 				raise
 
+	# For bug 554084, always apply permissions to a directory before
+	# that directory is traversed.
 	all_applied = True
-	for dirpath, dirnames, filenames in os.walk(top):
-		try:
-			applied = apply_secpass_permissions(dirpath,
-				uid=uid, gid=gid, mode=dirmode, mask=dirmask,
-				follow_links=follow_links)
-			if not applied:
-				all_applied = False
-		except PortageException as e:
+
+	try:
+		stat_cached = _do_stat(top, follow_links=follow_links)
+	except FileNotFound:
+		# backward compatibility
+		return True
+
+	if stat.S_ISDIR(stat_cached.st_mode):
+		mode = dirmode
+		mask = dirmask
+	else:
+		mode = filemode
+		mask = filemask
+
+	try:
+		applied = apply_secpass_permissions(top,
+			uid=uid, gid=gid, mode=mode, mask=mask,
+			stat_cached=stat_cached, follow_links=follow_links)
+		if not applied:
 			all_applied = False
-			onerror(e)
+	except PortageException as e:
+		all_applied = False
+		onerror(e)
 
-		for name in filenames:
+	for dirpath, dirnames, filenames in os.walk(top):
+		for name, mode, mask in chain(
+			((x, filemode, filemask) for x in filenames),
+			((x, dirmode, dirmask) for x in dirnames)):
 			try:
 				applied = apply_secpass_permissions(os.path.join(dirpath, name),
-					uid=uid, gid=gid, mode=filemode, mask=filemask,
+					uid=uid, gid=gid, mode=mode, mask=mask,
 					follow_links=follow_links)
 				if not applied:
 					all_applied = False
@@ -1216,21 +1237,7 @@ def apply_secpass_permissions(filename, uid=-1, gid=-1, mode=-1, mask=-1,
 	unapplied."""
 
 	if stat_cached is None:
-		try:
-			if follow_links:
-				stat_cached = os.stat(filename)
-			else:
-				stat_cached = os.lstat(filename)
-		except OSError as oe:
-			func_call = "stat('%s')" % filename
-			if oe.errno == errno.EPERM:
-				raise OperationNotPermitted(func_call)
-			elif oe.errno == errno.EACCES:
-				raise PermissionDenied(func_call)
-			elif oe.errno == errno.ENOENT:
-				raise FileNotFound(filename)
-			else:
-				raise
+		stat_cached = _do_stat(filename, follow_links=follow_links)
 
 	all_applied = True
 
-- 
2.4.6



^ permalink raw reply related	[flat|nested] 6+ messages in thread

* Re: [gentoo-portage-dev] [PATCH v4] unpack: use chmod-lite helper for bug 554084
  2015-10-03 16:15   ` [gentoo-portage-dev] [PATCH v4] " Zac Medico
@ 2015-10-03 16:24     ` Brian Dolbec
  0 siblings, 0 replies; 6+ messages in thread
From: Brian Dolbec @ 2015-10-03 16:24 UTC (permalink / raw
  To: gentoo-portage-dev

On Sat,  3 Oct 2015 09:15:56 -0700
Zac Medico <zmedico@gentoo.org> wrote:

> Use the apply_recursive_permissions function to minimize the number of
> chmod calls.
> 
> Also, fix an UnboundLocalError triggered in portage.data._get_global
> by chmod-lite.
> 
> X-Gentoo-Bug: 554084
> X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=554084
> ---
> [PATCH v4] adds backward compatibility to apply_recursive_permissions
> for the case where the given path does not exist
> 
>  bin/chmod-lite               | 10 +++++
>  bin/chmod-lite.py            | 30 ++++++++++++++
>  bin/phase-helpers.sh         |  2 +-
>  pym/portage/data.py          |  2 +-
>  pym/portage/util/__init__.py | 93
> ++++++++++++++++++++++++-------------------- 5 files changed, 92
> insertions(+), 45 deletions(-) create mode 100755 bin/chmod-lite
>  create mode 100755 bin/chmod-lite.py
> 
> diff --git a/bin/chmod-lite b/bin/chmod-lite
> new file mode 100755
> index 0000000..ffa8d4d
> --- /dev/null
> +++ b/bin/chmod-lite
> @@ -0,0 +1,10 @@
> +#!/bin/bash
> +# Copyright 2015 Gentoo Foundation
> +# Distributed under the terms of the GNU General Public License v2
> +
> +export __PORTAGE_HELPER_CWD=${PWD}
> +
> +# Use safe cwd, avoiding unsafe import for bug #469338.
> +cd "${PORTAGE_PYM_PATH}" || exit 1
> +PYTHONPATH=${PORTAGE_PYTHONPATH:-${PORTAGE_PYM_PATH}} \
> +	exec "${PORTAGE_PYTHON:-/usr/bin/python}"
> "$PORTAGE_BIN_PATH/chmod-lite.py" "$@" diff --git a/bin/chmod-lite.py
> b/bin/chmod-lite.py new file mode 100755
> index 0000000..177be7e
> --- /dev/null
> +++ b/bin/chmod-lite.py
> @@ -0,0 +1,30 @@
> +#!/usr/bin/python -b
> +# Copyright 2015 Gentoo Foundation
> +# Distributed under the terms of the GNU General Public License v2
> +
> +import os
> +import sys
> +
> +from portage.util import apply_recursive_permissions
> +
> +# Change back to original cwd _after_ all imports (bug #469338).
> +os.chdir(os.environ["__PORTAGE_HELPER_CWD"])
> +
> +def main(files):
> +
> +	if sys.hexversion >= 0x3000000:
> +		# We can't trust that the filesystem encoding
> (locale dependent)
> +		# correctly matches the arguments, so use
> surrogateescape to
> +		# pass through the original argv bytes for Python 3.
> +		fs_encoding = sys.getfilesystemencoding()
> +		files = [x.encode(fs_encoding, 'surrogateescape')
> for x in files] +
> +	for filename in files:
> +		# Emulate 'chmod -fR a+rX,u+w,g-w,o-w' with minimal
> chmod calls.
> +		apply_recursive_permissions(filename, filemode=0o644,
> +			filemask=0o022, dirmode=0o755, dirmask=0o022)
> +
> +	return os.EX_OK
> +
> +if __name__ == "__main__":
> +	sys.exit(main(sys.argv[1:]))
> diff --git a/bin/phase-helpers.sh b/bin/phase-helpers.sh
> index efd2cfa..0c25ffe 100644
> --- a/bin/phase-helpers.sh
> +++ b/bin/phase-helpers.sh
> @@ -532,7 +532,7 @@ unpack() {
>  	# Do not chmod '.' since it's probably ${WORKDIR} and
> PORTAGE_WORKDIR_MODE # should be preserved.
>  	find . -mindepth 1 -maxdepth 1 ! -type l -print0 | \
> -		${XARGS} -0 chmod -fR a+rX,u+w,g-w,o-w
> +		${XARGS} -0 "${PORTAGE_BIN_PATH}/chmod-lite"
>  }
>  
>  econf() {
> diff --git a/pym/portage/data.py b/pym/portage/data.py
> index 2fd287d..2c99548 100644
> --- a/pym/portage/data.py
> +++ b/pym/portage/data.py
> @@ -139,7 +139,7 @@ def _get_global(k):
>  			v = 2
>  		elif unprivileged:
>  			v = 2
> -		elif portage_gid in os.getgroups():
> +		elif _get_global('portage_gid') in os.getgroups():
>  			v = 1
>  
>  	elif k in ('portage_gid', 'portage_uid'):
> diff --git a/pym/portage/util/__init__.py
> b/pym/portage/util/__init__.py index c0b509b..2b7ff8d 100644
> --- a/pym/portage/util/__init__.py
> +++ b/pym/portage/util/__init__.py
> @@ -17,9 +17,9 @@ from copy import deepcopy
>  import errno
>  import io
>  try:
> -	from itertools import filterfalse
> +	from itertools import chain, filterfalse
>  except ImportError:
> -	from itertools import ifilterfalse as filterfalse
> +	from itertools import chain, ifilterfalse as filterfalse
>  import logging
>  import re
>  import shlex
> @@ -1041,6 +1041,23 @@ def unique_everseen(iterable, key=None):
>  				seen_add(k)
>  				yield element
>  
> +def _do_stat(filename, follow_links=True):
> +	try:
> +		if follow_links:
> +			return os.stat(filename)
> +		else:
> +			return os.lstat(filename)
> +	except OSError as oe:
> +		func_call = "stat('%s')" % filename
> +		if oe.errno == errno.EPERM:
> +			raise OperationNotPermitted(func_call)
> +		elif oe.errno == errno.EACCES:
> +			raise PermissionDenied(func_call)
> +		elif oe.errno == errno.ENOENT:
> +			raise FileNotFound(filename)
> +		else:
> +			raise
> +
>  def apply_permissions(filename, uid=-1, gid=-1, mode=-1, mask=-1,
>  	stat_cached=None, follow_links=True):
>  	"""Apply user, group, and mode bits to a file if the
> existing bits do not @@ -1058,21 +1075,7 @@ def
> apply_permissions(filename, uid=-1, gid=-1, mode=-1, mask=-1, gid =
> int(gid) 
>  	if stat_cached is None:
> -		try:
> -			if follow_links:
> -				stat_cached = os.stat(filename)
> -			else:
> -				stat_cached = os.lstat(filename)
> -		except OSError as oe:
> -			func_call = "stat('%s')" % filename
> -			if oe.errno == errno.EPERM:
> -				raise
> OperationNotPermitted(func_call)
> -			elif oe.errno == errno.EACCES:
> -				raise PermissionDenied(func_call)
> -			elif oe.errno == errno.ENOENT:
> -				raise FileNotFound(filename)
> -			else:
> -				raise
> +		stat_cached = _do_stat(filename,
> follow_links=follow_links) 
>  	if	(uid != -1 and uid != stat_cached.st_uid) or \
>  		(gid != -1 and gid != stat_cached.st_gid):
> @@ -1177,22 +1180,40 @@ def apply_recursive_permissions(top, uid=-1,
> gid=-1, else:
>  				raise
>  
> +	# For bug 554084, always apply permissions to a directory
> before
> +	# that directory is traversed.
>  	all_applied = True
> -	for dirpath, dirnames, filenames in os.walk(top):
> -		try:
> -			applied = apply_secpass_permissions(dirpath,
> -				uid=uid, gid=gid, mode=dirmode,
> mask=dirmask,
> -				follow_links=follow_links)
> -			if not applied:
> -				all_applied = False
> -		except PortageException as e:
> +
> +	try:
> +		stat_cached = _do_stat(top,
> follow_links=follow_links)
> +	except FileNotFound:
> +		# backward compatibility
> +		return True
> +
> +	if stat.S_ISDIR(stat_cached.st_mode):
> +		mode = dirmode
> +		mask = dirmask
> +	else:
> +		mode = filemode
> +		mask = filemask
> +
> +	try:
> +		applied = apply_secpass_permissions(top,
> +			uid=uid, gid=gid, mode=mode, mask=mask,
> +			stat_cached=stat_cached,
> follow_links=follow_links)
> +		if not applied:
>  			all_applied = False
> -			onerror(e)
> +	except PortageException as e:
> +		all_applied = False
> +		onerror(e)
>  
> -		for name in filenames:
> +	for dirpath, dirnames, filenames in os.walk(top):
> +		for name, mode, mask in chain(
> +			((x, filemode, filemask) for x in filenames),
> +			((x, dirmode, dirmask) for x in dirnames)):
>  			try:
>  				applied =
> apply_secpass_permissions(os.path.join(dirpath, name),
> -					uid=uid, gid=gid,
> mode=filemode, mask=filemask,
> +					uid=uid, gid=gid, mode=mode,
> mask=mask, follow_links=follow_links)
>  				if not applied:
>  					all_applied = False
> @@ -1216,21 +1237,7 @@ def apply_secpass_permissions(filename,
> uid=-1, gid=-1, mode=-1, mask=-1, unapplied."""
>  
>  	if stat_cached is None:
> -		try:
> -			if follow_links:
> -				stat_cached = os.stat(filename)
> -			else:
> -				stat_cached = os.lstat(filename)
> -		except OSError as oe:
> -			func_call = "stat('%s')" % filename
> -			if oe.errno == errno.EPERM:
> -				raise
> OperationNotPermitted(func_call)
> -			elif oe.errno == errno.EACCES:
> -				raise PermissionDenied(func_call)
> -			elif oe.errno == errno.ENOENT:
> -				raise FileNotFound(filename)
> -			else:
> -				raise
> +		stat_cached = _do_stat(filename,
> follow_links=follow_links) 
>  	all_applied = True
>  

LGTM

-- 
Brian Dolbec <dolsen>



^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2015-10-03 16:25 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-10-02  7:22 [gentoo-portage-dev] [PATCH] unpack: use chmod-lite helper for bug 554084 Zac Medico
2015-10-02  7:32 ` [gentoo-portage-dev] " Zac Medico
2015-10-02 18:57 ` [gentoo-portage-dev] [PATCH v2] " Zac Medico
2015-10-02 19:11   ` [gentoo-portage-dev] [PATCH v3] " Zac Medico
2015-10-03 16:15   ` [gentoo-portage-dev] [PATCH v4] " Zac Medico
2015-10-03 16:24     ` Brian Dolbec

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox