[gentoo-dev] Prelink, unpriv user compile and fakeroot

[gentoo-dev] Prelink, unpriv user compile and fakeroot

[Stefan Jones]

[-- Attachment #1: Type: text/plain, Size: 2130 bytes --]

Hi all,

When I started using prelink and portage some apps started to segfault
in the portage compile/install environment. This was traced to the
LD_PRELOAD environmental variable. Prelinked apps don't like being
LD_PRELOADed.

Most of / all these problems are due to buggy / nonstandard compile
methods and glibc bugs with preloading and prelink (one glibc bug has
already been fixed)

As a work around to the above problems and a permanent fix to sandbox
problems I started working on jrray's fakeroot portage patch ( look at
http://cvs.gentoo.org/~jrray )

What was needed was for compiles to be done by a normal unprivileged
user ( like nobody, or the new portage user ) without ANY sandbox or
other library being preloaded. This fixes most compile time problems
with things like emacs. This is also supported by all package I know as
it is the normal manual install method most people use.

This safe as the user has no write permissions to most of the file
system. Also the functions of sandbox are not needed during compile as
during this stage nothing needs to be installed properly.

Then the install stage can be done as root with libsandbox as normal as
during this stage no tricky library manipulations are being done so
LD_PRELOAD will not interfere.

The attached patch only uses sandbox, but can be easily hacked to use
fakeroot instead/as well. The patch is a radical hack of jrray's
version.

It can also be found at

http://cvs.gentoo.org/~cretin/portage-prelink+fakeroot.patch

Install:
Install prelink if you want to prelink, you can use the patch without
prelink to get the unpriv user compile feature.

Then apply patch:
This a patch against the portage-2.0.44.tar.bz2 tarball found in
/usr/portage so add to the portage ebuild
cd ${S} ; bzip2 -dc /path/to/portage-prelink+fakeroot.patch | patch -p1
|| die

Next add a "portage" group and a user called "portage" with home dir at
/home/portage. Finally add prelink to FEATURES in make.conf if you want
prelink.

Now watch the breakages ....
(only joking, it works fine for me )

Is software ever finished?

-- 
Stefan Jones <cretin@gentoo.org>
Gentoo Linux

[-- Attachment #2: portage-prelink+fakeroot.patch --]
[-- Type: text/x-patch, Size: 14510 bytes --]

diff -ruN portage-2.0.44.orig/bin/ebuild.sh portage-2.0.44/bin/ebuild.sh
--- portage-2.0.44.orig/bin/ebuild.sh	2002-11-11 19:13:08.000000000 +0000
+++ portage-2.0.44/bin/ebuild.sh	2002-11-20 20:53:38.000000000 +0000
@@ -130,7 +130,7 @@
 	export PATH="/usr/bin/ccache:${PATH}"
 	if [ -z "${CCACHE_DIR}" ]
 	then
-		CCACHE_DIR=/root/.ccache
+		export CCACHE_DIR=/home/portage/.ccache
 	fi
 	addread ${CCACHE_DIR}
 	addwrite ${CCACHE_DIR}
@@ -569,6 +569,8 @@
 	src_install 
 	#|| abort_install "fail"
 	prepall
+	echo ">>> Recording file permissions of ${D}"
+	/usr/lib/portage/bin/savestats ${T}/perms.db ${BUILDDIR}/image
 	cd ${D}
 	echo ">>> Completed installing into ${D}"
 	echo
@@ -692,6 +694,8 @@
 		
 		# default target
 		[ -n "$T" ] && echo $1 >> ${T}/eclass-debug.log
+		# let the portage user own/write to this file
+		[ -n "$T" ] && chown portage.portage ${T}/eclass-debug.log
 		
 		shift
 	done
diff -ruN portage-2.0.44.orig/bin/postallprelink portage-2.0.44/bin/postallprelink
--- portage-2.0.44.orig/bin/postallprelink	1970-01-01 01:00:00.000000000 +0100
+++ portage-2.0.44/bin/postallprelink	2002-11-20 20:53:38.000000000 +0000
@@ -0,0 +1,10 @@
+#!/bin/bash
+
+FILES=`cat $1`
+
+echo "prelinking:"
+for x in $FILES ; do
+	x=${x/*image/}
+	echo $x
+	prelink -mR $x
+done
diff -ruN portage-2.0.44.orig/bin/prepallstrip portage-2.0.44/bin/prepallstrip
--- portage-2.0.44.orig/bin/prepallstrip	2002-11-09 09:00:58.000000000 +0000
+++ portage-2.0.44/bin/prepallstrip	2002-11-20 20:53:38.000000000 +0000
@@ -6,6 +6,7 @@
 fi	
 echo "strip:"
 z=`find ${D} -type f \( -perm -0100 -or -perm -0010 -or -perm -0001 -or -name "*.so" -or -name "*.so.*" \)`
+/bin/rm -f /tmp/prelink_list
 
 for x in $z
 do
@@ -14,11 +15,13 @@
    then
 	echo $x
 	strip "${x}"
+	echo $x >> /tmp/prelink_list
    fi
    if [ "${f/*SB shared object*/1}" == "1" ]
    then
 	echo $x
 	strip --strip-debug "${x}"
+	echo $x >> /tmp/prelink_list
    fi
 done
 
diff -ruN portage-2.0.44.orig/bin/savestats portage-2.0.44/bin/savestats
--- portage-2.0.44.orig/bin/savestats	1970-01-01 01:00:00.000000000 +0100
+++ portage-2.0.44/bin/savestats	2002-11-20 20:53:38.000000000 +0000
@@ -0,0 +1,54 @@
+#!/usr/bin/python2.2
+# Copyright 1999-2002 Gentoo Technologies, Inc.
+# Distributed under the terms of the GNU General Public License v2
+# Author: J Robert Ray <jrray@gentoo.org>
+# $Header: /home/cvsroot/gentoo-src/portage/bin/repoman,v 1.5 2002/07/27 19:13:59 drobbins Exp $
+
+# Record all the stat info in a directory tree into a file
+#
+# Used to save file properties from within a fakeroot shell,
+# so that they may be really applied to the files outside the
+# shell.
+#
+# savestats <output file> <path to crawl>
+
+from stat import *
+import os,sys,cPickle
+
+
+
+db={}
+
+def walk(basepath,rest):
+	fullpath=basepath+rest
+	list = os.listdir(fullpath)
+	for file in list:
+		restpath=rest+'/'+file
+		filename=fullpath+'/'+file
+		print restpath
+		mystat=os.lstat(filename)
+		db[restpath]=mystat
+		if (S_ISDIR(mystat[ST_MODE])):
+			walk(basepath,restpath)
+
+
+if (len(sys.argv) != 3):
+	print "Usage:"
+	print "  savestats <outputfile> <path to crawl>"
+	sys.exit(1)
+
+dbfile=sys.argv[1]
+path=sys.argv[2]
+
+# verify the path is really a path
+try:
+	mystat=os.lstat(path)
+	if (not S_ISDIR(mystat[ST_MODE])):
+		print "!!!",path,"is not a directory!"
+		sys.exit(1)
+except OSError:
+	print "!!!",path,"does not exist!"
+	sys.exit(1)
+
+walk(path,'')
+cPickle.dump(db, open(dbfile,'wb'))
diff -ruN portage-2.0.44.orig/pym/portage.py portage-2.0.44/pym/portage.py
--- portage-2.0.44.orig/pym/portage.py	2002-11-13 00:07:07.000000000 +0000
+++ portage-2.0.44/pym/portage.py	2002-11-20 20:53:38.000000000 +0000
@@ -7,7 +7,7 @@
 from stat import *
 from commands import *
 from select import *
-import string,os,types,sys,shlex,shutil,xpak,fcntl,signal,time,missingos,cPickle,atexit,grp,traceback
+import string,os,types,sys,shlex,shutil,xpak,fcntl,signal,time,missingos,cPickle,atexit,grp,traceback,pwd
 
 #Secpass will be set to 1 if the user is root or in the wheel group.
 uid=os.getuid()
@@ -23,6 +23,16 @@
 	print "Please fix this so that Portage can operate correctly (It's normally GID 10)"
 	pass
 
+#Discover the uid and gid of the portage user/group
+try:
+	portage_uid=pwd.getpwnam("portage")[2]
+	portage_gid=grp.getgrnam("portage")[2]
+except KeyError:
+	print "portage initialization: your system doesn't have a \"portage\" user or group."
+	print "Please fix this so that Portage can operate correctly"
+	print "exiting."
+	sys.exit(1)
+
 incrementals=["USE","FEATURES","ACCEPT_KEYWORDS","ACCEPT_LICENSE","CONFIG_PROTECT_MASK","CONFIG_PROTECT"]
 stickies=["KEYWORDS_ACCEPT","USE","CFLAGS","CXXFLAGS","MAKEOPTS","EXTRA_ECONF","EXTRA_EMAKE"]
 
@@ -59,6 +69,19 @@
 
 try:
 	import fchksum
+	def perform_prelink_checksum(filename):
+		if not(prelink_enabled == 1):		
+			return fchksum.fmd5t(filename)
+		else:
+			# Don't delete tmp file for speed
+			ret = os.system("/usr/sbin/prelink --verify " + filename + " > /tmp/portage/prelink.tmp 2> /dev/null")
+			if(ret != 0):
+				# Error probably cos we got "at least one of file's dependencies has changed"
+				# undo prelink and do a normal md5sum
+				ret = os.spawnlp(os.P_WAIT,"/usr/sbin/prelink","/usr/sbin/prelink","--undo",filename)
+				return fchksum.fmd5t(filename)
+			else:
+				return fchksum.fmd5t("/tmp/portage/prelink.tmp")
 	def perform_checksum(filename):
 		return fchksum.fmd5t(filename)
 except ImportError:
@@ -68,9 +91,21 @@
 		for ix in xrange(len(md5sum)):
 			hexform = hexform + "%02x" % ord(md5sum[ix])
 		return(string.lower(hexform))
-	
+	def perform_prelink_checksum(filename):
+		return perform_checksum(filename)
 	def perform_checksum(filename):
-		f = open(filename, 'rb')
+		if(prelink_enabled == 1):
+			# Don't delete tmp file for speed
+			ret = os.system("/usr/sbin/prelink --verify " + filename + " > /tmp/portage/prelink.tmp 2> /dev/null")
+			if(ret != 0):
+				# Error probably cos we got "at least one of file's dependencies has changed"
+				# undo prelink and do a normal md5sum
+				ret = os.system("/usr/sbin/prelink --undo " + filename)
+				f = open(filename, 'rb')
+			else:
+				f = open("/tmp/portage/prelink.tmp", 'rb')
+		else:
+			f = open(filename, 'rb')
 		blocksize=32768
 		data = f.read(blocksize)
 		size = 0L
@@ -79,6 +114,7 @@
 			sum.update(data)
 			size = size + len(data)
 			data = f.read(blocksize)
+		f.close()
 		return (md5_to_hex(sum.digest()),size)
 
 starttime=int(time.time())
@@ -830,7 +866,7 @@
 			mydict[x]=self[x]
 		return mydict
 	
-def spawn(mystring,debug=0,free=0):
+def spawn(mystring,debug=0,free=0,nodrop=1,fakeroot=0):
 	"""spawn a subprocess with optional sandbox protection, 
 	depending on whether sandbox is enabled.  The "free" argument,
 	when set to 1, will disable sandboxing.  This allows us to 
@@ -847,13 +883,26 @@
 		myargs=[]
 		if ("sandbox" in features) and (not free):
 			mycommand="/usr/lib/portage/bin/sandbox"
-			myargs=["["+settings["PF"]+"] sandbox",mystring]
+			if ("fakeroot" in features) and fakeroot:
+				myargs=["["+settings["PF"]+"] sandbox","/usr/bin/fakeroot",mystring]
+			else:
+				myargs=["["+settings["PF"]+"] sandbox",mystring]
+		elif ("fakeroot" in features) and fakeroot:
+			mycommand="/usr/bin/fakeroot"
+			if debug:
+				myargs=["fakeroot","/bin/bash","-x","-c",mystring]
+			else:
+				myargs=["fakeroot","/bin/bash","-c",mystring]
 		else:
 			mycommand="/bin/bash"
 			if debug:
 				myargs=["bash","-x","-c",mystring]
 			else:
 				myargs=["bash","-c",mystring]
+		if (not nodrop):
+			#drop root privileges, become the 'portage' user
+			os.setgid(portage_gid)
+			os.setuid(portage_uid)
 		os.execve(mycommand,myargs,settings.environ())
 		# If the execve fails, we need to report it, and exit
 		# *carefully*
@@ -1074,6 +1123,20 @@
 			print ">>> md5 ;-)",x
 	return 1
 
+# parse actionmap to spawn ebuild with the appropriate args
+def spawnebuild(mydo,actionmap,debug,alwaysdep=0):
+	if alwaysdep or not ("noauto" in features):
+		# process dependency first
+		if "dep" in actionmap[mydo].keys():
+			retval=spawnebuild(actionmap[mydo]["dep"],actionmap,debug,alwaysdep)
+			if retval: return retval
+	# spawn ebuild.sh
+	return spawn("/usr/sbin/ebuild.sh " + mydo,debug,
+				actionmap[mydo]["args"][0],
+				actionmap[mydo]["args"][1],
+				actionmap[mydo]["args"][2]
+	)
+
 # "checkdeps" support has been deprecated.  Relying on emerge to handle it.
 def doebuild(myebuild,mydo,myroot,debug=0,listonly=0):
 	global settings
@@ -1124,10 +1187,12 @@
 	try:
 		if not os.path.exists(settings["BUILDDIR"]) and mydo!="depend":
 			os.makedirs(settings["BUILDDIR"])
+			os.chown(settings["BUILDDIR"], portage_uid, portage_gid)
 		# Should be ok again to set $T, as sandbox do not depend on it
 		settings["T"]=settings["BUILDDIR"]+"/temp"
 		if not os.path.exists(settings["T"]) and mydo!="depend":
 			os.makedirs(settings["T"])
+			os.chown(settings["T"], portage_uid, portage_gid)
 	except OSError, e:
 		print "!!! File system problem. (ReadOnly?)"
 		print "!!!"+str(e)
@@ -1224,22 +1289,20 @@
 		return 1
 	
 	#initial dep checks complete; time to process main commands
-	
-	actionmap={	"unpack":"setup unpack", 
-				"compile":"setup unpack compile",
-				"install":"setup unpack compile install",
-				"rpm":"setup unpack compile install rpm"
-				}
+
+	actionmap={	  "setup": {                 "args":(1,1,0)},  # as root,    no sandbox, no fakeroot
+			 "unpack": {"dep":"setup",   "args":(0,0,0)},  # as portage, w/ sandbox, no fakeroot
+			"compile": {"dep":"unpack",  "args":(1,0,0)},  # as portage, no sandbox, no fakeroot
+			"install": {"dep":"compile", "args":(0,1,0)},  # as root, w/ sandbox, no fakeroot
+			    "rpm": {"dep":"install", "args":(0,0,1)},  # as portage, w/ sandbox, w/ fakeroot
+	}
 	if mydo in actionmap.keys():	
-		if "noauto" in features:
-			return spawn("/usr/sbin/ebuild.sh "+mydo,debug)
-		else:
-			return spawn("/usr/sbin/ebuild.sh "+actionmap[mydo],debug)
+		return spawnebuild(mydo,actionmap,debug)
 	elif mydo=="qmerge": 
 		#qmerge is specifically not supposed to do a runtime dep check
 		return merge(settings["CATEGORY"],settings["PF"],settings["D"],settings["BUILDDIR"]+"/build-info",myroot)
 	elif mydo=="merge":
-		retval=spawn("/usr/sbin/ebuild.sh setup unpack compile install")
+		retval=spawnebuild("install",actionmap,debug,1)
 		if retval: return retval
 		return merge(settings["CATEGORY"],settings["PF"],settings["D"],settings["BUILDDIR"]+"/build-info",myroot,myebuild=settings["EBUILD"])
 	elif mydo=="package":
@@ -1462,6 +1525,9 @@
 def perform_md5(x):
 	return perform_checksum(x)[0]
 
+def perform_prelink_md5(x):
+	return perform_prelink_checksum(x)[0]
+
 def merge(mycat,mypkg,pkgloc,infloc,myroot,myebuild=None):
 	mylink=dblink(mycat,mypkg,myroot)
 	if not mylink.exists():
@@ -3468,7 +3534,7 @@
 				if not os.path.isfile(obj):
 					print "--- !obj  ","obj", obj
 					continue
-				mymd5=perform_md5(obj)
+				mymd5=perform_prelink_md5(obj)
 				# string.lower is needed because db entries used to be in upper-case.  The
 				# string.lower allows for backwards compatibility.
 				if mymd5 != string.lower(pkgfiles[obj][2]):
@@ -3642,6 +3708,12 @@
 			cfgfiledict=grabdict(destroot+"/var/cache/edb/config")
 		else:
 			cfgfiledict={}
+		# load up the file stat database produced while inside fakeroot
+		try:
+			dbfile=settings["T"]+"/perms.db"
+			self.statdb=cPickle.load(open(dbfile,'rb'))
+		except:
+			self.statdb={}
 		# set umask to 0 for merging; back up umask, save old one in prevmask (since this is a global change)
 		mymtime=int(time.time())
 		prevmask=os.umask(0)
@@ -3666,7 +3738,12 @@
 		if len(secondhand):
 			# force merge of remaining symlinks (broken or circular; oh well)
 			self.mergeme(srcroot,destroot,outfile,None,secondhand,cfgfiledict,mymtime)
-			
+		
+		# Prelink files in the list
+		if (prelink_enabled == 1):
+			os.spawnlp(os.P_WAIT,"/usr/lib/portage/bin/postallprelink","/usr/lib/portage/bin/postallprelink","/tmp/prelink_list")
+		os.spawnlp(os.P_WAIT,"/bin/rm","/bin/rm","-f","/tmp/prelink_list")
+
 		#restore umask
 		os.umask(prevmask)
 		#if we opened it, close it	
@@ -3740,8 +3817,16 @@
 			mydest=os.path.normpath(destroot+offset+x)
 			# myrealdest is mydest without the $ROOT prefix (makes a difference if ROOT!="/")
 			myrealdest="/"+offset+x
-			# stat file once, test using S_* macros many times (faster that way)
-			mystat=os.lstat(mysrc)
+			try:
+				# update the file permissions to the one from the fakeroot db, if possible
+				mystat=self.statdb[myrealdest]
+				missingos.lchown(mysrc,mystat[4],mystat[5]);
+				if (not S_ISLNK(mysrc)):
+					os.chmod(mysrc,mystat[0]);
+			except:
+				# stat file once, test using S_* macros many times (faster that way)
+				# but only stat if there wasn't a stat from the statdb
+				mystat=os.lstat(mysrc)
 			mymode=mystat[ST_MODE]
 			# handy variables; mydest is the target object on the live filesystems;
 			# mysrc is the source object in the temporary install dir 
@@ -4250,6 +4335,12 @@
 
 #,"porttree":portagetree(root,virts),"bintree":binarytree(root,virts)}
 features=settings["FEATURES"].split()
+# make this a variable to speed it up, it is used alot
+if ("prelink" in features) and (os.system("/usr/sbin/prelink --version > /dev/null 2>&1") == 0):
+	prelink_enabled=1
+else:
+	prelink_enabled=0
+
 dbcachedir=settings["PORTAGE_CACHEDIR"]
 if not dbcachedir:
 	#the auxcache is the only /var/cache/edb/ entry that stays at / even when "root" changes.
diff -ruN portage-2.0.44.orig/src/sandbox-1.1/sandbox.c portage-2.0.44/src/sandbox-1.1/sandbox.c
--- portage-2.0.44.orig/src/sandbox-1.1/sandbox.c	2002-09-24 18:13:45.000000000 +0100
+++ portage-2.0.44/src/sandbox-1.1/sandbox.c	2002-11-20 20:53:38.000000000 +0000
@@ -546,11 +546,16 @@
       exit(1);
     }
 
-    /* Our r+ also will create the file if it doesn't exist */
-    preload_file=file_open("/etc/ld.so.preload", "r+", 1, 0644);
-    if (-1 == preload_file) {
-      preload_adaptable = 0;
-/*      exit(1);*/
+    if (getuid() == 0) {
+            /* Our r+ also will create the file if it doesn't exist */
+            preload_file=file_open("/etc/ld.so.preload", "r+", 1, 0644);
+            if (-1 == preload_file) {
+              preload_adaptable = 0;
+              /*      exit(1);*/
+            }
+    } else {
+            /* avoid permissions warnings if we're not root */
+            preload_adaptable = 0;
     }
 
 #ifdef USE_LD_SO_PRELOAD


[-- Attachment #3: Type: text/plain, Size: 37 bytes --]

--
gentoo-dev@gentoo.org mailing list