From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 30500 invoked by uid 1002); 20 Nov 2002 21:11:11 -0000 Mailing-List: contact gentoo-dev-help@gentoo.org; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@gentoo.org Received: (qmail 30480 invoked from network); 20 Nov 2002 21:11:11 -0000 From: Stefan Jones To: gentoo-core , gentoo-dev Content-Type: multipart/mixed; boundary="=-/60hVr88s+IDAMKKD6H0" Organization: Gentoo Linux Message-Id: <1037826628.18993.46.camel@localhost> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.2.0- Date: 20 Nov 2002 21:10:28 +0000 Subject: [gentoo-dev] Prelink, unpriv user compile and fakeroot X-Archives-Salt: ff3520c9-b459-4d12-bfe7-3521931becb2 X-Archives-Hash: 8492c3469209cac073a6985020f0c779 --=-/60hVr88s+IDAMKKD6H0 Content-Type: text/plain Content-Transfer-Encoding: 7bit Hi all, When I started using prelink and portage some apps started to segfault in the portage compile/install environment. This was traced to the LD_PRELOAD environmental variable. Prelinked apps don't like being LD_PRELOADed. Most of / all these problems are due to buggy / nonstandard compile methods and glibc bugs with preloading and prelink (one glibc bug has already been fixed) As a work around to the above problems and a permanent fix to sandbox problems I started working on jrray's fakeroot portage patch ( look at http://cvs.gentoo.org/~jrray ) What was needed was for compiles to be done by a normal unprivileged user ( like nobody, or the new portage user ) without ANY sandbox or other library being preloaded. This fixes most compile time problems with things like emacs. This is also supported by all package I know as it is the normal manual install method most people use. This safe as the user has no write permissions to most of the file system. Also the functions of sandbox are not needed during compile as during this stage nothing needs to be installed properly. Then the install stage can be done as root with libsandbox as normal as during this stage no tricky library manipulations are being done so LD_PRELOAD will not interfere. The attached patch only uses sandbox, but can be easily hacked to use fakeroot instead/as well. The patch is a radical hack of jrray's version. It can also be found at http://cvs.gentoo.org/~cretin/portage-prelink+fakeroot.patch Install: Install prelink if you want to prelink, you can use the patch without prelink to get the unpriv user compile feature. Then apply patch: This a patch against the portage-2.0.44.tar.bz2 tarball found in /usr/portage so add to the portage ebuild cd ${S} ; bzip2 -dc /path/to/portage-prelink+fakeroot.patch | patch -p1 || die Next add a "portage" group and a user called "portage" with home dir at /home/portage. Finally add prelink to FEATURES in make.conf if you want prelink. Now watch the breakages .... (only joking, it works fine for me ) Is software ever finished? -- Stefan Jones Gentoo Linux --=-/60hVr88s+IDAMKKD6H0 Content-Disposition: attachment; filename=portage-prelink+fakeroot.patch Content-Type: text/x-patch; name=portage-prelink+fakeroot.patch; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit diff -ruN portage-2.0.44.orig/bin/ebuild.sh portage-2.0.44/bin/ebuild.sh --- portage-2.0.44.orig/bin/ebuild.sh 2002-11-11 19:13:08.000000000 +0000 +++ portage-2.0.44/bin/ebuild.sh 2002-11-20 20:53:38.000000000 +0000 @@ -130,7 +130,7 @@ export PATH="/usr/bin/ccache:${PATH}" if [ -z "${CCACHE_DIR}" ] then - CCACHE_DIR=/root/.ccache + export CCACHE_DIR=/home/portage/.ccache fi addread ${CCACHE_DIR} addwrite ${CCACHE_DIR} @@ -569,6 +569,8 @@ src_install #|| abort_install "fail" prepall + echo ">>> Recording file permissions of ${D}" + /usr/lib/portage/bin/savestats ${T}/perms.db ${BUILDDIR}/image cd ${D} echo ">>> Completed installing into ${D}" echo @@ -692,6 +694,8 @@ # default target [ -n "$T" ] && echo $1 >> ${T}/eclass-debug.log + # let the portage user own/write to this file + [ -n "$T" ] && chown portage.portage ${T}/eclass-debug.log shift done diff -ruN portage-2.0.44.orig/bin/postallprelink portage-2.0.44/bin/postallprelink --- portage-2.0.44.orig/bin/postallprelink 1970-01-01 01:00:00.000000000 +0100 +++ portage-2.0.44/bin/postallprelink 2002-11-20 20:53:38.000000000 +0000 @@ -0,0 +1,10 @@ +#!/bin/bash + +FILES=`cat $1` + +echo "prelinking:" +for x in $FILES ; do + x=${x/*image/} + echo $x + prelink -mR $x +done diff -ruN portage-2.0.44.orig/bin/prepallstrip portage-2.0.44/bin/prepallstrip --- portage-2.0.44.orig/bin/prepallstrip 2002-11-09 09:00:58.000000000 +0000 +++ portage-2.0.44/bin/prepallstrip 2002-11-20 20:53:38.000000000 +0000 @@ -6,6 +6,7 @@ fi echo "strip:" z=`find ${D} -type f \( -perm -0100 -or -perm -0010 -or -perm -0001 -or -name "*.so" -or -name "*.so.*" \)` +/bin/rm -f /tmp/prelink_list for x in $z do @@ -14,11 +15,13 @@ then echo $x strip "${x}" + echo $x >> /tmp/prelink_list fi if [ "${f/*SB shared object*/1}" == "1" ] then echo $x strip --strip-debug "${x}" + echo $x >> /tmp/prelink_list fi done diff -ruN portage-2.0.44.orig/bin/savestats portage-2.0.44/bin/savestats --- portage-2.0.44.orig/bin/savestats 1970-01-01 01:00:00.000000000 +0100 +++ portage-2.0.44/bin/savestats 2002-11-20 20:53:38.000000000 +0000 @@ -0,0 +1,54 @@ +#!/usr/bin/python2.2 +# Copyright 1999-2002 Gentoo Technologies, Inc. +# Distributed under the terms of the GNU General Public License v2 +# Author: J Robert Ray +# $Header: /home/cvsroot/gentoo-src/portage/bin/repoman,v 1.5 2002/07/27 19:13:59 drobbins Exp $ + +# Record all the stat info in a directory tree into a file +# +# Used to save file properties from within a fakeroot shell, +# so that they may be really applied to the files outside the +# shell. +# +# savestats + +from stat import * +import os,sys,cPickle + + + +db={} + +def walk(basepath,rest): + fullpath=basepath+rest + list = os.listdir(fullpath) + for file in list: + restpath=rest+'/'+file + filename=fullpath+'/'+file + print restpath + mystat=os.lstat(filename) + db[restpath]=mystat + if (S_ISDIR(mystat[ST_MODE])): + walk(basepath,restpath) + + +if (len(sys.argv) != 3): + print "Usage:" + print " savestats " + sys.exit(1) + +dbfile=sys.argv[1] +path=sys.argv[2] + +# verify the path is really a path +try: + mystat=os.lstat(path) + if (not S_ISDIR(mystat[ST_MODE])): + print "!!!",path,"is not a directory!" + sys.exit(1) +except OSError: + print "!!!",path,"does not exist!" + sys.exit(1) + +walk(path,'') +cPickle.dump(db, open(dbfile,'wb')) diff -ruN portage-2.0.44.orig/pym/portage.py portage-2.0.44/pym/portage.py --- portage-2.0.44.orig/pym/portage.py 2002-11-13 00:07:07.000000000 +0000 +++ portage-2.0.44/pym/portage.py 2002-11-20 20:53:38.000000000 +0000 @@ -7,7 +7,7 @@ from stat import * from commands import * from select import * -import string,os,types,sys,shlex,shutil,xpak,fcntl,signal,time,missingos,cPickle,atexit,grp,traceback +import string,os,types,sys,shlex,shutil,xpak,fcntl,signal,time,missingos,cPickle,atexit,grp,traceback,pwd #Secpass will be set to 1 if the user is root or in the wheel group. uid=os.getuid() @@ -23,6 +23,16 @@ print "Please fix this so that Portage can operate correctly (It's normally GID 10)" pass +#Discover the uid and gid of the portage user/group +try: + portage_uid=pwd.getpwnam("portage")[2] + portage_gid=grp.getgrnam("portage")[2] +except KeyError: + print "portage initialization: your system doesn't have a \"portage\" user or group." + print "Please fix this so that Portage can operate correctly" + print "exiting." + sys.exit(1) + incrementals=["USE","FEATURES","ACCEPT_KEYWORDS","ACCEPT_LICENSE","CONFIG_PROTECT_MASK","CONFIG_PROTECT"] stickies=["KEYWORDS_ACCEPT","USE","CFLAGS","CXXFLAGS","MAKEOPTS","EXTRA_ECONF","EXTRA_EMAKE"] @@ -59,6 +69,19 @@ try: import fchksum + def perform_prelink_checksum(filename): + if not(prelink_enabled == 1): + return fchksum.fmd5t(filename) + else: + # Don't delete tmp file for speed + ret = os.system("/usr/sbin/prelink --verify " + filename + " > /tmp/portage/prelink.tmp 2> /dev/null") + if(ret != 0): + # Error probably cos we got "at least one of file's dependencies has changed" + # undo prelink and do a normal md5sum + ret = os.spawnlp(os.P_WAIT,"/usr/sbin/prelink","/usr/sbin/prelink","--undo",filename) + return fchksum.fmd5t(filename) + else: + return fchksum.fmd5t("/tmp/portage/prelink.tmp") def perform_checksum(filename): return fchksum.fmd5t(filename) except ImportError: @@ -68,9 +91,21 @@ for ix in xrange(len(md5sum)): hexform = hexform + "%02x" % ord(md5sum[ix]) return(string.lower(hexform)) - + def perform_prelink_checksum(filename): + return perform_checksum(filename) def perform_checksum(filename): - f = open(filename, 'rb') + if(prelink_enabled == 1): + # Don't delete tmp file for speed + ret = os.system("/usr/sbin/prelink --verify " + filename + " > /tmp/portage/prelink.tmp 2> /dev/null") + if(ret != 0): + # Error probably cos we got "at least one of file's dependencies has changed" + # undo prelink and do a normal md5sum + ret = os.system("/usr/sbin/prelink --undo " + filename) + f = open(filename, 'rb') + else: + f = open("/tmp/portage/prelink.tmp", 'rb') + else: + f = open(filename, 'rb') blocksize=32768 data = f.read(blocksize) size = 0L @@ -79,6 +114,7 @@ sum.update(data) size = size + len(data) data = f.read(blocksize) + f.close() return (md5_to_hex(sum.digest()),size) starttime=int(time.time()) @@ -830,7 +866,7 @@ mydict[x]=self[x] return mydict -def spawn(mystring,debug=0,free=0): +def spawn(mystring,debug=0,free=0,nodrop=1,fakeroot=0): """spawn a subprocess with optional sandbox protection, depending on whether sandbox is enabled. The "free" argument, when set to 1, will disable sandboxing. This allows us to @@ -847,13 +883,26 @@ myargs=[] if ("sandbox" in features) and (not free): mycommand="/usr/lib/portage/bin/sandbox" - myargs=["["+settings["PF"]+"] sandbox",mystring] + if ("fakeroot" in features) and fakeroot: + myargs=["["+settings["PF"]+"] sandbox","/usr/bin/fakeroot",mystring] + else: + myargs=["["+settings["PF"]+"] sandbox",mystring] + elif ("fakeroot" in features) and fakeroot: + mycommand="/usr/bin/fakeroot" + if debug: + myargs=["fakeroot","/bin/bash","-x","-c",mystring] + else: + myargs=["fakeroot","/bin/bash","-c",mystring] else: mycommand="/bin/bash" if debug: myargs=["bash","-x","-c",mystring] else: myargs=["bash","-c",mystring] + if (not nodrop): + #drop root privileges, become the 'portage' user + os.setgid(portage_gid) + os.setuid(portage_uid) os.execve(mycommand,myargs,settings.environ()) # If the execve fails, we need to report it, and exit # *carefully* @@ -1074,6 +1123,20 @@ print ">>> md5 ;-)",x return 1 +# parse actionmap to spawn ebuild with the appropriate args +def spawnebuild(mydo,actionmap,debug,alwaysdep=0): + if alwaysdep or not ("noauto" in features): + # process dependency first + if "dep" in actionmap[mydo].keys(): + retval=spawnebuild(actionmap[mydo]["dep"],actionmap,debug,alwaysdep) + if retval: return retval + # spawn ebuild.sh + return spawn("/usr/sbin/ebuild.sh " + mydo,debug, + actionmap[mydo]["args"][0], + actionmap[mydo]["args"][1], + actionmap[mydo]["args"][2] + ) + # "checkdeps" support has been deprecated. Relying on emerge to handle it. def doebuild(myebuild,mydo,myroot,debug=0,listonly=0): global settings @@ -1124,10 +1187,12 @@ try: if not os.path.exists(settings["BUILDDIR"]) and mydo!="depend": os.makedirs(settings["BUILDDIR"]) + os.chown(settings["BUILDDIR"], portage_uid, portage_gid) # Should be ok again to set $T, as sandbox do not depend on it settings["T"]=settings["BUILDDIR"]+"/temp" if not os.path.exists(settings["T"]) and mydo!="depend": os.makedirs(settings["T"]) + os.chown(settings["T"], portage_uid, portage_gid) except OSError, e: print "!!! File system problem. (ReadOnly?)" print "!!!"+str(e) @@ -1224,22 +1289,20 @@ return 1 #initial dep checks complete; time to process main commands - - actionmap={ "unpack":"setup unpack", - "compile":"setup unpack compile", - "install":"setup unpack compile install", - "rpm":"setup unpack compile install rpm" - } + + actionmap={ "setup": { "args":(1,1,0)}, # as root, no sandbox, no fakeroot + "unpack": {"dep":"setup", "args":(0,0,0)}, # as portage, w/ sandbox, no fakeroot + "compile": {"dep":"unpack", "args":(1,0,0)}, # as portage, no sandbox, no fakeroot + "install": {"dep":"compile", "args":(0,1,0)}, # as root, w/ sandbox, no fakeroot + "rpm": {"dep":"install", "args":(0,0,1)}, # as portage, w/ sandbox, w/ fakeroot + } if mydo in actionmap.keys(): - if "noauto" in features: - return spawn("/usr/sbin/ebuild.sh "+mydo,debug) - else: - return spawn("/usr/sbin/ebuild.sh "+actionmap[mydo],debug) + return spawnebuild(mydo,actionmap,debug) elif mydo=="qmerge": #qmerge is specifically not supposed to do a runtime dep check return merge(settings["CATEGORY"],settings["PF"],settings["D"],settings["BUILDDIR"]+"/build-info",myroot) elif mydo=="merge": - retval=spawn("/usr/sbin/ebuild.sh setup unpack compile install") + retval=spawnebuild("install",actionmap,debug,1) if retval: return retval return merge(settings["CATEGORY"],settings["PF"],settings["D"],settings["BUILDDIR"]+"/build-info",myroot,myebuild=settings["EBUILD"]) elif mydo=="package": @@ -1462,6 +1525,9 @@ def perform_md5(x): return perform_checksum(x)[0] +def perform_prelink_md5(x): + return perform_prelink_checksum(x)[0] + def merge(mycat,mypkg,pkgloc,infloc,myroot,myebuild=None): mylink=dblink(mycat,mypkg,myroot) if not mylink.exists(): @@ -3468,7 +3534,7 @@ if not os.path.isfile(obj): print "--- !obj ","obj", obj continue - mymd5=perform_md5(obj) + mymd5=perform_prelink_md5(obj) # string.lower is needed because db entries used to be in upper-case. The # string.lower allows for backwards compatibility. if mymd5 != string.lower(pkgfiles[obj][2]): @@ -3642,6 +3708,12 @@ cfgfiledict=grabdict(destroot+"/var/cache/edb/config") else: cfgfiledict={} + # load up the file stat database produced while inside fakeroot + try: + dbfile=settings["T"]+"/perms.db" + self.statdb=cPickle.load(open(dbfile,'rb')) + except: + self.statdb={} # set umask to 0 for merging; back up umask, save old one in prevmask (since this is a global change) mymtime=int(time.time()) prevmask=os.umask(0) @@ -3666,7 +3738,12 @@ if len(secondhand): # force merge of remaining symlinks (broken or circular; oh well) self.mergeme(srcroot,destroot,outfile,None,secondhand,cfgfiledict,mymtime) - + + # Prelink files in the list + if (prelink_enabled == 1): + os.spawnlp(os.P_WAIT,"/usr/lib/portage/bin/postallprelink","/usr/lib/portage/bin/postallprelink","/tmp/prelink_list") + os.spawnlp(os.P_WAIT,"/bin/rm","/bin/rm","-f","/tmp/prelink_list") + #restore umask os.umask(prevmask) #if we opened it, close it @@ -3740,8 +3817,16 @@ mydest=os.path.normpath(destroot+offset+x) # myrealdest is mydest without the $ROOT prefix (makes a difference if ROOT!="/") myrealdest="/"+offset+x - # stat file once, test using S_* macros many times (faster that way) - mystat=os.lstat(mysrc) + try: + # update the file permissions to the one from the fakeroot db, if possible + mystat=self.statdb[myrealdest] + missingos.lchown(mysrc,mystat[4],mystat[5]); + if (not S_ISLNK(mysrc)): + os.chmod(mysrc,mystat[0]); + except: + # stat file once, test using S_* macros many times (faster that way) + # but only stat if there wasn't a stat from the statdb + mystat=os.lstat(mysrc) mymode=mystat[ST_MODE] # handy variables; mydest is the target object on the live filesystems; # mysrc is the source object in the temporary install dir @@ -4250,6 +4335,12 @@ #,"porttree":portagetree(root,virts),"bintree":binarytree(root,virts)} features=settings["FEATURES"].split() +# make this a variable to speed it up, it is used alot +if ("prelink" in features) and (os.system("/usr/sbin/prelink --version > /dev/null 2>&1") == 0): + prelink_enabled=1 +else: + prelink_enabled=0 + dbcachedir=settings["PORTAGE_CACHEDIR"] if not dbcachedir: #the auxcache is the only /var/cache/edb/ entry that stays at / even when "root" changes. diff -ruN portage-2.0.44.orig/src/sandbox-1.1/sandbox.c portage-2.0.44/src/sandbox-1.1/sandbox.c --- portage-2.0.44.orig/src/sandbox-1.1/sandbox.c 2002-09-24 18:13:45.000000000 +0100 +++ portage-2.0.44/src/sandbox-1.1/sandbox.c 2002-11-20 20:53:38.000000000 +0000 @@ -546,11 +546,16 @@ exit(1); } - /* Our r+ also will create the file if it doesn't exist */ - preload_file=file_open("/etc/ld.so.preload", "r+", 1, 0644); - if (-1 == preload_file) { - preload_adaptable = 0; -/* exit(1);*/ + if (getuid() == 0) { + /* Our r+ also will create the file if it doesn't exist */ + preload_file=file_open("/etc/ld.so.preload", "r+", 1, 0644); + if (-1 == preload_file) { + preload_adaptable = 0; + /* exit(1);*/ + } + } else { + /* avoid permissions warnings if we're not root */ + preload_adaptable = 0; } #ifdef USE_LD_SO_PRELOAD --=-/60hVr88s+IDAMKKD6H0 Content-Type: text/plain; charset=us-ascii -- gentoo-dev@gentoo.org mailing list --=-/60hVr88s+IDAMKKD6H0--