From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 10598 invoked from network); 7 Oct 2004 21:57:32 +0000 Received: from smtp.gentoo.org (156.56.111.197) by lists.gentoo.org with AES256-SHA encrypted SMTP; 7 Oct 2004 21:57:32 +0000 Received: from lists.gentoo.org ([156.56.111.196] helo=parrot.gentoo.org) by smtp.gentoo.org with esmtp (Exim 4.41) id 1CFgGR-0004Ev-P4 for arch-gentoo-portage-dev@lists.gentoo.org; Thu, 07 Oct 2004 21:57:32 +0000 Received: (qmail 10460 invoked by uid 89); 7 Oct 2004 21:57:24 +0000 Mailing-List: contact gentoo-portage-dev-help@gentoo.org; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail Reply-To: gentoo-portage-dev@lists.gentoo.org X-BeenThere: gentoo-portage-dev@gentoo.org Received: (qmail 14090 invoked from network); 7 Oct 2004 21:57:23 +0000 From: Dan Armak Reply-To: danarmak@gentoo.org To: gentoo-portage-dev@lists.gentoo.org Date: Thu, 7 Oct 2004 23:57:44 +0200 User-Agent: KMail/1.7 Organization: Gentoo Technologies, Inc. Cc: kde@gentoo.org, Simone Gotti , stuart@gentoo.org MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart2063065.EjAv9bkkYm"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit Message-Id: <200410072357.52578.danarmak@gentoo.org> Subject: [gentoo-portage-dev] Port of Stuart's confcache for .51_rc7 X-Archives-Salt: 2f08276f-07d4-4cbd-8120-46a4fc6f9132 X-Archives-Hash: 865b000ec7346d592a5ef546a8846470 --nextPart2063065.EjAv9bkkYm Content-Type: multipart/mixed; boundary="Boundary-01=_YvbZBg/TEVADbAT" Content-Transfer-Encoding: 7bit Content-Disposition: inline --Boundary-01=_YvbZBg/TEVADbAT Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Hi all, I ported Stuart's confcache patch from portage .50 to .51_rc7 (hope cvs hea= d=20 isn't too different...). I also made a few changes and fixes - see attached= =20 changelog if you used the orig patch. What this does: it keeps a central cache for autoconf-generated configure=20 scripts in /var/tmp. When econf is called, it checks the cache's validity=20 against a list of files & md5sums. If any files have changed, the entire=20 cache is cleared. During configure's run, the sandbox logs all files accessed. There is a sma= ll=20 patch to libsandbox.c, unchanged from Stuart's version, to allow this. Then= ,=20 the file/md5sum list is updated. The sandbox log then has to be manually=20 fixed up to remove these entries. Ebuild-side: FEATURES+=3Dconfcache enables this. Either use econf, or call= =20 confcache_start and confcache_stop before and after your configure run.=20 confcache_start adds the --with-cache=3Dfoo parameter to $ECONF_EXTRA; pass= =20 that to configure. Embedded people: I've been asked to keep a separate cache for every=20 combination of $CHOST, $CBUILD, $THOST. Please confirm that's how you want= =20 it. Myself, I need it to speed up the upcoming split kde ebuilds (eg separate=20 ebuilds for konqueror etc rather than one big kdebase ebuild). Stuart, I hope you're ok with my changes. Basically they just consist of=20 updating the patch for the new version's ebuild.sh and fixing a couple of=20 bugs. Comments welcome (my first patch to portage, hope it works :-). Credit goes= to=20 Stuart, really. =2D-=20 Dan Armak Gentoo Linux developer (KDE) Matan, Israel Public GPG key: http://dev.gentoo.org/~danarmak/danarmak-gpg-public.key =46ingerprint: DD70 DBF9 E3D4 6CB9 2FDD 0069 508D 9143 8D5F 8951 --Boundary-01=_YvbZBg/TEVADbAT Content-Type: text/x-diff; charset="us-ascii"; name="confcache-portage-2.0.51_rc7.patch" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="confcache-portage-2.0.51_rc7.patch" =2D-- portage-2.0.51_rc7/bin/ebuild.sh 2004-09-21 04:08:05.000000000 +0300 +++ portage-2.0.51_rc7.new/bin/ebuild.sh 2004-10-07 21:34:15.000000000 +0200 @@ -65,6 +65,15 @@ export SANDBOX_ON=3D"0" =20 # sandbox support functions; defined prior to profile.bashrc srcing, since= the profile might need to add a default exception (/usr/lib64/conftest fex= , bug #60147) +addlogread() +{ + if [ "$1" =3D "1" ]; then + export SANDBOX_LOGREAD=3D"1" + else + unset SANDBOX_LOGREAD + fi +} + addread() { export SANDBOX_READ=3D"$SANDBOX_READ:$1" @@ -389,7 +398,256 @@ done } =20 +CONFCACHE_MD5SUM=3D"$PORTAGE_TMPDIR/configure-$CBUILD-$CHOST-$THOST.md5sum= s" +CONFCACHE_SETTINGS=3D"$PORTAGE_TMPDIR/configure-$CBUILD-$CHOST-$THOST.cach= e" + +# $1 - file to write the configure cache out to + +confcache_prepare () { +=09 + echo ">>> Preparing configure cache" + + # step 1 - make sure that none of the files in our list of deps + # have changed since we last built something + + addwrite "$CONFCACHE_MD5SUM" + confcache_checksums + + case "$?" in + 2) + return + ;; + 1) + echo ">>> Configure deps have changed; using an empty cache" + rm "$CONFCACHE_MD5SUM" + return + ;; + esac + + # step 2 - now we copy the existing cache into the temporary file + + cat $CONFCACHE_SETTINGS > $1 +} + +confcache_checksums () { + + # if any of them have changed, we throw the whole cache away + # a future version, written in python, will be able to do + # per config-line deps + + if [ ! -f "$CONFCACHE_MD5SUM" ]; then + echo ">>> You have no md5sum cache file; assuming this is your first tim= e" + return 2 + fi + + while read x ; do + sum=3D"`echo $x | cut -d ' ' -f 1`" + file=3D"`echo $x | cut -d ' ' -f 2-`" + + new_sum=3D"`md5sum $file | awk '{ print $1 }'`" + if [ "$new_sum" !=3D "$sum" ]; then + return 1 + fi + done < $CONFCACHE_MD5SUM + + return 0 +} + +# $1 - file containing the updated configure cache +# $SANDBOX_LOG - file containing the sandbox log + +confcache_update () { + + # special cases + + if [ ! -f "$1" ]; then + echo "confcache_update(): unable to find temporary cache $1" + return + fi + + echo ">>> Updating global configure cache from $1" + + addwrite "$CONFCACHE_SETTINGS" + addwrite "$CONFCACHE_MD5SUM" + + # step 1 - replace our global configure cache + + cp -f "$1" "$CONFCACHE_SETTINGS" + + # step 2 - make a list of files from the sandbox log + + if [ ! -f "$SANDBOX_LOG" ]; then + # nothing we can do - let's bail + echo "confcache_update: sandbox log not found" + return + fi + + echo ">>> Reading list of files used by configure" + if [ -n "$CCACHE_DIR" ]; then + local REMOVE_CCACHE_DIR=3D"$CCACHE_DIR" + else + local REMOVE_CCACHE_DIR=3D"/root/.ccache" + fi + files=3D"`grep open_rd $SANDBOX_LOG | sed -e 's/^open_rd: \+//' | grep -v= /tmp | grep -v /var/tmp | grep -v $REMOVE_CCACHE_DIR | grep -v /dev | sort= | uniq`" + + # step 3 - add each file to the global md5 cache + # + # yes, this is a bit slow, but relying on egrep to search the file + # is risky. Sooner or later, some spanner package will rely on + # a file that contains spaces. + + # special case - do we *have* a CONFCACHE_MD5SUM file atm? + + if [ ! -f "$CONFCACHE_MD5SUM" ]; then + + echo ">>> No md5sum cache found; populating for first time" + + # create the file + touch $CONFCACHE_MD5SUM + + # populate it + + OLD_IFS=3D"$IFS" + NEW_IFS=3D"^M" + # IFS=3D"$NEW_IFS" + + for x in $files ; do + IFS=3D"$OLD_IFS" + + if [ ! -f "$x" ] ; then + continue + fi + + newsum=3D"`md5sum \"$x\" | awk '{ print $1 }'`" + echo "$newsum $x" >> $CONFCACHE_MD5SUM + + # IFS=3D"$NEW_IFS" + done + IFS=3D"$OLD_IFS" + fi + + # if we don't + + OLD_IFS=3D"$IFS" + NEW_IFS=3D" +" + + IFS=3D"$NEW_IFS" + + echo ">>> Updating md5sum cache" + + for x in $files ; do + if [ ! -f "$x" ]; then + continue + fi + + infile=3D0 + while read y < $CONFCACHE_MD5SUM ; do + IFS=3D"$OLD_IFS" + =09 + sum=3D"`echo $x | cut -d ' ' -f 1`" + file=3D"`echo $x | cut -d ' ' -f 2-`" + + if [ "$file" =3D "$x" ]; then + infile=3D1 + break + fi + IFS=3D"$NEW_IFS" + done + + IFS=3D"$OLD_IFS" + + if [ "$infile" =3D "0" ]; then + newsum=3D"`md5sum \"$x\" | awk '{ print $1 }'`" + echo "$newsum $x" >> $CONFCACHE_MD5SUM + fi + + IFS=3D"$NEW_IFS" + done + + IFS=3D"$OLD_IFS" +} + +confcache_start() { + # global configure cache - stuart@gentoo.org + # + # if FEATURES=3D"sandbox confcache" are set, we maintain a global + # cache of results from previous configure statements + # + # this global cache should benefit all machines, but especially + # multi-processor boxes + + if [ "${FEATURES//*confcache*/true}" =3D "true" -a "${FEATURES//*sandbox*= /true}" =3D "true" -a "${RESTRICT//*noconfcache*/true}" !=3D "true" ]; then + CONF_CACHE=3D"`/bin/tempfile`" + echo ">>> Temporary configure cache file is $CONF_CACHE" + addwrite "$CONF_CACHE" + confcache_prepare "$CONF_CACHE" + EXTRA_ECONF=3D"--cache-file=3D$CONF_CACHE $EXTRA_ECONF" + addlogread 1 + + # mark these variables read-only, so that they can't be + # changed to, say, /etc/passwd + + # typeset -r CONF_CACHE + # typeset -r READLOG + else + echo "!!! Not using global configure cache" + fi +} + +confcache_stop() { + if [ "${FEATURES//*confcache*/true}" =3D "true" -a "${FEATURES//*sandbox*= /true}" =3D "true" -a "${RESTRICT//*noconfcache*/true}" !=3D "true" ]; then + addlogread 0 + confcache_update "$CONF_CACHE" + [ -f "$CONF_CACHE" ] && rm -f "$CONF_CACHE" + + # now we need to clean up the log file, otherwise portage + # will have a fit ;-) + # + # here, we remove all the entries that the SANDBOX_LOGREAD + # feature will have added + # + # any entries that we don't remove are sandbox violations + # by definitions + echo ">>> Fixing sandbox log" + if [ -f "$SANDBOX_LOG" ]; then + SANDBOX_DIRS=3D"`echo $SANDBOX_WRITE | tr ':' ' '`" + + sed -i -e "s/^open_rd:.*//" \ + -e "s/^execve:.*//" \ + -e "s/^opendir:.*//" $SANDBOX_LOG + for x in $SANDBOX_DIRS ; do + sed -i -e "s/^open_wr: ${x//\//\\/}.*//" \ + -e "s/^unlink: ${x//\//\\/}.*//" \ + -e "s/^rename: ${x//\//\\/}.*//" \ + -e "s/^chmod: ${x//\//\\/}.*//" \ + -e "s/^chown: ${x//\//\\/}.*//" \ + -e "s/^symlink: ${x//\//\\/}.*//" \ + -e "s/^rmdir: ${x//\//\\/}.*//" \ + -e "s/^mkdir: ${x//\//\\/}.*//" $SANDBOX_LOG + done + + # now we remove the empty lines + + TMPFILE=3D"`tempfile`" + egrep -v '^$' $SANDBOX_LOG > $TMPFILE + cat $TMPFILE > $SANDBOX_LOG + rm $TMPFILE + + # if the file is empty, we can now remove the file + # + # this test may break on arches that do not use GNU ls +=09 + if [ "`ls -l $SANDBOX_LOG | awk '{ print $5 }'`" =3D "0" ]; then + rm -f $SANDBOX_LOG + fi + fi + fi +} + econf() { + confcache_start +=09 if [ -x ./configure ]; then if hasq autoconfig $FEATURES && ! hasq autoconfig $RESTRICT; then if [ -e /usr/share/gnuconfig/ -a -x /bin/basename ]; then @@ -440,8 +698,12 @@ ${EXTRA_ECONF} \ "$@" || die "econf failed" else + [ -n "$CONF_CACHE" ] && rm -f "$CONF_CACHE" + [ -n "$READLOG" ] && rm -f "$READLOG"=20 die "no configure script found" fi + + confcache_stop } =20 einstall() { =2D-- portage-2.0.51_rc7/src/sandbox-1.1/libsandbox.c 2004-08-02 21:49:31.0= 00000000 +0300 +++ portage-2.0.51_rc7.new/src/sandbox-1.1/libsandbox.c 2004-10-07 15:27:53= =2E000000000 +0200 @@ -1193,14 +1193,14 @@ debug_log_env =3D getenv("SANDBOX_DEBUG"); debug_log_path =3D getenv("SANDBOX_DEBUG_LOG"); =20 =2D if (((NULL =3D=3D log_path) || =2D (0 !=3D strncmp(absolute_path, log_path, strlen(log_path)))) && + if (((NULL !=3D log_path) || + (0 =3D=3D strncmp(absolute_path, log_path, strlen(log_path)))) && ((NULL =3D=3D debug_log_env) || (NULL =3D=3D debug_log_path) || (0 !=3D strncmp(absolute_path, debug_log_path, strlen(debug_log_path))= )) =2D && (0 =3D=3D check_access(sbcontext, func, absolute_path)) ) { =2D if (1 =3D=3D sbcontext->show_access_violation) { + + if ((0 =3D=3D check_access(sbcontext, func, absolute_path)) && (1 =3D=3D= sbcontext->show_access_violation)) { fprintf(stderr, "\e[31;01mACCESS DENIED\033[0m %s:%*s%s\n", func, (int) (10 - strlen(func)), "", absolute_path); @@ -1226,9 +1226,33 @@ } } } + + result =3D 0; } + else if (getenv("SANDBOX_LOGREAD") !=3D NULL) + { + /* create the log message */ + + sprintf ( + buffer,=20 + "%s:%*s%s\n",=20 + func,=20 + (int) (10 - strlen(func)),=20 + "", + absolute_path + ); + + /* fprintf(stderr, buffer); */ + + /* log the event to the log file anyway */ + log_file =3D true_open(log_path, O_APPEND | O_WRONLY | O_CREAT, S_IRUSR= | S_IWUSR | S_IRGRP | S_IROTH); + if (log_file >=3D 0) { + write(log_file, buffer, strlen(buffer)); + close(log_file); + } =20 =2D result =3D 0; + result =3D 1; + } } else if (NULL !=3D debug_log_env) { if (NULL !=3D debug_log_path) { if (0 !=3D strncmp(absolute_path, debug_log_path, strlen(debug_log_path= ))) { --Boundary-01=_YvbZBg/TEVADbAT Content-Type: text/plain; charset="us-ascii"; name="confcache.changelog" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="confcache.changelog" 20041007, danarmak: * ported stuart's orig patch to portage .51_rc7 * change: move the confcache code in econf into new functions confcache_st= art(), confcache_stop(), which are called at the beginning and end of econd(). this way ebuilds that don't use econf (eg kde.eclass) can still use confc= ache explicitly. * fix: confcache_stop: also remove allowed chown()s from sandbox log * fix: confcache_stop: fix sed invocations * optimize: confcache_stop: convert many sed invocations into one invocati= on with many scripts (can be further reduced to egrep()s, but i'm lazy and this seems fast eno= ugh) * change: move cache to PORTAGE_TMPDIR. /etc/portage is obviously the wron= g place, and carpaski advises not using /var/cache/edb like ferringb did as long as the impleme= ntation is in bash and runs under the sandbox. * change: store separate cache per combinations of $CHOST, $CBUILD, $TBUIL= D, as in ferringb's version; the embedded people are said to want this=09 --Boundary-01=_YvbZBg/TEVADbAT-- --nextPart2063065.EjAv9bkkYm Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.6 (GNU/Linux) iD8DBQBBZbvgUI2RQ41fiVERAq/bAJ45Hd7Rusi8QrvZfxtoyP1eE9v0owCeN2Ix nKdpW02IAq0r8mmvLkNgNlo= =Ayox -----END PGP SIGNATURE----- --nextPart2063065.EjAv9bkkYm--