[gentoo-dev] Script to clean old files from /usr/portage/distfiles

[gentoo-dev] Script to clean old files from /usr/portage/distfiles

[José Fonseca]

[-- Attachment #1: Type: text/plain, Size: 707 bytes --]

I never do `rm -f /usr/portage/distfiles'. I find very useful having the
sources of the installed packages ihandy as it avoids downloading
everything over andagain when updating to new releases of the same
version, or when installing Gentoo on another system. But after some
time that directory gets too fat and is too boring to eliminate the
deprecated packages by hand so I've made a script to automate this. You
can see it attached.

I've started with a shell script, then python, then I've read qpkg.sh,
ebuild.sh, portage.py and many others in the search of the easiest way
to do this. In the end everything boiled down to 2 lines - the rest is
just sintatic sugar!

I hope you enjoy it.

José Fonseca

[-- Attachment #2: distfiles-clean --]
[-- Type: text/plain, Size: 575 bytes --]

#!/bin/sh
#
# distfiles-clean
#
# Cleans unused files from Portage distfiles directory.
#
# José Fonseca <j_r_fonseca@yahoo.co.uk>

PORTDIR=/usr/portage
DISTDIR=$PORTDIR/distfiles
CACHEDIR=$PORTDIR/metadata/cache
PKGDBDIR=/var/db/pkg

TMPFILE=`mktemp /tmp/$0.XXXXXX`

find $PKGDBDIR -name '*.ebuild' | sed -n -e "s:^$PKGDBDIR/\([^/]*\)/\([^/]*\)/\([^/]*\)\.ebuild$:$CACHEDIR/\1/\3:p" | xargs -l sed -e '4!d;s/[[:space:]]\+/\n/g' | sed -n -e 's:^.*/::p' | sort -u > $TMPFILE

ls -1 $DISTDIR | sort -m - $TMPFILE | uniq -u | sed -e "s:^:$DISTDIR/:" | xargs rm -f

rm $TMPFILE


[-- Attachment #3: Type: text/plain, Size: 37 bytes --]

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Script to clean old files from /usr/portage/distfiles

[Johannes Ballé]

Am Monday 11 November 2002 21:38 schrieb José Fonseca:
> I never do `rm -f /usr/portage/distfiles'. I find very useful having the
> sources of the installed packages ihandy as it avoids downloading
> everything over andagain when updating to new releases of the same
> version, or when installing Gentoo on another system. But after some
> time that directory gets too fat and is too boring to eliminate the
> deprecated packages by hand so I've made a script to automate this. You
> can see it attached.

Sounds good! I was missing this feature ...

>
> I've started with a shell script, then python, then I've read qpkg.sh,
> ebuild.sh, portage.py and many others in the search of the easiest way
> to do this. In the end everything boiled down to 2 lines - the rest is
> just sintatic sugar!
>
> I hope you enjoy it.
>
> José Fonseca

When I run your script, though, I get this:

sed: can't read /usr/portage/metadata/cache/sys-kernel/linux-headers-2.4.19: 
No such file or directory
sed: can't read /usr/portage/metadata/cache/berlin-base/berlin-env-1.0: No 
such file or directory
sed: can't read /usr/portage/metadata/cache/sys-devel/binutils-2.13.90.0.4: No 
such file or directory
sed: can't read /usr/portage/metadata/cache/sys-devel/gcc-3.2-r1: No such file 
or directory
sed: can't read /usr/portage/metadata/cache/sys-apps/tcp-wrappers-7.6-r4: No 
such file or directory
sed: can't read /usr/portage/metadata/cache/sys-libs/ncurses-5.2.20020511-r3: 
No such file or directory
sed: can't read /usr/portage/metadata/cache/sys-libs/readline-4.2a-r1: No such 
file or directory
sed: can't read /usr/portage/metadata/cache/fresco-base/fresco-env-1.0: No 
such file or directory
sed: can't read /usr/portage/metadata/cache/fresco-base/fresco-20020808: No 
such file or directory
sed: can't read /usr/portage/metadata/cache/media-video/em8300-modules-0.13.0: 
No such file or directory
sed: can't read 
/usr/portage/metadata/cache/media-video/em8300-libraries-0.13.0: No such file 
or directory

Could this be my fault? Sometimes when I was messing with the files in 
/usr/portage I wasn't sure whether an 'emerge rsync' would fix it all up 
again ...

I also have some stuff in /usr/local/portage ...

Johannes


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Script to clean old files from /usr/portage/distfiles

[Marko Mikulicic]

José Fonseca wrote:
> I never do `rm -f /usr/portage/distfiles'. I find very useful having the
> sources of the installed packages ihandy as it avoids downloading
> everything over andagain when updating to new releases of the same
> version, or when installing Gentoo on another system. But after some
> time that directory gets too fat and is too boring to eliminate the
> deprecated packages by hand so I've made a script to automate this. You
> can see it attached.
You read my mind!
I was just thinking  "i'd like to have a script that ...."
and you come out with this. great.
thanks. I should think more often :-)

It would be nice if this script would appear in gentoolkit...

> 
> I've started with a shell script, then python, then I've read qpkg.sh,
> ebuild.sh, portage.py and many others in the search of the easiest way
> to do this. In the end everything boiled down to 2 lines - the rest is
> just sintatic sugar!
nice. I just love when sed transforms the command line in completely 
unreadable magic :-)

How would be to make it a bit shorter and more configurable in one shot ?:

-PORTDIR=/usr/portage
-DISTDIR=$PORTDIR/distfiles
-CACHEDIR=$PORTDIR/metadata/cache
-PKGDBDIR=/var/db/pkg
+. /etc/make.globals;. /etc/make.conf

(or: for i in /etc/make.{globals,conf} do . $i;done )
Marko


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Script to clean old files from /usr/portage/distfiles

[José Fonseca]

On Tue, Nov 12, 2002 at 01:29:44AM +0100, Marko Mikulicic wrote:
> You read my mind!
> I was just thinking  "i'd like to have a script that ...."
> and you come out with this. great.
> thanks. I should think more often :-)
> 

I'm glad you liked it.

> It would be nice if this script would appear in gentoolkit...
> 
> >
> >I've started with a shell script, then python, then I've read qpkg.sh,
> >ebuild.sh, portage.py and many others in the search of the easiest way
> >to do this. In the end everything boiled down to 2 lines - the rest is
> >just sintatic sugar!
> nice. I just love when sed transforms the command line in completely 
> unreadable magic :-)
> 
> How would be to make it a bit shorter and more configurable in one shot ?:
> 
> -PORTDIR=/usr/portage
> -DISTDIR=$PORTDIR/distfiles
> +. /etc/make.globals;. /etc/make.conf
> (or: for i in /etc/make.{globals,conf} do . $i;done )

This is ok...

> -CACHEDIR=$PORTDIR/metadata/cache
> -PKGDBDIR=/var/db/pkg

...but becare full with these - you should keep them because they're
really not defined on the above files and the getting the last one wrong
will lead to the deletion of all files in distfiles...!

The way the script work is first get the list of all sources related
with the installed files from portache cache, merge it with the list of
sources in distfiles, and delete those which are unique - i.e., only are
in distfiles. There are two caveats with this procedure: 
 - it will delete sources of packages in PORTAGE_OVERLAY, if you have
   any, since they aren't cached.
 - if the cache is busted (or can't be found), the most probable outcome
   is that all files in distfiles get deleted... so a good idea is to
   replace 'rm' by 'echo' to get an idea of what is gonna be deleted.

These two caveats could be overcomed by getting the sources directly
from the installed ebuilds, but that would be terribly slow compared
with this, due to the parsing of all ebuilds.

Regards,

José Fonseca
__________________________________________________
Do You Yahoo!?
Everything you'll ever need on one web page
from News and Sport to Email and Music Charts
http://uk.my.yahoo.com

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Script to clean old files from /usr/portage/distfiles

[José Fonseca]

On Tue, Nov 12, 2002 at 12:42:41AM +0100, Johannes Ball? wrote:
> Am Monday 11 November 2002 21:38 schrieb José Fonseca:
> > I never do `rm -f /usr/portage/distfiles'. I find very useful having the
> > sources of the installed packages ihandy as it avoids downloading
> > everything over andagain when updating to new releases of the same
> > version, or when installing Gentoo on another system. But after some
> > time that directory gets too fat and is too boring to eliminate the
> > deprecated packages by hand so I've made a script to automate this. You
> > can see it attached.
> 
> Sounds good! I was missing this feature ...
> 
> >
> > I've started with a shell script, then python, then I've read qpkg.sh,
> > ebuild.sh, portage.py and many others in the search of the easiest way
> > to do this. In the end everything boiled down to 2 lines - the rest is
> > just sintatic sugar!
> >
> > I hope you enjoy it.
> >
> > José Fonseca
> 
> When I run your script, though, I get this:
> 
> sed: can't read /usr/portage/metadata/cache/sys-kernel/linux-headers-2.4.19: 
> No such file or directory
[...]

These are packages which were installed from your '/usr/local/portage'
or were removed from portage.

> 
> Could this be my fault? Sometimes when I was messing with the files in 
> /usr/portage I wasn't sure whether an 'emerge rsync' would fix it all up 
> again ...
> 
> I also have some stuff in /usr/local/portage ...

It's not really your fault, but just a limitation of the script. Please
read my second post for more info about this.

I decided not to flush the error output to /dev/null so that one is
alerted that the sources specific to these package [ do not belong to
any other]will get deleted.

I could write a variation of the script without these shortcomings -
that actually read the installed ebuilds which is *much* slower. I
personally don't need it as I only have little stuff in
/usr/local/portage.

If there is enough interest I'll add a couple of operation modes to the
script: "dummy", and "safe"; to make a dry-run and/or a safe evaluation
of the installed packages repectively.

José Fonseca
__________________________________________________
Do You Yahoo!?
Everything you'll ever need on one web page
from News and Sport to Email and Music Charts
http://uk.my.yahoo.com

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Script to clean old files from /usr/portage/distfiles

[Marko Mikulicic]

José Fonseca wrote:
> On Tue, Nov 12, 2002 at 01:29:44AM +0100, Marko Mikulicic wrote: >
> 
> ...but becare full with these - you should keep them because they're
> really not defined on the above files and the getting the last one wrong
> will lead to the deletion of all files in distfiles...!
oh. I didn't checked. I just thought it was from there.

Now that I'm looking better I've seen: 
portage.settings["PORTAGE_CACHEDIR"] in /usr/bin/emerge
rsync (or sync). From what is the portage cache built ?
  Since it is a cache it should mean that i caches some other
information wich remains present on disk, and the cache could be rebuilt 
at any instant.
Obviously it doesnt cache /var/db/pkg because overlay ebuild are listed 
there. Where
is the base data (meta-) which /usr/portage/metadata/cache caches ?
This also should solve the errors Johannes and me are getting.
for example:

$ qpkg linux-headers -I -v
sys-kernel/linux-headers-2.4.19

but /usr/portage/metadata/cache/sys-kernel/linux-headers-2.4.19 doesn't 
exist

> 
> The way the script work is first get the list of all sources related
> with the installed files from portache cache, merge it with the list of
> sources in distfiles, and delete those which are unique - i.e., only are
> in distfiles.
 > There are two caveats with this procedure:
>  - it will delete sources of packages in PORTAGE_OVERLAY, if you have
I don't understand why packages in PORTAGE_OVERLAY are not put in the
/usr/portage/metadata/cache.
They are in /var/db/pkg/ and should be treaten like any other ebuild.

>  - if the cache is busted (or can't be found), the most probable outcome
>    is that all files in distfiles get deleted... so a good idea is to
>    replace 'rm' by 'echo' to get an idea of what is gonna be deleted.
I guess that simplicity or elegance would be completely destroyed
when introducing commandline parsing for the gentooish --pretend (or at 
least only '-p')
option.  However, my opinion is that first we should have a working 
script that helps
somebody who knows what he want, and then

> 
> These two caveats could be overcomed by getting the sources directly
> from the installed ebuilds, but that would be terribly slow compared
> with this, due to the parsing of all ebuilds.
yes, because of the ${P} or ${PN} or who knows what other triks are in 
ther (for example
xfree 4.2.1 X_DRIVERS, MS_FONT_URLS, ....). you should evaluate the 
ebuild in order
to resolve local script variables; not a simple sed. And then evaluating 
the script is not so
simple because of "inherits" and other commands not defined there. I had 
some problems
while writting emerge-rsync (it will be there in the next gentoolkit).
  I think your choice is not only the faster but also it will work with 
all packages which
have passed emerge. Why duplicate the work. Emerge produced the output, 
why don't use it.
I think that the question about portage overlay is not to be solved here 
but with a more general solution.

Marko


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Script to clean old files from /usr/portage/distfiles

[Marko Mikulicic]

José Fonseca wrote:
> On Tue, Nov 12, 2002 at 12:42:41AM +0100, Johannes Ball? wrote:
> 
>>Am Monday 11 November 2002 21:38 schrieb José Fonseca:
>>

>>
>>sed: can't read /usr/portage/metadata/cache/sys-kernel/linux-headers-2.4.19: 
>>No such file or directory
> 
> [...]
> 
> These are packages which were installed from your '/usr/local/portage'
> or were removed from portage.

I ran "emerge linux-headers"
emerge (1 of 1) sys-kernel/linux-headers-2.4.19

/var/db/pkg/sys-kernel/linux-headers-2.4.19     exists
/usr/portage/metadata/cache/sys-kernel/linux-headers-2.4.19  does not exist

/var/db/pkg/sys-kernel/linux-headers-2.4.18     does not exist
/usr/portage/metadata/cache/sys-kernel/linux-headers-2.4.18  exists

it seems strange to me. the cache is not up to date.
Is there a way to force the ebuild of the cache ?

> I could write a variation of the script without these shortcomings -
> that actually read the installed ebuilds which is *much* slower. I
> personally don't need it as I only have little stuff in
> /usr/local/portage
> 
> If there is enough interest I'll add a couple of operation modes to the
> script: "dummy", and "safe"; to make a dry-run and/or a safe evaluation
> of the installed packages repectively.

I'd have enough interest :-) . I never do an emerge whatever without "-p"
first. even if this data is not lost forever I have the habit of 
checking first.

Marko


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Script to clean old files from /usr/portage/distfiles

[José Fonseca]

[-- Attachment #1: Type: text/plain, Size: 981 bytes --]

On Mon, Nov 11, 2002 at 09:28:49PM -0600, Nick Jones wrote:
> metadata is _not_ something you should be using. metadata is a
> quick-search cache database that is updated remotely. It has a basic
> link to your system, but should in no way be used to determine
> settings and data as it can be out of date. It's primary purpose is
> for searching. /var/cache/edb/dep is a valid, local system cache.

Thanks! I hadn't noticed the existence of /var/cache/edb/dep. Using it
instead of /usr/portage/metadata/cache works flawlessly.

All interested, please check the attached new version. It includes all
suggestions which were made so far. It requires >=sed-4.0, otherwise
replace "xargs sed -s" by "xargs -l sed". If there aren't further
corrections/suggestions then I'll create a bug in bugzilla for it.

> Try this:
> 
> FEATURES="mirror" emerge -efp world |  ...

Unfortunately when doing this, emerge starts complaining that it can't
satisfy dependencies.

Regards,

José Fonseca

[-- Attachment #2: distfiles-clean --]
[-- Type: text/plain, Size: 1149 bytes --]

#!/bin/sh
#
# distfiles-clean
#
# Cleans unused files from Portage's distfiles directory.
#
# José Fonseca <j_r_fonseca@yahoo.co.uk>

PROGRAM=`basename "$0"`

while [ ${#} -gt 0 ]
do
	case "$1" in
		-h|--help)
			USAGE=y
			break
			;;
		-p|--pretend)
			PRETEND=y
			;;
		*)
			echo "$PROGRAM: Invalid option \'$1\'" 1>&2
			USAGE=y
			break
			;;
	esac
	shift
done

# For PORTDIR and DISTDIR
. /etc/make.globals
. /etc/make.conf

if [ "$USAGE" ]
then
	echo "Usage:	$PROGRAM [-h|--help] [-p|--pretend]"
	echo "Cleans unused files from $DISTDIR directory."
	exit
fi

DBDIR=/var/db/pkg
CACHEDIR=/var/cache/edb/dep

for DIR in "$PORTDIR" "$DISTDIR" "$DBDIR" "$CACHEDIR"
do
	if [ ! -d "$DIR" ]
	then
		echo "$PROGRAM: \'$DIR\' not found."
		exit
	fi
done

TMPFILE=`mktemp /tmp/$0.XXXXXX`

find $DBDIR -name '*.ebuild' | sed -n -e "s:^$DBDIR/\([^/]*\)/\([^/]*\)/\([^/]*\)\.ebuild$:$CACHEDIR/\1/\3:p" | xargs sed -s -e '4!d;s/[[:space:]]\+/\n/g' | sed -n -e 's:^.*/::p' | sort -u > $TMPFILE && ls -1 $DISTDIR | grep -v '^cvs-src$' | comm -23 - $TMPFILE | {
	if [ "$PRETEND" ]
	then
		cat
	else
		sed -e "s:^:$DISTDIR/:" | xargs rm -f
	fi
}

rm $TMPFILE


[-- Attachment #3: Type: text/plain, Size: 37 bytes --]

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Script to clean old files from /usr/portage/distfiles

[William Kenworthy]

I get the messages below before the duplicates after running with the -p
flag.  Is there a way to ensure the cache is up-to-date and clean before
running?  

BillK

sed: can't read /var/cache/edb/dep/app-admin/metalog-0.6-r9: No such
file or directory
sed: can't read /var/cache/edb/dep/x11-themes/crux-1.9.3: No such file
or directory
sed: can't read /var/cache/edb/dep/x11-themes/gtk-thinice-engine-2.0.1:
No such file or directory
sed: can't read /var/cache/edb/dep/gnome-base/libgnome-2.0.1-r1: No such
file or directory
sed: can't read /var/cache/edb/dep/gnome-base/ORBit2-2.4.0-r2: No such
file or directory
sed: can't read /var/cache/edb/dep/app-arch/rpm-4.0.4-r2: No such file
or directory
sed: can't read /var/cache/edb/dep/app-text/ispell-3.2.06-r1: No such
file or directory
sed: can't read /var/cache/edb/dep/app-text/gnome-spell-0.4.1-r1: No
such file or directory
sed: can't read /var/cache/edb/dep/sys-devel/gcc-3.1.1-r1: No such file
or directory
sed: can't read /var/cache/edb/dep/sys-devel/make-3.79.1-r3: No such
file or directory
sed: can't read /var/cache/edb/dep/sys-devel/perl-5.6.1-r4: No such file
or directory
sed: can't read /var/cache/edb/dep/net-www/lynx-2.8.4a-r4: No such file
or directory
sed: can't read /var/cache/edb/dep/media-libs/lcms-1.08: No such file or
directory
sed: can't read /var/cache/edb/dep/sys-apps/hdparm-5.2: No such file or
directory
sed: can't read /var/cache/edb/dep/sys-apps/gawk-3.1.0-r2: No such file
or directory
sed: can't read /var/cache/edb/dep/sys-libs/pam-0.75-r6: No such file or
directory
sed: can't read /var/cache/edb/dep/sys-libs/glibc-2.2.5-r4: No such file
or directory
sed: can't read /var/cache/edb/dep/dev-libs/libsigc++-1.1.13: No such
file or directory
sed: can't read /var/cache/edb/dep/dev-libs/libxslt-1.0.18: No such file
or directory
sed: can't read /var/cache/edb/dep/dev-libs/expat-1.95.2: No such file
or directory
sed: can't read /var/cache/edb/dep/media-sound/sox-12.17.3: No such file
or directory
sed: can't read /var/cache/edb/dep/media-sound/ecasound-2.0.4: No such
file or directory
sed: can't read /var/cache/edb/dep/media-video/nvidia-kernel-1.0.2960:
No such file or directory
sed: can't read /var/cache/edb/dep/gnome-extra/gtkhtml-1.0.2-r2: No such
file or directory
sed: can't read /var/cache/edb/dep/gnome-extra/gal2-0.0.4-r1: No such
file or directory
sed: can't read /var/cache/edb/dep/net-dialup/ppp-2.4.1-r8: No such file
or directory
4.2.0-libGLU-bad-extern.patch
4.2.0-xlib-i18n-module.patch
4.2.0-zlib-security.patch
ExtUtils-F77-1.13.tar.gz


On Tue, 2002-11-12 at 18:21, José Fonseca wrote:
> On Mon, Nov 11, 2002 at 09:28:49PM -0600, Nick Jones wrote:
> > metadata is _not_ something you should be using. metadata is a
> > quick-search cache database that is updated remotely. It has a basic
> > link to your system, but should in no way be used to determine
> > settings and data as it can be out of date. It's primary purpose is
> > for searching. /var/cache/edb/dep is a valid, local system cache.
> 
> Thanks! I hadn't noticed the existence of /var/cache/edb/dep. Using it
> instead of /usr/portage/metadata/cache works flawlessly.
> 
> All interested, please check the attached new version. It includes all
> suggestions which were made so far. It requires >=sed-4.0, otherwise
> replace "xargs sed -s" by "xargs -l sed". If there aren't further
> corrections/suggestions then I'll create a bug in bugzilla for it.
> 
> > Try this:
> > 
> > FEATURES="mirror" emerge -efp world |  ...
> 
> Unfortunately when doing this, emerge starts complaining that it can't
> satisfy dependencies.
> 
> Regards,
> 
> José Fonseca
> ----
> 

> #!/bin/sh
> #
> # distfiles-clean
> #
> # Cleans unused files from Portage's distfiles directory.
> #
> # José Fonseca <j_r_fonseca@yahoo.co.uk>
> 
> PROGRAM=`basename "$0"`
> 
> while [ ${#} -gt 0 ]
> do
> 	case "$1" in
> 		-h|--help)
> 			USAGE=y
> 			break
> 			;;
> 		-p|--pretend)
> 			PRETEND=y
> 			;;
> 		*)
> 			echo "$PROGRAM: Invalid option \'$1\'" 1>&2
> 			USAGE=y
> 			break
> 			;;
> 	esac
> 	shift
> done
> 
> # For PORTDIR and DISTDIR
> . /etc/make.globals
> . /etc/make.conf
> 
> if [ "$USAGE" ]
> then
> 	echo "Usage:	$PROGRAM [-h|--help] [-p|--pretend]"
> 	echo "Cleans unused files from $DISTDIR directory."
> 	exit
> fi
> 
> DBDIR=/var/db/pkg
> CACHEDIR=/var/cache/edb/dep
> 
> for DIR in "$PORTDIR" "$DISTDIR" "$DBDIR" "$CACHEDIR"
> do
> 	if [ ! -d "$DIR" ]
> 	then
> 		echo "$PROGRAM: \'$DIR\' not found."
> 		exit
> 	fi
> done
> 
> TMPFILE=`mktemp /tmp/$0.XXXXXX`
> 
> find $DBDIR -name '*.ebuild' | sed -n -e "s:^$DBDIR/\([^/]*\)/\([^/]*\)/\([^/]*\)\.ebuild$:$CACHEDIR/\1/\3:p" | xargs sed -s -e '4!d;s/[[:space:]]\+/\n/g' | sed -n -e 's:^.*/::p' | sort -u > $TMPFILE && ls -1 $DISTDIR | grep -v '^cvs-src$' | comm -23 - $TMPFILE | {
> 	if [ "$PRETEND" ]
> 	then
> 		cat
> 	else
> 		sed -e "s:^:$DISTDIR/:" | xargs rm -f
> 	fi
> }
> 
> rm $TMPFILE
> 
> ----
> 

> --
> gentoo-dev@gentoo.org mailing list



--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Script to clean old files from /usr/portage/distfiles

[José Fonseca]

On Tue, Nov 12, 2002 at 07:28:35PM +0800, William Kenworthy wrote:
> I get the messages below before the duplicates after running with the -p
> flag.  Is there a way to ensure the cache is up-to-date and clean before
> running?  
> 

Calling ebuild with depend updates the cache of that ebuild. For
example, doing:

  ebuild /usr/portage/x11-base/xfree/xfree-4.2.1-r1.ebuild depend

updates /var/cache/edb/dep/x11-base/xfree-4.2.1-r1 on my system.

But my understanding from looking at the sources is that emerge
automatically keeps the cache uptodate in the normal use. But then
again, it's not perfectly clear to me the roles of the two caches
(/var/cache/edb/dep and /usr/portage/metadata/cache), specifically in
which order they are used and updated.

> sed: can't read /var/cache/edb/dep/app-admin/metalog-0.6-r9: No such
> file or directory
> ...

If the list [of files that the script plans to delete] below is
complete, then the above error messages aren't important. If not then I
can update the script two use both caches in an attempt to find more.

> 4.2.0-libGLU-bad-extern.patch
> 4.2.0-xlib-i18n-module.patch
> 4.2.0-zlib-security.patch
> ExtUtils-F77-1.13.tar.gz

José Fonseca
__________________________________________________
Do You Yahoo!?
Everything you'll ever need on one web page
from News and Sport to Email and Music Charts
http://uk.my.yahoo.com

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Script to clean old files from /usr/portage/distfiles

[Nils Ohlmeier]

On Tuesday 12 November 2002 11:21, José Fonseca wrote:
> All interested, please check the attached new version. It includes all
> suggestions which were made so far. It requires >=sed-4.0, otherwise
> replace "xargs sed -s" by "xargs -l sed". If there aren't further
> corrections/suggestions then I'll create a bug in bugzilla for it.

First of all great script. I was thinking about such a script a long time but 
didn't find the time to do it myself. (And if i did it, it wouldn't be such a 
good scipt :-)

Feedback / Corrections:
the first run results in the following

cloudcity root # distfiles-clean -p
mktemp: cannot make temp file /tmp//usr/local/sbin/distfiles-clean.E63Npk: No 
such file or directory
/usr/local/sbin/distfiles-clean: $TMPFILE: ambiguous redirect
sed: can't read /var/cache/edb/dep/sys-kernel/linux-headers-2.4.19: No such 
file or directory
xargs: sed: terminated by signal 13
rm: too few arguments
Try `rm --help' for more information.

I think there is an error in the script in line 53. I changed /tmp/$0.XXXXX 
into /tmp/$PROGRAM.XXXXX and then the result looks like this:

cloudcity sbin # distfiles-clean -p
sed: can't read /var/cache/edb/dep/sys-kernel/linux-headers-2.4.19: No such 
file or directory
alsa-lib-0.5.10b.tar.bz2
gaim-0.59.3.tar.bz2
j2sdk-1_4_1_01-linux-i586.bin *
kde-i18n-3.0.4.tar.bz2 *
kde-i18n-gentoo.patch *
rp8_linux20_libc6_i386_cs2_rpm *

I marked the packets which are installed with an *. So removing of this would 
be to much :)

Thanks anyway for the script. Keep up the good work.

  Nils Ohlmeier

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Script to clean old files from /usr/portage/distfiles

[José Fonseca]

On Tue, Nov 12, 2002 at 04:37:39PM +0100, Nils Ohlmeier wrote:
> First of all great script. I was thinking about such a script a long time but 
> didn't find the time to do it myself. (And if i did it, it wouldn't be such a 
> good scipt :-)
> 
> Feedback / Corrections:
[...]
> I think there is an error in the script in line 53. I changed /tmp/$0.XXXXX 
> into /tmp/$PROGRAM.XXXXX and then the result looks like this:
> 

Thanks, applied.

> cloudcity sbin # distfiles-clean -p
> sed: can't read /var/cache/edb/dep/sys-kernel/linux-headers-2.4.19: No
> such file or directory
> alsa-lib-0.5.10b.tar.bz2
> gaim-0.59.3.tar.bz2
> j2sdk-1_4_1_01-linux-i586.bin *

Unfortunately there is nothing one can do to know that the
j2sdk-1_4_1_01-linux-i586.bin is used by sun-jdk-1.4.1.01.ebuild because
it isn't listed there (SRC_URI=""). The only solution is have an option
to pass a file with a list of wildcards of the packages wich should be
left alone, for these situations.

> kde-i18n-3.0.4.tar.bz2 *
> kde-i18n-gentoo.patch *

Strange, these two files are listed on
/var/cache/edb/dep/app-i18n/kde-i18n-3.0.4 in my system. Aren't they
there in yours too? If not, see if doing

  ebuild /usr/portage/app-i18n/kde-i18n/kde-i18n-3.0.4.ebuild depend

helps.

> rp8_linux20_libc6_i386_cs2_rpm *

rp8_linux20_libc6_i386_cs2_rpm isn't a full URI with slashes '/' so it
was being considered as junk by the script. It should be fixed now.

> I marked the packets which are installed with an *. So removing of
> this would  be to much :)

Try again, with the script from
http://mefriss1.swan.ac.uk/~jfonseca/chest/gentoo/distfiles-clean [to
avoid flood everybody inbox with more attachments], by doing

  distfiles-clean -p -i ignorelist

where ignorelist is a file containing

  j2sdk-*.bin

plus anything else you want.

Regards,

José Fonseca
__________________________________________________
Do You Yahoo!?
Everything you'll ever need on one web page
from News and Sport to Email and Music Charts
http://uk.my.yahoo.com

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Script to clean old files from /usr/portage/distfiles

[Nils Ohlmeier]

On Tuesday 12 November 2002 18:56, José Fonseca wrote:
> On Tue, Nov 12, 2002 at 04:37:39PM +0100, Nils Ohlmeier wrote:
> > cloudcity sbin # distfiles-clean -p
> > sed: can't read /var/cache/edb/dep/sys-kernel/linux-headers-2.4.19: No
> > such file or directory
> > alsa-lib-0.5.10b.tar.bz2
> > gaim-0.59.3.tar.bz2
> > j2sdk-1_4_1_01-linux-i586.bin *
>
> Unfortunately there is nothing one can do to know that the
> j2sdk-1_4_1_01-linux-i586.bin is used by sun-jdk-1.4.1.01.ebuild because
> it isn't listed there (SRC_URI=""). The only solution is have an option
> to pass a file with a list of wildcards of the packages wich should be
> left alone, for these situations.

Okay -i works good.

> > kde-i18n-3.0.4.tar.bz2 *
> > kde-i18n-gentoo.patch *
>
> Strange, these two files are listed on
> /var/cache/edb/dep/app-i18n/kde-i18n-3.0.4 in my system. Aren't they
> there in yours too? If not, see if doing
>
>   ebuild /usr/portage/app-i18n/kde-i18n/kde-i18n-3.0.4.ebuild depend
>
> helps.

Sorry my fault. I did not read carefull enough. The tar.bz2 can be removed, 
because i canceld the download after seeing the size of the file :)
And the patch is downloaded because of a bug.

> > rp8_linux20_libc6_i386_cs2_rpm *
>
> rp8_linux20_libc6_i386_cs2_rpm isn't a full URI with slashes '/' so it
> was being considered as junk by the script. It should be fixed now.

Yes fixed.

> > I marked the packets which are installed with an *. So removing of
> > this would  be to much :)
>
> Try again, with the script from
> http://mefriss1.swan.ac.uk/~jfonseca/chest/gentoo/distfiles-clean [to
> avoid flood everybody inbox with more attachments], by doing
>
>   distfiles-clean -p -i ignorelist
>
> where ignorelist is a file containing
>
>   j2sdk-*.bin
>
> plus anything else you want.

Everything works for me now perfectly.
BTW i tested your skript on my one year old 1.2 installation and i gave me 
over 400MB diskspace back!

Definalty a candidate for gentoolkit IMHO.

Thanks
  Nils Ohlmeier


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Script to clean old files from /usr/portage/distfiles

[DJ Cozatt]

I won't repost the thread but I liked this idea and wondered it if could
also be extended to remove old pkg's in PKGDIR set when usindg buildpkg
in /etc/make.conf Would yield a more general housecleaning app

bus_drivr

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Script to clean old files from /usr/portage/distfiles

[Chris Bainbridge]

On Tuesday 25 March 2003 18:55, DJ Cozatt wrote:
> I won't repost the thread but I liked this idea and wondered it if could
> also be extended to remove old pkg's in PKGDIR set when usindg buildpkg
> in /etc/make.conf Would yield a more general housecleaning app
>
> bus_drivr
>
> --
> gentoo-dev@gentoo.org mailing list

Heres my binary packages cleaning script. Its very simplistic but does the 
job.

#!/usr/bin/python

import os
import stat
import string

def samename(s0,s1):
  i=0
  for x in s0:
    if x in string.digits:
      break
    i+=1
  # same for s1
  j=0
  for x in s0:
    if x in string.digits:
      break
    j+=1
  return s0[:i]==s1[:j]

dirs=['All']
removed=[]
for dir in dirs:
  print 'Searching files'
  os.chdir('/usr/portage/packages/'+dir)
  list=os.listdir('.')
  list.sort()
  current=list[0]
  for x in list[1:]:
    if samename(current,x):
      time0=os.stat(current)[stat.ST_CTIME]
      time1=os.stat(x)[stat.ST_CTIME]
      if time0<time1:
        removed.append(current)
        os.remove(current)
        current=x
      else:
        removed.append(x)
        os.remove(x)
    else:
      current=x

print 'Removed',removed

print 'Searching for broken symlinks'
removed=[]
dirs=os.listdir('/usr/portage/packages/')
for dir in dirs:
  os.chdir('/usr/portage/packages/'+dir)
  list=os.listdir('.')
  for f in list:
    try:
      os.stat(f)
    except:
      if os.path.islink(f):
        removed.append(f)
        os.remove(f)

print 'Removed',removed



--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Script to clean old files from /usr/portage/distfiles

[DJ Cozatt]

On Tue, 25 Mar 2003 20:34:38 +0000
Chris Bainbridge <C.J.Bainbridge@ed.ac.uk> wrote:

> On Tuesday 25 March 2003 18:55, DJ Cozatt wrote:
> > I won't repost the thread but I liked this idea and wondered it if could
> > also be extended to remove old pkg's in PKGDIR set when usindg buildpkg
> > in /etc/make.conf Would yield a more general housecleaning app
> >
> > bus_drivr
> >
> > --
> > gentoo-dev@gentoo.org mailing list
> 
> Heres my binary packages cleaning script. Its very simplistic but does the 
> job.
> 
> #!/usr/bin/python
> 
> import os
> import stat
> import string
> 
> def samename(s0,s1):
>   i=0
>   for x in s0:
>     if x in string.digits:
>       break
>     i+=1
>   # same for s1
>   j=0
>   for x in s0:
>     if x in string.digits:
>       break
>     j+=1
>   return s0[:i]==s1[:j]
> 
> dirs=['All']
> removed=[]
> for dir in dirs:
>   print 'Searching files'
>   os.chdir('/usr/portage/packages/'+dir)
>   list=os.listdir('.')
>   list.sort()
>   current=list[0]
>   for x in list[1:]:
>     if samename(current,x):
>       time0=os.stat(current)[stat.ST_CTIME]
>       time1=os.stat(x)[stat.ST_CTIME]
>       if time0<time1:
>         removed.append(current)
>         os.remove(current)
>         current=x
>       else:
>         removed.append(x)
>         os.remove(x)
>     else:
>       current=x
> 
> print 'Removed',removed
> 
> print 'Searching for broken symlinks'
> removed=[]
> dirs=os.listdir('/usr/portage/packages/')
> for dir in dirs:
>   os.chdir('/usr/portage/packages/'+dir)
>   list=os.listdir('.')
>   for f in list:
>     try:
>       os.stat(f)
>     except:
>       if os.path.islink(f):
>         removed.append(f)
>         os.remove(f)
> 
> print 'Removed',removed
> 
> 
> 
> --
> gentoo-dev@gentoo.org mailing list
> 

Thank you :)

--
gentoo-dev@gentoo.org mailing list