[gentoo-dev] New feature proposition (make.conf)

[gentoo-dev] New feature proposition (make.conf)

[Philippe Lafoucrière]

I don't think it needs a new GLEP :

I'd like to see a new feature in the current list : mrproper.
This feature would allow portage to remove downloaded files in
/usr/portage/distfiles, and clean after emerging the /var/tmp/portage.

This would help users to save some disk space on little configurations.


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] New feature proposition (make.conf)

[Zack Gilburd]

[-- Attachment #1: signed data --]
[-- Type: text/plain, Size: 1084 bytes --]

On Sunday 17 August 2003 02:45 am, Philippe Lafoucrière wrote:
> I don't think it needs a new GLEP :
>
> I'd like to see a new feature in the current list : mrproper.
> This feature would allow portage to remove downloaded files in
> /usr/portage/distfiles, and clean after emerging the /var/tmp/portage.
>
> This would help users to save some disk space on little configurations.
>

One of the problems that plagues Gentoo is that the distro's growth is leading 
to an insane amount of distfiles downloads (and even excessive downloads) 
along with excessive rsync'ing.  Automatic removal of distfiles would 
substancially increase the amount of distfiles downloads due to the fact that 
packages often have revisions made to them and revisions mean that the 
software has to be recompiled -- thus the distfile would have to be 
redownloaded of already removed.

Until the general downloading (and rsyncing) etiqutte improves, I don't see 
how this could be The Right Thing(TM) to do.

-- 
Zack Gilburd
 http://tehunlose.com
  GnuPG Key ID: A79A45668240AB6C

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] New feature proposition (make.conf)

[Jason Wever]

[-- Attachment #1: Type: text/plain, Size: 908 bytes --]

On Sun, 17 Aug 2003 04:06:49 -0700
Zack Gilburd <klasikahl@gentoo.org> wrote:

> One of the problems that plagues Gentoo is that the distro's growth is
> leading to an insane amount of distfiles downloads (and even excessive
> downloads) along with excessive rsync'ing.  Automatic removal of
> distfiles would substancially increase the amount of distfiles downloads
> due to the fact that packages often have revisions made to them and
> revisions mean that the software has to be recompiled -- thus the
> distfile would have to be redownloaded of already removed.

My thought is rather than remove the distfiles for the version you are
building, remove the distfiles for versions that have just been
uninstalled by clean or update (granted they aren't being used by the new
version as well). This way you only have the distfiles for what's
currently on your system.

-- 
Jason Wever
Gentoo/Sparc Team Lead

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] New feature proposition (make.conf)

[Philippe Lafoucrière]

> My thought is rather than remove the distfiles for the version you are
> building, remove the distfiles for versions that have just been
> uninstalled by clean or update (granted they aren't being used by the new
> version as well). This way you only have the distfiles for what's
> currently on your system.

I agree. I have a lot of files that should go to Trash in my distfiles.
I have to remove them by hand. why keep

package_v1.0.tbz2
package_v1.1.tbz2
package_v1.2.tbz2
package_v1.3.tbz2

where package_v1.3.tbz2 will only be used (not for packages with
patches).



--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] New feature proposition (make.conf)

[Chris Gianelloni]

[-- Attachment #1.1: Type: text/plain, Size: 978 bytes --]

On Sun, 2003-08-17 at 07:59, Philippe Lafoucrière wrote:
> > My thought is rather than remove the distfiles for the version you are
> > building, remove the distfiles for versions that have just been
> > uninstalled by clean or update (granted they aren't being used by the new
> > version as well). This way you only have the distfiles for what's
> > currently on your system.
> 
> I agree. I have a lot of files that should go to Trash in my distfiles.
> I have to remove them by hand. why keep
> 
> package_v1.0.tbz2
> package_v1.1.tbz2
> package_v1.2.tbz2
> package_v1.3.tbz2
> 
> where package_v1.3.tbz2 will only be used (not for packages with
> patches).

I don't remember where I got this, so I cannot give credit to the
person, but I find this is a great tool for removing distfiles for
packages you no longer have installed.

It is also in python, so portage integration could be a possibility.

-- 
Chris Gianelloni
Developer, Gentoo Linux

[-- Attachment #1.2: distclean.py --]
[-- Type: text/x-python, Size: 1771 bytes --]

#!/usr/bin/env python
# distclean.py - remove source files for Gentoo
# packages that are no longer installed
# Fredrik Arnerup (e97_far@e.kth.se), 2003
# Use with '-p' (pretend) flag to just get a list of files
# that would be removed
import sys, os, os.path, getopt, portage

opt_p = 0
try:
    if len(sys.argv) > 1:
        opt_p = getopt.getopt(sys.argv[1:], "p")[0][0][0] == '-p'
except getopt.GetoptError:
    pass

distdir = portage.config().environ()['DISTDIR']
print 'DISTDIR =', distdir
    
vartree = portage.db["/"]["vartree"]
packages = []
for name in vartree.getallnodes():
    packages.extend(vartree.dep_match(name))

files = {}
for package in packages:
    try:
        package_files = portage.portdb.aux_get(package, ['SRC_URI'])[0].split()
        package_files = [(url.split('/')[-1]) for url in package_files]
        for filename in package_files:
            files[filename] = 1
    except:
        print "Failed to get file list for", package

if not files:
    sys.exit("No package files found.  This can't be right.\n")

try:
    list = portage.listdir(distdir)
except os.OSError:
    sys.exit("Failed to read " + distdir)

size = 0; count = 0
for file in list:
    abs_file = distdir + '/' + file
    if (os.path.isfile(abs_file) and (not os.path.islink(abs_file)) 
        and (not file in files)):
        size += os.stat(abs_file).st_size
        count += 1
        if opt_p:
            print 'Would remove', abs_file
        else:
            try:
                os.remove(abs_file)
                print 'Removed', abs_file
            except OSError:
                print 'Failed to remove', abs_file


size /= 1024  ## KB
print "%i files, total size: %i KB" % (count, size)

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] New feature proposition (make.conf)

[Mike Frysinger]

[-- Attachment #1: signed data --]
[-- Type: text/plain, Size: 347 bytes --]

On Monday 18 August 2003 08:38, Chris Gianelloni wrote:
> I don't remember where I got this, so I cannot give credit to the
> person, but I find this is a great tool for removing distfiles for
> packages you no longer have installed.
>
> It is also in python, so portage integration could be a possibility.

i'm pretty sure nick wrote it ;)
-mike

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] New feature proposition (make.conf)

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 667 bytes --]

On Mon, 2003-08-18 at 14:51, Mike Frysinger wrote:
> On Monday 18 August 2003 08:38, Chris Gianelloni wrote:
> > I don't remember where I got this, so I cannot give credit to the
> > person, but I find this is a great tool for removing distfiles for
> > packages you no longer have installed.
> >
> > It is also in python, so portage integration could be a possibility.
> 
> i'm pretty sure nick wrote it ;)

Actually, it is Fredrik Arnerup (e97_far@e.kth.se) who wrote it.  I use
it on my machines with diff to figure out what files are not in use by
any of my machines (I have a central portage tree).

-- 
Chris Gianelloni
Developer, Gentoo Linux

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] New feature proposition (make.conf)

[Dewet Diener]

[-- Attachment #1: Type: text/plain, Size: 750 bytes --]

On Sunday 17 August 2003 13:17, Jason Wever wrote:
> My thought is rather than remove the distfiles for the version you
> are building, remove the distfiles for versions that have just been
> uninstalled by clean or update (granted they aren't being used by
> the new version as well). This way you only have the distfiles for
> what's currently on your system.

Someone posted this to the list way back.  I'm still using it with good 
results to keep my distfiles directory fairly trim;  its just a simple 
shell script to delete old, unused distfiles.

Regards,
Dewet

-- 
Dewet Diener   <gentoo@dewet.org>   http://dewet.org
Professional Student, avid Gentoo user :)
Stellenbosch, South Africa (33º 55" 58.80'S  18º 51" 00.00'E)

[-- Attachment #2: distfiles-clean --]
[-- Type: application/x-shellscript, Size: 1504 bytes --]

[-- Attachment #3: Type: text/plain, Size: 37 bytes --]

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] New feature proposition (make.conf)

[Stewart Honsberger]

Zack Gilburd wrote:

> One of the problems that plagues Gentoo is that the distro's growth is leading 
> to an insane amount of distfiles downloads (and even excessive downloads) 
> along with excessive rsync'ing.  Automatic removal of distfiles would 
> substancially increase the amount of distfiles downloads due to the fact that 
> packages often have revisions made to them and revisions mean that the 
> software has to be recompiled -- thus the distfile would have to be 
> redownloaded of already removed.
> 
> Until the general downloading (and rsyncing) etiqutte improves, I don't see 
> how this could be The Right Thing(TM) to do.

I've been thinking along the lines of a cronjob being pulled in with, 
say, Portage. Said cron job would, on a daily/weekly basis remove old 
distfiles based upon age, and perhaps even have a setting to consider 
size. (Eg; always/never remove files greater/less than a certain size).

That's where it gets tricky. OpenOffice, Mozilla et al. are two great 
examples of packages whose source tarballs are *LARGE*. On one hand, 
those would, in one fell swoop, free up the most HDD space = most 
benefeit. On the other hand, they'd also cost more bandwidth to 
re-download = most detrimental.

Operating on a strictly age-based system based around file access time 
could potentially work, except that Gentoo's install defaults and/or 
suggests strongly the notion of 'noatime' in fstab entries.

The script for the cronjob is easy, especially if it's only date-based. 
The politics involved in implementing it, however, could be hairy. I'll 
throw my script out there for review if it interests anybody and let 
someone who's more proficient in Bash scripting add in the filesize details.

-- 
Stewart Honsberger
http://blackdeath.snerk.org/
"Capitalists, by nature, organize to protect themselves.
-- Geeks, by nature, resist organizaion."


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] New feature proposition (make.conf)

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 2569 bytes --]

On Thu, 2003-08-21 at 00:44, Stewart Honsberger wrote:
> Zack Gilburd wrote:
> 
> > One of the problems that plagues Gentoo is that the distro's growth is leading 
> > to an insane amount of distfiles downloads (and even excessive downloads) 
> > along with excessive rsync'ing.  Automatic removal of distfiles would 
> > substancially increase the amount of distfiles downloads due to the fact that 
> > packages often have revisions made to them and revisions mean that the 
> > software has to be recompiled -- thus the distfile would have to be 
> > redownloaded of already removed.
> > 
> > Until the general downloading (and rsyncing) etiqutte improves, I don't see 
> > how this could be The Right Thing(TM) to do.
> 
> I've been thinking along the lines of a cronjob being pulled in with, 
> say, Portage. Said cron job would, on a daily/weekly basis remove old 
> distfiles based upon age, and perhaps even have a setting to consider 
> size. (Eg; always/never remove files greater/less than a certain size).
> 
> That's where it gets tricky. OpenOffice, Mozilla et al. are two great 
> examples of packages whose source tarballs are *LARGE*. On one hand, 
> those would, in one fell swoop, free up the most HDD space = most 
> benefeit. On the other hand, they'd also cost more bandwidth to 
> re-download = most detrimental.
> 
> Operating on a strictly age-based system based around file access time 
> could potentially work, except that Gentoo's install defaults and/or 
> suggests strongly the notion of 'noatime' in fstab entries.
> 
> The script for the cronjob is easy, especially if it's only date-based. 
> The politics involved in implementing it, however, could be hairy. I'll 
> throw my script out there for review if it interests anybody and let 
> someone who's more proficient in Bash scripting add in the filesize details.

I use a script which deletes any files in distfiles which do not belong
to an installed package.  This keeps old packages out of my distfiles,
but also allows me to keep enough distfiles locally to keep from
hammering the mirrors constantly.  I think possibly a combination of the
two should be in order?  Maybe remove all non-installed distfiles on a
weekly basis and also remove all distfiles that have not been accessed
in something like 90 days?  This would keep things around long enough to
probably make it through any revision changes but also manage to clear
up room for the people that are complaining about space.

-- 
Chris Gianelloni
Developer, Gentoo Linux

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] New feature proposition (make.conf)

[Toby Dickenson]

On Thursday 21 August 2003 05:44, Stewart Honsberger wrote:

> That's where it gets tricky. OpenOffice, Mozilla et al. are two great
> examples of packages whose source tarballs are *LARGE*. On one hand,
> those would, in one fell swoop, free up the most HDD space = most
> benefeit. On the other hand, they'd also cost more bandwidth to
> re-download = most detrimental.
>
> Operating on a strictly age-based system based around file access time
> could potentially work, except that Gentoo's install defaults and/or
> suggests strongly the notion of 'noatime' in fstab entries.

I used something similar for a while (using 'find' and atimes) but it was 
never very satisfactory. The right time to delete these large tarballs is 
"after the user has decided he wont be needing it". Im not sure how to 
automate that.

Perhaps the best solution would be something like etc-update.... It finds the 
best candidates for deletion based on size and age, and lets the user select 
the ones to delete.



--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] New feature proposition (make.conf)

[Anders Eriksson]

Zack Gilburd wrote:

> One of the problems that plagues Gentoo is that the distro's growth is leading 
> to an insane amount of distfiles downloads (and even excessive downloads) 
> along with excessive rsync'ing.  Automatic removal of distfiles would 
> substancially increase the amount of distfiles downloads due to the fact that 
> packages often have revisions made to them and revisions mean that the 
> software has to be recompiled -- thus the distfile would have to be 
> redownloaded of already removed.
> 
> Until the general downloading (and rsyncing) etiqutte improves, I don't see 
> how this could be The Right Thing(TM) to do.


I for one have the distfiles in /var/cache/ which is the Right Place, 
imho
(and I believe the FHS thinks so too). Along with apache cache files, 
ccache cache files and cached man pages, it amounts to quite some 
data which can be removed when the space is needed. As a matted of 
fact, I have a script which does just that.

How about adopting that poicy in gentoo? The cleaning side of it 
(script, whatever) can be dressed to accept whatever policy the user 
sees fit.

Just my 2c.
/Anders


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] New feature proposition (make.conf)

[Lisa Marie Seelye]

[-- Attachment #1: Type: text/plain, Size: 269 bytes --]

On Thu, 2003-08-28 at 12:06, Anders Eriksson wrote:
> I for one have the distfiles in /var/cache/ which is the Right Place, 

You may set that in make.conf by setting DISTDIR.

-- 
Regards,
-Lisa
<Vix ulla tam iniqua pax, quin bello vel aequissimo sit potior>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]