[gentoo-dev] digest-* files in portage tree

[gentoo-dev] digest-* files in portage tree

[Francesco Riosa]

In addition to "Manifest" files in portage tree there are 19000 
"digest-${P}" files in the portage tree each of this uses 4k on ext{2,3} 
filesystem and has to be scanned each time rsync is run on the tree.
Is possible to avoid the use of theese files?

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] digest-* files in portage tree

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 545 bytes --]

On Thu, 2005-03-24 at 14:22 +0100, Francesco Riosa wrote:
> In addition to "Manifest" files in portage tree there are 19000 
> "digest-${P}" files in the portage tree each of this uses 4k on ext{2,3} 
> filesystem and has to be scanned each time rsync is run on the tree.
> Is possible to avoid the use of theese files?

Not if you ever want to ensure that what you're downloading hasn't been
trojaned or otherwise tampered with.

-- 
Chris Gianelloni
Release Engineering - Strategic Lead/QA Manager
Games - Developer
Gentoo Linux

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] digest-* files in portage tree

[Francesco Riosa]

Chris Gianelloni ha scritto:

>On Thu, 2005-03-24 at 14:22 +0100, Francesco Riosa wrote:
>  
>
>>In addition to "Manifest" files in portage tree there are 19000 
>>"digest-${P}" files in the portage tree each of this uses 4k on ext{2,3} 
>>filesystem and has to be scanned each time rsync is run on the tree.
>>Is possible to avoid the use of theese files?
>>    
>>
>
>Not if you ever want to ensure that what you're downloading hasn't been
>trojaned or otherwise tampered with.
>
>  
>
ok, this is true only when you have FEATURES="strict" or "cvs" that it's 
not the default.
I'm interested in understund better the way it work, have you any 
pointer (inside the code or some doc) ?

thanks in advance

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] digest-* files in portage tree

[Mike Frysinger]

On Thursday 24 March 2005 09:16 am, Francesco Riosa wrote:
> Chris Gianelloni ha scritto:
> >On Thu, 2005-03-24 at 14:22 +0100, Francesco Riosa wrote:
> >>In addition to "Manifest" files in portage tree there are 19000
> >>"digest-${P}" files in the portage tree each of this uses 4k on ext{2,3}
> >>filesystem and has to be scanned each time rsync is run on the tree.
> >>Is possible to avoid the use of theese files?
> >
> >Not if you ever want to ensure that what you're downloading hasn't been
> >trojaned or otherwise tampered with.
>
> ok, this is true only when you have FEATURES="strict" or "cvs" that it's
> not the default.

no it's not

Manifest is checked when you have FEATURES=strict, digest is checked everytime 
you fetch/unpack files ... all those little 'src_uri md5' lines is a check 
against the digest
-mike
--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] digest-* files in portage tree

[Francesco Riosa]

Mike Frysinger ha scritto:

>On Thursday 24 March 2005 09:16 am, Francesco Riosa wrote:
>  
>
>>Chris Gianelloni ha scritto:
>>    
>>
>>>On Thu, 2005-03-24 at 14:22 +0100, Francesco Riosa wrote:
>>>      
>>>
>>>>In addition to "Manifest" files in portage tree there are 19000
>>>>"digest-${P}" files in the portage tree each of this uses 4k on ext{2,3}
>>>>filesystem and has to be scanned each time rsync is run on the tree.
>>>>Is possible to avoid the use of theese files?
>>>>        
>>>>
>>>Not if you ever want to ensure that what you're downloading hasn't been
>>>trojaned or otherwise tampered with.
>>>      
>>>
>>ok, this is true only when you have FEATURES="strict" or "cvs" that it's
>>not the default.
>>    
>>
>
>no it's not
>
>Manifest is checked when you have FEATURES=strict, digest is checked everytime 
>you fetch/unpack files ... all those little 'src_uri md5' lines is a check 
>against the digest
>-mike
>  
>
ah ok, I was confused by the fact that I can change patches and ebuild 
in my *overlays* without the need of a "ebuild P digest"
already found "digestCheckFiles" & "digestcheck" in portage.py so 
pointers not needed anymore.

thanks
--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] digest-* files in portage tree

[Aron Griffis]

[-- Attachment #1: Type: text/plain, Size: 482 bytes --]

Francesco Riosa wrote:	[Thu Mar 24 2005, 08:22:40AM EST]
> In addition to "Manifest" files in portage tree there are 19000
> "digest-${P}" files in the portage tree each of this uses 4k on
> ext{2,3} filesystem and has to be scanned each time rsync is run on
> the tree.  Is possible to avoid the use of theese files?

A few mods to portage and it should be possible to give the Manifest
a structure to contain the digests...

Regards,
Aron

--
Aron Griffis
Gentoo Linux Developer


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] digest-* files in portage tree

[Marius Mauch]

[-- Attachment #1: Type: text/plain, Size: 855 bytes --]

On Thu, 24 Mar 2005 10:42:05 -0500
Aron Griffis <agriffis@gentoo.org> wrote:

> Francesco Riosa wrote:	[Thu Mar 24 2005, 08:22:40AM EST]
> > In addition to "Manifest" files in portage tree there are 19000
> > "digest-${P}" files in the portage tree each of this uses 4k on
> > ext{2,3} filesystem and has to be scanned each time rsync is run on
> > the tree.  Is possible to avoid the use of theese files?
> 
> A few mods to portage and it should be possible to give the Manifest
> a structure to contain the digests...

Yeah, I might reconsider my digest-redesign I sent here a few months ago
if there is actually interest in this.

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] digest-* files in portage tree

[Daniel]

[-- Attachment #1: Type: text/plain, Size: 309 bytes --]


> Yeah, I might reconsider my digest-redesign I sent here a few months ago
> if there is actually interest in this.

I think its a great idea to minimise the about of metadata associated with 
each package.

-- 
Daniel Black <dragonheart@gentoo.org>
Gentoo Crypto/PPC/dev-embedded/Forensics/NetMon

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] digest-* files in portage tree

[Daniel]

[-- Attachment #1: Type: text/plain, Size: 494 bytes --]

On Fri, 25 Mar 2005 01:58 pm, Daniel wrote:
> > Yeah, I might reconsider my digest-redesign I sent here a few months ago
> > if there is actually interest in this.
>
> I think its a great idea to minimise the about of metadata associated with
> each package.

I probably should of been clearer here. "Minimise the number files for the 
amount of metadata" is what I ment to say.

-- 
Daniel Black <dragonheart@gentoo.org>
Gentoo Crypto/PPC/dev-embedded/Forensics/NetMon/app-backup

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] digest-* files in portage tree

[Peter Cech]

On Fri, Mar 25, 2005 at 02:46:05AM +0100, Marius Mauch wrote:
> On Thu, 24 Mar 2005 10:42:05 -0500
> Aron Griffis <agriffis@gentoo.org> wrote:
> 
> > A few mods to portage and it should be possible to give the Manifest
> > a structure to contain the digests...
> 
> Yeah, I might reconsider my digest-redesign I sent here a few months ago
> if there is actually interest in this.

Please do. Less files in portage tree -> faster emerge sync and less
disk space consumed by portage tree. I don't think anybody will object
to this.

Regards,
Peter Cech
--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] digest-* files in portage tree

[Daniel Goller]

[-- Attachment #1: Type: text/plain, Size: 827 bytes --]

Francesco Riosa wrote:
> In addition to "Manifest" files in portage tree there are 19000
> "digest-${P}" files in the portage tree each of this uses 4k on ext{2,3}
> filesystem and has to be scanned each time rsync is run on the tree.
> Is possible to avoid the use of theese files?
>
> --
> gentoo-dev@gentoo.org mailing list
>
>
while not technically what you asked for, keeping the tree on a
partition with a smaller than default block and inode size would allow
you to decrease wasted space significantly, ext3 does 1024 in linux
iirc, docs suggest 512 should be possible, but i have not found a fs on
linux that did indeed allow actual use of 512byte blocks/inodes (yes you
want to make sure to adjust both, not just one)

hope this helps while people consider integrating them elsewhere to safe
the most space possible



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 256 bytes --]

Re: [gentoo-dev] digest-* files in portage tree

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 1431 bytes --]

On Fri, Mar 25, 2005 at 12:04:41AM -0600, Daniel Goller wrote:
> Francesco Riosa wrote:
> >In addition to "Manifest" files in portage tree there are 19000
> >"digest-${P}" files in the portage tree each of this uses 4k on ext{2,3}
> >filesystem and has to be scanned each time rsync is run on the tree.
> >Is possible to avoid the use of theese files?
> while not technically what you asked for, keeping the tree on a
> partition with a smaller than default block and inode size would allow
> you to decrease wasted space significantly, ext3 does 1024 in linux
> iirc, docs suggest 512 should be possible, but i have not found a fs on
> linux that did indeed allow actual use of 512byte blocks/inodes (yes you
> want to make sure to adjust both, not just one)
> 
> hope this helps while people consider integrating them elsewhere to safe
> the most space possible
As a workaround, I have my CVS checkout of the tree on a reiser3
partition with tail-packing enabled. This really helps, esp. as CVS has
4 files of overhead for per working directory. I'm not aware of any
other tail-packing read-write filesystems unfortunately. (I know that
read-only ones like squashfs etc. exist).

-- 
Robin Hugh Johnson
E-Mail     : robbat2@orbis-terrarum.net
Home Page  : http://www.orbis-terrarum.net/?l=people.robbat2
ICQ#       : 30269588 or 41961639
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 241 bytes --]

[gentoo-dev] Re: digest-* files in portage tree

[Duncan]

Robin H. Johnson posted
<20050325071250.GA11908@curie-int.orbis-terrarum.net>, excerpted below, 
on Thu, 24 Mar 2005 23:12:50 -0800:

> As a workaround, I have my CVS checkout of the tree on a reiser3 partition
> with tail-packing enabled. This really helps, esp. as CVS has 4 files of
> overhead for per working directory. I'm not aware of any other
> tail-packing read-write filesystems unfortunately. (I know that read-only
> ones like squashfs etc. exist).

I was just going to mention reiserfs, which I use exclusively (in
data=ordered mode).  I /believe/ one of the other journalled fs, either
jfs or <forgot>, does something similar, using large blocks but packing
files into them irrespective of size.  I read that somewhere and noted it
with interest, but reiserfs has been sufficient for me, particularly after
it got data=ordered (and now data=journal, tho ordered is sufficient for
me) support.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman in
http://www.linuxdevcenter.com/pub/a/linux/2004/12/22/rms_interview.html


--
gentoo-dev@gentoo.org mailing list