Re: [gentoo-dev] [pre-GLEP] Split distfile mirror directory structure

["Robin H. Johnson" <robbat2@gentoo.org>]

[-- Attachment #1: Type: text/plain, Size: 1509 bytes --]

On Sun, Jan 28, 2018 at 03:01:11PM +0800, Jason Zaman wrote:
> Another thing im wondering is if we can just use the same dir layout as
> the packages themselves. that would fix texlive since it has a whole lot
> of separate packages. eg /usr/portage/distfiles/app-cat/pkg/pkg-1.0.tgz
Texlive is worse than that:
dev-texlive/texlive-latexextra/Manifest contains 8556 DIST entries, ALL
starting with 'texlive-module-'.

> there is a problem if many packages use the same distfiles (quite
> extensive for SELinux, every single of the sec-policy/selinux-* packages
> has identical distfiles) so im not sure how to deal with it.
The new MetaManifest proposed that common distfiles could be moved to
the category level Manifest (but needs a long transition period).

> do you have an easy way to calculate how big the distfiles are per
> category or cat/pkg? i'd be interested to see.
very quick awk:
per-package, no-dedupe:
gawk '/^DIST/{f=gensub("/Manifest","",1,FILENAME); sum[f]+=$3}END{for(f in sum){print f,sum[f]}}' */*/Manifest

(games-board/tablebase-syzygy is NOT a typo, it has ~150GiB of distfiles)

per-category, no-dedupe:
awk '/^DIST/{f=gensub("/[^/]+/Manifest","",1,FILENAME); sum[f]+=$3}END{for(f in sum){print f,sum[f]}}' */*/Manifest |sort -k +2n

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail   : robbat2@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 1113 bytes --]