From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from lists.gentoo.org ([140.105.134.102] helo=robin.gentoo.org) by nuthatch.gentoo.org with esmtp (Exim 4.54) id 1Et9yW-0002mY-0r for garchives@archives.gentoo.org; Sun, 01 Jan 2006 20:38:44 +0000 Received: from robin.gentoo.org (localhost [127.0.0.1]) by robin.gentoo.org (8.13.5/8.13.5) with SMTP id k01KbZjZ022827; Sun, 1 Jan 2006 20:37:35 GMT Received: from mail.pnpitalia.it (85-18-21-122.ip.fastwebnet.it [85.18.21.122]) by robin.gentoo.org (8.13.5/8.13.5) with ESMTP id k01KZaMN026664 for ; Sun, 1 Jan 2006 20:35:36 GMT Received: from localhost (localhost [127.0.0.1]) by mail.pnpitalia.it (Postfix) with ESMTP id 6AD7A7AD139 for ; Sun, 1 Jan 2006 21:35:36 +0100 (CET) Received: from mail.pnpitalia.it ([127.0.0.1]) by localhost (db [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 07367-11 for ; Sun, 1 Jan 2006 21:35:35 +0100 (CET) Received: from [192.168.4.153] (host-4-153.pnpitalia.it [192.168.4.153]) by mail.pnpitalia.it (Postfix) with ESMTP id CBE6D7AD138 for ; Sun, 1 Jan 2006 21:35:35 +0100 (CET) Message-ID: <43B83D16.6060803@gentoo.org> Date: Sun, 01 Jan 2006 21:35:34 +0100 From: Francesco Riosa User-Agent: Mail/News 1.5 (X11/20051227) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@gentoo.org Reply-to: gentoo-dev@lists.gentoo.org MIME-Version: 1.0 To: gentoo-dev@lists.gentoo.org Subject: [gentoo-dev] ChangeLogs and rsync time Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Virus-Scanned: amavisd-new at db X-Archives-Salt: 78b5aaf4-2502-492c-8a38-0f2deb317517 X-Archives-Hash: ad01b947a0a63fe802da36a63e2edfa6 We have currently 10371 ChangeLog files, > 25 MB totally . 1365 == 13% with size >= 4096 B, 12 MB totally rsync from "emerge --sync" has "--whole-file" between its options, that mean transfer the whole file if changed. To make things worse the bigger ChangeLogs are (oh, surprise) those that change frequently (also the age count), following there is the list of first 25: 110062 ./x11-base/xorg-x11/ChangeLog 96906 ./sys-devel/gcc/ChangeLog 86916 ./sys-libs/glibc/ChangeLog 72429 ./net-www/apache/ChangeLog 65331 ./sys-apps/baselayout/ChangeLog 61801 ./media-video/mplayer/ChangeLog 57688 ./dev-db/mysql/ChangeLog 53938 ./sys-kernel/gentoo-sources/ChangeLog 53810 ./net-im/gaim/ChangeLog 53173 ./www-client/mozilla/ChangeLog 51891 ./dev-php/mod_php/ChangeLog 48127 ./dev-db/postgresql/ChangeLog 47014 ./sys-devel/binutils/ChangeLog 46742 ./kde-base/kdelibs/ChangeLog 45370 ./dev-lang/perl/ChangeLog 44998 ./sys-kernel/mm-sources/ChangeLog 41010 ./kde-base/kdebase/ChangeLog 37644 ./www-client/mozilla-firefox/ChangeLog 37524 ./net-fs/samba/ChangeLog 36411 ./mail-mta/postfix/ChangeLog 35269 ./app-office/openoffice-ximian/ChangeLog 34890 ./app-office/openoffice/ChangeLog 34773 ./sys-kernel/mips-sources/ChangeLog 33245 ./media-sound/xmms/ChangeLog 32769 ./dev-util/subversion/ChangeLog The information contained in the ChangeLogs is essential, and it must be kept, but, force the users to download all that data it's not optimal. That said I can see only two ways to reduce the ChangeLog files (a centralized one is obviously not viable) 1) bzip2 them in some way. Pros: - whole story is avaiable - much lower file transfer size Cons: - grepping them need dedicated tools - never ending cvs issues, may be needed an alternate non cvs tree - vim already has a wrapper to read bz2 files, other editors does not - also 247 bytes file will need to be zipped ? 2) "rotate" Changelogs, keeping only the last changes, until a size of 4000 or [choose a preferred size here] bytes. This would save only about 7Mb of data (max size < 4096). Pros: - still easily readable and parseable - save download of data in the right points - affect only 13% of the actual tree Cons: - need changes in repoman/echangelog to cut the ChangeLog in the right position - grepping of ChangeLog impossible (for cutted data) - ChangeLog cutted at _no_ definite point in time (maybe yesterday) - whole history only on viewcvs or with acks like a sys-apps/ChangeLogs package Thoughts ? It's doable in some way ? -- gentoo-dev@gentoo.org mailing list