[gentoo-user] PORTAGE_COMPRESS

[gentoo-user] PORTAGE_COMPRESS

[Douglas J Hunley]

[-- Attachment #1: Type: text/plain, Size: 710 bytes --]

I was reviewing my Portage settings yesterday and I noticed that I have
PORTAGE_COMPRESS set (to bzip2, the default) on both of my servers and it
occurred to me that both of these servers have filesystems that support
compression (btrfs on one, zfs on the other). So I'm wondering if it still
makes sense to have PORTAGE_COMPRESS set or if I should unset it and just
let the fs-level compression handle it.  Portage is already slow, why have
it take the time to do this when the fs does it better and
transparently? Thoughts on the matter?

--
Douglas J Hunley (doug.hunley@gmail.com)
Twitter: @hunleyd                                               Web:
douglasjhunley.com
G+: http://google.com/+DouglasHunley

[-- Attachment #2: Type: text/html, Size: 916 bytes --]

Re: [gentoo-user] PORTAGE_COMPRESS

[Alan McKinnon]

On 02/04/2014 14:27, Douglas J Hunley wrote:
> I was reviewing my Portage settings yesterday and I noticed that I have
> PORTAGE_COMPRESS set (to bzip2, the default) on both of my servers and
> it occurred to me that both of these servers have filesystems that
> support compression (btrfs on one, zfs on the other). So I'm wondering
> if it still makes sense to have PORTAGE_COMPRESS set or if I should
> unset it and just let the fs-level compression handle it.  Portage is
> already slow, why have it take the time to do this when the fs does it
> better and transparently? Thoughts on the matter?


I agree with your reasoning.

PORTAGE_COMPRESS is an ugly hack to get doc files smaller and the need
for it has long since gone away for the general case and it predates
filesystems with compression anyway. So do let the fs deal with this
transparently and avoid the cost of calling bunzip2 everytime you read a
doc.



-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] PORTAGE_COMPRESS

[fruktopus]

Am 02.04.2014 16:10, schrieb Alan McKinnon:
> On 02/04/2014 14:27, Douglas J Hunley wrote:
>> I was reviewing my Portage settings yesterday and I noticed that I have
>> PORTAGE_COMPRESS set (to bzip2, the default) on both of my servers and
>> it occurred to me that both of these servers have filesystems that
>> support compression (btrfs on one, zfs on the other). So I'm wondering
>> if it still makes sense to have PORTAGE_COMPRESS set or if I should
>> unset it and just let the fs-level compression handle it.  Portage is
>> already slow, why have it take the time to do this when the fs does it
>> better and transparently? Thoughts on the matter?
> I agree with your reasoning.
>
> PORTAGE_COMPRESS is an ugly hack to get doc files smaller and the need
> for it has long since gone away for the general case and it predates
> filesystems with compression anyway. So do let the fs deal with this
> transparently and avoid the cost of calling bunzip2 everytime you read a
> doc.

Where did you find PORTAGE_COMPRESS? I just checked /etc/portage and
some other locations but without luck. Also its not documented. Can you
give me a hint.
 
 ~frukto

Re: [gentoo-user] PORTAGE_COMPRESS

[Alan McKinnon]

On 02/04/2014 18:48, fruktopus wrote:
> Am 02.04.2014 16:10, schrieb Alan McKinnon:
>> On 02/04/2014 14:27, Douglas J Hunley wrote:
>>> I was reviewing my Portage settings yesterday and I noticed that I have
>>> PORTAGE_COMPRESS set (to bzip2, the default) on both of my servers and
>>> it occurred to me that both of these servers have filesystems that
>>> support compression (btrfs on one, zfs on the other). So I'm wondering
>>> if it still makes sense to have PORTAGE_COMPRESS set or if I should
>>> unset it and just let the fs-level compression handle it.  Portage is
>>> already slow, why have it take the time to do this when the fs does it
>>> better and transparently? Thoughts on the matter?
>> I agree with your reasoning.
>>
>> PORTAGE_COMPRESS is an ugly hack to get doc files smaller and the need
>> for it has long since gone away for the general case and it predates
>> filesystems with compression anyway. So do let the fs deal with this
>> transparently and avoid the cost of calling bunzip2 everytime you read a
>> doc.
> 
> Where did you find PORTAGE_COMPRESS? I just checked /etc/portage and
> some other locations but without luck. Also its not documented. Can you
> give me a hint.

man 5 make.conf

-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] PORTAGE_COMPRESS

[fruktopus]

Am 02.04.2014 20:29, schrieb Alan McKinnon:
> On 02/04/2014 18:48, fruktopus wrote:
>> Am 02.04.2014 16:10, schrieb Alan McKinnon:
>>> On 02/04/2014 14:27, Douglas J Hunley wrote:
>>>> I was reviewing my Portage settings yesterday and I noticed that I have
>>>> PORTAGE_COMPRESS set (to bzip2, the default) on both of my servers and
>>>> it occurred to me that both of these servers have filesystems that
>>>> support compression (btrfs on one, zfs on the other). So I'm wondering
>>>> if it still makes sense to have PORTAGE_COMPRESS set or if I should
>>>> unset it and just let the fs-level compression handle it.  Portage is
>>>> already slow, why have it take the time to do this when the fs does it
>>>> better and transparently? Thoughts on the matter?
>>> I agree with your reasoning.
>>>
>>> PORTAGE_COMPRESS is an ugly hack to get doc files smaller and the need
>>> for it has long since gone away for the general case and it predates
>>> filesystems with compression anyway. So do let the fs deal with this
>>> transparently and avoid the cost of calling bunzip2 everytime you read a
>>> doc.
>> Where did you find PORTAGE_COMPRESS? I just checked /etc/portage and
>> some other locations but without luck. Also its not documented. Can you
>> give me a hint.
> man 5 make.conf
>
Thank you.

Re: [gentoo-user] PORTAGE_COMPRESS

[Douglas J Hunley]

[-- Attachment #1: Type: text/plain, Size: 412 bytes --]

On Wed, Apr 2, 2014 at 2:29 PM, Alan McKinnon <alan.mckinnon@gmail.com>wrote:

> man 5 make.conf


What isn't listed in the man page is if you should simply unset it:
PORTAGE_COMPRESS=""
or set it to something like /bin/true

Any thoughts?


-- 
Douglas J Hunley (doug.hunley@gmail.com)
Twitter: @hunleyd                                               Web:
douglasjhunley.com
G+: http://google.com/+DouglasHunley

[-- Attachment #2: Type: text/html, Size: 1118 bytes --]

Re: [gentoo-user] PORTAGE_COMPRESS

[null_ptr]

On 02/04/14 16:02, Douglas J Hunley wrote:
>On Wed, Apr 2, 2014 at 2:29 PM, Alan McKinnon <alan.mckinnon@gmail.com>wrote:
>
>> man 5 make.conf
>
>
>What isn't listed in the man page is if you should simply unset it:
>PORTAGE_COMPRESS=""
>or set it to something like /bin/true
>
>Any thoughts?

How about setting PORTAGE_COMPRESS_EXCLUDE_SUFFIXES=".*" ?

Re: [gentoo-user] PORTAGE_COMPRESS

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 427 bytes --]

On Wed, 2 Apr 2014 22:10:41 +0200, null_ptr wrote:

> >What isn't listed in the man page is if you should simply unset it:
> >PORTAGE_COMPRESS=""
> >or set it to something like /bin/true
> 
> How about setting PORTAGE_COMPRESS_EXCLUDE_SUFFIXES=".*" ?

That would still compress files without an extension, like README, NEWS
etc.


-- 
Neil Bothwick

The original point and click interface was a Smith & Wesson.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] PORTAGE_COMPRESS

[fruktopus]

[-- Attachment #1: Type: text/plain, Size: 913 bytes --]

Am 02.04.2014 22:02, schrieb Douglas J Hunley:
>
> On Wed, Apr 2, 2014 at 2:29 PM, Alan McKinnon <alan.mckinnon@gmail.com
> <mailto:alan.mckinnon@gmail.com>> wrote:
>
>     man 5 make.conf
>
>
> What isn't listed in the man page is if you should simply unset it:
> PORTAGE_COMPRESS=""
> or set it to something like /bin/true
>
> Any thoughts?

I just checked portages git repository. The interesting file is
bin/ebuild-helpers/ecompress [1]

There we have:

    12 # setup compression stuff
    13 PORTAGE_COMPRESS=${PORTAGE_COMPRESS-bzip2}
    14 [[ -z ${PORTAGE_COMPRESS} ]] && exit 0

If PORTAGE_COMPRESS is not declared it defaults to bzip2. If it is
declared, but without value or empty the script exits. So a

PORTAGE_COMPRESS=""

should be fine.

 ~frukto

----
[1]
http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=blob;f=bin/ebuild-helpers/ecompress;hb=d8345b546c4212d928b38b972921879ef9fa466c

[-- Attachment #2: Type: text/html, Size: 2230 bytes --]

Re: [gentoo-user] PORTAGE_COMPRESS

[Mike Gilbert]

On Wed, Apr 2, 2014 at 4:02 PM, Douglas J Hunley <doug.hunley@gmail.com> wrote:
>
> On Wed, Apr 2, 2014 at 2:29 PM, Alan McKinnon <alan.mckinnon@gmail.com>
> wrote:
>>
>> man 5 make.conf
>
>
> What isn't listed in the man page is if you should simply unset it:
> PORTAGE_COMPRESS=""
> or set it to something like /bin/true
>
> Any thoughts?
>

From /usr/lib/portage/bin/ebuild-helpers/ecompress:

# setup compression stuff
PORTAGE_COMPRESS=${PORTAGE_COMPRESS-bzip2}
[[ -z ${PORTAGE_COMPRESS} ]] && exit 0

So, to disable it, just set PORTAGE_COMPRESS to an empty string:

PORTAGE_COMPRESS=""

Re: [gentoo-user] PORTAGE_COMPRESS

[Daniel Troeder]

[-- Attachment #1: Type: text/plain, Size: 2099 bytes --]

Am 02.04.2014 14:27, schrieb Douglas J Hunley:
> I was reviewing my Portage settings yesterday and I noticed that I have
> PORTAGE_COMPRESS set (to bzip2, the default) on both of my servers and
> it occurred to me that both of these servers have filesystems that
> support compression (btrfs on one, zfs on the other). So I'm wondering
> if it still makes sense to have PORTAGE_COMPRESS set or if I should
> unset it and just let the fs-level compression handle it.  Portage is
> already slow, why have it take the time to do this when the fs does it
> better and transparently? Thoughts on the matter?
I think you have a good point disabling portage compression, but I
wanted to correct one statement: Decompression in the kernel takes as
much time as decompression in userspace - if it's the same algorithm.

The differences here are:
* creation of an additional process (bz[ip2, cat]), negligible compared
to the cpu time compression takes
* bz[ip2, cat] runs in user space (safer) vs in kernel space ("risky")
* different [de]compression algorithm (fs usually uses weaker but faster
compression algorithms like lz4, lzo or gz)
* different [de]compression setting (portage uses -9, kernel at best -6
- the difference in the resulting file size is small, but the difference
in used cpu time is high)

For me two things decide the question:
* Assuming you write more man pages than you read, and disk space is not
an issue, a weaker compression ratio to save cpu time makes sense.
* If you have compression _already_ on for the fs, then compressing
textual data before storing it on that fs is a waist of cpu time:

For example LZ4 will detect already compressed data (compression ratio
<12.5%), abort compression and store the file uncompressed. That's nice,
but it means that it has already compressed some blocks, before it could
measure it - waisting cpu time.

Greetings,
Daniel


-- 
Get my PGP key at:
*
http://keyserver.ubuntu.com:11371/pks/lookup?op=get&search=0x837FB8B5BB9D4887
* $ gpg --recv-keys --keyserver keyserver.ubuntu.com 0xBB9D4887


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 901 bytes --]