public inbox for gentoo-dev@lists.gentoo.org
 help / color / mirror / Atom feed
* [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2
@ 2012-09-26 20:30 Michael Mol
  2012-09-26 20:43 ` Matt Turner
                   ` (2 more replies)
  0 siblings, 3 replies; 22+ messages in thread
From: Michael Mol @ 2012-09-26 20:30 UTC (permalink / raw
  To: gentoo-dev

A few months ago, I filed bug 423651 to ask that bzip2 on the install
media be replaced with
 pbzip2. It was closed a short while later, telling me that it'd
involve changing what's kept in @system, and that had to be discussed
here, rather than in a bug report.

Here's a detailed description of how pbzip2 operates, as described by
a friend of mine:

> pbzip2's compression routine splits the input into blocks (with a default of 900,000
> bytes), which it then feeds into the standard bzip2 compression routine. The output
> of the various calls to the bzip2 compression routine are then concatenated together.
> The end result is the same as if you had first used the "split" command on the input,
> run individual bzip2 commands on the split pieces, then recombined the individual
> bz2 files using cat.
>
> The down side to this is that you have multiple file headers, footers, and byte-align
> padding, plus the fact that bzip2 does a RLE compression stage to fill the buffer it
> feeds to the BWT, the main part of the compression routine. If you happen to have a
> section with 1MiB of the same byte, the pbzip2 front-end will split that into two blocks
> (at the default settings) and feed them to separate bzip2 compressors. bzip2 will
> then compress the first block down to a buffer of about 17kiB before passing it on
> to be compressed further, and the rest of the data would have fit within this block, if
> pbzip2 hadn't split it the way it had.
>
> As for decompression, pbzip2 can only really do parallel decompression of files that it
> created, since it seeks for the bz2 file header in order to split it to different workers. One
> reason for this is that the bz2 block header is not byte aligned.

I really don't know how to carry this discussion any further than
this; I'll answer any questions I can.

-- 
:wq


^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2
  2012-09-26 20:30 [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2 Michael Mol
@ 2012-09-26 20:43 ` Matt Turner
  2012-09-26 21:27   ` Florian Philipp
  2012-09-27  9:23   ` Piotr Szymaniak
  2012-09-26 21:49 ` Chí-Thanh Christopher Nguyễn
  2012-09-27  9:48 ` Ulrich Mueller
  2 siblings, 2 replies; 22+ messages in thread
From: Matt Turner @ 2012-09-26 20:43 UTC (permalink / raw
  To: gentoo-dev

On Wed, Sep 26, 2012 at 1:30 PM, Michael Mol <mikemol@gmail.com> wrote:
> A few months ago, I filed bug 423651 to ask that bzip2 on the install
> media be replaced with
>  pbzip2. It was closed a short while later, telling me that it'd
> involve changing what's kept in @system, and that had to be discussed
> here, rather than in a bug report.

If we're going to ship a parallel bzip2 implementation, it should be
lbzip2 and not pbzip2.

lbzip2 can decompress bz2 archives with multiple threads that haven't
been compressed with lbzip2/pbzip2.


^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2
  2012-09-26 20:43 ` Matt Turner
@ 2012-09-26 21:27   ` Florian Philipp
  2012-09-26 21:53     ` Michael Mol
  2012-09-27  9:23   ` Piotr Szymaniak
  1 sibling, 1 reply; 22+ messages in thread
From: Florian Philipp @ 2012-09-26 21:27 UTC (permalink / raw
  To: gentoo-dev

[-- Attachment #1: Type: text/plain, Size: 951 bytes --]

Am 26.09.2012 22:43, schrieb Matt Turner:
> On Wed, Sep 26, 2012 at 1:30 PM, Michael Mol <mikemol@gmail.com> wrote:
>> A few months ago, I filed bug 423651 to ask that bzip2 on the install
>> media be replaced with
>>  pbzip2. It was closed a short while later, telling me that it'd
>> involve changing what's kept in @system, and that had to be discussed
>> here, rather than in a bug report.
> 
> If we're going to ship a parallel bzip2 implementation, it should be
> lbzip2 and not pbzip2.
> 
> lbzip2 can decompress bz2 archives with multiple threads that haven't
> been compressed with lbzip2/pbzip2.
> 

This seems relevant, especially comment 12ff:
https://bugs.gentoo.org/show_bug.cgi?id=309683

For further anecdotal evidence: I've used pbzip2 with USE="symlink" for
several months now and never had trouble with it. Checking out lbzip2
now. I noticed it doesn't install a bunzip2 symlink.

Regards,
Florian Philipp


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2
  2012-09-26 20:30 [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2 Michael Mol
  2012-09-26 20:43 ` Matt Turner
@ 2012-09-26 21:49 ` Chí-Thanh Christopher Nguyễn
  2012-09-26 21:59   ` Michael Mol
  2012-09-27  8:58   ` Tobias Klausmann
  2012-09-27  9:48 ` Ulrich Mueller
  2 siblings, 2 replies; 22+ messages in thread
From: Chí-Thanh Christopher Nguyễn @ 2012-09-26 21:49 UTC (permalink / raw
  To: gentoo-dev

Michael Mol schrieb:
> A few months ago, I filed bug 423651 to ask that bzip2 on the install
> media be replaced with
>  pbzip2.

If I understand correctly, pbzip2 depends on bzip2. So what you are
asking is that pbzip2 is preferred over bzip2 when both are installed,
and that pbzip2 is installed by default?

I have so far encountered only one anecdotal case in #gentoo IRC where
pbzip2[symlink] caused problems in emerging a package. Disabling the
symlink flag made the problem go away. However I can't point to the
report right now, maybe someone with searchable backlog can uncover it.

A different question is whether in the cases where parallel bzip2 makes
sense, is it really the best solution? xz is outperforming bzip2's
compression ratio for large files (for an informal comparison, see bug
434350). And xz is faster at decompression, which offsets the parallel
advantage to some degree.


Best regards,
Chí-Thanh Christopher Nguyễn



^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2
  2012-09-26 21:27   ` Florian Philipp
@ 2012-09-26 21:53     ` Michael Mol
  2012-09-27  7:22       ` Florian Philipp
  0 siblings, 1 reply; 22+ messages in thread
From: Michael Mol @ 2012-09-26 21:53 UTC (permalink / raw
  To: gentoo-dev

On Wed, Sep 26, 2012 at 5:27 PM, Florian Philipp <lists@binarywings.net> wrote:
> Am 26.09.2012 22:43, schrieb Matt Turner:
>> On Wed, Sep 26, 2012 at 1:30 PM, Michael Mol <mikemol@gmail.com> wrote:
>>> A few months ago, I filed bug 423651 to ask that bzip2 on the install
>>> media be replaced with
>>>  pbzip2. It was closed a short while later, telling me that it'd
>>> involve changing what's kept in @system, and that had to be discussed
>>> here, rather than in a bug report.
>>
>> If we're going to ship a parallel bzip2 implementation, it should be
>> lbzip2 and not pbzip2.
>>
>> lbzip2 can decompress bz2 archives with multiple threads that haven't
>> been compressed with lbzip2/pbzip2.
>>
>
> This seems relevant, especially comment 12ff:
> https://bugs.gentoo.org/show_bug.cgi?id=309683
>
> For further anecdotal evidence: I've used pbzip2 with USE="symlink" for
> several months now and never had trouble with it. Checking out lbzip2
> now. I noticed it doesn't install a bunzip2 symlink.

Piotr Szymaniak asked me about lbzip2, and I bounced the question over
to my friend. He didn't investigate it deeply; it crashed (OOM or
something else, I don't know) when he tried it on a large file. Could
have been from 2GB to 2TB, from what he has laying around. I don't
know; I didn't get that one in writing. :)

But if it proves to be stable for small and very large files, I'd have
no complaint. :)

-- 
:wq


^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2
  2012-09-26 21:49 ` Chí-Thanh Christopher Nguyễn
@ 2012-09-26 21:59   ` Michael Mol
  2012-09-26 22:31     ` Mike Gilbert
  2012-09-27  8:58   ` Tobias Klausmann
  1 sibling, 1 reply; 22+ messages in thread
From: Michael Mol @ 2012-09-26 21:59 UTC (permalink / raw
  To: gentoo-dev

On Wed, Sep 26, 2012 at 5:49 PM, Chí-Thanh Christopher Nguyễn
<chithanh@gentoo.org> wrote:
> Michael Mol schrieb:
>> A few months ago, I filed bug 423651 to ask that bzip2 on the install
>> media be replaced with
>>  pbzip2.
>
> If I understand correctly, pbzip2 depends on bzip2. So what you are
> asking is that pbzip2 is preferred over bzip2 when both are installed,
> and that pbzip2 is installed by default?

pbzip2 uses libbzip2, which I understand bzip2 to also be a wrapper around.

>
> I have so far encountered only one anecdotal case in #gentoo IRC where
> pbzip2[symlink] caused problems in emerging a package. Disabling the
> symlink flag made the problem go away. However I can't point to the
> report right now, maybe someone with searchable backlog can uncover it.

pbzip2[symlink] is more or less the scenario I'd like to see.

>
> A different question is whether in the cases where parallel bzip2 makes
> sense, is it really the best solution? xz is outperforming bzip2's
> compression ratio for large files (for an informal comparison, see bug
> 434350). And xz is faster at decompression, which offsets the parallel
> advantage to some degree.

xz is faster for decompression, by my inspiration case was during
system installation, so compression. Last I looked, xz was still very
slow for threaded compression. And it's not block-oriented, so I don't
think that's really possible without loss of compression efficiency,
anyway...and my use cases range from 4-8 physical cores.

-- 
:wq


^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2
  2012-09-26 21:59   ` Michael Mol
@ 2012-09-26 22:31     ` Mike Gilbert
  2012-09-26 22:57       ` Christoph Junghans
  0 siblings, 1 reply; 22+ messages in thread
From: Mike Gilbert @ 2012-09-26 22:31 UTC (permalink / raw
  To: gentoo-dev

On Wed, Sep 26, 2012 at 5:59 PM, Michael Mol <mikemol@gmail.com> wrote:
> On Wed, Sep 26, 2012 at 5:49 PM, Chí-Thanh Christopher Nguyễn
> <chithanh@gentoo.org> wrote:
>> Michael Mol schrieb:
>>> A few months ago, I filed bug 423651 to ask that bzip2 on the install
>>> media be replaced with
>>>  pbzip2.
>>
>> If I understand correctly, pbzip2 depends on bzip2. So what you are
>> asking is that pbzip2 is preferred over bzip2 when both are installed,
>> and that pbzip2 is installed by default?
>
> pbzip2 uses libbzip2, which I understand bzip2 to also be a wrapper around.
>

libbz2 is built and installed by the app-arch/bzip2 package. Thus,
app-arch/pbzip2 depends on app-arch/bzip2, unless someone rips libbz2
out into a separate ebuild.


^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2
  2012-09-26 22:31     ` Mike Gilbert
@ 2012-09-26 22:57       ` Christoph Junghans
  2012-09-26 23:57         ` Michael Mol
  2012-09-27  0:55         ` Diego Elio Pettenò
  0 siblings, 2 replies; 22+ messages in thread
From: Christoph Junghans @ 2012-09-26 22:57 UTC (permalink / raw
  To: gentoo-dev

2012/9/26 Mike Gilbert <floppym@gentoo.org>:
> On Wed, Sep 26, 2012 at 5:59 PM, Michael Mol <mikemol@gmail.com> wrote:
>> On Wed, Sep 26, 2012 at 5:49 PM, Chí-Thanh Christopher Nguyễn
>> <chithanh@gentoo.org> wrote:
>>> Michael Mol schrieb:
>>>> A few months ago, I filed bug 423651 to ask that bzip2 on the install
>>>> media be replaced with
>>>>  pbzip2.
>>>
>>> If I understand correctly, pbzip2 depends on bzip2. So what you are
>>> asking is that pbzip2 is preferred over bzip2 when both are installed,
>>> and that pbzip2 is installed by default?
>>
>> pbzip2 uses libbzip2, which I understand bzip2 to also be a wrapper around.
>>
>
> libbz2 is built and installed by the app-arch/bzip2 package. Thus,
> app-arch/pbzip2 depends on app-arch/bzip2, unless someone rips libbz2
> out into a separate ebuild.
That sound like a plan. Maybe bzip2 should become a virtual as busybox
also provides an implementation.



-- 
Christoph Junghans
http://dev.gentoo.org/~ottxor/


^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2
  2012-09-26 22:57       ` Christoph Junghans
@ 2012-09-26 23:57         ` Michael Mol
  2012-09-27  0:55         ` Diego Elio Pettenò
  1 sibling, 0 replies; 22+ messages in thread
From: Michael Mol @ 2012-09-26 23:57 UTC (permalink / raw
  To: gentoo-dev

On Wed, Sep 26, 2012 at 6:57 PM, Christoph Junghans <ottxor@gentoo.org> wrote:
> 2012/9/26 Mike Gilbert <floppym@gentoo.org>:
>> On Wed, Sep 26, 2012 at 5:59 PM, Michael Mol <mikemol@gmail.com> wrote:
>>> On Wed, Sep 26, 2012 at 5:49 PM, Chí-Thanh Christopher Nguyễn
>>> <chithanh@gentoo.org> wrote:
>>>> Michael Mol schrieb:
>>>>> A few months ago, I filed bug 423651 to ask that bzip2 on the install
>>>>> media be replaced with
>>>>>  pbzip2.
>>>>
>>>> If I understand correctly, pbzip2 depends on bzip2. So what you are
>>>> asking is that pbzip2 is preferred over bzip2 when both are installed,
>>>> and that pbzip2 is installed by default?
>>>
>>> pbzip2 uses libbzip2, which I understand bzip2 to also be a wrapper around.
>>>
>>
>> libbz2 is built and installed by the app-arch/bzip2 package. Thus,
>> app-arch/pbzip2 depends on app-arch/bzip2, unless someone rips libbz2
>> out into a separate ebuild.
> That sound like a plan. Maybe bzip2 should become a virtual as busybox
> also provides an implementation.

This makes sense. And going back to my initial issue, I don't really
care which implementation gets used on the bootable media, so long as
it supports scaling to use my CPU cores.

-- 
:wq


^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2
  2012-09-26 22:57       ` Christoph Junghans
  2012-09-26 23:57         ` Michael Mol
@ 2012-09-27  0:55         ` Diego Elio Pettenò
  1 sibling, 0 replies; 22+ messages in thread
From: Diego Elio Pettenò @ 2012-09-27  0:55 UTC (permalink / raw
  To: gentoo-dev

On 26/09/2012 15:57, Christoph Junghans wrote:
> That sound like a plan. Maybe bzip2 should become a virtual as busybox
> also provides an implementation.

No, just, no.

-- 
Diego Elio Pettenò — Flameeyes
flameeyes@flameeyes.eu — http://blog.flameeyes.eu/


^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2
  2012-09-26 21:53     ` Michael Mol
@ 2012-09-27  7:22       ` Florian Philipp
  2012-09-27 15:09         ` Florian Philipp
  0 siblings, 1 reply; 22+ messages in thread
From: Florian Philipp @ 2012-09-27  7:22 UTC (permalink / raw
  To: gentoo-dev

[-- Attachment #1: Type: text/plain, Size: 1800 bytes --]

Am 26.09.2012 23:53, schrieb Michael Mol:
> On Wed, Sep 26, 2012 at 5:27 PM, Florian Philipp <lists@binarywings.net> wrote:
>> Am 26.09.2012 22:43, schrieb Matt Turner:
>>> On Wed, Sep 26, 2012 at 1:30 PM, Michael Mol <mikemol@gmail.com> wrote:
>>>> A few months ago, I filed bug 423651 to ask that bzip2 on the install
>>>> media be replaced with
>>>>  pbzip2. It was closed a short while later, telling me that it'd
>>>> involve changing what's kept in @system, and that had to be discussed
>>>> here, rather than in a bug report.
>>>
>>> If we're going to ship a parallel bzip2 implementation, it should be
>>> lbzip2 and not pbzip2.
>>>
>>> lbzip2 can decompress bz2 archives with multiple threads that haven't
>>> been compressed with lbzip2/pbzip2.
>>>
>>
>> This seems relevant, especially comment 12ff:
>> https://bugs.gentoo.org/show_bug.cgi?id=309683
>>
>> For further anecdotal evidence: I've used pbzip2 with USE="symlink" for
>> several months now and never had trouble with it. Checking out lbzip2
>> now. I noticed it doesn't install a bunzip2 symlink.
> 
> Piotr Szymaniak asked me about lbzip2, and I bounced the question over
> to my friend. He didn't investigate it deeply; it crashed (OOM or
> something else, I don't know) when he tried it on a large file. Could
> have been from 2GB to 2TB, from what he has laying around. I don't
> know; I didn't get that one in writing. :)
> 
> But if it proves to be stable for small and very large files, I'd have
> no complaint. :)
> 

I just encountered this:

bzip2 -c </srv/qemu/hpwin.img >/dev/null
bzip2:
/var/tmp/portage/app-arch/lbzip2-2.2/work/lbzip2-2.2/src/encode.c:794:
generate_initial_trees: Assertion `a < b' failed.

Something in that file is upsetting lbzip2. I'm investigating.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2
  2012-09-26 21:49 ` Chí-Thanh Christopher Nguyễn
  2012-09-26 21:59   ` Michael Mol
@ 2012-09-27  8:58   ` Tobias Klausmann
  1 sibling, 0 replies; 22+ messages in thread
From: Tobias Klausmann @ 2012-09-27  8:58 UTC (permalink / raw
  To: gentoo-dev

Hi! 

On Wed, 26 Sep 2012, Chí-Thanh Christopher Nguyễn wrote:
> A different question is whether in the cases where parallel bzip2 makes
> sense, is it really the best solution? xz is outperforming bzip2's
> compression ratio for large files (for an informal comparison, see bug
> 434350). And xz is faster at decompression, which offsets the parallel
> advantage to some degree.

As for the performance side of things: 
http://blog.i-no.de/archives/2008/05/08/index.html#e2008-05-08T16_35_13.txt

Yes, that's four years old and needs to be redone with modern
implementations (and machines). Will do so this weekend (I hope).

Regards,
Tobias

-- 
printk (KERN_ALERT "You are screwed! " ...);
        linux-2.6.6/arch/i386/kernel/efi.c


^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2
  2012-09-26 20:43 ` Matt Turner
  2012-09-26 21:27   ` Florian Philipp
@ 2012-09-27  9:23   ` Piotr Szymaniak
  2012-10-29 14:13     ` Rick "Zero_Chaos" Farina
  1 sibling, 1 reply; 22+ messages in thread
From: Piotr Szymaniak @ 2012-09-27  9:23 UTC (permalink / raw
  To: gentoo-dev

[-- Attachment #1: Type: text/plain, Size: 1368 bytes --]

On Wed, Sep 26, 2012 at 01:43:27PM -0700, Matt Turner wrote:
> On Wed, Sep 26, 2012 at 1:30 PM, Michael Mol <mikemol@gmail.com> wrote:
> > A few months ago, I filed bug 423651 to ask that bzip2 on the install
> > media be replaced with
> >  pbzip2. It was closed a short while later, telling me that it'd
> > involve changing what's kept in @system, and that had to be discussed
> > here, rather than in a bug report.
> 
> If we're going to ship a parallel bzip2 implementation, it should be
> lbzip2 and not pbzip2.
> 
> lbzip2 can decompress bz2 archives with multiple threads that haven't
> been compressed with lbzip2/pbzip2.

Afair I'm using PORTAGE_BZIP2_COMMAND with lbzip2 and it works fine.
Also some time ago I've changed a bit the (famous?) stage4 backup
script from g-wiki to support parallel gz/bz2 implementations (simple
check if there pbzip2/lbzip2/foobar installed and if yes, use it instead
of normal gzip/bzip2).

Maybe portage should be like my stage4 mod? If it finds some parallel
(de)compressor it should use it, if not fallback to standard gzip/bzip2?


Piotr Szymaniak.
-- 
 - Jeden hamburger na dziesiec ci zaszkodzi. Jeden moj stary przyjaciel
to sprawdzil.  Zjadal dziewiec hamburgerow i byl idealnie zdrowy, a gdy
probowal zjesc dziesiatego, z miejsca dostawal torsji.
  -- Graham Masterton, "Night Warriors"

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2
  2012-09-26 20:30 [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2 Michael Mol
  2012-09-26 20:43 ` Matt Turner
  2012-09-26 21:49 ` Chí-Thanh Christopher Nguyễn
@ 2012-09-27  9:48 ` Ulrich Mueller
  2 siblings, 0 replies; 22+ messages in thread
From: Ulrich Mueller @ 2012-09-27  9:48 UTC (permalink / raw
  To: gentoo-dev

>>>>> On Wed, 26 Sep 2012, Michael Mol wrote:

> A few months ago, I filed bug 423651 to ask that bzip2 on the
> install media be replaced with pbzip2. It was closed a short while
> later, telling me that it'd involve changing what's kept in @system,
> and that had to be discussed here, rather than in a bug report.

We need to be careful when we replace such standard tools. Often the
replacement isn't completely compatible. For example, pbzip2 suffers
from the same bug as pigz [1] when it encounters a zero-padded tarball:

   $ echo foo | bzip2 | dd conv=sync 2>/dev/null | pbzip2 -d
   foo
   pbzip2: *ERROR during BZ2_bzDecompress - trailing garbage: ret=4; block=0; seq=0; isLastInSeq=1; avail_in=472
   Terminator thread: premature exit requested - quitting...
   $ echo $?
   1

The same command line as above but with bzip2 -d will return a good
exit status.

Ulrich

[1] <https://bugs.gentoo.org/show_bug.cgi?id=417657#c17>


^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2
  2012-09-27  7:22       ` Florian Philipp
@ 2012-09-27 15:09         ` Florian Philipp
  0 siblings, 0 replies; 22+ messages in thread
From: Florian Philipp @ 2012-09-27 15:09 UTC (permalink / raw
  To: gentoo-dev

[-- Attachment #1: Type: text/plain, Size: 2194 bytes --]

Am 27.09.2012 09:22, schrieb Florian Philipp:
> Am 26.09.2012 23:53, schrieb Michael Mol:
>> On Wed, Sep 26, 2012 at 5:27 PM, Florian Philipp <lists@binarywings.net> wrote:
>>> Am 26.09.2012 22:43, schrieb Matt Turner:
>>>> On Wed, Sep 26, 2012 at 1:30 PM, Michael Mol <mikemol@gmail.com> wrote:
>>>>> A few months ago, I filed bug 423651 to ask that bzip2 on the install
>>>>> media be replaced with
>>>>>  pbzip2. It was closed a short while later, telling me that it'd
>>>>> involve changing what's kept in @system, and that had to be discussed
>>>>> here, rather than in a bug report.
>>>>
>>>> If we're going to ship a parallel bzip2 implementation, it should be
>>>> lbzip2 and not pbzip2.
>>>>
>>>> lbzip2 can decompress bz2 archives with multiple threads that haven't
>>>> been compressed with lbzip2/pbzip2.
>>>>
>>>
>>> This seems relevant, especially comment 12ff:
>>> https://bugs.gentoo.org/show_bug.cgi?id=309683
>>>
>>> For further anecdotal evidence: I've used pbzip2 with USE="symlink" for
>>> several months now and never had trouble with it. Checking out lbzip2
>>> now. I noticed it doesn't install a bunzip2 symlink.
>>
>> Piotr Szymaniak asked me about lbzip2, and I bounced the question over
>> to my friend. He didn't investigate it deeply; it crashed (OOM or
>> something else, I don't know) when he tried it on a large file. Could
>> have been from 2GB to 2TB, from what he has laying around. I don't
>> know; I didn't get that one in writing. :)
>>
>> But if it proves to be stable for small and very large files, I'd have
>> no complaint. :)
>>
> 
> I just encountered this:
> 
> bzip2 -c </srv/qemu/hpwin.img >/dev/null
> bzip2:
> /var/tmp/portage/app-arch/lbzip2-2.2/work/lbzip2-2.2/src/encode.c:794:
> generate_initial_trees: Assertion `a < b' failed.
> 
> Something in that file is upsetting lbzip2. I'm investigating.

Okay, reported and (hopefully) fixed in
https://bugs.gentoo.org/show_bug.cgi?id=436382

In my infinite confidence in my own coding and testing skills I suggest
copying the proposed patch to /etc/portage/patches/app-arch/lbzip2-2.2
before trying out lbzip2. ;-)

Regards,
Florian Philipp


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2
  2012-09-27  9:23   ` Piotr Szymaniak
@ 2012-10-29 14:13     ` Rick "Zero_Chaos" Farina
  2012-10-29 14:39       ` Rich Freeman
  0 siblings, 1 reply; 22+ messages in thread
From: Rick "Zero_Chaos" Farina @ 2012-10-29 14:13 UTC (permalink / raw
  To: gentoo-dev

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 09/27/2012 05:23 AM, Piotr Szymaniak wrote:
> On Wed, Sep 26, 2012 at 01:43:27PM -0700, Matt Turner wrote:
>> On Wed, Sep 26, 2012 at 1:30 PM, Michael Mol <mikemol@gmail.com> wrote:
>>> A few months ago, I filed bug 423651 to ask that bzip2 on the install
>>> media be replaced with
>>>  pbzip2. It was closed a short while later, telling me that it'd
>>> involve changing what's kept in @system, and that had to be discussed
>>> here, rather than in a bug report.
>>
>> If we're going to ship a parallel bzip2 implementation, it should be
>> lbzip2 and not pbzip2.
>>
>> lbzip2 can decompress bz2 archives with multiple threads that haven't
>> been compressed with lbzip2/pbzip2.
> 
> Afair I'm using PORTAGE_BZIP2_COMMAND with lbzip2 and it works fine.
> Also some time ago I've changed a bit the (famous?) stage4 backup
> script from g-wiki to support parallel gz/bz2 implementations (simple
> check if there pbzip2/lbzip2/foobar installed and if yes, use it instead
> of normal gzip/bzip2).

I've been testing PORTAGE_BZIP2_COMMAND=lbzip2 for the few weeks since
this thread as well, and I have to say it's been great.
> 
> Maybe portage should be like my stage4 mod? If it finds some parallel
> (de)compressor it should use it, if not fallback to standard gzip/bzip2?

The releng team has begun using lbzip2 for decompressing things as much
as possible for speeding up catalyst builds, and speaking only for
myself I think it's been useful.  I spoke to zmedico about replacing the
default portage decompresser and he suggested far more use could be
attained by simply creating an eselect-bzip2 (possibly defaulting to
lbzip2 as this thread seemed to agree).  I'm confident no one would
attempt to block my adding eselect-bzip2 to the tree (aside from my poor
coding skills), but would anyone be interested in blocking using lbzip2
by default?  It seems pretty safe and I've done dozens of full system
builds etc.

Thanks,
Zero_Chaos
> 
> 
> Piotr Szymaniak.
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iQIcBAEBAgAGBQJQjo79AAoJEKXdFCfdEflKQNQQAIur3Mx4eD6kVv3US6kDkKpn
Z8aHyZqWIOws9VqcPCJUb1c1bn8aj2QTjL63obh//rJby1Qd1P3X3OA9hKfaiyiJ
hbgWRu5RkMSNPIdXhaslD3h/kL78BPr9eVz9O3R/TMp01puz4su0j1JVOtAT5Ny2
qAVof+H6RC5aCP2pKjIkvDq9vma/jd9QT98rM3UIw3BorBpkl0vTTa+sLtGA+U6p
mBkYZJFUUnkW3AtNC1ucPvQS9a7NwOxH5EDGEy4QeSJh9nvQZ8zCFivNtr4SUS6A
SGV0j/PB0Uqf42y1NOBfFSEQoQGT3aIlZzkbGdJKf6/jIdGPHhxWt3npQ6U6gOg9
TA/8zCJkJk0GZKAIq+K/y8STczeuIJ2hgrHUup4/I6CYwI66eBRiqnayNEmah7Cy
rpqR7xwqh+F2JrjljNYfE9qVQKwqhLU5MsOONlqRS+FPw+SuxxbJM6BiLCk4s0Qz
GPs5PXTz92MzzQ18XUuMq53bi9IaEmJJa8E83IBm3PD4zq67KsiWd2YxQo/JYX4p
ZTBayLabbY6PRwMVx0k9jKG7aOOKFZ5t2k5BmZDv6JBD2Sww9q6+xUPoTDwsZPgZ
pVv2m3AwaOXbGpvMrpqmzOTxbovUALp9WhdyTxlR5NmbTeXIuaKW/lQTU0cl/LKP
ACg4IADjv3hQz7gfsHAP
=hCz9
-----END PGP SIGNATURE-----


^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2
  2012-10-29 14:13     ` Rick "Zero_Chaos" Farina
@ 2012-10-29 14:39       ` Rich Freeman
  2012-10-31 13:56         ` Sergey Popov
  2013-01-02 23:11         ` Pacho Ramos
  0 siblings, 2 replies; 22+ messages in thread
From: Rich Freeman @ 2012-10-29 14:39 UTC (permalink / raw
  To: gentoo-dev

On Mon, Oct 29, 2012 at 10:13 AM, Rick "Zero_Chaos" Farina
<zerochaos@gentoo.org> wrote:
> I'm confident no one would
> attempt to block my adding eselect-bzip2 to the tree (aside from my poor
> coding skills),

++

> but would anyone be interested in blocking using lbzip2
> by default?  It seems pretty safe and I've done dozens of full system
> builds etc.

Why not just make it an option to start, advertise it by all means,
and then see how it turns out with volunteers before we make it the
default.  Going from nobody-has-heard-of-it to default overnight seems
a bit much.

Rich


^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2
  2012-10-29 14:39       ` Rich Freeman
@ 2012-10-31 13:56         ` Sergey Popov
  2013-01-02 23:11         ` Pacho Ramos
  1 sibling, 0 replies; 22+ messages in thread
From: Sergey Popov @ 2012-10-31 13:56 UTC (permalink / raw
  To: gentoo-dev

[-- Attachment #1: Type: text/plain, Size: 596 bytes --]

29.10.2012 18:39, Rich Freeman wrote:
> On Mon, Oct 29, 2012 at 10:13 AM, Rick "Zero_Chaos" Farina
>> but would anyone be interested in blocking using lbzip2
>> by default?  It seems pretty safe and I've done dozens of full system
>> builds etc.
> 
> Why not just make it an option to start, advertise it by all means,
> and then see how it turns out with volunteers before we make it the
> default.  Going from nobody-has-heard-of-it to default overnight seems
> a bit much.
> 

+1 for that

-- 
Best regards, Sergey Popov
Gentoo Linux Developer
Desktop-effects project lead


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 554 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2
  2012-10-29 14:39       ` Rich Freeman
  2012-10-31 13:56         ` Sergey Popov
@ 2013-01-02 23:11         ` Pacho Ramos
  2013-01-03  1:28           ` Rick "Zero_Chaos" Farina
  1 sibling, 1 reply; 22+ messages in thread
From: Pacho Ramos @ 2013-01-02 23:11 UTC (permalink / raw
  To: gentoo-dev

[-- Attachment #1: Type: text/plain, Size: 765 bytes --]

El lun, 29-10-2012 a las 10:39 -0400, Rich Freeman escribió:
> On Mon, Oct 29, 2012 at 10:13 AM, Rick "Zero_Chaos" Farina
> <zerochaos@gentoo.org> wrote:
> > I'm confident no one would
> > attempt to block my adding eselect-bzip2 to the tree (aside from my poor
> > coding skills),
> 
> ++
> 
> > but would anyone be interested in blocking using lbzip2
> > by default?  It seems pretty safe and I've done dozens of full system
> > builds etc.
> 
> Why not just make it an option to start, advertise it by all means,
> and then see how it turns out with volunteers before we make it the
> default.  Going from nobody-has-heard-of-it to default overnight seems
> a bit much.
> 
> Rich
> 
> 

How this ended finally? :/

Thanks for the info!

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2
  2013-01-02 23:11         ` Pacho Ramos
@ 2013-01-03  1:28           ` Rick "Zero_Chaos" Farina
  2013-01-03 10:30             ` Samuli Suominen
  0 siblings, 1 reply; 22+ messages in thread
From: Rick "Zero_Chaos" Farina @ 2013-01-03  1:28 UTC (permalink / raw
  To: gentoo-dev

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 01/02/2013 06:11 PM, Pacho Ramos wrote:
> El lun, 29-10-2012 a las 10:39 -0400, Rich Freeman escribió:
>> On Mon, Oct 29, 2012 at 10:13 AM, Rick "Zero_Chaos" Farina
>> <zerochaos@gentoo.org> wrote:
>>> I'm confident no one would
>>> attempt to block my adding eselect-bzip2 to the tree (aside from my poor
>>> coding skills),
>>
>> ++
>>
>>> but would anyone be interested in blocking using lbzip2
>>> by default?  It seems pretty safe and I've done dozens of full system
>>> builds etc.
>>
>> Why not just make it an option to start, advertise it by all means,
>> and then see how it turns out with volunteers before we make it the
>> default.  Going from nobody-has-heard-of-it to default overnight seems
>> a bit much.
>>
>> Rich
>>
>>
> 
> How this ended finally? :/
It ended with my setting PORTAGE_BZIP2_COMMAND="lbzip2" in my profile
and moving on with my life.  I am very not good at eselect scripts and
I've not had the time to dig at it. You are welcome to work on the
eselect script if you like.

> Thanks for the info!
> 

- -Zero
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iQIcBAEBAgAGBQJQ5N63AAoJEKXdFCfdEflKdn0P/R9W6idHgpd+n7y5nyKQM9si
noHmFC6KHJSZJnjO8BlrhsN2HZrgZyPqPIlWLn5VwK+KcDTh0s1y4UGGh22a93qD
JodVbmmSJlnWA+EkzNloG+nTk1FxJYnmoL7o6/dhqv5x1Q/FD/xf0rAGCgTvDPSb
RuoMbCfkgSwZZlSSJtHr7Xd3w1psAgVemXNcG1C49AmnNrrmEkvawk5SNzg5Ruyn
UVMKSbLt1zEkRmUZHWS+lizN+GjFMVlaPEb6KK7ELbYWEMIjBihZ0WqdSWQ5Yxcn
yaZN9RTM2xel9tS+PEZjxwCBwb4ndXxuj51zK91WeTeWL+1wvwRXQ0nL+jFi3WfA
9nIHe26Cop5btYnCw2/5wlmKYReAxasql0ZVS8mrob5IgiptMIrxCAhCq8raJVZn
VD3+GiWzHYQRBUEkwQq/Qq7n7d5hgyd8l2/CCSgNUxmNbtFLoNWxPK5aVvEeb5EH
zyLH0eOQp4JA2ViZtnYgkviFRaEr9pa+Kc2S4BbvPTqYOX7Q7Ah2LMOw8uXQMefB
cNlVBa6ar5feAybWkmKH+c9n6mjAU9iCWyuoATSUMpSQ1dPmDnmKmxst4kARgMVK
Ou1F4mREQk7N23AqAnREF3b4RNLQqv6ff5GuxG4COjXZpIynFcm4o/x4nnzan6xr
ALCXZ5LIRk5D5K//oYJU
=yYs4
-----END PGP SIGNATURE-----


^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2
  2013-01-03  1:28           ` Rick "Zero_Chaos" Farina
@ 2013-01-03 10:30             ` Samuli Suominen
  2013-01-03 10:35               ` Diego Elio Pettenò
  0 siblings, 1 reply; 22+ messages in thread
From: Samuli Suominen @ 2013-01-03 10:30 UTC (permalink / raw
  To: gentoo-dev

On 03/01/13 03:28, Rick "Zero_Chaos" Farina wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 01/02/2013 06:11 PM, Pacho Ramos wrote:
>> El lun, 29-10-2012 a las 10:39 -0400, Rich Freeman escribió:
>>> On Mon, Oct 29, 2012 at 10:13 AM, Rick "Zero_Chaos" Farina
>>> <zerochaos@gentoo.org> wrote:
>>>> I'm confident no one would
>>>> attempt to block my adding eselect-bzip2 to the tree (aside from my poor
>>>> coding skills),
>>>
>>> ++
>>>
>>>> but would anyone be interested in blocking using lbzip2
>>>> by default?  It seems pretty safe and I've done dozens of full system
>>>> builds etc.
>>>
>>> Why not just make it an option to start, advertise it by all means,
>>> and then see how it turns out with volunteers before we make it the
>>> default.  Going from nobody-has-heard-of-it to default overnight seems
>>> a bit much.
>>>
>>> Rich
>>>
>>>
>>
>> How this ended finally? :/
> It ended with my setting PORTAGE_BZIP2_COMMAND="lbzip2" in my profile
> and moving on with my life.  I am very not good at eselect scripts and
> I've not had the time to dig at it. You are welcome to work on the
> eselect script if you like.

Feel free to steal the code used in eselect-pinentry.

It was originally written by mgorny for eselect-sh, but then later 
modified by me, and has not had a single bug ever since.



^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2
  2013-01-03 10:30             ` Samuli Suominen
@ 2013-01-03 10:35               ` Diego Elio Pettenò
  0 siblings, 0 replies; 22+ messages in thread
From: Diego Elio Pettenò @ 2013-01-03 10:35 UTC (permalink / raw
  To: gentoo-dev

On 03/01/2013 11:30, Samuli Suominen wrote:
>>
> 
> Feel free to steal the code used in eselect-pinentry.
> 
> It was originally written by mgorny for eselect-sh, but then later
> modified by me, and has not had a single bug ever since.

Can I reiterate that it'd be cool to have an "eselect tools"? :)

-- 
Diego Elio Pettenò — Flameeyes
flameeyes@flameeyes.eu — http://blog.flameeyes.eu/


^ permalink raw reply	[flat|nested] 22+ messages in thread

end of thread, other threads:[~2013-01-03 10:35 UTC | newest]

Thread overview: 22+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-09-26 20:30 [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2 Michael Mol
2012-09-26 20:43 ` Matt Turner
2012-09-26 21:27   ` Florian Philipp
2012-09-26 21:53     ` Michael Mol
2012-09-27  7:22       ` Florian Philipp
2012-09-27 15:09         ` Florian Philipp
2012-09-27  9:23   ` Piotr Szymaniak
2012-10-29 14:13     ` Rick "Zero_Chaos" Farina
2012-10-29 14:39       ` Rich Freeman
2012-10-31 13:56         ` Sergey Popov
2013-01-02 23:11         ` Pacho Ramos
2013-01-03  1:28           ` Rick "Zero_Chaos" Farina
2013-01-03 10:30             ` Samuli Suominen
2013-01-03 10:35               ` Diego Elio Pettenò
2012-09-26 21:49 ` Chí-Thanh Christopher Nguyễn
2012-09-26 21:59   ` Michael Mol
2012-09-26 22:31     ` Mike Gilbert
2012-09-26 22:57       ` Christoph Junghans
2012-09-26 23:57         ` Michael Mol
2012-09-27  0:55         ` Diego Elio Pettenò
2012-09-27  8:58   ` Tobias Klausmann
2012-09-27  9:48 ` Ulrich Mueller

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox