[gentoo-user] Speed up with pbzip2 or not!?

[gentoo-user] Speed up with pbzip2 or not!?

[Justin]

[-- Attachment #1: Type: text/plain, Size: 978 bytes --]

Hey guys!

Yesterday I found this article
http://gentoo-wiki.com/HOWTO_Speed_up_decompression_with_pbzip2 and was
very happy, because I'm an owner of an Q6600.

So I tried to reproduce the benchmark from the wiki article but got that
bad results:

test # time *bzip2* -d -kf linux-2.6.23.tar.bz2

real    0m11.672s
user    0m11.306s
sys     0m0.367s


test # time *pbzip2* -d -p4 -kf linux-2.6.23.tar.bz2

real    0m25.554s
user    0m24.862s
sys     0m0.683s

So the parallel version took more than the double time!

To test whether this is a problem of my Pc I tested this on an Dual Core
with the same result.

An test with 7z was much better:


 test # time *7za* x -y linux-2.6.23.tar.bz2

real    0m4.642s
user    0m8.379s
sys     0m0.327s


All tests where done in a tmpfs off 1GB on 2GB RAM.


So my questions is what did I do wrong?

I also tested it with different CFLAGS, but my CFLAGS are basically very
conservative. (-O2 -march=prescott -pipe -fomit-frame-pointer)

[-- Attachment #2: Type: text/html, Size: 1528 bytes --]

Re: [gentoo-user] Speed up with pbzip2 or not!?

[Dirk Heinrichs]

[-- Attachment #1: Type: text/plain, Size: 404 bytes --]

Am Sonntag, 27. Januar 2008 schrieb Justin:

> Yesterday I found this article
> http://gentoo-wiki.com/HOWTO_Speed_up_decompression_with_pbzip2 and was
> very happy, because I'm an owner of an Q6600.
>
> So I tried to reproduce the benchmark from the wiki article but got that
> bad results:
>
> So my questions is what did I do wrong?

Your kernel is compiled with SMP support, isn't it?

Bye...

	Dirk

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] Speed up with pbzip2 or not!?

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 1183 bytes --]


On Sun, 2008-01-27 at 10:58 +0100, Justin wrote:
> Hey guys!
> 
> Yesterday I found this article
> http://gentoo-wiki.com/HOWTO_Speed_up_decompression_with_pbzip2 and
> was very happy, because I'm an owner of an Q6600.
> 
> So I tried to reproduce the benchmark from the wiki article but got
> that bad results:
> 
> test # time bzip2 -d -kf linux-2.6.23.tar.bz2 
> 
> real    0m11.672s
> user    0m11.306s
> sys     0m0.367s
> 
> 
> test # time pbzip2 -d -p4 -kf linux-2.6.23.tar.bz2 
> 
> real    0m25.554s
> user    0m24.862s
> sys     0m0.683s
> 
> So the parallel version took more than the double time!
> 
> To test whether this is a problem of my Pc I tested this on an Dual
> Core with the same result.
> 
> An test with 7z was much better:
> 
> 
>  test # time 7za x -y linux-2.6.23.tar.bz2 
> 
> real    0m4.642s
> user    0m8.379s
> sys     0m0.327s
> 

> So my questions is what did I do wrong?


Did you compress it with pbzip2 in the first place?

Pbzip2 can only speed up decompression of files created by pbzip2, as
well. Read the docs, dude ...

p7zip has got its own implementation of bzip2, which might be faster. 

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] Speed up with pbzip2 or not!?

[Justin]

Yes it is! But a question at this point: Where can I find the "Enhanced
Real Time Clock Support" or is it gone in the 2.6.23?

Dirk Heinrichs schrieb:
> Am Sonntag, 27. Januar 2008 schrieb Justin:
>
>   
>> Yesterday I found this article
>> http://gentoo-wiki.com/HOWTO_Speed_up_decompression_with_pbzip2 and was
>> very happy, because I'm an owner of an Q6600.
>>
>> So I tried to reproduce the benchmark from the wiki article but got that
>> bad results:
>>
>> So my questions is what did I do wrong?
>>     
>
> Your kernel is compiled with SMP support, isn't it?
>
> Bye...
>
> 	Dirk
>   
-- 
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] Speed up with pbzip2 or not!?

[Justin]

Thats a good point. Now it worked really fast.
But then the questions is why should I use pbzip2 for decompression with
portage? I think most tarballs are packed only with the normal
compression algorithm!
The WIKI articel pretends a gain of speed which wont be!

Florian Philipp schrieb:
> On Sun, 2008-01-27 at 10:58 +0100, Justin wrote:
>   
>> Hey guys!
>>
>> Yesterday I found this article
>> http://gentoo-wiki.com/HOWTO_Speed_up_decompression_with_pbzip2 and
>> was very happy, because I'm an owner of an Q6600.
>>
>> So I tried to reproduce the benchmark from the wiki article but got
>> that bad results:
>>
>> test # time bzip2 -d -kf linux-2.6.23.tar.bz2 
>>
>> real    0m11.672s
>> user    0m11.306s
>> sys     0m0.367s
>>
>>
>> test # time pbzip2 -d -p4 -kf linux-2.6.23.tar.bz2 
>>
>> real    0m25.554s
>> user    0m24.862s
>> sys     0m0.683s
>>
>> So the parallel version took more than the double time!
>>
>> To test whether this is a problem of my Pc I tested this on an Dual
>> Core with the same result.
>>
>> An test with 7z was much better:
>>
>>
>>  test # time 7za x -y linux-2.6.23.tar.bz2 
>>
>> real    0m4.642s
>> user    0m8.379s
>> sys     0m0.327s
>>
>>     
>
>   
>> So my questions is what did I do wrong?
>>     
>
>
> Did you compress it with pbzip2 in the first place?
>
> Pbzip2 can only speed up decompression of files created by pbzip2, as
> well. Read the docs, dude ...
>
> p7zip has got its own implementation of bzip2, which might be faster. 
>   
-- 
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] Speed up with pbzip2 or not!?

[Justin]

Is there a way to use 7zip for decompression with emerge?




Justin schrieb:
> Hey guys!
>
> Yesterday I found this article
> http://gentoo-wiki.com/HOWTO_Speed_up_decompression_with_pbzip2 and
> was very happy, because I'm an owner of an Q6600.
>
> So I tried to reproduce the benchmark from the wiki article but got
> that bad results:
>
> test # time *bzip2* -d -kf linux-2.6.23.tar.bz2
>
> real    0m11.672s
> user    0m11.306s
> sys     0m0.367s
>
>
> test # time *pbzip2* -d -p4 -kf linux-2.6.23.tar.bz2
>
> real    0m25.554s
> user    0m24.862s
> sys     0m0.683s
>
> So the parallel version took more than the double time!
>
> To test whether this is a problem of my Pc I tested this on an Dual
> Core with the same result.
>
> An test with 7z was much better:
>
>
>  test # time *7za* x -y linux-2.6.23.tar.bz2
>
> real    0m4.642s
> user    0m8.379s
> sys     0m0.327s
>
>
> All tests where done in a tmpfs off 1GB on 2GB RAM.
>
>
> So my questions is what did I do wrong?
>
> I also tested it with different CFLAGS, but my CFLAGS are basically
> very conservative. (-O2 -march=prescott -pipe -fomit-frame-pointer)
-- 
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] Speed up with pbzip2 or not!?

[Dirk Heinrichs]

[-- Attachment #1: Type: text/plain, Size: 237 bytes --]

Am Sonntag, 27. Januar 2008 schrieb Justin:

> But a question at this point: Where can I find the "Enhanced
> Real Time Clock Support" or is it gone in the 2.6.23?

Run "make menuconfig" and use the search function ("/").

HTH...

	Dirk

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] Speed up with pbzip2 or not!?

[Justin]

I did but without success. Thats why im asking!!

Dirk Heinrichs schrieb:
> Am Sonntag, 27. Januar 2008 schrieb Justin:
>
>   
>> But a question at this point: Where can I find the "Enhanced
>> Real Time Clock Support" or is it gone in the 2.6.23?
>>     
>
> Run "make menuconfig" and use the search function ("/").
>
> HTH...
>
> 	Dirk
>   
-- 
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] Speed up with pbzip2 or not!?

[Justin]

Google is my friend:

Device Drivers  --->
    Character devices  --->
      <M> Enhanced Real Time Clock Support



Justin schrieb:
> I did but without success. Thats why im asking!!
>
> Dirk Heinrichs schrieb:
>   
>> Am Sonntag, 27. Januar 2008 schrieb Justin:
>>
>>   
>>     
>>> But a question at this point: Where can I find the "Enhanced
>>> Real Time Clock Support" or is it gone in the 2.6.23?
>>>     
>>>       
>> Run "make menuconfig" and use the search function ("/").
>>
>> HTH...
>>
>> 	Dirk
>>   
>>     
-- 
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] Speed up with pbzip2 or not!?

[Benno Schulenberg]

Justin wrote:
> Is there a way to use 7zip for decompression with emerge?

Can this top-posting stop, please?  Interleave, and snip.

http://en.wikipedia.org/wiki/Top-posting

Benno
-- 
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] Speed up with pbzip2 or not!?

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 583 bytes --]


On Sun, 2008-01-27 at 11:40 +0100, Justin wrote:
> Is there a way to use 7zip for decompression with emerge?

I fear, no easy one. Of course, since emerge is just a python script,
you could rewrite it but 7z-syntax isn't compatible with bzip2 or
gnu-tar so it will be some work. I also tried to use star as a drop-in
replacement for gnu-tar, once. Didn't work well although their syntax is
nearly identical, but just nearly...

By the way: Please don't top-post (e.g. write your answers to a mail
below the quote. It makes reading long threads with many quotes easier.)

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] Speed up with pbzip2 or not!?

[Hemmann, Volker Armin]

On Sonntag, 27. Januar 2008, Justin wrote:
> Thats a good point. Now it worked really fast.
> But then the questions is why should I use pbzip2 for decompression with
> portage? I think most tarballs are packed only with the normal
> compression algorithm!
> The WIKI articel pretends a gain of speed which wont be!

and that is why you should never trust wiki-articles. Everybody can write them 
and say whatever they want.
-- 
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] Speed up with pbzip2 or not!?

[Enrico Weigelt]

* Florian Philipp <lists@f_philipp.fastmail.net> wrote:

Hi folks,

> On Sun, 2008-01-27 at 11:40 +0100, Justin wrote:
> > Is there a way to use 7zip for decompression with emerge?
> 
> I fear, no easy one. Of course, since emerge is just a python script,
> you could rewrite it but 7z-syntax isn't compatible with bzip2 or
> gnu-tar so it will be some work. I also tried to use star as a drop-in
> replacement for gnu-tar, once. Didn't work well although their syntax is
> nearly identical, but just nearly...

Why not an universal wrapper script ?

maybe something like:

    magic-uncompress [-t <format>] [-c] [-o <output>] <input>
    
This script could try to find out the input format automatically
(the optional -t parameter allows to explicitly specifiy the format)

Once we have this script, ebuilds could be rewritten step by step
and no one (else than magic-uncompress) has to care about the actual 
commands behind. 

Some could be done w/ tar and zip, etc.


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------
-- 
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] Speed up with pbzip2 or not!?

[Bo Ørsted Andresen]

[-- Attachment #1: Type: text/plain, Size: 701 bytes --]

On Sunday 27 January 2008 18:29:56 Enrico Weigelt wrote:
> Why not an universal wrapper script ?
>
> maybe something like:
>
>     magic-uncompress [-t <format>] [-c] [-o <output>] <input>
>    
> This script could try to find out the input format automatically
> (the optional -t parameter allows to explicitly specifiy the format)
>
> Once we have this script, ebuilds could be rewritten step by step
> and no one (else than magic-uncompress) has to care about the actual
> commands behind.
>
> Some could be done w/ tar and zip, etc.

You've just reinvented unpack(). It's been a part of portage for ages and is 
used in just about every ebuild in the tree...

-- 
Bo Andresen

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] Speed up with pbzip2 or not!?

[Justin]

Enrico Weigelt schrieb:
> * Florian Philipp <lists@f_philipp.fastmail.net> wrote:
>
> Hi folks,
>
>   
>> On Sun, 2008-01-27 at 11:40 +0100, Justin wrote:
>>     
>>> Is there a way to use 7zip for decompression with emerge?
>>>       
>> I fear, no easy one. Of course, since emerge is just a python script,
>> you could rewrite it but 7z-syntax isn't compatible with bzip2 or
>> gnu-tar so it will be some work. I also tried to use star as a drop-in
>> replacement for gnu-tar, once. Didn't work well although their syntax is
>> nearly identical, but just nearly...
>>     
>
> Why not an universal wrapper script ?
>
> maybe something like:
>
>     magic-uncompress [-t <format>] [-c] [-o <output>] <input>
>     
> This script could try to find out the input format automatically
> (the optional -t parameter allows to explicitly specifiy the format)
>
> Once we have this script, ebuilds could be rewritten step by step
> and no one (else than magic-uncompress) has to care about the actual 
> commands behind. 
>
> Some could be done w/ tar and zip, etc.
>
>
> cu
>   

But than there is still the incompability problem with pbzip2 with bzip2
compressed files. The best solution at this point would be to compress
all official tarballs with pbzip2, because now and  especially in future
parallel processing is the solution and second there is no problem
decompressing parallel compressed tarballs in single mode.


For an temporary work around I figured out this hack:

in /usr/lib/portage/bin/ebuild.sh there is a function defined for each
type of compression algorithm. I changed following lines

--- ebuild.sh.orig      2008-01-27 19:16:22.000000000 +0100
+++ ebuild.sh   2008-01-27 19:15:36.000000000 +0100
@@ -479,7 +479,7 @@
                                ;;
                        bz2|bz)
                                if [ "${y}" == "tar" ]; then
-                                       7za x -so "${srcdir}${x}" | tar
xof - ${tar_opts}
+                                       bzip2 -dc "${srcdir}${x}" | tar
xof - ${tar_opts}
                                        assert "$myfail"
                                else
                                        bzip2 -dc "${srcdir}${x}" >
${x%.*} || die "$myfail"

ebuild.sh.orig:
real    0m16.962s
user    0m16.289s
sys     0m2.780s


ebuild.sh:
real    0m12.805s
user    0m13.209s
sys     0m2.666s

With this trick you can change every kind of decompression command.

In this example you can save 25% of the time.
-- 
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] Speed up with pbzip2 or not!?

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 1178 bytes --]


On Sun, 2008-01-27 at 13:54 +0100, Hemmann, Volker Armin wrote:
> On Sonntag, 27. Januar 2008, Justin wrote:
> > Thats a good point. Now it worked really fast.
> > But then the questions is why should I use pbzip2 for decompression with
> > portage? I think most tarballs are packed only with the normal
> > compression algorithm!
> > The WIKI articel pretends a gain of speed which wont be!
> 
> and that is why you should never trust wiki-articles. Everybody can write them 
> and say whatever they want.

Well, many eyes see much. It's all a matter of checking, just as with
open source software...

To the topic:
I've unmerged pbzip2 after reading its docs. ... seemed too much
trouble. I'll try it again when it is suitable as a drop-in replacement
or when portage can make use of it. In the mean time, if I need good
compression with more than one thread, I use p7zip's lzma
implementation.

By the way, as soon as I come into contact with some decent scripting
languages (and no longer this closed source LabVIEW I currently have to
work with), I'll try to build a wrapper around p7zip to create a drop-in
replacement for gzip, bzip2 and zip. 

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]