[gentoo-user] Core 2 duo: Building threaded program versions

[gentoo-user] Core 2 duo: Building threaded program versions

[Nico Schümann]

Hi folks,

I recently set up a Core 2 duo system. Following the AMD64 howto, the
system now works. Kernel compilation took 1:40 minutes passing -j6 to
make.

Then I tried to bunzip a large file and found out that the machine was
about 40 % in idle which is clear because bzip2 is only one process.
So is there any way to build those utilities like bzip2 or gzip in a
thread-using way? It's not that nice to see the machine in idle at
all^^ Maybe there's any hidden use flag I haven't heard of or so :)

Greets,

Nico
-- 
gentoo-user@gentoo.org mailing list

[gentoo-user] Re: Core 2 duo: Building threaded program versions

[Sven Köhler]

[-- Attachment #1: Type: text/plain, Size: 929 bytes --]

> I recently set up a Core 2 duo system. Following the AMD64 howto, the
> system now works. Kernel compilation took 1:40 minutes passing -j6 to
> make.
> 
> Then I tried to bunzip a large file and found out that the machine was
> about 40 % in idle which is clear because bzip2 is only one process.
> So is there any way to build those utilities like bzip2 or gzip in a
> thread-using way? It's not that nice to see the machine in idle at
> all^^ Maybe there's any hidden use flag I haven't heard of or so :)

I think, that is a misconception within bzip2.

The alternative would be:
- decompress one chunk, and write it to the stdout while the next block
is already being decompressed.
- compress a chunk, while reading the next one from stdin

That would eat up at least twice the memory as it does now.


And after i searched portage, i found "pbzip2" - what about that? It may
solve your problems ;-)


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 249 bytes --]

Re: [gentoo-user] Core 2 duo: Building threaded program versions

[Etaoin Shrdlu]

On Monday 8 January 2007 21:16, Nico Schümann wrote:

> Hi folks,
>
> I recently set up a Core 2 duo system. Following the AMD64 howto, the
> system now works. Kernel compilation took 1:40 minutes passing -j6 to
> make.
>
> Then I tried to bunzip a large file and found out that the machine was
> about 40 % in idle which is clear because bzip2 is only one process.
> So is there any way to build those utilities like bzip2 or gzip in a
> thread-using way?

Use pbzip2, a parallel version of bzip2. It's in portage.

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Core 2 duo: Building threaded program versions

[Boyd Stephen Smith Jr.]

[-- Attachment #1: Type: text/plain, Size: 1137 bytes --]

On Monday 08 January 2007 14:16, "Nico Schümann" 
<nico.schuemann@googlemail.com> wrote about '[gentoo-user] Core 2 duo: 
Building threaded program versions':
> Then I tried to bunzip a large file and found out that the machine was
> about 40 % in idle which is clear because bzip2 is only one process.
> So is there any way to build those utilities like bzip2 or gzip in a
> thread-using way?

In general, no.  If there was a magic way to turn non-parallel activities 
into parallel activities it would be done by the kernel and/or gcc 
automagically AND we would have seen desktops be multi-core years ago.

> It's not that nice to see the machine in idle at 
> all^^

Multi-task.
I'm doing very little right now and I have 207 processes running.
The great thing about my 4 cores is I be doing all kinds of crazy stuff in 
the background and my games/full-screen video never miss a beat.

-- 
"If there's one thing we've established over the years,
it's that the vast majority of our users don't have the slightest
clue what's best for them in terms of package stability."
-- Gentoo Developer Ciaran McCreesh

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] Core 2 duo: Building threaded program versions

[Richard Fish]

On 1/8/07, Nico Schümann <nico.schuemann@googlemail.com> wrote:
> thread-using way? It's not that nice to see the machine in idle at
> all^^ Maybe there's any hidden use flag I haven't heard of or so :)

Just turn on disk encryption with dm-crypt.  That will take care of
the "extra core is idle" problem! ;->

-Richard

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Core 2 duo: Building threaded program versions

[Kent Fredric]

On 1/9/07, Richard Fish <bigfish@asmallpond.org> wrote:
> On 1/8/07, Nico Schümann <nico.schuemann@googlemail.com> wrote:
> > thread-using way? It's not that nice to see the machine in idle at
> > all^^ Maybe there's any hidden use flag I haven't heard of or so :)
>
> Just turn on disk encryption with dm-crypt.  That will take care of
> the "extra core is idle" problem! ;->
>
> -Richard
>
> --
> gentoo-user@gentoo.org mailing list
>
>
and theres the rare chance you could be decompressing faster than you
have hard drive access, but that might be a bit far fetched. I
shouldn't post so early in the AM.

-- 
/<ent Fredric
(aka theJackal)

-- 
gentoo-user@gentoo.org mailing list

RE: [gentoo-user] Core 2 duo: Building threaded program versions

[Nelson, David (ED, PAR&D)]

> -----Original Message-----
> From: Kent Fredric [mailto:kentfredric@gmail.com]
> Sent: 09 January 2007 18:48
> To: gentoo-user@lists.gentoo.org
> Subject: Re: [gentoo-user] Core 2 duo: Building threaded program
> versions
> 
> 
> On 1/9/07, Richard Fish <bigfish@asmallpond.org> wrote:
> > On 1/8/07, Nico Schümann <nico.schuemann@googlemail.com> wrote:
> > > thread-using way? It's not that nice to see the machine in idle at
> > > all^^ Maybe there's any hidden use flag I haven't heard 
> of or so :)
> >
> > Just turn on disk encryption with dm-crypt.  That will take care of
> > the "extra core is idle" problem! ;->
> >
> > -Richard
> >
> > --
> > gentoo-user@gentoo.org mailing list
> >
> >
> and theres the rare chance you could be decompressing faster than you
> have hard drive access, but that might be a bit far fetched. I
> shouldn't post so early in the AM.
> 
> -- 
> /<ent Fredric
> (aka theJackal)
> 

Regarding pbzip2, by the way, I gave it a shot last night. Definately faster for the type of archive I was compressing (mp3s in a tar archive). If I remember rightly it was approx 1min30s vs 0min50s. Not quite twice as fast, but a definate improvement. I plan to do a bit of further testing/playing if anyone is interested in the results.

David

Note: These views are my own, advice is provided with no guarantee of success. I do not represent anyone else in any emails I send to this list.

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Core 2 duo: Building threaded program versions

[Uwe Thiem]

On 10 January 2007 09:46, Nelson, David (ED, PAR&D) wrote:

> Regarding pbzip2, by the way, I gave it a shot last night. Definately
> faster for the type of archive I was compressing (mp3s in a tar archive).
> If I remember rightly it was approx 1min30s vs 0min50s. Not quite twice as
> fast, but a definate improvement. I plan to do a bit of further
> testing/playing if anyone is interested in the results.

... and was the resulting tarball smaller than an uncompressed one?

Uwe

-- 
A fast and easy generator of fractals for KDE:
http://www.SysEx.com.na/iwy-1.0.tar.bz2
Proof of concept of a TSP solver for KDE:
http://www.SysEx.com.na/epat-0.1.tar.bz2
-- 
gentoo-user@gentoo.org mailing list

RE: [gentoo-user] Core 2 duo: Building threaded program versions

[Nelson, David (ED, PAR&D)]

-----Original Message-----
From: Uwe Thiem [mailto:uwix@iway.na]
Sent: 10 January 2007 08:40
To: gentoo-user@lists.gentoo.org
Subject: Re: [gentoo-user] Core 2 duo: Building threaded program
versions
>
>
>On 10 January 2007 09:46, Nelson, David (ED, PAR&D) wrote:
>
>> Regarding pbzip2, by the way, I gave it a shot last night. Definately
>> faster for the type of archive I was compressing (mp3s in a tar archive).
>> If I remember rightly it was approx 1min30s vs 0min50s. Not quite twice as
>> fast, but a definate improvement. I plan to do a bit of further
>> testing/playing if anyone is interested in the results.
>
>... and was the resulting tarball smaller than an uncompressed one?
>
>Uwe

By about 1% .... agreed MP3s are a poor example but it was late last night and I didn't have time to mess around with it much. Going to try with a mix of documents (word processed documents, text, pictures, some video) to see how it performs then.

David

Disclaimer: I represent no-one else in my emails to this list. Use any advice given at your own risk.

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Core 2 duo: Building threaded program versions

[Etaoin Shrdlu]

On Wednesday 10 January 2007 12:20, Nelson, David (ED, PAR&D) wrote:

> By about 1% .... agreed MP3s are a poor example but it was late last
> night and I didn't have time to mess around with it much. Going to try
> with a mix of documents (word processed documents, text, pictures,
> some video) to see how it performs then.

The site says that pbzip2 is compatible with bzip2, so it may be faster 
but I think it does not compress more than its single-threaded cousin.
-- 
gentoo-user@gentoo.org mailing list

RE: [gentoo-user] Core 2 duo: Building threaded program versions

[Nelson, David (ED, PAR&D)]

> -----Original Message-----
> From: Etaoin Shrdlu [mailto:shrdlu@unlimitedmail.org]
> Sent: 10 January 2007 12:15
> To: gentoo-user@lists.gentoo.org
> Subject: Re: [gentoo-user] Core 2 duo: Building threaded program
> versions
> 
> > By about 1% .... agreed MP3s are a poor example but it was late last
> > night and I didn't have time to mess around with it much. 
> Going to try
> > with a mix of documents (word processed documents, text, pictures,
> > some video) to see how it performs then.
> 
> The site says that pbzip2 is compatible with bzip2, so it may 
> be faster 
> but I think it does not compress more than its single-threaded cousin.

I was answering in reply to:

>... and was the resulting tarball smaller than an uncompressed one?

i.e. was mp3.tar.bz2 smaller than mp3.tar. Which it was, but only slightly as MP3s are already compressed as they are so don't compress so well.

David
Note: These views are my own, advice is provided with no guarantee of success. I do not represent anyone else in any emails I send to this list.

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Core 2 duo: Building threaded program versions

[Uwe Thiem]

On 10 January 2007 13:20, Nelson, David (ED, PAR&D) wrote:
> -----Original Message-----
> From: Uwe Thiem [mailto:uwix@iway.na]
> Sent: 10 January 2007 08:40
> To: gentoo-user@lists.gentoo.org
> Subject: Re: [gentoo-user] Core 2 duo: Building threaded program
> versions
>
> >On 10 January 2007 09:46, Nelson, David (ED, PAR&D) wrote:
> >> Regarding pbzip2, by the way, I gave it a shot last night. Definately
> >> faster for the type of archive I was compressing (mp3s in a tar
> >> archive). If I remember rightly it was approx 1min30s vs 0min50s. Not
> >> quite twice as fast, but a definate improvement. I plan to do a bit of
> >> further testing/playing if anyone is interested in the results.
> >
> >... and was the resulting tarball smaller than an uncompressed one?
> >
> >Uwe
>
> By about 1% .... agreed MP3s are a poor example but it was late last night
> and I didn't have time to mess around with it much. Going to try with a mix
> of documents (word processed documents, text, pictures, some video) to see
> how it performs then.

I was trying to make a point that is was rather pointless to check a 
compressor on files that were basically incompressible. ;-)

Uwe

-- 
A fast and easy generator of fractals for KDE:
http://www.SysEx.com.na/iwy-1.0.tar.bz2
Proof of concept of a TSP solver for KDE:
http://www.SysEx.com.na/epat-0.1.tar.bz2
-- 
gentoo-user@gentoo.org mailing list