[gentoo-user] bad jack(?) performance

[gentoo-user] bad jack(?) performance

[Matt Garman]

I installed jack (jack-audio-connection-kit), and configured it, as
far as I can tell, correctly.  At least I can get multiple audio
streams multiplexed together.

However, the performance leaves something to be desired: it is
*extremely* prone to skipping, if my system is under any kind of
load.  For example, doing an "emerge search <package>" will usually
cause my XMMS playback to skip.  *Moving a window* will almost
always cause a skip---not just skip, but freeze for the whole time
the window is being moved.  Even worse, whenever the screensaver
comes on, the player skips.  And while the screensaver is active,
the music playback will regularly skip every couple seconds.  The
kicker?  My screensaver is just a blank screen!

I'd like to think that my system is powerful enough to do "normal"
desktop use without skipping: Athlon XP 2500, 1 GB RAM.

Any thoughts as to why the jack performance is so shoddy?

Thanks!
Matt

-- 
Matt Garman
email at: http://raw-sewage.net/index.php?file=email
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] bad jack(?) performance

[Volker Armin Hemmann]

On Wednesday 21 September 2005 03:30, Matt Garman wrote:
> I installed jack (jack-audio-connection-kit), and configured it, as
> far as I can tell, correctly.  At least I can get multiple audio
> streams multiplexed together.
>
> However, the performance leaves something to be desired: it is
> *extremely* prone to skipping, if my system is under any kind of
> load.  For example, doing an "emerge search <package>" will usually
> cause my XMMS playback to skip.  *Moving a window* will almost
> always cause a skip---not just skip, but freeze for the whole time
> the window is being moved.  Even worse, whenever the screensaver
> comes on, the player skips.  And while the screensaver is active,
> the music playback will regularly skip every couple seconds.  The
> kicker?  My screensaver is just a blank screen!
>
> I'd like to think that my system is powerful enough to do "normal"
> desktop use without skipping: Athlon XP 2500, 1 GB RAM.
>
> Any thoughts as to why the jack performance is so shoddy?
>

do you get this skips also with alsaplayer or amarok?
xmms is a little bit skippy, but I never had skips with the other two.
Oh, and when you just want to mulitplex together - shouldn't the dmix alsa 
plugin all you need?
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] bad jack(?) performance

[Mark Knecht]

On 9/20/05, Matt Garman <garman@raw-sewage.net> wrote:
>
> I installed jack (jack-audio-connection-kit), and configured it, as
> far as I can tell, correctly.  At least I can get multiple audio
> streams multiplexed together.
>
> However, the performance leaves something to be desired: it is
> *extremely* prone to skipping, if my system is under any kind of
> load.  For example, doing an "emerge search <package>" will usually
> cause my XMMS playback to skip.  *Moving a window* will almost
> always cause a skip---not just skip, but freeze for the whole time
> the window is being moved.  Even worse, whenever the screensaver
> comes on, the player skips.  And while the screensaver is active,
> the music playback will regularly skip every couple seconds.  The
> kicker?  My screensaver is just a blank screen!
>
> I'd like to think that my system is powerful enough to do "normal"
> desktop use without skipping: Athlon XP 2500, 1 GB RAM.
>
> Any thoughts as to why the jack performance is so shoddy?
>
> Thanks!
> Matt
>
> --
> Matt Garman
> email at: http://raw-sewage.net/index.php?file=email
> --
> gentoo-user@gentoo.org mailing list
>
>

Matt,
   I'm very sorry but I somehow never saw this thread. Please accept
my apologies.

   Did you get this working better?

   Anyway, I'd say that your machine is certainly fast enough to do
good audio work, barring some specific chipset or driver issue which
most likely won't be the case. In my experience there are usually just
a few things required to get this working.

1) First, let's determine whether you need a new kernel. su to root
and run Jack and some simple Jack apps. I'd recommend alsaplayer since
it's simple, small and very stable:

(Note - this is best done through QJackCtl, but you can do it at the
command line if you want.) Choose latency numbers that make sense for
your card and your needs. I run at 128/2:

jackd -R -dalsa -r44100 -dhw -p128 -n2

Then with Jack running run alsaplayer

alsaplayer -o jack

Tell alsaplayer to play a wave file or play a CD for a better, longer
test. Any skipping?

If you have skipping at this point then you most likely need a
real-time kernel. My 32-bit machines do not. They run fine with
gentoo-sources, but my amd64 doesn't run well and needed a new
realtime kernel to work right.

2) Assuming that your tests as root go well, then emerge realtime-lsm.
This may require a new kernel if you don't have the right Linux
Securities stuff enabled:

[ ] Enable access key retention support
   [*] Enable different security models
   [ ]   Socket and Networking Security Hooks
  <M>   Default Linux Capabilities
  < >   Root Plug Support
  < >   BSD Secure Levels
   [ ]   NSA SELinux Support

Reboot into your new kernel if necessary, emerge realtime-lsm at this point.

3) Create a group for realtime access. Here's mine:

lightning linux # cat /etc/group | grep realtime
realtime:x:600:mark
lightning linux #

4) Modprobe realtime at the command line:

modprobe realtime gid=600 any=1

5) Assuming this all goes well, redo your tests with alsaplayer and
make sure it's working as a user.

   Enabling realtime operation thorugh realtime-lsm is CRITICAL to
making this work well over time. For instance this morning I'm
rebuilding glibc as part of an emerge sync / emerge world operation.
At the same time I'm doing email on the web as well as playing music
with Aqualung on my AMD64 machine. Jack is running at 128/2 and I can
go for hours without any xruns. True, we are still working bugs out of
the newest kernel (2.6.14-rc3-rt10 as of this morning) but things are
starting to work pretty well even on AMD64. My gentoo-sources 32-bit
machines can go for days with out an xrun. (Granted, very good RME
sound cards that cost as much as the machine are being used....)

   Hope this helps. Write back if you want or need more info.

Cheers,
Mark

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] bad jack(?) performance

[Matt Garman]

On Thu, Oct 06, 2005 at 10:39:44AM -0700, Mark Knecht wrote:
> 1) First, let's determine whether you need a new kernel. su to
> ...
> jackd -R -dalsa -r44100 -dhw -p128 -n2
> alsaplayer -o jack
> ...
> longer test. Any skipping?

Nope, not when running jackd+alsaplayer as root.

However, when running jackd+alsaplayer as a regular user, I get lots
of skipping.

> If you have skipping at this point then you most likely need a
> real-time kernel. My 32-bit machines do not. They run fine with
> gentoo-sources, but my amd64 doesn't run well and needed a new
> realtime kernel to work right.

FWIW, before posting this message, I followed a jack howto (can't
remember the exact source), which walked me through recompiling my
kernel (with "[*] Enable different security models" and "<M>
Default Linux Capabilities"), as well as installing and setting
realtime-lsm up correctly...

> 2) Assuming that your tests as root go well, then emerge
> realtime-lsm.  This may require a new kernel if you don't have the
> right Linux Securities stuff enabled:

...but because I'm error-prone, I double-check my configuration.  As
far as I can tell, I have everything set up correctly.

>From what I can tell, it appears that when I run jackd and
alsaplayer as a non-root user, they automatically get nice'ed, and I
believe this is what is causing the skipping.

For example, as root:

# ps ax | grep jack
9430 pts/1    SLl    0:08 jackd -R -dalsa -r44100 -dhw -p128 -n2
9434 pts/1    SLl    0:09 alsaplayer -o jack

# top
 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
9430 root      18   0 28196  27m 2344 S  2.3  2.7   0:08.68 jackd              
9434 root      15   0 61852  60m 9336 S  2.0  6.0   0:09.89 alsaplayer         

But as a regular user:

# ps ax | grep jack
9661 pts/11   SNLl   0:00 jackd -R -dalsa -r44100 -dhw -p128 -n2
9665 pts/11   SNLl   0:00 alsaplayer -o jack

# top

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
9665 garman    20   5 61868  60m 9336 S  2.0  6.0   0:00.86 alsaplayer         
9661 garman    22   5 28200  27m 2344 S  1.7  2.7   0:00.82 jackd              
      
Notice the "N" (nice) flag for ps, and the niceness value of 5 in
top?

I even tried invoking jackd with the nice program (e.g. "nice -n 0
jackd ..."), but still got stuck the result above.

Hopefully I'm missing something simple... any thoughts?

Thanks!
Matt

-- 
Matt Garman
email at: http://raw-sewage.net/index.php?file=email
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] bad jack(?) performance

[Mark Knecht]

On 10/12/05, Matt Garman <garman@raw-sewage.net> wrote:
> On Thu, Oct 06, 2005 at 10:39:44AM -0700, Mark Knecht wrote:
> > 1) First, let's determine whether you need a new kernel. su to
> > ...
> > jackd -R -dalsa -r44100 -dhw -p128 -n2
> > alsaplayer -o jack
> > ...
> > longer test. Any skipping?
>
> Nope, not when running jackd+alsaplayer as root.
>
> However, when running jackd+alsaplayer as a regular user, I get lots
> of skipping.
>
> > If you have skipping at this point then you most likely need a
> > real-time kernel. My 32-bit machines do not. They run fine with
> > gentoo-sources, but my amd64 doesn't run well and needed a new
> > realtime kernel to work right.
>
> FWIW, before posting this message, I followed a jack howto (can't
> remember the exact source), which walked me through recompiling my
> kernel (with "[*] Enable different security models" and "<M>
> Default Linux Capabilities"), as well as installing and setting
> realtime-lsm up correctly...
>
> > 2) Assuming that your tests as root go well, then emerge
> > realtime-lsm.  This may require a new kernel if you don't have the
> > right Linux Securities stuff enabled:
>
> ...but because I'm error-prone, I double-check my configuration.  As
> far as I can tell, I have everything set up correctly.
>
> From what I can tell, it appears that when I run jackd and
> alsaplayer as a non-root user, they automatically get nice'ed, and I
> believe this is what is causing the skipping.
>
> For example, as root:
>
> # ps ax | grep jack
> 9430 pts/1    SLl    0:08 jackd -R -dalsa -r44100 -dhw -p128 -n2

This is good but expected since root canalways access the capabilities
required to run realtime...but...

> 9434 pts/1    SLl    0:09 alsaplayer -o jack

...alsaplayer requires that you say you want to use realtime capabilities:

alsaplayer -r -o jack

This should work better. Give it a try. man alsaplayer for more
options and info.)

>
> # top
>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 9430 root      18   0 28196  27m 2344 S  2.3  2.7   0:08.68 jackd
> 9434 root      15   0 61852  60m 9336 S  2.0  6.0   0:09.89 alsaplayer
>
> But as a regular user:
>
> # ps ax | grep jack
> 9661 pts/11   SNLl   0:00 jackd -R -dalsa -r44100 -dhw -p128 -n2
> 9665 pts/11   SNLl   0:00 alsaplayer -o jack
>
> # top
>
>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 9665 garman    20   5 61868  60m 9336 S  2.0  6.0   0:00.86 alsaplayer
> 9661 garman    22   5 28200  27m 2344 S  1.7  2.7   0:00.82 jackd
>
> Notice the "N" (nice) flag for ps, and the niceness value of 5 in
> top?

Nice should not be necessary.

>
> I even tried invoking jackd with the nice program (e.g. "nice -n 0
> jackd ..."), but still got stuck the result above.
>
> Hopefully I'm missing something simple... any thoughts?
>

Yeah, just the -r most likely. Also, depending on your sound card
128/2 might be a bit tight, but let's try for it and see what happens.

- Mark

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] bad jack(?) performance

[Matt Garman]

On Wed, Oct 12, 2005 at 06:48:48PM -0700, Mark Knecht wrote:
> ...alsaplayer requires that you say you want to use realtime
> capabilities:
> alsaplayer -r -o jack
> ...
> Yeah, just the -r most likely. Also, depending on your sound card
> 128/2 might be a bit tight, but let's try for it and see what happens.

Unfortunately, adding the -r had no effect as far as I can tell.

According to the alsaplayer manpage, 

          -r, --realtime
          Enable realtime scheduling.  To  use  this  as a  normal
          user,  alsaplayer must be SUID root.

So I tried setting alsaplayer SUID root:

# chmod u+s `which alsaplayer`

Then as a regular (non-root) user:

# alsaplayer -r -o jack & 
Gtk-WARNING **: This process is currently running setuid or setgid.
This is not a supported use of GTK+. You must create a helper
program instead. For further details, see:

    http://www.gtk.org/setuid.html

Refusing to initialize GTK+.
[2]  + exit 1     alsaplayer -r -o jack

I'm guessing that most folks don't have to worry about the whole
SUID root thing (or creating a "help program")?

Any more thoughts?

Thank you for all your help!

Matt

-- 
Matt Garman
email at: http://raw-sewage.net/index.php?file=email
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] bad jack(?) performance

[Mark Knecht]

On 10/12/05, Matt Garman <garman@raw-sewage.net> wrote:
> On Wed, Oct 12, 2005 at 06:48:48PM -0700, Mark Knecht wrote:
> > ...alsaplayer requires that you say you want to use realtime
> > capabilities:
> > alsaplayer -r -o jack
> > ...
> > Yeah, just the -r most likely. Also, depending on your sound card
> > 128/2 might be a bit tight, but let's try for it and see what happens.
>
> Unfortunately, adding the -r had no effect as far as I can tell.
>
> According to the alsaplayer manpage,
>
>           -r, --realtime
>           Enable realtime scheduling.  To  use  this  as a  normal
>           user,  alsaplayer must be SUID root.
>
> So I tried setting alsaplayer SUID root:
>
> # chmod u+s `which alsaplayer`
>
> Then as a regular (non-root) user:
>
> # alsaplayer -r -o jack &
> Gtk-WARNING **: This process is currently running setuid or setgid.
> This is not a supported use of GTK+. You must create a helper
> program instead. For further details, see:
>
>     http://www.gtk.org/setuid.html
>
> Refusing to initialize GTK+.
> [2]  + exit 1     alsaplayer -r -o jack
>
> I'm guessing that most folks don't have to worry about the whole
> SUID root thing (or creating a "help program")?

No. None of that is required for me on any kernel - Gentoo or Vanilla.
I just set up realtime-lsm and then run with realtime capabilites. I
would suggest that you use QJackCtl to run Jack as it will save your
settings nicely for you and give you patch bay access to hooking Jack
apps up to the server.

Note that I use pretty expensive RME cards. They work exceedingly well
for me. There are a lot of people out there that report they never go
faster than 128/2 or 256/2. 256/2 is about as good as any of my
Windows systems have every worked, and better then Pro Tools worked
when I owned it. You should not think that going a bit slower is
necessarily a problem. If you cannot hear the latency it doesn't
matte, and even if you can hear it, you can nudge recorded tracks
after recording to get a better sound if you can lay down a good
track.

>
> Any more thoughts?

Yes, but not sure you're going to like them... ;-)

The first one is easy. Try some different Jack settings. Instead of
128/2 try 64/4, or 128/3, etc., and see if some other setting works.
You might get the same latency, or you might have to go a bit slower.
The only time I actually use low latency is when recording. It's never
needed for playback only. Most of the time I run 512/2 just to ensure
no xruns causing clicks in my work.

On my 32-bit machines I've always been able to run Jack the standard
Gentoo-sources  kernel and get good realtime results. I have had to be
careful about what options I choose, and on a couple of machines
different kernel options have caused xruns (such as networking) but
I've always managed to get it to work and work well. Sometimes it has
taken some time, but it has worked. Maybe we need to look at how you
are configuring the kernel. Possibly send your config file off list or
I'll send you one of mine.

That said, on my new AMD64 machine gentoo-sources just doesn't cut it.
I had to go to a custom kernel to get realtime to work. I first tried
ck-sources, which lots of people report as working for them, but that
did not work for me, so I went with Ingo's realtime preempt patches
and I'm getting pretty good results. I get a few xruns/day at 64/2,
none so far at 128/2 running 20 track sessions in Ardour. I'm using
2.6.14-rc4-rt1. Here's the patches required to do that, should you
choose to go there:

http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.13.tar.bz2
http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.14-rc4.bz2
http://redhat.com/~mingo/realtime-preempt/patch-2.6.14-rc4-rt1

(This one is VERY new. There are more stable, tested versions out
there based on 2.6.13. I needed this due to AMD64)

Once this is up and running you get access to setting priorities for
all devices and things work pretty well. (I.e. - don't be disappointed
if it doesn't do any better the first time you boot it.)
Unfortunately, since Gentoo doesn't support an 'audio kernel' yet you
and I would have to manage updates on our own. That said, this is the
way most people interested in good realtime performance have gone.
Maybe I've just been excessively lucky up until now.

It's probably worth it to review how you've set up realtime-lsm one
more time, just in case, and possibly to look at your hardware setup a
bit.

lspci
lsmod
cat /proc/asound/cards

>
> Thank you for all your help!

Wish it was more successful. We should just keep plugging away.

What audio stuff are you going to use this machine for, BTW?

- Mark

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] bad jack(?) performance

[Mark Knecht]

On 10/13/05, Mark Knecht <markknecht@gmail.com> wrote:
<SNIP>
>
> The first one is easy. Try some different Jack settings. Instead of
> 128/2 try 64/4, or 128/3, etc., and see if some other setting works.
> You might get the same latency, or you might have to go a bit slower.
> The only time I actually use low latency is when recording. It's never
> needed for playback only. Most of the time I run 512/2 just to ensure
> no xruns causing clicks in my work.
>
<SNIP>

Also, some inexpensive cards like 48K operation and some like 44.1K
operation. Make sure you explore different sample rates. Sorry for
forgetting that before.

- Mark

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] bad jack(?) performance

[Matt Garman]

On Thu, Oct 13, 2005 at 05:06:22AM -0700, Mark Knecht wrote:
> The first one is easy. Try some different Jack settings. Instead
> of 128/2 try 64/4, or 128/3, etc., and see if some other setting
> works.  You might get the same latency, or you might have to go a
> bit slower.  The only time I actually use low latency is when
> recording. It's never needed for playback only. Most of the time I
> run 512/2 just to ensure no xruns causing clicks in my work.

I haven't had a chance to try that yet, but still I wonder: why
should I have to tweak jack's settings, if 128/2 worked fine for
root?

> On my 32-bit machines I've always been able to run Jack the
> standard Gentoo-sources  kernel and get good realtime results. I
> have had to be careful about what options I choose, and on a
> couple of machines different kernel options have caused xruns
> (such as networking) but I've always managed to get it to work and
> work well. Sometimes it has taken some time, but it has worked.
> Maybe we need to look at how you are configuring the kernel.
> Possibly send your config file off list or I'll send you one of
> mine.

Ehhh, that scares me a bit (the thought that random kernel options
can affect Jack performance)!  My kernel config is fairly custom, as
I went through each option one-by-one and (at least tried to) set it
most appropriately for my hardware.

But I'm still hung up on the fact that things work as expected as
root, but automatically get "nice'ed" as a regular user.

> and I would have to manage updates on our own. That said, this is
> the way most people interested in good realtime performance have
> gone.  Maybe I've just been excessively lucky up until now.

And at this point I'm not too interested in that good of realtime
performance, though I will in the future (see below).

> It's probably worth it to review how you've set up realtime-lsm
> one more time, just in case, and possibly to look at your hardware
> setup a bit.

I agree, as I alluded to above, I'm thinking it might be a
permissions/setup issue.

> lspci

0000:00:00.0 Host bridge: nVidia Corporation nForce2 AGP (different version?) (rev c1)
0000:00:00.1 RAM memory: nVidia Corporation nForce2 Memory Controller 1 (rev c1)
0000:00:00.2 RAM memory: nVidia Corporation nForce2 Memory Controller 4 (rev c1)
0000:00:00.3 RAM memory: nVidia Corporation nForce2 Memory Controller 3 (rev c1)
0000:00:00.4 RAM memory: nVidia Corporation nForce2 Memory Controller 2 (rev c1)
0000:00:00.5 RAM memory: nVidia Corporation nForce2 Memory Controller 5 (rev c1)
0000:00:01.0 ISA bridge: nVidia Corporation nForce2 ISA Bridge (rev a4)
0000:00:01.1 SMBus: nVidia Corporation nForce2 SMBus (MCP) (rev a2)
0000:00:02.0 USB Controller: nVidia Corporation nForce2 USB Controller (rev a4)
0000:00:02.1 USB Controller: nVidia Corporation nForce2 USB Controller (rev a4)
0000:00:02.2 USB Controller: nVidia Corporation nForce2 USB Controller (rev a4)
0000:00:05.0 Multimedia audio controller: nVidia Corporation nForce Audio Processing Unit (rev a2)
0000:00:06.0 Multimedia audio controller: nVidia Corporation nForce2 AC97 Audio Controler (MCP) (rev a1)
0000:00:08.0 PCI bridge: nVidia Corporation nForce2 External PCI Bridge (rev a3)
0000:00:09.0 IDE interface: nVidia Corporation nForce2 IDE (rev a2)
0000:00:0c.0 PCI bridge: nVidia Corporation nForce2 PCI Bridge (rev a3)
0000:00:1e.0 PCI bridge: nVidia Corporation nForce2 AGP (rev c1)
0000:01:0a.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1010 Ultra3 SCSI Adapter (rev 01)
0000:01:0a.1 SCSI storage controller: LSI Logic / Symbios Logic 53c1010 Ultra3 SCSI Adapter (rev 01)
0000:02:01.0 Ethernet controller: 3Com Corporation 3C920B-EMB Integrated Fast Ethernet Controller [Tornado] (rev 40)
0000:03:00.0 VGA compatible controller: nVidia Corporation NV28 [GeForce4 Ti 4200 AGP 8x] (rev a1)

> lsmod

Module                  Size  Used by
realtime                7752  0 
parport_pc             31812  1 
lp                      8260  0 
parport                20992  2 parport_pc,lp
usblp                  11072  0 
uhci_hcd               29200  0 
ehci_hcd               27272  0 
ohci_hcd               16584  0 
nvidia_agp              5916  1 
snd_pcm_oss            48288  0 
snd_mixer_oss          17664  1 snd_pcm_oss
snd_intel8x0           28032  3 
snd_ac97_codec         72192  1 snd_intel8x0
snd_pcm                81352  5 snd_pcm_oss,snd_intel8x0,snd_ac97_codec
snd_timer              21700  1 snd_pcm
snd                    45156  10 snd_pcm_oss,snd_mixer_oss,snd_intel8x0,snd_ac97_codec,snd_pcm,snd_timer
soundcore               7776  1 snd
snd_page_alloc          7620  2 snd_intel8x0,snd_pcm
w83l785ts               5716  0 
asb100                 21908  0 
i2c_sensor              2944  2 w83l785ts,asb100
i2c_nforce2             5504  0 
i2c_isa                 1728  0 
i2c_dev                 8064  0 
i2c_core               18512  6 w83l785ts,asb100,i2c_sensor,i2c_nforce2,i2c_isa,i2c_dev
nvidia               3707080  12 
agpgart                28648  2 nvidia_agp,nvidia

> cat /proc/asound/cards

0 [nForce2        ]: NFORCE - NVidia nForce2
                     NVidia nForce2 with ALC650F at 0xe2083000, irq 21

> What audio stuff are you going to use this machine for, BTW?

Right now, I just want multiplexing of audio streams for listening
to music/watching movies/hearing sound effects---typical "desktop
stuff" where more than one audio stream would be nice.

I'm starting to think that maybe jack+realtime is way overkill for
what I need... Though, in the future, I intend to build an amatuer
recording studio in my home, and I'd like to use Linux and OSS apps
as my multi-tracking software and drum machine.  So I thought it
would be nice to learn jack now :)

Anyway, thank you very much for all you help!  It is very much
appreciated.

Matt

-- 
Matt Garman
email at: http://raw-sewage.net/index.php?file=email
-- 
gentoo-user@gentoo.org mailing list