[gentoo-dev] Parallizing ebuilds - 'trivial' ebuilds

[gentoo-dev] Parallizing ebuilds - 'trivial' ebuilds

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 1078 bytes --]

I've been cleaning up media-fonts/ to work with modular-X, and I see a
lot of ebuilds with stuff like this:
    for font in *.bdf; do
        /usr/X11R6/bin/bdftopcf ${font} > `basename $font .bdf`.pcf
    done                               
    gzip *.pcf

For having 100 files in *bdf, this is so serial it's painful.

While this is a daunting task to accomplish, it would be worthwhile for
developers to attack poor coding like this and replace it with nicer stuff.

The above for example should be replaced with a small makefile:
%.pcf: %.bdf
	bdftopcf $< -o $@
%.pcf.gz: %.pcf
	gzip -9 $<
BDFFILES = # files go here
PCFFILES = $(BDFFILES:.bdf=.pcf)
PCFFILES_GZ = $(BDFFILES:.bdf=.pcf.gz)
all: $(PCFFILES_GZ)
clean:
	rm -f $(PCFFILES) $(PCFFILES_GZ)

Is this really too much work to get developers to do? I don't think so.

(And no, this email wasn't prompted by one of ciaranm's diversions about SPARC having many CPUs).

-- 
Robin Hugh Johnson
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 241 bytes --]

Re: [gentoo-dev] Parallizing ebuilds - 'trivial' ebuilds

[Lisa Seelye]

[-- Attachment #1: Type: text/plain, Size: 579 bytes --]

On Wed, 2006-01-11 at 14:51 -0800, Robin H. Johnson wrote:
> I've been cleaning up media-fonts/ to work with modular-X, and I see a
> lot of ebuilds with stuff like this:
>     for font in *.bdf; do
>         /usr/X11R6/bin/bdftopcf ${font} > `basename $font .bdf`.pcf
>     done                               
>     gzip *.pcf
> 
> For having 100 files in *bdf, this is so serial it's painful.

And here I was hoping Distcc would get some usage. :(

-- 
Regards,
Lisa Seelye
GPG: 09CF5 2D6B8 2B72B 997A7 601BC B46B5 561E4 96FC5
http://www.thedoh.com/~lisa/site

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Parallizing ebuilds - 'trivial' ebuilds

[Ferris McCormick]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wed, 11 Jan 2006, Lisa Seelye wrote:

> On Wed, 2006-01-11 at 14:51 -0800, Robin H. Johnson wrote:
>> I've been cleaning up media-fonts/ to work with modular-X, and I see a
>> lot of ebuilds with stuff like this:
>>     for font in *.bdf; do
>>         /usr/X11R6/bin/bdftopcf ${font} > `basename $font .bdf`.pcf
>>     done
>>     gzip *.pcf
>>
>> For having 100 files in *bdf, this is so serial it's painful.
>
> And here I was hoping Distcc would get some usage. :(
>

Distcc gets lots of usage with modular X.  But for the fonts? :)

> -- 
> Regards,
Regards,
Ferris
- --
Ferris McCormick (P44646, MI) <fmccor@gentoo.org>
Developer, Gentoo Linux (sparc, devrel)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)

iD8DBQFDxaBWQa6M3+I///cRAv/UAKCmOh2vFoZoAstFIOoynlYk5IhdFQCgntpt
XYfbsIWLAmbdu38ksgKdmP4=
=YDDI
-----END PGP SIGNATURE-----
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Parallizing ebuilds - 'trivial' ebuilds

[Lisa Seelye]

[-- Attachment #1: Type: text/plain, Size: 847 bytes --]

On Thu, 2006-01-12 at 00:18 +0000, Ferris McCormick wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On Wed, 11 Jan 2006, Lisa Seelye wrote:
> 
> > On Wed, 2006-01-11 at 14:51 -0800, Robin H. Johnson wrote:
> >> I've been cleaning up media-fonts/ to work with modular-X, and I see a
> >> lot of ebuilds with stuff like this:
> >>     for font in *.bdf; do
> >>         /usr/X11R6/bin/bdftopcf ${font} > `basename $font .bdf`.pcf
> >>     done
> >>     gzip *.pcf
> >>
> >> For having 100 files in *bdf, this is so serial it's painful.
> >
> > And here I was hoping Distcc would get some usage. :(
> >
> 
> Distcc gets lots of usage with modular X.  But for the fonts? :)

Time for distfont? ;)

-- 
Regards,
Lisa Seelye
GPG: 09CF5 2D6B8 2B72B 997A7 601BC B46B5 561E4 96FC5
http://www.thedoh.com/~lisa/site

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Parallizing ebuilds - 'trivial' ebuilds

[Philippe Trottier]

Lisa Seelye wrote:
> On Thu, 2006-01-12 at 00:18 +0000, Ferris McCormick wrote:
> 
>>-----BEGIN PGP SIGNED MESSAGE-----
>>Hash: SHA1
>>
>>On Wed, 11 Jan 2006, Lisa Seelye wrote:
>>
>>
>>>On Wed, 2006-01-11 at 14:51 -0800, Robin H. Johnson wrote:
>>>
>>>>I've been cleaning up media-fonts/ to work with modular-X, and I see a
>>>>lot of ebuilds with stuff like this:
>>>>    for font in *.bdf; do
>>>>        /usr/X11R6/bin/bdftopcf ${font} > `basename $font .bdf`.pcf
>>>>    done
>>>>    gzip *.pcf
>>>>
>>>>For having 100 files in *bdf, this is so serial it's painful.
>>>
>>>And here I was hoping Distcc would get some usage. :(
>>>
>>
>>Distcc gets lots of usage with modular X.  But for the fonts? :)
> 
> Time for distfont? ;)

Make this distributed tool for tar zip bzip2 and gzip and I'm in, I don't think 
it would be useful with anything else than Gigabit Ethernet.

We might want to have in the make.conf 2 separate variables, one of them saying 
how many threads can be run on the machine, then How many threads/process across 
a cluster.

For example, my Dual Xeon EM64T file server can do make -j4  locally, like in 
make install, make docs etc etc, But for compiling I can use -j20, really not 
useful over -j8 anyway. But the point is, it would be usefully to separate the 
load distribution on the local machine and cluster nodes.

Philippe Trottier
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Parallizing ebuilds - 'trivial' ebuilds

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 973 bytes --]

On Thu, 2006-01-12 at 11:23 +0200, Philippe Trottier wrote:
> Make this distributed tool for tar zip bzip2 and gzip and I'm in, I don't think 
> it would be useful with anything else than Gigabit Ethernet.

app-arch/pbzip2

It covers only bzip2, but proves that it can be done.  I tend to like it
for catalyst, since it helps a lot on SMP machines.

> We might want to have in the make.conf 2 separate variables, one of them saying 
> how many threads can be run on the machine, then How many threads/process across 
> a cluster.
> 
> For example, my Dual Xeon EM64T file server can do make -j4  locally, like in 
> make install, make docs etc etc, But for compiling I can use -j20, really not 
> useful over -j8 anyway. But the point is, it would be usefully to separate the 
> load distribution on the local machine and cluster nodes.

-- 
Chris Gianelloni
Release Engineering - Strategic Lead
x86 Architecture Team
Games - Developer
Gentoo Linux

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[gentoo-dev] Re: Parallizing ebuilds - 'trivial' ebuilds

[Duncan]

Chris Gianelloni posted <1137075087.14567.1.camel@cgianelloni.nuvox.net>,
excerpted below,  on Thu, 12 Jan 2006 09:11:27 -0500:

> app-arch/pbzip2
> 
> It covers only bzip2, but proves that it can be done.  I tend to like it
> for catalyst, since it helps a lot on SMP machines.

Very cool!  I didn't know that existed.  It could be quite useful here on
my dual Opteron.  Thanks!

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman in
http://www.linuxdevcenter.com/pub/a/linux/2004/12/22/rms_interview.html


-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: Parallizing ebuilds - 'trivial' ebuilds

[Simon Stelling]

Duncan wrote:
>>app-arch/pbzip2
>>
>>It covers only bzip2, but proves that it can be done.  I tend to like it
>>for catalyst, since it helps a lot on SMP machines.
> 
> 
> Very cool!  I didn't know that existed.  It could be quite useful here on
> my dual Opteron.  Thanks!

For those who wondered what might be the purpose of that mail, re-read the last 
sentence with stress on the last three words.

SCNR

-- 
Simon Stelling
Gentoo/AMD64 Operational Co-Lead
blubb@gentoo.org
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Parallizing ebuilds - 'trivial' ebuilds

[Kalin KOZHUHAROV]

Philippe Trottier wrote:
> Lisa Seelye wrote:
> 
>> On Thu, 2006-01-12 at 00:18 +0000, Ferris McCormick wrote:
>>
>>> -----BEGIN PGP SIGNED MESSAGE-----
>>> Hash: SHA1
>>>
>>> On Wed, 11 Jan 2006, Lisa Seelye wrote:
>>>
>>>
>>>> On Wed, 2006-01-11 at 14:51 -0800, Robin H. Johnson wrote:
>>>>
>>>>> I've been cleaning up media-fonts/ to work with modular-X, and I see a
>>>>> lot of ebuilds with stuff like this:
>>>>>    for font in *.bdf; do
>>>>>        /usr/X11R6/bin/bdftopcf ${font} > `basename $font .bdf`.pcf
>>>>>    done
>>>>>    gzip *.pcf
>>>>>
>>>>> For having 100 files in *bdf, this is so serial it's painful.
>>>>
>>>>
>>>> And here I was hoping Distcc would get some usage. :(
>>>>
>>>
>>> Distcc gets lots of usage with modular X.  But for the fonts? :)
>>
>>
>> Time for distfont? ;)
> 
> 
> Make this distributed tool for tar zip bzip2 and gzip and I'm in, I
> don't think it would be useful with anything else than Gigabit Ethernet.
> 
> We might want to have in the make.conf 2 separate variables, one of them
> saying how many threads can be run on the machine, then How many
> threads/process across a cluster.
> 
> For example, my Dual Xeon EM64T file server can do make -j4  locally,
> like in make install, make docs etc etc, But for compiling I can use
> -j20, really not useful over -j8 anyway. But the point is, it would be
> usefully to separate the load distribution on the local machine and
> cluster nodes.

As the discusison started...

I would like to be able to limit the -jN when there is no distcc host
available or when compiling c++ code, otherwise my poor laptop is dead with
-j5 compiling pwlib when the network is down....

It is particular example, but being able to limit portage in some way as
total CPU, total MEM might be interesting (just nice-ing is not enough)

Kalin.
-- 
|[ ~~~~~~~~~~~~~~~~~~~~~~ ]|
+-> http://ThinRope.net/ <-+
|[ ______________________ ]|

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Parallizing ebuilds - 'trivial' ebuilds

[Patrick Lauer]

[-- Attachment #1: Type: text/plain, Size: 1765 bytes --]

On Fri, 2006-01-13 at 19:53 +0900, Kalin KOZHUHAROV wrote:
> > Make this distributed tool for tar zip bzip2 and gzip and I'm in, I
> > don't think it would be useful with anything else than Gigabit Ethernet.
One 2Ghz CPU can't even saturate a 100Mbit line with bzip2 as far as I
can tell.
Although the speedups won't be extreme it could just work.

> > We might want to have in the make.conf 2 separate variables, one of them
> > saying how many threads can be run on the machine, then How many
> > threads/process across a cluster.
> > 
> > For example, my Dual Xeon EM64T file server can do make -j4  locally,
> > like in make install, make docs etc etc, But for compiling I can use
> > -j20, really not useful over -j8 anyway. But the point is, it would be
> > usefully to separate the load distribution on the local machine and
> > cluster nodes.
> 
> As the discusison started...
> 
> I would like to be able to limit the -jN when there is no distcc host
> available or when compiling c++ code, otherwise my poor laptop is dead with
> -j5 compiling pwlib when the network is down....
As far as I can tell distcc isn't smart enough for dynamic load balancing.
One could hack portage to "test" each server in the distcc host list and
remove missing servers for each run - doesn't look elegant to me.

> It is particular example, but being able to limit portage in some way as
> total CPU, total MEM might be interesting (just nice-ing is not enough)
Very difficult - usually gcc uses ~25M per process (small source files), but 
I've seen >100M (most larger C++ files) and heard of ~600M per process for MySQL

Limiting that is beyond the scope of portage.

wkr,
Patrick
-- 
Stand still, and let the rest of the universe move

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-dev] Re: Parallizing ebuilds - 'trivial' ebuilds

[Duncan]

Patrick Lauer posted <1137150388.5924.44.camel@localhost>, excerpted
below,  on Fri, 13 Jan 2006 12:06:28 +0100:

> Very difficult - usually gcc uses ~25M per process (small source files),
> but I've seen >100M (most larger C++ files) and heard of ~600M per
> process for MySQL
> 
> Limiting that is beyond the scope of portage.

There's one point in the kmail/kdepim (split/monolithic) build where with
USE=kdeenablefinal on AMD64, a single process takes > 700 meg, based on my
results.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman in
http://www.linuxdevcenter.com/pub/a/linux/2004/12/22/rms_interview.html


-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Parallizing ebuilds - 'trivial' ebuilds

[Kalin KOZHUHAROV]

Patrick Lauer wrote:
> On Fri, 2006-01-13 at 19:53 +0900, Kalin KOZHUHAROV wrote:
> 
>>>Make this distributed tool for tar zip bzip2 and gzip and I'm in, I
>>>don't think it would be useful with anything else than Gigabit Ethernet.
> 
> One 2Ghz CPU can't even saturate a 100Mbit line with bzip2 as far as I
> can tell.
> Although the speedups won't be extreme it could just work.
> 
> 
>>>We might want to have in the make.conf 2 separate variables, one of them
>>>saying how many threads can be run on the machine, then How many
>>>threads/process across a cluster.
>>>
>>>For example, my Dual Xeon EM64T file server can do make -j4  locally,
>>>like in make install, make docs etc etc, But for compiling I can use
>>>-j20, really not useful over -j8 anyway. But the point is, it would be
>>>usefully to separate the load distribution on the local machine and
>>>cluster nodes.
>>
>>As the discusison started...
>>
>>I would like to be able to limit the -jN when there is no distcc host
>>available or when compiling c++ code, otherwise my poor laptop is dead with
>>-j5 compiling pwlib when the network is down....
> 
> As far as I can tell distcc isn't smart enough for dynamic load balancing.
> One could hack portage to "test" each server in the distcc host list and
> remove missing servers for each run - doesn't look elegant to me.

Yes, might be a solution, even if not elegant. I am thinking also of
automating distcc configuration (i.e. no need to run --set-hosts) and one
idea is to use DNS with some TXT record, but that is just an idea - no
patching is done yet.

Not sure if distcc has local limiter, i.e. if it it set with "localhost/2"
and portage user (or some other user != root) tries to start 3 processes,
the 3rd just blocks (and not take memory). I think not, so this thing might
be interesting to implement (for "old" laptops with less memory).

I think I should resubscribe to the distcc list :-)


>>It is particular example, but being able to limit portage in some way as
>>total CPU, total MEM might be interesting (just nice-ing is not enough)
> 
> Very difficult - usually gcc uses ~25M per process (small source files), but 
> I've seen >100M (most larger C++ files) and heard of ~600M per process for MySQL
> 
> Limiting that is beyond the scope of portage.

Hmm, may be not limiting the total usage, but more like just adjusting
MAKEOPTS='-j1' in some cases (NOTE: to /me, define "some cases").
Implementing the above "Local limiter" in distcc will solve that automagically.

Kalin.

-- 
|[ ~~~~~~~~~~~~~~~~~~~~~~ ]|
+-> http://ThinRope.net/ <-+
|[ ______________________ ]|

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Parallizing ebuilds - 'trivial' ebuilds

[Francesco Riosa]

>>> I would like to be able to limit the -jN when there is no distcc host
>>> available or when compiling c++ code, otherwise my poor laptop is dead with
>>> -j5 compiling pwlib when the network is down....
>> As far as I can tell distcc isn't smart enough for dynamic load balancing.
>> One could hack portage to "test" each server in the distcc host list and
>> remove missing servers for each run - doesn't look elegant to me.
> dn
> Yes, might be a solution, even if not elegant. I am thinking also of
> automating distcc configuration (i.e. no need to run --set-hosts) and one
> idea is to use DNS with some TXT record, but that is just an idea - no
> patching is done yet.

you may be interested in the SRV DNS record, take a look at
http://www.dns-sd.org/ServiceTypes.html

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Parallizing ebuilds - 'trivial' ebuilds

[Philippe Trottier]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Kalin KOZHUHAROV wrote:
> Patrick Lauer wrote:
>> On Fri, 2006-01-13 at 19:53 +0900, Kalin KOZHUHAROV wrote:
>>
>>>> Make this distributed tool for tar zip bzip2 and gzip and I'm in, I
>>>> don't think it would be useful with anything else than Gigabit Ethernet.
>> One 2Ghz CPU can't even saturate a 100Mbit line with bzip2 as far as I
>> can tell.
>> Although the speedups won't be extreme it could just work.
>>
>>
>>>> We might want to have in the make.conf 2 separate variables, one of them
>>>> saying how many threads can be run on the machine, then How many
>>>> threads/process across a cluster.
>>>>
>>>> For example, my Dual Xeon EM64T file server can do make -j4  locally,
>>>> like in make install, make docs etc etc, But for compiling I can use
>>>> -j20, really not useful over -j8 anyway. But the point is, it would be
>>>> usefully to separate the load distribution on the local machine and
>>>> cluster nodes.
>>> As the discusison started...
>>>
>>> I would like to be able to limit the -jN when there is no distcc host
>>> available or when compiling c++ code, otherwise my poor laptop is dead with
>>> -j5 compiling pwlib when the network is down....
>> As far as I can tell distcc isn't smart enough for dynamic load balancing.
>> One could hack portage to "test" each server in the distcc host list and
>> remove missing servers for each run - doesn't look elegant to me.
> 
> Yes, might be a solution, even if not elegant. I am thinking also of
> automating distcc configuration (i.e. no need to run --set-hosts) and one
> idea is to use DNS with some TXT record, but that is just an idea - no
> patching is done yet.

Recipe for disaster, specially in a place like mine where sparc, alpha, x86_64
and ppc32/64 mix... not counting ia64 for a test run soon...

If you really want to do this, someone has to make a rendezvous a la Apple.
Where not only distcc says I am available but I am also doing the right stuff.



-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFDx/nsP0/FkJ0eBc0RAn8aAJ9h8C1QFO0HyQmBINl2erPljgaEYwCfcOBF
yPpDJ/KElWfotWqNvWYPq3s=
=XM29
-----END PGP SIGNATURE-----
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Parallizing ebuilds - 'trivial' ebuilds

[Philipp Riegger]

On Jan 13, 2006, at 8:05 PM, Philippe Trottier wrote:

> Recipe for disaster, specially in a place like mine where sparc,  
> alpha, x86_64
> and ppc32/64 mix... not counting ia64 for a test run soon...
>
> If you really want to do this, someone has to make a rendezvous a  
> la Apple.
> Where not only distcc says I am available but I am also doing the  
> right stuff.

This all is something, icecream[1] can do, i think. I am very  
interested in getting icecream to work with gentoo.

[1] http://wiki.kde.org/icecream

Philipp
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Parallizing ebuilds - 'trivial' ebuilds

[Kalin KOZHUHAROV]

Philipp Riegger wrote:
> On Jan 13, 2006, at 8:05 PM, Philippe Trottier wrote:
> 
>> Recipe for disaster, specially in a place like mine where sparc,
>> alpha, x86_64
>> and ppc32/64 mix... not counting ia64 for a test run soon...
>>
>> If you really want to do this, someone has to make a rendezvous a la
>> Apple.
>> Where not only distcc says I am available but I am also doing the
>> right stuff.
> 
> This all is something, icecream[1] can do, i think. I am very interested
> in getting icecream to work with gentoo.
> 
> [1] http://wiki.kde.org/icecream

Yeah, this looks nice, will give it a try. Unfortunately it does not seem a
drop-in-replacement for distcc (see the bottom of [1] above for Gentoo part)

Kalin.

-- 
|[ ~~~~~~~~~~~~~~~~~~~~~~ ]|
+-> http://ThinRope.net/ <-+
|[ ______________________ ]|

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Parallizing ebuilds - 'trivial' ebuilds

[Colin Kingsley]

Kalin KOZHUHAROV wrote:
> I would like to be able to limit the -jN when there is no distcc host
> available or when compiling c++ code, otherwise my poor laptop is dead with
> -j5 compiling pwlib when the network is down....
> 
> It is particular example, but being able to limit portage in some way as
> total CPU, total MEM might be interesting (just nice-ing is not enough)

You might be interested in `man make`. The '-l' option in particular 
would probably do what you want.

Tercel
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Parallizing ebuilds - 'trivial' ebuilds

[Kevin F. Quinn (Gentoo)]

[-- Attachment #1: Type: text/plain, Size: 1071 bytes --]

On Wed, 11 Jan 2006 14:51:46 -0800
"Robin H. Johnson" <robbat2@gentoo.org> wrote:

> I've been cleaning up media-fonts/ to work with modular-X, and I see a
> lot of ebuilds with stuff like this:
>     for font in *.bdf; do
>         /usr/X11R6/bin/bdftopcf ${font} > `basename $font .bdf`.pcf
>     done                               
>     gzip *.pcf
>
> ['make' suggestion and example makefile snipped]

You could just do it directly in bash:

    for font in *.bdf; do
        pcf=`basename $font`.pcf && \
          /usr/X11R6/bin/bdftopcf ${font} > ${pcf} &&
          gzip ${pcf} &
    done
    wait

You may want to limit the number of parallel processes somewhat (as
with 'make -jN'), which is also easy enough to do:

    n=${MAX_PARALLEL}
    for font in *.bdf; do
        pcf=`basename $font .bdf`.pcf && \
          /usr/X11R6/bin/bdftopcf ${font} > ${pcf} && \
          gzip ${pcf} &
        [[ ${n} -eq 0 ]] && wait && n=${MAX_PARALLEL}
       ((n=${n}-1))
    done
    wait

-- 
Kevin F. Quinn


-- 
Kevin F. Quinn

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]