[gentoo-amd64] experiences with Tyan S4881?

[gentoo-amd64] experiences with Tyan S4881?

[Tamas Karpati]

Dear all,

After no success at the gentoo-server list I hope to get some opinions here
on Tyan's S4881 mob with 4 or 8 professors and memory usage.

Please execuse me if you see this message for the second time.

Do you have any experience with an S4881 with 4 single or dual core Opterons?
The goal is to have a system flexible enough to run
 - a single thread job using all the available memory and
 - one or more multithreaded jobs sharing the physical RAM.

Have you observed any problems? About dual core processors?

I have good experiences with Gentoo on dual AMD (both Athlon and
Opteron) systems,
as well as with dual core Intel so I just want to be sure before
recommending the
S4881+Gentoo combination for scientific computations...

Any hints are welcome,
 Tamas
-- 
gentoo-amd64@gentoo.org mailing list

[gentoo-amd64] Re: experiences with Tyan S4881?

[Duncan]

"Tamas Karpati" <tkarpati@gmail.com> posted
bd453d7f0606200445o335db209rc64b9a3940c22314@mail.gmail.com, excerpted
below, on  Tue, 20 Jun 2006 11:45:39 +0000:

> I have good experiences with Gentoo on dual AMD (both Athlon and Opteron)
> systems,
> as well as with dual core Intel so I just want to be sure before
> recommending the
> S4881+Gentoo combination for scientific computations...

Tyan is one of the better choices for high-end multi-socket solutions on
Linux.  They tend to be one of the more Linux friendly board
manufacturers.  

I'm not running anything quite as high end as this board, but I am running
a dual-socket s2885 that I've been /very/ happy with, originally
single-core but now dual-core ready due to a BIOS update (I'll probably
be upgrading to dual-cores later this year). Before purchase, I verified
that Linux supported the chipset and various board hardware had drivers.
Tyan even had an lm_sensors config file for the board!  They also have
various Linux specific support pages. After purchase, I noticed the manual
even listed certifications on Red Hat, TurboLinux, and SuSE.  

I haven't checked recently but back then (~2 years ago now) the Linux
support pages did still assume access to an MSDOS floppy for BIOS updates.
I wrote in, asking about FreeDOS for BIOS updates, as I no longer have
anything MS around, noting that one manufacturer (ASUS maybe? IDR)
actually shipped FreeDOS BIOS update floppy images. The reply was friendly
but non-committal. After I went ahead and tried it, I wrote back
confirming that it worked, and suggesting they at least add it to their
Linux documentation pages as a suggestion from a user, or something of the
like.  Again, I've not checked to see if they actually did it or not, but
I've used FreeDOS images for all my updates since then, mounting the
generic FreeDOS OEM image and copying the BIOS updater to it before dd-ing
the image directly to the floppy.

Checking the tyan pages for that board, I don't see the 4881 listed for an
lm_sensors config file (there's one for the 4882).  I'd verify the chipset
and hardware has Linux drivers, then, given the price of that class of
solution, I'd mail Tyan directly and ask about it.  As I said, they are
quite Linux friendly and will likely be able to tell you it's certified
with various Linux distributions.  If not, they'll almost certainly be
able to point you to one that is.

Or...  try the 4882-D, if you don't specifically need PCI-E and can use
PCI-X instead.  It's the same base AMD chipset I'm running in my 2885, and
it does have an lm_sensors config available on the Tyan site, so support
for it should be a given.  If I were in the market for a 4-socket model,
I'd certainly not hesitate on it, given my experience on this one.

Meanwhile, there's another angle you can try.  A number of Gentoo devs
(and users) have participated in the Gentoo exhibit at various
conferences.  Given the from-source nature of Gentoo, it's popular with
various hardware exhibitors as well, and I recall mention in GWN and on
the dev list of at least a couple times when Gentoo was installed on 4-way
or higher systems "live" at the conference exhibit.  Going back thru the
GWN issues and getting in touch with the devs involved to get the details
on what the hardware was exactly might be informative and give you
additional specific details of Gentoo on specific hardware.



-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Re: experiences with Tyan S4881?

[Ryan A. Rice]

We sell a configuration that utilizes the S4881.  It works very well 
with nix.  We loaded Gentoo on it at LinuxWorld '06 in Boston.  
(actually, some of the gentoo guys, namely Chris Gianelloni loaded it)

We also had the M4881 plug in there, allowing us 16 cores with 64GB ram 
installed.  See the GWN on it :-)

http://www.gentoo.org/news/en/gwn/20060417-newsletter.xml

Regards,

Ryan A. Rice
Operations Manager
i*hydra, Where Price and Power Converge.
http://www.ihydra.com
rrice@ihydra.com
Direct: (207) 226-7360
Toll Free: (866)-433-1238
Fax: (207)-563-6601

Any and all information/attachments are strictly proprietary and confidential.  If you have received this message in error, please delete it immediately, and notify rrice@ihydra.com  Thank you. 

+---------------------------------+



Duncan wrote:
> "Tamas Karpati" <tkarpati@gmail.com> posted
> bd453d7f0606200445o335db209rc64b9a3940c22314@mail.gmail.com, excerpted
> below, on  Tue, 20 Jun 2006 11:45:39 +0000:
>
>   
>> I have good experiences with Gentoo on dual AMD (both Athlon and Opteron)
>> systems,
>> as well as with dual core Intel so I just want to be sure before
>> recommending the
>> S4881+Gentoo combination for scientific computations...
>>     
>
> Tyan is one of the better choices for high-end multi-socket solutions on
> Linux.  They tend to be one of the more Linux friendly board
> manufacturers.  
>
> I'm not running anything quite as high end as this board, but I am running
> a dual-socket s2885 that I've been /very/ happy with, originally
> single-core but now dual-core ready due to a BIOS update (I'll probably
> be upgrading to dual-cores later this year). Before purchase, I verified
> that Linux supported the chipset and various board hardware had drivers.
> Tyan even had an lm_sensors config file for the board!  They also have
> various Linux specific support pages. After purchase, I noticed the manual
> even listed certifications on Red Hat, TurboLinux, and SuSE.  
>
> I haven't checked recently but back then (~2 years ago now) the Linux
> support pages did still assume access to an MSDOS floppy for BIOS updates.
> I wrote in, asking about FreeDOS for BIOS updates, as I no longer have
> anything MS around, noting that one manufacturer (ASUS maybe? IDR)
> actually shipped FreeDOS BIOS update floppy images. The reply was friendly
> but non-committal. After I went ahead and tried it, I wrote back
> confirming that it worked, and suggesting they at least add it to their
> Linux documentation pages as a suggestion from a user, or something of the
> like.  Again, I've not checked to see if they actually did it or not, but
> I've used FreeDOS images for all my updates since then, mounting the
> generic FreeDOS OEM image and copying the BIOS updater to it before dd-ing
> the image directly to the floppy.
>
> Checking the tyan pages for that board, I don't see the 4881 listed for an
> lm_sensors config file (there's one for the 4882).  I'd verify the chipset
> and hardware has Linux drivers, then, given the price of that class of
> solution, I'd mail Tyan directly and ask about it.  As I said, they are
> quite Linux friendly and will likely be able to tell you it's certified
> with various Linux distributions.  If not, they'll almost certainly be
> able to point you to one that is.
>
> Or...  try the 4882-D, if you don't specifically need PCI-E and can use
> PCI-X instead.  It's the same base AMD chipset I'm running in my 2885, and
> it does have an lm_sensors config available on the Tyan site, so support
> for it should be a given.  If I were in the market for a 4-socket model,
> I'd certainly not hesitate on it, given my experience on this one.
>
> Meanwhile, there's another angle you can try.  A number of Gentoo devs
> (and users) have participated in the Gentoo exhibit at various
> conferences.  Given the from-source nature of Gentoo, it's popular with
> various hardware exhibitors as well, and I recall mention in GWN and on
> the dev list of at least a couple times when Gentoo was installed on 4-way
> or higher systems "live" at the conference exhibit.  Going back thru the
> GWN issues and getting in touch with the devs involved to get the details
> on what the hardware was exactly might be informative and give you
> additional specific details of Gentoo on specific hardware.
>
>
>
>   
-- 
gentoo-amd64@gentoo.org mailing list

[gentoo-amd64] Re: Re: experiences with Tyan S4881?

[Duncan]

"Ryan A. Rice" <rrice@ihydra.com> posted 4499AD6B.1070609@ihydra.com,
excerpted below, on  Wed, 21 Jun 2006 16:34:51 -0400:

> We sell a configuration that utilizes the S4881.  It works very well 
> with nix.  We loaded Gentoo on it at LinuxWorld '06 in Boston.  
> (actually, some of the gentoo guys, namely Chris Gianelloni loaded it)
> 
> We also had the M4881 plug in there, allowing us 16 cores with 64GB ram 
> installed.  See the GWN on it :-)
> 
> http://www.gentoo.org/news/en/gwn/20060417-newsletter.xml

There you go. =8^)  I did suggest checking old GWNs for conference exhibit
stories.  Sure enough!

I was looking at that 4-way with 4-way plugin board thing, and thinking
about dual-cores on it, and my mouth was watering!  That would be
seriously fun to play with!  =8^)  OTOH, how slow my old 2-way single-core
with 8 gigs of RAM and a 4-disk RAID-6 desktop/workstation would seem
afterward! =8^(  I AM looking forward to the dual-core upgrade, later this
year, however. =8^)



-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Re: Re: experiences with Tyan S4881?

[Tamas Karpati]

Hi Duncan & Ryan,

Thanks for the hints so far. I'm going to check out Tyan's support for Linux
before getting more astonished (by the possibilities Duncan mentioned :)

I think the 4-way plugin and the possibility to substitute single to dual core
procs are one of S4881's strengths but I'm not sure that next year or later
(when one gets the money) it will still be possible to order the plugin.

One more promising fact is that I haven't seen the half-of-four gigs waste of
RAM with Tyan S2882 (as opposed to MSI MasterX-Far mobs which,
in addition, wouldn't let me install >4G). As I'v learnt from one of
Duncan's earlier messages, it was most probably due to address space
allocation for various buses by the BIOS. On S2882, I could also mix up RAM
modules of various sizes and clock rates (I suppose the mob syncronized to
the slowest) on any of the two ranks...

Anyways, I hope memory can be shared on S4881 (or -D, S4882) as well
as on S2882, meaning that any thread can utilize any part of (or all) the
physical memory mounted on the mob. They want to use all the 64G memory
(only "all" because (i) no mob supports more and (ii) resources are limited).

Thanks again,
  Tamas
-- 
gentoo-amd64@gentoo.org mailing list

[gentoo-amd64] Re: Re: Re: experiences with Tyan S4881?

[Duncan]

"Tamas Karpati" <tkarpati@gmail.com> posted
bd453d7f0606220440n65ae201ftc7a5ebf17416eb8e@mail.gmail.com, excerpted
below, on  Thu, 22 Jun 2006 13:40:48 +0200:

> Anyways, I hope memory can be shared on S4881 (or -D, S4882) as well
> as on S2882, meaning that any thread can utilize any part of (or all) the
> physical memory mounted on the mob. They want to use all the 64G memory
> (only "all" because (i) no mob supports more and (ii) resources are limited).

That should be no problem.  It's part of the AMD64 architecture, and
anything /not/ supporting it would therefore be considered crippled.

I'm not sure about 4-way, tho I would assume it would be similar to 2-way,
but with my 2-way (s2885), the BIOS has memory per-node and across-node
interleaving options.  For highest bandwidth access by a single thread,
one turns on the across-node interleaving.  For running multiple
threads, where each thread has faster access to the memory local to the
processor/node it is on, but /can/ access memory cross-nodes if necessary,
one turns off across-node interleaving, and enabled NUMA mode memory
management in the kernel.  The kernel process and memory control then
attempts to keep the processes balanced both by CPU use and memory use,
also accounting for threads sharing memory and trying to keep them on the
same node and therefore both accessing local memory, if possible. 
However, any thread or process can still access memory on any other node
if need be.  (There are separate kernel options available to wall off
particular applications, limiting them to specific CPUs and memory, if
desired, but I know little about these beyond the fact that they are still
somewhat experimental, from what I've read.)



-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Re: Re: Re: experiences with Tyan S4881?

[Vladimir G. Ivanovic]

FYI: The Daily Tech has a story on the next generation of Opterons that
have just been announced by AMD.

http://dailytech.com/article.aspx?newsid=2971

--- Vladimir

-- 
Vladimir G. Ivanovic <vgivanovic@comcast.net>
-- 
gentoo-amd64@gentoo.org mailing list

[gentoo-amd64] Re: Re: Re: Re: experiences with Tyan S4881?

[Duncan]

"Vladimir G. Ivanovic" <vgivanovic@comcast.net> posted
1151002139.5761.37.camel@scarlatti.leonora.org, excerpted below, on  Thu,
22 Jun 2006 11:48:59 -0700:

> FYI: The Daily Tech has a story on the next generation of Opterons that
> have just been announced by AMD.
> 
> http://dailytech.com/article.aspx?newsid=2971

So they'll move to quad-core in 2K7, shortly after I upgrade my
dual-socket-940 system to dual-cores later this year.  However, at that,
having already upgraded memory on this board to 8 gig, I think I'll hold
for awhile (2 years minimum, possibly 4-5).

Observation: the hardware folks should be really glad I switched to Linux.
Since then, the money that /used/ to be split between software (mostly MS)
and hardware has been all hardware!  =8^)  However, I've been headed to
the point of diminishing returns, and my hardware now does what I need it
to do, so it'll be time to hold and wait for technology to catch up and
make my next upgrade to what by then will be middle of the line again, tho
it'll be a big upgrade for me.



-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Re: Re: Re: Re: experiences with Tyan S4881?

[William Tetrault]

On Thursday 22 June 2006 23:28, Duncan wrote:
>
> Observation: the hardware folks should be really glad I switched to Linux.
> Since then, the money that /used/ to be split between software (mostly MS)
> and hardware has been all hardware!  =8^)  However, I've been headed to
> the point of diminishing returns, and my hardware now does what I need it
> to do, so it'll be time to hold and wait for technology to catch up and
> make my next upgrade to what by then will be middle of the line again, tho
> it'll be a big upgrade for me.

That's a very acute observation, too.  Makes one wonder why, this being the 
case, there isn't more and better support for open source!

Me, I'm salivating for the K8L's, regardless of the number of cores.
-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Re: Re: Re: Re: experiences with Tyan S4881?

[Tamas Karpati]

Hi,

(Just back from a short holiday...)

Thanks for all the comments on the  S4881 subject!

Cheers,
  Tamas
-- 
gentoo-amd64@gentoo.org mailing list