[gentoo-mirrors] Configuration Request

[gentoo-mirrors] Configuration Request

[Brath, Shane]

I know this is a bit off topic, but I'm trying to tune my mirrors server to
perform as fast as possible.

I'm looking at a new hardware purchase to add more capacity, but it's
difficult to know if the new hardware is enough.
The general config is a HP DL380 2x3.4Ghz processors, 4G ram, Dual Raid
Controllers (SmartArray 6404) with 256M/each, and 2x14 drive HP Drive Arrays
(HP MSA 30 JBOD) with 73G drives at 15Krpm. GigE.

Are any sites willing to share their "Config" I'm interested in the hardware
setup, as well as the OS/Filesystem/Layout/Raid, type of thing.

Any help would be great.

Thanks in advance.

> Shane S Brath
	mirrors@tds.net



--
gentoo-mirrors@gentoo.org mailing list

Re: [gentoo-mirrors] Configuration Request

[Andrew D. Clark]

Time to show off my hardware :)

ftp.ucsb.edu is a FreeBSD 4.8 box, 512MB RAM, 733mhz PIII with a 
100Mb/fdx connection.  Storage for the ftp stuff is a JetStor III IDE 
box.  8 80GB IDE drives in a RAID 5 with a U160 connection back to the 
host.  I'm planning on adding another 1GB of RAM in the very near 
future and about 1.5TB more disk in the not so near future (stealing a 
12 bay Jetstor III IDE RAID with 100GB disks off another box I plan to 
kill fairly soon)
In November, the ftp server pushed 1317GB of Gentoo, 1593GB total for 
all the content on the box.

--
Andrew Clark
Campus Network Programmer
Office of Information Technology
University of California, Santa Barbara
andrew.clark@ucsb.edu (805) 893-5311

--On Wednesday, December 01, 2004 10:34:17 AM -0600 "Brath, Shane" 
<shane.brath@tdstelecom.com> wrote:

>
> I know this is a bit off topic, but I'm trying to tune my mirrors
> server to perform as fast as possible.
>
> I'm looking at a new hardware purchase to add more capacity, but it's
> difficult to know if the new hardware is enough.
> The general config is a HP DL380 2x3.4Ghz processors, 4G ram, Dual
> Raid Controllers (SmartArray 6404) with 256M/each, and 2x14 drive HP
> Drive Arrays (HP MSA 30 JBOD) with 73G drives at 15Krpm. GigE.
>
> Are any sites willing to share their "Config" I'm interested in the
> hardware setup, as well as the OS/Filesystem/Layout/Raid, type of
> thing.
>
> Any help would be great.
>
> Thanks in advance.
>
>> Shane S Brath
> 	mirrors@tds.net
>
>
>
> --
> gentoo-mirrors@gentoo.org mailing list
>



--
gentoo-mirrors@gentoo.org mailing list

Re: [gentoo-mirrors] Configuration Request

[Rob Baxter]

OS: gentoo of course. 2.6 kernel (gentoo-dev-sources)
Filesystem: reiserfs. in my experience, ext3 may be the norm, but it is damned slow.
I highly recommend vsftpd if you're running an ftp mirror, it's very secure >:]
Besides the obvious, it is also extremely fast and can take a beating.

There are a few handy config files here:
http://criticaldamage.com/forum/ - Linux/Config Share
the most useful for you would probably be the rsync server run from a ramdrive, which is so much 
faster than disk it's not funny and the the vsftpd config if you choose to use it.

If you're looking for anything more specific, let me know off list.

Rob


Brath, Shane wrote:
> I know this is a bit off topic, but I'm trying to tune my mirrors server to
> perform as fast as possible.
> 
> I'm looking at a new hardware purchase to add more capacity, but it's
> difficult to know if the new hardware is enough.
> The general config is a HP DL380 2x3.4Ghz processors, 4G ram, Dual Raid
> Controllers (SmartArray 6404) with 256M/each, and 2x14 drive HP Drive Arrays
> (HP MSA 30 JBOD) with 73G drives at 15Krpm. GigE.
> 
> Are any sites willing to share their "Config" I'm interested in the hardware
> setup, as well as the OS/Filesystem/Layout/Raid, type of thing.
> 
> Any help would be great.
> 
> Thanks in advance.
> 
> 
>>Shane S Brath
> 
> 	mirrors@tds.net
> 
> 
> 
> --
> gentoo-mirrors@gentoo.org mailing list
> 


--
gentoo-mirrors@gentoo.org mailing list

Re: [gentoo-mirrors] Configuration Request

[Sebastian Werner]

* Brath, Shane <shane.brath@tdstelecom.com> [041201 17:34]:
> I know this is a bit off topic, but I'm trying to tune my mirrors server to
> perform as fast as possible.
> 
> I'm looking at a new hardware purchase to add more capacity, but it's
> difficult to know if the new hardware is enough.
> The general config is a HP DL380 2x3.4Ghz processors, 4G ram, Dual Raid
> Controllers (SmartArray 6404) with 256M/each, and 2x14 drive HP Drive Arrays
> (HP MSA 30 JBOD) with 73G drives at 15Krpm. GigE.
> 
> Are any sites willing to share their "Config" I'm interested in the hardware
> setup, as well as the OS/Filesystem/Layout/Raid, type of thing.

yap, time for some hardware showoff :P

ftp.uni-erlangen.de is running at a dual 3ghz xeon setup using 4gb ram.
(in fact it IS a HP/Compaq DL380)

our diskspace is 4tb on an external transtec ata<->u320scsi array using raid5.
this array is running at xfs with some individually tuned parameters.

we are using suse 9.1, running a suse-out-of-the-box 2.4.21-243-smp4G
but we will soon switch over to a brand new DL380 with dual 3,4ghz with sles9.
(we switch, cause the current server will be a dedicated rsync frontend)

as you can see at  http://www.ftp.uni-erlangen.de/server/
we constantly run under pretty nice load having total traffic of about 29 TB last month.
(gentoo was about 4 TB last month).
our outbound connection is currently 1gig.

we are using apache2 with mpm-worker for http-service, which performed great during
the firefox-burn-in (at over 1300 threads with load 30 at ~600mbit avg output).
ftp service is currently done by pureftpd, cause we need some features of it, that
vsftpd does not provide.
thats bad, cause vsftpd is supposed to do a better job than vsftpd.

if you need specific setup parameters, feel free to contact me.

hope this helps.

--                                                                                                                                                                    
Sebastian Werner

--
gentoo-mirrors@gentoo.org mailing list

Re: [gentoo-mirrors] Configuration Request

[Georgi Georgiev]

maillog: 01/12/2004-14:06:11(-0700): Rob Baxter types
> OS: gentoo of course. 2.6 kernel (gentoo-dev-sources)
> Filesystem: reiserfs. in my experience, ext3 may be the norm, but it is 
> damned slow.
> I highly recommend vsftpd if you're running an ftp mirror, it's very secure 
> >:]
> Besides the obvious, it is also extremely fast and can take a beating.
> 
> There are a few handy config files here:
> http://criticaldamage.com/forum/ - Linux/Config Share
> the most useful for you would probably be the rsync server run from a 
> ramdrive, which is so much faster than disk it's not funny and the the 
> vsftpd config if you choose to use it.

Naaah, ramdrive. The whole thing is constantly cached anyway if you have
enough RAM (1GB does the job in my case). It of course depends on how
busy your rsync server is, but if it's not -- why do you care about
speed?

-- 
 >   Georgi Georgiev    > "Given the choice between accomplishing       >
<     chutz@gg3.net    <  something and just lying around, I'd         <
 >  +81(90)6266-1163    > rather lie around. No contest." ng -- Eric    >
<  ------------------- <  Clapton                                      <

--
gentoo-mirrors@gentoo.org mailing list

Re: [gentoo-mirrors] Configuration Request

[Rob Baxter]

i think the point is that the data that gets cached can and does change every 30 minutes, so caching 
doesn't really help. all 5 servers in the rsync.gentoo.org rotation are currently running rsync out 
of ram. i think i can safely say they wouldn't be nearly as fast as they are, running off a hard 
disk. besides, ram is cheap and has a small footprint (smaller blocks) using ramdrive, why not use it.

rob

Georgi Georgiev wrote:
> maillog: 01/12/2004-14:06:11(-0700): Rob Baxter types
> 
>>OS: gentoo of course. 2.6 kernel (gentoo-dev-sources)
>>Filesystem: reiserfs. in my experience, ext3 may be the norm, but it is 
>>damned slow.
>>I highly recommend vsftpd if you're running an ftp mirror, it's very secure 
>>
>>>:]
>>
>>Besides the obvious, it is also extremely fast and can take a beating.
>>
>>There are a few handy config files here:
>>http://criticaldamage.com/forum/ - Linux/Config Share
>>the most useful for you would probably be the rsync server run from a 
>>ramdrive, which is so much faster than disk it's not funny and the the 
>>vsftpd config if you choose to use it.
> 
> 
> Naaah, ramdrive. The whole thing is constantly cached anyway if you have
> enough RAM (1GB does the job in my case). It of course depends on how
> busy your rsync server is, but if it's not -- why do you care about
> speed?
> 


--
gentoo-mirrors@gentoo.org mailing list

Re: [gentoo-mirrors] Configuration Request

[Georgi Georgiev]

maillog: 01/12/2004-16:30:40(-0700): Rob Baxter types
> i think the point is that the data that gets cached can and does change 
> every 30 minutes, so caching doesn't really help.

It does help. If data changes, it is first updated in the cache. New
data is served directly out of the cache, while the hard disk is synced
in the background. That's unless I am majorly mistaken about how caching
works.

> all 5 servers in the rsync.gentoo.org rotation are currently running
> rsync out of ram. i think i can safely say they wouldn't be nearly as
> fast as they are, running off a hard disk.

Do you have some real numbers? I am not trying to doubt you too much. I
am genuinely curious how big the speedup is and if it is worth the
effort. All I wanted to point in my post is that the speedup is probably
not that great, but I'd really like to see benchmark numbers if someone
went to the trouble of doing it.

> besides, ram is cheap

That's pretty relative. $70 for 512MB is not cheap for me. One reason
why my poor "server" is still running with 2x512MB *PC133*.

> and has a small footprint (smaller blocks) using
> ramdrive, why not use it.

Well, you may be right. I guess I could try serving two trees and do
some tests myself.

-- 
*)   Georgi Georgiev   *) What fools these morals be!                  *)
(*    chutz@gg3.net    (*                                              (*
*)  +81(90)6266-1163   *)                                              *)

--
gentoo-mirrors@gentoo.org mailing list

Re: [gentoo-mirrors] Configuration Request

[Rob Baxter]

i don't have any official numbers, as i don't run any of the master mirrors, and never bothered to 
test it out. i manage 2 here in canada though, and friend manages a 3rd, all 3 are running from 
ramdisk, you're welcome to try them out for yourself if you wish. rsync4, rsync5, & 
rsync9.ca.gentoo.org.

i noticed an instant jump in the speed when i moved my mirror from u160 scsi to ramdisk, it does 
make sense if you think about it. you're serving roughly 100k tiny files to a bunch of people all at 
once. that's a lot of seek time on any hard disk. i also sync the ramdisk every 30 mins directly to 
ram. basically, it's a lot less disk thrashing. ;)

rob

Georgi Georgiev wrote:
> maillog: 01/12/2004-16:30:40(-0700): Rob Baxter types
> 
>>i think the point is that the data that gets cached can and does change 
>>every 30 minutes, so caching doesn't really help.
> 
> 
> It does help. If data changes, it is first updated in the cache. New
> data is served directly out of the cache, while the hard disk is synced
> in the background. That's unless I am majorly mistaken about how caching
> works.
> 
> 
>>all 5 servers in the rsync.gentoo.org rotation are currently running
>>rsync out of ram. i think i can safely say they wouldn't be nearly as
>>fast as they are, running off a hard disk.
> 
> 
> Do you have some real numbers? I am not trying to doubt you too much. I
> am genuinely curious how big the speedup is and if it is worth the
> effort. All I wanted to point in my post is that the speedup is probably
> not that great, but I'd really like to see benchmark numbers if someone
> went to the trouble of doing it.
> 
> 
>>besides, ram is cheap
> 
> 
> That's pretty relative. $70 for 512MB is not cheap for me. One reason
> why my poor "server" is still running with 2x512MB *PC133*.
> 
> 
>>and has a small footprint (smaller blocks) using
>>ramdrive, why not use it.
> 
> 
> Well, you may be right. I guess I could try serving two trees and do
> some tests myself.
> 


--
gentoo-mirrors@gentoo.org mailing list

Re: [gentoo-mirrors] Configuration Request

[Kurt Lieber]

[-- Attachment #1: Type: text/plain, Size: 1375 bytes --]

On Thu, Dec 02, 2004 at 10:55:57AM +0900 or thereabouts, Georgi Georgiev wrote:
> > all 5 servers in the rsync.gentoo.org rotation are currently running
> > rsync out of ram. i think i can safely say they wouldn't be nearly as
> > fast as they are, running off a hard disk.
> 
> Do you have some real numbers? I am not trying to doubt you too much. I
> am genuinely curious how big the speedup is and if it is worth the
> effort. All I wanted to point in my post is that the speedup is probably
> not that great, but I'd really like to see benchmark numbers if someone
> went to the trouble of doing it.

I can't give real comparison numbers, but I can say that we did see
dramatic improvements when moving from an IDE drive to a ram drive on the
servers in the rsync.g.o rotation.  It's a bit hazy, but I seem to recall
getting killed with much more than 20 or so simultaneous connections on an
IDE-based rsync.g.o server and no more than 30-40 with a SCSI-based one.
With the ram-based servers, we've never seen a ceiling.  Most of our
servers are serving up ~30 simultaneous connections and ~6Mbps on average
and their loads never spike much beyond .4 or so.  

Out of the 5 servers we have in the rotation, we could probably make do
with 2 of them.  The rest are mainly there for spikes during releases and
redundancy in case of failure.

--kurt

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]