Re: [gentoo-user] Re: File system testing

["J. Roeleveld" <joost@antarean.org>]

[-- Attachment #1: Type: text/plain, Size: 1657 bytes --]


On Wednesday, September 17, 2014 04:20:24 PM Alec Ten Harmsel wrote:
> As far as HDFS goes, I would only set that up if you will use it for
> Hadoop or related tools. It's highly specific, and the performance is
> not good unless you're doing a massively parallel read (what it was
> designed for). I can elaborate why if anyone is actually interested.
> 
> We use Lustre for our high performance general storage. I don't have 
any
> numbers, but I'm pretty sure it is *really* fast (10Gbit/s over IB
> sounds familiar, but don't quote me on that).

I think any shared filesystem will be fast if you have a lot of bandwidth :)
When comparing network filesystems it makes sense to keep the hardware 
identical reduce the overhead to a percentage. Eg. What is the theoretical 
maximum speed for the used network. (10Gbit/s) and what is the actual 
maximum speed you get with:
1) a single really large file (200GB)
2) a lot (100,000) smaller files (2MB)

Then you can make an estimate on what to expect when using a 1Gbit/s 
network. I somehow don't expect James to have InfiniBand available for his 
research?
Personally, when choosing between InfiniBand and Ethernet, I'm tempted 
to go with dedicated bonded 10Gbit/s links because of the price-
difference. (A quick research shows me that Infiniband is about 3x as 
expensive for the same throughput)

> > Personally, I would read up on these and see how they work. Then,
> > based on that, decide if they are likely to assist in the specific
> > situation you are interested in.
> 
> Always good advice.

It saves time to do some simple research (the reading type) before 
actually doing tests.

--
Joost

[-- Attachment #2: Type: text/html, Size: 6197 bytes --]