[gentoo-user] Solid state disks...

[gentoo-user] Solid state disks...

[Steve]

I'm playing around with an application that requires me to manage a 
large (multi-gigabyte to terabyte), bespoke, frequently-updating data 
structure in real-time... key concerns are for durability and 
efficiency.  While a traditional approach might be to employ an 
expensive DBMS on expensive hardware... I'm looking to be more 
innovative.  I want to achieve big-iron beating performance on a 
shoestring budget... and I'm optimistic since the problem domain doesn't 
translate well to traditional RDBMS approaches.

An obvious alternative to a DBMS is to use the file-system directly... 
in principle this could work - but it would be a laborious process 
fraught with potential pitfalls with respect to atomicity of updates, 
transactional recovery (in case of a fail-stop while processing a large 
update) etc.  Another issue is that in order to establish an efficient 
and reliable implementation, it becomes necessary to second guess 
details about the implementation of file-systems... this vastly 
complicates any implementation and might render it unacceptably fragile 
(subject to unexpected deviations in behaviour as the implementation is 
moved between hardware/OS-versions etc.

I've recently discovered that SSDs are becoming more affordable... and 
this might present new options.  There were major hurdles in attempting 
to establish a strategy to interact with hard-disk block devices... 
including, but not limited to, a significant difficulty in establishing 
the extent to which locality of reference affected performance.  Another 
worry was that it might be difficult to establish that a write had 
actually completed (i.e. the data reliably and durably stored - not just 
that the responsibility for recording the data was now exclusively with 
the drive.)  My hope is that SSD technology simplifies some of these 
concerns - allowing a clear model for access performance that should 
allow an efficient and reliable implementation.

I'd like to hear about anyone who has experience with configuring SSDs 
for use with (Gentoo) Linux - and especially from anyone who's 
investigated performance issues.  I've read that SSDs typically have a 
64Kib block size... this would work fine for me (though I understand 
that it is a significant impediment for high performance with existing 
file systems.  I'd be interested to know if anyone has done performance 
analysis of SSDs at the device level under Linux... and am intrigued if 
there is more to interacting with them than establishing the block size 
from manufacturer data - then reading/writing appropriately many bytes 
from block devices... and/or flushing appropriately aligned and sized 
blocks of memory mapped data.  For example, is there an interface to 
quiz an SSD about its block-size?  I'm intrigued to establish if I can 
rely upon my data being durably stored on an SSD when a flush/write returns.

In a practical sense, I'd like to experiment with some SSD hardware - 
but there seems to be a lot to chose from.  For development purposes, 
I'd not need more than, say, 32GB - and I'm not all that fussed about 
absolute performance - as long as the relative performance of various 
interactions will increase proportionally were I to move to more 
expensive SSDs in future.  I'm interested to establish any practical 
anecdotes (or hard statistical data) about the relative merits of 
various interfaces for SSDs - and to establish if RAID needs to be taken 
into account when establishing a performance model.

Any feedback would be appreciated... especially from any gentooist who 
is interested in SSD performance/reliability/configuration.