[gentoo-amd64] Re: Hard drive (installation)

[Duncan <1i5t5.duncan@cox.net>]

Rich Freeman posted on Fri, 30 Aug 2013 21:50:47 -0400 as excerpted:

> Anybody with a motherboard supporting USB2 almost certainly had a
> motherboard supporting PATA at a faster transfer rate.

[Long winded and thread tangential.  TL;DR readers just skip it, but I 
know some people find these things interesting/useful.  There's some SSD 
deployment discussion further down that I certainly could have used a few 
months ago!]

Well, yes, but...

With my 8-year-old native USB1 (with a USB2 addon card, it wouldn't do 
USB3 as it didn't have PCIE, only PCI-X, dual socket original 3-digit 
Opteron maxed out at dual Opteron-290, which are dual-core @ 2.8 GHz, so 
it was effectively quad-core at 2.8, not /too/ shabby, just getting 
seriously dated in terms of buses, etc) system that died last year, I 
setup an unbootable USB external spinning rust drive as a backup, with a 
USB thumbdrive on the bootable native USB1 for /boot to load the kernel, 
which could then handle the PCI-card USB2 that the BIOS couldn't, and 
thus get to the backup root on the external USB2 spinning rust.

That's how I know that not all external USB devices are bootable, as that 
external drive wasn't, even when I had it on the bios-supported native 
USB1, thus the earlier warning about that, and how to work around it.

But the point is, that setup wasn't /significantly/ slower in normal 
operation than my fancy multi-disk md-raid, and in fact, was NOTICEABLY 
faster than my original mdraid-6 setup (thus the discussion on it a few 
months ago), tho mdraid-1 was a bit faster after I switched to it, 
particularly for multi-thread read as typically happens during boot.

Now for transferring hundreds of megabytes (as when actually making the 
backups), yeah, doing that over the USB2 was a bit slow, roughly 
comparable to backing up to a different partition on the same spindle 
internally (with the disk seeks that means, except that backing up to the 
external was to a separate external physical device so without the USB2 
bottleneck it should have been faster).  However, for ordinary use, the 6 
gigs of RAM (at one point 8, two 2-gig sticks per socket, but one stick 
died and I never replaced it) was plenty to cover both my normal memory 
usage and working set disk cache with reasonable margin to spare as I 
normally run only 2-3 gigs apps+buffers+cache (top line of free), except 
when I'm doing updates or large media files and thus have all that cached 
too.

But even cold-booting off the USB2 external was enough faster than the 
mdraid-6 to demonstrate how bad mdraid6 can be, and convince me to switch 
to mdraid-1, which was indeed faster.

> I do agree that random access speed does lower the effective rate.  My
> hard drives are running at 3GB/s transfer rates each on a dedicated
> channel, and yet they're probably not any faster than they would have
> been under PATA (assuming one drive per cable).
> 
> Hopefully one of these days there will be a decent SSD cache option for
> Linux.  Bcache is still fairly experimental, and I'm not sure how well
> it performs in practice with btrfs - plus it is a device layer and not
> filesystem layer implementation (ie if you have mirrored drives you end
> up with mirrored cache which seems a bit dumb, especially if the mirrors
> end up being on separate partitions on the same device).

Here, I ended up putting root and home (and log and the gentoo tree and 
overlays and...) on btrfs on SSD, with, effectively, only my media (and 
secondary backups) remaining on spinning rust, as I couldn't justify the 
cost to put it on SSD.

A 64-gig SSD is now well below the best-price-point knee at least around 
here, and I calculated that would be minimal for working copy and primary 
backup of the OS and /home.  However, once I looked at prices (as of a 
few months ago), I found 128 gig SSDs were actually the low end of the 
price-point-knee, with 256 gig the high end, and decided the over-
provisioning would be a good idea due to the limited write-cycle issue on 
multi-level SSDs, so actually planned on that.

But when I went in to Fry's Electronics to buy, turned out all their good 
deals on 128 gig SSDs were selling out faster than they could get them in 
stock, and they happened to be out of the good deals there.  I use public 
transit and didn't feel like going home empty handed only to come back 
the day the truck came in, so I ended up with 256 gig SSDs, still at the 
price-point-knee tho at the upper end of it, and was able to put even 
MORE on the SSDs -- while still keeping a very healthy nearly 100% over-
provisioning, tho I expect that might dwindle a bit over time.  (With 16 
GiB RAM I decided I didn't need swap.)

I did spend a bit more than planned, buying three of them, two for the 
main machine, now mostly configured in btrfs raid1 mode in ordered to 
actually make use of btrfs' data integrity checksumming and scrub 
abilities, with the third to eventually be used in my netbook (which 
being SATA2 will bottleneck on that, but the slower ones weren't 
significantly cheaper, and I expect it'll outlast my netbook, to be used 
either in my main machine or a netbook replacement, later, at which point 
I'll probably actually use the full speed, as I do with the other two 
now).  But I've been working extra hours and will have it paid off  
~three months from purchase so not so much interest, and I'm happy with 
it.

The point being... if you do your partitioning right, you can get 90% of 
the benefits of full SSD at *FAR* more reasonable costs than full SSD.  
Basically SSD caching, only without the cache, simply putting the stuff 
that will truly benefit from SSD on SSD, while leaving the stuff that 
won't on cheaper spinning rust.  I calculated that I only NEEDED about 64 
GB for the most important stuff, with backups on spinning rust, and that 
would have left a healthy over-provisioning too.  (I figured I could fit 
what I really wanted on SSD in 32 gig if I had to, but it would have been 
a pretty tight fit!)

Here's my gdisk -l output for one of the SSD pair (I've been running GUID 
Partition Tables, GPT, for some time now, some boilerplate omitted for 
posting):

Disk /dev/sda: 500118192 sectors, 238.5 GiB
Partition table holds up to 128 entries
Partitions will be aligned on 2048-sector boundaries
Total free space is 246364781 sectors (117.5 GiB)

No. Start(sector) End(sector)  Size        Code  Name
 1          2048        8191   3.0 MiB     EF02  bi0238gcn1+35l0
 2          8192      262143   124.0 MiB   EF00  ef0238gcn1+35l0
 3        262144      786431   256.0 MiB   8300  bt0238gcn1+35l0
 4        786432     2097151   640.0 MiB   8300  lg0238gcn1+35l0
 5       2097152    18874367   8.0 GiB     8300  rt0238gcn1+35l0
 6      18874368    60817407   20.0 GiB    8300  hm0238gcn1+35l0
 7      60817408   111149055   24.0 GiB    8300  pk0238gcn1+35l0
 8     111149056   127926271   8.0 GiB     8300  nr0238gcn1+35l0
 9     127926272   144703487   8.0 GiB     8300  rt0238gcn1+35l1
10     144703488   186646527   20.0 GiB    8300  hm0238gcn1+35l1
11     186646528   236978175   24.0 GiB    8300  pk0238gcn1+35l1
12     236978176   253755391   8.0 GiB     8300  nr0238gcn1+35l1

I wanted my partitions 4 MiB aligned for efficient erase-block handling, 
tho the first one's only 1 MiB aligned.

One of the features of GPT partitioning is that it allows partition
names/labels much like filesystems normally do.  That's what's shown in 
the last column.  I have a standard naming scheme, developed back when I 
was running multiple mdraid devices, some of which were themselves 
partitioned, on a 4-spindle set, that I use for both my partition labels 
and my filesystem labels, for both my main machine and my netbook:

* 15 characters long
123456789012345
ff     bbB ymd
  ssssS   t   n

Using a different example from my mdraid days:

rt0005gmd3+9bc0

ff:	2-char function abbreviation

bi=bios (gpt dedicated legacy BIOS boot partition, used by grub2).
ef=efi (gpt dedicated efi partition, unused here ATM but reserved for 
forward compatibility)
bt=boot, lg=log, rt=root, hm=home, lg=log, pk=package (gentoo tree, 
layman overlays, sources, binpkgs, kernel tree), nr=netbook-root 
(separate 32-bit chroot build-image filesystem/partition).

Example rt=root

Device-ID consisting of ssssSbbB:
ssssS:	4-digit size, 1-char multiplier.  This is the size of the 
underlying/containing media, NOT the partition/filesystem (I'm IDing the 
containing device).

bbB: 2-char media brand ID, 1-digit sequence number.

The md example is a 5 GiB mdraid volume (/dev/md3), which might itself be 
partitioned (IIRC I was keeping /usr/local/ on a separate filesystem/
partition on the same mdraid as root, back then).

In the above output, 0238 GiB, corsair neutron.  The paired SSDs are 0 
and 1 (0 is installed as sdb, 1 as sda, as I reversed them in 
installation).  So the device-ids are 0238gcn1 and 0238gcn0, with the 
common btrfs on top of both device-IDed as 0238gcnx.

t: single-char target/separator.  This serves as both a target ID and a 
visual separator.

I use . for my netbook (dot 'cause it's small), + for the main machine, 
and % for portable disk partitions intended to be used on both.

Both the output and the md example are for my main machine, so +.

ymd: 1-char-each year/month/day.

y=last digit of year (I might use storage devices for years but it's 
unlikely I'll need to track decade wrap).
m=month (1-9abc)
d=day (1-9a-v)

This is generally the day I setup the partition.  Back when I was still 
on MBR I used to relabel the filesystem on my backup partitions with the 
new date when I did a mkfs and a clean backup, but when I switched to GPT 
and could label the partitions too I decided to keep them date-matched as 
my fstab uses LABEL= mounting, and it was a hassle updating the fstab 
when I blew away an old backup and redid it.

The md example is 2009-11-12.  The table is using 2013-05-21.

n: 1-digit copy number.  The working copy is 0, primary backup 1...

The md example is the working copy.  The table has the working copy and 
for some partitions the primary backup.


mdraid example all together: rt0005gmd3+9bc0
rt		root
0005gmd3	5-gig /dev/md3
+		for use on the main machine
9bc		2009-11-12 (Nov 12)
0		working copy


So from the above you can see I have a 3 MiB BIOS partition (grub2 uses, 
per-drive, size chosen to put all later partitions on 4 MiB boundaries), 
a 124 MiB EFI partition (reserved for future use, later partitions are 
now on 128 MiB boundaries).

A 256 MiB /boot comes next.  (This is a separate filesystem for each 
drive, the second one for backup, working copy updated every time I 
install a new kernel, which I do a lot as I run git kernels, backup 
updated once per cycle with the 3.x.0 release, I can direct the BIOS at 
either one or at the spinning rust /boot, my secondary /boot backup).

That's followed by /var/log at 640 MiB, leaving me at 1 GiB boundaries.  
This partition and later are btrfs raid1 mode, but I only keep the 
working copy of log, not the working and backup copies I keep for 
everything else.

Then come the root (8 GiB), home (20 GiB), package (24 GiB), and netbook-
root-build-image (8 GiB) partitions, working copies followed by primary 
backups.  Secondary backups and media remain as I said on spinning rust.  
That's the bulk of my data but it's also the least speed-critical.

As you can see with a bit of math (I can cheat here and just look it up 
in gdisk), that's 121 GiB used, 117.5 GiB free, just worse than 50/50, so 
near 100% over-provisioning.  I shouldn't have to worry too much about 
write-cycle wear-out with that, even if I add a few gigs of partitions 
later. (Recommended SSD overprovisioning is 25-33%, thus actually using 
3/4-4/5.  I'm barely over half! =:^)

With a bit more squeezing I could have fit home and both main root and 
netbook root on a 32-gig with no overprovisioning, and a 64 gig would 
have fit 1 copy of everything listed, with the primary backups on 
spinning rust, and that or an 80 gig to allow a bit of overprovisioning 
is what I was originally targeting.  Then I looked at the prices and saw 
128 gigs at not /that/ much over 80 gigs and at a cheaper per-gig, so 
that's what I actually thought I'd buy.  But I'm not complaining about 
the 256 gig (238 GiB, unfortunately they're carrying over the marketing 
practices from spinning rust, I AM complaining about that!), and I'm 
*DEFINITELY* not complaining about boot or emerge --sync speed on the 
SSDs! =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman