Re: [gentoo-user] PCIe x1 or PCIe x4 SATA controller card

[Dale <rdalek1967@gmail.com>]

Rich Freeman wrote:
> On Mon, Mar 13, 2023 at 8:24 AM Dale <rdalek1967@gmail.com> wrote:
>>  According to my google searches, PCIe x4 is faster
>> than PCIe x1.  It's why some cards are PCIe x8 or x16.  I think video
>> cards are usually x16.  My question is, given the PCIe x4 card costs
>> more, is it that much faster than a PCIe x1?
> It could be slower than PCIe x1, because you didn't specify the version.
>
> PCIe uses lanes.  Each lane provides a certain amount of bandwidth
> depending on the version in use.
>
> For example, a v1 4x card has 1 GB/s of bandwidth.  A v4 1x card has
> 2GB/s of bandwidth.
>
> Note that slot size is only loosely coupled with the number of lanes.
> Lots of motherboards have a second 16x slot that only provides 4-8
> lanes to save on the cost of a PCIe swich.  You can also use adapters
> to connect a 16x card to a 1x slot, or you might find a motherboard
> that has an open-ended slot so that you can just fit a 16x card onto
> the 1x slot.  It will of course only use a single lane that way.
>
> So what you need to do is consider the following:
>
> 1. How much bandwidth do you actually need?  If you're using spinning
> disks you aren't going to sustain more than 200MB/s to a single drive,
> and the odds of having 10 drives using all that bandwidth are pretty
> low.  If you're using SSDs then you're more likely to max them out
> since the seek cost is much lower.
> 2. What PCIe version does your motherboard support?  Sticking a v4
> card on an old motherboard that only supports v2 is going to result in
> it running at v2 speeds, so don't pay a premium for something you
> won't use.  Likewise, if they cut down on the number of lanes assuming
> they'll have more bandwidth you might have less than you expected to
> have.
>
> Then look up the number of lanes and the PCIe version and see what you
> can expect:
> https://en.wikipedia.org/wiki/PCI_Express#History_and_revisions
>
> I think odds are you aren't going to want to pay a premium if you're
> just using spinning disks.  If you actually wanted solid state storage
> then I'd also be avoiding SATA and trying to use NVMe, though doing
> that at scale requires a lot of IO, and that will cost you quite a
> bit.  There is a reason your motherboard has mostly 1x slots - PCIe
> lanes are expensive to support.  On most consumer motherboards they're
> only handled by the CPU, and consumer CPUs are very limited in how
> many they offer.  Higher end motherboards may have a switch and offer
> more lanes, but they'll still bottleneck if they're all maxed out
> getting into the CPU.  If you buy a server CPU for several thousand
> dollars one of the main features they offer is a LOT more PCIe lanes,
> so you can load up on NVMes and have them running at v4-5.  (Typical
> NVMe uses a 4x M.2 slot, and of course you can have 16x cards offering
> multiples of those.)
>
> The whole setup is pretty analogous to networking.  If you have a
> computer with 4 network ports you can bond them together and run them
> to a switch that supports this with 4 cables, and get 4x the
> bandwidth.  However, you can also get a single connection to run at
> higher speeds (1Gb, 2.5Gb, 10Gb, etc), and you can do both.  PCIe
> lanes are just like bonded network cables - they are just pairs of
> signal wires that use differential signaling, just like twisted pairs
> in an ethernet cable.  Longer slots just add more of them.  Everything
> is packet switched, so if there are more lanes it just spreads the
> packets across them.  Higher versions mean higher speeds in each lane.
>


This is why I asked.  I didn't even think about the different PCIe
versions available.  My mobo, had to go dig out the manual, says it is
PCIe 2.0.  This is a Gigabyte 970A-UD3P motherboard.  Yes, I'm thinking
about building a new rig.  Turn this into a NAS maybe.  Anyway, I'm
assuming 2.0 isn't the slowest or fastest but as you point out, it'll be
a bottleneck.  Everything has a bottleneck somewhere. 

The difference in price isn't that large.  What I did was, I saw the
PCIe x1 and bought it.  It supported Linux.  Later on I noticed the PCIe
x4 and then wondered if I should upgrade to that.  Given the limits of
my mobo and the fact I won't be maxing out the drives anyway, I don't
see the need to upgrade.  Your info pretty much makes that clear.  I
might, just might, see a small difference when using pvmove.  Maybe. 
Given that it generally maxes out the drive as it is, even if it does go
faster, it won't be much.  Add in that when I start a pvmove, I go nap
and do other things anyway, I won't notice it.  Last pvmove took a
little over 19 hours.  Even 20 or 30 minutes isn't much difference in
the grand scheme of things. 

Another question.  My rig is getting a bit aged.  I have a AMD FX-8350 8
core CPU running at 4GHz.  I also have 32GBs of memory.  I've read that
Intel currently has the best bang for buck on CPUs nowadays.  I'm open
to the idea of switching.  As far as speed goes, if I built a new rig
that is using a reasonably cost CPU and memory, would I see any real
improvements?  When I say 'reasonably cost', I usually find the
fastest/newest then drop down a bit to get out of that 'brand new' price
point.  Generally, the difference in price is quite large but the
difference in speed isn't that much.  Also, I got hard drives, I don't
spend much on video cards either since I don't game, except solitaire. 
So, let's say a mobo, CPU and memory.  What price range would I need to
look for?  Just a rough idea.  I figure the CPU will be a few hundred. 
Memory may be half that.  Mobo will likely be close to $200 or so.  I'm
thinking $500 to $700 or so.  Then comes case, video card and all that. 
The CPU, memory and mobo is the ones I try to buy all at once from the
same vendor, like Newegg or Tigerdirect.  Thoughts?  Am I close?

Thanks for the info.  At least I know I'm good on drive speed.  For my
use anyway. 

Dale

:-)  :-)