[gentoo-user] Hard drive pricing and the near future

[gentoo-user] Hard drive pricing and the near future

[Dale]

Howdy all,

I still have quite a bit of drive storage but I've read that prices on
drives are on the rise.  Thing is, I don't track them much.  I'm looking
at buying a 8TB drive and I've researched to make sure I'm getting a
PMR/CMR drive.  I'm avoiding a SMR since it doesn't perform as well in
my use case.  I tend to stick with Seagate, WD and other major makers.

If anyone reading this does track the pricing of drives, are they on the
rise, stable, dropping or what?  Is this a good time to expand while it
is more cost effective?  I shop around on ebay, Amazon and others before
buying.  I'm not opposed to buying used since I can sometimes find one
that was pulled and sometimes has only a few hours of use.  I found one
once that only had like 10 hours on it.  Still got it too. 

One reason I'm wanting to do this now is price.  However, in a year or
so, I'm getting fiber internet, dang fast at that.  It's starts at 200Mb
but still over a 100 times faster than current connection.  It goes all
the way up to 1Gb.  God help us all.  ROFL

Thoughts??

Dale

:-)  :-) 

Re: [gentoo-user] Hard drive pricing and the near future

[Rich Freeman]

On Tue, Oct 5, 2021 at 7:32 PM Dale <rdalek1967@gmail.com> wrote:
>
> If anyone reading this does track the pricing of drives, are they on the
> rise, stable, dropping or what?  Is this a good time to expand while it
> is more cost effective?  I shop around on ebay, Amazon and others before
> buying.  I'm not opposed to buying used since I can sometimes find one
> that was pulled and sometimes has only a few hours of use.  I found one
> once that only had like 10 hours on it.  Still got it too.

Dropping I would say.  For a while the supply was interrupted, most
likely due to Chia.  Fortunately the price of Chia dropped and it
became the network had gotten so large that payback was going to be
very slow except for a few weeks in the beginning.  I suspect that
people with a lot of storage might be farming Chia with their spare
storage, but I doubt anybody is buying pallets of hard drives just to
farm it.

If you aren't in a hurry or picky about the model I suggest setting up
searches on slickdeals.  Then be sure to check online to see if the
drive is known to be SMR.  When I buy a drive I do a bit of
benchmarking just to make sure - I think just running more than one
pass on badblocks would probably catch it (granted the access is all
sequential, but the drive has no way of knowing that and so on each
pass it would have to do two passes to consolidate writes).

Usually the best prices are on USB3 10+TB hard drives.  The good 3.5"
drives tend to be more expensive since they're targeted at commercial
use.  You can generally shuck the drive out of a USB3 enclosure if you
want to, but if your PSU isn't compatible you have to do a bit of
workarounds because they use the latest SATA power standard and some
genius decided not to make that backwards-compatible with the SATA
power found all over the place.  Usually that is only used in
enterprise drives and the USB3 enclosures often use surplus enterprise
disks (so you're getting a really good value with them).  If you keep
it in the enclosure you don't have to worry about it.  I've found
about half my PSUs work fine them, and half require polyamide tape
games to work.

-- 
Rich

Re: [gentoo-user] Hard drive pricing and the near future

[Dale]

Rich Freeman wrote:
> On Tue, Oct 5, 2021 at 7:32 PM Dale <rdalek1967@gmail.com> wrote:
>> If anyone reading this does track the pricing of drives, are they on the
>> rise, stable, dropping or what?  Is this a good time to expand while it
>> is more cost effective?  I shop around on ebay, Amazon and others before
>> buying.  I'm not opposed to buying used since I can sometimes find one
>> that was pulled and sometimes has only a few hours of use.  I found one
>> once that only had like 10 hours on it.  Still got it too.
> Dropping I would say.  For a while the supply was interrupted, most
> likely due to Chia.  Fortunately the price of Chia dropped and it
> became the network had gotten so large that payback was going to be
> very slow except for a few weeks in the beginning.  I suspect that
> people with a lot of storage might be farming Chia with their spare
> storage, but I doubt anybody is buying pallets of hard drives just to
> farm it.
>
> If you aren't in a hurry or picky about the model I suggest setting up
> searches on slickdeals.  Then be sure to check online to see if the
> drive is known to be SMR.  When I buy a drive I do a bit of
> benchmarking just to make sure - I think just running more than one
> pass on badblocks would probably catch it (granted the access is all
> sequential, but the drive has no way of knowing that and so on each
> pass it would have to do two passes to consolidate writes).
>
> Usually the best prices are on USB3 10+TB hard drives.  The good 3.5"
> drives tend to be more expensive since they're targeted at commercial
> use.  You can generally shuck the drive out of a USB3 enclosure if you
> want to, but if your PSU isn't compatible you have to do a bit of
> workarounds because they use the latest SATA power standard and some
> genius decided not to make that backwards-compatible with the SATA
> power found all over the place.  Usually that is only used in
> enterprise drives and the USB3 enclosures often use surplus enterprise
> disks (so you're getting a really good value with them).  If you keep
> it in the enclosure you don't have to worry about it.  I've found
> about half my PSUs work fine them, and half require polyamide tape
> games to work.
>

I may give it a bit and see what they do then.  My /home is at 65% so I
got time, especially while on this whimpy DSL.  I recently discovered
torrents and its advantages and now my DSL stays busy.  I only pause it
when I need the internet for something else.  When fiber gets here, oh
dear. 

You the one who introduced me to SMR.  I bought one and started a thread
about why my external drive had this bumpy feel long after my backups
were done.  You posted about SMR and how they work.  For the backup
drive, I don't mind much but if I had known before I bought it, I would
have avoided it.  I let the drive sit until the bumpy feel goes away
after I do my backups, which at times takes a while.  Now I try to avoid
them and research before hitting the buy button. 

I've looked into buying external drives and removing them for internal
use.  It seems to be a little risky given the power problem.  At one
point I thought I found a adapter, maybe a China made thing, that plugs
into the drive and then regular power supply cables plug into the
adapter.  I never bought one since I think it may be best to just buy
drives made for going in my puter case and hooking directly to my
cables.  I've read where you can save quite a bit of money doing that tho. 

May give this a bit more time.  See what the prices do.

Thanks for the info.

Dale

:-)  :-) 

Re: [gentoo-user] Hard drive pricing and the near future

[mad.scientist.at.large]

SMR drive prices are hopefully dropping.  A couple of months ago I bought a couple of used Hitachi 4 GB drives, for my raid setup.  Being raid I got recertified drives on ebay.  They were about $65, up from about $50 for the same drives a couple of years ago.  Larger new drives are hopefully coming down (I'd still like to have proper backup when I can afford it)

RE: the SATA power connection change, it's stupid but easy to fix.  Just cut the orange wire going to the drive power connector.  I don't think anyone uses 3.3V on hard drives (the orange wire), and letting it float works with the rather damaged change to the spec.  Of course, only do this on the drive connectors.  Doesn't cause a problem on older drives.  Alternately, you can use a "Molex" to sata power connector, since the 4 pin Molex connectors don't have a 3.3V line the adapters always leave it unconnected.  Might be preferable if your' system has some of the older connectors and you are nervous about cutting wires though in the case of the drives it's absolutely OK.  As these drive power connector are usually daisy chained it's best to cut the orange wire on all the SATA power connectors.


--"Fascism begins the moment a ruling class, fearing the people may use their political democracy to gain economic democracy, begins to destroy political democracy in order to retain its power of exploitation and special privilege." Tommy Douglas




Oct 5, 2021, 17:32 by rdalek1967@gmail.com:

> Howdy all,
>
> I still have quite a bit of drive storage but I've read that prices on
> drives are on the rise.  Thing is, I don't track them much.  I'm looking
> at buying a 8TB drive and I've researched to make sure I'm getting a
> PMR/CMR drive.  I'm avoiding a SMR since it doesn't perform as well in
> my use case.  I tend to stick with Seagate, WD and other major makers.
>
> If anyone reading this does track the pricing of drives, are they on the
> rise, stable, dropping or what?  Is this a good time to expand while it
> is more cost effective?  I shop around on ebay, Amazon and others before
> buying.  I'm not opposed to buying used since I can sometimes find one
> that was pulled and sometimes has only a few hours of use.  I found one
> once that only had like 10 hours on it.  Still got it too. 
>
> One reason I'm wanting to do this now is price.  However, in a year or
> so, I'm getting fiber internet, dang fast at that.  It's starts at 200Mb
> but still over a 100 times faster than current connection.  It goes all
> the way up to 1Gb.  God help us all.  ROFL
>
> Thoughts??
>
> Dale
>
> :-)  :-) 
>

Re: [gentoo-user] Hard drive pricing and the near future

[Matt Connell]

On Tue, 2021-10-05 at 18:32 -0500, Dale wrote:
> If anyone reading this does track the pricing of drives, are they on
> the rise, stable, dropping or what?

I can't speak to trends, but I've used this site in the past to keep an
eye out for a deal when it comes up.  It only indexes Amazon prices,
but its usually a good bellweather for how things look in general.

https://diskprices.com/

Re: [gentoo-user] Hard drive pricing and the near future

[Dale]

Matt Connell wrote:
> On Tue, 2021-10-05 at 18:32 -0500, Dale wrote:
>> If anyone reading this does track the pricing of drives, are they on
>> the rise, stable, dropping or what?
> I can't speak to trends, but I've used this site in the past to keep an
> eye out for a deal when it comes up.  It only indexes Amazon prices,
> but its usually a good bellweather for how things look in general.
>
> https://diskprices.com/
>
>
>
>

That's a nifty site.  Wish it would allow me to select brands but this
is a really good starting point. 

Thanks much.

Dale

:-)  :-) 

RE: [gentoo-user] Hard drive pricing and the near future

[Laurence Perkins]

> -----Original Message-----
> From: Dale <rdalek1967@gmail.com> 
> Sent: Tuesday, October 5, 2021 4:32 PM
> To: gentoo-user@lists.gentoo.org
> Subject: [gentoo-user] Hard drive pricing and the near future
> 
> 
> Howdy all,
> 
> I still have quite a bit of drive storage but I've read that prices on drives are on the rise.  Thing is, I don't track them much.  I'm looking at buying a 8TB drive and I've researched to make sure I'm getting a PMR/CMR drive.  I'm avoiding a SMR since it doesn't perform as well in my use case.  I tend to stick with Seagate, WD and other major makers.
> 
> If anyone reading this does track the pricing of drives, are they on the rise, stable, dropping or what?  Is this a good time to expand while it is more cost effective?  I shop around on ebay, Amazon and others before buying.  I'm not opposed to buying used since I can sometimes find one that was pulled and sometimes has only a few hours of use.  I found one once that only had like 10 hours on it.  Still got it too.
> 
> One reason I'm wanting to do this now is price.  However, in a year or so, I'm getting fiber internet, dang fast at that.  It's starts at 200Mb but still over a 100 times faster than current connection.  It goes all the way up to 1Gb.  God help us all.  ROFL
> 
> Thoughts??
> 
> Dale
> 
> :-)  :-)
> 

If you want to avoid SMR, look for the drives built for video surveillance systems.  Most manufacturers have a line for that purpose.  For WD it's the "purple" ones.

Other option, depending on exactly what your use case is would be to look into your choice of filesystem.  SMR doesn't like random writes into one of its chunks unless it has enough idle time to go back and straighten it out later is all.  There are now format options for ext4 to align its metadata to the SMR sections and to make it avoid random writes as much as it can.  Additionally BTRFS, ZFS, and NILFS2 are all structured such that they tend to write from one end of the disk to the other and then jump back to the beginning, so they see little if any degradation from SMR.

LMP

Re: [gentoo-user] Hard drive pricing and the near future

[Rich Freeman]

On Wed, Oct 6, 2021 at 12:16 PM Laurence Perkins <lperkins@openeye.net> wrote:
>
> Other option, depending on exactly what your use case is would be to look into your choice of filesystem.  SMR doesn't like random writes into one of its chunks unless it has enough idle time to go back and straighten it out later is all.  There are now format options for ext4 to align its metadata to the SMR sections and to make it avoid random writes as much as it can.  Additionally BTRFS, ZFS, and NILFS2 are all structured such that they tend to write from one end of the disk to the other and then jump back to the beginning, so they see little if any degradation from SMR.

Unless something has changed, it was ZFS rebuilds that caused a lot of
the initial fuss on Linux.  Drives were getting dropped from pools due
to timeouts/etc during rebuilds.  I'm not sure how sequential the IO
is for ZFS rebuilds.  I think btrfs seems a bit smarter about scrubs
in general.

-- 
Rich

RE: [gentoo-user] Hard drive pricing and the near future

[Laurence Perkins]

> On Wed, Oct 6, 2021 at 12:16 PM Laurence Perkins <lperkins@openeye.net> wrote:
> >
> > Other option, depending on exactly what your use case is would be to look into your choice of filesystem.  SMR doesn't like random writes into one of its chunks unless it has enough idle time to go back and straighten it out later is all.  There are now format options for ext4 to align its metadata to the SMR sections and to make it avoid random writes as much as it can.  Additionally BTRFS, ZFS, and NILFS2 are all structured such that they tend to write from one end of the disk to the other and then jump back to the beginning, so they see little if any degradation from SMR.
> 
> Unless something has changed, it was ZFS rebuilds that caused a lot of the initial fuss on Linux.  Drives were getting dropped from pools due to timeouts/etc during rebuilds.  I'm not sure how sequential the IO is for ZFS rebuilds.  I think btrfs seems a bit smarter about scrubs in general.
> 
> --
> Rich
> 

Good to know.  I don't use ZFS myself, and I suspect the benchmarks I looked at when I was checking on how best to deal with SMR drives didn't try a pool rebuild for ZFS.

Scrub on BTRFS is read-only unless there are errors that need correcting, which should be rare.  BTRFS balance operations might be affected, but those aren't needed as often as they used to be and should still operate linearly on whole chunks.  

I think what to look for there would be if there's a way to align the BTRFS chunks to the SMR blocks.  But the manufacturers decided to continue manufacturing CBR disks for the surveillance industry, so I haven't had to worry about it just yet.

LMP

Re: [gentoo-user] Hard drive pricing and the near future

[Rich Freeman]

On Wed, Oct 6, 2021 at 3:37 PM Laurence Perkins <lperkins@openeye.net> wrote:
> I think what to look for there would be if there's a way to align the BTRFS chunks to the SMR blocks.

There are definitely ways to implement filesystems that are more
compatible with SMR.  You basically want something like a log-based
filesystem.  A COW filesystem is actually a really good candidate as
they don't do in-place writes ever, and all you need to do is defer
block frees and garbage collect and so on to make it more log-based.

One of the issues though is that these drives obfuscate how they work.
The filesystem has no way to intentionally write to the CMR vs SMR
regions.

This is why drive-managed SMR really shouldn't be a thing.
Host-managed SMR makes a LOT more sense, because then the filesystem
can mitigate most of the issues and not end up fighting the drive
firmware, whose behavior isn't even standardized.

> But the manufacturers decided to continue manufacturing CBR disks for the surveillance industry, so I haven't had to worry about it just yet.

There are lots of CMR drives out there.  It is just that you have to
be careful as nothing is well-documented and it is all subject to
change.  It is like trying to figure out how many channels a DIMM has
or what its timing capabilities are.

I saw a good price on an Exos drive and I believe those are all CMR.
This is why I use slickdeals - you can set up searches and get alerts
when a price drops.  It also picks up stuff like Best Buy who often
has some of the best prices on USB enclosures for whatever reason when
they go on sale.

-- 
Rich

RE: [gentoo-user] Hard drive pricing and the near future

[Laurence Perkins]

> On Wed, Oct 6, 2021 at 3:37 PM Laurence Perkins <lperkins@openeye.net> wrote:
> > I think what to look for there would be if there's a way to align the BTRFS chunks to the SMR blocks.
> 
> There are definitely ways to implement filesystems that are more compatible with SMR.  You basically want something like a log-based filesystem.  A COW filesystem is actually a really good candidate as they don't do in-place writes ever, and all you need to do is defer block frees and garbage collect and so on to make it more log-based.

COW is a good compromise.  Log-based filesystems like NILFS2 generally have terrible performance.  Their only advantage in this situation is that they don't suffer any detectable performance loss from SMR.
EXT4 has new formatting options that tell it to sequentialize the metadata writes as much as possible.  Supposedly it helps quite a bit, but I haven't tested it myself yet.

> 
> One of the issues though is that these drives obfuscate how they work.
> The filesystem has no way to intentionally write to the CMR vs SMR regions.

It depends on the drive.  There are ATA commands you can use and at least some drives will tell you what their zones are and can be told to give the host more control over what goes where.
https://zonedstorage.io/getting-started/smr-disk/  has a good description.
Definitely avoid the Drive Managed Interface ones except maybe for cheap, bulk storage.  But the Host Aware and Host Managed ones can perform pretty well on some IO loads with proper FS tuning.

> 
> This is why drive-managed SMR really shouldn't be a thing.
> Host-managed SMR makes a LOT more sense, because then the filesystem can mitigate most of the issues and not end up fighting the drive firmware, whose behavior isn't even standardized.

Yup.  Only reason for drive-managed is for drop-in replacement of low-throughput systems that couldn't otherwise handle it.

> 
> > But the manufacturers decided to continue manufacturing CMR disks for the surveillance industry, so I haven't had to worry about it just yet.
> 
> There are lots of CMR drives out there.  It is just that you have to be careful as nothing is well-documented and it is all subject to change.  It is like trying to figure out how many channels a DIMM has or what its timing capabilities are.
> 
> I saw a good price on an Exos drive and I believe those are all CMR.
> This is why I use slickdeals - you can set up searches and get alerts when a price drops.  It also picks up stuff like Best Buy who often has some of the best prices on USB enclosures for whatever reason when they go on sale.

Almost everything marketed as being for the surveillance industry is going to be CMR.  We often run the drives at 80% or better throughput and a lot of it is non-sequential.  SMRs just can't keep up.  So there was a lot of pushback from the surveillance industry about not changing the underlying storage technology without clearly labeling it as such.

Prior to that the announcement from Seagate and I think WD was that everything larger than 12TB was going to be SMR.  (I work on the software side of things, I get hardware stuff second hand, so my understanding may be fuzzy.)

But currently the WD Purples and the Seagate Skyhawks should all be CMR.

Obviously I have no power to hold them to that, so do not take this as any kind of guarantee.  But if they change it without warning they're likely to have a lot of unhappy customers.

> 
> --
> Rich
> 
> 

LMP

Re: [gentoo-user] Hard drive pricing and the near future

[antlists]

On 06/10/2021 21:20, Laurence Perkins wrote:
> But currently the WD Purples and the Seagate Skyhawks should all be CMR.
> 
Seagate have said the Iromwolf range will remain CMR. BarraCuda is now 
all SMR (renamed from Barracuda, presumably to say it's still the budget 
range, but the old drives are pre-SMR). I get the impression that 
FireCuda are the "budget CMR" range, but don't quote me on that.

> Obviously I have no power to hold them to that, so do not take this as any kind of guarantee.  But if they change it without warning they're likely to have a lot of unhappy customers.

AS WD found out when loads of "marketed for NAS" WD Reds got returned 
"not fit for purpose" when people put them in NASs.

Cheers,
Wol

Re: [gentoo-user] Hard drive pricing and the near future

[Frank Steinmetzger]

[-- Attachment #1: Type: text/plain, Size: 1988 bytes --]

Am Tue, Oct 05, 2021 at 06:32:24PM -0500 schrieb Dale:
> Howdy all,
> […]
> If anyone reading this does track the pricing of drives, are they on the
> rise, stable, dropping or what?  Is this a good time to expand while it
> is more cost effective?

Well I’m on ye Olde Continent and here I use a price search engine called
geizhals (German for scrooge). But they have an English equivalent ;-) and
of course use European pricing schemes (meaning including taxes, which you
do different).
https://skinflint.co.uk/?m=1

The big difference here: it allows you to filter and sort by all sorts of
criteria, such as being (non-)SMR or price/TB. And it shows you the prices
of lots of different shops, including Amazon. So while it may not give you
an idea of the US market, it gives you one of available drive models.

For instance, here are all internal non-SMR 3.5″ drives of 6 TB and up:
https://skinflint.co.uk/?cat=hde7s&xf=1080_SATA+1.5Gb%2Fs%7E1080_SATA+3Gb%2Fs%7E1080_SATA+6Gb%2Fs%7E13810_6000%7E3772_3.5%7E8457_non-SMR

I like to buy naked drives, because I don’t like “cable salad”, i.e. having
multiple external cases, each needing its own special power supply. Instead,
I’d opt for a hard drive dock which lets you insert a hard disk as if it
were an SD card. Actually, I use this internal bay for my PC case:

https://skinflint.co.uk/sharkoon-sata-quickport-internal-multi-a862667.html
It eats standard SATA drives and is connected via standard connectors to my
PSU and mobo.

> However, in a year or so, I'm getting fiber internet, dang fast at that. 
> It's starts at 200Mb but still over a 100 times faster than current
> connection.  It goes all the way up to 1Gb.  God help us all.  ROFL

But when are you ever going to watch it all?

-- 
Grüße | Greetings | Qapla’
Please do not share anything from, with or about me on any social network.

Filling your car becomes much cheaper if you install a smaller tank.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] Hard drive pricing and the near future

[Frank Steinmetzger]

[-- Attachment #1: Type: text/plain, Size: 1329 bytes --]

Am Wed, Oct 06, 2021 at 10:24:07PM +0200 schrieb Frank Steinmetzger:
> Am Tue, Oct 05, 2021 at 06:32:24PM -0500 schrieb Dale:
> > Howdy all,
> > […]
> > If anyone reading this does track the pricing of drives, are they on the
> > rise, stable, dropping or what?  Is this a good time to expand while it
> > is more cost effective?
> 
> Well I’m on ye Olde Continent and here I use a price search engine called
> geizhals (German for scrooge). But they have an English equivalent ;-) and
> of course use European pricing schemes (meaning including taxes, which you
> do different).
> https://skinflint.co.uk/?m=1
> 
> The big difference here: it allows you to filter and sort by all sorts of
> criteria, such as being (non-)SMR or price/TB. And it shows you the prices
> of lots of different shops, including Amazon. So while it may not give you
> an idea of the US market, it gives you one of available drive models.

Ah, and another reason why I wanted to suggest this site: it also remembers
the price development of an item. If you view an item’s details, you have
its price history on the top right of the screen.

-- 
Grüße | Greetings | Qapla’
Please do not share anything from, with or about me on any social network.

The only thing that makes some people bearable is their absence.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] Hard drive pricing and the near future

[Dale]

Frank Steinmetzger wrote:
> Am Wed, Oct 06, 2021 at 10:24:07PM +0200 schrieb Frank Steinmetzger:
>> Am Tue, Oct 05, 2021 at 06:32:24PM -0500 schrieb Dale:
>>> Howdy all,
>>> […]
>>> If anyone reading this does track the pricing of drives, are they on the
>>> rise, stable, dropping or what?  Is this a good time to expand while it
>>> is more cost effective?
>> Well I’m on ye Olde Continent and here I use a price search engine called
>> geizhals (German for scrooge). But they have an English equivalent ;-) and
>> of course use European pricing schemes (meaning including taxes, which you
>> do different).
>> https://skinflint.co.uk/?m=1
>>
>> The big difference here: it allows you to filter and sort by all sorts of
>> criteria, such as being (non-)SMR or price/TB. And it shows you the prices
>> of lots of different shops, including Amazon. So while it may not give you
>> an idea of the US market, it gives you one of available drive models.
> Ah, and another reason why I wanted to suggest this site: it also remembers
> the price development of an item. If you view an item’s details, you have
> its price history on the top right of the screen.
>


I used to use pricewatch.com but it seems to be gone now.  I also ran up
on a site once that shows a graph of prices going back to when a product
was first introduced.  Since my brain has no elephant genes, I have no
idea what the name of the site was nor can I find it now either.  :-/

Neat sites tho. 

Dale

:-)  :-) 

Re: [gentoo-user] Hard drive pricing and the near future

[Peter Humphrey]

On Wednesday, 6 October 2021 21:24:07 BST Frank Steinmetzger wrote:

> Well I’m on ye Olde Continent and here I use a price search engine called
> geizhals (German for scrooge). But they have an English equivalent ;-) and
> of course use European pricing schemes (meaning including taxes, which you
> do different).
> https://skinflint.co.uk/?m=1

What a useful site! Thank you Frank.

-- 
Regards,
Peter.