[gentoo-user] Sata hard drive speed question

[gentoo-user] Sata hard drive speed question

[Dale]

Howdy,

I bought a 8TB hard drive.  Seagate 8TB 5E8 Exos ST8000AS0003 is the
exact model info.  It seems to be slow.  First, I had it hooked to a
adapter to a USB port.  I expected it to be a little slow but it gave me
memories of the old dial-up days.  When it shows KBs/second, it's
getting slow for a sata drive.  So, I moved it inside the case with a
sata connection directly to the mobo.  I unhooked my DVD burner for
this.  It's somewhat faster but still slow in my opinion.  I found this
for specs on a website.


Max. Sustained Transfer Rate OD (MB/s)
190MB/s


OK, can I get half that now?  One quarter would be better even.  This is
a sample of what I get when using --progress with rsync while copying
files from another drive to it, backup thing. 


102,782,342 100%    4.68MB/s    0:00:20 (xfr#122, ir-chk=1135/1995)

65,330,688 100%    5.34MB/s    0:00:11 (xfr#123, ir-chk=1134/1995)

59,338,843 100%    2.04MB/s    0:00:27 (xfr#124, ir-chk=1133/1995)

64,996,691 100%   10.99MB/s    0:00:05 (xfr#125, ir-chk=1132/1995)

467,837,625 100%    5.42MB/s    0:01:22 (xfr#126, ir-chk=1131/1995)

39,236,581 100%    5.42MB/s    0:00:06 (xfr#127, ir-chk=1130/1995)

302,340,815 100%    3.95MB/s    0:01:12 (xfr#128, ir-chk=1129/1995)



This is what I get from hdparm:


root@fireball / # hdparm -Tt /dev/sdb

/dev/sdb:
 Timing cached reads:   8222 MB in  2.00 seconds = 4114.05 MB/sec
 Timing buffered disk reads:   2 MB in  3.59 seconds = 570.26 kB/sec
root@fireball / #


First one looks reasonable but second one just plain sucks.  Note the KB
instead of a MB.  I get this on a much older drive:


root@fireball / # hdparm -Tt /dev/sda

/dev/sda:
 Timing cached reads:   8664 MB in  2.00 seconds = 4335.98 MB/sec
 Timing buffered disk reads: 328 MB in  3.01 seconds = 108.82 MB/sec
root@fireball / #


And smartctrl gives me this on the new drive:


SMART Self-test log structure revision number 1
Num  Test_Description    Status                              Remaining 
LifeTime(hours)  LBA_of_first_error
# 1    Extended offline    Self-test routine in progress  90%      
544         -
# 2    Short offline          Completed without error       00%      
543         -
# 3    Short offline          Completed without error       00%      
528         -


I've ran those tests in the past and it not affect the copy speed. 
Still, it shows the drive is OK.  I'm running the long one to be 100%
sure.  I was getting the same before I started the selftest tho.  I
created one large partition with gfdisk.  It is formatted with ext4 file
system.  Most files are videos but some are other file types and
smaller.  Thing is, it seems slow no matter what size the file is. 
Large files just take longer naturally.  This is what mount shows
including options.


/dev/sdb1 on /mnt/tmpdisk type ext4 (rw,relatime)


I have a few other drives on this system.  They work fine and perform
fine. Heck, a 6TB drive in a external enclosure connected by USB does
better than this.  Can someone explain why this drive is so terribly
slow?  Did I do something wrong?  Is there something special about a
drive this large that I need to do? 

Thanks.

Dale

:-)  :-) 

Re: [gentoo-user] Sata hard drive speed question

[Taiidan]

Here are some theories.

* You gotta properly align the sectors for 4K advanced format
* USB doesn't have NCQ which really slows things down.
* Copying many small files is almost always slow since they are located
on various parts of the drive not in a contiguous block (again see NCQ)
* System is set to use IDE not AHCI thus no NCQ etc
* You are using a secondary SATA chip such as the terrible ones from
JMicron or what not instead of what is on your systems northbridge or a
quality PCI-e HBA.

Re: [gentoo-user] Sata hard drive speed question

[Dale]

Taiidan@gmx.com wrote:
> Here are some theories.
>
> * You gotta properly align the sectors for 4K advanced format
> * USB doesn't have NCQ which really slows things down.
> * Copying many small files is almost always slow since they are located
> on various parts of the drive not in a contiguous block (again see NCQ)
> * System is set to use IDE not AHCI thus no NCQ etc
> * You are using a secondary SATA chip such as the terrible ones from
> JMicron or what not instead of what is on your systems northbridge or a
> quality PCI-e HBA.
>
>


Googled to see how to find out if it is aligned correctly and found this.

root@fireball / # cat /sys/block/sdb/queue/physical_block_size
4096
root@fireball / #

I thought cgdisk did that automatically so I guess it did.  Drive is
currently connected to my motherboard's Sata port.  If it was a
bad/cheap controller, I'd think the other drives would also give slow
speeds.  They work fine.  While I have a Sata PCI-e card installed, I'm
not using it yet.  It has a Marvel chipset which others say works fine. 
Once I get some more power cables in, I'll test it to see how it does. 
At this point tho, all drives are connected to the Gigabyte Sata ports. 
Sorry if that caused confusion.

It seems we can eliminate some possible problems at least.  I need more
ideas to check on it seems.  Still, I may dd the thing, at least the
first bit of it anyway, and start again.  I did repartition and format
the drive after the move tho.  Still, maybe dd-ing it for a fresh start
will help.  At this point, I don't need the data on it.  I can redo
whatever until I get it working correctly. 

Thanks for the ideas. 

Dale

:-)  :-) 

[gentoo-user] Re: Sata hard drive speed question

[Nikos Chantziaras]

On 13/12/2018 02:48, Dale wrote:
> Howdy,
> 
> I bought a 8TB hard drive.  Seagate 8TB 5E8 Exos ST8000AS0003 is the
> exact model info.  It seems to be slow.

What's the output of:

sudo fdisk -l /dev/sda

(Assuming it's the sda drive.)

Re: [gentoo-user] Re: Sata hard drive speed question

[Dale]

Nikos Chantziaras wrote:
> On 13/12/2018 02:48, Dale wrote:
>> Howdy,
>>
>> I bought a 8TB hard drive.  Seagate 8TB 5E8 Exos ST8000AS0003 is the
>> exact model info.  It seems to be slow.
>
> What's the output of:
>
> sudo fdisk -l /dev/sda
>
> (Assuming it's the sda drive.)
>
>
>


Well, after a lot more googling, I decided to start over and then
decided to use a different tool.  I ran dd for several GBs and then used
gparted to partition and format the drive with ext4.  Right now, it is
doing the format part. 

One thing I noticed.  When it is formatting, it takes HOURS.  When I did
it the first time, from command line using mkfs.ext4, it took hours.  So
far, it's been working on it for well over 30 minutes.  I don't recall
it taking anywhere near this long on the 6TB drive I have.  I might add,
I did it through a USB port.  The fact it takes so long to format makes
me thing something is up somewhere.  Is that normal??  I also got this
during a attempt to put a file system on it a bit ago.


root@fireball / # mkfs.ext4 -m 0 -L 8tb-backup -b 4096 /dev/sdb1
mke2fs 1.43.9 (8-Feb-2018)
Creating filesystem with 1953506129 4k blocks and 244191232 inodes
Filesystem UUID: 2b987f80-b9e2-45e0-8dda-b25f0901e213
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632,
2654208,
        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
        102400000, 214990848, 512000000, 550731776, 644972544, 1934917632

Allocating group tables: done                           
Writing inode tables: done                           
Creating journal (262144 blocks): done
Writing superblocks and filesystem accounting information:    
Warning, had trouble writing out superblocks.


That last line is something I've never seen before.  If it doesn't
finish soon, I may check the sata cables and such.  Maybe one of them
isn't plugged in good, has dust on it or something.  Something isn't
working right here. 

Open to ideas tho. 

Dale

:-) :-) 

[gentoo-user] Re: Sata hard drive speed question

[Nikos Chantziaras]

On 13/12/2018 09:11, Dale wrote:
> Nikos Chantziaras wrote:
>> On 13/12/2018 02:48, Dale wrote:
>>> Howdy,
>>>
>>> I bought a 8TB hard drive.  Seagate 8TB 5E8 Exos ST8000AS0003 is the
>>> exact model info.  It seems to be slow.
>>
>> What's the output of:
>>
>> sudo fdisk -l /dev/sda
>>
>> (Assuming it's the sda drive.)
>>
>>
>>
> 
> 
> Well, after a lot more googling, I decided to start over and then
> decided to use a different tool.  I ran dd for several GBs and then used
> gparted to partition and format the drive with ext4.  Right now, it is
> doing the format part.

I'd still like to know what the output of "sudo fdisk -l /dev/sda" is :P

Re: [gentoo-user] Re: Sata hard drive speed question

[Dale]

Nikos Chantziaras wrote:
> On 13/12/2018 09:11, Dale wrote:
>> Nikos Chantziaras wrote:
>>> On 13/12/2018 02:48, Dale wrote:
>>>> Howdy,
>>>>
>>>> I bought a 8TB hard drive.  Seagate 8TB 5E8 Exos ST8000AS0003 is the
>>>> exact model info.  It seems to be slow.
>>>
>>> What's the output of:
>>>
>>> sudo fdisk -l /dev/sda
>>>
>>> (Assuming it's the sda drive.)
>>>
>>>
>>>
>>
>>
>> Well, after a lot more googling, I decided to start over and then
>> decided to use a different tool.  I ran dd for several GBs and then used
>> gparted to partition and format the drive with ext4.  Right now, it is
>> doing the format part.
>
> I'd still like to know what the output of "sudo fdisk -l /dev/sda" is :P
>
>
>


This is what it says right now. 

root@fireball / # fdisk -l /dev/sdb
Disk /dev/sdb: 7.3 TiB, 8001563222016 bytes, 15628053168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 16E55D4E-BA7D-463B-807F-0BE27A488E21

Device     Start         End     Sectors  Size Type
/dev/sdb1   2048 15628052479 15628050432  7.3T Linux filesystem
root@fireball / #

BTW, it's sdb but I know what you wanted.  ;-)  As it is, that was done
with gparted.  It is still trying to put a ext4 file system on it and it
has been about a hour.  If I recall correctly, it took several minutes
on the 6TB drive a while back but nowhere near this long.  There's not
that much difference between 6TB and 8TB.  I might add, I did a
smartctrl -a for that drive, it took a good long while to retrieve the
data.  Generally, it comes back in seconds for other drives.  It seems
that everything is slow for that specific drive. 

While I was typing all that in, it came back with this.


create new ext4 file system  01:05:26    ( ERROR )
        
mkfs.ext4 -F -O ^64bit -L "8tb-backup" /dev/sdb1  01:05:26    ( ERROR )
        
Creating filesystem with 1953506304 4k blocks and 244191232 inodes
Filesystem UUID: 49241f90-62c0-47bf-b3a0-32f2efaa3fed
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000, 214990848, 512000000, 550731776, 644972544, 1934917632

Allocating group tables: done
Writing inode tables: done
Creating journal (262144 blocks): done
Writing superblocks and filesystem accounting information:
mke2fs 1.43.9 (8-Feb-2018)

Warning, had trouble writing out superblocks.


Yea, something isn't right here.  Given I've tried two different tools,
I'm going to check those cables and such.  ;-)

Dale

:-)  :-) 

Re: [gentoo-user] Sata hard drive speed question

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 475 bytes --]

On Wed, 12 Dec 2018 21:36:20 -0600, Dale wrote:

> Googled to see how to find out if it is aligned correctly and found
> this.
> 
> root@fireball / # cat /sys/block/sdb/queue/physical_block_size
> 4096
> root@fireball / #
> 
> I thought cgdisk did that automatically so I guess it did.

gdisk -l will tell you if it is. If the first partition starts at sector
2048 you re OK on that.


-- 
Neil Bothwick

Growing old is mandatory; growing up is optional!!

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] Sata hard drive speed question

[Dale]

Neil Bothwick wrote:
> On Wed, 12 Dec 2018 21:36:20 -0600, Dale wrote:
>
>> Googled to see how to find out if it is aligned correctly and found
>> this.
>>
>> root@fireball / # cat /sys/block/sdb/queue/physical_block_size
>> 4096
>> root@fireball / #
>>
>> I thought cgdisk did that automatically so I guess it did.
> gdisk -l will tell you if it is. If the first partition starts at sector
> 2048 you re OK on that.
>
>

I remember seeing that so it did.  I generally notice when it does that
but I don't give it much thought.  I think it is one of those things
that if I didn't see it there, I'd know something wasn't right and I'd
notice it and check into it. 

I reseated the cables but it's still taking a long time to do anything. 
Given my drive led is on, it's doing something.  I'm just not sure how
fast it is doing it.  o_O 

Dale

:-)  :-) 

Re: [gentoo-user] Sata hard drive speed question

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 442 bytes --]

On Thu, 13 Dec 2018 02:54:07 -0600, Dale wrote:

> I reseated the cables but it's still taking a long time to do anything. 
> Given my drive led is on, it's doing something.  I'm just not sure how
> fast it is doing it.  o_O 

Have you tried running the smartctl selftests?


-- 
Neil Bothwick

WinErr 079: Mouse not found - A mouse driver has not been installed.
            Please click the left mouse button to continue.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[gentoo-user] Re: Sata hard drive speed question

[Nikos Chantziaras]

On 13/12/2018 09:49, Dale wrote:
> This is what it says right now.
> 
> /dev/sdb1   2048 15628052479 15628050432  7.3T Linux filesystem

Just wanted to make sure it's not a 4K alignment issue. It starts at 
2048 so it's fine.


> It is still trying to put a ext4 file system on it and it
> has been about a hour.

I'd recommend just using mkfs instead of using your own parameters:

mkfs -t ext4 /dev/sdb1

It will use the parameters from /etc/mke2fs.conf. This is the safest way 
to format a partition.

Re: [gentoo-user] Sata hard drive speed question

[Dale]

Neil Bothwick wrote:
> On Thu, 13 Dec 2018 02:54:07 -0600, Dale wrote:
>
>> I reseated the cables but it's still taking a long time to do anything. 
>> Given my drive led is on, it's doing something.  I'm just not sure how
>> fast it is doing it.  o_O 
> Have you tried running the smartctl selftests?
>
>


I ran a short one and it said it was all good.  When I try to run the
long one, it keeps aborting.  I'm not sure why it is doing that tho.  I
may just change the sata cable completely.  Bad thing is, it is right
next to the drive my OS is on so I want to shutdown to do that.  Just in
case the wrong one comes unplugged. 

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining 
LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Interrupted (host reset)      00%      
559         -
# 2  Extended offline    Interrupted (host reset)      00%      
556         -
# 3  Short offline       Completed without error       00%      
543         -
# 4  Short offline       Completed without error       00%      
528         -
# 5  Extended offline    Aborted by host               90%      
527         -


I think #4 and 5 were done before I got it. 

Dale

:-)  :-) 

Re: [gentoo-user] Re: Sata hard drive speed question

[Dale]

Nikos Chantziaras wrote:
> On 13/12/2018 09:49, Dale wrote:
>> This is what it says right now.
>>
>> /dev/sdb1   2048 15628052479 15628050432  7.3T Linux filesystem
>
> Just wanted to make sure it's not a 4K alignment issue. It starts at
> 2048 so it's fine.
>
>
>> It is still trying to put a ext4 file system on it and it
>> has been about a hour.
>
> I'd recommend just using mkfs instead of using your own parameters:
>
> mkfs -t ext4 /dev/sdb1
>
> It will use the parameters from /etc/mke2fs.conf. This is the safest
> way to format a partition.
>
>
>

May try that next, if it ever finishes this current attempt.  It's been
a hour for the current format attempt.  I won't be surprised if it gives
up too. 

Dale

:-)  :-) 

P. S.  Where's my sledge hammer at?? 

[gentoo-user] Re: Sata hard drive speed question

[Nikos Chantziaras]

On 13/12/2018 11:18, Dale wrote:
> Nikos Chantziaras wrote:
>> On 13/12/2018 09:49, Dale wrote:
>>> This is what it says right now.
>>>
>>> /dev/sdb1   2048 15628052479 15628050432  7.3T Linux filesystem
>>
>> Just wanted to make sure it's not a 4K alignment issue. It starts at
>> 2048 so it's fine.
>>
>>
>>> It is still trying to put a ext4 file system on it and it
>>> has been about a hour.
>>
>> I'd recommend just using mkfs instead of using your own parameters:
>>
>> mkfs -t ext4 /dev/sdb1
>>
>> It will use the parameters from /etc/mke2fs.conf. This is the safest
>> way to format a partition.
>>
>>
>>
> 
> May try that next, if it ever finishes this current attempt.  It's been
> a hour for the current format attempt.  I won't be surprised if it gives
> up too.

Did you check for any errors in dmesg?

Re: [gentoo-user] Re: Sata hard drive speed question

[Dale]

Nikos Chantziaras wrote:
> On 13/12/2018 11:18, Dale wrote:
>> Nikos Chantziaras wrote:
>>>
>>> I'd recommend just using mkfs instead of using your own parameters:
>>>
>>> mkfs -t ext4 /dev/sdb1
>>>
>>> It will use the parameters from /etc/mke2fs.conf. This is the safest
>>> way to format a partition.
>>>
>>>
>>>
>>
>> May try that next, if it ever finishes this current attempt.  It's been
>> a hour for the current format attempt.  I won't be surprised if it gives
>> up too.
>
> Did you check for any errors in dmesg?
>
>
>


OK.  This is what I did this time.  First, I dd'd the drive, the first
several gigs worth to be sure the partition table etc is gone.  Second,
I ran portprobe for it to see the partition was gone.  It would still
show up in /proc/partitions.  I then used gdisk to create the
partition.  I might add, cgdisk would not run.  It spit out a error and
quit.  Then I ran partprobe again.  May have ran it twice.  Then it
showed up in /proc/partitons as it should.  Then I used your advice and
used mkfs -t ext4 and other options for label etc to format the
partition.  That gave me this:


root@fireball / # time mkfs -v -t ext4 -m 0 -L 8tb-backup /dev/sde1
mke2fs 1.43.9 (8-Feb-2018)
fs_types for mke2fs.conf resolution: 'ext4', 'big'
Filesystem label=8tb-backup
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
244191232 inodes, 1953506385 blocks
0 blocks (0.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4102029312
59617 block groups
32768 blocks per group, 32768 fragments per group
4096 inodes per group
Filesystem UUID: ebcd0ad4-f25f-466e-9b5c-acac33886df0
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632,
2654208,
        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
        102400000, 214990848, 512000000, 550731776, 644972544, 1934917632

Allocating group tables: done                           
Writing inode tables: done                           
Creating journal (262144 blocks): done
Writing superblocks and filesystem accounting information: done      


real    37m50.570s
user    0m0.121s
sys     0m1.639s
root@fireball / #


Before you freak out, I did move the drive to another port when I
changed the cable.  It moved from sdb to sde.  I always confirm using
smartctrl -i until I find the right device. 

After all that, I get this:


    204,807,599 100%  120.79MB/s    0:00:01 (xfr#7946, ir-chk=3715/13065)

120,136,339 100%   77.20MB/s    0:00:01 (xfr#7947, ir-chk=3714/13065)

119,445,345 100%   94.38MB/s    0:00:01 (xfr#7948, ir-chk=3713/13065)

109,298,753 100%  100.81MB/s    0:00:01 (xfr#7949, ir-chk=3712/13065)

116,704,897 100%   82.38MB/s    0:00:01 (xfr#7950, ir-chk=3711/13065)

110,075,610 100%   92.49MB/s    0:00:01 (xfr#7951, ir-chk=3710/13065)

115,757,218 100%  106.46MB/s    0:00:01 (xfr#7952, ir-chk=3709/13065)

111,693,138 100%  128.49MB/s    0:00:00 (xfr#7953, ir-chk=3708/13065)

208,458,508 100%   56.93MB/s    0:00:03 (xfr#7954, ir-chk=3707/13065)

113,847,275 100%   88.92MB/s    0:00:01 (xfr#7955, ir-chk=3706/13065)

181,249,801 100%   79.22MB/s    0:00:02 (xfr#7956, ir-chk=3705/13065)

215,941,705 100%  146.99MB/s    0:00:01 (xfr#7957, ir-chk=3704/13065)


Now I knew this wasn't the fastest drive out there.  It puts a little
more on living a long life at the expense of a little speed.  However,
this is MUCH MUCH better than I was getting.  Since I have a good size
drive now, I'm backing up /home and excluding things I don't care about
like cache and files in the trash etc.  It's a progressive thing. 

At this point, I don't know if it was the cable, me running partprobe or
both that did this.  It could also be running mkfs instead of mkfs.ext4
as well.  Who knows.  I'm just glad to have some SPEED.  O_O 

Thanks much to all. 

Dale

:-)  :-)