From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 62130158011 for ; Sun, 21 Aug 2022 04:23:23 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 039C5E0993; Sun, 21 Aug 2022 04:23:18 +0000 (UTC) Received: from icp-osb-irony-out4.external.iinet.net.au (icp-osb-irony-out4.external.iinet.net.au [203.59.1.220]) by pigeon.gentoo.org (Postfix) with ESMTP id 7AF92E0977 for ; Sun, 21 Aug 2022 04:23:15 +0000 (UTC) IronPort-SDR: Ngklo0uJOQ3Wm/NAfS/eZF0ggs0ea+5KZX4IJ9Ym6ucJcDNM6uIq6m1fWsBf/37kNCJQdzMZJD +d3Toi+cnNV+5zP8oCbatWXZGB8YDtpQoLtIOE0tcpnt1ckERr/mGOc6+VIVp0JDgijulzASjX v9+9DPSpwDeF2MkjlEXXY0xw6+Dm9FAuFEkoMqFnF4vZmzcw5t5ywkpJLRFVVF1VDPROhnvg6R FwAmFqe6dD574/CBArv6P/OZz6rc3VYZem5nhcy8msY/Gvw/LJX4NqDMPtfjZVBBlX36rg7zdI V9o= X-SMTP-MATCH: 0 X-IPAS-Result: =?us-ascii?q?A2AUAQABsgFj/ylZ69xaHQEBAQEJARIBBQUBQAmBNQUBC?= =?us-ascii?q?wGJS5BeCCYDizSFKIw7gWgLAQEBAQEBAQEBCUIEAQGCE4J0AoRiJjcGDgECB?= =?us-ascii?q?AEBAQEDAgMBAQcBAQEFAQEBAQEBBgMBgRyFL0aGQgEBAQECASMPAQVGCwsYA?= =?us-ascii?q?gIRAxICAlcTCAEBgnmCfiWneXqBMYEBigKBESwBhHyKWAcBO32BEIEVJwwDg?= =?us-ascii?q?XNRMD6DfQERAQERAQtVEIMHgmUEhnSGCYQxiDwmBA4DGiseQgIBC0gICRcSE?= =?us-ascii?q?BACBBEaCwYDFj4JAgQOA0AIDQMRBAMPGAkSCBAEBgMxDCULAxQMAQYDBgUDA?= =?us-ascii?q?QMbAxQDBSQHAwIXDyMNDQQYBwwDAwUlAwICGwcCAgMCBhUGAgJOOQgECAQrI?= =?us-ascii?q?w8FAgcvBQQvAh4EBQYRCAIWAgYEBAQEFQIQCAIIJxcHExgbGQEFWRAJIRwKB?= =?us-ascii?q?BoKBgUGEwMgbwUKOw8oMzU5Kx0bCoESKikVAwQEAwIGEwMDIgIQLjEDFQYpE?= =?us-ascii?q?xItByt1CQIDImkFAwMEKCwDCT4HCSImPQUFWzoBBAMDECI9BgMJAwIpOwKab?= =?us-ascii?q?IEbEygkAlkFIAEKCRkRVBYtAgORc5BpnlKDXKAPBg8ELoN2knsIMJFLlwehd?= =?us-ascii?q?zOFNYE/OIEPcE0fGTuCaFAZD1iNVBaOO4EjAgYBCgEBAwmIFSWCIgEB?= IronPort-Data: A9a23:BmNQHq7QyRWlyll77RQagwxRtJ3FchMFZxGqfqrLsTDasY5as4F+v mYWCzuPafuPZjemKIsjbYmxoE1QusfXzNdqTlds/3pkEysa+MHILOrCIxarNUt+DCFjoGGLT ik6QoOdRCzhZiaE/n9BClVgxJVF/fngqoDUUYYoAQgsA14/IMsdoUg7wbRh0tQw2YLR7z6l4 LseneWOYDdJ5BYpagr424rbwP+4lK2v0N+wlgVWicFj5DcypVFMZH4sDfjZw0/Df2VhNrXSq 9AvbV2O1jixEx8FUrtJm56nKRdSGua60QKm0hK6UID66vROjnBpiP5jbJLwZG8P4whlkeydx /1Btt+qdyonDJTvmc8YDjVIHXtaF5JvreqvzXiX6aR/zmXDenrohfRoAls/e4kf8e9zDWZL/ P0eQNwPRkrZ36Tsm+/9Grgq2p1LwMrDZevzvllqzDXdDt4nQJbOX+PM6MMe1SpYasVmRqaGO ZNIMGIxBPjGSwxONFcGLqo5p7ipviPgNDxch2PEgbVitgA/yyQ0itABKuH9ddGMWcJS21uDq 3ju+2XiHgpcO9GZ1T2CtHW2iYfycTjTAthKUefjq7s60RjPnyofGRtQVFq9rOX/jEOiHdtCQ 6AJxhcTQWEJ3BTDZrHAs9eQ+RZoYjZ0twJsLtAH IronPort-HdrOrdr: A9a23:5vFXMKvzL15NrWK7vVqeIQy+7skDUNV00zEX/kB9WHVpm62j+/ xG+c5x6faaslkssR0b9+xoQZPwJ080l6QU3WBhB9aftWDd0QPDQb2KhrGSoAEIdReOjdJ15O NNdLV/Fc21LXUSt7eB3OG0eexQpOVu/MqT9ILjJ30Gd3AJV0luhT0JbTpzy3cGPTWu06BJbK ah2g== X-IronPort-Anti-Spam-Filtered: true X-IronPort-AV: E=Sophos;i="5.93,251,1654531200"; d="scan'208";a="380619106" Received: from 220-235-89-41.dyn.iinet.net.au (HELO mail.infra.localdomain) ([220.235.89.41]) by icp-osb-irony-out4.iinet.net.au with ESMTP; 21 Aug 2022 12:23:12 +0800 Received: from localhost (mail.infra.localdomain [127.0.0.1]) by mail.infra.localdomain (Postfix) with ESMTP id 3E33023AF0 for ; Sun, 21 Aug 2022 12:23:12 +0800 (AWST) X-Virus-Scanned: amavisd-new at localdomain Received: from mail.infra.localdomain ([127.0.0.1]) by localhost (mail.infra.localdomain [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id nbzA6uleh5TJ for ; Sun, 21 Aug 2022 12:22:23 +0800 (AWST) Message-ID: <4c28eb9f-2285-5580-aba3-eca051db3ca5@iinet.net.au> Date: Sun, 21 Aug 2022 12:22:22 +0800 Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-user@lists.gentoo.org Reply-to: gentoo-user@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.12.0 Subject: Re: [gentoo-user] Getting maximum space out of a hard drive Content-Language: en-AU To: gentoo-user@lists.gentoo.org References: <7f23ce9f-7871-c76f-8f50-212e2ff637cf@gmail.com> From: William Kenworthy In-Reply-To: <7f23ce9f-7871-c76f-8f50-212e2ff637cf@gmail.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Archives-Salt: 5850d32b-ec82-4c1f-b92e-cb4348f1db3d X-Archives-Hash: 3f6437bf1e698ff607a980be868dfb69 What are you measuring the speed with - hdparm or rsync or ? hdparm is best for profiling just the harddisk (tallks to the interface and can bypass the cache depending on settings, rsync/cp/?? usually have the whole OS storage chain including encryption affecting throughput.  Encryption itself can be highly variable depending on what you use and usually though not always includes compression before encryption.  There are tools you can use to isolate where the slowdown occurs.  atop is another one that may help. [test using a USB3 shingled drive on a 32 it arm system] xu4 ~ # hdparm -Tt /dev/sda /dev/sda:  Timing cached reads:   1596 MB in  2.00 seconds = 798.93 MB/sec  Timing buffered disk reads: 526 MB in  3.01 seconds = 174.99 MB/sec xu4 ~ # BillK On 21/8/22 06:45, Dale wrote: > Grant Taylor wrote: >> Sorry for the duplicate post.  I had an email client error that >> accidentally caused me to hit send on the window I was composing in. > I figured it was something like that.  ;-) > >> On 8/20/22 1:15 PM, Dale wrote: >>> Howdy, >> Hi, >> >>> Related question.  Does encryption slow the read/write speeds of a >>> drive down a fair amount? >> My experience has been the opposite.  I know that it's unintuitive >> that encryption would make things faster.  But my understanding is >> that it alters how data is read from / written to the disk such that >> it's done in more optimized batches and / or optimized caching. >> >> This was so surprising that I decrypted a drive / re-encrypted a drive >> multiple times to compare things to come to the conclusion that >> encryption was noticeably better. >> >> Plus, encryption has the advantage of destroying the key rendering the >> drive safe to use independent of the data that was on it. >> >> N.B. The actual encryption key is encrypted with the passphrase.  The >> passphrase isn't the encryption key itself. >> >>> This new 10TB drive is maxing out at about 49.51MB/s or so. >> I wonder if you are possibly running into performance issues related >> to shingled drives.  Their raw capacity comes at a performance penalty. > This drive is not supposed to be SMR.  It's a 10TB and according to a > site I looked on, none of them are SMR, yet.  I found another site that > said it was CMR.  So, pretty sure it isn't SMR.  Nothing is 100% tho.  I > might add, it's been at about that speed since I started the backup.  If > you have a better source of info, it's a WD model WD101EDBZ-11B1DA0 drive. > > >>> I actually copied that from the progress of rsync and a nice sized >>> file.  It's been running over 24 hours now so I'd think buffer and >>> cache would be well done with.  LOL >> Ya, you have /probably/ exceeded the write back cache in the system's >> memory. >> >>> It did pass both a short and long self test.  I used cryptsetup -s 512 >>> to encrypt with, nice password too.  My rig has a FX-8350 8 core running >>> at 4GHz CPU and 32GBs of memory.  The CPU is fairly busy.  A little more >>> than normal anyway.  Keep in mind, I have two encrypted drives connected >>> right now. >> The last time I looked at cryptsetup / LUKS, I found that there was a >> [kernel] process per encrypted block device. >> >> A hack that I did while testing things was to slice up a drive into >> multiple partitions, encrypt each one, and then re-aggregate the LUKS >> devices as PVs in LVM.  This surprisingly was a worthwhile performance >> boost. > I noticed there is a kcrypt something thread running, a few actually but > it's hard to keep up since I see it on gkrellm's top process list.  The > CPU is running at about 40% or so average but I do have mplayer, a > couple Firefox profiles, Seamonkey and other stuff running as well.  I > still got plenty of CPU pedal left if needed.  Having Ktorrent and > qbittorrent running together isn't helping.  Thinking of switching > torrent software.  Qbit does seem to use more memory tho. > > >>> Just curious if that speed is normal or not. >> I suspect that your drive is FAR more the bottleneck than the >> encryption itself is.  There is a chance that the encryption's access >> pattern is exascerbating a drive performance issue. >> >>> Thoughts? >> Conceptually working in 512 B blocks on a drive that is natively 4 kB >> sectors.  Thus causing the drive to do lots of extra work to account >> for the other seven 512 B blocks in a 4 kB sector. > I think the 512 has something to do with key size or something.  Am I > wrong on that?  If I need to use 256 or something, I can.  My > understanding was that 512 was stronger than 256 as far as the > encryption goes. > > >>> P. S.  The pulled drive I bought had like 60 hours on it.  Dang near >>> new. >> :-) > I'm going to try some tests Rich mentioned after it is done doing its > backup.  I don't want to stop it if I can avoid it.  It's about half way > through, give or take a little. > > Dale > > :-)  :-) >