[gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[gevisz]

I have bought an external 5TB Western Digital hard drive
that I am going to use mainly for backing up some files
in my home directory and carrying a very big files, for
example a virtual machine image file, from one computer
to another. This hard drive is preformatted with NTFS.
Now, I am going to format it with ext4 which probably
will take a lot of time taking into account that it is
going to be done via USB connection. So, before formatting
this hard drive I would like to know if it is still
advisable to partition big hard drives into smaller
logical ones.

For about 20 last years, following an advice of my older
colleague, I always partitioned all my hard drives into
the smaller logical ones and do very well know all
disadvantages of doing so. :)

But what are disadvantages of not partitioning a big
hard drive into smaller logical ones?

Is it still advisable to partition a big hard drive
into smaller logical ones and why?

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Alan McKinnon]

On 01/09/2016 08:04, gevisz wrote:
> I have bought an external 5TB Western Digital hard drive
> that I am going to use mainly for backing up some files
> in my home directory and carrying a very big files, for
> example a virtual machine image file, from one computer
> to another. This hard drive is preformatted with NTFS.
> Now, I am going to format it with ext4 which probably
> will take a lot of time taking into account that it is
> going to be done via USB connection. So, before formatting
> this hard drive I would like to know if it is still
> advisable to partition big hard drives into smaller
> logical ones.

it will take about 5 seconds to partition it.
And a few more to mkfs it.

Are you sure you aren't thinking of mkfs with ext2 (which did take hours
for a drive that size?

> 
> For about 20 last years, following an advice of my older
> colleague, I always partitioned all my hard drives into
> the smaller logical ones and do very well know all
> disadvantages of doing so. :)

So you are following 20 year-old advice for hardware relevant to 20
years ago and not taking tech advances into account ? :-)

> 
> But what are disadvantages of not partitioning a big
> hard drive into smaller logical ones?

You only get 1 mount point
Some ancient software might whinge and complain about not having a
partition table present.
The drive vendor no longer has a place to put their magic sekrit
phone-home data collection stuff. Oh wait, that's a benefit and belongs
below

> 
> Is it still advisable to partition a big hard drive
> into smaller logical ones and why?

The only reason to partition a drive is to get 2 or more
smaller ones that differ somehow (size, inode ratio, mount options, etc)

Go with no partition table by all means, but if you one day find you
need one, you will have to copy all your data off, repartition, and copy
your data back. If you are certain that will not happen (eg you will
rather buy a second drive) then by all means dispense with partitions.

They are after all nothing more than a Microsoft invention from the 80s
so people could install UCSD Pascal next to MS-DOS


-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[gevisz]

2016-09-01 9:13 GMT+03:00 Alan McKinnon <alan.mckinnon@gmail.com>:
> On 01/09/2016 08:04, gevisz wrote:
>> I have bought an external 5TB Western Digital hard drive
>> that I am going to use mainly for backing up some files
>> in my home directory and carrying a very big files, for
>> example a virtual machine image file, from one computer
>> to another. This hard drive is preformatted with NTFS.
>> Now, I am going to format it with ext4 which probably
>> will take a lot of time taking into account that it is
>> going to be done via USB connection. So, before formatting
>> this hard drive I would like to know if it is still
>> advisable to partition big hard drives into smaller
>> logical ones.
>
> it will take about 5 seconds to partition it.
> And a few more to mkfs it.

Just to partition - may be, but I very much doubt
that it will take seconds to create a full-fledged
ext4 file system on these 5TB via USB2 connention.

Even more: my aquiantance from the Window world
that recomended me this disc scared me that it may
take days...

> Are you sure you aren't thinking of mkfs with ext2
> (which did take hours for a drive that size?
>
>>
>> For about 20 last years, following an advice of my older
>> colleague, I always partitioned all my hard drives into
>> the smaller logical ones and do very well know all
>> disadvantages of doing so. :)
>
> So you are following 20 year-old advice for hardware relevant to 20
> years ago and not taking tech advances into account ? :-)

Yes. But, please, take into account that after these 20 years
I decided to reconsider the old "rule of thumb." :)

>> But what are disadvantages of not partitioning a big
>> hard drive into smaller logical ones?
>
> You only get 1 mount point
> Some ancient software might whinge and complain about not having a
> partition table present.
> The drive vendor no longer has a place to put their magic sekrit
> phone-home data collection stuff. Oh wait, that's a benefit and belongs
> below
>
>>
>> Is it still advisable to partition a big hard drive
>> into smaller logical ones and why?
>
> The only reason to partition a drive is to get 2 or more
> smaller ones that differ somehow (size, inode ratio, mount options, etc)
>
> Go with no partition table by all means, but if you one day find you
> need one, you will have to copy all your data off, repartition, and copy
> your data back. If you are certain that will not happen (eg you will
> rather buy a second drive) then by all means dispense with partitions.
>
> They are after all nothing more than a Microsoft invention from the 80s
> so people could install UCSD Pascal next to MS-DOS

I definitely will not need more than one mount point for this hard drive
but I do remember some arguments that partitioning a large hard drive
into smaller logical ones gives me more safety in case a file system
suddenly will get corrupted because in this case I will loose my data
only on one of the logical partitions and not on the whole drive.

Is this argument still valid nowadays?

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Frank Steinmetzger]

On Thu, Sep 01, 2016 at 08:13:23AM +0200, Alan McKinnon wrote:

> it will take about 5 seconds to partition it.
> And a few more to mkfs it.
>
> Are you sure you aren't thinking of mkfs with ext2 (which did take hours
> for a drive that size?

Some people do a full systems check (i.e. badblocks) before entrusting a
drive with anything important.

> > Is it still advisable to partition a big hard drive
> > into smaller logical ones and why?
> 
> The only reason to partition a drive is to get 2 or more
> smaller ones that differ somehow (size, inode ratio, mount options, etc)

If you want to do backups, then of course the file system is important, so
it retains permissions and stuff. Your ext4 choice is the right one in that
case. However, I partitioned by backupdrive into two partitions, so the one
with the sensitive data can be encrypted. The big partition that holds media
files has not got that treatment.

> Go with no partition table by all means, but if you one day find you
> need one, you will have to copy all your data off, repartition, and copy
> your data back.

When I do the mentioned partitioning sceme, I put the biggest partition at
the beginning of the drive and the smaller one(s) at the back. That way,
should I ever actually need to resize a partition, I only have to export the
smaller partition for the process (or none at all, if it’s just a backup
itself and I have another backup on another drive).
Of course there’s LVM these days, but up until recently, I used NTFS for the
media partition so I could also read it in $DUMB_OS, which doesn’t know LVM.
Only a short while back, I also switched to ext4 for that, so I can retain
file names with : and ? in them. But I still refrained from using LVM,
though.

-- 
Gruß | Greetings | Qapla’
’ve been using vi for 15 years, because I don’t know with which command
to close it.

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Matthias Hanft]

gevisz wrote:
> 
> But what are disadvantages of not partitioning a big
> hard drive into smaller logical ones?

If your filesystem becomes corrupt (and you are unable to
repair it), *all* of your data is lost (instead of just
one partition). That's the only disadvantage I can think
of. I don't like partitions either (after some years, I
always found that sizes don't match my requirements any
more), and therefore, on my new server, I didn't create
any other partitions than "boot":

home01 ~ # df -h -T
Filesystem     Type          Size  Used Avail Use% Mounted on
/dev/sda4      xfs            17T   14T  2.9T  83% /
devtmpfs       devtmpfs       10M     0   10M   0% /dev
tmpfs          tmpfs         3.2G  644K  3.2G   1% /run
shm            tmpfs          16G  512K   16G   1% /dev/shm
/dev/sda2      ext2          124M   46M   72M  39% /boot
ACDFuse        fuse.ACDFuse  100T  284M  100T   1% /mnt/acd
/dev/sdb1      ext3          2.7T  707G  1.9T  28% /mnt/toshiba
home01 ~ #

Backup of important and/or secret files goes to an external
USB hard drive (sdb1) which is formatted with ext3 for
maximum compatibility (every other Linux can read this
without kernel hacks); backup of not-so-secret files goes
to Amazon Cloud Drive (acd) or some other "cloud"; unimportant
files (videos which I have on DVD or BD anyway, or can re-buy)
just aren't backed up.

-Matt

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 624 bytes --]

On Thu, 1 Sep 2016 10:18:29 +0300, gevisz wrote:

> > it will take about 5 seconds to partition it.
> > And a few more to mkfs it.  
> 
> Just to partition - may be, but I very much doubt
> that it will take seconds to create a full-fledged
> ext4 file system on these 5TB via USB2 connention.

Even if that were the case, does it really matter? You said you wanted to
use this drive for backups, surely doing it right is more important than
doing it quickly. It's not like you have to hold its hand while mkfs is
running.


-- 
Neil Bothwick

Bagpipe for free: Stuff cat under arm. Pull legs, chew tail.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 163 bytes --]

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[gevisz]

2016-09-01 10:23 GMT+03:00 Frank Steinmetzger <Warp_7@gmx.de>:
> On Thu, Sep 01, 2016 at 08:13:23AM +0200, Alan McKinnon wrote:
>
>> it will take about 5 seconds to partition it.
>> And a few more to mkfs it.
>>
>> Are you sure you aren't thinking of mkfs with ext2 (which did take hours
>> for a drive that size?
>
> Some people do a full systems check (i.e. badblocks) before entrusting a
> drive with anything important.

It is a good advice! I have already thought of this but I am sorry to
acknowledge
that, since the "old good times" of MS DOS 6.22, I never did this in Linux. :(

And except for one 2.5" disk failure on my old laptop about 7 years ago,
I had no problem with this so far. :)

All other my hard disks work for about 10 years without any intervention
from my side and even without any backups so far. That's why I started
to think about it now. :)

So, can you, please, advice me about the program or utility that can do
badblocks check for me?

>> > Is it still advisable to partition a big hard drive
>> > into smaller logical ones and why?
>>
>> The only reason to partition a drive is to get 2 or more
>> smaller ones that differ somehow (size, inode ratio, mount options, etc)
>
> If you want to do backups, then of course the file system is important, so
> it retains permissions and stuff. Your ext4 choice is the right one in that
> case. However, I partitioned by backupdrive into two partitions, so the one
> with the sensitive data can be encrypted. The big partition that holds media
> files has not got that treatment.

It is, again, a good advice but, again, returning to the "old good times"
of MS DOS 6.22, I do remember that working then on 40MB (yes, megabytes)
hard drive I used some program that compressed all the data before saving
them on that hard drive. Unfortunately, one day, because of the corruption,
I lost all the data on that hard drive. Since then, I am very much afraid of
compressed or encrypted hard drives.

>> Go with no partition table by all means, but if you one day find you
>> need one, you will have to copy all your data off, repartition, and copy
>> your data back.
>
> When I do the mentioned partitioning scheme, I put the biggest partition at
> the beginning of the drive and the smaller one(s) at the back. That way,
> should I ever actually need to resize a partition, I only have to export the
> smaller partition for the process (or none at all, if it’s just a backup
> itself and I have another backup on another drive).
> Of course there’s LVM these days, but up until recently, I used NTFS for the
> media partition so I could also read it in $DUMB_OS, which doesn’t know LVM.
> Only a short while back, I also switched to ext4 for that, so I can retain
> file names with : and ? in them. But I still refrained from using LVM,
> though.

I am afraid of LVM because of the same reason I described above.

> Gruß | Greetings | Qapla’
> ’ve been using vi for 15 years, because I don’t know with which command
> to close it.

:)

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[gevisz]

2016-09-01 10:30 GMT+03:00 Matthias Hanft <mh@hanft.de>:
> gevisz wrote:
>>
>> But what are disadvantages of not partitioning a big
>> hard drive into smaller logical ones?
>
> If your filesystem becomes corrupt (and you are unable to
> repair it), *all* of your data is lost (instead of just
> one partition). That's the only disadvantage I can think
> of.

That is exactly what I am afraid of!

So, the 20-years old rule of thumb is still valid. :(

> I don't like partitions either (after some years, I
> always found that sizes don't match my requirements any
> more),

And this is exactly the reason why I do not want to partition
my new hard drive! :)

> and therefore, on my new server, I didn't create
> any other partitions than "boot":
>
> home01 ~ # df -h -T
> Filesystem     Type          Size  Used Avail Use% Mounted on
> /dev/sda4      xfs            17T   14T  2.9T  83% /
> devtmpfs       devtmpfs       10M     0   10M   0% /dev
> tmpfs          tmpfs         3.2G  644K  3.2G   1% /run
> shm            tmpfs          16G  512K   16G   1% /dev/shm
> /dev/sda2      ext2          124M   46M   72M  39% /boot
> ACDFuse        fuse.ACDFuse  100T  284M  100T   1% /mnt/acd
> /dev/sdb1      ext3          2.7T  707G  1.9T  28% /mnt/toshiba
> home01 ~ #
>
> Backup of important and/or secret files goes to an external
> USB hard drive (sdb1) which is formatted with ext3 for
> maximum compatibility (every other Linux can read this
> without kernel hacks); backup of not-so-secret files goes
> to Amazon Cloud Drive (acd) or some other "cloud"; unimportant
> files (videos which I have on DVD or BD anyway, or can re-buy)
> just aren't backed up.
>
> -Matt
>
>

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 839 bytes --]

On Thu, 1 Sep 2016 11:49:43 +0300, gevisz wrote:

> > If your filesystem becomes corrupt (and you are unable to
> > repair it), *all* of your data is lost (instead of just
> > one partition). That's the only disadvantage I can think
> > of.  
> 
> That is exactly what I am afraid of!
> 
> So, the 20-years old rule of thumb is still valid. :(
> 
> > I don't like partitions either (after some years, I
> > always found that sizes don't match my requirements any
> > more),  
> 
> And this is exactly the reason why I do not want to partition
> my new hard drive! :)

Have you considered LVM? You get the benefits of separate filesystems
without the limitations of inflexible partitioning.


-- 
Neil Bothwick

For a list of all the ways technology has failed to improve the
quality of life, please press three.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 163 bytes --]

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Frank Steinmetzger]

On Thu, Sep 01, 2016 at 11:44:19AM +0300, gevisz wrote:

> > Some people do a full systems check (i.e. badblocks) before entrusting a
> > drive with anything important.
> 
> It is a good advice! I have already thought of this but I am sorry to
> acknowledge that, since the "old good times" of MS DOS 6.22, I never did
> this in Linux. :(
> […]
> So, can you, please, advice me about the program or utility that can do
> badblocks check for me?

Badblocks is part of e2fsprogs. But since you’re using USB2, this will
really take a while. At best I get 39 MB/s out of it. Another way is a
S.M.A.R.T. test, methinks `smartctl -t full` is the command for that. But I
don’t know what exactly is being tested there. But it runs fully internal of
the disk, so no USB2-bottleneck. Others may chime in if I tell fairy tales.

-- 
Gruß | Greetings | Qapla'
I cna ytpe 300 wrods pre mniuet!!!

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[gevisz]

2016-09-01 11:03 GMT+03:00 Neil Bothwick <neil@digimed.co.uk>:
> On Thu, 1 Sep 2016 10:18:29 +0300, gevisz wrote:
>
>> > it will take about 5 seconds to partition it.
>> > And a few more to mkfs it.
>>
>> Just to partition - may be, but I very much doubt
>> that it will take seconds to create a full-fledged
>> ext4 file system on these 5TB via USB2 connention.
>
> Even if that were the case, does it really matter?

You are right: it does not matter for the main question.
However, it is an additional reason to think twice and
ask a knowledgeable people before starting. :)

> You said you wanted to use this drive for backups,
> surely doing it right is more important than doing
> it quickly. It's not like you have to hold its hand
> while mkfs is running.

But I would have to keep my fingers crossed so that
there would not be a sudden blackout during the formatting
hard disk anyway. :)

> Neil Bothwick
>
> Bagpipe for free: Stuff cat under arm. Pull legs, chew tail.

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Alan McKinnon]

On 01/09/2016 09:18, gevisz wrote:
> 2016-09-01 9:13 GMT+03:00 Alan McKinnon <alan.mckinnon@gmail.com>:
>> On 01/09/2016 08:04, gevisz wrote:

[snip]

>> it will take about 5 seconds to partition it.
>> And a few more to mkfs it.
> 
> Just to partition - may be, but I very much doubt
> that it will take seconds to create a full-fledged
> ext4 file system on these 5TB via USB2 connention.


Do it. Tell me how long it tool.

Discussing it without doing it and offering someone else's opinion is a
100% worthless activity

> 
> Even more: my aquiantance from the Window world
> that recomended me this disc scared me that it may
> take days...

Mickey Mouse told me it takes microseconds. So what?

Do it. Tell me how long it took.

>>> Is it still advisable to partition a big hard drive
>>> into smaller logical ones and why?
>>
>> The only reason to partition a drive is to get 2 or more
>> smaller ones that differ somehow (size, inode ratio, mount options, etc)
>>
>> Go with no partition table by all means, but if you one day find you
>> need one, you will have to copy all your data off, repartition, and copy
>> your data back. If you are certain that will not happen (eg you will
>> rather buy a second drive) then by all means dispense with partitions.
>>
>> They are after all nothing more than a Microsoft invention from the 80s
>> so people could install UCSD Pascal next to MS-DOS
> 
> I definitely will not need more than one mount point for this hard drive
> but I do remember some arguments that partitioning a large hard drive
> into smaller logical ones gives me more safety in case a file system
> suddenly will get corrupted because in this case I will loose my data
> only on one of the logical partitions and not on the whole drive.
> 
> Is this argument still valid nowadays?

That is the most stupid dumbass argument I've heard in weeks.
It doesn't even deserve a response.

Who the fuck is promoting this shit?


-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 667 bytes --]

On Thu, 1 Sep 2016 11:59:55 +0300, gevisz wrote:

> > You said you wanted to use this drive for backups,
> > surely doing it right is more important than doing
> > it quickly. It's not like you have to hold its hand
> > while mkfs is running.  
> 
> But I would have to keep my fingers crossed so that
> there would not be a sudden blackout during the formatting
> hard disk anyway. :)

Nah, just start again. It's only after the power goes out for the third
time that you decide the Universe hates you and give up!


-- 
Neil Bothwick

Q: What's the second worst sound you can hear a sysadmin make?
A: Uh-oh
Q: And the worst sound?
A: Oops....

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 163 bytes --]

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[gevisz]

2016-09-01 11:55 GMT+03:00 Frank Steinmetzger <Warp_7@gmx.de>:
> On Thu, Sep 01, 2016 at 11:44:19AM +0300, gevisz wrote:
>
>> > Some people do a full systems check (i.e. badblocks) before entrusting a
>> > drive with anything important.
>>
>> It is a good advice! I have already thought of this but I am sorry to
>> acknowledge that, since the "old good times" of MS DOS 6.22, I never did
>> this in Linux. :(
>> […]
>> So, can you, please, advice me about the program or utility that can do
>> badblocks check for me?
>
> Badblocks is part of e2fsprogs. But since you’re using USB2, this will
> really take a while. At best I get 39 MB/s out of it. Another way is a
> S.M.A.R.T. test, methinks `smartctl -t full` is the command for that. But I
> don’t know what exactly is being tested there. But it runs fully internal of
> the disk, so no USB2-bottleneck. Others may chime in if I tell fairy tales.

Thank you for your advices. I will try both.

> Gruß | Greetings | Qapla'
> I cna ytpe 300 wrods pre mniuet!!!

:)

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[gevisz]

2016-09-01 11:54 GMT+03:00 Neil Bothwick <neil@digimed.co.uk>:
> On Thu, 1 Sep 2016 11:49:43 +0300, gevisz wrote:
>
>> > If your filesystem becomes corrupt (and you are unable to
>> > repair it), *all* of your data is lost (instead of just
>> > one partition). That's the only disadvantage I can think
>> > of.
>>
>> That is exactly what I am afraid of!
>>
>> So, the 20-years old rule of thumb is still valid. :(
>>
>> > I don't like partitions either (after some years, I
>> > always found that sizes don't match my requirements any
>> > more),
>>
>> And this is exactly the reason why I do not want to partition
>> my new hard drive! :)
>
> Have you considered LVM? You get the benefits of separate filesystems
> without the limitations of inflexible partitioning.

I am afraid of LVM because of the same reason as described below:

returning to the "old good times" of MS DOS 6.22, I do remember that working
then on 40MB (yes, megabytes) hard drive I used some program that compressed
all the data before saving them on that hard drive. Unfortunately, one day,
because of the corruption, I lost all the data on that hard drive. Since then,
I am very much afraid of compressed or encrypted hard drives.

> Neil Bothwick
>
> For a list of all the ways technology has failed to improve the
> quality of life, please press three.

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[gevisz]

2016-09-01 12:04 GMT+03:00 Neil Bothwick <neil@digimed.co.uk>:
> On Thu, 1 Sep 2016 11:59:55 +0300, gevisz wrote:
>
>> > You said you wanted to use this drive for backups,
>> > surely doing it right is more important than doing
>> > it quickly. It's not like you have to hold its hand
>> > while mkfs is running.
>>
>> But I would have to keep my fingers crossed so that
>> there would not be a sudden blackout during the formatting
>> hard disk anyway. :)
>
> Nah, just start again. It's only after the power goes out for the third
> time that you decide the Universe hates you and give up!

:)

> Neil Bothwick
>
> Q: What's the second worst sound you can hear a sysadmin make?
> A: Uh-oh
> Q: And the worst sound?
> A: Oops....

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Alan McKinnon]

On 01/09/2016 10:59, gevisz wrote:
> 2016-09-01 11:03 GMT+03:00 Neil Bothwick <neil@digimed.co.uk>:
>> On Thu, 1 Sep 2016 10:18:29 +0300, gevisz wrote:
>>
>>>> it will take about 5 seconds to partition it.
>>>> And a few more to mkfs it.
>>>
>>> Just to partition - may be, but I very much doubt
>>> that it will take seconds to create a full-fledged
>>> ext4 file system on these 5TB via USB2 connention.
>>
>> Even if that were the case, does it really matter?
> 
> You are right: it does not matter for the main question.
> However, it is an additional reason to think twice and
> ask a knowledgeable people before starting. :)
> 
>> You said you wanted to use this drive for backups,
>> surely doing it right is more important than doing
>> it quickly. It's not like you have to hold its hand
>> while mkfs is running.
> 
> But I would have to keep my fingers crossed so that
> there would not be a sudden blackout during the formatting
> hard disk anyway. :)

Even if it does, so what?

Unplug, start over. It's a mkfs, it lays down where the inodes are. You
can do it over and over and over and over .... safely


-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Alan McKinnon]

On 01/09/2016 10:44, gevisz wrote:
> 2016-09-01 10:23 GMT+03:00 Frank Steinmetzger <Warp_7@gmx.de>:
>> On Thu, Sep 01, 2016 at 08:13:23AM +0200, Alan McKinnon wrote:
>>
>>> it will take about 5 seconds to partition it.
>>> And a few more to mkfs it.
>>>
>>> Are you sure you aren't thinking of mkfs with ext2 (which did take hours
>>> for a drive that size?
>>
>> Some people do a full systems check (i.e. badblocks) before entrusting a
>> drive with anything important.
> 
> It is a good advice! I have already thought of this but I am sorry to
> acknowledge
> that, since the "old good times" of MS DOS 6.22, I never did this in Linux. :(
> 
> And except for one 2.5" disk failure on my old laptop about 7 years ago,
> I had no problem with this so far. :)
> 
> All other my hard disks work for about 10 years without any intervention
> from my side and even without any backups so far. That's why I started
> to think about it now. :)
> 
> So, can you, please, advice me about the program or utility that can do
> badblocks check for me?
> 
>>>> Is it still advisable to partition a big hard drive
>>>> into smaller logical ones and why?
>>>
>>> The only reason to partition a drive is to get 2 or more
>>> smaller ones that differ somehow (size, inode ratio, mount options, etc)
>>
>> If you want to do backups, then of course the file system is important, so
>> it retains permissions and stuff. Your ext4 choice is the right one in that
>> case. However, I partitioned by backupdrive into two partitions, so the one
>> with the sensitive data can be encrypted. The big partition that holds media
>> files has not got that treatment.
> 
> It is, again, a good advice but, again, returning to the "old good times"
> of MS DOS 6.22, I do remember that working then on 40MB (yes, megabytes)
> hard drive I used some program that compressed all the data before saving
> them on that hard drive. Unfortunately, one day, because of the corruption,
> I lost all the data on that hard drive. Since then, I am very much afraid of
> compressed or encrypted hard drives.
> 
>>> Go with no partition table by all means, but if you one day find you
>>> need one, you will have to copy all your data off, repartition, and copy
>>> your data back.
>>
>> When I do the mentioned partitioning scheme, I put the biggest partition at
>> the beginning of the drive and the smaller one(s) at the back. That way,
>> should I ever actually need to resize a partition, I only have to export the
>> smaller partition for the process (or none at all, if it’s just a backup
>> itself and I have another backup on another drive).
>> Of course there’s LVM these days, but up until recently, I used NTFS for the
>> media partition so I could also read it in $DUMB_OS, which doesn’t know LVM.
>> Only a short while back, I also switched to ext4 for that, so I can retain
>> file names with : and ? in them. But I still refrained from using LVM,
>> though.
> 
> I am afraid of LVM because of the same reason I described above.

You are allowing your fears of 20 years ago to determine your present
day attitudes. That's silly.

There's a misconception that a drive is somehow a pristine storage
device with pigeon holes where data goes and nothing below the fs level
can have any effect. Nothing could be further from the truth.

The sheer amount of encoding that goes into a drive is almost beyond
belief. It is NOT a digital device, it is analog - exactly like tape,
just many times more complex. It doesn't store bits, it stores
fluctuating regions of local magnetism and all of that gets decoded by
analog circuitry to represent digital bits. About 50% of your drive's
capacity (measured by what the heads see) is devoted to marking where
the tracks go, where the sectors start and end, checksumming and may
other firmware safeguards.

All that stuff can go wrong.

Yes, encryption on-drive can go wrong. So can SSL traffic, gpg keys,
encrypted mail, vpns and all your traffic over the internet (if you're
on a corporate network I can almost assure you it's all encrypted inside
an IPSec tunnel).

That stuff doesn't break. Neither does your disk encryption.

LVM can break, but it's really hard. All a PV is, is a block device with
a 2k signature at the start then some metadata. All a VG is, is a bunch
of PVs and some metadata to list them. An LV is really nothing more than
an indirection lookup table: The fs knows which inode and sectors the
file uses, and the LV has a mapping table to track which real disk
sectors that is.

The kernel does this all the time with every smidgen of RAM you have
(it's how a vmm works). It's the same technology in essence.

So yeah, stuff can break. But that same stuff is used in many other
highly critical areas where you likely don't know of it, and that
doesn't change that it's there.

I know of know recent reports where disk encryption or volume management
broke data solely due to a code bug in stable production versions. Devs
are not that stupid :-)

So honestly, your fears are baseless and have more to do with the
chemical reactions inside your brain that reality.

Sorry to be so blunt, but that's how it is.

-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[gevisz]

2016-09-01 12:01 GMT+03:00 Alan McKinnon <alan.mckinnon@gmail.com>:
> On 01/09/2016 09:18, gevisz wrote:
>> 2016-09-01 9:13 GMT+03:00 Alan McKinnon <alan.mckinnon@gmail.com>:
>>> On 01/09/2016 08:04, gevisz wrote:
>
> [snip]
>
>>>> Is it still advisable to partition a big hard drive
>>>> into smaller logical ones and why?
>>>
>>> The only reason to partition a drive is to get 2 or more
>>> smaller ones that differ somehow (size, inode ratio, mount options, etc)
>>>
>>> Go with no partition table by all means, but if you one day find you
>>> need one, you will have to copy all your data off, repartition, and copy
>>> your data back. If you are certain that will not happen (eg you will
>>> rather buy a second drive) then by all means dispense with partitions.
>>>
>>> They are after all nothing more than a Microsoft invention from the 80s
>>> so people could install UCSD Pascal next to MS-DOS
>>
>> I definitely will not need more than one mount point for this hard drive
>> but I do remember some arguments that partitioning a large hard drive
>> into smaller logical ones gives me more safety in case a file system
>> suddenly will get corrupted because in this case I will loose my data
>> only on one of the logical partitions and not on the whole drive.
>>
>> Is this argument still valid nowadays?
>
> That is the most stupid dumbass argument I've heard in weeks.
> It doesn't even deserve a response.
>
> Who the fuck is promoting this shit?

Even somebody in this thread (in addition to me and independent of me)
made the same arguments.

But I do not state anything, I am just asking.

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Rich Freeman]

On Thu, Sep 1, 2016 at 2:04 AM, gevisz <gevisz@gmail.com> wrote:
>
> Is it still advisable to partition a big hard drive
> into smaller logical ones and why?
>

Assuming this is only used on Linux machines (you mentioned moving
files around), here is what I would do:

1. Definitely create a partition table.  Yes, I know some like to
stick filesystems on raw drives, but you're basically going to fight
all the automation in existence if you do this.
2. Set it up as an LVM partition.  Unless you're using filesystems
like zfs/btrfs that have their own way of doing volume management,
this just makes things less painful down the road.
3. I'd probably just set it up as one big logical volume, unless you
know you don't need all the space and you think you might use it for
something else later.  You can change your mind on this with ext4+lvm
either way, but better to start out whichever way seems best.

It will take you all of 30 seconds to format this, unless you're
running badblocks (which almost nobody does, because...).

You seem to be concerned about losing data.  You should be.  This is a
physical storage device.  You WILL lose everything stored on it at
some point in time.  You mitigate this by one or more of:
1.  Not storing anything you mind losing on the drive, and then not
complaining when you lose it.
2.  Keeping backups, preferably at a different physical location,
using a periodically tested recovery methodology.
3.  Availability solutions like RAID (not the same as a backup, but it
will mean less downtime WHEN you WILL have a drive failure).  Some
filesystems like zfs/btrfs have specific ways of achieving this (and
are generally more resistant to unreliable storage devices, which all
storage devices are).

I've actually had LVM eat my data once due to some kind of really rare
bug (found one discussion of similar issues on some forum somewhere).
That isn't a good reason not to use LVM.  Wanting to plug the drive
into a bunch of Windows machines would be a good reason not to use
LVM, or ext4 for that matter.

Most of the historic reasons for not having large volumes had to do
with addressing limits, whether it be drive geometry limits,
filesystem limits, etc.  Modern partition tables like GPT and
filesystems can handle volumes MUCH larger than 5TB.

Most modern journaling filesystems should also tend to avoid failure
modes like losing the entire filesystem during a power failure (when
correctly used, heaven help you if you follow a random friend's advice
with mount options, like not using at least ordered data or disabling
barriers).  But, bugs can exist, which is a big reason to have backups
and not just trust your filesystem unless you don't care much about
the data.

-- 
Rich

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 1249 bytes --]

On Thu, 1 Sep 2016 12:09:09 +0300, gevisz wrote:

> > Have you considered LVM? You get the benefits of separate filesystems
> > without the limitations of inflexible partitioning.  
> 
> I am afraid of LVM because of the same reason as described below:
> 
> returning to the "old good times" of MS DOS 6.22, I do remember that
> working then on 40MB (yes, megabytes) hard drive I used some program
> that compressed all the data before saving them on that hard drive.
> Unfortunately, one day, because of the corruption, I lost all the data
> on that hard drive. Since then, I am very much afraid of compressed or
> encrypted hard drives.

LVM is neither encrypted nor compressed. The filesystems on it are no
different to the filesystems on physical partitions, and subject to the
same risks. An LVM logical volume is just a block device that is treated
the same as a physical partition on a non-LVM setup.

Sp far, you have come up with reasons, good or otherwise, for not taking
each of the available choices. You need to decide what you really need
and what is important to you. Only then can you decide on the best
arrangement for your needs.


-- 
Neil Bothwick

Evolution stops when stupidity is no longer fatal!

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 163 bytes --]

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Dale]

gevisz wrote:
> 2016-09-01 9:13 GMT+03:00 Alan McKinnon <alan.mckinnon@gmail.com>:
>> On 01/09/2016 08:04, gevisz wrote:
>>> I have bought an external 5TB Western Digital hard drive
>>> that I am going to use mainly for backing up some files
>>> in my home directory and carrying a very big files, for
>>> example a virtual machine image file, from one computer
>>> to another. This hard drive is preformatted with NTFS.
>>> Now, I am going to format it with ext4 which probably
>>> will take a lot of time taking into account that it is
>>> going to be done via USB connection. So, before formatting
>>> this hard drive I would like to know if it is still
>>> advisable to partition big hard drives into smaller
>>> logical ones.
>> it will take about 5 seconds to partition it.
>> And a few more to mkfs it.
> Just to partition - may be, but I very much doubt
> that it will take seconds to create a full-fledged
> ext4 file system on these 5TB via USB2 connention.
>
> Even more: my aquiantance from the Window world
> that recomended me this disc scared me that it may
> take days...
>

Something to think on.  You have a 5TB drive.  You format the whole
thing and let's say it takes 30 seconds.  Or, you break it into two
2.5TB partitions and then format those, which take 20 or 25 seconds
each.  That adds up to 40 to 50 seconds format time.  Isn't it faster to
format one large partition instead of two?  After all, you have to type
the command in to format it too which also takes a few seconds, assuming
you up arrow and just edit the partition letter.  No matter whether you
break the drive up into parts or not, you are still formatting 5TBs
worth of drive.  The only way you can save time is to not format the
whole thing. 

Things break.  They always have and always will.  Sure you can prepare
for that lose but if not careful, you could lose it while you are
second, third, forth etc etc etc guessing yourself and what tool you are
going to use.  I suspect that every file system out there has caused a
person to lose data before.  I'm sure that every brand and even model of
hard drive out there has caused someone to lose data before.  When it
gets as complex has a hard drive and the tools used on them, it has to
break at some point.  The best bet, duplicate your files just in case
something in the above list goes bad.  The only advice I think would be
good on this, don't use the same brand and model drive for both main and
backup.  One could even say not to use the same file system, that way if
one goes bad due to bad coding in the kernel, likely the other shouldn't
be affected, but even that can't be a for sure thing. 

I hope you aren't to worried about making a backup now that you can
think on how everything fails eventually.  ;-)

Dale

:-)  :-) 

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Michael Mol]

[-- Attachment #1: Type: text/plain, Size: 2147 bytes --]


On Thursday, September 01, 2016 12:09:09 PM gevisz wrote:
> 2016-09-01 11:54 GMT+03:00 Neil Bothwick <neil@digimed.co.uk>:
> > On Thu, 1 Sep 2016 11:49:43 +0300, gevisz wrote:
> >> > If your filesystem becomes corrupt (and you are unable to
> >> > repair it), *all* of your data is lost (instead of just
> >> > one partition). That's the only disadvantage I can think
> >> > of.
> >> 
> >> That is exactly what I am afraid of!
> >> 
> >> So, the 20-years old rule of thumb is still valid. :(
> >> 
> >> > I don't like partitions either (after some years, I
> >> > always found that sizes don't match my requirements any
> >> > more),
> >> 
> >> And this is exactly the reason why I do not want to partition
> >> my new hard drive! :)
> > 
> > Have you considered LVM? You get the benefits of separate filesystems
> > without the limitations of inflexible partitioning.
> 
> I am afraid of LVM because of the same reason as described below:
> 
> returning to the "old good times" of MS DOS 6.22, I do remember that working
> then on 40MB (yes, megabytes) hard drive I used some program that
> compressed all the data before saving them on that hard drive.
> Unfortunately, one day, because of the corruption, I lost all the data on
> that hard drive. Since then, I am very much afraid of compressed or
> encrypted hard drives.

LVM doesn't *need* to do any of that. It will only do as much as you tell it 
to do. If you only want to use it as a way of reshaping relatively simple 
partitions, you can use it for that.

Honestly, I tend not to create separate partitions for separate mount points 
these days. At least, not on personal systems. For servers, it's can be 
beneficial to have /var separate from /, or /var/log separate from /var, or 
/var/spool, or /var/lib/mysql, or what have you. But the biggest driver for 
that, IME, is if one of those fills up, it can't take down the rest of the 
host.

In your case, I'd suggest using a single / filesystem. If it works, it works. 
If it doesn't, you'll know in the future where you need to be more flexible; 
there's no single panacea.

-- 
:wq

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Rich Freeman]

On Thu, Sep 1, 2016 at 8:51 AM, Michael Mol <mikemol@gmail.com> wrote:
>
> Honestly, I tend not to create separate partitions for separate mount points
> these days. At least, not on personal systems. For servers, it's can be
> beneficial to have /var separate from /, or /var/log separate from /var, or
> /var/spool, or /var/lib/mysql, or what have you. But the biggest driver for
> that, IME, is if one of those fills up, it can't take down the rest of the
> host.
>

The other big use case these days would be SSDs.  I tend to have one
SSD filesystem for root, and one SSD filesystem for everything else.
That means a lot of bind mounts, but it all works.  I'm not about to
get into separate filesystems for random directories in var that tend
to get big.

-- 
Rich

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 813 bytes --]

On Thu, 1 Sep 2016 09:27:39 -0400, Rich Freeman wrote:

> > Honestly, I tend not to create separate partitions for separate mount
> > points these days. At least, not on personal systems. For servers,
> > it's can be beneficial to have /var separate from /, or /var/log
> > separate from /var, or /var/spool, or /var/lib/mysql, or what have
> > you. But the biggest driver for that, IME, is if one of those fills
> > up, it can't take down the rest of the host.

> The other big use case these days would be SSDs.  I tend to have one
> SSD filesystem for root, and one SSD filesystem for everything else.
> That means a lot of bind mounts, but it all works.

Bind mounts? I thought you would use btrfs subvolumes!


-- 
Neil Bothwick

Make it idiot proof and someone will make a better idiot.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 163 bytes --]

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Rich Freeman]

On Thu, Sep 1, 2016 at 1:42 PM, Neil Bothwick <neil@digimed.co.uk> wrote:
> On Thu, 1 Sep 2016 09:27:39 -0400, Rich Freeman wrote:
>
>> > Honestly, I tend not to create separate partitions for separate mount
>> > points these days. At least, not on personal systems. For servers,
>> > it's can be beneficial to have /var separate from /, or /var/log
>> > separate from /var, or /var/spool, or /var/lib/mysql, or what have
>> > you. But the biggest driver for that, IME, is if one of those fills
>> > up, it can't take down the rest of the host.
>
>> The other big use case these days would be SSDs.  I tend to have one
>> SSD filesystem for root, and one SSD filesystem for everything else.
>> That means a lot of bind mounts, but it all works.
>
> Bind mounts? I thought you would use btrfs subvolumes!
>

Often the bind mounts point to btrfs subvolumes.

Yeah, I guess I could directly mount all those subvolumes, but I find
symlinks or bind mounts easier.  The other factor is that if I have
unnecessary subvolumes then I'm having to manage snapshots across more
of them and my snapshots are less atomic, since snapshots don't cross
subvolume boundaries (which is something which ought to be
configurable).


-- 
Rich

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[gevisz]

2016-09-01 14:55 GMT+03:00 Rich Freeman <rich0@gentoo.org>:
> On Thu, Sep 1, 2016 at 2:04 AM, gevisz <gevisz@gmail.com> wrote:
>>
>> Is it still advisable to partition a big hard drive
>> into smaller logical ones and why?
>>
>
> Assuming this is only used on Linux machines (you mentioned moving
> files around), here is what I would do:
>
> 1. Definitely create a partition table.  Yes, I know some like to
> stick filesystems on raw drives, but you're basically going to fight
> all the automation in existence if you do this.

I will do it with gparted and guess that it will create a partition table
for me anyway.

> 2. Set it up as an LVM partition.  Unless you're using filesystems
> like zfs/btrfs that have their own way of doing volume management,
> this just makes things less painful down the road.
>
> 3. I'd probably just set it up as one big logical volume, unless you
> know you don't need all the space and you think you might use it for
> something else later.  You can change your mind on this with ext4+lvm
> either way, but better to start out whichever way seems best.

I had to refresh my memory about LVM before replying to you
but still can not see why I may need LVM on an external
hard drive...

> It will take you all of 30 seconds to format this, unless you're
> running badblocks (which almost nobody does, because...

it takes too much time?

I currently running a smart test on it, and it promised to take
10 hours to complete...

> You seem to be concerned about losing data.  You should be.  This is a
> physical storage device.  You WILL lose everything stored on it at
> some point in time.

Last time, I have managed to restore all the data from my 2.5" hard
drive that suddenly died about 7 years ago and hope to do it again
if any. :)

>  You mitigate this by one or more of:
> 1.  Not storing anything you mind losing on the drive, and then not
> complaining when you lose it.
> 2.  Keeping backups, preferably at a different physical location,
> using a periodically tested recovery methodology.
> 3.  Availability solutions like RAID (not the same as a backup, but it
> will mean less downtime WHEN you WILL have a drive failure).  Some
> filesystems like zfs/btrfs have specific ways of achieving this (and
> are generally more resistant to unreliable storage devices, which all
> storage devices are).
>
> I've actually had LVM eat my data once due to some kind of really rare
> bug (found one discussion of similar issues on some forum somewhere).

Aha!

> That isn't a good reason not to use LVM.  Wanting to plug the drive
> into a bunch of Windows machines would be a good reason not to use
> LVM, or ext4 for that matter.
>
> Most of the historic reasons for not having large volumes had to do
> with addressing limits, whether it be drive geometry limits,
> filesystem limits, etc.  Modern partition tables like GPT and
> filesystems can handle volumes MUCH larger than 5TB.
>
> Most modern journaling filesystems should also tend to avoid failure
> modes like losing the entire filesystem during a power failure (when
> correctly used, heaven help you if you follow a random friend's advice
> with mount options, like not using at least ordered data or disabling
> barriers).  But, bugs can exist, which is a big reason to have backups
> and not just trust your filesystem unless you don't care much about
> the data.

Thank you for replying.

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[gevisz]

2016-09-01 15:21 GMT+03:00 Neil Bothwick <neil@digimed.co.uk>:
> On Thu, 1 Sep 2016 12:09:09 +0300, gevisz wrote:
>
>> > Have you considered LVM? You get the benefits of separate filesystems
>> > without the limitations of inflexible partitioning.
>>
>> I am afraid of LVM because of the same reason as described below:
>>
>> returning to the "old good times" of MS DOS 6.22, I do remember that
>> working then on 40MB (yes, megabytes) hard drive I used some program
>> that compressed all the data before saving them on that hard drive.
>> Unfortunately, one day, because of the corruption, I lost all the data
>> on that hard drive. Since then, I am very much afraid of compressed or
>> encrypted hard drives.
>
> LVM is neither encrypted nor compressed. The filesystems on it are no
> different to the filesystems on physical partitions, and subject to the
> same risks. An LVM logical volume is just a block device that is treated
> the same as a physical partition on a non-LVM setup.

Thank you for the explanation, I have also just refreshed my memory
about LVM before replying to you but still can not see any reason why
I may need LVM on an external hard drive...

> Sp far, you have come up with reasons, good or otherwise, for not taking
> each of the available choices. You need to decide what you really need
> and what is important to you. Only then can you decide on the best
> arrangement for your needs.
>
> Neil Bothwick
>
> Evolution stops when stupidity is no longer fatal!

:)

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Rich Freeman]

On Thu, Sep 1, 2016 at 2:58 PM, gevisz <gevisz@gmail.com> wrote:
> 2016-09-01 14:55 GMT+03:00 Rich Freeman <rich0@gentoo.org>:
>
>> 2. Set it up as an LVM partition.  Unless you're using filesystems
>> like zfs/btrfs that have their own way of doing volume management,
>> this just makes things less painful down the road.
>>
>> 3. I'd probably just set it up as one big logical volume, unless you
>> know you don't need all the space and you think you might use it for
>> something else later.  You can change your mind on this with ext4+lvm
>> either way, but better to start out whichever way seems best.
>
> I had to refresh my memory about LVM before replying to you
> but still can not see why I may need LVM on an external
> hard drive...

It just gives you more options in the future, it is easy to move LVM
volumes to other drives, re-partition them later, and so on.  I agree
it is probably overkill on a removable device, but it doesn't hurt.
This is a 5TB drive after all.  But, I don't think it is
super-critical either.

>
>> It will take you all of 30 seconds to format this, unless you're
>> running badblocks (which almost nobody does, because...
>
> it takes too much time?
>
> I currently running a smart test on it, and it promised to take
> 10 hours to complete...

That's basically it.  If it didn't take time people would of course
run it first.  I think a SMART test would be about as good and likely
a lot faster.  However, the drive should be managing bad blocks on its
own (granted, many drives seem to get that wrong in my experience,
which is part of why I run btrfs, but I probably wouldn't use
btrfs/zfs for a drive you're moving all over the place since who knows
what kind of kernel you'll have when you use it and heaven help you if
you ever need to read it on Windows).

>
>> You seem to be concerned about losing data.  You should be.  This is a
>> physical storage device.  You WILL lose everything stored on it at
>> some point in time.
>
> Last time, I have managed to restore all the data from my 2.5" hard
> drive that suddenly died about 7 years ago and hope to do it again
> if any. :)

Well, if the data is redundant then you're fine (it is essentially
already backed up).  But, you should check those backups from time to
time.

You should never rely on the ability to recover data from a hard
drive.  For starters, if you just lose the thing (portable things can
sometimes grow legs; you're talking about 5 libraries of congress in a
bag that could get stolen) or it is catastrophically destroyed that
isn't going to work.  Short of that there is a fair chance you can get
a lot of data off the drive, and it is fairly likely if you're using
some kind of expensive recovery service, but you can't promise that
the specific file you care about most will get recovered.

Backups are annoying.  I don't do them as well as ideally I should
(way too much data to get it all offsite), but I make a conscious
decision about what does/doesn't get backed up and how.  I
occasionally restore my encrypted cloud backups to confirm they
contain what I expect them to.  I actually get the log summary emailed
daily to make sure it is running (if I had more hosts I could use some
kind of monitoring for that...).  I've never needed to use the online
cloud backups, but they're there for a reason and they cover anything
I actually care about (documents and such).  I also backup all my
cloud services (evernote, google drive, etc) to local storage
occassionally; that doesn't require further backup since it is the
backup.  You just need two copies of everything, with one copy
preferably being inaccessible from the other and not at the same
physical site.

-- 
Rich

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[gevisz]

2016-09-01 15:51 GMT+03:00 Michael Mol <mikemol@gmail.com>:
>
> On Thursday, September 01, 2016 12:09:09 PM gevisz wrote:
>> 2016-09-01 11:54 GMT+03:00 Neil Bothwick <neil@digimed.co.uk>:
>> > On Thu, 1 Sep 2016 11:49:43 +0300, gevisz wrote:
>> >> > If your filesystem becomes corrupt (and you are unable to
>> >> > repair it), *all* of your data is lost (instead of just
>> >> > one partition). That's the only disadvantage I can think
>> >> > of.
>> >>
>> >> That is exactly what I am afraid of!
>> >>
>> >> So, the 20-years old rule of thumb is still valid. :(
>> >>
>> >> > I don't like partitions either (after some years, I
>> >> > always found that sizes don't match my requirements any
>> >> > more),
>> >>
>> >> And this is exactly the reason why I do not want to partition
>> >> my new hard drive! :)
>> >
>> > Have you considered LVM? You get the benefits of separate filesystems
>> > without the limitations of inflexible partitioning.
>>
>> I am afraid of LVM because of the same reason as described below:
>>
>> returning to the "old good times" of MS DOS 6.22, I do remember that working
>> then on 40MB (yes, megabytes) hard drive I used some program that
>> compressed all the data before saving them on that hard drive.
>> Unfortunately, one day, because of the corruption, I lost all the data on
>> that hard drive. Since then, I am very much afraid of compressed or
>> encrypted hard drives.
>
> LVM doesn't *need* to do any of that. It will only do as much as you tell it
> to do. If you only want to use it as a way of reshaping relatively simple
> partitions, you can use it for that.
>
> Honestly, I tend not to create separate partitions for separate mount points
> these days. At least, not on personal systems. For servers, it's can be
> beneficial to have /var separate from /, or /var/log separate from /var, or
> /var/spool, or /var/lib/mysql, or what have you. But the biggest driver for
> that, IME, is if one of those fills up, it can't take down the rest of the
> host.
>
> In your case, I'd suggest using a single / filesystem. If it works, it works.
> If it doesn't, you'll know in the future where you need to be more flexible;
> there's no single panacea.

Thank you for the reply. And I even agree with you to the point that
on a Linux desktop it may be enough to have just 3 different partitions:
one - for /, second - for swap (yes, one can do without it nowadays),
and third - for /home. But you probably missed the point that it goes
about an external drive dedicated to backups only.

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[gevisz]

2016-09-01 22:12 GMT+03:00 Rich Freeman <rich0@gentoo.org>:
> On Thu, Sep 1, 2016 at 2:58 PM, gevisz <gevisz@gmail.com> wrote:
>> 2016-09-01 14:55 GMT+03:00 Rich Freeman <rich0@gentoo.org>:
>>
>>> 2. Set it up as an LVM partition.  Unless you're using filesystems
>>> like zfs/btrfs that have their own way of doing volume management,
>>> this just makes things less painful down the road.
>>>
>>> 3. I'd probably just set it up as one big logical volume, unless you
>>> know you don't need all the space and you think you might use it for
>>> something else later.  You can change your mind on this with ext4+lvm
>>> either way, but better to start out whichever way seems best.
>>
>> I had to refresh my memory about LVM before replying to you
>> but still can not see why I may need LVM on an external
>> hard drive...
>
> It just gives you more options in the future,

Yes, thank you.

> it is easy to move LVM volumes to other drives, re-partition them later,
> and so on.

I still suspect that this extra level of complexity can complicate recovery
of the data, if anything happens to the disk under LVM management (except
for stealing the hard drive, of course :).

>  I agree it is probably overkill on a removable device, but it doesn't hurt.
> This is a 5TB drive after all.  But, I don't think it is super-critical either.
>
>>
>>> It will take you all of 30 seconds to format this, unless you're
>>> running badblocks (which almost nobody does, because...
>>
>> it takes too much time?
>>
>> I currently running a smart test on it, and it promised to take
>> 10 hours to complete...
>
> That's basically it.  If it didn't take time people would of course
> run it first.  I think a SMART test would be about as good and likely
> a lot faster.  However, the drive should be managing bad blocks on its
> own (granted, many drives seem to get that wrong in my experience,
> which is part of why I run btrfs, but I probably wouldn't use
> btrfs/zfs for a drive you're moving all over the place since who knows
> what kind of kernel you'll have when you use it and heaven help you if
> you ever need to read it on Windows).

It is not a question of using the disk with Windows, but I too often see
some reports about problems in using btrfs on this list to try using it
myself...

>>> You seem to be concerned about losing data.  You should be.  This is a
>>> physical storage device.  You WILL lose everything stored on it at
>>> some point in time.
>>
>> Last time, I have managed to restore all the data from my 2.5" hard
>> drive that suddenly died about 7 years ago and hope to do it again
>> if any. :)
>
> Well, if the data is redundant then you're fine (it is essentially
> already backed up).

No, that data was not backed up.

But I am not guaranteed to be so lucky again, of course. :(

That is why I decided to finally start to back up my data. :)

>  But, you should check those backups from time to time.
>
> You should never rely on the ability to recover data from a hard
> drive.  For starters, if you just lose the thing (portable things can
> sometimes grow legs; you're talking about 5 libraries of congress in a
> bag that could get stolen) or it is catastrophically destroyed that
> isn't going to work.  Short of that there is a fair chance you can get
> a lot of data off the drive, and it is fairly likely if you're using
> some kind of expensive recovery service, but you can't promise that
> the specific file you care about most will get recovered.
>
> Backups are annoying.

Yes. :)

>  I don't do them as well as ideally I should

Who does? :)

Well, probably, one who just lost a lot of data because of not doing backup. :)

> (way too much data to get it all offsite), but I make a conscious
> decision about what does/doesn't get backed up and how.  I
> occasionally restore my encrypted cloud backups to confirm they
> contain what I expect them to.  I actually get the log summary emailed
> daily to make sure it is running (if I had more hosts I could use some
> kind of monitoring for that...).  I've never needed to use the online
> cloud backups, but they're there for a reason and they cover anything
> I actually care about (documents and such).  I also backup all my
> cloud services (evernote, google drive, etc) to local storage
> occassionally; that doesn't require further backup since it is the
> backup.  You just need two copies of everything, with one copy
> preferably being inaccessible from the other and not at the same
> physical site.

Well, thank you for your advices.

[gentoo-user] Re: [OT] Is it still advisable to partition a big hard drive?

[Kai Krakow]

Am Thu, 1 Sep 2016 22:56:06 +0300
schrieb gevisz <gevisz@gmail.com>:

> > Backups are annoying.  
> 
> Yes. :)

No, try borgbackup with a cronjob.

> >  I don't do them as well as ideally I should  
> 
> Who does? :)

I do.

> Well, probably, one who just lost a lot of data because of not doing
> backup. :)

Data that you do not backup is unimportant data - by definition. It's
that simple. ;-)

You also care about your money and put it on a bank account, keeping
your PIN secret, and neither let money or PIN lying around for everyone
to grab.


-- 
Regards,
Kai

Replies to list-only preferred.

[gentoo-user] Re: [OT] Is it still advisable to partition a big hard drive?

[Kai Krakow]

Am Thu, 1 Sep 2016 09:04:53 +0300
schrieb gevisz <gevisz@gmail.com>:

> I have bought an external 5TB Western Digital hard drive
> that I am going to use mainly for backing up some files
> in my home directory and carrying a very big files, for
> example a virtual machine image file, from one computer
> to another. This hard drive is preformatted with NTFS.
> Now, I am going to format it with ext4 which probably
> will take a lot of time taking into account that it is
> going to be done via USB connection. So, before formatting
> this hard drive I would like to know if it is still
> advisable to partition big hard drives into smaller
> logical ones.
> 
> For about 20 last years, following an advice of my older
> colleague, I always partitioned all my hard drives into
> the smaller logical ones and do very well know all
> disadvantages of doing so. :)

This has been a bad advice for at least the last 15 years when the last
DOS-based machines died.

The reasoning behind this:

Hard drives are really bad at performance after the first third of
storage space (do a benchmark, transfer speed will almost half).

Next, how do you decide in front how big a partition should be? Your OS
partition will become too small one day or another - your are going to
put big files (swap files, program files) into the other partition. See
previous point: This is the slow one.

By this process, you will now artificially put a big gap into OS
related files - this clearly counterfeits your original intention of
keeping OS files close together.

Most current OSes are good at keeping related files close together
(except maybe Windows after a few Windows Updates runs, but there's
software like MyDefrag to fix this and restore original performance),
or there's technology to mitigate this issue (like bcache in Linux). I
think even Windows has an optimization of allocating swap space nearby
the heads current position, so swap fragmentation isn't even an issue.

The advice which I was always given and refused, since more than 15
years:

	"But if you reinstall, you then don't have to restore all your
	data, and settings, and you can even install your programs to
	the other partition to not loose data and programs..."

Sorry, bu****it. If you expect this to work (at least on Windows, but
that's were the example is from), you will be really disappointed if
you relied on that in case of a disaster: Windows simply stored all
your settings and secret program data files on its C drive - which is
gone. The installed programs are not there or do not work because
Windows simply has no knowledge of them in the other partition after
you reinstall, and even when you manage to start/reinstall them: Their
state is kinda unknown or reset because "ProgramData" is missing. So
this setup is a complete waste of performance and time. And there's no
easy way to fix this. Tricks like symlinking C:\Users to another drive
or use a submount are unsupported and updates will eventually fail to
do this.

I selected Windows here as the example because I expect the advice you
mentioned comes from Windows installations.

Linux, by design, works a lot better here. But still my advice is:
Never ever partition for this reasoning. Even less, if performance is
your concern.

> But what are disadvantages of not partitioning a big
> hard drive into smaller logical ones?

Usually, none. At least for ordinary usage. Performance-wise it's
always a better choice to use multiple physical disks if you need
different partitions. A valid reason for separate partitions (read:
physical drives) is special purpose software like a doctor's office
software which puts all its data and shares below a single directory
structure.

> Is it still advisable to partition a big hard drive
> into smaller logical ones and why?

No. If you want it for logical management, there are much better ways of
achieving this (like fs-integrated pooling, LVM, separate physical
drives selected for their special purpose).

Regarding performance:

I wish Linux had options to relocate files (not just defragment) back
into logical groups for nearby access. Fragmentation is less of a
problem, the bigger problem is data block dislocation over time due to
updates. In Windows, there's the wonderful tool MyDefrag which does
magic and puts your aging Windows installation back into a state of an
almost fresh installation by relocating files to sane positions.

Is there anything similar for Linux?


-- 
Regards,
Kai

Replies to list-only preferred.

[gentoo-user] Re: [OT] Is it still advisable to partition a big hard drive?

[Kai Krakow]

[-- Attachment #1: Type: text/plain, Size: 1512 bytes --]

Am Thu, 1 Sep 2016 09:03:17 +0100
schrieb Neil Bothwick <neil@digimed.co.uk>:

> On Thu, 1 Sep 2016 10:18:29 +0300, gevisz wrote:
> 
> > > it will take about 5 seconds to partition it.
> > > And a few more to mkfs it.    
> > 
> > Just to partition - may be, but I very much doubt
> > that it will take seconds to create a full-fledged
> > ext4 file system on these 5TB via USB2 connention.  
> 
> Even if that were the case, does it really matter? You said you
> wanted to use this drive for backups, surely doing it right is more
> important than doing it quickly. It's not like you have to hold its
> hand while mkfs is running.

If you want to use it as a backup, and it's external (which it should
be), by all means: partition it. It acts as a protection layer against
silly OSes that may simply wipe data at the beginning (maybe by
accident) because there is no partition and they cannot detect that
there is data stored on the disk, and this destroy your fs superblock -
which is what you don't want.

History shows, that in case of disaster, you may attach your disk to
some recovery environment, just to find: your backup has been destroyed
now by accident.

Or someone else attaches this drive (out of curiosity or whatever) to
some silly OS, it asks "do you want to initialize this drive?" - "ah,
yes, of course sir". Bam. Gone. Not good. With a partition table this
does not happen.

PS: Yes, I mean Windows.

-- 
Regards,
Kai

Replies to list-only preferred.

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[gentoo-user] Re: [OT] Is it still advisable to partition a big hard drive?

[Kai Krakow]

Am Thu, 1 Sep 2016 12:09:09 +0300
schrieb gevisz <gevisz@gmail.com>:

> 2016-09-01 11:54 GMT+03:00 Neil Bothwick <neil@digimed.co.uk>:
> > On Thu, 1 Sep 2016 11:49:43 +0300, gevisz wrote:
> >  
>  [...]  
> >>
> >> That is exactly what I am afraid of!
> >>
> >> So, the 20-years old rule of thumb is still valid. :(
> >>  
>  [...]  
> >>
> >> And this is exactly the reason why I do not want to partition
> >> my new hard drive! :)  
> >
> > Have you considered LVM? You get the benefits of separate
> > filesystems without the limitations of inflexible partitioning.  
> 
> I am afraid of LVM because of the same reason as described below:
> 
> returning to the "old good times" of MS DOS 6.22, I do remember that
> working then on 40MB (yes, megabytes) hard drive I used some program
> that compressed all the data before saving them on that hard drive.
> Unfortunately, one day, because of the corruption, I lost all the
> data on that hard drive. Since then, I am very much afraid of
> compressed or encrypted hard drives.

You are talking of software like Double Disk or what it was called.
This is a complete different scenario, it was a virtual filesystem
inside a filesystem. You layered two indirections on top of each other.
Chances were, if the top layer broke somewhere, the lower layer became
inaccessible. You have been hit by this problem.

LVM works different. It allocates huge blocks as virtual partitions, and
doesn't indirect each single fs-level block in a fine-grained
structure. Of course, there's still the chance that the descriptive
block of LVM can become corrupted - but so can your partition table.
The solution is simple: dump the LVM configuration block that holds the
on-disk structure - then you can replay it. The same, as you would do
with partition tables: Back them up to replay them.

Backup the whole fs if data is important. If you are low on storage but
retention is important, I'd recommend borgbackup.


-- 
Regards,
Kai

Replies to list-only preferred.

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 1122 bytes --]

On Thu, 1 Sep 2016 22:04:22 +0300, gevisz wrote:

> > LVM is neither encrypted nor compressed. The filesystems on it are no
> > different to the filesystems on physical partitions, and subject to
> > the same risks. An LVM logical volume is just a block device that is
> > treated the same as a physical partition on a non-LVM setup.  
> 
> Thank you for the explanation, I have also just refreshed my memory
> about LVM before replying to you but still can not see any reason why
> I may need LVM on an external hard drive...

You gave on in the post that I replied to suggesting LVM in the first
place - that's why I suggested it.

You were worrying about the difficulty of altering a partition layout
once it is committed to disk and filled with data. LVM removes that
problem, because volumes and filesystems can be resized, added and
deleted at will.

However, at no point did I state that you "need" it, only that it may be
useful. The location of the drive is less relevant that its capacity when
considering this.


-- 
Neil Bothwick

A clean desk is a sign of a cluttered desk drawer.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 163 bytes --]

Re: [gentoo-user] Re: [OT] Is it still advisable to partition a big hard drive?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 752 bytes --]

On Thu, 1 Sep 2016 23:06:56 +0200, Kai Krakow wrote:

> If you want to use it as a backup, and it's external (which it should
> be), by all means: partition it. It acts as a protection layer against
> silly OSes that may simply wipe data at the beginning (maybe by
> accident) because there is no partition and they cannot detect that
> there is data stored on the disk, and this destroy your fs superblock -
> which is what you don't want.

That's an excellent point. I've just started using unpartitioned disks
with btrfs, but they are safely tucked away inside my computer. I can see
how having no partition table on an external disk is asking for trouble.


-- 
Neil Bothwick

Data to Picard: 'No, Captain, I do NOT run WINDOWS!'

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 163 bytes --]

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Alan McKinnon]

On 01/09/2016 10:49, gevisz wrote:
> 2016-09-01 10:30 GMT+03:00 Matthias Hanft <mh@hanft.de>:
>> gevisz wrote:
>>>
>>> But what are disadvantages of not partitioning a big
>>> hard drive into smaller logical ones?
>>
>> If your filesystem becomes corrupt (and you are unable to
>> repair it), *all* of your data is lost (instead of just
>> one partition). That's the only disadvantage I can think
>> of.
>
> That is exactly what I am afraid of!
>
> So, the 20-years old rule of thumb is still valid. :(

No, it is not valid, and it is not true.

Data corruption on-disk does not by and large (unless you are very 
unlucky) corrupt file systems. It corrupts files.

Secondly, by and large, most people have all the files they really care 
about on one partition, called DATA or similar. Everything else except 
your data can usually be reconstructed, especially the OS itself. You 
probably store all that data in one volume simply because it makes 
logical sense to do so. Data is read and written far more than anything 
else on your disk so if you are unlucky enough to suffer volume 
corruption it's likely to be on a) the biggest volume and b) the busiest 
volume. In both cases it is your data, meaning your data is what is 
exposed to risk and everything else not so much.

Yes, this is a real factor you mention. It is detectable and 
measureable. It's also minute and statistically irrelevant if you 
haven't dealt with environmental factors that cause data damage (dodgy 
ram, cables, psus, over-temps, brownouts). If those things happen, and 
they WILL happen, you are 10-20 times at least more likely to lose your 
data than anything else, no matter how you partitioned the disk.

[gentoo-user] Re: [OT] Is it still advisable to partition a big hard drive?

[Kai Krakow]

Am Fri, 2 Sep 2016 01:53:31 +0200
schrieb Alan McKinnon <alan.mckinnon@gmail.com>:

> On 01/09/2016 10:49, gevisz wrote:
> > 2016-09-01 10:30 GMT+03:00 Matthias Hanft <mh@hanft.de>:  
> >> gevisz wrote:  
>  [...]  
> >>
> >> If your filesystem becomes corrupt (and you are unable to
> >> repair it), *all* of your data is lost (instead of just
> >> one partition). That's the only disadvantage I can think
> >> of.  
> >
> > That is exactly what I am afraid of!
> >
> > So, the 20-years old rule of thumb is still valid. :(  
> 
> No, it is not valid, and it is not true.
> 
> Data corruption on-disk does not by and large (unless you are very 
> unlucky) corrupt file systems. It corrupts files.
> 
> Secondly, by and large, most people have all the files they really
> care about on one partition, called DATA or similar. Everything else
> except your data can usually be reconstructed, especially the OS
> itself. You probably store all that data in one volume simply because
> it makes logical sense to do so. Data is read and written far more
> than anything else on your disk so if you are unlucky enough to
> suffer volume corruption it's likely to be on a) the biggest volume
> and b) the busiest volume. In both cases it is your data, meaning
> your data is what is exposed to risk and everything else not so much.

This is one of the best points, and very easy to follow. *thumbsup*

> Yes, this is a real factor you mention. It is detectable and 
> measureable. It's also minute and statistically irrelevant if you 
> haven't dealt with environmental factors that cause data damage
> (dodgy ram, cables, psus, over-temps, brownouts). If those things
> happen, and they WILL happen, you are 10-20 times at least more
> likely to lose your data than anything else, no matter how you
> partitioned the disk.

So you can store everything in the same partition anyways. Especially
since Windows doesn't have this distinction that all data should be
where the user thinks it is (on "DATA or similar"). So an important part
of your data is still on the OS drive anyways. Instead, make a backup
of the complete user profile.

But one should take into account: Not only the data has value. Also the
work needed to reconstruct the OS and applications has value. So better
put it in the backup, too. One more point for using one partition.

-- 
Regards,
Kai

Replies to list-only preferred.

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[gevisz]

2016-09-01 11:55 GMT+03:00 Frank Steinmetzger <Warp_7@gmx.de>:
> On Thu, Sep 01, 2016 at 11:44:19AM +0300, gevisz wrote:
>
>> > Some people do a full systems check (i.e. badblocks) before entrusting a
>> > drive with anything important.
>>
>> It is a good advice! I have already thought of this but I am sorry to
>> acknowledge that, since the "old good times" of MS DOS 6.22, I never did
>> this in Linux. :(
>> […]
>> So, can you, please, advice me about the program or utility that can do
>> badblocks check for me?
>
> Badblocks is part of e2fsprogs. But since you’re using USB2, this will
> really take a while. At best I get 39 MB/s out of it. Another way is a
> S.M.A.R.T. test, methinks `smartctl -t full` is the command for that. But I
> don’t know what exactly is being tested there. But it runs fully internal of
> the disk, so no USB2-bottleneck. Others may chime in if I tell fairy tales.

So far, the hard drive passed two (small) smart tests started by commands:
# smartctl -c -t short -d sat /dev/sdc
and
# smartctl -t conveyance -d sat /dev/sdc
as is indicated by
# smartctl -l selftest -d sat /dev/sdc
smartctl 6.4 2015-06-04 r4109 [x86_64-linux-4.4.6-gentoo] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining
LifeTime(hours)  LBA_of_first_error
# 1  Conveyance offline  Completed without error       00%         0         -
# 2  Short offline       Completed without error       00%         0         -

However, after running
# smartctl -t long -d sat /dev/sdc
I have no indication that it has been passed:
# smartctl -t long -d sat /dev/sdc
smartctl 6.4 2015-06-04 r4109 [x86_64-linux-4.4.6-gentoo] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
Sending command: "Execute SMART Extended self-test routine immediately
in off-line mode".
Drive command "Execute SMART Extended self-test routine immediately in
off-line mode" successful.
Testing has begun.
Please wait 571 minutes for test to complete.
Test will complete after Fri Sep  2 04:02:18 2016

Use smartctl -X to abort test.

Fri Sep 2 6:10
# smartctl -l selftest -d sat /dev/sdc
smartctl 6.4 2015-06-04 r4109 [x86_64-linux-4.4.6-gentoo] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org

Read Device Identity failed: scsi error device will be ready soon

A mandatory SMART command failed: exiting. To continue, add one or
more '-T permissive' options.

# smartctl -a -d sat /dev/sdc
smartctl 6.4 2015-06-04 r4109 [x86_64-linux-4.4.6-gentoo] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     WDC WD50EZRZ-00GZ5B1
Serial Number:   <snipped>
LU WWN Device Id: <snipped>
Firmware Version: 80.00A80
User Capacity:    5,000,981,078,016 bytes [5.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5700 rpm
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Fri Sep  2 06:12:50 2016 EEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x80)    Offline data collection activity
                    was never started.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)    The previous self-test
routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:         (57180) seconds.
Offline data collection
capabilities:              (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   2) minutes.
Extended self-test routine
recommended polling time:      ( 571) minutes.
Conveyance self-test routine
recommended polling time:      (   5) minutes.
SCT capabilities:            (0x3035)    SCT Status supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE
UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   100   253   051    Pre-fail
Always       -       0
  3 Spin_Up_Time            0x0027   198   198   021    Pre-fail
Always       -       9083
  4 Start_Stop_Count        0x0032   100   100   000    Old_age
Always       -       7
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail
Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age
Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age
Always       -       1
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age
Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age
Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age
Always       -       7
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age
Always       -       1
193 Load_Cycle_Count        0x0032   200   200   000    Old_age
Always       -       10
194 Temperature_Celsius     0x0022   124   105   000    Old_age
Always       -       28
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age
Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age
Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age
Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age
Always       -       0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age
Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining
LifeTime(hours)  LBA_of_first_error
# 1  Conveyance offline  Completed without error       00%         0         -
# 2  Short offline       Completed without error       00%         0         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[gevisz]

2016-09-02 7:23 GMT+03:00 gevisz <gevisz@gmail.com>:
> 2016-09-01 11:55 GMT+03:00 Frank Steinmetzger <Warp_7@gmx.de>:
>> On Thu, Sep 01, 2016 at 11:44:19AM +0300, gevisz wrote:
>>
>>> > Some people do a full systems check (i.e. badblocks) before entrusting a
>>> > drive with anything important.
>>>
>>> It is a good advice! I have already thought of this but I am sorry to
>>> acknowledge that, since the "old good times" of MS DOS 6.22, I never did
>>> this in Linux. :(
>>> […]
>>> So, can you, please, advice me about the program or utility that can do
>>> badblocks check for me?
>>
>> Badblocks is part of e2fsprogs. But since you’re using USB2, this will
>> really take a while. At best I get 39 MB/s out of it. Another way is a
>> S.M.A.R.T. test, methinks `smartctl -t full` is the command for that. But I
>> don’t know what exactly is being tested there. But it runs fully internal of
>> the disk, so no USB2-bottleneck. Others may chime in if I tell fairy tales.
>
> So far, the hard drive passed two (small) smart tests started by commands:
> # smartctl -c -t short -d sat /dev/sdc
> and
> # smartctl -t conveyance -d sat /dev/sdc

<skipped>

> However, after running
> # smartctl -t long -d sat /dev/sdc
> I have no indication that it has been passed:
> # smartctl -t long -d sat /dev/sdc
> smartctl 6.4 2015-06-04 r4109 [x86_64-linux-4.4.6-gentoo] (local build)
> Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org
>
> === START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
> Sending command: "Execute SMART Extended self-test routine immediately
> in off-line mode".
> Drive command "Execute SMART Extended self-test routine immediately in
> off-line mode" successful.
> Testing has begun.
> Please wait 571 minutes for test to complete.
> Test will complete after Fri Sep  2 04:02:18 2016
>
> Use smartctl -X to abort test.
>
> Fri Sep 2 6:10
> # smartctl -l selftest -d sat /dev/sdc
> smartctl 6.4 2015-06-04 r4109 [x86_64-linux-4.4.6-gentoo] (local build)
> Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org
>
> Read Device Identity failed: scsi error device will be ready soon
>
> A mandatory SMART command failed: exiting. To continue, add one or
> more '-T permissive' options.

Well, may be, it has not been finished yet.

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 882 bytes --]

On Thu, 1 Sep 2016 13:54:40 -0400, Rich Freeman wrote:

> > Bind mounts? I thought you would use btrfs subvolumes!
> >  
> 
> Often the bind mounts point to btrfs subvolumes.
> 
> Yeah, I guess I could directly mount all those subvolumes, but I find
> symlinks or bind mounts easier.  The other factor is that if I have
> unnecessary subvolumes then I'm having to manage snapshots across more
> of them and my snapshots are less atomic, since snapshots don't cross
> subvolume boundaries (which is something which ought to be
> configurable).

I use a script to handle my snapshots, so snapshotting multiple
subvolumes is less of an issue, but an option to snapshot a subvolume and
all its children, or even the whole filesystem, would be nice.


-- 
Neil Bothwick

Remember, it takes 47 muscles to frown
And only 4 to pull the trigger of a sniper rifle....

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 163 bytes --]

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Rich Freeman]

On Fri, Sep 2, 2016 at 6:59 AM, Neil Bothwick <neil@digimed.co.uk> wrote:
> On Thu, 1 Sep 2016 13:54:40 -0400, Rich Freeman wrote:
>
>> > Bind mounts? I thought you would use btrfs subvolumes!
>> >
>>
>> Often the bind mounts point to btrfs subvolumes.
>>
>> Yeah, I guess I could directly mount all those subvolumes, but I find
>> symlinks or bind mounts easier.  The other factor is that if I have
>> unnecessary subvolumes then I'm having to manage snapshots across more
>> of them and my snapshots are less atomic, since snapshots don't cross
>> subvolume boundaries (which is something which ought to be
>> configurable).
>
> I use a script to handle my snapshots, so snapshotting multiple
> subvolumes is less of an issue, but an option to snapshot a subvolume and
> all its children, or even the whole filesystem, would be nice.
>

Note that what I want is a snapshot that crosses subvolume boundaries,
so that it is atomic.  Not a program that just iterates creating
individual snapshots that don't all happen at the exact same time.

I'd have to look a little more closely at how the filesystem roots
work to see if that is actually possible.  I don't know if the root
node actually covers all the subvolumes it contains, and how exactly
subvolumes are bound to their containing directories.

I guess if the structure of the tree doesn't allow a single snapshot
at the data structure level another option would be for the filesystem
to create a write barrier / lock of some kind while the snapshots are
being created, so they end up being consistent anyway.  This approach
could work even across different filesystems.

-- 
Rich

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 683 bytes --]

On Fri, 2 Sep 2016 07:48:17 -0400, Rich Freeman wrote:

> > I use a script to handle my snapshots, so snapshotting multiple
> > subvolumes is less of an issue, but an option to snapshot a subvolume
> > and all its children, or even the whole filesystem, would be nice.
> >  
> 
> Note that what I want is a snapshot that crosses subvolume boundaries,
> so that it is atomic.  Not a program that just iterates creating
> individual snapshots that don't all happen at the exact same time.

Yes, I get that. It would be a nice feature.


-- 
Neil Bothwick

If the bank returns your cheque marked "Insufficient Funds," call them
 and ask if they mean you or them. :-)

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 163 bytes --]

Re: [gentoo-user] Re: [OT] Is it still advisable to partition a big hard drive?

[Mick]

[-- Attachment #1: Type: text/plain, Size: 826 bytes --]

On Thursday 01 Sep 2016 22:57:12 Kai Krakow wrote:

> Regarding performance:
> 
> I wish Linux had options to relocate files (not just defragment) back
> into logical groups for nearby access. Fragmentation is less of a
> problem, the bigger problem is data block dislocation over time due to
> updates. In Windows, there's the wonderful tool MyDefrag which does
> magic and puts your aging Windows installation back into a state of an
> almost fresh installation by relocating files to sane positions.
> 
> Is there anything similar for Linux?

Dale will pop in soon to mention the defrag application he was running on 
reiserfs, but a potentially more effective defrag method irrespective of fs 
(we're talking about spinning disks where this issue applies) is tar off/tar on 
your data.
-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [gentoo-user] Re: [OT] Is it still advisable to partition a big hard drive?

[Dale]

[-- Attachment #1: Type: text/plain, Size: 1826 bytes --]

Mick wrote:
> On Thursday 01 Sep 2016 22:57:12 Kai Krakow wrote:
>
>> Regarding performance:
>>
>> I wish Linux had options to relocate files (not just defragment) back
>> into logical groups for nearby access. Fragmentation is less of a
>> problem, the bigger problem is data block dislocation over time due to
>> updates. In Windows, there's the wonderful tool MyDefrag which does
>> magic and puts your aging Windows installation back into a state of an
>> almost fresh installation by relocating files to sane positions.
>>
>> Is there anything similar for Linux?
> Dale will pop in soon to mention the defrag application he was running on 
> reiserfs, but a potentially more effective defrag method irrespective of fs 
> (we're talking about spinning disks where this issue applies) is tar off/tar on 
> your data.

Now someone is asking for me to post something.  ROFL 

Script should be attached.  Be forewarned, I have not used this script
in ages.  I have no clue if it works or not or if it will totally screw
up anything and everything.  I would recommend trying it on something
that doesn't matter or maybe a directory full of copied files to be
sure.  If it hoses your system, it's not my script and you been warned. 
I'm not even sure where I got it from.  Might be the forums but could be
anywhere. 

By the way, I switched to ext4 and it has a defrag command of its own. 
Just man e4defrag for details, assuming you have the ext utilities
package installed.  That would be sys-fs/e2fsprogs by the way.  I
*think* it works on ext3 as well but not sure.  Everything here is ext4
except /boot which is ext2. 

I guess this is the benefit of large hard drives.  I don't have to
delete stuff even if I don't use it for a long time.  lol 

Y'all have fun. 

Dale

:-)  :-) 

[-- Attachment #2: fragck.pl --]
[-- Type: application/x-perl, Size: 1303 bytes --]

Re: [gentoo-user] Re: [OT] Is it still advisable to partition a big hard drive?

[J. Roeleveld]

On Friday, September 02, 2016 05:42:13 PM Dale wrote:
> Mick wrote:
> > On Thursday 01 Sep 2016 22:57:12 Kai Krakow wrote:
> >> Regarding performance:
> >> 
> >> I wish Linux had options to relocate files (not just defragment) back
> >> into logical groups for nearby access. Fragmentation is less of a
> >> problem, the bigger problem is data block dislocation over time due to
> >> updates. In Windows, there's the wonderful tool MyDefrag which does
> >> magic and puts your aging Windows installation back into a state of an
> >> almost fresh installation by relocating files to sane positions.
> >> 
> >> Is there anything similar for Linux?
> > 
> > Dale will pop in soon to mention the defrag application he was running on
> > reiserfs, but a potentially more effective defrag method irrespective of
> > fs
> > (we're talking about spinning disks where this issue applies) is tar
> > off/tar on your data.
> 
> Now someone is asking for me to post something.  ROFL
> 
> Script should be attached.  Be forewarned, I have not used this script
> in ages.  I have no clue if it works or not or if it will totally screw
> up anything and everything.  I would recommend trying it on something
> that doesn't matter or maybe a directory full of copied files to be
> sure.  If it hoses your system, it's not my script and you been warned.
> I'm not even sure where I got it from.  Might be the forums but could be
> anywhere.
> 
> By the way, I switched to ext4 and it has a defrag command of its own.
> Just man e4defrag for details, assuming you have the ext utilities
> package installed.  That would be sys-fs/e2fsprogs by the way.  I
> *think* it works on ext3 as well but not sure.  Everything here is ext4
> except /boot which is ext2.
> 
> I guess this is the benefit of large hard drives.  I don't have to
> delete stuff even if I don't use it for a long time.  lol
> 
> Y'all have fun.

How does that script work?
From a quick look, it depends on some application called "filefrag".
I can't seem to find that on my system.

--
Joost

[gentoo-user] Re: [OT] Is it still advisable to partition a big hard drive?

[Kai Krakow]

Am Fri, 2 Sep 2016 17:42:13 -0500
schrieb Dale <rdalek1967@gmail.com>:

> Mick wrote:
> > On Thursday 01 Sep 2016 22:57:12 Kai Krakow wrote:
> >  
> >> Regarding performance:
> >>
> >> I wish Linux had options to relocate files (not just defragment)
> >> back into logical groups for nearby access. Fragmentation is less
> >> of a problem, the bigger problem is data block dislocation over
> >> time due to updates. In Windows, there's the wonderful tool
> >> MyDefrag which does magic and puts your aging Windows installation
> >> back into a state of an almost fresh installation by relocating
> >> files to sane positions.
> >>
> >> Is there anything similar for Linux?  
> > Dale will pop in soon to mention the defrag application he was
> > running on reiserfs, but a potentially more effective defrag method
> > irrespective of fs (we're talking about spinning disks where this
> > issue applies) is tar off/tar on your data.  
> 
> Now someone is asking for me to post something.  ROFL 
> 
> Script should be attached.  Be forewarned, I have not used this script
> in ages.  I have no clue if it works or not or if it will totally
> screw up anything and everything.  I would recommend trying it on
> something that doesn't matter or maybe a directory full of copied
> files to be sure.  If it hoses your system, it's not my script and
> you been warned. I'm not even sure where I got it from.  Might be the
> forums but could be anywhere. 
> 
> By the way, I switched to ext4 and it has a defrag command of its
> own. Just man e4defrag for details, assuming you have the ext
> utilities package installed.  That would be sys-fs/e2fsprogs by the
> way.  I *think* it works on ext3 as well but not sure.  Everything
> here is ext4 except /boot which is ext2. 
> 
> I guess this is the benefit of large hard drives.  I don't have to
> delete stuff even if I don't use it for a long time.  lol 
> 
> Y'all have fun. 

Well, this is not exactly what I was asking for. I think defragmenting
files is really not that important as long as the fragments have some
sane minimum sizes. I think something like contiguous chunks of 4 MB
are enough for performance, SuSE seems to suggest 32 MB when you are
looking at their btrfs maintenance script (it doesn't consider extents
of more than 32MB for defragmentation).

Much more important is to have executables, libs and data files nearby
that are typically loaded at same time. The preload application
(adaptive preload daemon) already does the right analysis by recording
which files are needed and uses markov chains to predict which files
you are going to need next to preload them into the page cache. I think,
this data could also be used to rearrange files into better on-disk
locations. Also, I think exploiting the page cache for this may not
always be the best idea because in the end you may not need this data
and it will push other important data out of cache.

I think there's e4rat which already rearranges boot-related files to
the start of the disk but it's ext[34] only. I think this technology
could be developed further by clustering files needed by applications
you start together.

-- 
Regards,
Kai

Replies to list-only preferred.

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Stroller]

> On 2 Sep 2016, at 23:03, Mick <michaelkintzios@gmail.com> wrote:
> 
> … a potentially more effective defrag method irrespective of fs 
> (we're talking about spinning disks where this issue applies) i

I understood that fragmentation can also occur on flash-based disks.

Although the effect of it is not so noticeable, I understood that it still has one.

Stroller.

Re: [gentoo-user] Re: [OT] Is it still advisable to partition a big hard drive?

[Dale]

J. Roeleveld wrote:
> On Friday, September 02, 2016 05:42:13 PM Dale wrote:
>> Mick wrote:
>>> On Thursday 01 Sep 2016 22:57:12 Kai Krakow wrote:
>>>> Regarding performance:
>>>>
>>>> I wish Linux had options to relocate files (not just defragment) back
>>>> into logical groups for nearby access. Fragmentation is less of a
>>>> problem, the bigger problem is data block dislocation over time due to
>>>> updates. In Windows, there's the wonderful tool MyDefrag which does
>>>> magic and puts your aging Windows installation back into a state of an
>>>> almost fresh installation by relocating files to sane positions.
>>>>
>>>> Is there anything similar for Linux?
>>> Dale will pop in soon to mention the defrag application he was running on
>>> reiserfs, but a potentially more effective defrag method irrespective of
>>> fs
>>> (we're talking about spinning disks where this issue applies) is tar
>>> off/tar on your data.
>> Now someone is asking for me to post something.  ROFL
>>
>> Script should be attached.  Be forewarned, I have not used this script
>> in ages.  I have no clue if it works or not or if it will totally screw
>> up anything and everything.  I would recommend trying it on something
>> that doesn't matter or maybe a directory full of copied files to be
>> sure.  If it hoses your system, it's not my script and you been warned.
>> I'm not even sure where I got it from.  Might be the forums but could be
>> anywhere.
>>
>> By the way, I switched to ext4 and it has a defrag command of its own.
>> Just man e4defrag for details, assuming you have the ext utilities
>> package installed.  That would be sys-fs/e2fsprogs by the way.  I
>> *think* it works on ext3 as well but not sure.  Everything here is ext4
>> except /boot which is ext2.
>>
>> I guess this is the benefit of large hard drives.  I don't have to
>> delete stuff even if I don't use it for a long time.  lol
>>
>> Y'all have fun.
> How does that script work?
> >From a quick look, it depends on some application called "filefrag".
> I can't seem to find that on my system.
>
> --
> Joost
>
>


As I said, it's not my script.  I just got it from somewhere.  I have no
idea if it works much less how it works.  Based on the time stamp, it's
at least 10 years old and no telling how old it was before I got it. 

Dale

:-)  :-) 

Re: [gentoo-user] Re: [OT] Is it still advisable to partition a big hard drive?

[Dale]

Kai Krakow wrote:
> Am Fri, 2 Sep 2016 17:42:13 -0500
> schrieb Dale <rdalek1967@gmail.com>:
>
>> Mick wrote:
>>> On Thursday 01 Sep 2016 22:57:12 Kai Krakow wrote:
>>>  
>>>> Regarding performance:
>>>>
>>>> I wish Linux had options to relocate files (not just defragment)
>>>> back into logical groups for nearby access. Fragmentation is less
>>>> of a problem, the bigger problem is data block dislocation over
>>>> time due to updates. In Windows, there's the wonderful tool
>>>> MyDefrag which does magic and puts your aging Windows installation
>>>> back into a state of an almost fresh installation by relocating
>>>> files to sane positions.
>>>>
>>>> Is there anything similar for Linux?  
>>> Dale will pop in soon to mention the defrag application he was
>>> running on reiserfs, but a potentially more effective defrag method
>>> irrespective of fs (we're talking about spinning disks where this
>>> issue applies) is tar off/tar on your data.  
>> Now someone is asking for me to post something.  ROFL 
>>
>> Script should be attached.  Be forewarned, I have not used this script
>> in ages.  I have no clue if it works or not or if it will totally
>> screw up anything and everything.  I would recommend trying it on
>> something that doesn't matter or maybe a directory full of copied
>> files to be sure.  If it hoses your system, it's not my script and
>> you been warned. I'm not even sure where I got it from.  Might be the
>> forums but could be anywhere. 
>>
>> By the way, I switched to ext4 and it has a defrag command of its
>> own. Just man e4defrag for details, assuming you have the ext
>> utilities package installed.  That would be sys-fs/e2fsprogs by the
>> way.  I *think* it works on ext3 as well but not sure.  Everything
>> here is ext4 except /boot which is ext2. 
>>
>> I guess this is the benefit of large hard drives.  I don't have to
>> delete stuff even if I don't use it for a long time.  lol 
>>
>> Y'all have fun. 
> Well, this is not exactly what I was asking for. I think defragmenting
> files is really not that important as long as the fragments have some
> sane minimum sizes. I think something like contiguous chunks of 4 MB
> are enough for performance, SuSE seems to suggest 32 MB when you are
> looking at their btrfs maintenance script (it doesn't consider extents
> of more than 32MB for defragmentation).
>
> Much more important is to have executables, libs and data files nearby
> that are typically loaded at same time. The preload application
> (adaptive preload daemon) already does the right analysis by recording
> which files are needed and uses markov chains to predict which files
> you are going to need next to preload them into the page cache. I think,
> this data could also be used to rearrange files into better on-disk
> locations. Also, I think exploiting the page cache for this may not
> always be the best idea because in the end you may not need this data
> and it will push other important data out of cache.
>
> I think there's e4rat which already rearranges boot-related files to
> the start of the disk but it's ext[34] only. I think this technology
> could be developed further by clustering files needed by applications
> you start together.
>

All that sounds good but I don't know of any such tool to do that.  Some
people wanted a way to defrag things so someone wrote the script I
posted.  It worked back then but to be honest, I don't think defragging
is even really needed on Linux and any reasonably modern file system,
excluding the windozish ones that Linux can access like fat etc. 
Whenever I use such a tool or run some tool that shows fragmentation, it
is either none existent or so small that it doesn't matter.  Then there
is always those files that because of size, will always be fragmented. 

I guess no matter how fast hard drives get, someone will always want to
squeeze out just a little tiny fraction of more speed.  ;-)

Dale

:-)  :-) 

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Mick]

[-- Attachment #1: Type: text/plain, Size: 738 bytes --]

On Saturday 03 Sep 2016 16:39:03 Stroller wrote:
> > On 2 Sep 2016, at 23:03, Mick <michaelkintzios@gmail.com> wrote:
> > 
> > … a potentially more effective defrag method irrespective of fs
> > (we're talking about spinning disks where this issue applies) i
> 
> I understood that fragmentation can also occur on flash-based disks.
> 
> Although the effect of it is not so noticeable, I understood that it still
> has one.
> 
> Stroller.

Yes, flash drives (unlike spinning drivers) are completely digital.  In 
addition, wear levelling algorithms invariably kick in and bits and bytes are 
sprayed all over the pages/modules of the memory chips.  So you could say they 
are fragmented by design.

-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Stroller]

> On 3 Sep 2016, at 17:50, Mick <michaelkintzios@gmail.com> wrote:
>> 
>> I understood that fragmentation can also occur on flash-based disks.
>> 
>> Although the effect of it is not so noticeable, I understood that it still
>> has one.
> 
> Yes, flash drives (unlike spinning drivers) are completely digital.  In 
> addition, wear levelling algorithms invariably kick in and bits and bytes are 
> sprayed all over the pages/modules of the memory chips.  So you could say they 
> are fragmented by design.

That would seem to dismiss the problem, "oh, they're fragmented by design, thus it's unimportant".

My understanding is that defragmenting a flash device (although I think, personally, I would only do this by deleting all the files on the drive, and copying them back) can make for faster access.

• http://www.lagom.nl/misc/flash_fragmentation.html
• http://www.wizcode.com/articles/comments/flash_memory_fragmentation_myths_and_facts/

Stroller.

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Mick]

[-- Attachment #1: Type: text/plain, Size: 2591 bytes --]

On Sunday 04 Sep 2016 17:48:14 Stroller wrote:
> > On 3 Sep 2016, at 17:50, Mick <michaelkintzios@gmail.com> wrote:

> > Yes, flash drives (unlike spinning drivers) are completely digital.  In
> > addition, wear levelling algorithms invariably kick in and bits and bytes
> > are sprayed all over the pages/modules of the memory chips.  So you could
> > say they are fragmented by design.
> 
> That would seem to dismiss the problem, "oh, they're fragmented by design,
> thus it's unimportant".

I'm far from an expert on NOR/NAND flash drives and therefore I didn't mean to 
sound dismissive.  I was merely highlighting the fact that the memory 
controller on these cards interferes with whatever our OS is trying to write 
on them, as the card's chip controller implements various wear levelling 
algorithms.


> My understanding is that defragmenting a flash device (although I think,
> personally, I would only do this by deleting all the files on the drive,
> and copying them back) can make for faster access.
> 
> • http://www.lagom.nl/misc/flash_fragmentation.html
> •
> http://www.wizcode.com/articles/comments/flash_memory_fragmentation_myths_a
> nd_facts/
> 
> Stroller.

Some of these tests assume that flushing the OS cache *also* flushes the cache 
on the flash drive.  This is not so, especially on more modern flush drives.  
I've been watching the behaviour of a Verbatim 32G USB stick I use more or 
less daily and I am convinced that running sync following a copy operation on 
my PC, in no way means the cache on the flash controller is also flushed.

What these tests prove is that when the card is full it takes longer to write 
content on it, because blocks will have to be erased before they can be 
written on.  The cluster size is quite important for this performance, as is 
the size of the file(s) being copied.

What I am saying is that the write operation performance is determined by the 
cluster size, the file size, the flash drive's cache size and most importantly 
by the flash drive controller's wear levelling algorithms.  There is no 
guarantee that data will be written contiguously, although they will be 
written in one-block-at-a-time.  The blocks themselves almost certainly will 
not be contiguous on a used drive.  Formatting it with unsuitable logical 
block sizes for its physical block size will almost certainly incur a write 
penalty (always depending on the size of the file being written).

This is what I meant when I said USB flash drives are fragmented by design.
-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

[gentoo-user] Re: [OT] Is it still advisable to partition a big hard drive?

[Hans]

On 01/09/16 16:04, gevisz wrote:
> I have bought an external 5TB Western Digital hard drive
> that I am going to use mainly for backing up some files
> in my home directory and carrying a very big files, for
> example a virtual machine image file, from one computer
> to another. This hard drive is preformatted with NTFS.
> Now, I am going to format it with ext4 which probably
> will take a lot of time taking into account that it is
> going to be done via USB connection. So, before formatting
> this hard drive I would like to know if it is still
> advisable to partition big hard drives into smaller
> logical ones.
>
> For about 20 last years, following an advice of my older
> colleague, I always partitioned all my hard drives into
> the smaller logical ones and do very well know all
> disadvantages of doing so. :)
>
> But what are disadvantages of not partitioning a big
> hard drive into smaller logical ones?
>
> Is it still advisable to partition a big hard drive
> into smaller logical ones and why?
>
>
I use 2TB USB drive with one EXT4 partition. Took about 30 seconds to 
format connected to a USB2 port. Testing the drive with dd and copying 
files to the drive is very slow. Don't touch "Green Drives". They die 
like flies.

Re: [gentoo-user] Re: [OT] Is it still advisable to partition a big hard drive?

[Mick]

[-- Attachment #1: Type: text/plain, Size: 1700 bytes --]

On Monday 05 Sep 2016 10:42:34 Hans wrote:
> On 01/09/16 16:04, gevisz wrote:
> > I have bought an external 5TB Western Digital hard drive
> > that I am going to use mainly for backing up some files
> > in my home directory and carrying a very big files, for
> > example a virtual machine image file, from one computer
> > to another. This hard drive is preformatted with NTFS.
> > Now, I am going to format it with ext4 which probably
> > will take a lot of time taking into account that it is
> > going to be done via USB connection. So, before formatting
> > this hard drive I would like to know if it is still
> > advisable to partition big hard drives into smaller
> > logical ones.
> > 
> > For about 20 last years, following an advice of my older
> > colleague, I always partitioned all my hard drives into
> > the smaller logical ones and do very well know all
> > disadvantages of doing so. :)
> > 
> > But what are disadvantages of not partitioning a big
> > hard drive into smaller logical ones?
> > 
> > Is it still advisable to partition a big hard drive
> > into smaller logical ones and why?
> 
> I use 2TB USB drive with one EXT4 partition. Took about 30 seconds to
> format connected to a USB2 port. Testing the drive with dd and copying
> files to the drive is very slow. Don't touch "Green Drives". They die
> like flies.

Did you get the logical and physical sector aligned when you partitioned them? 
(if not sure, google for 4k sector drives).  All recent versions of 
fdisk/gdisk/parted and friends will align them by default.

How did you test it with dd and how are you copying files?

How slow is slow in this case?

-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [gentoo-user] Re: [OT] Is it still advisable to partition a big hard drive?

[gevisz]

2016-09-05 3:42 GMT+03:00 Hans <linux@c5ace.com>:
> On 01/09/16 16:04, gevisz wrote:
>>
>> I have bought an external 5TB Western Digital hard drive
>> that I am going to use mainly for backing up some files
>> in my home directory and carrying a very big files, for
>> example a virtual machine image file, from one computer
>> to another. This hard drive is preformatted with NTFS.
>> Now, I am going to format it with ext4 which probably
>> will take a lot of time taking into account that it is
>> going to be done via USB connection. So, before formatting
>> this hard drive I would like to know if it is still
>> advisable to partition big hard drives into smaller
>> logical ones.
>>
>> For about 20 last years, following an advice of my older
>> colleague, I always partitioned all my hard drives into
>> the smaller logical ones and do very well know all
>> disadvantages of doing so. :)
>>
>> But what are disadvantages of not partitioning a big
>> hard drive into smaller logical ones?
>>
>> Is it still advisable to partition a big hard drive
>> into smaller logical ones and why?
>>
>>
> I use 2TB USB drive with one EXT4 partition.
> Took about 30 seconds to format connected to a USB2 port.

May be. But I have just finished testing its first 41 GB with
# badblocks -sw -b4096

Three passes with different write patterns took about 2 hours 10 minutes,
that is about 33 minutes per pass.

So, the full one-pass write test with badblocks should take about 3 days,
if I do not err in my calculations.

The same amount of time should take formatting it with
# mke2fs -cc
But I have not tried that so far.

> Testing the drive with dd and copying files to the
> drive is very slow. Don't touch "Green Drives".
> They die like flies.


What do you mean by this?

My WDC WD15EADS (it is Green) already works (hosting my /home)
for more than 10 years and the systems reports that it is still ok.
(I work at this computer from 6 to 8 hours daily.)

[gentoo-user] Re: [OT] Is it still advisable to partition a big hard drive?

[Hans]

On 05/09/16 17:22, gevisz wrote:
> 2016-09-05 3:42 GMT+03:00 Hans <linux@c5ace.com>:
>> On 01/09/16 16:04, gevisz wrote:
>>>
>>> I have bought an external 5TB Western Digital hard drive
>>> that I am going to use mainly for backing up some files
>>> in my home directory and carrying a very big files, for
>>> example a virtual machine image file, from one computer
>>> to another. This hard drive is preformatted with NTFS.
>>> Now, I am going to format it with ext4 which probably
>>> will take a lot of time taking into account that it is
>>> going to be done via USB connection. So, before formatting
>>> this hard drive I would like to know if it is still
>>> advisable to partition big hard drives into smaller
>>> logical ones.
>>>
>>> For about 20 last years, following an advice of my older
>>> colleague, I always partitioned all my hard drives into
>>> the smaller logical ones and do very well know all
>>> disadvantages of doing so. :)
>>>
>>> But what are disadvantages of not partitioning a big
>>> hard drive into smaller logical ones?
>>>
>>> Is it still advisable to partition a big hard drive
>>> into smaller logical ones and why?
>>>
>>>
>> I use 2TB USB drive with one EXT4 partition.
>> Took about 30 seconds to format connected to a USB2 port.
>
> May be. But I have just finished testing its first 41 GB with
> # badblocks -sw -b4096
>
> Three passes with different write patterns took about 2 hours 10 minutes,
> that is about 33 minutes per pass.
>
> So, the full one-pass write test with badblocks should take about 3 days,
> if I do not err in my calculations.
>
> The same amount of time should take formatting it with
> # mke2fs -cc
> But I have not tried that so far.
>
>> Testing the drive with dd and copying files to the
>> drive is very slow. Don't touch "Green Drives".
>> They die like flies.
>
>
> What do you mean by this?
>
> My WDC WD15EADS (it is Green) already works (hosting my /home)
> for more than 10 years and the systems reports that it is still ok.
> (I work at this computer from 6 to 8 hours daily.)
>
>
I look after 4 laptops. They came with "Green Drives". Had on all 4 
multiple drive failures during the first year. They could not stand 24/7 
use in mining equipment. Replaced them with HGST drives. No failure 
since 18 month.

[gentoo-user] Re: [OT] Is it still advisable to partition a big hard drive?

[Hans]

On 05/09/16 15:31, Mick wrote:
> On Monday 05 Sep 2016 10:42:34 Hans wrote:
>> On 01/09/16 16:04, gevisz wrote:
>>> I have bought an external 5TB Western Digital hard drive
>>> that I am going to use mainly for backing up some files
>>> in my home directory and carrying a very big files, for
>>> example a virtual machine image file, from one computer
>>> to another. This hard drive is preformatted with NTFS.
>>> Now, I am going to format it with ext4 which probably
>>> will take a lot of time taking into account that it is
>>> going to be done via USB connection. So, before formatting
>>> this hard drive I would like to know if it is still
>>> advisable to partition big hard drives into smaller
>>> logical ones.
>>>
>>> For about 20 last years, following an advice of my older
>>> colleague, I always partitioned all my hard drives into
>>> the smaller logical ones and do very well know all
>>> disadvantages of doing so. :)
>>>
>>> But what are disadvantages of not partitioning a big
>>> hard drive into smaller logical ones?
>>>
>>> Is it still advisable to partition a big hard drive
>>> into smaller logical ones and why?
>>
>> I use 2TB USB drive with one EXT4 partition. Took about 30 seconds to
>> format connected to a USB2 port. Testing the drive with dd and copying
>> files to the drive is very slow. Don't touch "Green Drives". They die
>> like flies.
>
> Did you get the logical and physical sector aligned when you partitioned them?
> (if not sure, google for 4k sector drives).  All recent versions of
> fdisk/gdisk/parted and friends will align them by default.
>
> How did you test it with dd and how are you copying files?
>
> How slow is slow in this case?
>
Can't remember. Much slower than copying 1TB Video files from and to 
SATA disks.

Re: [gentoo-user] Re: [OT] Is it still advisable to partition a big hard drive?

[Mick]

[-- Attachment #1: Type: text/plain, Size: 2231 bytes --]

On Monday 05 Sep 2016 22:56:33 Hans wrote:
> On 05/09/16 15:31, Mick wrote:
> > On Monday 05 Sep 2016 10:42:34 Hans wrote:
> >> On 01/09/16 16:04, gevisz wrote:
> >>> I have bought an external 5TB Western Digital hard drive
> >>> that I am going to use mainly for backing up some files
> >>> in my home directory and carrying a very big files, for
> >>> example a virtual machine image file, from one computer
> >>> to another. This hard drive is preformatted with NTFS.
> >>> Now, I am going to format it with ext4 which probably
> >>> will take a lot of time taking into account that it is
> >>> going to be done via USB connection. So, before formatting
> >>> this hard drive I would like to know if it is still
> >>> advisable to partition big hard drives into smaller
> >>> logical ones.
> >>> 
> >>> For about 20 last years, following an advice of my older
> >>> colleague, I always partitioned all my hard drives into
> >>> the smaller logical ones and do very well know all
> >>> disadvantages of doing so. :)
> >>> 
> >>> But what are disadvantages of not partitioning a big
> >>> hard drive into smaller logical ones?
> >>> 
> >>> Is it still advisable to partition a big hard drive
> >>> into smaller logical ones and why?
> >> 
> >> I use 2TB USB drive with one EXT4 partition. Took about 30 seconds to
> >> format connected to a USB2 port. Testing the drive with dd and copying
> >> files to the drive is very slow. Don't touch "Green Drives". They die
> >> like flies.
> > 
> > Did you get the logical and physical sector aligned when you partitioned
> > them? (if not sure, google for 4k sector drives).  All recent versions of
> > fdisk/gdisk/parted and friends will align them by default.
> > 
> > How did you test it with dd and how are you copying files?
> > 
> > How slow is slow in this case?
> 
> Can't remember. Much slower than copying 1TB Video files from and to
> SATA disks.

Ah, yes, it would be so because you would be copying over a USB 2.0, port 
which can read at around 30MB/s and write at about half that.  Had you used an 
e-sata interface instead of USB 2.0, the same drive would perform comparably 
to SATA disks/controllers.

-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Volker Armin Hemmann]

Am 01.09.2016 um 11:01 schrieb Alan McKinnon:
> On 01/09/2016 09:18, gevisz wrote:
>> 2016-09-01 9:13 GMT+03:00 Alan McKinnon <alan.mckinnon@gmail.com>:
>>> On 01/09/2016 08:04, gevisz wrote:
> [snip]
>
>>> it will take about 5 seconds to partition it.
>>> And a few more to mkfs it.
>> Just to partition - may be, but I very much doubt
>> that it will take seconds to create a full-fledged
>> ext4 file system on these 5TB via USB2 connention.
>
> Do it. Tell me how long it tool.
>
> Discussing it without doing it and offering someone else's opinion is a
> 100% worthless activity
>
>> Even more: my aquiantance from the Window world
>> that recomended me this disc scared me that it may
>> take days...
> Mickey Mouse told me it takes microseconds. So what?
>
> Do it. Tell me how long it took.
>
>>>> Is it still advisable to partition a big hard drive
>>>> into smaller logical ones and why?
>>> The only reason to partition a drive is to get 2 or more
>>> smaller ones that differ somehow (size, inode ratio, mount options, etc)
>>>
>>> Go with no partition table by all means, but if you one day find you
>>> need one, you will have to copy all your data off, repartition, and copy
>>> your data back. If you are certain that will not happen (eg you will
>>> rather buy a second drive) then by all means dispense with partitions.
>>>
>>> They are after all nothing more than a Microsoft invention from the 80s
>>> so people could install UCSD Pascal next to MS-DOS
>> I definitely will not need more than one mount point for this hard drive
>> but I do remember some arguments that partitioning a large hard drive
>> into smaller logical ones gives me more safety in case a file system
>> suddenly will get corrupted because in this case I will loose my data
>> only on one of the logical partitions and not on the whole drive.
>>
>> Is this argument still valid nowadays?
> That is the most stupid dumbass argument I've heard in weeks.
> It doesn't even deserve a response.
>
> Who the fuck is promoting this shit?
>
>
people who had to deal with corrupted filesystems in the past?

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Alan McKinnon]

On 07/09/2016 01:57, Volker Armin Hemmann wrote:
> Am 01.09.2016 um 11:01 schrieb Alan McKinnon:
>> On 01/09/2016 09:18, gevisz wrote:
>>> 2016-09-01 9:13 GMT+03:00 Alan McKinnon <alan.mckinnon@gmail.com>:
>>>> On 01/09/2016 08:04, gevisz wrote:
>> [snip]
>>
>>>> it will take about 5 seconds to partition it.
>>>> And a few more to mkfs it.
>>> Just to partition - may be, but I very much doubt
>>> that it will take seconds to create a full-fledged
>>> ext4 file system on these 5TB via USB2 connention.
>>
>> Do it. Tell me how long it tool.
>>
>> Discussing it without doing it and offering someone else's opinion is a
>> 100% worthless activity
>>
>>> Even more: my aquiantance from the Window world
>>> that recomended me this disc scared me that it may
>>> take days...
>> Mickey Mouse told me it takes microseconds. So what?
>>
>> Do it. Tell me how long it took.
>>
>>>>> Is it still advisable to partition a big hard drive
>>>>> into smaller logical ones and why?
>>>> The only reason to partition a drive is to get 2 or more
>>>> smaller ones that differ somehow (size, inode ratio, mount options, etc)
>>>>
>>>> Go with no partition table by all means, but if you one day find you
>>>> need one, you will have to copy all your data off, repartition, and copy
>>>> your data back. If you are certain that will not happen (eg you will
>>>> rather buy a second drive) then by all means dispense with partitions.
>>>>
>>>> They are after all nothing more than a Microsoft invention from the 80s
>>>> so people could install UCSD Pascal next to MS-DOS
>>> I definitely will not need more than one mount point for this hard drive
>>> but I do remember some arguments that partitioning a large hard drive
>>> into smaller logical ones gives me more safety in case a file system
>>> suddenly will get corrupted because in this case I will loose my data
>>> only on one of the logical partitions and not on the whole drive.
>>>
>>> Is this argument still valid nowadays?
>> That is the most stupid dumbass argument I've heard in weeks.
>> It doesn't even deserve a response.
>>
>> Who the fuck is promoting this shit?
>>
>>
> people who had to deal with corrupted filesystems in the past?
> 
> 

The way to deal with the problem of fs corruption is to have reliable
tested backups.

The wrong way to deal with the problem of fs corruption is to get into
cargo-cult manoeuvrers thinking that lots of little bits making a whole
is going to solve the problem.

Especially when the part of the disk statistically most at risk is the
valuable data itself. OS code can be rebuilt easily, without backups
data can't.

-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Volker Armin Hemmann]

Am 07.09.2016 um 08:18 schrieb Alan McKinnon:
> On 07/09/2016 01:57, Volker Armin Hemmann wrote:
>> Am 01.09.2016 um 11:01 schrieb Alan McKinnon:
>>> On 01/09/2016 09:18, gevisz wrote:
>>>> 2016-09-01 9:13 GMT+03:00 Alan McKinnon <alan.mckinnon@gmail.com>:
>>>>> On 01/09/2016 08:04, gevisz wrote:
>>> [snip]
>>>
>>>>> it will take about 5 seconds to partition it.
>>>>> And a few more to mkfs it.
>>>> Just to partition - may be, but I very much doubt
>>>> that it will take seconds to create a full-fledged
>>>> ext4 file system on these 5TB via USB2 connention.
>>> Do it. Tell me how long it tool.
>>>
>>> Discussing it without doing it and offering someone else's opinion is a
>>> 100% worthless activity
>>>
>>>> Even more: my aquiantance from the Window world
>>>> that recomended me this disc scared me that it may
>>>> take days...
>>> Mickey Mouse told me it takes microseconds. So what?
>>>
>>> Do it. Tell me how long it took.
>>>
>>>>>> Is it still advisable to partition a big hard drive
>>>>>> into smaller logical ones and why?
>>>>> The only reason to partition a drive is to get 2 or more
>>>>> smaller ones that differ somehow (size, inode ratio, mount options, etc)
>>>>>
>>>>> Go with no partition table by all means, but if you one day find you
>>>>> need one, you will have to copy all your data off, repartition, and copy
>>>>> your data back. If you are certain that will not happen (eg you will
>>>>> rather buy a second drive) then by all means dispense with partitions.
>>>>>
>>>>> They are after all nothing more than a Microsoft invention from the 80s
>>>>> so people could install UCSD Pascal next to MS-DOS
>>>> I definitely will not need more than one mount point for this hard drive
>>>> but I do remember some arguments that partitioning a large hard drive
>>>> into smaller logical ones gives me more safety in case a file system
>>>> suddenly will get corrupted because in this case I will loose my data
>>>> only on one of the logical partitions and not on the whole drive.
>>>>
>>>> Is this argument still valid nowadays?
>>> That is the most stupid dumbass argument I've heard in weeks.
>>> It doesn't even deserve a response.
>>>
>>> Who the fuck is promoting this shit?
>>>
>>>
>> people who had to deal with corrupted filesystems in the past?
>>
>>
> The way to deal with the problem of fs corruption is to have reliable
> tested backups.
>
> The wrong way to deal with the problem of fs corruption is to get into
> cargo-cult manoeuvrers thinking that lots of little bits making a whole
> is going to solve the problem.
>
> Especially when the part of the disk statistically most at risk is the
> valuable data itself. OS code can be rebuilt easily, without backups
> data can't.
>

the bigger the drive, the greater the chance of fs corruption. Just by
statistics. Better one minor partition is lost than everything.

You can disagree as much as you like, but with the size of drives and
the current error rate of consumer hard drives it is not a question of
'if' but just a matter of 'when'.

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Alan McKinnon]

On 08/09/2016 00:12, Volker Armin Hemmann wrote:
> Am 07.09.2016 um 08:18 schrieb Alan McKinnon:
>> On 07/09/2016 01:57, Volker Armin Hemmann wrote:
>>> Am 01.09.2016 um 11:01 schrieb Alan McKinnon:
>>>> On 01/09/2016 09:18, gevisz wrote:
>>>>> 2016-09-01 9:13 GMT+03:00 Alan McKinnon <alan.mckinnon@gmail.com>:
>>>>>> On 01/09/2016 08:04, gevisz wrote:
>>>> [snip]
>>>>
>>>>>> it will take about 5 seconds to partition it.
>>>>>> And a few more to mkfs it.
>>>>> Just to partition - may be, but I very much doubt
>>>>> that it will take seconds to create a full-fledged
>>>>> ext4 file system on these 5TB via USB2 connention.
>>>> Do it. Tell me how long it tool.
>>>>
>>>> Discussing it without doing it and offering someone else's opinion is a
>>>> 100% worthless activity
>>>>
>>>>> Even more: my aquiantance from the Window world
>>>>> that recomended me this disc scared me that it may
>>>>> take days...
>>>> Mickey Mouse told me it takes microseconds. So what?
>>>>
>>>> Do it. Tell me how long it took.
>>>>
>>>>>>> Is it still advisable to partition a big hard drive
>>>>>>> into smaller logical ones and why?
>>>>>> The only reason to partition a drive is to get 2 or more
>>>>>> smaller ones that differ somehow (size, inode ratio, mount options, etc)
>>>>>>
>>>>>> Go with no partition table by all means, but if you one day find you
>>>>>> need one, you will have to copy all your data off, repartition, and copy
>>>>>> your data back. If you are certain that will not happen (eg you will
>>>>>> rather buy a second drive) then by all means dispense with partitions.
>>>>>>
>>>>>> They are after all nothing more than a Microsoft invention from the 80s
>>>>>> so people could install UCSD Pascal next to MS-DOS
>>>>> I definitely will not need more than one mount point for this hard drive
>>>>> but I do remember some arguments that partitioning a large hard drive
>>>>> into smaller logical ones gives me more safety in case a file system
>>>>> suddenly will get corrupted because in this case I will loose my data
>>>>> only on one of the logical partitions and not on the whole drive.
>>>>>
>>>>> Is this argument still valid nowadays?
>>>> That is the most stupid dumbass argument I've heard in weeks.
>>>> It doesn't even deserve a response.
>>>>
>>>> Who the fuck is promoting this shit?
>>>>
>>>>
>>> people who had to deal with corrupted filesystems in the past?
>>>
>>>
>> The way to deal with the problem of fs corruption is to have reliable
>> tested backups.
>>
>> The wrong way to deal with the problem of fs corruption is to get into
>> cargo-cult manoeuvrers thinking that lots of little bits making a whole
>> is going to solve the problem.
>>
>> Especially when the part of the disk statistically most at risk is the
>> valuable data itself. OS code can be rebuilt easily, without backups
>> data can't.
>>
>
> the bigger the drive, the greater the chance of fs corruption. Just by
> statistics. Better one minor partition is lost than everything.

What are the statistical chances of that one minor partition being the 
one that gets corrupted? Statistically the odds are very small.

Think about it, if the minor partition is say 5% of the disk and if all 
other things are exactly equal, the odds are 1 in 20.

Apart from inherent defects in the drive itself, the sectors that are 
more prone to failing are those that are read the most and to a larger 
extent those that are written the most.

What is read the most? OS and Data
What is written the most? Data
What has by far the greatest likelihood of suffering fs corruption? Data



>
> You can disagree as much as you like, but with the size of drives and
> the current error rate of consumer hard drives it is not a question of
> 'if' but just a matter of 'when'.
>

I don't disagree with you. I'm disagreeing with cargo cult mentality 
that dividing a disk up into lots of smaller partitions somehow 
magically confers significant safety margins of some magical kind. Go 
read the OPs opening statement again, he's quoting a friend from 20 
years ago and the statement consists entirely of woo-woo magic 
hand-wavey statements, the kind of shit I have to deal with every day 
from twits with just enough IQ to read executive white papers.

Yes, drives fail. Yes, consumer drives are crap. With 3TB now being 
common place and prices plunging, we have 20G or so for OS and 2980GB 
full of data. That 20G is so small and immaterial in terms of risk we 
can just disregard it and assume the only thing that can be damaged is 
2980G of data.

Solution: back up the whole damn lot properly and forget what we did 20 
years ago. That was farting in a breeze, nowadays it's farting in a 
hurricane.

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[waltdnes]

On Thu, Sep 08, 2016 at 12:12:07AM +0200, Volker Armin Hemmann wrote

> You can disagree as much as you like, but with the size of drives and
> the current error rate of consumer hard drives it is not a question of
> 'if' but just a matter of 'when'.

  It's not just the drive; it's the entire PC.  My main desktop at home
has had a few panics recently.  It's several years old, and I'll be
getting rid of it, because I can't really trust it.  I've switched to my
"hot backup", and am currently setting up a new machine as the new "hot
backup".  After doing the initial Gentoo install, I copied over the
config files, with appropriate changes.  I copied /var/lib/portage/world
and launched "emerge --changed-use --deep --update @world".  A few
minutes ago, emerge was on package 228 of 337.

  I have 3 USB backup drives and I use them all.  This does not include
the monthly copying over of /home/waltdnes and /home/misc from the main
desktop to the "hot backup".

  BTW, this is probably the first email sent out from this machine to
the Gentoo list.

-- 
Walter Dnes <waltdnes@waltdnes.org>
I don't run "desktop environments"; I run useful applications

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Volker Armin Hemmann]

Am 08.09.2016 um 00:47 schrieb Alan McKinnon:
> On 08/09/2016 00:12, Volker Armin Hemmann wrote:
>> Am 07.09.2016 um 08:18 schrieb Alan McKinnon:
>>> On 07/09/2016 01:57, Volker Armin Hemmann wrote:
>>>> Am 01.09.2016 um 11:01 schrieb Alan McKinnon:
>>>>> On 01/09/2016 09:18, gevisz wrote:
>>>>>> 2016-09-01 9:13 GMT+03:00 Alan McKinnon <alan.mckinnon@gmail.com>:
>>>>>>> On 01/09/2016 08:04, gevisz wrote:
>>>>> [snip]
>>>>>
>>>>>>> it will take about 5 seconds to partition it.
>>>>>>> And a few more to mkfs it.
>>>>>> Just to partition - may be, but I very much doubt
>>>>>> that it will take seconds to create a full-fledged
>>>>>> ext4 file system on these 5TB via USB2 connention.
>>>>> Do it. Tell me how long it tool.
>>>>>
>>>>> Discussing it without doing it and offering someone else's opinion
>>>>> is a
>>>>> 100% worthless activity
>>>>>
>>>>>> Even more: my aquiantance from the Window world
>>>>>> that recomended me this disc scared me that it may
>>>>>> take days...
>>>>> Mickey Mouse told me it takes microseconds. So what?
>>>>>
>>>>> Do it. Tell me how long it took.
>>>>>
>>>>>>>> Is it still advisable to partition a big hard drive
>>>>>>>> into smaller logical ones and why?
>>>>>>> The only reason to partition a drive is to get 2 or more
>>>>>>> smaller ones that differ somehow (size, inode ratio, mount
>>>>>>> options, etc)
>>>>>>>
>>>>>>> Go with no partition table by all means, but if you one day find
>>>>>>> you
>>>>>>> need one, you will have to copy all your data off, repartition,
>>>>>>> and copy
>>>>>>> your data back. If you are certain that will not happen (eg you
>>>>>>> will
>>>>>>> rather buy a second drive) then by all means dispense with
>>>>>>> partitions.
>>>>>>>
>>>>>>> They are after all nothing more than a Microsoft invention from
>>>>>>> the 80s
>>>>>>> so people could install UCSD Pascal next to MS-DOS
>>>>>> I definitely will not need more than one mount point for this
>>>>>> hard drive
>>>>>> but I do remember some arguments that partitioning a large hard
>>>>>> drive
>>>>>> into smaller logical ones gives me more safety in case a file system
>>>>>> suddenly will get corrupted because in this case I will loose my
>>>>>> data
>>>>>> only on one of the logical partitions and not on the whole drive.
>>>>>>
>>>>>> Is this argument still valid nowadays?
>>>>> That is the most stupid dumbass argument I've heard in weeks.
>>>>> It doesn't even deserve a response.
>>>>>
>>>>> Who the fuck is promoting this shit?
>>>>>
>>>>>
>>>> people who had to deal with corrupted filesystems in the past?
>>>>
>>>>
>>> The way to deal with the problem of fs corruption is to have reliable
>>> tested backups.
>>>
>>> The wrong way to deal with the problem of fs corruption is to get into
>>> cargo-cult manoeuvrers thinking that lots of little bits making a whole
>>> is going to solve the problem.
>>>
>>> Especially when the part of the disk statistically most at risk is the
>>> valuable data itself. OS code can be rebuilt easily, without backups
>>> data can't.
>>>
>>
>> the bigger the drive, the greater the chance of fs corruption. Just by
>> statistics. Better one minor partition is lost than everything.
>
> What are the statistical chances of that one minor partition being the
> one that gets corrupted? Statistically the odds are very small.
>
> Think about it, if the minor partition is say 5% of the disk and if
> all other things are exactly equal, the odds are 1 in 20.
>
> Apart from inherent defects in the drive itself, the sectors that are
> more prone to failing are those that are read the most and to a larger
> extent those that are written the most.
>
> What is read the most? OS and Data
> What is written the most? Data
> What has by far the greatest likelihood of suffering fs corruption? Data

and that is why spreading data over several partitions is not a bad idea.

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Andrew Lowe]

On 01/09/16 17:01, Alan McKinnon wrote:
> On 01/09/2016 09:18, gevisz wrote:
>> 2016-09-01 9:13 GMT+03:00 Alan McKinnon <alan.mckinnon@gmail.com>:
>>> On 01/09/2016 08:04, gevisz wrote:
>
> [snip]
>
[snip]
...
...
[snip]
>
> That is the most stupid dumbass argument I've heard in weeks.
> It doesn't even deserve a response.
>
> Who the fuck is promoting this shit?
>
>

	I've just spent 7 hours in a bottleshop in an entertainment area, 
putting up with idiots swearing all night. Now I come home and catch up 
on what's happening on this list and what do I get? More drop kicks 
swearing their heads off. It's just an indication of the lack of your 
grasp of the English language when you start carrying on like this.

	Don't get me wrong, I can swear with the best of them, I spent my late 
teens working as a barman in a pub opposite one of the hardest gaols you 
will find anywhere. There is a time and a place for everything, a public 
technical email list is not one of them for swearing.

	How about you just pull your head in until you learn some "big boy" 
words and can contribute in a civil manner?

	Just my 5c worth,
		Andrew

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Peter Humphrey]

On Friday 09 Sep 2016 23:25:10 Andrew Lowe wrote:

> 	I've just spent 7 hours in a bottleshop in an entertainment area,
> putting up with idiots swearing all night. Now I come home and catch up
> on what's happening on this list and what do I get? More drop kicks
> swearing their heads off. It's just an indication of the lack of your
> grasp of the English language when you start carrying on like this.
> 
> ... There is a time and a place for everything, a public technical email
> list is not one of them for swearing.

A man after my own heart.

When a new couple took over my local pub last year, they split the village* 
beer drinkers down the middle by enforcing a no-swearing policy. Now all 
those men who can't say anything without an f*** every other word go to the 
other main pub, leaving the one I use to those who can. Trade has soared, 
and local people have been appearing who were never seen in a pub before. 
Winners all round.

* Tideswell, Derbyshire, in case you're interested.

-- 
Rgds
Peter