[gentoo-user] {OT} zflashpoint for Linux? (SSD performance "accelerator")

[gentoo-user] {OT} zflashpoint for Linux? (SSD performance "accelerator")

[Grant]

Here is some info on zflashpoint:

http://forum.notebookreview.com/showthread.php?p=5163549

It is supposed to be an SSD performance "accelerator".

"Gen1 SSDs suffer with small file writes and I read that 90% of
Windows writes are small, so that introduces overall system lag.
zflashpoint uses a 32MB buffer for small file writes, combining them
and dumping them as one big write."

Has anyone heard of development of a similar project for Linux?

- Grant

Re: [gentoo-user] {OT} zflashpoint for Linux? (SSD performance "accelerator")

[Paul Hartman]

On Thu, Aug 6, 2009 at 9:45 AM, Grant<emailgrant@gmail.com> wrote:
> Here is some info on zflashpoint:
>
> http://forum.notebookreview.com/showthread.php?p=5163549
>
> It is supposed to be an SSD performance "accelerator".
>
> "Gen1 SSDs suffer with small file writes and I read that 90% of
> Windows writes are small, so that introduces overall system lag.
> zflashpoint uses a 32MB buffer for small file writes, combining them
> and dumping them as one big write."
>
> Has anyone heard of development of a similar project for Linux?

Sounds like a disk cache to me... maybe i'm missing something.

Re: [gentoo-user] {OT} zflashpoint for Linux? (SSD performance "accelerator")

[Grant]

>> Here is some info on zflashpoint:
>>
>> http://forum.notebookreview.com/showthread.php?p=5163549
>>
>> It is supposed to be an SSD performance "accelerator".
>>
>> "Gen1 SSDs suffer with small file writes and I read that 90% of
>> Windows writes are small, so that introduces overall system lag.
>> zflashpoint uses a 32MB buffer for small file writes, combining them
>> and dumping them as one big write."
>>
>> Has anyone heard of development of a similar project for Linux?
>
> Sounds like a disk cache to me... maybe i'm missing something.

Maybe it's just making up for the too-small disc cache on the
stuttering SSDs?  The 64MB cache on the non-stuttering ones is
supposed to have something to do with it.  Can this sort of a
"software disk cache" be set up on Linux?

- Grant

Re: [gentoo-user] {OT} zflashpoint for Linux? (SSD performance "accelerator")

[Paul Hartman]

On Thu, Aug 6, 2009 at 11:37 AM, Grant<emailgrant@gmail.com> wrote:
>>> Here is some info on zflashpoint:
>>>
>>> http://forum.notebookreview.com/showthread.php?p=5163549
>>>
>>> It is supposed to be an SSD performance "accelerator".
>>>
>>> "Gen1 SSDs suffer with small file writes and I read that 90% of
>>> Windows writes are small, so that introduces overall system lag.
>>> zflashpoint uses a 32MB buffer for small file writes, combining them
>>> and dumping them as one big write."
>>>
>>> Has anyone heard of development of a similar project for Linux?
>>
>> Sounds like a disk cache to me... maybe i'm missing something.
>
> Maybe it's just making up for the too-small disc cache on the
> stuttering SSDs?  The 64MB cache on the non-stuttering ones is
> supposed to have something to do with it.  Can this sort of a
> "software disk cache" be set up on Linux?

I always just assumed Linux used a cache with SSD just like it does
with everything else but I have no experience or evidence to prove
that. I know windows doesn't cache writes to USB flash drives, maybe
it doesn't cache SSD either.

Re: [gentoo-user] {OT} zflashpoint for Linux? (SSD performance "accelerator")

[Grant]

>>>> Here is some info on zflashpoint:
>>>>
>>>> http://forum.notebookreview.com/showthread.php?p=5163549
>>>>
>>>> It is supposed to be an SSD performance "accelerator".
>>>>
>>>> "Gen1 SSDs suffer with small file writes and I read that 90% of
>>>> Windows writes are small, so that introduces overall system lag.
>>>> zflashpoint uses a 32MB buffer for small file writes, combining them
>>>> and dumping them as one big write."
>>>>
>>>> Has anyone heard of development of a similar project for Linux?
>>>
>>> Sounds like a disk cache to me... maybe i'm missing something.
>>
>> Maybe it's just making up for the too-small disc cache on the
>> stuttering SSDs?  The 64MB cache on the non-stuttering ones is
>> supposed to have something to do with it.  Can this sort of a
>> "software disk cache" be set up on Linux?
>
> I always just assumed Linux used a cache with SSD just like it does
> with everything else but I have no experience or evidence to prove
> that. I know windows doesn't cache writes to USB flash drives, maybe
> it doesn't cache SSD either.

Alright, but my girlfriend's Gentoo SSD netbook has fairly bad
stuttering problems.

- Grant

Re: [gentoo-user] {OT} zflashpoint for Linux? (SSD performance "accelerator")

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 654 bytes --]

Grant schrieb:
> Here is some info on zflashpoint:
> 
> http://forum.notebookreview.com/showthread.php?p=5163549
> 
> It is supposed to be an SSD performance "accelerator".
> 
> "Gen1 SSDs suffer with small file writes and I read that 90% of
> Windows writes are small, so that introduces overall system lag.
> zflashpoint uses a 32MB buffer for small file writes, combining them
> and dumping them as one big write."
> 
> Has anyone heard of development of a similar project for Linux?
> 
> - Grant
> 

Sounds like laptop-mode to me (app-laptop/laptop-mode-tools). For ext*
file systems, you can also try the 'commit' mount option.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 261 bytes --]

Re: [gentoo-user] {OT} zflashpoint for Linux? (SSD performance "accelerator")

[Grant]

>> Here is some info on zflashpoint:
>>
>> http://forum.notebookreview.com/showthread.php?p=5163549
>>
>> It is supposed to be an SSD performance "accelerator".
>>
>> "Gen1 SSDs suffer with small file writes and I read that 90% of
>> Windows writes are small, so that introduces overall system lag.
>> zflashpoint uses a 32MB buffer for small file writes, combining them
>> and dumping them as one big write."
>>
>> Has anyone heard of development of a similar project for Linux?
>>
>> - Grant
>>
>
> Sounds like laptop-mode to me (app-laptop/laptop-mode-tools). For ext*
> file systems, you can also try the 'commit' mount option.

On her laptop, I get:

# cat /proc/sys/vm/laptop_mode
0

There is lots of talk on the internet about the stuttering problems on
SSD netbooks.  Are you thinking this problem shouldn't manifest itself
on a Linux system?

- Grant

Re: [gentoo-user] {OT} zflashpoint for Linux? (SSD performance "accelerator")

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 2385 bytes --]

Grant schrieb:
>>> Here is some info on zflashpoint:
>>>
>>> http://forum.notebookreview.com/showthread.php?p=5163549
>>>
>>> It is supposed to be an SSD performance "accelerator".
>>>
>>> "Gen1 SSDs suffer with small file writes and I read that 90% of
>>> Windows writes are small, so that introduces overall system lag.
>>> zflashpoint uses a 32MB buffer for small file writes, combining them
>>> and dumping them as one big write."
>>>
>>> Has anyone heard of development of a similar project for Linux?
>>>
>>> - Grant
>>>
>> Sounds like laptop-mode to me (app-laptop/laptop-mode-tools). For ext*
>> file systems, you can also try the 'commit' mount option.
> 
> On her laptop, I get:
> 
> # cat /proc/sys/vm/laptop_mode
> 0
> 
> There is lots of talk on the internet about the stuttering problems on
> SSD netbooks.  Are you thinking this problem shouldn't manifest itself
> on a Linux system?
> 
> - Grant
> 

Well, as I see it, there are two distinct problems here:
1. (Cheap) SSDs are slow
2. (Cheap) SSDs are terribly slow on random write

Both reasons can produce stuttering and you can't completely fix them.
However, you can try to cache random writes until you can commit them as
one sequential write.

There are several things which will help here:

1. A file system that does not scatter data unnecessarily and which
might even align its (meta)data to the erase block boundaries of the SSD.[1]

2. A long commit interval in which writes can be collected until they
are actually performed in optimal order(see my last post).

3. A disk scheduler that optimizes for SSD writes instead of HDD writes
(like Completely Fair Scheduler on kernel >=2.6.29)

4. An enterprise grade RAID controller

5. An SSD with built-in cache and enough "intelligence" to use it properly

I don't see a reason why a Microsoft OS might not make these
optimizations if their devs would care about it and I'd really like to
see a benchmark between a bleeding edge Linux and a Windows 7 on this topic.

Of course, none of these tricks will help with problem 1 and in the end,
when your usage pattern provoke it, you will still see stuttering while
your system struggles to flush its overly large write caches to the much
too slow disk.

[1]
http://thunk.org/tytso/blog/2009/02/20/aligning-filesystems-to-an-ssds-erase-block-size/


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 261 bytes --]

Re: [gentoo-user] {OT} zflashpoint for Linux? (SSD performance "accelerator")

[Grant]

>>>> Here is some info on zflashpoint:
>>>>
>>>> http://forum.notebookreview.com/showthread.php?p=5163549
>>>>
>>>> It is supposed to be an SSD performance "accelerator".
>>>>
>>>> "Gen1 SSDs suffer with small file writes and I read that 90% of
>>>> Windows writes are small, so that introduces overall system lag.
>>>> zflashpoint uses a 32MB buffer for small file writes, combining them
>>>> and dumping them as one big write."
>>>>
>>>> Has anyone heard of development of a similar project for Linux?
>>>>
>>>> - Grant
>>>>
>>> Sounds like laptop-mode to me (app-laptop/laptop-mode-tools). For ext*
>>> file systems, you can also try the 'commit' mount option.
>>
>> On her laptop, I get:
>>
>> # cat /proc/sys/vm/laptop_mode
>> 0
>>
>> There is lots of talk on the internet about the stuttering problems on
>> SSD netbooks.  Are you thinking this problem shouldn't manifest itself
>> on a Linux system?
>>
>> - Grant
>>
>
> Well, as I see it, there are two distinct problems here:
> 1. (Cheap) SSDs are slow
> 2. (Cheap) SSDs are terribly slow on random write
>
> Both reasons can produce stuttering and you can't completely fix them.
> However, you can try to cache random writes until you can commit them as
> one sequential write.
>
> There are several things which will help here:
>
> 1. A file system that does not scatter data unnecessarily and which
> might even align its (meta)data to the erase block boundaries of the SSD.[1]
>
> 2. A long commit interval in which writes can be collected until they
> are actually performed in optimal order(see my last post).

OK, I'm sorry, I thought you were telling me before she must be in
laptop-mode.  Now I see that laptop-mode can act as the cache.  Has
anyone actually reported laptop-mode helping ease the stuttering of
slow SSDs?

> 3. A disk scheduler that optimizes for SSD writes instead of HDD writes
> (like Completely Fair Scheduler on kernel >=2.6.29)

She's on CFS.

> 4. An enterprise grade RAID controller
>
> 5. An SSD with built-in cache and enough "intelligence" to use it properly

That seems like the best solution.  As I posted about recently, the
firmware for the type of drive you describe seems to be in flux right
now, and reflashing requires reformatting.  I'd also like to buy an
SLC drive instead of an MLC drive and they're still too expensive.

- Grant


> I don't see a reason why a Microsoft OS might not make these
> optimizations if their devs would care about it and I'd really like to
> see a benchmark between a bleeding edge Linux and a Windows 7 on this topic.
>
> Of course, none of these tricks will help with problem 1 and in the end,
> when your usage pattern provoke it, you will still see stuttering while
> your system struggles to flush its overly large write caches to the much
> too slow disk.
>
> [1]
> http://thunk.org/tytso/blog/2009/02/20/aligning-filesystems-to-an-ssds-erase-block-size/

Re: [gentoo-user] {OT} zflashpoint for Linux? (SSD performance "accelerator")

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 3293 bytes --]

Grant schrieb:
>>>>> Here is some info on zflashpoint:
>>>>>
>>>>> http://forum.notebookreview.com/showthread.php?p=5163549
>>>>>
>>>>> It is supposed to be an SSD performance "accelerator".
>>>>>
>>>>> "Gen1 SSDs suffer with small file writes and I read that 90% of
>>>>> Windows writes are small, so that introduces overall system lag.
>>>>> zflashpoint uses a 32MB buffer for small file writes, combining them
>>>>> and dumping them as one big write."
>>>>>
>>>>> Has anyone heard of development of a similar project for Linux?
>>>>>
>>>>> - Grant
>>>>>
>>>> Sounds like laptop-mode to me (app-laptop/laptop-mode-tools). For ext*
>>>> file systems, you can also try the 'commit' mount option.
>>> On her laptop, I get:
>>>
>>> # cat /proc/sys/vm/laptop_mode
>>> 0
>>>
>>> There is lots of talk on the internet about the stuttering problems on
>>> SSD netbooks.  Are you thinking this problem shouldn't manifest itself
>>> on a Linux system?
>>>
>>> - Grant
>>>
>> Well, as I see it, there are two distinct problems here:
>> 1. (Cheap) SSDs are slow
>> 2. (Cheap) SSDs are terribly slow on random write
>>
>> Both reasons can produce stuttering and you can't completely fix them.
>> However, you can try to cache random writes until you can commit them as
>> one sequential write.
>>
>> There are several things which will help here:
>>
>> 1. A file system that does not scatter data unnecessarily and which
>> might even align its (meta)data to the erase block boundaries of the SSD.[1]
>>
>> 2. A long commit interval in which writes can be collected until they
>> are actually performed in optimal order(see my last post).
> 
> OK, I'm sorry, I thought you were telling me before she must be in
> laptop-mode.  Now I see that laptop-mode can act as the cache.  Has
> anyone actually reported laptop-mode helping ease the stuttering of
> slow SSDs?
> 

No need to apologize.

Some additional thoughts:

When you think about the situation, laptop-mode might actually make the
situation worse. You see, it was originally developed to help HDDs
staying in standby for longer periods by delaying writes until a read
action causes the drive to spin up or some period of time has passed.

At this point, all writes should happen in one short burst. However,
with slow SSDs, these bursts might actually cause the stuttering you
experience. This is especially true when the writes delay a read action.
I'm not sure whether the disk scheduler prefers reads over writes but it
certainly would help.

There is no need to keep an SSD idle as there is no kind of standby like
HDDs have.[1] Therefore I think a better solution would be treating
write actions as batch jobs: You do them only when there is nothing
better to do (i.e. no read action). Until then, you keep them in a large
write cache.

I'm not sure if there is such a system, yet. Maybe you should try out
XFS as it already implements a very aggressive write cache. I'd be very
interested in benchmarks for Ext4 vs. XFS on slow SSDs
but I wouldn't bet on seeing one soon. I suppose simulating and
measuring such a usage pattern isn't a simple task.

[1] Sure, an idle SSD still consumes less power than a busy one but you
can't really compare that to HDDs.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 261 bytes --]

Re: [gentoo-user] {OT} zflashpoint for Linux? (SSD performance "accelerator")

[Grant]

>>>>>> Here is some info on zflashpoint:
>>>>>>
>>>>>> http://forum.notebookreview.com/showthread.php?p=5163549
>>>>>>
>>>>>> It is supposed to be an SSD performance "accelerator".
>>>>>>
>>>>>> "Gen1 SSDs suffer with small file writes and I read that 90% of
>>>>>> Windows writes are small, so that introduces overall system lag.
>>>>>> zflashpoint uses a 32MB buffer for small file writes, combining them
>>>>>> and dumping them as one big write."
>>>>>>
>>>>>> Has anyone heard of development of a similar project for Linux?
>>>>>>
>>>>>> - Grant
>>>>>>
>>>>> Sounds like laptop-mode to me (app-laptop/laptop-mode-tools). For ext*
>>>>> file systems, you can also try the 'commit' mount option.
>>>> On her laptop, I get:
>>>>
>>>> # cat /proc/sys/vm/laptop_mode
>>>> 0
>>>>
>>>> There is lots of talk on the internet about the stuttering problems on
>>>> SSD netbooks.  Are you thinking this problem shouldn't manifest itself
>>>> on a Linux system?
>>>>
>>>> - Grant
>>>>
>>> Well, as I see it, there are two distinct problems here:
>>> 1. (Cheap) SSDs are slow
>>> 2. (Cheap) SSDs are terribly slow on random write
>>>
>>> Both reasons can produce stuttering and you can't completely fix them.
>>> However, you can try to cache random writes until you can commit them as
>>> one sequential write.
>>>
>>> There are several things which will help here:
>>>
>>> 1. A file system that does not scatter data unnecessarily and which
>>> might even align its (meta)data to the erase block boundaries of the SSD.[1]
>>>
>>> 2. A long commit interval in which writes can be collected until they
>>> are actually performed in optimal order(see my last post).
>>
>> OK, I'm sorry, I thought you were telling me before she must be in
>> laptop-mode.  Now I see that laptop-mode can act as the cache.  Has
>> anyone actually reported laptop-mode helping ease the stuttering of
>> slow SSDs?
>>
>
> No need to apologize.
>
> Some additional thoughts:
>
> When you think about the situation, laptop-mode might actually make the
> situation worse. You see, it was originally developed to help HDDs
> staying in standby for longer periods by delaying writes until a read
> action causes the drive to spin up or some period of time has passed.
>
> At this point, all writes should happen in one short burst. However,
> with slow SSDs, these bursts might actually cause the stuttering you
> experience. This is especially true when the writes delay a read action.
> I'm not sure whether the disk scheduler prefers reads over writes but it
> certainly would help.
>
> There is no need to keep an SSD idle as there is no kind of standby like
> HDDs have.[1] Therefore I think a better solution would be treating
> write actions as batch jobs: You do them only when there is nothing
> better to do (i.e. no read action). Until then, you keep them in a large
> write cache.
>
> I'm not sure if there is such a system, yet. Maybe you should try out
> XFS as it already implements a very aggressive write cache. I'd be very
> interested in benchmarks for Ext4 vs. XFS on slow SSDs
> but I wouldn't bet on seeing one soon. I suppose simulating and
> measuring such a usage pattern isn't a simple task.
>
> [1] Sure, an idle SSD still consumes less power than a busy one but you
> can't really compare that to HDDs.

Thank you Florian.

- Grant

Re: [gentoo-user] {OT} zflashpoint for Linux? (SSD performance "accelerator")

[Eray Aslan]

On 09.08.2009 16:13, Florian Philipp wrote:
[..]
> When you think about the situation, laptop-mode might actually make the
> situation worse. You see, it was originally developed to help HDDs
> staying in standby for longer periods by delaying writes until a read
> action causes the drive to spin up or some period of time has passed.
> 
> At this point, all writes should happen in one short burst. However,
> with slow SSDs, these bursts might actually cause the stuttering you
> experience. This is especially true when the writes delay a read action.
> I'm not sure whether the disk scheduler prefers reads over writes but it
> certainly would help.

Reads do get higher priority by default.  They are synchronous afterall.
 Problem usually occurs when reads get interspersed with random writes,
i.e. when start you getting lots of seeks.

But good SSDs don't care.  Only HDDs do.  And maybe bad SSDs, too.

> There is no need to keep an SSD idle as there is no kind of standby like
> HDDs have.[1] Therefore I think a better solution would be treating
> write actions as batch jobs: You do them only when there is nothing
> better to do (i.e. no read action). Until then, you keep them in a large
> write cache.

It is not that easy (it never is?).  There are a lot of trade-offs as
can be witnessed by the variety and complexity of the disk schedulers.

> I'm not sure if there is such a system, yet. Maybe you should try out
> XFS as it already implements a very aggressive write cache. I'd be very
> interested in benchmarks for Ext4 vs. XFS on slow SSDs
> but I wouldn't bet on seeing one soon. I suppose simulating and
> measuring such a usage pattern isn't a simple task.

Well, work with email (email causes a lot of filesystem syncs typically)
while dd'ing a big file repeatedly in the background.  Should be close
enough.  Both latency (stutters) and throughput are important.

-- 
Eray

Re: [gentoo-user] {OT} zflashpoint for Linux? (SSD performance "accelerator")

[Paul Hartman]

Josh Saddler had a couple blog posts recently about his adventures
with SSD and Gentoo:

http://blogs.gentoo.org/nightmorph/2009/08/02/ssds-and-filesystems
http://blogs.gentoo.org/nightmorph/2009/08/09/ssds-and-filesystems-part-2

Re: [gentoo-user] {OT} zflashpoint for Linux? (SSD performance "accelerator")

[Grant]

> Josh Saddler had a couple blog posts recently about his adventures
> with SSD and Gentoo:
>
> http://blogs.gentoo.org/nightmorph/2009/08/02/ssds-and-filesystems
> http://blogs.gentoo.org/nightmorph/2009/08/09/ssds-and-filesystems-part-2

I've been following those (actually posted the first link in another
thread a few days ago).  It sounds like pretty exhaustive research.
The second link has removed ext4 from my "new install" list.

- Grant

Re: [gentoo-user] {OT} zflashpoint for Linux? (SSD performance "accelerator")

[Stroller]

On 11 Aug 2009, at 16:14, Grant wrote:
>> Josh Saddler had a couple blog posts recently about his adventures
>> with SSD and Gentoo:
>>
>> http://blogs.gentoo.org/nightmorph/2009/08/02/ssds-and-filesystems
>> http://blogs.gentoo.org/nightmorph/2009/08/09/ssds-and-filesystems-part-2
>
> I've been following those (actually posted the first link in another
> thread a few days ago).  It sounds like pretty exhaustive research.
> The second link has removed ext4 from my "new install" list.

The posts seemed really anecdotal to me. I mean, I've got ext4 working  
just fine here, so it's not really clear what he's doing differently.  
Is it merely that he's using SSDs & I'm using rotating platters? He's  
going back to ReiserFS, but I've had some bad experiences with that,  
myself.

I'm not saying that he's wrong, but it's one bad experience. It's one  
well-documented bad experience, admittedly, but a description of  
"exhaustive research" (if you'll excuse me saying so) should be  
reserved for tests in which they line up half a dozen systems down a  
test bench, and try all the different file systems on each of them.

I wish there were some definitive way to say which filing system(s)  
are safe & reliable. Mostly you're left to trust either Hans Reiser,  
or Linus & his buddies (for ext[234]) or a filesystem designed for  
Irix (XFS), ported over and no longer maintained by its original  
designers.

I don't mean that in a bad way about XFS, because it works just fine  
on one of my disks here, but choosing a filesystem is inherently a bit  
unscientific and about perspective and trust and intangibles.

It's relatively easy to do performance testing on filesystems -  
although we seem to see relatively few benchmarks published - but of  
course it depends on what kind of data you're storing (random access  
of lots of small files vs. reading through a single large file, for  
instance).

It's much harder to unequivocally state how often different  
filesystems get accidentally corrupted or how bad the consequences  
will be when they do or how well they'll recover from it. If you put  
half a dozen systems on a test bench and pulled the power plugs on all  
of them mid-write, any filesystems boffin would tell you that this  
test failed to consider many other possible circumstances. Should this  
test be performed whilst `mv` is being performed, or `cp`? Different  
file systems will probably behave differently.

So at the end of the day, all we can do is make characterisations  
based on our own experience and bias and - sure - upon anecdotal  
evidence like this.

When I came to Linux, ext2 was the "reliable filesystem" that was  
"designed for Linux", ext3 featured incremental improvements to that  
to add journalling; in those days ReiserFS was relatively new and  
"racey" but "showed promise" with solid computer science behind its  
technology. There is at least one former ReiserCo. employee still  
maintaining & improving ReiserFS, but no longer the same dedicated  
team. Now ext3 is proven - does anyone dispute that? - and ext4 is  
merely "incremental improvements" to its predecessor.

So what are you going to choose? I'm sure I'm not the only person on  
this list who's using ext4 just fine - in fact, I chose it because  
others on this list were already using it. Here deletes of large files  
are a heck of a lot faster on ext4  than on ext3.

Stroller.