Re: [gentoo-user] Fast file system for cache directory with lot's of files

[Michael Hampicke <mgehampicke@gmail.com>]

Am 13.08.2012 19:14, schrieb Florian Philipp:
> Am 13.08.2012 16:52, schrieb Michael Mol:
>> On Mon, Aug 13, 2012 at 10:42 AM, Michael Hampicke
>> <mgehampicke@gmail.com <mailto:mgehampicke@gmail.com>> wrote:
>>
>>         Have you indexed your ext4 partition?
>>
>>         # tune2fs -O dir_index /dev/your_partition
>>         # e2fsck -D /dev/your_partition
>>
>>     Hi, the dir_index is active. I guess that's why delete operations
>>     take as long as they take (index has to be updated every time) 
>>
>>
>> 1) Scan for files to remove
>> 2) disable index
>> 3) Remove files
>> 4) enable index
>>
>> ?
>>
>> -- 
>> :wq
> 
> Other things to think about:
> 
> 1. Play around with data=journal/writeback/ordered. IIRC, data=journal
> actually used to improve performance depending on the workload as it
> delays random IO in favor of sequential IO (when updating the journal).
> 
> 2. Increase the journal size.
> 
> 3. Take a look at `man 1 chattr`. Especially the 'T' attribute. Of
> course this only helps after re-allocating everything.
> 
> 4. Try parallelizing. Ext4 requires relatively few locks nowadays (since
> 2.6.39 IIRC). For example:
> find $TOP_DIR -mindepth 1 -maxdepth 1 -print0 | \
> xargs -0 -n 1 -r -P 4 -I '{}' find '{}' -type f
> 
> 5. Use a separate device for the journal.
> 
> 6. Temporarily deactivate the journal with tune2fs similar to MM's idea.
> 
> Regards,
> Florian Philipp
> 

Trying out different journals-/options was already on my list, but the
manpage on chattr regarding the T attribute is an interesting read.
Definitely worth trying.

Parallelizing multiple finds was something I already did, but the only
thing that increased was the IO wait :) But now having read all the
suggestions in this thread, I might try it again.

Separate device for the journal is a good idea, but not possible atm
(machine is abroad in a data center)