[gentoo-user] Filesystem with lowest CPU load, acceptable emerge performance, and stable?

[gentoo-user] Filesystem with lowest CPU load, acceptable emerge performance, and stable?

[Pandu Poluan]

So, can anyone recommend me a filesystem that fulfills my following needs:

Scenario: vFirewall (virtual Firewall) that is going to be deployed at
my IaaS Cloud Provider.

Disk I/O Characteristic: Occasional writes during 'normal' usage,
once-a-week eix-sync + emerge -avuD

Priority: Stable (i.e., less chance of corruption), least CPU usage.

My Google-Fu seems to indicate either XFS or JFS; what do you think?

Rgds,


-- 
--
Pandu E Poluan - IT Optimizer
My website: http://pandu.poluan.info/

[gentoo-user] Re: Filesystem with lowest CPU load, acceptable emerge performance, and stable?

[Pandu Poluan]

Sorry, forgot one thing: For the time being, I'm sticking with
2.6.39-hardened. Saw too many incompatibility bug with 3.0 (due to
packages hard-wired to expect the kernel version to begin with "2.6").

Rgds,


On 2011-09-07, Pandu Poluan <pandu@poluan.info> wrote:
> So, can anyone recommend me a filesystem that fulfills my following needs:
>
> Scenario: vFirewall (virtual Firewall) that is going to be deployed at
> my IaaS Cloud Provider.
>
> Disk I/O Characteristic: Occasional writes during 'normal' usage,
> once-a-week eix-sync + emerge -avuD
>
> Priority: Stable (i.e., less chance of corruption), least CPU usage.
>
> My Google-Fu seems to indicate either XFS or JFS; what do you think?
>
> Rgds,
>
>
> --
> --
> Pandu E Poluan - IT Optimizer
> My website: http://pandu.poluan.info/
>


-- 
--
Pandu E Poluan - IT Optimizer
My website: http://pandu.poluan.info/

Re: [gentoo-user] Re: Filesystem with lowest CPU load, acceptable emerge performance, and stable?

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 1157 bytes --]

Am 06.09.2011 19:52, schrieb Pandu Poluan:
> On 2011-09-07, Pandu Poluan <pandu@poluan.info> wrote:
>> So, can anyone recommend me a filesystem that fulfills my following needs:
>>
>> Scenario: vFirewall (virtual Firewall) that is going to be deployed at
>> my IaaS Cloud Provider.
>>
>> Disk I/O Characteristic: Occasional writes during 'normal' usage,
>> once-a-week eix-sync + emerge -avuD
>>
>> Priority: Stable (i.e., less chance of corruption), least CPU usage.
>>
>> My Google-Fu seems to indicate either XFS or JFS; what do you think?
>>
>> Rgds,
>>
> Sorry, forgot one thing: For the time being, I'm sticking with
> 2.6.39-hardened. Saw too many incompatibility bug with 3.0 (due to
> packages hard-wired to expect the kernel version to begin with "2.6").
>
> Rgds,
>
>

JFS is a pretty good and care-free choice for this. Low resource usage.
Good performance, especially with large files. Although I must admit, I
wouldn't use it anymore since Ext4 is usually good enough for just about
every use-case and tested by more people in new kernel versions
(therefore presumably more stable).

Regards,
Florian Philipp


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [gentoo-user] Filesystem with lowest CPU load, acceptable emerge performance, and stable?

[kashani]

On 9/6/2011 10:26 AM, Pandu Poluan wrote:
> So, can anyone recommend me a filesystem that fulfills my following needs:
>
> Scenario: vFirewall (virtual Firewall) that is going to be deployed at
> my IaaS Cloud Provider.
>
> Disk I/O Characteristic: Occasional writes during 'normal' usage,
> once-a-week eix-sync + emerge -avuD
>
> Priority: Stable (i.e., less chance of corruption), least CPU usage.
>
> My Google-Fu seems to indicate either XFS or JFS; what do you think?

	I think it's a useless local optimization for no real world gain which 
only increases the complexity of your systems. Use the same filesystem 
you use on all your other servers.

kashani

Re: [gentoo-user] Filesystem with lowest CPU load, acceptable emerge performance, and stable?

[Pandu Poluan]

On Wed, Sep 7, 2011 at 01:15, kashani <kashani-list@badapple.net> wrote:
> On 9/6/2011 10:26 AM, Pandu Poluan wrote:
>>
>> So, can anyone recommend me a filesystem that fulfills my following needs:
>>
>> Scenario: vFirewall (virtual Firewall) that is going to be deployed at
>> my IaaS Cloud Provider.
>>
>> Disk I/O Characteristic: Occasional writes during 'normal' usage,
>> once-a-week eix-sync + emerge -avuD
>>
>> Priority: Stable (i.e., less chance of corruption), least CPU usage.
>>
>> My Google-Fu seems to indicate either XFS or JFS; what do you think?
>
>        I think it's a useless local optimization for no real world gain
> which only increases the complexity of your systems. Use the same filesystem
> you use on all your other servers.
>

Well, for all my other servers, I standardized on ext4.

Since a vFirewall have to perform lots of packet-juggling, I'd rather
dedicate the CPU time to the kernel rather than the HD I/O.

Of course, a vFirewall needs to be updated every now and then, but
everytime an update is called for, it should not overly tax the CPU
and degrade the netfilter framework.

Rgds,
-- 
FdS Pandu E Poluan
~ IT Optimizer ~

 • LOPSA Member #15248
 • Blog : http://pepoluan.tumblr.com
 • Linked-In : http://id.linkedin.com/in/pepoluan

Re: [gentoo-user] Filesystem with lowest CPU load, acceptable emerge performance, and stable?

[kashani]

On 9/7/2011 5:25 AM, Pandu Poluan wrote:
> On Wed, Sep 7, 2011 at 01:15, kashani<kashani-list@badapple.net>  wrote:
>> On 9/6/2011 10:26 AM, Pandu Poluan wrote:
>>>
>>> So, can anyone recommend me a filesystem that fulfills my following needs:
>>>
>>> Scenario: vFirewall (virtual Firewall) that is going to be deployed at
>>> my IaaS Cloud Provider.
>>>
>>> Disk I/O Characteristic: Occasional writes during 'normal' usage,
>>> once-a-week eix-sync + emerge -avuD
>>>
>>> Priority: Stable (i.e., less chance of corruption), least CPU usage.
>>>
>>> My Google-Fu seems to indicate either XFS or JFS; what do you think?
>>
>>         I think it's a useless local optimization for no real world gain
>> which only increases the complexity of your systems. Use the same filesystem
>> you use on all your other servers.
>>
>
> Well, for all my other servers, I standardized on ext4.
>
> Since a vFirewall have to perform lots of packet-juggling, I'd rather
> dedicate the CPU time to the kernel rather than the HD I/O.
>
> Of course, a vFirewall needs to be updated every now and then, but
> everytime an update is called for, it should not overly tax the CPU
> and degrade the netfilter framework.
>
> Rgds,

	You are making my point for me, but not realizing the end result of the 
logic. There isn't any filesystem change that is going to affect CPU 
usage by more than a few percentage points in the use case you've 
described. Rsync, portage, and gcc use a massive amount of CPU compared 
to the amount the filesystem changes will use other than brief points 
during the rsync. Additionally most benchmarks are testing filesystem 
throughput and comparing it to CPU. Because disk IO isn't under pressure 
in your scenario you're unlikely to see the pathological use of CPU that 
can highlight the differences between filesystems.
	That said, you have a few reasonable choices.

1. Move to a binary distro
2. Use buildpkg on a clone of this server and only install packages on 
your Firewall.
3. NFS mount /usr/portage when you need it and dist build on another server
4. Don't upgrade
5. Get a firewall server with more CPU so that it doesn't matter
6. Script a new firewall server install every x months and swap it into 
place and drop the original server.
7. Some combination of the above.

kashani

Re: [gentoo-user] Filesystem with lowest CPU load, acceptable emerge performance, and stable?

[Pandu Poluan]

On Thu, Sep 8, 2011 at 05:15, kashani <kashani-list@badapple.net> wrote:
> On 9/7/2011 5:25 AM, Pandu Poluan wrote:
>>
>> Well, for all my other servers, I standardized on ext4.
>>
>> Since a vFirewall have to perform lots of packet-juggling, I'd rather
>> dedicate the CPU time to the kernel rather than the HD I/O.
>>
>> Of course, a vFirewall needs to be updated every now and then, but
>> everytime an update is called for, it should not overly tax the CPU
>> and degrade the netfilter framework.
>>
>> Rgds,
>
>        You are making my point for me, but not realizing the end result of
> the logic. There isn't any filesystem change that is going to affect CPU
> usage by more than a few percentage points in the use case you've described.
> Rsync, portage, and gcc use a massive amount of CPU compared to the amount
> the filesystem changes will use other than brief points during the rsync.
> Additionally most benchmarks are testing filesystem throughput and comparing
> it to CPU. Because disk IO isn't under pressure in your scenario you're
> unlikely to see the pathological use of CPU that can highlight the
> differences between filesystems.

Gosh, you're right! (And Jesús' reply also remind me).

What was I thinking >.<

>        That said, you have a few reasonable choices.
>
> 1. Move to a binary distro
> 2. Use buildpkg on a clone of this server and only install packages on your
> Firewall.
> 3. NFS mount /usr/portage when you need it and dist build on another server
> 4. Don't upgrade
> 5. Get a firewall server with more CPU so that it doesn't matter
> 6. Script a new firewall server install every x months and swap it into
> place and drop the original server.
> 7. Some combination of the above.
>

I think I'll do (6). Attach a HD to another VM, install a similar
system on that HD (chroot-ed, of course), update that regularly, make
a stage5 (or 6 or whatevs) of the (ch)root, then do a 'tar xJf' on the
firewall proper.

So, a different scenario, then: Sometimes I need to log stuffs (via
ULOG) or do a tcpdump. Will JFS give me additional benefit to ext4? Or
should I just stick with ext4?

Rgds,
-- 
FdS Pandu E Poluan
~ IT Optimizer ~

 • LOPSA Member #15248
 • Blog : http://pepoluan.tumblr.com
 • Linked-In : http://id.linkedin.com/in/pepoluan

Re: [gentoo-user] Filesystem with lowest CPU load, acceptable emerge performance, and stable?

[kashani]

On 9/8/2011 12:52 AM, Pandu Poluan wrote:
>
> So, a different scenario, then: Sometimes I need to log stuffs (via
> ULOG) or do a tcpdump. Will JFS give me additional benefit to ext4? Or
> should I just stick with ext4?

	Simplest performance gain for appends is to drop atime/dirtime from the 
file or filesystem. It's a fairly common practice on database servers 
though the gains are relatively minor. I'm not sure how much it would 
affect logging, but it would be fairly easy to test.

kashani

Re: [gentoo-user] Filesystem with lowest CPU load, acceptable emerge performance, and stable?

[Permjacov Evgeniy]

On 09/06/2011 09:26 PM, Pandu Poluan wrote:
> So, can anyone recommend me a filesystem that fulfills my following needs:
>
> Scenario: vFirewall (virtual Firewall) that is going to be deployed at
> my IaaS Cloud Provider.
>
> Disk I/O Characteristic: Occasional writes during 'normal' usage,
> once-a-week eix-sync + emerge -avuD
>
> Priority: Stable (i.e., less chance of corruption), least CPU usage.
>
> My Google-Fu seems to indicate either XFS or JFS; what do you think?
>
> Rgds,
>
>
The best fs for emerge is tmpfs on TMP_PORTDIR. I run box with tmpfs on
both /var/tmp and /tmp and happy with it -)

For fs CPU usage is nothing, IO usage is a real problem and weak point.
Thus, you are free to choose any fs with full journaling. ext3 allows
full journaling as option, as well as ext4 and ext4 is little faster if
tuned properly. JFS/XFS journals metadata only. Remember that journaling
makes writes (i.e. emerge) a bit slower.

Re: [gentoo-user] Filesystem with lowest CPU load, acceptable emerge performance, and stable?

[Michael Mol]

On Tue, Sep 6, 2011 at 2:55 PM, Permjacov Evgeniy <permeakra@gmail.com> wrote:
> On 09/06/2011 09:26 PM, Pandu Poluan wrote:
>> So, can anyone recommend me a filesystem that fulfills my following needs:
>>
>> Scenario: vFirewall (virtual Firewall) that is going to be deployed at
>> my IaaS Cloud Provider.
>>
>> Disk I/O Characteristic: Occasional writes during 'normal' usage,
>> once-a-week eix-sync + emerge -avuD
>>
>> Priority: Stable (i.e., less chance of corruption), least CPU usage.
>>
>> My Google-Fu seems to indicate either XFS or JFS; what do you think?
>>
>> Rgds,
>>
>>
> The best fs for emerge is tmpfs on TMP_PORTDIR. I run box with tmpfs on
> both /var/tmp and /tmp and happy with it -)

Watch out that some ebuilds can and will fail if you exceed the
capacity of your tmpfs. Numerous factors will contribute to the space
required by portage during an emerge.

-- 
:wq

Re: [gentoo-user] Filesystem with lowest CPU load, acceptable emerge performance, and stable?

[James Broadhead]

On 6 September 2011 19:55, Permjacov Evgeniy <permeakra@gmail.com> wrote:
> On 09/06/2011 09:26 PM, Pandu Poluan wrote:
>> Disk I/O Characteristic: Occasional writes during 'normal' usage,
>> once-a-week eix-sync + emerge -avuD
>> Priority: Stable (i.e., less chance of corruption), least CPU usage.

You would have to profile this, but I imagine that the best approach
would be to compile in a RAM disk and copy. I think that you're
probably trying to optimise the wrong part of this problem.

As for ext3/ext4, the improvements to fsck alone make ext4 the FS of
choice between the two.

JB

Re: [gentoo-user] Filesystem with lowest CPU load, acceptable emerge performance, and stable?

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 888 bytes --]

Am 06.09.2011 21:24, schrieb James Broadhead:
> On 6 September 2011 19:55, Permjacov Evgeniy <permeakra@gmail.com> wrote:
>> On 09/06/2011 09:26 PM, Pandu Poluan wrote:
>>> Disk I/O Characteristic: Occasional writes during 'normal' usage,
>>> once-a-week eix-sync + emerge -avuD
>>> Priority: Stable (i.e., less chance of corruption), least CPU usage.
> 
> You would have to profile this, but I imagine that the best approach
> would be to compile in a RAM disk and copy. I think that you're
> probably trying to optimise the wrong part of this problem.
> 
> As for ext3/ext4, the improvements to fsck alone make ext4 the FS of
> choice between the two.
> 
> JB
> 

Pandu is building a firewall. Putting a ton of RAM in it just for the
sake for system updates is plain overkill and -- depending on his IaaS
provider -- pretty expensive.

Regards,
Florian Philipp


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [gentoo-user] Filesystem with lowest CPU load, acceptable emerge performance, and stable?

[Pandu Poluan]

On Wed, Sep 7, 2011 at 19:06, Florian Philipp <lists@binarywings.net> wrote:
> Am 06.09.2011 21:24, schrieb James Broadhead:
>> On 6 September 2011 19:55, Permjacov Evgeniy <permeakra@gmail.com> wrote:
>>> On 09/06/2011 09:26 PM, Pandu Poluan wrote:
>>>> Disk I/O Characteristic: Occasional writes during 'normal' usage,
>>>> once-a-week eix-sync + emerge -avuD
>>>> Priority: Stable (i.e., less chance of corruption), least CPU usage.
>>
>> You would have to profile this, but I imagine that the best approach
>> would be to compile in a RAM disk and copy. I think that you're
>> probably trying to optimise the wrong part of this problem.
>>
>> As for ext3/ext4, the improvements to fsck alone make ext4 the FS of
>> choice between the two.
>>
>> JB
>>
>
> Pandu is building a firewall. Putting a ton of RAM in it just for the
> sake for system updates is plain overkill and -- depending on his IaaS
> provider -- pretty expensive.
>

Indeed. If I need more RAM, they will only sell a complete package of
vCPU+RAM+Storage, which will then be merged with my current package.

They do offer Storage-only add-on packages, though. But that's beside the point.

Rgds,
-- 
FdS Pandu E Poluan
~ IT Optimizer ~

 • LOPSA Member #15248
 • Blog : http://pepoluan.tumblr.com
 • Linked-In : http://id.linkedin.com/in/pepoluan

Re: [gentoo-user] Filesystem with lowest CPU load, acceptable emerge performance, and stable?

[Pandu Poluan]

On Wed, Sep 7, 2011 at 02:24, James Broadhead <jamesbroadhead@gmail.com> wrote:
> On 6 September 2011 19:55, Permjacov Evgeniy <permeakra@gmail.com> wrote:
>> On 09/06/2011 09:26 PM, Pandu Poluan wrote:
>>> Disk I/O Characteristic: Occasional writes during 'normal' usage,
>>> once-a-week eix-sync + emerge -avuD
>>> Priority: Stable (i.e., less chance of corruption), least CPU usage.
>
> You would have to profile this, but I imagine that the best approach
> would be to compile in a RAM disk and copy. I think that you're
> probably trying to optimise the wrong part of this problem.
>

Hmmm... that gives me an idea...

If I have some free time, I'll experiment with doing an 'emerge -e
@world' on the various filesystems, and recording their total time
*and* CPU load.

Is the `sar` utility good enough to record CPU load?

Rgds,
-- 
FdS Pandu E Poluan
~ IT Optimizer ~

 • LOPSA Member #15248
 • Blog : http://pepoluan.tumblr.com
 • Linked-In : http://id.linkedin.com/in/pepoluan

Re: [gentoo-user] Filesystem with lowest CPU load, acceptable emerge performance, and stable?

[Jesús J. Guerrero Botella]

JFS is very soft on cpu usage, and ext4 does a very good job overall.
fsck times for ext4 makes it probably the best choice for a server,
plus it has more eyes watching over it.

In any case, I wanted to call your attention that this might not be
the best choice anyway. If you truly want portage to have the minimum
possible impact on the performance of your machine(s) you should
probably be using other machine(s) to build binary packages, then use
these package in the target system(s) (i.e. with emerge -K).
-- 
Jesús Guerrero Botella

Re: [gentoo-user] Filesystem with lowest CPU load, acceptable emerge performance, and stable?

[Andrea Conti]

> So, can anyone recommend me a filesystem that fulfills my following needs:
> 
> Scenario: vFirewall (virtual Firewall) that is going to be deployed at
> my IaaS Cloud Provider.
> 
> Disk I/O Characteristic: Occasional writes during 'normal' usage,
> once-a-week eix-sync + emerge -avuD
> 
> Priority: Stable (i.e., less chance of corruption), least CPU usage.
> 
> My Google-Fu seems to indicate either XFS or JFS; what do you think?

IMHO a firewall (physical or virtual) is something that fits strictly
into the "appliance" category. It must do only one thing and do it well,
with as little complexity and maintenance overhead as possible. Why in
the world would anyone want to run gentoo (which among the rest needs
portage and a whole compiler stack) -- or for that matter any other
full-fledged linux distribution -- on something like that in production
is beyond me...

That said, XFS and JFS are targeted at completely different use cases
and are way too complex for your scenario. Without appropriately-sized
hardware I'm not even sure XFS fits in the "stable" category. Stick to
ext3, keeping an eye on the inode count for /usr/portage as the default
value on a small partition probably won't be enough.

Fs-related CPU usage in a firewall (which has nearly zero disk activity
when up and running) is mostly a non-issue unless you need some form of
heavy logging or you're doing something wrong.

Weekly updates, on the other hand are exposing you to the risk of random
breakages and -- if you compile from source -- are going to cost you a
serious amount of CPU. My advice would be to limit updates to those
fixing known vulnerabilities, and even then compiling somewhere else and
doing binary installs would be preferable.

andrea

Re: [gentoo-user] Filesystem with lowest CPU load, acceptable emerge performance, and stable?

[Walter Dnes]

On Wed, Sep 07, 2011 at 12:26:15AM +0700, Pandu Poluan wrote
> So, can anyone recommend me a filesystem that fulfills my following needs:
> 
> Scenario: vFirewall (virtual Firewall) that is going to be deployed at
> my IaaS Cloud Provider.
> 
> Disk I/O Characteristic: Occasional writes during 'normal' usage,
> once-a-week eix-sync + emerge -avuD
> 
> Priority: Stable (i.e., less chance of corruption), least CPU usage.
> 
> My Google-Fu seems to indicate either XFS or JFS; what do you think?

  Try "thinking outside the box".  Do you really need more than extfs2?
That should be the ultimate in low-overhead writing on the device.
Another option is to send the log data out on UDP port 514 to be logged
on another machine.  A cute trick is to have /etc/conf.d/net as follows

config_eth0="
192.168.123.2/24 broadcast 192.168.123.255
routes_eth0="
default via 192.168.123.254

  And then send the log data to the broadcast address 192.168.123.255
UDP port 514.  Any computer with the same broadcast address can receive
the log data.  You can even have multiple computers sending out, and
multiple computers receiving.  One of the first things an attacker does
after compromising a machine is to wipe the logs on that machine to
cover his tracks.  If the log data goes to multiple different machines,
it will be much more difficult to wipe.

  Another strategy, on the paranoid side, is to have the router sending
logs to a machine like 192.168.123.45, and also have a machine on a
totally different IP address (e.g. 10.0.0.1) with its NIC set to
"promiscuous mode", listen for and save the log data.

-- 
Walter Dnes <waltdnes@waltdnes.org>

Re: [gentoo-user] Filesystem with lowest CPU load, acceptable emerge performance, and stable?

[Jesús J. Guerrero Botella]

It all comes down to what do you want to prioritize here.

If you want minimal downtimes in case that there's a power source
failure of any kind, then you probably want ext4 which will give you
the fastest fsck times. Or, you might want to check into sqashfs on a
flash drive for your rootfs and use whatever else for writable parts
(/tmp,/var/log/, etc.), and update only when strictly necessary (GLSAs
can probably help you there). After all, as someone else said above,
this machine just needs to do one thing, and do it well. If you plan
to make stage4/5/6 or whatever the trend is nowadays to name it, you
don't even need portage or a toolchain in that box, and having it will
only be a security risk since some rootkits comes in the form of a
kernel module that needs to be compiled for your specific kernel and
architecture (eliminating the kernel sources and the compiler you sort
that out from the very root).

In any case, the cpu won't be a limiting factor or a bottleneck,
whatever your definitive choice shall be.


-- 
Jesús Guerrero Botella