[gentoo-user] vmWare HowTo / best practices

[gentoo-user] vmWare HowTo / best practices

[Tanstaafl]

Hi all,

Previously I had asked for some help with a preconfigured image, but 
decided against that, and have been playing and reading.

I'm ready to get down to brass tacks with the ultimate goal of getting a 
new gentoo vm up and running on my esxi host this weekend.

Can someone point me to some recent/decent docs on best practices for 
this? Ideally gentoo related, but just general linux related would be ok 
too.

Things like vmware-tools installation (is open-vm-tools good enough 
nowadays?), time syncing, snapshots/backups, etc is what I'm looking for.

Thanks,

Charles

Re: [gentoo-user] vmWare HowTo / best practices

[Jarry]

On 19-Apr-13 16:21, Tanstaafl wrote:

> Previously I had asked for some help with a preconfigured image, but
> decided against that, and have been playing and reading.
>
> I'm ready to get down to brass tacks with the ultimate goal of getting a
> new gentoo vm up and running on my esxi host this weekend.
>
> Can someone point me to some recent/decent docs on best practices for
> this? Ideally gentoo related, but just general linux related would be ok
> too.
>
> Things like vmware-tools installation (is open-vm-tools good enough
> nowadays?), time syncing, snapshots/backups, etc is what I'm looking for.

May I join the club? I have been running a few Gentoo-VMs
for some time, but I'm still quite new to this "ESXi-world".
But one I know for sure is that hypervisor-virtualization
is much more complex than OS-virtualization (i.e. VServer
or OpenVZ which I have used previously).

vmware-tools: I have tested open-vm-tools but now I'm running
my VMs without them because every kernel upgrade was a real
pain in a**. And trully I did not see any benefit in running
vm-tools (maybe it would be different on desktop). For
shutdown of Gentoo-VMs from ESXi I use ssh-script or
hibernation.

Snapshots are very well covered by esxi and for backup I use
ghetto-vcb tool (script). It tried backup&restore on one
of my running Gentoo-VM servers and it works like charm.

For VM-hardware I used (iirc) CentOS template, because
with "other linux 64b" I did not get hw-options I wanted
to use (LSI-Logic Parallel SCSI controller, and VMXNET3
network adapter).

Unfortunatelly there is not a lot info about Gentoo & ESXi
and what exists is quite outdated (i.e. Gentoo-wiki). But
I used guidelines for general linux-VM, and I consulted
problems on VMware community web-page...

Jarry
-- 
_______________________________________________________________
This mailbox accepts e-mails only from selected mailing-lists!
Everything else is considered to be spam and therefore deleted.

Re: [gentoo-user] vmWare HowTo / best practices

[Pandu Poluan]

[-- Attachment #1: Type: text/plain, Size: 2305 bytes --]

On Apr 19, 2013 10:24 PM, "Jarry" <mr.jarry@gmail.com> wrote:
>
> On 19-Apr-13 16:21, Tanstaafl wrote:
>
>> Previously I had asked for some help with a preconfigured image, but
>> decided against that, and have been playing and reading.
>>
>> I'm ready to get down to brass tacks with the ultimate goal of getting a
>> new gentoo vm up and running on my esxi host this weekend.
>>
>> Can someone point me to some recent/decent docs on best practices for
>> this? Ideally gentoo related, but just general linux related would be ok
>> too.
>>
>> Things like vmware-tools installation (is open-vm-tools good enough
>> nowadays?), time syncing, snapshots/backups, etc is what I'm looking for.
>
>
> May I join the club? I have been running a few Gentoo-VMs
> for some time, but I'm still quite new to this "ESXi-world".
> But one I know for sure is that hypervisor-virtualization
> is much more complex than OS-virtualization (i.e. VServer
> or OpenVZ which I have used previously).
>
> vmware-tools: I have tested open-vm-tools but now I'm running
> my VMs without them because every kernel upgrade was a real
> pain in a**. And trully I did not see any benefit in running
> vm-tools (maybe it would be different on desktop). For
> shutdown of Gentoo-VMs from ESXi I use ssh-script or
> hibernation.
>
> Snapshots are very well covered by esxi and for backup I use
> ghetto-vcb tool (script). It tried backup&restore on one
> of my running Gentoo-VM servers and it works like charm.
>
> For VM-hardware I used (iirc) CentOS template, because
> with "other linux 64b" I did not get hw-options I wanted
> to use (LSI-Logic Parallel SCSI controller, and VMXNET3
> network adapter).
>
> Unfortunatelly there is not a lot info about Gentoo & ESXi
> and what exists is quite outdated (i.e. Gentoo-wiki). But
> I used guidelines for general linux-VM, and I consulted
> problems on VMware community web-page...
>
> Jarry
> --

Well, for me, XenServer-based virtualization is very very simple. And if I
compile the kernel with all Xen PV (paravirtualized) 'FrontEnds', it runs
near-natively.

Only the xend daemon need some 'tweaking' to run properly.

Do a Google search for "gentoo xenserver" and if you find pages written by
me, those are my experiences running Gentoo on top of XenServer,
successfully.

Rgds,
--

[-- Attachment #2: Type: text/html, Size: 2835 bytes --]

Re: [gentoo-user] vmWare HowTo / best practices

[J. Roeleveld]

[-- Attachment #1: Type: text/plain, Size: 2653 bytes --]

Pandu Poluan <pandu@poluan.info> wrote:

>On Apr 19, 2013 10:24 PM, "Jarry" <mr.jarry@gmail.com> wrote:
>>
>> On 19-Apr-13 16:21, Tanstaafl wrote:
>>
>>> Previously I had asked for some help with a preconfigured image, but
>>> decided against that, and have been playing and reading.
>>>
>>> I'm ready to get down to brass tacks with the ultimate goal of
>getting a
>>> new gentoo vm up and running on my esxi host this weekend.
>>>
>>> Can someone point me to some recent/decent docs on best practices
>for
>>> this? Ideally gentoo related, but just general linux related would
>be ok
>>> too.
>>>
>>> Things like vmware-tools installation (is open-vm-tools good enough
>>> nowadays?), time syncing, snapshots/backups, etc is what I'm looking
>for.
>>
>>
>> May I join the club? I have been running a few Gentoo-VMs
>> for some time, but I'm still quite new to this "ESXi-world".
>> But one I know for sure is that hypervisor-virtualization
>> is much more complex than OS-virtualization (i.e. VServer
>> or OpenVZ which I have used previously).
>>
>> vmware-tools: I have tested open-vm-tools but now I'm running
>> my VMs without them because every kernel upgrade was a real
>> pain in a**. And trully I did not see any benefit in running
>> vm-tools (maybe it would be different on desktop). For
>> shutdown of Gentoo-VMs from ESXi I use ssh-script or
>> hibernation.
>>
>> Snapshots are very well covered by esxi and for backup I use
>> ghetto-vcb tool (script). It tried backup&restore on one
>> of my running Gentoo-VM servers and it works like charm.
>>
>> For VM-hardware I used (iirc) CentOS template, because
>> with "other linux 64b" I did not get hw-options I wanted
>> to use (LSI-Logic Parallel SCSI controller, and VMXNET3
>> network adapter).
>>
>> Unfortunatelly there is not a lot info about Gentoo & ESXi
>> and what exists is quite outdated (i.e. Gentoo-wiki). But
>> I used guidelines for general linux-VM, and I consulted
>> problems on VMware community web-page...
>>
>> Jarry
>> --
>
>Well, for me, XenServer-based virtualization is very very simple. And
>if I
>compile the kernel with all Xen PV (paravirtualized) 'FrontEnds', it
>runs
>near-natively.
>
>Only the xend daemon need some 'tweaking' to run properly.
>
>Do a Google search for "gentoo xenserver" and if you find pages written
>by
>me, those are my experiences running Gentoo on top of XenServer,
>successfully.
>
>Rgds,
>--

Pandu.

Do you still use xend on your Xen hosts?
I thought that was deprecated?

--
Joost
-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

[-- Attachment #2: Type: text/html, Size: 3392 bytes --]

Re: [gentoo-user] vmWare HowTo / best practices

[J. Roeleveld]

Jarry <mr.jarry@gmail.com> wrote:

>On 19-Apr-13 16:21, Tanstaafl wrote:
>
>> Previously I had asked for some help with a preconfigured image, but
>> decided against that, and have been playing and reading.
>>
>> I'm ready to get down to brass tacks with the ultimate goal of
>getting a
>> new gentoo vm up and running on my esxi host this weekend.
>>
>> Can someone point me to some recent/decent docs on best practices for
>> this? Ideally gentoo related, but just general linux related would be
>ok
>> too.
>>
>> Things like vmware-tools installation (is open-vm-tools good enough
>> nowadays?), time syncing, snapshots/backups, etc is what I'm looking
>for.
>
>May I join the club? I have been running a few Gentoo-VMs
>for some time, but I'm still quite new to this "ESXi-world".
>But one I know for sure is that hypervisor-virtualization
>is much more complex than OS-virtualization (i.e. VServer
>or OpenVZ which I have used previously).
>
>vmware-tools: I have tested open-vm-tools but now I'm running
>my VMs without them because every kernel upgrade was a real
>pain in a**. And trully I did not see any benefit in running
>vm-tools (maybe it would be different on desktop). For
>shutdown of Gentoo-VMs from ESXi I use ssh-script or
>hibernation.
>
>Snapshots are very well covered by esxi and for backup I use
>ghetto-vcb tool (script). It tried backup&restore on one
>of my running Gentoo-VM servers and it works like charm.
>
>For VM-hardware I used (iirc) CentOS template, because
>with "other linux 64b" I did not get hw-options I wanted
>to use (LSI-Logic Parallel SCSI controller, and VMXNET3
>network adapter).
>
>Unfortunatelly there is not a lot info about Gentoo & ESXi
>and what exists is quite outdated (i.e. Gentoo-wiki). But
>I used guidelines for general linux-VM, and I consulted
>problems on VMware community web-page...
>
>Jarry
>-- 
>_______________________________________________________________
>This mailbox accepts e-mails only from selected mailing-lists!
>Everything else is considered to be spam and therefore deleted.

I would suggest installing the vmaare tools when the VM is running on a VMWare server.

You get better driver performance and the management tools actually work.

--
Joost
-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

Re: [gentoo-user] vmWare HowTo / best practices

[Jarry]

On 19-Apr-13 17:52, Pandu Poluan wrote:

> Well, for me, XenServer-based virtualization is very very simple. And if
> I compile the kernel with all Xen PV (paravirtualized) 'FrontEnds', it
> runs near-natively.
>
> Only the xend daemon need some 'tweaking' to run properly.
>
> Do a Google search for "gentoo xenserver" and if you find pages written
> by me, those are my experiences running Gentoo on top of XenServer,
> successfully.

What I had in mind is administration of hypervisor itself.
ESXi is feature-rich product, and to handle all its possibilities
(i.e. vMotion, vShield, HA, FT, vCenter, DRS/DPM, FW, etc) one have
to spend quite long time by studying and the learning curve is
very steep (again, I'm comparing with VServer or OpenVZ/Virtuozzo,
I do not know XenServer).

Deploying Gentoo-guest (or "VM" / "DomU" as they call it) is
actually very easy. And after reading your wiki-page I'd say
it is easier on ESXi then on XenServer, because there is actually
no difference between installing Gentoo on VM, or real hardware
(no need for special compile options or special device-files,
no limit on boot-loader, etc.).

Jarry
-- 
_______________________________________________________________
This mailbox accepts e-mails only from selected mailing-lists!
Everything else is considered to be spam and therefore deleted.

Re: [gentoo-user] vmWare HowTo / best practices

[Kvothe Tech]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Tanstaafl <tanstaafl@libertytrek.org> wrote:

>Hi all,
>
>Previously I had asked for some help with a preconfigured image, but
>decided against that, and have been playing and reading.
>
>I'm ready to get down to brass tacks with the ultimate goal of getting
>a
>new gentoo vm up and running on my esxi host this weekend.
>
>Can someone point me to some recent/decent docs on best practices for
>this? Ideally gentoo related, but just general linux related would be
>ok
>too.
>
>Things like vmware-tools installation (is open-vm-tools good enough
>nowadays?), time syncing, snapshots/backups, etc is what I'm looking
>for.
>
>Thanks,
>
>Charles

I used VMware tools but I suspect I didn't need to.  As was already mentioned ESX should take care of most of that.  Other than that I just followed the gentoo handbook and it all works for me
- --
Kvothe Tech
Kvothe@kvothetech.com
http://kvothetech.com/
-----BEGIN PGP SIGNATURE-----
Version: APG v1.0.8

iQJBBAEBCAArBQJRcYtaJBxLdm90aGUgVGVjaCA8a3ZvdGhlQGt2b3RoZXRlY2gu
Y29tPgAKCRDpajPkVlu7OjMdD/9CjIt1GVXKC5u1YbBkaV66ThgGQ2p0xArEsoJB
ME4hHQGDjrMZy2EmtweRhfHlrK33ln7j/FYlndoUMET93/QvdQ+x5iEnPNsqOkql
pdWJxvVV0YqbLXdhJbkhvR9NckkUE+NdIv3eUjh8OmGOOPbb1dzQzR6GmnwxIBZT
dtPZb1wET39qC14YFNdOlsmgu6rnO+qkwjJ0XjewlkXkzBVk5p318YLL970R7dAw
nYomoVRi36ZYqK97NbBZDMtd6inXS5ARVcs8zjXrmUr49NqypkX0c8HWB57C9BR0
YBYeHPczJ4ZlUQXMot22ccbjM2KFrdASwCp9Ot1BdO1yJOlOO8COVIZHFPryAw1D
6HRaj7C8x5JjXRSFLnuluX/laQUdwFpaZtqyuxXf9tOlxTBJ5XehQLkM954nczjd
quex+iHERsLViSg0BwEFnjIrFje8BtLVSlvZ2CSgSUBkvz8KdqyVDsNSsve87E6w
+I9NzXAuxb/COLywMwoDI4qstmLGllscdyhbgJQri5N/EweosKtLXviML46cRZx6
Rpw+yLMqz3tCKcTVJpofTx6cYc2IHKapGSZpvo3yP1DMibZfB21SebVRkJ58OjJw
SYxzuUfjgxptQkumI/DEoiuoKdzyLo4BDSZsuMToUT1fbTnbSd+hhaHuyWhra2dG
Lwyhpg==
=bi6L
-----END PGP SIGNATURE-----

Re: [gentoo-user] vmWare HowTo / best practices

[J. Roeleveld]

On Fri, April 19, 2013 18:42, Jarry wrote:
> On 19-Apr-13 17:52, Pandu Poluan wrote:
>
>> Well, for me, XenServer-based virtualization is very very simple. And if
>> I compile the kernel with all Xen PV (paravirtualized) 'FrontEnds', it
>> runs near-natively.
>>
>> Only the xend daemon need some 'tweaking' to run properly.
>>
>> Do a Google search for "gentoo xenserver" and if you find pages written
>> by me, those are my experiences running Gentoo on top of XenServer,
>> successfully.
>
> What I had in mind is administration of hypervisor itself.
> ESXi is feature-rich product, and to handle all its possibilities
> (i.e. vMotion, vShield, HA, FT, vCenter, DRS/DPM, FW, etc) one have
> to spend quite long time by studying and the learning curve is
> very steep (again, I'm comparing with VServer or OpenVZ/Virtuozzo,
> I do not know XenServer).
>
> Deploying Gentoo-guest (or "VM" / "DomU" as they call it) is
> actually very easy. And after reading your wiki-page I'd say
> it is easier on ESXi then on XenServer, because there is actually
> no difference between installing Gentoo on VM, or real hardware
> (no need for special compile options or special device-files,
> no limit on boot-loader, etc.).

Actually, deploying it on ESXi and on XenServer is both very easy.
The difference is, XenServer has 2 options for the guests:
1) Fully Virtualised
2) Paravirtualised

ESXi only supports the first.
If you install all VMs using the first option, it is very simple.

But, if you want maximum (as in, near native) performance, the 2nd option
is definitely worth the extra effort.

I use a Gentoo Dom0 (Xen Host) with several Gentoo VMs running on top of
it. I only had to add a few options to the kernel configuration to get the
VMs working.
Similar effort to installing a Gentoo guest on ESXi, but on ESXi, I would
need to add the VMWare tools to get the VMs to shutdown correctly when I
need to shutdown the host.

--
Joost Roeleveld

[gentoo-user] Best filesystem for virtualized gentoo mail server - WAS: vmWare HowTo / best practices

[Tanstaafl]

Thanks for the responses so far...

Another question - are there any caveats as to which filesystem to use 
for a mail server, for virtualized systems? Ir do the same 
issues/questions apply (ie, does the fact that it is virtualized not 
change anything)?

If there are none, I'm curious what others prefer.

I've been using reiserfs on my old mail server since it was first set up 
(over 8 years ago). I have had no issues with it whatsoever, and even 
had one scare with a bad UPS causing the system to experienc an unclean 
shutdown - but it came back up, auto fsck'd, and there was no 'apparent' 
data loss (this was a very long time ago, so if there had been any 
serious problems, I'd have known about it long go).

I've been considering using XFS, but have never used it before.

So, anyway, opinions are welcome...

Thanks again

Charles

Re: [gentoo-user] Best filesystem for virtualized gentoo mail server - WAS: vmWare HowTo / best practices

[Alan McKinnon]

On 20/04/2013 17:00, Tanstaafl wrote:
> Thanks for the responses so far...
> 
> Another question - are there any caveats as to which filesystem to use
> for a mail server, for virtualized systems? Ir do the same
> issues/questions apply (ie, does the fact that it is virtualized not
> change anything)?
> 
> If there are none, I'm curious what others prefer.
> 
> I've been using reiserfs on my old mail server since it was first set up
> (over 8 years ago). I have had no issues with it whatsoever, and even
> had one scare with a bad UPS causing the system to experienc an unclean
> shutdown - but it came back up, auto fsck'd, and there was no 'apparent'
> data loss (this was a very long time ago, so if there had been any
> serious problems, I'd have known about it long go).
> 
> I've been considering using XFS, but have never used it before.
> 
> So, anyway, opinions are welcome...


Virtualization can change things, and it's not really intuitive.

Regardless of what optimizations you apply to the VM, and regardless of
what kind of virtualization is in use on the host, you are still going
to be bound by the disk and fs behaviour of the host. If VMWare gives
you a really shitty host driver, then something really shitty is going
to be the best you can achieve.

Disks aren't like eg NICs, you can't easily virtualize them and give the
guest exclusive access in the style of para-virtualization (I can't
imagine how that would even be done).

You also didn't mention what mail server you use - implementations vary
a great deal. Gut feel tells me that unless you are dealing with many
1000s of mails in a short period you won't really need XFS's aggressive
caching. But I'm happy to be proved wrong and numbers tell the truth :-)

I think the best you will get here is a list of combinations that are
unlikely to suit, and you will have to do your own extensive testing to
find what works best in your area.

FWIW, I have two mail relays (no mail storage) running old postfix
versions on FreeBSD. I expected throughput to differ when virtualized on
ESXi, but in practice I couldn't see a difference at all - maybe the
mail servers were very under-utilized. Considering this pair deal with
anything between 500,000 to a million mails a day total, I would not
have considered them "under-utilized". Just goes to show how opinions
are often worthless but numbers buys the whiskey :-)


-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] Best filesystem for virtualized gentoo mail server - WAS: vmWare HowTo / best practices

[Jarry]

On 20-Apr-13 17:00, Tanstaafl wrote:

> Another question - are there any caveats as to which filesystem to use
> for a mail server, for virtualized systems? Ir do the same
> issues/questions apply (ie, does the fact that it is virtualized not
> change anything)?

Problem of virtualized filesystem is not that it is virtualized,
but that it is located on datastore with more virtual systems,
all of them competing for the same i/o. *That* is the bottleneck.
If you switch reiser for xfs or btrfs, you might win (or loose)
a few %. If you optimize your esxi-datastore design, you might
win much more than what you have ever dreamed of.

I have 8 VMs (out of them 6 are Gentoo) hosted on ESXi, intended
for various tasks (mail, dns, mysql, web, etc), moderately loaded.
I used hw-raid controller with 2x sata-hdd in raid1 but performance
was quite dissapointing and I experienced all sorts of i/o jams.
Then I switched hdd for ssd (yes I use 2 of them in raid1, even
if this is not generally recommended) and performance rocks now!
I can start now kernel compilation on all 6 VMs at the same time,
with near-zero performance penalty (depending on cpu/vcpu ratio
and number of threads used). Unthinkable with hdd-based datastore.

I would definitely recommend using SSD. Either directly as
datastore for VMs, or at least as EXSi host-cache. There is
also possibility of "hybrid-raid" (1xSSD and 1xHDD in raid1)
on some raid-controllers. Or if your pocket is really deep,
you could grab one of those FusionIO-cards to avoid being
limited by rather slow sata-interface (SSD for PCIe)...

Jarry

-- 
_______________________________________________________________
This mailbox accepts e-mails only from selected mailing-lists!
Everything else is considered to be spam and therefore deleted.

Re: [gentoo-user] vmWare HowTo / best practices

[Pandu Poluan]

[-- Attachment #1: Type: text/plain, Size: 3185 bytes --]

On Apr 19, 2013 11:14 PM, "J. Roeleveld" <joost@antarean.org> wrote:i
>
> Pandu Poluan <pandu@poluan.info> wrote:
>>
>>
>> On Apr 19, 2013 10:24 PM, "Jarry" <mr.jarry@gmail.com> wrote:
>> >
>> > On 19-Apr-13 16:21, Tanstaafl wrote:
>> >
>> >> Previously I had asked for some help with a preconfigured image, but
>> >> decided against that, and have been playing and reading.
>> >>
>> >> I'm ready to get down to brass tacks with the ultimate goal of
getting a
>> >> new gentoo vm up and running on my esxi host this weekend.
>> >>
>> >> Can someone point me to some recent/decent docs on best practices for
>> >> this? Ideally gentoo related, but just general linux related would be
ok
>> >> too.
>> >>
>> >> Things like vmware-tools installation (is open-vm-tools good enough
>> >> nowadays?), time syncing, snapshots/backups, etc is what I'm looking
for.
>> >
>> >
>> > May I join the club? I have been running a few Gentoo-VMs
>> > for some time, but I'm still quite new to this "ESXi-world".
>> > But one I know for sure is that hypervisor-virtualization
>> > is much more complex than OS-virtualization (i.e. VServer
>> > or OpenVZ which I have used previously).
>> >
>> > vmware-tools: I have tested open-vm-tools but now I'm running
>> > my VMs without them because every kernel upgrade was a real
>> > pain in a**. And trully I did not see any benefit in running
>> > vm-tools (maybe it would be different on desktop). For
>> > shutdown of Gentoo-VMs from ESXi I use ssh-script or
>> > hibernation.
>> >
>> > Snapshots are very well covered by esxi and for backup I use
>> > ghetto-vcb tool (script). It tried backup&restore on one
>> > of my running Gentoo-VM servers and it works like charm.
>> >
>> > For VM-hardware I used (iirc) CentOS template, because
>> > with "other linux 64b" I did not get hw-options I wanted
>> > to use (LSI-Logic Parallel SCSI controller, and VMXNET3
>> > network adapter).
>> >
>> > Unfortunatelly there is not a lot info about Gentoo & ESXi
>> > and what exists is quite outdated (i.e. Gentoo-wiki). But
>> > I used guidelines for general linux-VM, and I consulted
>> > problems on VMware community web-page...
>> >
>> > Jarry
>> > --
>>
>> Well, for me, XenServer-based virtualization is very very simple. And if
I compile the kernel with all Xen PV (paravirtualized) 'FrontEnds', it runs
near-natively.
>>
>> Only the xend daemon need some 'tweaking' to run properly.
>>
>> Do a Google search for "gentoo xenserver" and if you find pages written
by me, those are my experiences running Gentoo on top of XenServer,
successfully.
>>
>> Rgds,
>> --
>
>
> Pandu.
>
> Do you still use xend on your Xen hosts?
> I thought that was deprecated?
>

Ah, sorry. What I meant was xstools daemon. It's necessary to properly
monitor Linux PV guests on XenServer.

It's apparently a simple shell script, which commits suicide when it can't
determine what Linux distro it's running in. Need to add several lines of
code within the script to enable it to recognize Gentoo and not commit
suicide.

XenCenter/CloudStack will.afterwards detect that the machine is running
Gentoo. Won't affect anything, but better than it reporting "unknown Linux"
:-)

Rgds,
--

[-- Attachment #2: Type: text/html, Size: 4305 bytes --]

Re: [gentoo-user] vmWare HowTo / best practices

[Pandu Poluan]

[-- Attachment #1: Type: text/plain, Size: 2948 bytes --]

On Apr 20, 2013 9:31 PM, "J. Roeleveld" <joost@antarean.org> wrote:
>
> On Fri, April 19, 2013 18:42, Jarry wrote:
> > On 19-Apr-13 17:52, Pandu Poluan wrote:
> >
> >> Well, for me, XenServer-based virtualization is very very simple. And
if
> >> I compile the kernel with all Xen PV (paravirtualized) 'FrontEnds', it
> >> runs near-natively.
> >>
> >> Only the xend daemon need some 'tweaking' to run properly.
> >>
> >> Do a Google search for "gentoo xenserver" and if you find pages written
> >> by me, those are my experiences running Gentoo on top of XenServer,
> >> successfully.
> >
> > What I had in mind is administration of hypervisor itself.
> > ESXi is feature-rich product, and to handle all its possibilities
> > (i.e. vMotion, vShield, HA, FT, vCenter, DRS/DPM, FW, etc) one have
> > to spend quite long time by studying and the learning curve is
> > very steep (again, I'm comparing with VServer or OpenVZ/Virtuozzo,
> > I do not know XenServer).
> >
> > Deploying Gentoo-guest (or "VM" / "DomU" as they call it) is
> > actually very easy. And after reading your wiki-page I'd say
> > it is easier on ESXi then on XenServer, because there is actually
> > no difference between installing Gentoo on VM, or real hardware
> > (no need for special compile options or special device-files,
> > no limit on boot-loader, etc.).
>
> Actually, deploying it on ESXi and on XenServer is both very easy.
> The difference is, XenServer has 2 options for the guests:
> 1) Fully Virtualised
> 2) Paravirtualised
>
> ESXi only supports the first.
> If you install all VMs using the first option, it is very simple.
>
> But, if you want maximum (as in, near native) performance, the 2nd option
> is definitely worth the extra effort.
>
> I use a Gentoo Dom0 (Xen Host) with several Gentoo VMs running on top of
> it. I only had to add a few options to the kernel configuration to get the
> VMs working.
> Similar effort to installing a Gentoo guest on ESXi, but on ESXi, I would
> need to add the VMWare tools to get the VMs to shutdown correctly when I
> need to shutdown the host.
>
> --
> Joost Roeleveld
>
>

True. Since on Gentoo we have access to the source code (and have to
compile our own kernel anyways), it's just wasteful top not leverage the
near-native performance of full PV (paravirtualized) mode, in which all
important devices, namely storage and networking, are offloaded to the dom0
via the proper hypercalls.

XenServer also has all the so-called 'advanced' features of VMware, even in
the free edition. Citrix only charged for the 'automation' tools (e.g.,
automatic workload balancing). But if one is well versed with programming
and the CloudStack API, one can easily create one's own automation system.

In my current employment, I have been busy rebuilding the data center using
XenServer, and decommissioning many VMware ESXi hosts while at it. By the
end of this year, I expect we will be VMware-free.

Rgds,
--

Rgds,
--

[-- Attachment #2: Type: text/html, Size: 3657 bytes --]

Re: [gentoo-user] Best filesystem for virtualized gentoo mail server - WAS: vmWare HowTo / best practices

[Pandu Poluan]

[-- Attachment #1: Type: text/plain, Size: 1841 bytes --]

On Apr 20, 2013 10:01 PM, "Tanstaafl" <tanstaafl@libertytrek.org> wrote:
>
> Thanks for the responses so far...
>
> Another question - are there any caveats as to which filesystem to use
for a mail server, for virtualized systems? Ir do the same issues/questions
apply (ie, does the fact that it is virtualized not change anything)?
>
> If there are none, I'm curious what others prefer.
>
> I've been using reiserfs on my old mail server since it was first set up
(over 8 years ago). I have had no issues with it whatsoever, and even had
one scare with a bad UPS causing the system to experienc an unclean
shutdown - but it came back up, auto fsck'd, and there was no 'apparent'
data loss (this was a very long time ago, so if there had been any serious
problems, I'd have known about it long go).
>
> I've been considering using XFS, but have never used it before.
>
> So, anyway, opinions are welcome...
>
> Thanks again
>
> Charles
>

Reiterating what others have said, in a virtualized environment, it's how
you build the underlying storage that will have the greatest effect on
performance.

Just an illustration: in my current employment, we have a very heavily used
database (SQL Server). To ensure good performance, I dedicated a RAID array
of 8 drives (15k RPM each), ensure that the space allocation is 'thick' not
'thin', and dedicate the whole RAID array to just that one VM. Performance
went through the roof with that one... especially since it was originally a
physical server running on top of 4 x 7200 RPM drives ;-)

If you have the budget, you really should invest in a SAN Storage solution
that can provide "tiered storage", in which frequently used blocks will be
'cached' in SSD, while less frequently used blocks are migrated first to
slower SAS drives, and later on (if 'cold') to even slower SATA drives.

Rgds,
--

[-- Attachment #2: Type: text/html, Size: 2175 bytes --]

Re: [gentoo-user] vmWare HowTo / best practices

[J. Roeleveld]

On Sat, April 20, 2013 18:06, Pandu Poluan wrote:
> On Apr 19, 2013 11:14 PM, "J. Roeleveld" <joost@antarean.org> wrote:i

<SNIPPED>

>> Pandu.
>>
>> Do you still use xend on your Xen hosts?
>> I thought that was deprecated?
>>
>
> Ah, sorry. What I meant was xstools daemon. It's necessary to properly
> monitor Linux PV guests on XenServer.
>
> It's apparently a simple shell script, which commits suicide when it can't
> determine what Linux distro it's running in. Need to add several lines of
> code within the script to enable it to recognize Gentoo and not commit
> suicide.
>
> XenCenter/CloudStack will.afterwards detect that the machine is running
> Gentoo. Won't affect anything, but better than it reporting "unknown
> Linux"
> :-)

That's useful to know. I tend to use binary distro's on the XCP-server
myself. But that is because that machine is used to quickly test large
applications where I need to ensure the presence of libraries packaged
with certain Redhat versions.

--
Joost

Re: [gentoo-user] Best filesystem for virtualized gentoo mail server - WAS: vmWare HowTo / best practices

[J. Roeleveld]

On Sat, April 20, 2013 17:33, Alan McKinnon wrote:
> On 20/04/2013 17:00, Tanstaafl wrote:
>> Thanks for the responses so far...
>>
>> Another question - are there any caveats as to which filesystem to use
>> for a mail server, for virtualized systems? Ir do the same
>> issues/questions apply (ie, does the fact that it is virtualized not
>> change anything)?
>>
>> If there are none, I'm curious what others prefer.
>>
>> I've been using reiserfs on my old mail server since it was first set up
>> (over 8 years ago). I have had no issues with it whatsoever, and even
>> had one scare with a bad UPS causing the system to experienc an unclean
>> shutdown - but it came back up, auto fsck'd, and there was no 'apparent'
>> data loss (this was a very long time ago, so if there had been any
>> serious problems, I'd have known about it long go).
>>
>> I've been considering using XFS, but have never used it before.
>>
>> So, anyway, opinions are welcome...
>
>
> Virtualization can change things, and it's not really intuitive.
>
> Regardless of what optimizations you apply to the VM, and regardless of
> what kind of virtualization is in use on the host, you are still going
> to be bound by the disk and fs behaviour of the host. If VMWare gives
> you a really shitty host driver, then something really shitty is going
> to be the best you can achieve.

This can be improved by not using file-backed disk devices.
Not sure if ESXi can do this. XenServer supports LVM-backed disk devices,
which reduces the overhead for the FS significantly.

> Disks aren't like eg NICs, you can't easily virtualize them and give the
> guest exclusive access in the style of para-virtualization (I can't
> imagine how that would even be done).

There is one option for this, but then you need to push the whole
disk-device for the VM. In other words you pass "sda" to the VM, instead
of a file or partition.
To avoid lousy I/O driver on the host, you could try passing the
disk-controller directly to the VM. But then the VM has it's own set of
disks that can not be used for other VMs.


> FWIW, I have two mail relays (no mail storage) running old postfix
> versions on FreeBSD. I expected throughput to differ when virtualized on
> ESXi, but in practice I couldn't see a difference at all - maybe the
> mail servers were very under-utilized. Considering this pair deal with
> anything between 500,000 to a million mails a day total, I would not
> have considered them "under-utilized". Just goes to show how opinions
> are often worthless but numbers buys the whiskey :-)

If Postfix only passes emails through, then it only uses the mail-spool
for temporary storage. For that, it doesn't require a lot of disk I/O.
Most filtering is done in memory, afaik.

My experience with ESXi is that it can have issues with networking when
the versions of the host don't match perfectly or the time is not synched
correctly or VMs are moved around a lot by an Admin that likes to play
with that...
Doing large multi-system installations on an ESXi cluster while the VMs
are moved around can occasionally fail because of a bad network-layer.

--
Joost

>
>
> --
> Alan McKinnon
> alan.mckinnon@gmail.com
>
>
>
>

Re: [gentoo-user] Best filesystem for virtualized gentoo mail server - WAS: vmWare HowTo / best practices

[J. Roeleveld]

On Sat, April 20, 2013 17:38, Jarry wrote:
> On 20-Apr-13 17:00, Tanstaafl wrote:
>
>> Another question - are there any caveats as to which filesystem to use
>> for a mail server, for virtualized systems? Ir do the same
>> issues/questions apply (ie, does the fact that it is virtualized not
>> change anything)?
>
> Problem of virtualized filesystem is not that it is virtualized,
> but that it is located on datastore with more virtual systems,
> all of them competing for the same i/o. *That* is the bottleneck.
> If you switch reiser for xfs or btrfs, you might win (or loose)
> a few %. If you optimize your esxi-datastore design, you might
> win much more than what you have ever dreamed of.

If the underlying I/O is fast enough with low seek-times and high
throughput, that handling multiple VMs using a lot of disk I/O
simultaneously isn't a problem. Provided the Host has sufficient resources
(think memory and dedicated CPU) to handle it.

> I have 8 VMs (out of them 6 are Gentoo) hosted on ESXi, intended
> for various tasks (mail, dns, mysql, web, etc), moderately loaded.
> I used hw-raid controller with 2x sata-hdd in raid1 but performance
> was quite dissapointing and I experienced all sorts of i/o jams.

Which hw-raid controller did you use?
RAID-1 (mirroring) isn't actually known for high performance.

> Then I switched hdd for ssd (yes I use 2 of them in raid1, even
> if this is not generally recommended) and performance rocks now!
> I can start now kernel compilation on all 6 VMs at the same time,
> with near-zero performance penalty (depending on cpu/vcpu ratio
> and number of threads used). Unthinkable with hdd-based datastore.

I have HDD-based datastores and can do this on 4 VMs (single quad-core
CPU) without any penalty.

> I would definitely recommend using SSD. Either directly as
> datastore for VMs, or at least as EXSi host-cache. There is
> also possibility of "hybrid-raid" (1xSSD and 1xHDD in raid1)
> on some raid-controllers. Or if your pocket is really deep,
> you could grab one of those FusionIO-cards to avoid being
> limited by rather slow sata-interface (SSD for PCIe)...

A decent hardware raid-controller with multiple disks running in a higher
raid version is cheaper then the same storage capacity in SSDs.

--
Joost

Re: [gentoo-user] Best filesystem for virtualized gentoo mail server - WAS: vmWare HowTo / best practices

[J. Roeleveld]

On Sat, April 20, 2013 18:22, Pandu Poluan wrote:
> On Apr 20, 2013 10:01 PM, "Tanstaafl" <tanstaafl@libertytrek.org> wrote:
>>
>> Thanks for the responses so far...
>>
>> Another question - are there any caveats as to which filesystem to use
> for a mail server, for virtualized systems? Ir do the same
> issues/questions
> apply (ie, does the fact that it is virtualized not change anything)?
>>
>> If there are none, I'm curious what others prefer.
>>
>> I've been using reiserfs on my old mail server since it was first set up
> (over 8 years ago). I have had no issues with it whatsoever, and even had
> one scare with a bad UPS causing the system to experienc an unclean
> shutdown - but it came back up, auto fsck'd, and there was no 'apparent'
> data loss (this was a very long time ago, so if there had been any serious
> problems, I'd have known about it long go).
>>
>> I've been considering using XFS, but have never used it before.
>>
>> So, anyway, opinions are welcome...
>>
>> Thanks again
>>
>> Charles
>>
>
> Reiterating what others have said, in a virtualized environment, it's how
> you build the underlying storage that will have the greatest effect on
> performance.
>
> Just an illustration: in my current employment, we have a very heavily
> used
> database (SQL Server). To ensure good performance, I dedicated a RAID
> array
> of 8 drives (15k RPM each), ensure that the space allocation is 'thick'
> not
> 'thin', and dedicate the whole RAID array to just that one VM. Performance
> went through the roof with that one... especially since it was originally
> a
> physical server running on top of 4 x 7200 RPM drives ;-)
>
> If you have the budget, you really should invest in a SAN Storage solution
> that can provide "tiered storage", in which frequently used blocks will be
> 'cached' in SSD, while less frequently used blocks are migrated first to
> slower SAS drives, and later on (if 'cold') to even slower SATA drives.

4-tier sounds nicer: 1 TB in high speed RAM for the high-speed layer, with
dedicated UPS to ensure this is backed up to disk on shutdown.

--
Joost

Re: [gentoo-user] Best filesystem for virtualized gentoo mail server - WAS: vmWare HowTo / best practices

[Pandu Poluan]

[-- Attachment #1: Type: text/plain, Size: 2654 bytes --]

On Apr 21, 2013 4:51 PM, "J. Roeleveld" <joost@antarean.org> wrote:
>
> On Sat, April 20, 2013 18:22, Pandu Poluan wrote:
> > On Apr 20, 2013 10:01 PM, "Tanstaafl" <tanstaafl@libertytrek.org> wrote:
> >>
> >> Thanks for the responses so far...
> >>
> >> Another question - are there any caveats as to which filesystem to use
> > for a mail server, for virtualized systems? Ir do the same
> > issues/questions
> > apply (ie, does the fact that it is virtualized not change anything)?
> >>
> >> If there are none, I'm curious what others prefer.
> >>
> >> I've been using reiserfs on my old mail server since it was first set
up
> > (over 8 years ago). I have had no issues with it whatsoever, and even
had
> > one scare with a bad UPS causing the system to experienc an unclean
> > shutdown - but it came back up, auto fsck'd, and there was no 'apparent'
> > data loss (this was a very long time ago, so if there had been any
serious
> > problems, I'd have known about it long go).
> >>
> >> I've been considering using XFS, but have never used it before.
> >>
> >> So, anyway, opinions are welcome...
> >>
> >> Thanks again
> >>
> >> Charles
> >>
> >
> > Reiterating what others have said, in a virtualized environment, it's
how
> > you build the underlying storage that will have the greatest effect on
> > performance.
> >
> > Just an illustration: in my current employment, we have a very heavily
> > used
> > database (SQL Server). To ensure good performance, I dedicated a RAID
> > array
> > of 8 drives (15k RPM each), ensure that the space allocation is 'thick'
> > not
> > 'thin', and dedicate the whole RAID array to just that one VM.
Performance
> > went through the roof with that one... especially since it was
originally
> > a
> > physical server running on top of 4 x 7200 RPM drives ;-)
> >
> > If you have the budget, you really should invest in a SAN Storage
solution
> > that can provide "tiered storage", in which frequently used blocks will
be
> > 'cached' in SSD, while less frequently used blocks are migrated first to
> > slower SAS drives, and later on (if 'cold') to even slower SATA drives.
>
> 4-tier sounds nicer: 1 TB in high speed RAM for the high-speed layer, with
> dedicated UPS to ensure this is backed up to disk on shutdown.
>

Indeed! But 1 TB is kind of overkill, if you ask me... :-D

VMware and XenServer can 'talk' with some Storage controllers, where they
conspire in the background to provide 'victim cache' on the virtualization
host. Not sure about Hyper-V.

I myself had had good experience relying on EMC VNX's internal 8 GB cache;
apparently the workload is not high enough to stress the system.

Rgds,
--

[-- Attachment #2: Type: text/html, Size: 3465 bytes --]

Re: [gentoo-user] Best filesystem for virtualized gentoo mail server - WAS: vmWare HowTo / best practices

[Tanstaafl]

Thanks for the reply Alan...

On 2013-04-20 11:33 AM, Alan McKinnon <alan.mckinnon@gmail.com> wrote:
> If VMWare gives you a really shitty host driver, then something
> really shitty is going to be the best you can achieve.

The host is a Dell R515 with the Perc H700.

Windows VMs see get an 'LSI Logic SAS', and my gentoo VM gets an 'LSI 
Logic Parallel' controller.

All disks are 15k rpm SAS (6G) drives.

> Disks aren't like eg NICs, you can't easily virtualize them and give the
> guest exclusive access in the style of para-virtualization (I can't
> imagine how that would even be done).
>
> You also didn't mention what mail server you use - implementations vary
> a great deal. Gut feel tells me that unless you are dealing with many
> 1000s of mails in a short period you won't really need XFS's aggressive
> caching.

Postfix+dovecot, still debating between maildir or mdbox, leaning toward 
mdbox.

Re: [gentoo-user] Best filesystem for virtualized gentoo mail server - WAS: vmWare HowTo / best practices

[Tanstaafl]

On 2013-04-21 5:47 AM, J. Roeleveld <joost@antarean.org> wrote:
> On Sat, April 20, 2013 17:38, Jarry wrote:
>> Problem of virtualized filesystem is not that it is virtualized,
>> but that it is located on datastore with more virtual systems,
>> all of them competing for the same i/o. *That* is the bottleneck.
>> If you switch reiser for xfs or btrfs, you might win (or loose)
>> a few %. If you optimize your esxi-datastore design, you might
>> win much more than what you have ever dreamed of.
>
> If the underlying I/O is fast enough with low seek-times and high
> throughput, that handling multiple VMs using a lot of disk I/O
> simultaneously isn't a problem. Provided the Host has sufficient resources
> (think memory and dedicated CPU) to handle it.

My host specs:

Dual AMD Opteron 4180 (6-core, 2.6Ghz)
128GB RAM
2x internal SSDs in RAID1 for Host OS
6x 300G SAS 6Gb 15k hard drives in RAID10 for Guest OSs

I allocate each Guest 1 virtual CPU with 2 cores

> A decent hardware raid-controller with multiple disks running in a higher
> raid version is cheaper then the same storage capacity in SSDs.

Yep... I toyed with the idea of SSDs, but the cost was considerably more 
as compared to even these SAS drives...

[gentoo-user] LVM on VM or not? - WAS Re: Best filesystem for virtualized gentoo mail server - WAS: vmWare HowTo / best practices

[Tanstaafl]

On 2013-04-20 11:00 AM, Tanstaafl <tanstaafl@libertytrek.org> wrote:
> Another question - are there any caveats as to which filesystem to use
> for a mail server, for virtualized systems?

Ok, googling reveals lots of conflicting opinions about using LVM in a 
VM environment.

I was wanting to use it mainly for its snapshot ability (to get 
consistent backups of my mailstore and mysql DBs).

Also it would be very nice to be able to resize things if needed (I have 
adequate storage available).

But I've found lots of opinions that using LVM in a virtualized 
environment can lead to data corruption, and if this is true, I'd rather 
not risk it...

So, LVM or not?

Re: [gentoo-user] LVM on VM or not? - WAS Re: Best filesystem for virtualized gentoo mail server - WAS: vmWare HowTo / best practices

[Jarry]

On 21-Apr-13 18:15, Tanstaafl wrote:
>
> Ok, googling reveals lots of conflicting opinions about using LVM in a
> VM environment.
>
> I was wanting to use it mainly for its snapshot ability (to get
> consistent backups of my mailstore and mysql DBs).
>
> Also it would be very nice to be able to resize things if needed (I have
> adequate storage available).
>
> But I've found lots of opinions that using LVM in a virtualized
> environment can lead to data corruption, and if this is true, I'd rather
> not risk it...
>
> So, LVM or not?

You can make snapshots from ESXi (btw snapshot is *not* backup),
and you can resize VM-disks as well. So the right question is:
What are the LVM features I need? If I do not need any, then why
should I bother with it?

Jarry

-- 
_______________________________________________________________
This mailbox accepts e-mails only from selected mailing-lists!
Everything else is considered to be spam and therefore deleted.

Re: [gentoo-user] LVM on VM or not? - WAS Re: Best filesystem for virtualized gentoo mail server - WAS: vmWare HowTo / best practices

[Randy Barlow]

On Sun, 2013-04-21 at 12:15 -0400, Tanstaafl wrote:
> But I've found lots of opinions that using LVM in a virtualized 
> environment can lead to data corruption, and if this is true, I'd rather 
> not risk it...
> 
> So, LVM or not?

This is surprising to me, because at my former employer we used LVM for
all of our virtual machines in Xen. It performed quite well, and we
never experienced any data loss due to that setup. LVM gives a lot of
flexibility in managing virtual machines, so I'd highly recommend it.

I believe LVM devices can be passed to KVM guests as well, though I have
never personally tried that.

-- 
Randy Barlow

Re: [gentoo-user] LVM on VM or not? - WAS Re: Best filesystem for virtualized gentoo mail server - WAS: vmWare HowTo / best practices

[Randy Barlow]

On Sun, 2013-04-21 at 12:32 -0400, Randy Barlow wrote:
> LVM gives a lot of
> flexibility in managing virtual machines, so I'd highly recommend it.

I should mention one specific advantage to using LVM over file-based
images: I believe you will find that LVM performs better. This is due to
avoiding the duplicated filesystem overhead that would occur in the
file-based image approach. If the guest wants to fsync(), for example,
both filesystems need to be involved (the guest's, and the host's). With
LVM, you still have the host processing the LVM bits of that process,
but at least the host's filesystem doesn't need to be involved.

Of course, giving the guest it's own raw block device (a disk, or a
partition) would also have this advantage, so here I'm mostly just
comparing LVM to the "easy" file-backed disk image approach.

-- 
Randy Barlow

[gentoo-user] Partition layout questions - WAS Re: Best filesystem for virtualized gentoo mail server - WAS: vmWare HowTo / best practices

[Tanstaafl]

Next, as to partition layout.

I was considering this partition layout:

/boot (ext2), 100M
/swap, 2048G
/ (ext4), 40G
/tmp (ext2), 2G
/var (xfs), 600G

But doing some reading, I stumbled on some other suggestions, like:

Bind /tmp to tmpfs, ie:

tmpfs   /tmp         tmpfs   nodev,nosuid                  0  0

Then I read another suggestion to bind /var/tmp to /tmp:

/tmp /var/tmp none rw,noexec,nosuid,nodev,bind 0 0

Which means that both /tmp and /var/tmp are now bound to /tmp?

But, I also read one one of these pages that tmpfs should NOT be used 
for /var/tmp, because it stores files that need to be persistent across 
reboots - is this still true?

My main concerns are security (which dirs should be separate so they can 
be mounted as securely as possible, is, nodev noexec and nosuid mount 
options)?

1. Should I go ahead and make separate smallish (maybe 1 or 2GB) /home 
so I can mount it nodev,noexec,nosuid?

2. Should I make a separate partition for /var/tmp so I can mount it as 
nodev,noexec,nosuid, and bind /tmp to /tmpfs as above? Or does the 
caveat about /var/tmp storing files that need to be persistent across 
reboots no longer apply, and I can bind them both to tmpfs?

3. Dumb question (google didn't give me an answer) - can I mount all of 
/var noexec and nosuid? Assuming not...

4. Since I'm running dovecot with a single user (vmail), and dovecot 
stores sieve scripts in the users 'home' dir, does this mean I can't 
mount that directory with nodev noexec and/or nosuid?

5. Webapps... can I mount the dir where these are installed with 
nodev,noexec,nosuid (I still use webapp-config to manage my website 
installations, and currently these are in /var/www)?

I'm thinking an alternative would be to put all data that can be stored 
on a partition that is mounted nodev,noexec,nosuid, ie:

/virtual

which would contain:

/virtual/home
/virtual/mail
/virtual/www

Maybe I'm overthinking/overcomplicating this, but obviously now is the 
time to make these decisions...

So, comments/criticisms welcome as always...

Re: [gentoo-user] LVM on VM or not? - WAS Re: Best filesystem for virtualized gentoo mail server - WAS: vmWare HowTo / best practices

[Tanstaafl]

On 2013-04-21 12:27 PM, Jarry <mr.jarry@gmail.com> wrote:
> On 21-Apr-13 18:15, Tanstaafl wrote:
>>
>> Ok, googling reveals lots of conflicting opinions about using LVM in a
>> VM environment.
>>
>> I was wanting to use it mainly for its snapshot ability (to get
>> consistent backups of my mailstore and mysql DBs).
>>
>> Also it would be very nice to be able to resize things if needed (I have
>> adequate storage available).
>>
>> But I've found lots of opinions that using LVM in a virtualized
>> environment can lead to data corruption, and if this is true, I'd rather
>> not risk it...
>>
>> So, LVM or not?
>
> You can make snapshots from ESXi (btw snapshot is *not* backup),
> and you can resize VM-disks as well. So the right question is:
> What are the LVM features I need? If I do not need any, then why
> should I bother with it?

Yes I can't take snapshots with ESXi, but everything I've read says that 
for these to be consistent, they need to be done when the VM is shutdown.

Also, they take a LONG time, whereas an LVM snapshot happens almost 
immediately.

Re: [gentoo-user] LVM on VM or not? - WAS Re: Best filesystem for virtualized gentoo mail server - WAS: vmWare HowTo / best practices

[Tanstaafl]

On 2013-04-21 12:56 PM, Tanstaafl <tanstaafl@libertytrek.org> wrote:
> Yes I can't take snapshots with ESXi,

Sorry, should have read 'Yes I can take snapshots with ESXi...'

Re: [gentoo-user] LVM on VM or not? - WAS Re: Best filesystem for virtualized gentoo mail server - WAS: vmWare HowTo / best practices

[Michael Hampicke]

Am 21.04.2013 18:32, schrieb Randy Barlow:
> On Sun, 2013-04-21 at 12:15 -0400, Tanstaafl wrote:
>> But I've found lots of opinions that using LVM in a virtualized
>> environment can lead to data corruption, and if this is true, I'd 
>> rather
>> not risk it...
>> 
>> So, LVM or not?
> 
> This is surprising to me, because at my former employer we used LVM 
> for
> all of our virtual machines in Xen. It performed quite well, and we
> never experienced any data loss due to that setup. LVM gives a lot of
> flexibility in managing virtual machines, so I'd highly recommend it.
> 
> I believe LVM devices can be passed to KVM guests as well, though I 
> have
> never personally tried that.

Correct, in almost every case where I set up a VM Host, I use LVM to 
manage the "disks" that the guests will get.

Re: [gentoo-user] LVM on VM or not? - WAS Re: Best filesystem for virtualized gentoo mail server - WAS: vmWare HowTo / best practices

[Tanstaafl]

On 2013-04-21 12:38 PM, Randy Barlow <randy@electronsweatshop.com> wrote:
> I should mention one specific advantage to using LVM over file-based
> images: I believe you will find that LVM performs better. This is due to
> avoiding the duplicated filesystem overhead that would occur in the
> file-based image approach. If the guest wants to fsync(), for example,
> both filesystems need to be involved (the guest's, and the host's). With
> LVM, you still have the host processing the LVM bits of that process,

???

This doesn't make sense to me.

Unless you're talking about using LVM on the HOST.

I'm not. I didn't specify this in this particular post, but I'm using 
vmWare ESXi, and installing a gentoo VM to run on it.

So, I'm asking about using the LVM2 installation manual from Gentoo and 
using LVM2 for just my gentoo VM...

So, in this case, is it still recommended/fully supported/safe?

Thanks

Re: [gentoo-user] Best filesystem for virtualized gentoo mail server - WAS: vmWare HowTo / best practices

[Alan McKinnon]

On 21/04/2013 16:33, Tanstaafl wrote:
> Thanks for the reply Alan...
> 
> On 2013-04-20 11:33 AM, Alan McKinnon <alan.mckinnon@gmail.com> wrote:
>> If VMWare gives you a really shitty host driver, then something
>> really shitty is going to be the best you can achieve.
> 
> The host is a Dell R515 with the Perc H700.
> 
> Windows VMs see get an 'LSI Logic SAS', and my gentoo VM gets an 'LSI
> Logic Parallel' controller.
> 
> All disks are 15k rpm SAS (6G) drives.

That's very similar to our setups with the exception of the host
hardware - we have R700 series hosts.

I'd be interested to keep track of what you find out and feed that back
to the sysadmins

> 
>> Disks aren't like eg NICs, you can't easily virtualize them and give the
>> guest exclusive access in the style of para-virtualization (I can't
>> imagine how that would even be done).
>>
>> You also didn't mention what mail server you use - implementations vary
>> a great deal. Gut feel tells me that unless you are dealing with many
>> 1000s of mails in a short period you won't really need XFS's aggressive
>> caching.
> 
> Postfix+dovecot, still debating between maildir or mdbox, leaning toward
> mdbox.

I always found postfix to be the best choice for an MTA in the large
picture (mostly because everyone else can also figure out what's going
on). I'd also be testing mdbox first just because maildir always leaves
me feeling a bit like there's a little too much disk IO going on and it
could be better


-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] LVM on VM or not? - WAS Re: Best filesystem for virtualized gentoo mail server - WAS: vmWare HowTo / best practices

[Pandu Poluan]

[-- Attachment #1: Type: text/plain, Size: 1435 bytes --]

On Apr 22, 2013 2:05 AM, "Tanstaafl" <tanstaafl@libertytrek.org> wrote:
>
> On 2013-04-21 12:38 PM, Randy Barlow <randy@electronsweatshop.com> wrote:
>>
>> I should mention one specific advantage to using LVM over file-based
>> images: I believe you will find that LVM performs better. This is due to
>> avoiding the duplicated filesystem overhead that would occur in the
>> file-based image approach. If the guest wants to fsync(), for example,
>> both filesystems need to be involved (the guest's, and the host's). With
>> LVM, you still have the host processing the LVM bits of that process,
>
>
> ???
>
> This doesn't make sense to me.
>
> Unless you're talking about using LVM on the HOST.
>
> I'm not. I didn't specify this in this particular post, but I'm using
vmWare ESXi, and installing a gentoo VM to run on it.
>
> So, I'm asking about using the LVM2 installation manual from Gentoo and
using LVM2 for just my gentoo VM...
>
> So, in this case, is it still recommended/fully supported/safe?
>
> Thanks
>

Honestly, I don't see how LVM can interact with VMware's VMDK... unless one
use VMware's thin provisioning over a SAN Storage Thin Provisioning, in
which case all hell will break loose once the actual disk size is reached...

Stick with VMware Thin Provisioning XOR SAN Storage Thin Provisioning.
Never both.

One thing you have to think about, is whether to implement
LVM/partition-less, or LVM/partitions.

Rgds,
--

[-- Attachment #2: Type: text/html, Size: 1839 bytes --]

Re: [gentoo-user] LVM on VM or not? - WAS Re: Best filesystem for virtualized gentoo mail server - WAS: vmWare HowTo / best practices

[Randy Barlow]

On Sun, 2013-04-21 at 15:04 -0400, Tanstaafl wrote:
> ???
> 
> This doesn't make sense to me.
> 
> Unless you're talking about using LVM on the HOST.

Ah, apologies, I think I had misunderstood. Given that you are using
ESXi, I should have thought that LVM on the host wouldn't be possible.
Are you set on using ESXi? If not, I think there are some compelling
advantages to some of the open source solutions. If so, you are correct
that my suggestion about performance advantages wouldn't apply to you.

> I'm not. I didn't specify this in this particular post, but I'm using 
> vmWare ESXi, and installing a gentoo VM to run on it.
> 
> So, I'm asking about using the LVM2 installation manual from Gentoo
> and 
> using LVM2 for just my gentoo VM...
> 
> So, in this case, is it still recommended/fully supported/safe?

In this case, I think you may lose some of the LVM advantages. Assuming
that a volume resize in ESXi is pretty easy, you won't need the guest to
use LVM to take advantage of it.

Here's one though, and I'm not sure the answer: can you easily mount an
ESXi snapshot somewhere, and make a real backup from it? I use LVM to
snapshot my FS, mount that snapshot somewhere, and then I have my backup
software back that up instead of the live system. This has the advantage
of allowing the backup software to have a consistent view of the disk as
it does its work, and gives you sort of a "crash consistent" backup. If
you can still do that with ESXi, that's about the only other advantage.

As to whether its supported, I'm not an expert of ESXi (so don't count
me as an expert), but as far as I know, there should be nothing stopping
you from using LVM in the guests. LVM just works with block devices, and
the ESXi disks should work like block devices, so everything should be
fine. Again, I've never used VMWare very extensively, so this isn't
coming from experience.

-- 
Randy Barlow

Re: [gentoo-user] LVM on VM or not? - WAS Re: Best filesystem for virtualized gentoo mail server - WAS: vmWare HowTo / best practices

[Randy Barlow]

On Sun, 2013-04-21 at 12:56 -0400, Tanstaafl wrote:
> Yes I can't take snapshots with ESXi, but everything I've read says
> that 
> for these to be consistent, they need to be done when the VM is
> shutdown.
> 
> Also, they take a LONG time, whereas an LVM snapshot happens almost 
> immediately.

This reveals a significant advantage to using LVM, if you wish to use
snapshots as the target for backups. I personally prefer this, as I
mentioned in another e-mail in this thread, due to getting a "crash
consistent" view of the disk since it will not change while the backup
is in progress. The fact that LVM snapshots happen instantly is an
advantage, and also that you can perform the snapshots while the system
is running.

-- 
Randy Barlow

Re: [gentoo-user] Best filesystem for virtualized gentoo mail server - WAS: vmWare HowTo / best practices

[J. Roeleveld]

On Sun, April 21, 2013 17:09, Tanstaafl wrote:
> On 2013-04-21 5:47 AM, J. Roeleveld <joost@antarean.org> wrote:
>> On Sat, April 20, 2013 17:38, Jarry wrote:
>>> Problem of virtualized filesystem is not that it is virtualized,
>>> but that it is located on datastore with more virtual systems,
>>> all of them competing for the same i/o. *That* is the bottleneck.
>>> If you switch reiser for xfs or btrfs, you might win (or loose)
>>> a few %. If you optimize your esxi-datastore design, you might
>>> win much more than what you have ever dreamed of.
>>
>> If the underlying I/O is fast enough with low seek-times and high
>> throughput, that handling multiple VMs using a lot of disk I/O
>> simultaneously isn't a problem. Provided the Host has sufficient
>> resources
>> (think memory and dedicated CPU) to handle it.
>
> My host specs:
>
> Dual AMD Opteron 4180 (6-core, 2.6Ghz)
> 128GB RAM
> 2x internal SSDs in RAID1 for Host OS
> 6x 300G SAS 6Gb 15k hard drives in RAID10 for Guest OSs

Sounds like a nice machine for testing :)

> I allocate each Guest 1 virtual CPU with 2 cores

Do you limit the Guest to use any of 2 specific cores? Or are you giving 2
vCPUs to each Guest?

>> A decent hardware raid-controller with multiple disks running in a
>> higher
>> raid version is cheaper then the same storage capacity in SSDs.
>
> Yep... I toyed with the idea of SSDs, but the cost was considerably more
> as compared to even these SAS drives...

I am planning on using SSDs when getting new desktops, but for servers I
prefer spinning disks. They're higher capacity and cheaper.
For speed, I just put a bunch of them together with hardware raid.

--
Joost

Re: [gentoo-user] Best filesystem for virtualized gentoo mail server - WAS: vmWare HowTo / best practices

[J. Roeleveld]

On Sun, April 21, 2013 12:06, Pandu Poluan wrote:
> On Apr 21, 2013 4:51 PM, "J. Roeleveld" <joost@antarean.org> wrote:
>>
>> On Sat, April 20, 2013 18:22, Pandu Poluan wrote:
> > > If you have the budget, you really should invest in a SAN
> > > Storage solution that can provide "tiered storage", in which
> > > frequently used blocks will be 'cached' in SSD, while less
> > > frequently used blocks are migrated first to slower SAS
> > > drives, and later on (if 'cold') to even slower SATA
> > > drives.
> >
> > 4-tier sounds nicer: 1 TB in high speed RAM for the high-speed
> > layer, with dedicated UPS to ensure this is backed up to disk
> > on shutdown.
>
> Indeed! But 1 TB is kind of overkill, if you ask me... :-D

Maybe, but when using that for VMs in a lab environment where you want to
create snapshots quickly.

> VMware and XenServer can 'talk' with some Storage controllers, where they
> conspire in the background to provide 'victim cache' on the virtualization
> host. Not sure about Hyper-V.
>
> I myself had had good experience relying on EMC VNX's internal 8 GB cache;
> apparently the workload is not high enough to stress the system.

When the time comes to upgrade the hardware, I will look into that. By
then, this technology should be more common as well.

--
Joost

Re: [gentoo-user] Best filesystem for virtualized gentoo mail server - WAS: vmWare HowTo / best practices

[Andre Lucas Falco]

[-- Attachment #1: Type: text/plain, Size: 238 bytes --]

2013/4/21 Tanstaafl <tanstaafl@libertytrek.org>

> Windows VMs see get an 'LSI Logic SAS', and my gentoo VM gets an 'LSI
> Logic Parallel' controller.
>


Did you tested using pvscsi? It's improve performance with less cost to CPU
usage.

[-- Attachment #2: Type: text/html, Size: 618 bytes --]

Re: [gentoo-user] LVM on VM or not? - WAS Re: Best filesystem for virtualized gentoo mail server - WAS: vmWare HowTo / best practices

[Tanstaafl]

On 2013-04-21 6:15 PM, Pandu Poluan <pandu@poluan.info> wrote:
> One thing you have to think about, is whether to implement
> LVM/partition-less, or LVM/partitions.

Well, I was going to just allocate a new 'drive' (create one within 
vmWare and then attach it to my gentoo VM), and use that entire virtual 
'disk' for LVM... is that what you mean by 'partition-less'?

Hadn't really thought about the differences much... what are the 
advantages or disadvantages to using partitions? Seems like it would be 
more flexible without partitions?

PVSCSI vs LSI Logic Parallel/SAS - WAS: Re: [gentoo-user] Best filesystem for virtualized gentoo mail server - WAS: vmWare HowTo / best practices

[Tanstaafl]

On 2013-04-22 8:56 AM, Andre Lucas Falco <alfalco@gmail.com> wrote:
> 2013/4/21 Tanstaafl <tanstaafl@libertytrek.org> wrote:
>> Windows VMs see get an 'LSI Logic SAS', and my gentoo VM gets an
>> 'LSI Logic Parallel' controller.

> Did you tested using pvscsi? It's improve performance with less cost to
> CPU usage.

No, I didn't...

It appears there is pvscsi support in the mainline linux kernel, but is 
it rock-solid? Anyone else here running gentoo linux with this driver 
for their primary/boot disk controller?

Also, for my windows server 2008r2 vms, I used the default, which was 
the LSI SAS... I did search and found the knowledgebase article 
describing how to change them, but is the gain really worth the trouble 
(and more importantly, the risk)?

Re: PVSCSI vs LSI Logic Parallel/SAS - WAS: Re: [gentoo-user] Best filesystem for virtualized gentoo mail server - WAS: vmWare HowTo / best practices

[Tanstaafl]

Anyone?

On 2013-04-23 3:28 PM, Tanstaafl <tanstaafl@libertytrek.org> wrote:
> On 2013-04-22 8:56 AM, Andre Lucas Falco <alfalco@gmail.com> wrote:
>> 2013/4/21 Tanstaafl <tanstaafl@libertytrek.org> wrote:
>>> Windows VMs see get an 'LSI Logic SAS', and my gentoo VM gets an
>>> 'LSI Logic Parallel' controller.
>
>> Did you tested using pvscsi? It's improve performance with less cost to
>> CPU usage.
>
> No, I didn't...
>
> It appears there is pvscsi support in the mainline linux kernel, but is
> it rock-solid? Anyone else here running gentoo linux with this driver
> for their primary/boot disk controller?
>
> Also, for my windows server 2008r2 vms, I used the default, which was
> the LSI SAS... I did search and found the knowledgebase article
> describing how to change them, but is the gain really worth the trouble
> (and more importantly, the risk)?
>

Re: PVSCSI vs LSI Logic Parallel/SAS - WAS: Re: [gentoo-user] Best filesystem for virtualized gentoo mail server - WAS: vmWare HowTo / best practices

[Pandu Poluan]

[-- Attachment #1: Type: text/plain, Size: 1125 bytes --]

On Apr 24, 2013 2:29 AM, "Tanstaafl" <tanstaafl@libertytrek.org> wrote:
>
> On 2013-04-22 8:56 AM, Andre Lucas Falco <alfalco@gmail.com> wrote:
>>
>> 2013/4/21 Tanstaafl <tanstaafl@libertytrek.org> wrote:
>>>
>>> Windows VMs see get an 'LSI Logic SAS', and my gentoo VM gets an
>>> 'LSI Logic Parallel' controller.
>
>
>> Did you tested using pvscsi? It's improve performance with less cost to
>> CPU usage.
>
>
> No, I didn't...
>
> It appears there is pvscsi support in the mainline linux kernel, but is
it rock-solid? Anyone else here running gentoo linux with this driver for
their primary/boot disk controller?
>
> Also, for my windows server 2008r2 vms, I used the default, which was the
LSI SAS... I did search and found the knowledgebase article describing how
to change them, but is the gain really worth the trouble (and more
importantly, the risk)?
>

My Gentoo VMs in the cloud (using VMware's vCloud) uses PV-SCSI. It's
stable... but kind of sensitive: Everytime the cloud provider do something
with their storage, my VMs become Read-Only.

Other than that, performance is good, no fs corruption, etc.

Rgds,
--

[-- Attachment #2: Type: text/html, Size: 1526 bytes --]

Re: PVSCSI vs LSI Logic Parallel/SAS - WAS: Re: [gentoo-user] Best filesystem for virtualized gentoo mail server - WAS: vmWare HowTo / best practices

[Tanstaafl]

On 2013-04-24 10:23 PM, Pandu Poluan <pandu@poluan.info> wrote:
> My Gentoo VMs in the cloud (using VMware's vCloud) uses PV-SCSI. It's
> stable... but kind of sensitive: Everytime the cloud provider do
> something with their storage, my VMs become Read-Only.

Ouch... so, how do you fix it?

Re: PVSCSI vs LSI Logic Parallel/SAS - WAS: Re: [gentoo-user] Best filesystem for virtualized gentoo mail server - WAS: vmWare HowTo / best practices

[Pandu Poluan]

[-- Attachment #1: Type: text/plain, Size: 1404 bytes --]

On Apr 25, 2013 5:54 PM, "Tanstaafl" <tanstaafl@libertytrek.org> wrote:
>
> On 2013-04-24 10:23 PM, Pandu Poluan <pandu@poluan.info> wrote:
>>
>> My Gentoo VMs in the cloud (using VMware's vCloud) uses PV-SCSI. It's
>> stable... but kind of sensitive: Everytime the cloud provider do
>> something with their storage, my VMs become Read-Only.
>
>
> Ouch... so, how do you fix it?
>

For the Read-Only problem, a simple reboot suffices.

But I made sure to fire an angry email to the Customer Support, telling
them in no uncertain terms that the next time they want to do something re:
their storage, they contact me first.

So, when the time comes for then to do some storage management things, they
tell me exactly at what time. I scheduled the VMs to shut down at the
agreed time, and turn them back on as soon as they text me that their
maintenance had finished. Only one VM I left on, the "gatewall" VM. And
every single time the gatewall VM detected that something is being done on
the storage level, and changed to a Read-Only mode.

I consider myself lucky that when the 'problem' manifested itself, the VMs
are not yet in Production. And no corruption happened.

(PS: Strangely enough, the RO switch happens only on Gentoo VMs; the
FreeBSD VMs and Debian VMs are unaffected. Maybe because I've pared down
each and every Gentoo VM to their bare minimum, so they are much more
responsive)

Rgds,
--

[-- Attachment #2: Type: text/html, Size: 1705 bytes --]

Re: PVSCSI vs LSI Logic Parallel/SAS - WAS: Re: [gentoo-user] Best filesystem for virtualized gentoo mail server - WAS: vmWare HowTo / best practices

[Tanstaafl]

On 2013-04-25 7:44 AM, Pandu Poluan <pandu@poluan.info> wrote:
> On Apr 25, 2013 5:54 PM, "Tanstaafl" wrote:
>> On 2013-04-24 10:23 PM, Pandu Poluan wrote:
>>> My Gentoo VMs in the cloud (using VMware's vCloud) uses PV-SCSI. It's
>>> stable... but kind of sensitive: Everytime the cloud provider do
>>> something with their storage, my VMs become Read-Only.

>> Ouch... so, how do you fix it?

> For the Read-Only problem, a simple reboot suffices.
>
> But I made sure to fire an angry email to the Customer Support, telling
> them in no uncertain terms that the next time they want to do something
> re: their storage, they contact me first.

Interesting... thanks for the heads up, but since this is on my own 
host, I won't need to worry about this.

Re: [gentoo-user] vmWare HowTo / best practices

[Tanstaafl]

On 2013-04-19 11:22 AM, Jarry <mr.jarry@gmail.com> wrote:
> vmware-tools: I have tested open-vm-tools but now I'm running
> my VMs without them because every kernel upgrade was a real
> pain in a**. And trully I did not see any benefit in running
> vm-tools (maybe it would be different on desktop). For
> shutdown of Gentoo-VMs from ESXi I use ssh-script or
> hibernation.

Hi Jarry,

I missed the significance of this (didn't realize that the vmware tools 
required modules to be enabled...

Can you elaborate on how you execute safe shutdowns of your gentoo vms 
from the ESXi host?

Thanks!

Re: [gentoo-user] vmWare HowTo / best practices

[Tanstaafl]

On 2013-04-19 11:22 AM, Jarry <mr.jarry@gmail.com> wrote:
> vmware-tools: I have tested open-vm-tools but now I'm running
> my VMs without them because every kernel upgrade was a real
> pain in a**.

Jarry - can you elaborate on this?

I'm trying to see why updating the kernel would be a problem, as long as 
you keep the same config options?