[gentoo-user] installing Gentoo as a xen guest

[gentoo-user] installing Gentoo as a xen guest

[hw]

Hi,

I'm installing Gentoo as a xen PV guest.  Do I need to install a 
bootloader like grub, or should I rather just specify the kernel to boot 
in the definition file of the guest?  If I do the latter, what about the 
kernel command line?

Is there anything I should consider?  The host is also running Gentoo.

Re: [gentoo-user] installing Gentoo as a xen guest

[Rich Freeman]

On Thu, Sep 24, 2015 at 6:05 AM, hw <hw@gc-24.de> wrote:
>
> Hi,
>
> I'm installing Gentoo as a xen PV guest.  Do I need to install a bootloader
> like grub, or should I rather just specify the kernel to boot in the
> definition file of the guest?  If I do the latter, what about the kernel
> command line?
>
> Is there anything I should consider?  The host is also running Gentoo.
>

You can do either.  I suspect it would be easier to just use grub.  I
don't know if xen provides a way to provide a command-line, if not you
could build a default command-line into your kernel.

I haven't messed with it recently, but some of my docs on running
Gentoo on EC2 might be useful, since EC2 is ultimately running xen
guests.  Other than using the right kernel settings for xen support in
your guest there really isn't much needed to get it working with
Gentoo.  Most of my write-ups were about the Amazon process (creating
disk images, configuring grub.cfg for Amazon's grub and device
numbering, etc).

-- 
Rich

Re: [gentoo-user] installing Gentoo as a xen guest

[J. Roeleveld]

On 24 September 2015 14:53:07 CEST, Rich Freeman <rich0@gentoo.org> wrote:
>On Thu, Sep 24, 2015 at 6:05 AM, hw <hw@gc-24.de> wrote:
>>
>> Hi,
>>
>> I'm installing Gentoo as a xen PV guest.  Do I need to install a
>bootloader
>> like grub, or should I rather just specify the kernel to boot in the
>> definition file of the guest?  If I do the latter, what about the
>kernel
>> command line?
>>
>> Is there anything I should consider?  The host is also running
>Gentoo.
>>
>
>You can do either.  I suspect it would be easier to just use grub.  I
>don't know if xen provides a way to provide a command-line, if not you
>could build a default command-line into your kernel.

For PV, grub is actually more work to get working. There is a config option for the commandline.
I will send one of mine later today.

>I haven't messed with it recently, but some of my docs on running
>Gentoo on EC2 might be useful, since EC2 is ultimately running xen
>guests.  Other than using the right kernel settings for xen support in
>your guest there really isn't much needed to get it working with
>Gentoo.  Most of my write-ups were about the Amazon process (creating
>disk images, configuring grub.cfg for Amazon's grub and device
>numbering, etc).

Does EC2 actually provide PV guests?
With PV, the guest knows it's a guest and communicates with Xen. Non-PV has an emulation layer (qemu) running on the host that hides the virtualisation from the guest.
Special drivers on the guest can help with performance, but isn't necessary to get it to work.

--
Joost


-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [gentoo-user] installing Gentoo as a xen guest

[Rich Freeman]

On Thu, Sep 24, 2015 at 9:22 AM, J. Roeleveld <joost@antarean.org> wrote:
>
> For PV, grub is actually more work to get working. There is a config option for the commandline.
> I will send one of mine later today.

I can believe that.  My only experience is with Amazon, which doesn't
give you any control over the host xen.  It just runs grub with a
grub.cfg you provide if you want to run your own kernel (unless that
has changed).

>
> Does EC2 actually provide PV guests?
> With PV, the guest knows it's a guest and communicates with Xen. Non-PV has an emulation layer (qemu) running on the host that hides the virtualisation from the guest.
> Special drivers on the guest can help with performance, but isn't necessary to get it to work.

I believe that EC2 ONLY provides PV guests.  I don't believe it will
do full virtualization for linux guests.  They do provide windows
guests, and I'm not sure of the details of how that is done.  If you
want to run a linux guest you either use one of their kernels, or you
can run your own as long as it supports Xen PV.

-- 
Rich

Re: [gentoo-user] installing Gentoo as a xen guest

[J. Roeleveld]

On Thursday, September 24, 2015 09:45:09 AM Rich Freeman wrote:
> On Thu, Sep 24, 2015 at 9:22 AM, J. Roeleveld <joost@antarean.org> wrote:
> > For PV, grub is actually more work to get working. There is a config
> > option for the commandline. I will send one of mine later today.

Please see email from Håkon Alstadheim for the example.

> I can believe that.  My only experience is with Amazon, which doesn't
> give you any control over the host xen.  It just runs grub with a
> grub.cfg you provide if you want to run your own kernel (unless that
> has changed).

Sounds like pvgrub. I never looked into setting that up, afaiui, it mounts the 
guest filesystem, grabs the grub.cfg, grabs the kernel listed there, umounts, 
then boots the guest.

> > Does EC2 actually provide PV guests?
> > With PV, the guest knows it's a guest and communicates with Xen. Non-PV
> > has an emulation layer (qemu) running on the host that hides the
> > virtualisation from the guest. Special drivers on the guest can help with
> > performance, but isn't necessary to get it to work.
> I believe that EC2 ONLY provides PV guests.  I don't believe it will
> do full virtualization for linux guests.  They do provide windows
> guests, and I'm not sure of the details of how that is done.  If you
> want to run a linux guest you either use one of their kernels, or you
> can run your own as long as it supports Xen PV.

Windows guests will run in PVH mode as I doubt Microsoft has a PV-enabled 
kernel available for Amazon :)

--
Joost

Re: [gentoo-user] installing Gentoo as a xen guest

[Rich Freeman]

On Thu, Sep 24, 2015 at 9:50 AM, J. Roeleveld <joost@antarean.org> wrote:
>
> Sounds like pvgrub. I never looked into setting that up, afaiui, it mounts the
> guest filesystem, grabs the grub.cfg, grabs the kernel listed there, umounts,
> then boots the guest.
>

Indeed, that is exactly the behavior Amazon has, and I'm pretty sure
their "logs" show pvgrub in the output.

From their standpoint it probably gives them a really easy way to make
guest filesystems self-contained without the need to tweak the
hypervisor environment to make each guest run.


-- 
Rich

Re: [gentoo-user] installing Gentoo as a xen guest

[Håkon Alstadheim]

Den 24. sep. 2015 14:53, skrev Rich Freeman:
> On Thu, Sep 24, 2015 at 6:05 AM, hw <hw@gc-24.de> wrote:
>> Hi,
>>
>> I'm installing Gentoo as a xen PV guest.  Do I need to install a bootloader
>> like grub, or should I rather just specify the kernel to boot in the
>> definition file of the guest?  If I do the latter, what about the kernel
>> command line?
>>
>> Is there anything I should consider?  The host is also running Gentoo.
>>
> You can do either.  I suspect it would be easier to just use grub.  I
> don't know if xen provides a way to provide a command-line, if not you
> could build a default command-line into your kernel.
>
Depending on your hardware you will want to use hvm or pvm for
efficiency. (VT-x means hvm is more efficient). If running hvm on
quemu-xen-traditional, you HAVE to use a bootloader inside the VM, or
some kind of netboot/pvgrub thing. If running upstream quemu for a hvm,
you can choose. I find it less of a hassle to use bootloader inside the
VM. If running pv, I BELIEVE you have to specify kernel and boot options
outside the VM-image, possibly through pvgrub. Pvgrub is supposed to be
able to fish out kernel and initrd from the VM disk. I never bothered to
get that working. Here is an example excerpt from a PVM I use (for a
debian vm):
--print.pvm---
builder = "generic"
kernel = "/etc/xen/wheezy/vmlinuz-3.16.0-0.bpo.4-amd64"
ramdisk = "/etc/xen/wheezy/initrd.img-3.16.0-0.bpo.4-amd64"
extra = "root=/dev/xvda console=hvc0 tmem"
--
For explanation check "man xl.cfg". When you specify
kernel&ramdisk&parameters in the vm config, you do NOT need a /boot/
directory on your vm.  Obviously when you build a kernel inside the vm,
it will end up in /boot, so you have to provide a way for the
dom0/hypervisor to load your kernel, either (when using full
virtualization) a regular boot-loader inside the vm, or (for a PV guest)
through some other means to let the dom0/hypervisor load your image.

Re: [gentoo-user] installing Gentoo as a xen guest

[J. Roeleveld]

On Thursday, September 24, 2015 03:33:28 PM Håkon Alstadheim wrote:
> Den 24. sep. 2015 14:53, skrev Rich Freeman:
> > On Thu, Sep 24, 2015 at 6:05 AM, hw <hw@gc-24.de> wrote:
> >> Hi,
> >> 
> >> I'm installing Gentoo as a xen PV guest.  Do I need to install a
> >> bootloader
> >> like grub, or should I rather just specify the kernel to boot in the
> >> definition file of the guest?  If I do the latter, what about the kernel
> >> command line?
> >> 
> >> Is there anything I should consider?  The host is also running Gentoo.
> > 
> > You can do either.  I suspect it would be easier to just use grub.  I
> > don't know if xen provides a way to provide a command-line, if not you
> > could build a default command-line into your kernel.
> 
> Depending on your hardware you will want to use hvm or pvm for
> efficiency. (VT-x means hvm is more efficient).

What do you base this on?
Without VT-x, HVM doesn't even work, which means PV is only option.

With VT-x, PV still has higher performance as the drivers inside the guest 
talk directly to the host.

> If running hvm on
> quemu-xen-traditional, you HAVE to use a bootloader inside the VM, or
> some kind of netboot/pvgrub thing. If running upstream quemu for a hvm,
> you can choose. I find it less of a hassle to use bootloader inside the
> VM.

It's simple, if you don't have full access to the host.
If you have full access, it's actually simpler as you don't have to worry 
about boot-order, partitioning and a bootloader.

> If running pv, I BELIEVE you have to specify kernel and boot options
> outside the VM-image, possibly through pvgrub. Pvgrub is supposed to be
> able to fish out kernel and initrd from the VM disk. I never bothered to
> get that working. Here is an example excerpt from a PVM I use (for a
> debian vm):
> --print.pvm---
> builder = "generic"
> kernel = "/etc/xen/wheezy/vmlinuz-3.16.0-0.bpo.4-amd64"
> ramdisk = "/etc/xen/wheezy/initrd.img-3.16.0-0.bpo.4-amd64"
> extra = "root=/dev/xvda console=hvc0 tmem"
> --
> For explanation check "man xl.cfg". When you specify
> kernel&ramdisk&parameters in the vm config, you do NOT need a /boot/
> directory on your vm.  Obviously when you build a kernel inside the vm,
> it will end up in /boot, so you have to provide a way for the
> dom0/hypervisor to load your kernel, either (when using full
> virtualization) a regular boot-loader inside the vm, or (for a PV guest)
> through some other means to let the dom0/hypervisor load your image.

--
Joost

Re: [gentoo-user] installing Gentoo as a xen guest

[Håkon Alstadheim]

Den 26. sep. 2015 14:00, skrev J. Roeleveld:
>
>> Depending on your hardware you will want to use hvm or pvm for
>> efficiency. (VT-x means hvm is more efficient).
> What do you base this on?
> Without VT-x, HVM doesn't even work, which means PV is only option.
I stand corrected :/ .

I'l refrain from confusing the issue further, I believe I was thinking
VT-d. This
<http://wiki.xen.org/wiki/Xen_Project_Software_Overview#Guest_Types> is
the right place to learn about guest types :). Some of the alphabet-soup
is explained here <https://en.wikipedia.org/wiki/X86_virtualization>

The rule of thumb I live by for my private, ad-hoc systems is: since I
have hardware support for virtualization and also VT-d, I use it, and I
keep references handy while i configure things :-D.


> With VT-x, PV still has higher performance as the drivers inside the guest 
> talk directly to the host.
>

PV on HVM will be best of both worlds, and drivers are available at
least for windows and linux, so I was taking pv drivers as a given.
Also, where I said VT-x read VT-d.

>> If running hvm on
>> quemu-xen-traditional, you HAVE to use a bootloader inside the VM, or
>> some kind of netboot/pvgrub thing. If running upstream quemu for a hvm,
>> you can choose. I find it less of a hassle to use bootloader inside the
>> VM.
> It's simple, if you don't have full access to the host.
> If you have full access, it's actually simpler as you don't have to worry 
> about boot-order, partitioning and a bootloader.
I'm sure you are right, it is just pvgrub is an extra piece of kit I
haven't bothered learning. I'll go back to lurking now :-~

Re: [gentoo-user] installing Gentoo as a xen guest

[J. Roeleveld]

On 24 September 2015 12:05:31 CEST, hw <hw@gc-24.de> wrote:
>
>Hi,
>
>I'm installing Gentoo as a xen PV guest.  Do I need to install a 
>bootloader like grub, or should I rather just specify the kernel to
>boot 
>in the definition file of the guest?  If I do the latter, what about
>the 
>kernel command line?
>
>Is there anything I should consider?  The host is also running Gentoo.

For PVs, I put the kernel on the host and specify it in the domain config. (Including the kernel commandline)

I only install the kernel and grub inside the VM for non PV domains.

--
Joost
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [gentoo-user] installing Gentoo as a xen guest: installing grub

[hw]

grub2-install /dev/xvda
Installing for i386-pc platform.
grub2-install: warning: File system `ext2' doesn't support embedding.
grub2-install: warning: Embedding is not possible.  GRUB can only be 
installed in this setup by using blocklists.  However, blocklists are 
UNRELIABLE and their use is discouraged..
grub2-install: error: will not proceed with blocklists.


Now what?

I'm guessing I can get away with grub not actually installed as long as 
I stick to PV.

However, the guests all reside in partitions on an SSD, so my plan was 
to have one partition as swap and another one for the system.  Both 
partitions would be let to the guest by entries in the config for the VM.

Now I'm wondering if I should rather have a larger partition for each 
guest and then partition the guest's /dev/xvda into swap and system from 
within the guest.

(I'm not really fond of lvm because it lacks some very important 
features which greatly reduce its benefits.)

What would you guys do?


On a side note, would grub even run, considering it's i386 and not 
amd64?  32bit support is kinda disabled.


hw wrote:
>
> Hi,
>
> I'm installing Gentoo as a xen PV guest.  Do I need to install a
> bootloader like grub, or should I rather just specify the kernel to boot
> in the definition file of the guest?  If I do the latter, what about the
> kernel command line?
>
> Is there anything I should consider?  The host is also running Gentoo.
>

Re: [gentoo-user] installing Gentoo as a xen guest: installing grub

[Rich Freeman]

On Fri, Sep 25, 2015 at 7:11 AM, hw <hw@gc-24.de> wrote:
> grub2-install /dev/xvda
> Installing for i386-pc platform.
> grub2-install: warning: File system `ext2' doesn't support embedding.
> grub2-install: warning: Embedding is not possible.  GRUB can only be
> installed in this setup by using blocklists.  However, blocklists are
> UNRELIABLE and their use is discouraged..
> grub2-install: error: will not proceed with blocklists.
>

My understanding is that if you're using pvgrub you wouldn't actually
install grub from within the guest.  You just set up a grub.cfg and
the pvgrub bootloader resides outside the guest.

If you are using full virtualization and want to run grub, then you
need to partition your disk, so your root wouldn't be /dev/xvda but
instead it would be /dev/xvda1.

Or you can just not use grub at all.

-- 
Rich

Re: [gentoo-user] installing Gentoo as a xen guest: installing grub

[J. Roeleveld]

On Friday, September 25, 2015 01:11:53 PM hw wrote:
> grub2-install /dev/xvda
> Installing for i386-pc platform.
> grub2-install: warning: File system `ext2' doesn't support embedding.
> grub2-install: warning: Embedding is not possible.  GRUB can only be
> installed in this setup by using blocklists.  However, blocklists are
> UNRELIABLE and their use is discouraged..
> grub2-install: error: will not proceed with blocklists.
> 
> Now what?

Partition first, then install grub

> I'm guessing I can get away with grub not actually installed as long as
> I stick to PV.

PV does not supply a "BIOS", which means, Grub is not used.

> However, the guests all reside in partitions on an SSD, so my plan was
> to have one partition as swap and another one for the system.  Both
> partitions would be let to the guest by entries in the config for the VM.

Most PVs are done that way.

> Now I'm wondering if I should rather have a larger partition for each
> guest and then partition the guest's /dev/xvda into swap and system from
> within the guest.

If using HVM, then yes.
If using PV, then up to you.

> (I'm not really fond of lvm because it lacks some very important
> features which greatly reduce its benefits.)

Which features are you missing that native partitions have?

Most Xen hosts use LVM.

> What would you guys do?

LVM for partitions
Pass LVs to Xen domains
For PV: Store kernel on Host and point to the kernel in the domain config.
For HVM: pretend it's a physical box

> On a side note, would grub even run, considering it's i386 and not
> amd64?  32bit support is kinda disabled.

When using PV, no. (And that's got nothing to do with the CPU architecture.)

> hw wrote:
> > Hi,
> > 
> > I'm installing Gentoo as a xen PV guest.  Do I need to install a
> > bootloader like grub, or should I rather just specify the kernel to boot
> > in the definition file of the guest?  If I do the latter, what about the
> > kernel command line?
> > 
> > Is there anything I should consider?  The host is also running Gentoo.