[gentoo-user] virtualbox in headless configuration broken after update: delayed echo

[gentoo-user] virtualbox in headless configuration broken after update: delayed echo

[n952162]

[-- Attachment #1: Type: text/plain, Size: 737 bytes --]

I updated my system and now characters typed into vbox over ssh are not
echo-ed until *after* a CR is entered.

I diffed the stty output, to see if I could spot anything:

10~>cat /tmp/sttydiff
2,3c2,3
<  rows 37
<  columns 100
---
 >  rows 44
 >  columns 88
21d20
<  discard = ^O
23c22,23
< min = 1
---
 > discard = ^O
 >  min = 1
30c30
< hupcl
---
 > -hupcl
36c36
< brkint
---
 > -brkint
48,49c48,49
< imaxbel
< iutf8
---
 > -imaxbel
 > -iutf8

Also, the font seems to be screwed up, because the last line of the
window only shows the top half of the line.

Anybody else encounter this or know what's wrong?

Vbox seems to work okay when run locally, on the machine it's installed on.



[-- Attachment #2: Type: text/html, Size: 1565 bytes --]

Re: [gentoo-user] virtualbox in headless configuration broken after update: delayed echo [ RESOLVED, kinda ]

[n952162]

[-- Attachment #1: Type: text/plain, Size: 1264 bytes --]

On 06/10/20 15:19, n952162 wrote:
>
> I updated my system and now characters typed into vbox over ssh are
> not echo-ed until *after* a CR is entered.
>
> I diffed the stty output, to see if I could spot anything:
>
> 10~>cat /tmp/sttydiff
> 2,3c2,3
> <  rows 37
> <  columns 100
> ---
> >  rows 44
> >  columns 88
> 21d20
> <  discard = ^O
> 23c22,23
> < min = 1
> ---
> > discard = ^O
> >  min = 1
> 30c30
> < hupcl
> ---
> > -hupcl
> 36c36
> < brkint
> ---
> > -brkint
> 48,49c48,49
> < imaxbel
> < iutf8
> ---
> > -imaxbel
> > -iutf8
>
> Also, the font seems to be screwed up, because the last line of the
> window only shows the top half of the line.
>
> Anybody else encounter this or know what's wrong?
>
> Vbox seems to work okay when run locally, on the machine it's
> installed on.
>
>

I think this is resolved, kinda.
I just discovered that if I turn off the vbox menu bar, the command
entry line works properly again, both in X-less console mode and in X.
     Settings -> User Interface -> Enable menu bar (disable this)

I've always had that menu bar, and need it, so something got
changed/broken, and I still have a problem, but at least now I don't
have to enter commands in blindly.

[-- Attachment #2: Type: text/html, Size: 2399 bytes --]

Re: [gentoo-user] virtualbox in headless configuration broken after update: delayed echo [ RESOLVED, kinda ]

[J. Roeleveld]

On 16 June 2020 21:07:56 CEST, n952162 <n952162@web.de> wrote:
>On 06/10/20 15:19, n952162 wrote:
>>
>> I updated my system and now characters typed into vbox over ssh are
>> not echo-ed until *after* a CR is entered.
>>
>> I diffed the stty output, to see if I could spot anything:
>>
>> 10~>cat /tmp/sttydiff
>> 2,3c2,3
>> <  rows 37
>> <  columns 100
>> ---
>> >  rows 44
>> >  columns 88
>> 21d20
>> <  discard = ^O
>> 23c22,23
>> < min = 1
>> ---
>> > discard = ^O
>> >  min = 1
>> 30c30
>> < hupcl
>> ---
>> > -hupcl
>> 36c36
>> < brkint
>> ---
>> > -brkint
>> 48,49c48,49
>> < imaxbel
>> < iutf8
>> ---
>> > -imaxbel
>> > -iutf8
>>
>> Also, the font seems to be screwed up, because the last line of the
>> window only shows the top half of the line.
>>
>> Anybody else encounter this or know what's wrong?
>>
>> Vbox seems to work okay when run locally, on the machine it's
>> installed on.
>>
>>
>
>I think this is resolved, kinda.
>I just discovered that if I turn off the vbox menu bar, the command
>entry line works properly again, both in X-less console mode and in X.
>     Settings -> User Interface -> Enable menu bar (disable this)
>
>I've always had that menu bar, and need it, so something got
>changed/broken, and I still have a problem, but at least now I don't
>have to enter commands in blindly.

Are these Virtualbox VMs critical?
If yes, I would suggest migrating them to a more reliable virtualisation technology.

I do not consider Virtualbox suitable for anything but a desktop based VM method for a quick test or simulation.

Gor anything serious, I would suggest Xen, KVM or VMWare.

--
Joost
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [gentoo-user] virtualbox in headless configuration broken after update: delayed echo [ RESOLVED, kinda ]

[n952162]

On 06/16/20 22:36, J. Roeleveld wrote:
> On 16 June 2020 21:07:56 CEST, n952162 <n952162@web.de> wrote:
>> On 06/10/20 15:19, n952162 wrote:
>>> I updated my system and now characters typed into vbox over ssh are
>>> not echo-ed until *after* a CR is entered.
>>>
>>> I diffed the stty output, to see if I could spot anything:
>>>
>>> 10~>cat /tmp/sttydiff
>>> 2,3c2,3
>>> <  rows 37
>>> <  columns 100
>>> ---
>>>>    rows 44
>>>>    columns 88
>>> 21d20
>>> <  discard = ^O
>>> 23c22,23
>>> < min = 1
>>> ---
>>>> discard = ^O
>>>>    min = 1
>>> 30c30
>>> < hupcl
>>> ---
>>>> -hupcl
>>> 36c36
>>> < brkint
>>> ---
>>>> -brkint
>>> 48,49c48,49
>>> < imaxbel
>>> < iutf8
>>> ---
>>>> -imaxbel
>>>> -iutf8
>>> Also, the font seems to be screwed up, because the last line of the
>>> window only shows the top half of the line.
>>>
>>> Anybody else encounter this or know what's wrong?
>>>
>>> Vbox seems to work okay when run locally, on the machine it's
>>> installed on.
>>>
>>>
>> I think this is resolved, kinda.
>> I just discovered that if I turn off the vbox menu bar, the command
>> entry line works properly again, both in X-less console mode and in X.
>>      Settings -> User Interface -> Enable menu bar (disable this)
>>
>> I've always had that menu bar, and need it, so something got
>> changed/broken, and I still have a problem, but at least now I don't
>> have to enter commands in blindly.
> Are these Virtualbox VMs critical?
> If yes, I would suggest migrating them to a more reliable virtualisation technology.
>
> I do not consider Virtualbox suitable for anything but a desktop based VM method for a quick test or simulation.
>
> Gor anything serious, I would suggest Xen, KVM or VMWare.
>
> --
> Joost

Well, no, they're really not critical, but your comment surprises me. 
I've been using vbox for years, on various assignments, and never
encountered anything else.  Can you say a word or two to that, or
provide a URL?  Which free vm is "the best"?

Re: [gentoo-user] virtualbox in headless configuration broken after update: delayed echo [ RESOLVED, kinda ]

[J. Roeleveld]

On Tuesday, June 16, 2020 11:08:23 PM CEST n952162 wrote:
> On 06/16/20 22:36, J. Roeleveld wrote:
> > On 16 June 2020 21:07:56 CEST, n952162 <n952162@web.de> wrote:
> >> On 06/10/20 15:19, n952162 wrote:
> >>> I updated my system and now characters typed into vbox over ssh are
> >>> not echo-ed until *after* a CR is entered.
> >>> 
> >>> I diffed the stty output, to see if I could spot anything:
> >>> 
> >>> 10~>cat /tmp/sttydiff
> >>> 2,3c2,3
> >>> <  rows 37
> >>> <  columns 100
> >>> ---
> >>> 
> >>>>    rows 44
> >>>>    columns 88
> >>> 
> >>> 21d20
> >>> <  discard = ^O
> >>> 23c22,23
> >>> < min = 1
> >>> ---
> >>> 
> >>>> discard = ^O
> >>>> 
> >>>>    min = 1
> >>> 
> >>> 30c30
> >>> < hupcl
> >>> ---
> >>> 
> >>>> -hupcl
> >>> 
> >>> 36c36
> >>> < brkint
> >>> ---
> >>> 
> >>>> -brkint
> >>> 
> >>> 48,49c48,49
> >>> < imaxbel
> >>> < iutf8
> >>> ---
> >>> 
> >>>> -imaxbel
> >>>> -iutf8
> >>> 
> >>> Also, the font seems to be screwed up, because the last line of the
> >>> window only shows the top half of the line.
> >>> 
> >>> Anybody else encounter this or know what's wrong?
> >>> 
> >>> Vbox seems to work okay when run locally, on the machine it's
> >>> installed on.
> >> 
> >> I think this is resolved, kinda.
> >> I just discovered that if I turn off the vbox menu bar, the command
> >> entry line works properly again, both in X-less console mode and in X.
> >> 
> >>      Settings -> User Interface -> Enable menu bar (disable this)
> >> 
> >> I've always had that menu bar, and need it, so something got
> >> changed/broken, and I still have a problem, but at least now I don't
> >> have to enter commands in blindly.
> > 
> > Are these Virtualbox VMs critical?
> > If yes, I would suggest migrating them to a more reliable virtualisation
> > technology.
> > 
> > I do not consider Virtualbox suitable for anything but a desktop based VM
> > method for a quick test or simulation.
> > 
> > Gor anything serious, I would suggest Xen, KVM or VMWare.
> > 
> > --
> > Joost
> 
> Well, no, they're really not critical, but your comment surprises me. 
> I've been using vbox for years, on various assignments, and never
> encountered anything else.  Can you say a word or two to that, or
> provide a URL?  Which free vm is "the best"?

I never bothered bookmarking URLs about this, but can elaborate on my 
reasoning and experience.

Virtualbox is a nice product and I do use it when it is convenient. It is 
perfect for quickly starting a VM to test something. It integrates nicely with 
the desktop to be able to quickly copy/paste data across and also easy to 
connect to the filesystem on the host.

This also mentions the reason why it is NOT suitable for actual production 
use. It is a virtualisation tool for a desktop.

If you want your VMs to run as fast and stable as possible, you want the host 
to be as minimal as possible. This means:
- it runs headless (no GUI, just text) and the host has only 1 task: Run VMs.
- it doesn't contain anything else (only exception is stuff for monitoring)

Virtualbox does not (afaik) support block-devices for VMs. It only supports 
file-based disks. This is fine as it allows you to "quickly" move these to 
different storage. But it adds another layer between the hardware and VM 
(filesystem on the host) which adds it's own write-caching and potential 
corruption (I have had this on several occasions).

The virtualisation systems I mentioned in my previous email (Xen, KVM, VMWare) 
all support block-devices and sit as close to hardware as is possible. In the 
case of VMWare, I am talking about the server product, not the desktop 
product. The VMWare desktop product has the same problems as VirtualBox.

As for which free one is best, I am reluctant to answer specifically as both 
Xen and KVM are good.

Personally, I use Xen. I have been using it since one of the 2.x versions and 
KVM didn't exist back then.
Xen has the hypervisor in a small "kernel" and the host runs as a VM with full 
privileges. You can add additional privileges VMs to provide storage, further 
seperating tasks between VMs.
Citrix also provides a free version of their Xen-product which can be managed 
remotely, but their remote-tool is windows-only last time I checked. I run Xen 
on top of Gentoo and manage everything from the CLI.

KVM runs inside a Linux kernel and this instance automatically is the host. (I 
don't know enough to properly compare the 2, there are plenty of resources 
about comparisons online, most are biased to one or the other)

Both Xen and KVM can be managed with other tools like virt-manager. I don't as 
I don't like the way those tools want to manage the whole environment.

As for use of these systems, when only looking at companies where I have 
experience with:

- VMWare is often used for virtualising servers
- Xen (Citrix) is often used to provide Virtual Desktop to users
- KVM is used by most VPS providers
- Virtualbox is used for training sessions

I have not come across MS HyperV outside of small businesses that need some 
local VMs. These companies tend to put all their infrastructure with one of 
the big cloud-VM providers (Like AWS, Azure, Googles,...)

--
Joost

Re: [gentoo-user] virtualbox in headless configuration broken after update: delayed echo [ RESOLVED, kinda ]

[n952162]

On 06/17/20 06:48, J. Roeleveld wrote:
> On Tuesday, June 16, 2020 11:08:23 PM CEST n952162 wrote:
>> On 06/16/20 22:36, J. Roeleveld wrote:
>>>
>>> Are these Virtualbox VMs critical?
>>> If yes, I would suggest migrating them to a more reliable virtualisation
>>> technology.
>>>
>>> I do not consider Virtualbox suitable for anything but a desktop based VM
>>> method for a quick test or simulation.
>>>
>>> Gor anything serious, I would suggest Xen, KVM or VMWare.
>>>
>>> --
>>> Joost
>> Well, no, they're really not critical, but your comment surprises me.
>> I've been using vbox for years, on various assignments, and never
>> encountered anything else.  Can you say a word or two to that, or
>> provide a URL?  Which free vm is "the best"?
> I never bothered bookmarking URLs about this, but can elaborate on my
> reasoning and experience.
>
> Virtualbox is a nice product and I do use it when it is convenient. It is
> perfect for quickly starting a VM to test something. It integrates nicely with
> the desktop to be able to quickly copy/paste data across and also easy to
> connect to the filesystem on the host.
>
> This also mentions the reason why it is NOT suitable for actual production
> use. It is a virtualisation tool for a desktop.
>
> If you want your VMs to run as fast and stable as possible, you want the host
> to be as minimal as possible. This means:
> - it runs headless (no GUI, just text) and the host has only 1 task: Run VMs.
> - it doesn't contain anything else (only exception is stuff for monitoring)
>
> Virtualbox does not (afaik) support block-devices for VMs. It only supports
> file-based disks. This is fine as it allows you to "quickly" move these to
> different storage. But it adds another layer between the hardware and VM
> (filesystem on the host) which adds it's own write-caching and potential
> corruption (I have had this on several occasions).
>
> The virtualisation systems I mentioned in my previous email (Xen, KVM, VMWare)
> all support block-devices and sit as close to hardware as is possible. In the
> case of VMWare, I am talking about the server product, not the desktop
> product. The VMWare desktop product has the same problems as VirtualBox.
>
> As for which free one is best, I am reluctant to answer specifically as both
> Xen and KVM are good.
>
> Personally, I use Xen. I have been using it since one of the 2.x versions and
> KVM didn't exist back then.
> Xen has the hypervisor in a small "kernel" and the host runs as a VM with full
> privileges. You can add additional privileges VMs to provide storage, further
> seperating tasks between VMs.
> Citrix also provides a free version of their Xen-product which can be managed
> remotely, but their remote-tool is windows-only last time I checked. I run Xen
> on top of Gentoo and manage everything from the CLI.
>
> KVM runs inside a Linux kernel and this instance automatically is the host. (I
> don't know enough to properly compare the 2, there are plenty of resources
> about comparisons online, most are biased to one or the other)
>
> Both Xen and KVM can be managed with other tools like virt-manager. I don't as
> I don't like the way those tools want to manage the whole environment.
>
> As for use of these systems, when only looking at companies where I have
> experience with:
>
> - VMWare is often used for virtualising servers
> - Xen (Citrix) is often used to provide Virtual Desktop to users
> - KVM is used by most VPS providers
> - Virtualbox is used for training sessions
>
> I have not come across MS HyperV outside of small businesses that need some
> local VMs. These companies tend to put all their infrastructure with one of
> the big cloud-VM providers (Like AWS, Azure, Googles,...)
>
> --
> Joost
>
>

Thank you for this excellent survey/summary.  It tells me that vbox is
good for my current usages, but I should start exposing myself to Xen as
a possible migration path.

Re: [gentoo-user] virtualbox in headless configuration broken after update: delayed echo [ RESOLVED, kinda ]

[J. Roeleveld]

On Wednesday, June 17, 2020 7:42:30 AM CEST n952162 wrote:
> On 06/17/20 06:48, J. Roeleveld wrote:
> > On Tuesday, June 16, 2020 11:08:23 PM CEST n952162 wrote:
> >> On 06/16/20 22:36, J. Roeleveld wrote:

<snipped>

> > I have not come across MS HyperV outside of small businesses that need
> > some
> > local VMs. These companies tend to put all their infrastructure with one
> > of
> > the big cloud-VM providers (Like AWS, Azure, Googles,...)
> > 
> > --
> > Joost
> 
> Thank you for this excellent survey/summary.  It tells me that vbox is
> good for my current usages, but I should start exposing myself to Xen as
> a possible migration path.

I would actually suggest to read up on both Xen and KVM and try both on spare 
machines.
See which best fits your requirements and also see if the existing management 
tools actually do things in a way that you can work with.

My systems have evolved over the past 25-odd years and I started using Xen to 
reduce the amount of physical systems I had running. At the time, VMWare was 
expensive, KVM didn't exist yet and was missing some important features for a 
few years after it appeared (not sure if this exists yet, not found anything 
about it on KVM):
- limit memory footprint of host-VM during boot.
- Dedicate CPU-core(s) to the host

Limiting the memory size is important, because there are several parts of the 
kernel (and userspace) that base their memory-settings on this amount. This is 
really noticable when the host thinks it has 384GB available when 370GB is 
passed to VMs.

Dedicating CPU-cores exclusively to the host means the host will always have 
CPU-resources available. This is necessary because all the context-switching 
is handled by the host and if this stalls, the whole environment is impacted.

For a lab-system, I was also missing the ability to save the full state of a 
VM for a snapshot. All the howto's and guides I can find online only talk 
about making a snapshot of the disks. Not of the memory as well. Especially 
when used to Virtualbox, you will notice this issue. When only snapshotting 
the disk, your snapshot is basically the state of when you literally pulled 
the plug of your VM if you want to restore back to this.

For KVM, I have found a few hints that this was planned. But I have not found 
anything about this. Virt-manager does not (last time I looked) support Xen's 
functionality of storing the memory when creating snapshots either. Which is 
why I don't use that even for my lab/testing-server.

As for tips/tricks (below works for Xen, but should also work with KVM):

The way I create a new Gentoo-VM is simply to create a new block-device 
(Either LVM or ZFS), do all the initial steps in the chroot from the host and 
when it comes to the first-reboot, umount the filesystems, hook it up to a new 
VM and start that.

Because of this, I can update the host as follows:
- create new "partitions" for the host-system.
- Install the latest versions, migrate the config across
- reboot into the new host.

If all goes fine, I can clean up the "old" partitions and prepare them for 
next time. If there are issues, I have a working "old" version I can quickly 
revert to.

--
Joost

Re: [gentoo-user] virtualbox in headless configuration broken after update: delayed echo [ RESOLVED, kinda ]

[Michael]

[-- Attachment #1: Type: text/plain, Size: 3996 bytes --]

On Wednesday, 17 June 2020 07:32:10 BST J. Roeleveld wrote:
> On Wednesday, June 17, 2020 7:42:30 AM CEST n952162 wrote:
> > On 06/17/20 06:48, J. Roeleveld wrote:
> > > On Tuesday, June 16, 2020 11:08:23 PM CEST n952162 wrote:
> > >> On 06/16/20 22:36, J. Roeleveld wrote:
> <snipped>
> 
> > > I have not come across MS HyperV outside of small businesses that need
> > > some
> > > local VMs. These companies tend to put all their infrastructure with one
> > > of
> > > the big cloud-VM providers (Like AWS, Azure, Googles,...)
> > > 
> > > --
> > > Joost
> > 
> > Thank you for this excellent survey/summary.  It tells me that vbox is
> > good for my current usages, but I should start exposing myself to Xen as
> > a possible migration path.
> 
> I would actually suggest to read up on both Xen and KVM and try both on
> spare machines.
> See which best fits your requirements and also see if the existing
> management tools actually do things in a way that you can work with.
> 
> My systems have evolved over the past 25-odd years and I started using Xen
> to reduce the amount of physical systems I had running. At the time, VMWare
> was expensive, KVM didn't exist yet and was missing some important features
> for a few years after it appeared (not sure if this exists yet, not found
> anything about it on KVM):
> - limit memory footprint of host-VM during boot.
> - Dedicate CPU-core(s) to the host
> 
> Limiting the memory size is important, because there are several parts of
> the kernel (and userspace) that base their memory-settings on this amount.
> This is really noticable when the host thinks it has 384GB available when
> 370GB is passed to VMs.
> 
> Dedicating CPU-cores exclusively to the host means the host will always have
> CPU-resources available. This is necessary because all the
> context-switching is handled by the host and if this stalls, the whole
> environment is impacted.
> 
> For a lab-system, I was also missing the ability to save the full state of a
> VM for a snapshot. All the howto's and guides I can find online only talk
> about making a snapshot of the disks. Not of the memory as well. Especially
> when used to Virtualbox, you will notice this issue. When only snapshotting
> the disk, your snapshot is basically the state of when you literally pulled
> the plug of your VM if you want to restore back to this.
> 
> For KVM, I have found a few hints that this was planned. But I have not
> found anything about this. Virt-manager does not (last time I looked)
> support Xen's functionality of storing the memory when creating snapshots
> either. Which is why I don't use that even for my lab/testing-server.

As far as I know QEMU with KVM can take snapshots of the current state of RAM, 
disk(s), CPU - it can take snapshots of images while online.

https://wiki.qemu.org/Features/Snapshots2


However, I've only taken snapshots of qcow2 images after I shut down the VM.  
These work as advertised and they are quite handy before major updates/
upgrades as temporary backups.


> As for tips/tricks (below works for Xen, but should also work with KVM):
> 
> The way I create a new Gentoo-VM is simply to create a new block-device
> (Either LVM or ZFS), do all the initial steps in the chroot from the host
> and when it comes to the first-reboot, umount the filesystems, hook it up
> to a new VM and start that.
> 
> Because of this, I can update the host as follows:
> - create new "partitions" for the host-system.
> - Install the latest versions, migrate the config across
> - reboot into the new host.
> 
> If all goes fine, I can clean up the "old" partitions and prepare them for
> next time. If there are issues, I have a working "old" version I can quickly
> revert to.
> 
> --
> Joost

I've wanted to migrate a qemu qcow2 image file or two of different OS', all 
currently stored on an ext4 partition on my desktop, to a dedicated partition 
on the disk.  Would this be possible - how?  Would I need to change the qcow2 
to a raw image?

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] virtualbox in headless configuration broken after update: delayed echo [ RESOLVED, kinda ]

[J. Roeleveld]

On 17 June 2020 19:01:54 CEST, Michael <confabulate@kintzios.com> wrote:
>On Wednesday, 17 June 2020 07:32:10 BST J. Roeleveld wrote:
>> On Wednesday, June 17, 2020 7:42:30 AM CEST n952162 wrote:
>> > On 06/17/20 06:48, J. Roeleveld wrote:
>> > > On Tuesday, June 16, 2020 11:08:23 PM CEST n952162 wrote:
>> > >> On 06/16/20 22:36, J. Roeleveld wrote:
>> <snipped>
>> 
>> > > I have not come across MS HyperV outside of small businesses that
>need
>> > > some
>> > > local VMs. These companies tend to put all their infrastructure
>with one
>> > > of
>> > > the big cloud-VM providers (Like AWS, Azure, Googles,...)
>> > > 
>> > > --
>> > > Joost
>> > 
>> > Thank you for this excellent survey/summary.  It tells me that vbox
>is
>> > good for my current usages, but I should start exposing myself to
>Xen as
>> > a possible migration path.
>> 
>> I would actually suggest to read up on both Xen and KVM and try both
>on
>> spare machines.
>> See which best fits your requirements and also see if the existing
>> management tools actually do things in a way that you can work with.
>> 
>> My systems have evolved over the past 25-odd years and I started
>using Xen
>> to reduce the amount of physical systems I had running. At the time,
>VMWare
>> was expensive, KVM didn't exist yet and was missing some important
>features
>> for a few years after it appeared (not sure if this exists yet, not
>found
>> anything about it on KVM):
>> - limit memory footprint of host-VM during boot.
>> - Dedicate CPU-core(s) to the host
>> 
>> Limiting the memory size is important, because there are several
>parts of
>> the kernel (and userspace) that base their memory-settings on this
>amount.
>> This is really noticable when the host thinks it has 384GB available
>when
>> 370GB is passed to VMs.
>> 
>> Dedicating CPU-cores exclusively to the host means the host will
>always have
>> CPU-resources available. This is necessary because all the
>> context-switching is handled by the host and if this stalls, the
>whole
>> environment is impacted.
>> 
>> For a lab-system, I was also missing the ability to save the full
>state of a
>> VM for a snapshot. All the howto's and guides I can find online only
>talk
>> about making a snapshot of the disks. Not of the memory as well.
>Especially
>> when used to Virtualbox, you will notice this issue. When only
>snapshotting
>> the disk, your snapshot is basically the state of when you literally
>pulled
>> the plug of your VM if you want to restore back to this.
>> 
>> For KVM, I have found a few hints that this was planned. But I have
>not
>> found anything about this. Virt-manager does not (last time I looked)
>> support Xen's functionality of storing the memory when creating
>snapshots
>> either. Which is why I don't use that even for my lab/testing-server.
>
>As far as I know QEMU with KVM can take snapshots of the current state
>of RAM, 
>disk(s), CPU - it can take snapshots of images while online.
>
>https://wiki.qemu.org/Features/Snapshots2

Can you point to where in the commands above the memory anf cpu state is actually stored and loaded back when reverting to the snapshot?
From what I see, it is only fhe disk image.
I really need this feature for lab environments where I need the ability to fully roll back to a running instance.


>However, I've only taken snapshots of qcow2 images after I shut down
>the VM.  
>These work as advertised and they are quite handy before major updates/
>upgrades as temporary backups.
>
>
>> As for tips/tricks (below works for Xen, but should also work with
>KVM):
>> 
>> The way I create a new Gentoo-VM is simply to create a new
>block-device
>> (Either LVM or ZFS), do all the initial steps in the chroot from the
>host
>> and when it comes to the first-reboot, umount the filesystems, hook
>it up
>> to a new VM and start that.
>> 
>> Because of this, I can update the host as follows:
>> - create new "partitions" for the host-system.
>> - Install the latest versions, migrate the config across
>> - reboot into the new host.
>> 
>> If all goes fine, I can clean up the "old" partitions and prepare
>them for
>> next time. If there are issues, I have a working "old" version I can
>quickly
>> revert to.
>> 
>> --
>> Joost
>
>I've wanted to migrate a qemu qcow2 image file or two of different OS',
>all 
>currently stored on an ext4 partition on my desktop, to a dedicated
>partition 
>on the disk.  Would this be possible - how?  Would I need to change the
>qcow2 
>to a raw image?

I don't know. One of the reasons I dislike file based images is the lack of transparency and tools. LVM is much simpler for disk based snapshots and management.

--
Joost



-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [gentoo-user] virtualbox in headless configuration broken after update: delayed echo [ RESOLVED, kinda ]

[Michael]

[-- Attachment #1: Type: text/plain, Size: 1713 bytes --]

On Wednesday, 17 June 2020 18:31:42 BST J. Roeleveld wrote:
> On 17 June 2020 19:01:54 CEST, Michael <confabulate@kintzios.com> wrote:

> >https://wiki.qemu.org/Features/Snapshots2
> 
> Can you point to where in the commands above the memory anf cpu state is
> actually stored and loaded back when reverting to the snapshot? From what I
> see, it is only fhe disk image.
> I really need this feature for lab environments where I need the ability to
> fully roll back to a running instance.

I understand runtime parameters (inc. RAM, CPU cores, et al.) are also 
reflected in the snapshot and have seen this mentioned in the interwebs, but 
as I have not performed an online snapshot myself and therefore I can't 
confirm its validity.  :(

However, all I see in the previous link plus the two below is manipulation of 
VM images.

https://wiki.qemu.org/Features/Snapshots
https://wiki.qemu.org/Features/LiveBlockMigration


> >I've wanted to migrate a qemu qcow2 image file or two of different OS',
> >all
> >currently stored on an ext4 partition on my desktop, to a dedicated
> >partition
> >on the disk.  Would this be possible - how?  Would I need to change the
> >qcow2
> >to a raw image?
> 
> I don't know. One of the reasons I dislike file based images is the lack of
> transparency and tools. LVM is much simpler for disk based snapshots and
> management.
> 
> --
> Joost

I have found QEMU rather esoteric in its command range and options, which has 
changed over time;  with older commands deprecated (e.g. '-drive 
if=virtio,...' replaced with '-blockdev file,...'.  I'd like to migrate a 
Win10 VM to a disk partition, but would not want to mess this up, because I 
would hate to have to reinstall it.

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] virtualbox in headless configuration broken after update: delayed echo [ RESOLVED, kinda ]

[J. Roeleveld]

On 17 June 2020 21:32:19 CEST, Michael <confabulate@kintzios.com> wrote:
>On Wednesday, 17 June 2020 18:31:42 BST J. Roeleveld wrote:
>> On 17 June 2020 19:01:54 CEST, Michael <confabulate@kintzios.com>
>wrote:
>
>> >https://wiki.qemu.org/Features/Snapshots2
>> 
>> Can you point to where in the commands above the memory anf cpu state
>is
>> actually stored and loaded back when reverting to the snapshot? From
>what I
>> see, it is only fhe disk image.
>> I really need this feature for lab environments where I need the
>ability to
>> fully roll back to a running instance.
>
>I understand runtime parameters (inc. RAM, CPU cores, et al.) are also 
>reflected in the snapshot and have seen this mentioned in the
>interwebs, but 
>as I have not performed an online snapshot myself and therefore I can't
>
>confirm its validity.  :(
>
>However, all I see in the previous link plus the two below is
>manipulation of 
>VM images.
>
>https://wiki.qemu.org/Features/Snapshots

This one mentions the need for guest agent functionality to put the filesystem in a consistent state, to avoid having to fsck at restore time.
This does not sound like it includes actual memory amd cpu status in the snapshot.

VMWare, Virtualbox and Xen have had this functionality for decades. It really is a shame KVM and Qemu don't have this.


>https://wiki.qemu.org/Features/LiveBlockMigration
>
>
>> >I've wanted to migrate a qemu qcow2 image file or two of different
>OS',
>> >all
>> >currently stored on an ext4 partition on my desktop, to a dedicated
>> >partition
>> >on the disk.  Would this be possible - how?  Would I need to change
>the
>> >qcow2
>> >to a raw image?
>> 
>> I don't know. One of the reasons I dislike file based images is the
>lack of
>> transparency and tools. LVM is much simpler for disk based snapshots
>and
>> management.
>> 
>> --
>> Joost
>
>I have found QEMU rather esoteric in its command range and options,
>which has 
>changed over time;  with older commands deprecated (e.g. '-drive 
>if=virtio,...' replaced with '-blockdev file,...'.  I'd like to migrate
>a 
>Win10 VM to a disk partition, but would not want to mess this up,
>because I 
>would hate to have to reinstall it.

This brings another problem I have with KVM/QEMU: all howtos and documents I find show long commandline options to just start the VM.
I have not found one where I can provide all the config in a single file and use that. Allowing me to duplicate settings by simply copying the file and changing a few lines.

KVM/Qemu seems to be written to be used together with virt-manager which, for me, lacks important features to make it usable in production.

--
Joost


-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [gentoo-user] virtualbox in headless configuration broken after update: delayed echo [ RESOLVED, kinda ]

[Michael]

[-- Attachment #1: Type: text/plain, Size: 2528 bytes --]

On Wednesday, 17 June 2020 20:55:19 BST J. Roeleveld wrote:
> On 17 June 2020 21:32:19 CEST, Michael <confabulate@kintzios.com> wrote:

> This brings another problem I have with KVM/QEMU: all howtos and documents I
> find show long commandline options to just start the VM. I have not found
> one where I can provide all the config in a single file and use that.
> Allowing me to duplicate settings by simply copying the file and changing a
> few lines.
> 
> KVM/Qemu seems to be written to be used together with virt-manager which,
> for me, lacks important features to make it usable in production.
> 
> --
> Joost

The long line can be extremely short, if you are happy with defaults, or 
silly-long if you have specific needs.  The same long line can also go into a 
script and invoked with bash.  For example, this will boot a pfSense VM on an 
EFI BIOS platform, 4G RAM and Opteron_G5-v1 CPU:

qemu-system-x86_64 -bios /usr/share/edk2-ovmf/OVMF_CODE.fd -m 4096 -cpu 
Opteron_G5-v1,+topoext -smp cpus=4,maxcpus=8,cores=4,threads=2,sockets=1 -
enable-kvm -accel kvm,thread=multi -display sdl,gl=on -vga virtio -soundhw hda 
-drive if=virtio,file=pfSense.qcow2,cache=none -usb -device qemu-xhci,id=xhci 
-net nic,model=virtio -net user,hostfwd=tcp::53260-:443

or, you could set it all up in a script file, one or more lines at a time:

#!/bin/bash
#
        qemu-system-x86_64 \
        -bios /usr/share/edk2-ovmf/OVMF_CODE.fd \
        -m 4096 \
        -cpu Opteron_G5-v1,+topoext \
[snip ...]

Then call the script from a terminal whenever you want to run it.  I prefer 
using a script file when I experiment with new machines and options, because 
it allows me to comment out options as I change them to alternatives and 
backtrack as necessary.

If you prefer to use RHL's libvirt to execute in 'C' API calls onto QEMU/KVM, 
then virt-manager will offer a VirtualBox GUI equivalent, or virsh a CLI 
option.  Virt-manager can be the front end for other hypervisors/VMs, e.g. 
Xen, ESX, etc. so it is not QEMU/KVM specific.

QEMU also offers QMP for JSON scripting, because ... apps, and a rather basic 
console interface called Human Monitor Interface (HMP), should you wish to 
interact with it while it's running to modify inputs/interfaces, plug/unplug 
devices, etc.

I'm not starting and stopping VMs all day long, so installing and running a 
libvirt GUI, or some of the various front ends for QEMU has not been necessary 
for me.  However, they may offer more sophisticated options than my basic CLI 
input above.

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] virtualbox in headless configuration broken after update: delayed echo [ RESOLVED, kinda ]

[William Kenworthy]

On 18/6/20 3:55 am, J. Roeleveld wrote:
> On 17 June 2020 21:32:19 CEST, Michael <confabulate@kintzios.com> wrote:
>> On Wednesday, 17 June 2020 18:31:42 BST J. Roeleveld wrote:
>>> On 17 June 2020 19:01:54 CEST, Michael <confabulate@kintzios.com>
>
> This brings another problem I have with KVM/QEMU: all howtos and documents I find show long commandline options to just start the VM.
> I have not found one where I can provide all the config in a single file and use that. Allowing me to duplicate settings by simply copying the file and changing a few lines.
>
> KVM/Qemu seems to be written to be used together with virt-manager which, for me, lacks important features to make it usable in production.
>
> --
> Joost
>
>
Until a few months ago I was using Qemu/KVM/VirtManager with snapshots
for linux and Windows VM's running network services - the trick is
redundancy and suspend.

Using libvirt, suspend the VM which copies ram to disk, then snapshot
((I used btrfs) both the ram and storage and restart the VM - takes
seconds (so redundancy may be needed) - at this point I backup'ed the
snapshot and deleted it.


When using snapshots of live systems, I had problems restoring some of
the more active VM's which this avoids.


BillK

Re: [gentoo-user] virtualbox in headless configuration broken after update: delayed echo [ RESOLVED, kinda ]

[Michael]

[-- Attachment #1: Type: text/plain, Size: 2492 bytes --]

On Thursday, 18 June 2020 00:09:45 BST William Kenworthy wrote:
> On 18/6/20 3:55 am, J. Roeleveld wrote:
> > On 17 June 2020 21:32:19 CEST, Michael <confabulate@kintzios.com> wrote:
> >> On Wednesday, 17 June 2020 18:31:42 BST J. Roeleveld wrote:
> >>> On 17 June 2020 19:01:54 CEST, Michael <confabulate@kintzios.com>
> > 
> > This brings another problem I have with KVM/QEMU: all howtos and documents
> > I find show long commandline options to just start the VM. I have not
> > found one where I can provide all the config in a single file and use
> > that. Allowing me to duplicate settings by simply copying the file and
> > changing a few lines.
> > 
> > KVM/Qemu seems to be written to be used together with virt-manager which,
> > for me, lacks important features to make it usable in production.
> > 
> > --
> > Joost
> 
> Until a few months ago I was using Qemu/KVM/VirtManager with snapshots
> for linux and Windows VM's running network services - the trick is
> redundancy and suspend.
> 
> Using libvirt, suspend the VM which copies ram to disk, then snapshot
> ((I used btrfs) both the ram and storage and restart the VM - takes
> seconds (so redundancy may be needed) - at this point I backup'ed the
> snapshot and deleted it.

Ahh!  Yes!

I had seen the Virsh command which involves pausing a live image and then 
taking a snapshot of it.  William's pointer makes sense.

However, there is also the option to define a memory file as an object

 '-object memory-backend-file,size=1M,share,mem-path=/dev/shm/blah...'

and this can be attached/detached/migrated/snapshoted/etc between hosts as 
desired.

More details here (search for "Generic object creation"):

https://qemu.readthedocs.io/en/latest/system/invocation.html

> When using snapshots of live systems, I had problems restoring some of
> the more active VM's which this avoids.
> 
> 
> BillK

Perhaps the technique with busy systems, other than pausing > snapshot, could 
involve using a migration of the image to a new location?  As I understand it 
the newly migrated image will take over and start running in place of the old 
image, the moment the migration is completed.

Scratching around I found the way to move an image from a qcow2 file on the 
host fs, to a raw format on a disk/partition.  It is simpler than I thought 
(even with no libvirt):

qemu-img convert -O raw image.qcow2 /dev/sda5

Or, 

the output can be 'image.raw' then keep a backup copy of it as a file and then 
dd it to a partition.disk.

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]