[gentoo-user] virt-manager and ssh

[gentoo-user] virt-manager and ssh

[Stefan G. Weichinger]

Does anyone of you use virt-manager to control QEMU/KVM-hosts?

I do or try ... and I always get hickups when I edit VM-settings.

Tried virt-manager-0.9.5, 0.10.0, 0.10.0-r1  ... same behavior, when I
click Save for setting the disks properties it runs into a timeout or
something.

The connection uses ssh and I access the host via an IPSEC-VPN (running
on my router). Normal ssh and/or mosh-sessions are fine.

I allow all TCP/UDP-ports through their firewall for my IPs ... but on
the target host I see:

libvirtd[5742]: No response from client 0x7fa8060eef80 after 5 keepalive
messages in 31 seconds

which seems to be the issue somehow.

Does anyone know this problem?

(feels like this server is my test case for nearly all topics right now)

Stefan

Re: [gentoo-user] virt-manager and ssh

[William Kenworthy]

On 03/10/13 17:18, Stefan G. Weichinger wrote:
> 
> Does anyone of you use virt-manager to control QEMU/KVM-hosts?
> 
> I do or try ... and I always get hickups when I edit VM-settings.
> 
> Tried virt-manager-0.9.5, 0.10.0, 0.10.0-r1  ... same behavior, when I
> click Save for setting the disks properties it runs into a timeout or
> something.
> 
> The connection uses ssh and I access the host via an IPSEC-VPN (running
> on my router). Normal ssh and/or mosh-sessions are fine.
> 
> I allow all TCP/UDP-ports through their firewall for my IPs ... but on
> the target host I see:
> 
> libvirtd[5742]: No response from client 0x7fa8060eef80 after 5 keepalive
> messages in 31 seconds
> 
> which seems to be the issue somehow.
> 
> Does anyone know this problem?
> 
> (feels like this server is my test case for nearly all topics right now)
> 
> Stefan
> 

Are you running stp.  I was having a problem that when I did anything to
libvirt it would disconnect the bridge and stp reconfigured.  Turning it
off fixed it (am using openvswitch)

BillK

Re: [gentoo-user] virt-manager and ssh

[Stefan G. Weichinger]

Am 03.10.2013 11:43, schrieb William Kenworthy:
> On 03/10/13 17:18, Stefan G. Weichinger wrote:
>>
>> Does anyone of you use virt-manager to control QEMU/KVM-hosts?
>>
>> I do or try ... and I always get hickups when I edit VM-settings.
>>
>> Tried virt-manager-0.9.5, 0.10.0, 0.10.0-r1  ... same behavior, when I
>> click Save for setting the disks properties it runs into a timeout or
>> something.
>>
>> The connection uses ssh and I access the host via an IPSEC-VPN (running
>> on my router). Normal ssh and/or mosh-sessions are fine.
>>
>> I allow all TCP/UDP-ports through their firewall for my IPs ... but on
>> the target host I see:
>>
>> libvirtd[5742]: No response from client 0x7fa8060eef80 after 5 keepalive
>> messages in 31 seconds
>>
>> which seems to be the issue somehow.
>>
>> Does anyone know this problem?
>>
>> (feels like this server is my test case for nearly all topics right now)
>>
>> Stefan
>>
> 
> Are you running stp.  I was having a problem that when I did anything to
> libvirt it would disconnect the bridge and stp reconfigured.  Turning it
> off fixed it (am using openvswitch)

Could you specify where to turn that off? I don't know stp ... sorry.
Thanks!

(and no openvswitch here)

Re: [gentoo-user] virt-manager and ssh

[Stefan G. Weichinger]

Am 03.10.2013 11:45, schrieb Stefan G. Weichinger:

>> Are you running stp.  I was having a problem that when I did anything to
>> libvirt it would disconnect the bridge and stp reconfigured.  Turning it
>> off fixed it (am using openvswitch)
> 
> Could you specify where to turn that off? I don't know stp ... sorry.

Ah, it's a bridge "feature" ;-)

I assume I should turn it off on the KVM server. Will do now.

Re: [gentoo-user] virt-manager and ssh

[Stefan G. Weichinger]

Am 03.10.2013 11:49, schrieb Stefan G. Weichinger:
> Am 03.10.2013 11:45, schrieb Stefan G. Weichinger:
> 
>>> Are you running stp.  I was having a problem that when I did anything to
>>> libvirt it would disconnect the bridge and stp reconfigured.  Turning it
>>> off fixed it (am using openvswitch)
>>
>> Could you specify where to turn that off? I don't know stp ... sorry.
> 
> Ah, it's a bridge "feature" ;-)
> 
> I assume I should turn it off on the KVM server. Will do now.
> 
> 

No change:

I get


Fehler beim Anwenden von Änderungen: Interner Fehler: Client Socket ist
geschlossen

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/details.py", line 2022, in
config_apply
    ret = self.config_disk_apply(key)
  File "/usr/share/virt-manager/virtManager/details.py", line 2323, in
config_disk_apply
    return self._change_config_helper(df, da, hf, ha)
  File "/usr/share/virt-manager/virtManager/details.py", line 2566, in
_change_config_helper
    self.vm.refresh_xml()
  File "/usr/share/virt-manager/virtManager/libvirtobject.py", line 115,
in refresh_xml
    self._xml = self._XMLDesc(self._active_xml_flags)
  File "/usr/share/virt-manager/virtManager/domain.py", line 924, in
_XMLDesc
    return self._backend.XMLDesc(flags)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 400, in XMLDesc
    if ret is None: raise libvirtError ('virDomainGetXMLDesc() failed',
dom=self)
libvirtError: Interner Fehler: Client Socket ist geschlossen

(german parts in there ...) and still the

libvirtd[5742]: No response from client 0x7fa8060eef10 after 5 keepalive
messages in 31 seconds

on the server.

Re: [gentoo-user] virt-manager and ssh

[Michael Hampicke]

[-- Attachment #1: Type: text/plain, Size: 709 bytes --]

Am 03.10.2013 11:52, schrieb Stefan G. Weichinger:
> Am 03.10.2013 11:49, schrieb Stefan G. Weichinger:
>> Am 03.10.2013 11:45, schrieb Stefan G. Weichinger:
>>
>>>> Are you running stp.  I was having a problem that when I did anything to
>>>> libvirt it would disconnect the bridge and stp reconfigured.  Turning it
>>>> off fixed it (am using openvswitch)
>>>
>>> Could you specify where to turn that off? I don't know stp ... sorry.
>>
>> Ah, it's a bridge "feature" ;-)
>>
>> I assume I should turn it off on the KVM server. Will do now.
>>
>>
> 
> No change:
> 
> I get
>  *** snipped error ***

Did you restart libvirt after the change?

virt-manager over ssh here works fine.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-user] virt-manager and ssh

[Stefan G. Weichinger]

Am 03.10.2013 13:42, schrieb Michael Hampicke:

> Did you restart libvirt after the change?
> 
> virt-manager over ssh here works fine.

Restarted, rebuilt ...

which release do you use? Which USE-flags?

I get "closed client socket" again right now.

Re: [gentoo-user] virt-manager and ssh

[Michael Hampicke]

[-- Attachment #1: Type: text/plain, Size: 1250 bytes --]

Am 03.10.2013 13:52, schrieb Stefan G. Weichinger:
> Am 03.10.2013 13:42, schrieb Michael Hampicke:
> 
>> Did you restart libvirt after the change?
>>
>> virt-manager over ssh here works fine.
> 
> Restarted, rebuilt ...
> 
> which release do you use? Which USE-flags?
> 

Client side:
[ebuild   R    ] app-emulation/libvirt-1.1.2-r3  USE="caps libvirtd lvm
macvtap nls numa policykit python qemu systemd udev vepa virt-network
virtualbox -audit -avahi -firewalld -fuse -iscsi -lxc -nfs -openvz
-parted -pcap -phyp -rbd -sasl (-selinux) -uml -xen"
PYTHON_SINGLE_TARGET="python2_7 -python2_6" PYTHON_TARGETS="python2_7
-python2_6" 0 kB
[ebuild   R    ] app-emulation/virt-manager-0.10.0-r1
USE="gnome-keyring policykit -debug -sasl"
PYTHON_SINGLE_TARGET="python2_7 -python2_6" PYTHON_TARGETS="python2_7
-python2_6" 0 kB


Server side:

[ebuild   R   ~] app-emulation/libvirt-1.1.2-r3  USE="caps libvirtd lvm
macvtap nls numa python qemu udev vepa virt-network -audit -avahi
-firewalld -fuse -iscsi -lxc -nfs -openvz -parted -pcap -phyp -policykit
-rbd -sasl (-selinux) -systemd -uml -virtualbox -xen"
PYTHON_SINGLE_TARGET="python2_7 -python2_6" PYTHON_TARGETS="python2_7
-python2_6" 0 kB

I connect via ssh+pubkey


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-user] virt-manager and ssh

[Stefan G. Weichinger]

Am 03.10.2013 15:39, schrieb Michael Hampicke:

> Server side:
> 
> [ebuild   R   ~] app-emulation/libvirt-1.1.2-r3  USE="caps libvirtd
> lvm macvtap nls numa python qemu udev vepa virt-network -audit
> -avahi -firewalld -fuse -iscsi -lxc -nfs -openvz -parted -pcap
> -phyp -policykit -rbd -sasl (-selinux) -systemd -uml -virtualbox
> -xen" PYTHON_SINGLE_TARGET="python2_7 -python2_6"
> PYTHON_TARGETS="python2_7 -python2_6" 0 kB
> 
> I connect via ssh+pubkey

Thanks ... looks quite the same for me ... rebuilding things now.

I even consider to create a VM locally and copy the XML file over ...
just to get things running ... frustrating!

Stefan

Re: [gentoo-user] virt-manager and ssh

[Stefan G. Weichinger]

Am 03.10.2013 16:03, schrieb Stefan G. Weichinger:
> Am 03.10.2013 15:39, schrieb Michael Hampicke:
> 
>> Server side:
>>
>> [ebuild   R   ~] app-emulation/libvirt-1.1.2-r3  USE="caps libvirtd
>> lvm macvtap nls numa python qemu udev vepa virt-network -audit
>> -avahi -firewalld -fuse -iscsi -lxc -nfs -openvz -parted -pcap
>> -phyp -policykit -rbd -sasl (-selinux) -systemd -uml -virtualbox
>> -xen" PYTHON_SINGLE_TARGET="python2_7 -python2_6"
>> PYTHON_TARGETS="python2_7 -python2_6" 0 kB
>>
>> I connect via ssh+pubkey
> 
> Thanks ... looks quite the same for me ... rebuilding things now.
> 
> I even consider to create a VM locally and copy the XML file over ...
> just to get things running ... frustrating!

Whenever I want to edit anything ... I get these hangups.
*sigh*

Re: [gentoo-user] virt-manager and ssh

[Stefan G. Weichinger]

Maybe my bridge-setup is wrong?

# cat /etc/systemd/system/bridge.service
[Unit]
Description=network bridge for KVM
After=network.target

[Service]
Type=oneshot
RemainAfterExit=yes

EnvironmentFile=/etc/conf.d/network_systemd

ExecStart=/sbin/brctl addbr ${bridge_name}
ExecStart=/usr/bin/tunctl -b -u ${user} -t ${tap_name}

ExecStart=/bin/ifconfig ${bridge_name} ${address} netmask ${netmask} up
ExecStart=/bin/ifconfig ${interface} up
ExecStart=/bin/ifconfig ${tap_name} up 0.0.0.0 promisc
ExecStart=/sbin/brctl addif ${bridge_name} ${tap_name} ${interface}
ExecStart=/sbin/sysctl net.ipv4.conf.${bridge_name}.forwarding=1
#ExecStart=/sbin/iptables -t nat -A POSTROUTING -o ${interface} -j
MASQUERADE
ExecStart=/bin/ip route add default via ${gateway}

ExecStop=/bin/ip route del default via ${gateway}
ExecStop=/sbin/sysctl net.ipv4.conf.${bridge_name}.forwarding=0
ExecStop=/bin/ifconfig ${tap_name} down
ExecStop=/sbin/brctl delif ${bridge_name} ${tap_name}
ExecStop=/usr/bin/tunctl -d ${tap_name}
ExecStop=/bin/ifconfig ${bridge_name} down
ExecStop=/bin/ifconfig ${interface} down
ExecStop=/sbin/brctl delbr ${bridge_name}
#ExecStop=/sbin/iptables -t nat -D POSTROUTING -o ${interface} -j MASQUERADE

[Install]
WantedBy=multi-user.target


----

# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0
state UP qlen 1000
    link/ether d8:9d:67:1a:bc:80 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::da9d:67ff:fe1a:bc80/64 scope link
       valid_lft forever preferred_lft forever
3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP
qlen 1000
    link/ether d8:9d:67:1a:bc:81 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.201/24 brd 192.168.1.255 scope global eno2
       valid_lft forever preferred_lft forever
    inet6 fe80::da9d:67ff:fe1a:bc81/64 scope link
       valid_lft forever preferred_lft forever
4: eno3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether d8:9d:67:1a:bc:82 brd ff:ff:ff:ff:ff:ff
5: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether d8:9d:67:1a:bc:83 brd ff:ff:ff:ff:ff:ff
6: sit0: <NOARP> mtu 1480 qdisc noop state DOWN
    link/sit 0.0.0.0 brd 0.0.0.0
7: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether d6:ce:b7:07:92:a5 brd ff:ff:ff:ff:ff:ff
    inet 10.96.25.131/25 brd 10.96.25.255 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fe80::4481:f2ff:fe56:12bf/64 scope link
       valid_lft forever preferred_lft forever
8: qtap0: <NO-CARRIER,BROADCAST,MULTICAST,PROMISC,UP> mtu 1500 qdisc
pfifo_fast master br0 state DOWN qlen 500
    link/ether d6:ce:b7:07:92:a5 brd ff:ff:ff:ff:ff:ff


--- I am really scratching my head here ... this box takes way more time
and effort than expected ...

I run sshd.socket ... and not a sshd.service ... tested sshd.service
now, no difference.

When I edit (for example) the RAM within virt-manager and click Apply
... it takes some seconds and drops the connection:


Fehler beim Anwenden von Änderungen: Interner Fehler: Client Socket ist
geschlossen

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/details.py", line 1998, in
config_apply
    ret = self.config_memory_apply()
  File "/usr/share/virt-manager/virtManager/details.py", line 2196, in
config_memory_apply
    return self._change_config_helper(df, da, hf, ha)
  File "/usr/share/virt-manager/virtManager/details.py", line 2549, in
_change_config_helper
    self.vm.refresh_xml()
  File "/usr/share/virt-manager/virtManager/libvirtobject.py", line 105,
in refresh_xml
    self._xml = self._XMLDesc(self._active_xml_flags)
  File "/usr/share/virt-manager/virtManager/domain.py", line 881, in
_XMLDesc
    return self._backend.XMLDesc(flags)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 400, in XMLDesc
    if ret is None: raise libvirtError ('virDomainGetXMLDesc() failed',
dom=self)
libvirtError: Interner Fehler: Client Socket ist geschlossen


---

I checked for permission issues.

Created group libvirt on server, edited libvirtd.conf to use it, created
my user "sgw" on the server and added it to the libvirt-group.

The URI is: qemu+ssh://sgw@jupi/system

and it connects fine ... but then drops.

I am pretty much lost right now .. tired and frustrated ... and in need
of some break ... otoh it can't be that hard to find the reason, I assume.

Stefan

Re: [gentoo-user] virt-manager and ssh

[Stefan G. Weichinger]

Am 03.10.2013 17:15, schrieb Stefan G. Weichinger:
> 
> Maybe my bridge-setup is wrong?

unlikely ... I use the same setup on at least 2 others systems w/
QEMU/KVM and systemd.

compared USE-flags ... same.

The main difference is the connection ... IPSEC and stuff ...

I tested and edited a VM on a host in my basement, same virt-manager,
ssh+pubkey ... works fast and correct, no delays, no timeout.

Very strange.

Re: [gentoo-user] virt-manager and ssh

[Stefan G. Weichinger]

Am 03.10.2013 17:32, schrieb Stefan G. Weichinger:
> Am 03.10.2013 17:15, schrieb Stefan G. Weichinger:
>>
>> Maybe my bridge-setup is wrong?
> 
> unlikely ... I use the same setup on at least 2 others systems w/
> QEMU/KVM and systemd.
> 
> compared USE-flags ... same.
> 
> The main difference is the connection ... IPSEC and stuff ...
> 
> I tested and edited a VM on a host in my basement, same virt-manager,
> ssh+pubkey ... works fast and correct, no delays, no timeout.
> 
> Very strange.

seems to be the server or the connection ...

Took the effort and installed ubuntu into a VM ;-) and virt-manager in
there. Connected to the server ... same behavior.

plain ssh session works fine and stable.

Maybe I should file a bug somewhere.

Stefan

Re: [gentoo-user] virt-manager and ssh

[Stefan G. Weichinger]

Am 03.10.2013 15:39, schrieb Michael Hampicke:

> Server side:
> 
> [ebuild   R   ~] app-emulation/libvirt-1.1.2-r3  USE="caps libvirtd
> lvm macvtap nls numa python qemu udev vepa virt-network -audit
> -avahi -firewalld -fuse -iscsi -lxc -nfs -openvz -parted -pcap
> -phyp -policykit -rbd -sasl (-selinux) -systemd -uml -virtualbox
> -xen" PYTHON_SINGLE_TARGET="python2_7 -python2_6"
> PYTHON_TARGETS="python2_7 -python2_6" 0 kB
> 
> I connect via ssh+pubkey

Would you mind sharing your libvirtd.conf as well?
Did you add a separate user/group for libvirtd on the server side?

Thanks!

Re: [gentoo-user] virt-manager and ssh

[Michael Hampicke]

[-- Attachment #1: Type: text/plain, Size: 1413 bytes --]

Am 07.10.2013 11:29, schrieb Stefan G. Weichinger:
> Am 03.10.2013 15:39, schrieb Michael Hampicke:
> 
>> Server side:
>>
>> [ebuild   R   ~] app-emulation/libvirt-1.1.2-r3  USE="caps libvirtd
>> lvm macvtap nls numa python qemu udev vepa virt-network -audit
>> -avahi -firewalld -fuse -iscsi -lxc -nfs -openvz -parted -pcap
>> -phyp -policykit -rbd -sasl (-selinux) -systemd -uml -virtualbox
>> -xen" PYTHON_SINGLE_TARGET="python2_7 -python2_6"
>> PYTHON_TARGETS="python2_7 -python2_6" 0 kB
>>
>> I connect via ssh+pubkey
> 
> Would you mind sharing your libvirtd.conf as well?
> Did you add a separate user/group for libvirtd on the server side?
> 
> Thanks!
> 
> 

Hi Stefan, I did not change my libvirtd.conf
There are no options set there, only comments.

# egrep -i 'qemu|kvm|libvirt' /etc/passwd /etc/group
/etc/passwd:qemu:x:77:77:added by portage for
libvirt:/dev/null:/sbin/nologin
/etc/group:kvm:x:78:qemu
/etc/group:qemu:x:77:


Once I had a problem with libvirt too. I remember that there was some
debug option. When enabled you could see what libvirt was doing exactly.
But I don't remeber where that options was. Maybe it was some enviroment
variable that you had to set? Something like

DEBUGOPTIONS="whatever" virsh your commands

and

DEBUGOPTIONS="whatever" libvirt --option-to-not-fork-in-background

I don't have the man page handy right now. HTH


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-user] virt-manager and ssh

[Stefan G. Weichinger]

Am 07.10.2013 19:24, schrieb Michael Hampicke:
> Am 07.10.2013 11:29, schrieb Stefan G. Weichinger:
>> Am 03.10.2013 15:39, schrieb Michael Hampicke:
>> 
>>> Server side:
>>> 
>>> [ebuild   R   ~] app-emulation/libvirt-1.1.2-r3  USE="caps
>>> libvirtd lvm macvtap nls numa python qemu udev vepa
>>> virt-network -audit -avahi -firewalld -fuse -iscsi -lxc -nfs
>>> -openvz -parted -pcap -phyp -policykit -rbd -sasl (-selinux)
>>> -systemd -uml -virtualbox -xen" PYTHON_SINGLE_TARGET="python2_7
>>> -python2_6" PYTHON_TARGETS="python2_7 -python2_6" 0 kB
>>> 
>>> I connect via ssh+pubkey
>> 
>> Would you mind sharing your libvirtd.conf as well? Did you add a
>> separate user/group for libvirtd on the server side?
>> 
>> Thanks!
>> 
>> 
> 
> Hi Stefan, I did not change my libvirtd.conf There are no options
> set there, only comments.
> 
> # egrep -i 'qemu|kvm|libvirt' /etc/passwd /etc/group 
> /etc/passwd:qemu:x:77:77:added by portage for 
> libvirt:/dev/null:/sbin/nologin /etc/group:kvm:x:78:qemu 
> /etc/group:qemu:x:77:
> 
> 
> Once I had a problem with libvirt too. I remember that there was
> some debug option. When enabled you could see what libvirt was
> doing exactly. But I don't remeber where that options was. Maybe it
> was some enviroment variable that you had to set? Something like
> 
> DEBUGOPTIONS="whatever" virsh your commands
> 
> and
> 
> DEBUGOPTIONS="whatever" libvirt --option-to-not-fork-in-background
> 
> I don't have the man page handy right now. HTH

Thanks for your reply ... I will check tomorrow when I am back at my
office (the IPSEC-VPN is locked to my static IP etc etc).

You use an URI like

qemu+ssh://server

?

I read about transports:

http://libvirt.org/remote.html#Remote_transports

and libvirtd is definitely running at the remote host.

I somehow suspect that systemd does *something* to the used sockets
... dunno.

Regards, Stefan