[gentoo-user] KVM on AMD ... why doesn't it just work?

[gentoo-user] KVM on AMD ... why doesn't it just work?

[Stefan G. Weichinger]

I try to set up KVM/QEMU on that new and shiny AMD server

24 cores of Opteron:

processor	: 23
vendor_id	: AuthenticAMD
cpu family	: 21
model		: 2
model name	: AMD Opteron(tm) Processor 6344

nice

I want gentoo-sources-3.10.7 (stable and long term supported) or
3.10.7-r1 (not yet on the machine).

I don't get /dev/kvm :-(

Right now I want this as a module "kvm_amd" or so.

# zgrep -i kvm /proc/config.gz
CONFIG_HAVE_KVM=y
CONFIG_HAVE_KVM_IRQCHIP=y
CONFIG_HAVE_KVM_IRQ_ROUTING=y
CONFIG_HAVE_KVM_EVENTFD=y
CONFIG_KVM_APIC_ARCHITECTURE=y
CONFIG_KVM_MMIO=y
CONFIG_KVM_ASYNC_PF=y
CONFIG_HAVE_KVM_MSI=y
CONFIG_HAVE_KVM_CPU_RELAX_INTERCEPT=y
CONFIG_KVM=m
CONFIG_KVM_INTEL=m
CONFIG_KVM_AMD=m

I compile the kernel with Canek's kerninst-script (see other thread).

kvm_amd does not load and gives me dozens of:

# dmesg | grep kvm  | head
[   13.460423] kvm_amd: Unknown symbol kvm_rdpmc (err 0)
[   13.460429] kvm_amd: Unknown symbol kvm_read_guest_page (err 0)
[   13.460437] kvm_amd: Unknown symbol kvm_requeue_exception (err 0)
[   13.460439] kvm_amd: Unknown symbol kvm_exit (err 0)
[   13.460441] kvm_amd: Unknown symbol kvm_init (err 0)
[   13.460444] kvm_amd: Unknown symbol kvm_enable_efer_bits (err 0)
[   13.460448] kvm_amd: Unknown symbol kvm_fast_pio_out (err 0)
[   13.460453] kvm_amd: Unknown symbol gfn_to_page (err 0)


*sigh*

With the kernel booted I now do:

cd /usr/src/linux
make clean modules modules_install

and see if I can load the module(s) then.

In general I am rather disappointed by the performance of this server. I
expected way more *bang* for the bucks ...

still unsure if that clocksource-topic might be relevant or if my kernel
config is somehow stupid.

Stefan

Re: [gentoo-user] KVM on AMD ... why doesn't it just work?

[Stefan G. Weichinger]

Am 02.10.2013 14:27, schrieb Stefan G. Weichinger:

> In general I am rather disappointed by the performance of this server. I
> expected way more *bang* for the bucks ...
> 
> still unsure if that clocksource-topic might be relevant or if my kernel
> config is somehow stupid.

Additional (for the performance issues):

# emerge --info

Portage 2.2.1 (default/linux/amd64/13.0, gcc-4.6.3, glibc-2.15-r3,
3.10.7-gentoo x86_64)
=================================================================
System uname:
Linux-3.10.7-gentoo-x86_64-AMD_Opteron-tm-_Processor_6344-with-gentoo-2.2
KiB Mem:    65965636 total,  64395888 free
KiB Swap:    8388604 total,   8388604 free
Timestamp of tree: Wed, 02 Oct 2013 00:45:01 +0000
ld GNU ld (GNU Binutils) 2.23.1
app-shells/bash:          4.2_p45
dev-lang/python:          2.7.5-r2, 3.2.5-r2
dev-util/cmake:           2.8.10.2-r2
dev-util/pkgconfig:       0.28
sys-apps/baselayout:      2.2
sys-apps/openrc:          0.11.8
sys-apps/sandbox:         2.6-r1
sys-devel/autoconf:       2.69
sys-devel/automake:       1.10.3, 1.11.6, 1.12.6, 1.13.4
sys-devel/binutils:       2.23.1
sys-devel/gcc:            4.5.4, 4.6.3
sys-devel/gcc-config:     1.7.3
sys-devel/libtool:        2.4-r1
sys-devel/make:           3.82-r4
sys-kernel/linux-headers: 3.9 (virtual/os-headers)
sys-libs/glibc:           2.15-r3
Repositories: gentoo hiro-oops-intern
ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="* -@EULA"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -march=native -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/gconf
/etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O2 -march=native -pipe"
DISTDIR="/usr/portage/distfiles"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs buildpkg config-protect-if-modified
distlocks ebuild-locks fixlafiles merge-sync news nodoc noinfo noman
notitles parallel-fetch preserve-libs protect-owned sandbox sfperms
strict unknown-features-warn unmerge-logs unmerge-orphans userfetch
userpriv usersandbox usersync"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j16"
PKGDIR="/var/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times
--omit-dir-times --compress --force --whole-file --delete --stats
--human-readable --timeout=180 --exclude=/distfiles --exclude=/local
--exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/portage"
USE="acl amd64 berkdb bzip2 cli cracklib crypt cxx dri fortran gdbm
iconv ipv6 mmx modules mudflap multilib ncurses nls nptl openmp pam pcre
readline session sse sse2 ssl systemd tcpd udev unicode zlib"
ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x
ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel
intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem
ymfpci" APACHE2_MODULES="alias cgi headers filter deflate perl"
CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon
braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load
memory rrdtool swap syslog" CURL_SSL="openssl"
DRACUT_MODULES="biosdevname caps lvm mdraid" ELIBC="glibc"
GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt
gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore
rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx"
INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad
cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text"
LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer"
OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-5"
PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_2"
RUBY_TARGETS="ruby19 ruby18" USERLAND="GNU" VIDEO_CARDS="fbdev glint
intel mach64 mga nouveau nv r128 radeon savage sis tdfx trident vesa via
vmware dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2
ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq
steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG,
LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS,
PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, SYNC, USE_PYTHON

Re: [gentoo-user] KVM on AMD ... why doesn't it just work?

[Stefan G. Weichinger]

Am 02.10.2013 14:38, schrieb Stefan G. Weichinger:
> Am 02.10.2013 14:27, schrieb Stefan G. Weichinger:
> 
>> In general I am rather disappointed by the performance of this server. I
>> expected way more *bang* for the bucks ...

And our new and ambitious friend also comes along ;-)

systemd-tmpfiles ... hogs CPU ... stopped it now ... and will look for
details in the evening.

Now the load is lower and compiling the modules is much faster. Maybe
this even solves the network-issue!?

Still the "Unknown symbol" for kvm_amd ...

Re: [gentoo-user] KVM on AMD ... why doesn't it just work?

[Kerin Millar]

On 02/10/2013 13:27, Stefan G. Weichinger wrote:
>
> I try to set up KVM/QEMU on that new and shiny AMD server
>
> 24 cores of Opteron:
>
> processor	: 23
> vendor_id	: AuthenticAMD
> cpu family	: 21
> model		: 2
> model name	: AMD Opteron(tm) Processor 6344
>
> nice
>
> I want gentoo-sources-3.10.7 (stable and long term supported) or
> 3.10.7-r1 (not yet on the machine).
>
> I don't get /dev/kvm :-(
>
> Right now I want this as a module "kvm_amd" or so.
>
> # zgrep -i kvm /proc/config.gz
> CONFIG_HAVE_KVM=y
> CONFIG_HAVE_KVM_IRQCHIP=y
> CONFIG_HAVE_KVM_IRQ_ROUTING=y
> CONFIG_HAVE_KVM_EVENTFD=y
> CONFIG_KVM_APIC_ARCHITECTURE=y
> CONFIG_KVM_MMIO=y
> CONFIG_KVM_ASYNC_PF=y
> CONFIG_HAVE_KVM_MSI=y
> CONFIG_HAVE_KVM_CPU_RELAX_INTERCEPT=y
> CONFIG_KVM=m
> CONFIG_KVM_INTEL=m
> CONFIG_KVM_AMD=m
>
> I compile the kernel with Canek's kerninst-script (see other thread).
>
> kvm_amd does not load and gives me dozens of:
>
> # dmesg | grep kvm  | head
> [   13.460423] kvm_amd: Unknown symbol kvm_rdpmc (err 0)
> [   13.460429] kvm_amd: Unknown symbol kvm_read_guest_page (err 0)
> [   13.460437] kvm_amd: Unknown symbol kvm_requeue_exception (err 0)
> [   13.460439] kvm_amd: Unknown symbol kvm_exit (err 0)
> [   13.460441] kvm_amd: Unknown symbol kvm_init (err 0)
> [   13.460444] kvm_amd: Unknown symbol kvm_enable_efer_bits (err 0)
> [   13.460448] kvm_amd: Unknown symbol kvm_fast_pio_out (err 0)
> [   13.460453] kvm_amd: Unknown symbol gfn_to_page (err 0)

Run "modinfo kvm kvm-amd" and check for discrepancies between the 
details of the two, especially regarding the vermagic field.

>
> *sigh*
>
> With the kernel booted I now do:
>
> cd /usr/src/linux
> make clean modules modules_install

I would suggest "make mrproper" to clean the source tree. Ensure that 
the .config has been backed up because it will be deleted as a result.

>
> and see if I can load the module(s) then.
>
> In general I am rather disappointed by the performance of this server. I
> expected way more *bang* for the bucks ...
>
> still unsure if that clocksource-topic might be relevant or if my kernel
> config is somehow stupid.
>
> Stefan

--Kerin

Re: [gentoo-user] KVM on AMD ... why doesn't it just work?

[Stefan G. Weichinger]

Am 02.10.2013 15:54, schrieb Kerin Millar:

> Run "modinfo kvm kvm-amd" and check for discrepancies between the
> details of the two, especially regarding the vermagic field.

[...]

> I would suggest "make mrproper" to clean the source tree. Ensure that
> the .config has been backed up because it will be deleted as a result.

I had no module "kvm" because one was built as module and one into the
kernel. I now did mrproper and recompile both kvm and kvm_amd into the
kernel. The next reboot will show ...

Thanks!

Re: [gentoo-user] KVM on AMD ... why doesn't it just work?

[Stefan G. Weichinger]

Am 02.10.2013 18:31, schrieb Stefan G. Weichinger:
> Am 02.10.2013 15:54, schrieb Kerin Millar:
> 
>> Run "modinfo kvm kvm-amd" and check for discrepancies between the
>> details of the two, especially regarding the vermagic field.
> 
> [...]
> 
>> I would suggest "make mrproper" to clean the source tree. Ensure that
>> the .config has been backed up because it will be deleted as a result.
> 
> I had no module "kvm" because one was built as module and one into the
> kernel. I now did mrproper and recompile both kvm and kvm_amd into the
> kernel. The next reboot will show ...

Rather simple issue:

I have a kernel in boot with the suffix -safe ... and this one got
listed first in grub.conf.

kerninst found that one and always set that one as default so my various
recompilings never changed the kernel itself but only the modules ...
which lead to strange mismatches ...

Now I am on a clean and mrproper 3.10.7-r1 ... WITH /dev/kvm !

Thanks!

And the performance of the first VM looks good ... even with the vmdk
... conversion will follow.

Stefan

Re: [gentoo-user] KVM on AMD ... why doesn't it just work?

[Kerin Millar]

On 02/10/2013 17:47, Stefan G. Weichinger wrote:
> Am 02.10.2013 18:31, schrieb Stefan G. Weichinger:
>> Am 02.10.2013 15:54, schrieb Kerin Millar:
>>
>>> Run "modinfo kvm kvm-amd" and check for discrepancies between the
>>> details of the two, especially regarding the vermagic field.
>>
>> [...]
>>
>>> I would suggest "make mrproper" to clean the source tree. Ensure that
>>> the .config has been backed up because it will be deleted as a result.
>>
>> I had no module "kvm" because one was built as module and one into the
>> kernel. I now did mrproper and recompile both kvm and kvm_amd into the
>> kernel. The next reboot will show ...
>
> Rather simple issue:
>
> I have a kernel in boot with the suffix -safe ... and this one got
> listed first in grub.conf.
>
> kerninst found that one and always set that one as default so my various
> recompilings never changed the kernel itself but only the modules ...
> which lead to strange mismatches ...

You could customize EXTRAVERSION for your safe kernel to prevent this 
from happening. It can be modified in the Makefile or - if I recall 
correctly - by creating a file named "localversion-safe" in the root of 
the source tree and enabling CONFIG_LOCALVERSION. That way, its module 
directory would be distinct.

>
> Now I am on a clean and mrproper 3.10.7-r1 ... WITH /dev/kvm !
>
> Thanks!
>
> And the performance of the first VM looks good ... even with the vmdk
> ... conversion will follow.
>
> Stefan
>
>

Re: [gentoo-user] KVM on AMD ... why doesn't it just work?

[Canek Peláez Valdés]

On Wed, Oct 2, 2013 at 11:47 AM, Stefan G. Weichinger <lists@xunil.at> wrote:
> Am 02.10.2013 18:31, schrieb Stefan G. Weichinger:
>> Am 02.10.2013 15:54, schrieb Kerin Millar:
>>
>>> Run "modinfo kvm kvm-amd" and check for discrepancies between the
>>> details of the two, especially regarding the vermagic field.
>>
>> [...]
>>
>>> I would suggest "make mrproper" to clean the source tree. Ensure that
>>> the .config has been backed up because it will be deleted as a result.
>>
>> I had no module "kvm" because one was built as module and one into the
>> kernel. I now did mrproper and recompile both kvm and kvm_amd into the
>> kernel. The next reboot will show ...
>
> Rather simple issue:
>
> I have a kernel in boot with the suffix -safe ... and this one got
> listed first in grub.conf.

kerninst is pretty stupid. To create the list of kernels (in GRUB, not
GRUB2), it simply lists the kernels in /boot, sorts them inverted by
version, and then uses that order for the listing.

Therefore, if you have:

vmlinuz-3.9
vmlinuz-3.8
vmlinuz-3.10
vmlinuz-blah

the inverted sort by version will be:

vmlinuz-blah
vmlinuz-3.10
vmlinuz-3.9
vmlinuz-3.8

and the kernel with an alphanumeric "version" will be always be the default.

> kerninst found that one and always set that one as default so my various
> recompilings never changed the kernel itself but only the modules ...
> which lead to strange mismatches ...

That should explain everything.

As I said before, kerninst was not written to test configurations; is
for when you have nailed the configuration and want to automatize the
kernel update.

Regards.
-- 
Canek Peláez Valdés
Posgrado en Ciencia e Ingeniería de la Computación
Universidad Nacional Autónoma de México

Re: [gentoo-user] KVM on AMD ... why doesn't it just work?

[Stefan G. Weichinger]

Am 02.10.2013 19:11, schrieb Canek Peláez Valdés:

> the inverted sort by version will be:
> 
> vmlinuz-blah
> vmlinuz-3.10
> vmlinuz-3.9
> vmlinuz-3.8
> 
> and the kernel with an alphanumeric "version" will be always be the default.

Exactly what I saw, yes. No complaints ... ok with me, I should have
checked my grub.conf as well (and did in the end).

>> kerninst found that one and always set that one as default so my various
>> recompilings never changed the kernel itself but only the modules ...
>> which lead to strange mismatches ...
> 
> That should explain everything.

> As I said before, kerninst was not written to test configurations; is
> for when you have nailed the configuration and want to automatize the
> kernel update.

Yep.

I just want to settle on one method to generate the kernel(s) even on
that customer server. Right now I am still tuning and toggling around ;-)

S

Re: [gentoo-user] KVM on AMD ... why doesn't it just work?

[Stefan G. Weichinger]

Am 02.10.2013 14:45, schrieb Stefan G. Weichinger:
> Am 02.10.2013 14:38, schrieb Stefan G. Weichinger:
>> Am 02.10.2013 14:27, schrieb Stefan G. Weichinger:
>>
>>> In general I am rather disappointed by the performance of this server. I
>>> expected way more *bang* for the bucks ...
> 
> And our new and ambitious friend also comes along ;-)
> 
> systemd-tmpfiles ... hogs CPU ... stopped it now ... and will look for
> details in the evening.

that one is still unsolved ... although since the last booting (now with
systemd-204-r1) I don't see that hogging process anymore.

Maybe something in the changelog:

http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/sys-apps/systemd/ChangeLog?view=markup

the tmpfiles-related stuff seems for 206 only.

Anyway, it seems to work now. great.

Re: [gentoo-user] KVM on AMD ... why doesn't it just work?

[Bruce Hill]

On Wed, Oct 02, 2013 at 06:47:21PM +0200, Stefan G. Weichinger wrote:
> 
> Rather simple issue:
> 
> I have a kernel in boot with the suffix -safe ... and this one got
> listed first in grub.conf.
> 
> kerninst found that one and always set that one as default so my various
> recompilings never changed the kernel itself but only the modules ...
> which lead to strange mismatches ...
> 
> Now I am on a clean and mrproper 3.10.7-r1 ... WITH /dev/kvm !
> 
> Thanks!
> 
> And the performance of the first VM looks good ... even with the vmdk
> ... conversion will follow.
> 
> Stefan

rm kerninst
less /usr/src/linux/README (proper instructions)

wash, rinse, repeat :-)
-- 
Happy Penguin Computers               >')
126 Fenco Drive                       ( \
Tupelo, MS 38801                       ^^
support@happypenguincomputers.com
662-269-2706 662-205-6424
http://happypenguincomputers.com/

A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?

Don't top-post: http://en.wikipedia.org/wiki/Top_post#Top-posting

Re: [gentoo-user] KVM on AMD ... why doesn't it just work?

[Stefan G. Weichinger]

Am 02.10.2013 20:51, schrieb Bruce Hill:

> rm kerninst
> less /usr/src/linux/README (proper instructions)
> 
> wash, rinse, repeat :-)

;-)  ... I take this as an option ;-)

Re: [gentoo-user] KVM on AMD ... why doesn't it just work?

[Stefan G. Weichinger]

Am 02.10.2013 14:38, schrieb Stefan G. Weichinger:

> CFLAGS="-O2 -march=native -pipe"

Do you agree with me on these CFLAGS ?

AMD Opteron 6344 ...

And I assume I should rebuild the kernel with CONFIG_MK8 ?

Thanks, Stefan