[gentoo-user] Dual or Quad CPU complications?

[gentoo-user] Dual or Quad CPU complications?

[Grant]

[-- Attachment #1: Type: text/plain, Size: 278 bytes --]

I've only ever used systems with a single CPU.  I'm looking for a new host
for a dedicated server (suggestions?) and it looks like I'll probably
choose a machine with two or four CPUs.  What sort of complications does
that add to set up and/or maintenance with Gentoo?

- Grant

[-- Attachment #2: Type: text/html, Size: 317 bytes --]

Re: [gentoo-user] Dual or Quad CPU complications?

[Alan McKinnon]

On Wed, 12 Dec 2012 22:12:18 -0800
Grant <emailgrant@gmail.com> wrote:

> I've only ever used systems with a single CPU.  I'm looking for a new
> host for a dedicated server (suggestions?) and it looks like I'll
> probably choose a machine with two or four CPUs.  What sort of
> complications does that add to set up and/or maintenance with Gentoo?

No complication.

Configure CONFIG_SMP in the the kernel for multicore.
Everything else is transparent.

Cores make threads work better, so you'd want to investigate if
USE="threads" is useful for you.


-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] Dual or Quad CPU complications?

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 701 bytes --]

Am 13.12.2012 07:23, schrieb Alan McKinnon:
> On Wed, 12 Dec 2012 22:12:18 -0800
> Grant <emailgrant@gmail.com> wrote:
> 
>> I've only ever used systems with a single CPU.  I'm looking for a new
>> host for a dedicated server (suggestions?) and it looks like I'll
>> probably choose a machine with two or four CPUs.  What sort of
>> complications does that add to set up and/or maintenance with Gentoo?
> 
> No complication.
> 
> Configure CONFIG_SMP in the the kernel for multicore.
> Everything else is transparent.
> 
> Cores make threads work better, so you'd want to investigate if
> USE="threads" is useful for you.
> 
> 

I think he's looking for advice on NUMA, not SMP.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [gentoo-user] Dual or Quad CPU complications?

[J. Roeleveld]

[-- Attachment #1: Type: text/plain, Size: 1056 bytes --]

Florian Philipp <lists@binarywings.net> wrote:

>Am 13.12.2012 07:23, schrieb Alan McKinnon:
>> On Wed, 12 Dec 2012 22:12:18 -0800
>> Grant <emailgrant@gmail.com> wrote:
>> 
>>> I've only ever used systems with a single CPU.  I'm looking for a
>new
>>> host for a dedicated server (suggestions?) and it looks like I'll
>>> probably choose a machine with two or four CPUs.  What sort of
>>> complications does that add to set up and/or maintenance with
>Gentoo?
>> 
>> No complication.
>> 
>> Configure CONFIG_SMP in the the kernel for multicore.
>> Everything else is transparent.
>> 
>> Cores make threads work better, so you'd want to investigate if
>> USE="threads" is useful for you.
>> 
>> 
>
>I think he's looking for advice on NUMA, not SMP.

NUMA is also an option in the kernel. Should also be fully transparent.
I got one machine with NUMA and only had to set an option for it.

Does anyone know how to check it's working properly?

--
Joost
-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

[-- Attachment #2: Type: text/html, Size: 1741 bytes --]

Re: [gentoo-user] Dual or Quad CPU complications?

[Luis Gustavo Vilela de Oliveira]

[-- Attachment #1: Type: text/plain, Size: 1269 bytes --]

I believe NUMA is only used on multiprocessor machine and not on only
multicore.

http://en.wikipedia.org/wiki/Non-Uniform_Memory_Access


2012/12/13 J. Roeleveld <joost@antarean.org>

> Florian Philipp <lists@binarywings.net> wrote:
>>
>> Am 13.12.2012 07:23, schrieb Alan McKinnon:
>>
>>>
>>> On Wed, 12 Dec 2012 22:12:18 -0800
>>> Grant <emailgrant@gmail.com> wrote:
>>>
>>>
>>>> I've only ever used systems with a single CPU.  I'm looking for a new
>>>> host for a dedicated server (suggestions?) and it looks like I'll
>>>> probably choose a machine with two or four CPUs.  What sort of
>>>> complications does that add to set up and/or maintenance with Gentoo?
>>>
>>>
>>> No complication.
>>>
>>> Configure CONFIG_SMP in the the kernel for multicore.
>>> Everything else is transparent.
>>>
>>> Cores make threads work better, so you'd want to investigate if
>>> USE="threads" is
>>> useful for you.
>>
>>
>>
>>
>> I think he's looking for advice on NUMA, not SMP.
>>
>>
> NUMA is also an option in the kernel. Should also be fully transparent.
> I got one machine with NUMA and only had to set an option for it.
>
> Does anyone know how to check it's working properly?
>
> --
> Joost
> --
> Sent from my Android phone with K-9 Mail. Please excuse my brevity.
>

[-- Attachment #2: Type: text/html, Size: 2449 bytes --]

Re: [gentoo-user] Dual or Quad CPU complications?

[Rafa Griman]

On Thu, Dec 13, 2012 at 2:01 PM, Luis Gustavo Vilela de Oliveira
<luisgustavo.vilela@gmail.com> wrote:
> I believe NUMA is only used on multiprocessor machine and not on only
> multicore.


NUMA's about memory access so it's about
cores/CPUs/processors/whatever_you_want_to_call_it and how they access
memory.

[...]

>> NUMA is also an option in the kernel. Should also be fully transparent.
>> I got one machine with NUMA and only had to set an option for it.
>>
>> Does anyone know how to check it's working properly?


You could use numactl and place your binary on certain nodes (cores)
and check whether that works.

You should also check your code uses OpenMP, threading, ... for example.

Monitor your CPU and memory also while your software is running ;)

What SW are you going to be running? Do you know if your software is
SMP and/or NUMA aware?

   Rafa

Re: [gentoo-user] Dual or Quad CPU complications?

[Bruce Hill]

On Thu, Dec 13, 2012 at 08:44:45AM +0100, J. Roeleveld wrote:
> 
> NUMA is also an option in the kernel. Should also be fully transparent.
> I got one machine with NUMA and only had to set an option for it.
> 
> Does anyone know how to check it's working properly?

dmesg | grep NUMA
-- 
Happy Penguin Computers               >')
126 Fenco Drive                       ( \
Tupelo, MS 38801                       ^^
support@happypenguincomputers.com
662-269-2706 662-205-6424
http://happypenguincomputers.com/

Don't top-post: http://en.wikipedia.org/wiki/Top_post#Top-posting

Re: [gentoo-user] Dual or Quad CPU complications?

[Mick]

[-- Attachment #1: Type: Text/Plain, Size: 934 bytes --]

On Thursday 13 Dec 2012 14:13:56 Bruce Hill wrote:
> On Thu, Dec 13, 2012 at 08:44:45AM +0100, J. Roeleveld wrote:
> > NUMA is also an option in the kernel. Should also be fully transparent.
> > I got one machine with NUMA and only had to set an option for it.
> > 
> > Does anyone know how to check it's working properly?
> 
> dmesg | grep NUMA

Hmm ... it seems that it can't find NUMA configuration:

$ dmesg | grep UMA
No NUMA configuration found

Am I supposed to configure something in userspace?  This is what the kernel 
has:

$ uname -a
Linux dell_xps 3.5.7-gentoo #2 SMP PREEMPT Mon Nov 26 10:36:47 GMT 2012 x86_64 
Intel(R) Core(TM) i7 CPU Q 720 @ 1.60GHz GenuineIntel GNU/Linux

$ cat /usr/src/linux/.config | grep -i NUMA
CONFIG_NUMA=y
# CONFIG_AMD_NUMA is not set
CONFIG_X86_64_ACPI_NUMA=y
# CONFIG_NUMA_EMU is not set
CONFIG_USE_PERCPU_NUMA_NODE_ID=y
CONFIG_ACPI_NUMA=y
-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Dual or Quad CPU complications?

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 1176 bytes --]

Am 15.12.2012 01:40, schrieb Mick:
> On Thursday 13 Dec 2012 14:13:56 Bruce Hill wrote:
>> On Thu, Dec 13, 2012 at 08:44:45AM +0100, J. Roeleveld wrote:
>>> NUMA is also an option in the kernel. Should also be fully transparent.
>>> I got one machine with NUMA and only had to set an option for it.
>>>
>>> Does anyone know how to check it's working properly?
>>
>> dmesg | grep NUMA
> 
> Hmm ... it seems that it can't find NUMA configuration:
> 
> $ dmesg | grep UMA
> No NUMA configuration found
> 
> Am I supposed to configure something in userspace?  This is what the kernel 
> has:
> 
> $ uname -a
> Linux dell_xps 3.5.7-gentoo #2 SMP PREEMPT Mon Nov 26 10:36:47 GMT 2012 x86_64 
> Intel(R) Core(TM) i7 CPU Q 720 @ 1.60GHz GenuineIntel GNU/Linux
> 
[...]

"dell_xps" as in XPS laptop? There are no NUMA laptops.

Despite all the stuff about terminology, we are basically talking about
multi-socket systems. Things with mainboards like these [1] as opposed
to these [2].

[1] http://www.newegg.com/Product/Product.aspx?Item=N82E16813131378
[2] http://www.newegg.com/Product/Product.aspx?Item=N82E16813131725

Regards,
Florian Philipp


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [gentoo-user] Dual or Quad CPU complications?

[Grant]

[-- Attachment #1: Type: text/plain, Size: 776 bytes --]

> >> I've only ever used systems with a single CPU.  I'm looking for a new
> >> host for a dedicated server (suggestions?) and it looks like I'll
> >> probably choose a machine with two or four CPUs.  What sort of
> >> complications does that add to set up and/or maintenance with Gentoo?
> >
> > No complication.
> >
> > Configure CONFIG_SMP in the the kernel for multicore.
> > Everything else is transparent.
> >
> > Cores make threads work better, so you'd want to investigate if
> > USE="threads" is useful for you.
> >
> >
>
> I think he's looking for advice on NUMA, not SMP.

So if I have 2 physical CPU's with 4 cores each and I enable SMP, I'm using
8 cores?  Can NUMA be either enabled or disabled when using more than one
physical CPU, or is it required?

- Grant

[-- Attachment #2: Type: text/html, Size: 1011 bytes --]

Re: [gentoo-user] Dual or Quad CPU complications?

[Rafa Griman]

On Fri, Dec 14, 2012 at 8:43 AM, Grant <emailgrant@gmail.com> wrote:

[...]

> So if I have 2 physical CPU's with 4 cores each and I enable SMP, I'm using
> 8 cores?  Can NUMA be either enabled or disabled when using more than one
> physical CPU, or is it required?


NUMA is a hardware architecture. It's how you access memory on a
hardware level: NUMA = Non Uniform Memory Access vs a UMA architecture
of typical (old/legacy) SMP systems (UMA = Uniform Memory Access).

In a UMA system, all the memory belongs to all the sockets. In a NUMA
system, each socket has it's "own" local memory. In modern (x86-64)
processors, each socket has it's own memory controller so each socket
controls its own local memory. If one socket runs out of memory it can
ask another socket to lend him some memory. In a UMA system, no socket
has to ask since memory is global and belongs to all sockets so if one
socket uses up all the memory ... the rest "starve". In NUMA, there's
more control over who uses what (be it cores or RAM).

If you have a modern dual or quad (or higher #) socket system ...
you've got NUMA architecture and you can't get rid of it, it's
hardware, not software.

NUMA is not bad nor good. It's "transparent" to you. If your SW
supports threads, OpenMP, ... you'll be using it without knowing. That
doesn't mean you can't tweak performance and use numactl tools,
cgroups, ... to increase performance. You can :)

HTH

   Rafa

Re: [gentoo-user] Dual or Quad CPU complications?

[Grant]

[-- Attachment #1: Type: text/plain, Size: 1523 bytes --]

> > So if I have 2 physical CPU's with 4 cores each and I enable SMP, I'm
using
> > 8 cores?  Can NUMA be either enabled or disabled when using more than
one
> > physical CPU, or is it required?
>
>
> NUMA is a hardware architecture. It's how you access memory on a
> hardware level: NUMA = Non Uniform Memory Access vs a UMA architecture
> of typical (old/legacy) SMP systems (UMA = Uniform Memory Access).
>
> In a UMA system, all the memory belongs to all the sockets. In a NUMA
> system, each socket has it's "own" local memory. In modern (x86-64)
> processors, each socket has it's own memory controller so each socket
> controls its own local memory. If one socket runs out of memory it can
> ask another socket to lend him some memory. In a UMA system, no socket
> has to ask since memory is global and belongs to all sockets so if one
> socket uses up all the memory ... the rest "starve". In NUMA, there's
> more control over who uses what (be it cores or RAM).
>
> If you have a modern dual or quad (or higher #) socket system ...
> you've got NUMA architecture and you can't get rid of it, it's
> hardware, not software.

So I must enable CONFIG_NUMA for more than one physical CPU, and disable it
for only one physical CPU?

- Grant


> NUMA is not bad nor good. It's "transparent" to you. If your SW
> supports threads, OpenMP, ... you'll be using it without knowing. That
> doesn't mean you can't tweak performance and use numactl tools,
> cgroups, ... to increase performance. You can :)
>
> HTH
>
>    Rafa

[-- Attachment #2: Type: text/html, Size: 1847 bytes --]

Re: [gentoo-user] Dual or Quad CPU complications?

[Rafa Griman]

On Fri, Dec 14, 2012 at 10:44 AM, Grant <emailgrant@gmail.com> wrote:
>> > So if I have 2 physical CPU's with 4 cores each and I enable SMP, I'm
>> > using
>> > 8 cores?  Can NUMA be either enabled or disabled when using more than
>> > one
>> > physical CPU, or is it required?
>>
>>
>> NUMA is a hardware architecture. It's how you access memory on a
>> hardware level: NUMA = Non Uniform Memory Access vs a UMA architecture
>> of typical (old/legacy) SMP systems (UMA = Uniform Memory Access).
>>
>> In a UMA system, all the memory belongs to all the sockets. In a NUMA
>> system, each socket has it's "own" local memory. In modern (x86-64)
>> processors, each socket has it's own memory controller so each socket
>> controls its own local memory. If one socket runs out of memory it can
>> ask another socket to lend him some memory. In a UMA system, no socket
>> has to ask since memory is global and belongs to all sockets so if one
>> socket uses up all the memory ... the rest "starve". In NUMA, there's
>> more control over who uses what (be it cores or RAM).
>>
>> If you have a modern dual or quad (or higher #) socket system ...
>> you've got NUMA architecture and you can't get rid of it, it's
>> hardware, not software.
>
> So I must enable CONFIG_NUMA for more than one physical CPU, and disable it
> for only one physical CPU?


Yup. But ... Why would you want to disable a socket (CPU)? If you
disable a socket (CPU) ... you lose the memory attached to that socket
(CPU) not to mention you lose those cores ;)

A better solution would be to use cgroups or numactl tools to pin a
certain process to a set of cores and a memory region.

If you really want to deactivate cores (but not the whole socket), you can type:

          echo 0 > /sys/devices/system/cpu/cpu1/online

This would deactivate core #1. You can deactivate as many cores as you
wish, except for core #0.

This can be done without rebooting your server (aka during run time).
Your memory will not be affected, but you will have less cores (and
theoretically more memory bandwidth). I say "theoretically" because
you always have to benchmark these things with YOUR application
(remember logic NEVER applies to real life ;)

If you want to check the # of cores you've got:

     cat /proc/interrupts | grep CPU

Other possibilities such as cat /proc/cpuinfo or dmesg, ... can be
useful too for this: your choice, FLOSS gives you options.

If you want to activate the previously deactivated core, you can run:

          echo 1 > /sys/devices/system/cpu/cpu1/online

Now ... be sure your core numbering is the expected core numbering.
IOW, not all server vendors follow the same numbering scheme so core
#1 in vendor A's server could be core #2 in vendor B's server. Never
trust logic ;)

As I mentioned previously: test/benchmark YOUR software. DON'T trust
logic or generic benchmarks or web pages with results. Trust YOUR
results only.

HTH

   Rafa

Re: [gentoo-user] Dual or Quad CPU complications?

[Grant]

[-- Attachment #1: Type: text/plain, Size: 3090 bytes --]

> >> > So if I have 2 physical CPU's with 4 cores each and I enable SMP, I'm
> >> > using
> >> > 8 cores?  Can NUMA be either enabled or disabled when using more than
> >> > one
> >> > physical CPU, or is it required?
> >>
> >>
> >> NUMA is a hardware architecture. It's how you access memory on a
> >> hardware level: NUMA = Non Uniform Memory Access vs a UMA architecture
> >> of typical (old/legacy) SMP systems (UMA = Uniform Memory Access).
> >>
> >> In a UMA system, all the memory belongs to all the sockets. In a NUMA
> >> system, each socket has it's "own" local memory. In modern (x86-64)
> >> processors, each socket has it's own memory controller so each socket
> >> controls its own local memory. If one socket runs out of memory it can
> >> ask another socket to lend him some memory. In a UMA system, no socket
> >> has to ask since memory is global and belongs to all sockets so if one
> >> socket uses up all the memory ... the rest "starve". In NUMA, there's
> >> more control over who uses what (be it cores or RAM).
> >>
> >> If you have a modern dual or quad (or higher #) socket system ...
> >> you've got NUMA architecture and you can't get rid of it, it's
> >> hardware, not software.
> >
> > So I must enable CONFIG_NUMA for more than one physical CPU, and
disable it
> > for only one physical CPU?
>
>
> Yup. But ... Why would you want to disable a socket (CPU)? If you
> disable a socket (CPU) ... you lose the memory attached to that socket
> (CPU) not to mention you lose those cores ;)

Sure but it sounds like if my system only has one CPU socket, CONFIG_NUMA
should be disabled.

- Grant


> A better solution would be to use cgroups or numactl tools to pin a
> certain process to a set of cores and a memory region.
>
> If you really want to deactivate cores (but not the whole socket), you
can type:
>
>           echo 0 > /sys/devices/system/cpu/cpu1/online
>
> This would deactivate core #1. You can deactivate as many cores as you
> wish, except for core #0.
>
> This can be done without rebooting your server (aka during run time).
> Your memory will not be affected, but you will have less cores (and
> theoretically more memory bandwidth). I say "theoretically" because
> you always have to benchmark these things with YOUR application
> (remember logic NEVER applies to real life ;)
>
> If you want to check the # of cores you've got:
>
>      cat /proc/interrupts | grep CPU
>
> Other possibilities such as cat /proc/cpuinfo or dmesg, ... can be
> useful too for this: your choice, FLOSS gives you options.
>
> If you want to activate the previously deactivated core, you can run:
>
>           echo 1 > /sys/devices/system/cpu/cpu1/online
>
> Now ... be sure your core numbering is the expected core numbering.
> IOW, not all server vendors follow the same numbering scheme so core
> #1 in vendor A's server could be core #2 in vendor B's server. Never
> trust logic ;)
>
> As I mentioned previously: test/benchmark YOUR software. DON'T trust
> logic or generic benchmarks or web pages with results. Trust YOUR
> results only.
>
> HTH
>
>    Rafa

[-- Attachment #2: Type: text/html, Size: 3835 bytes --]

Re: [gentoo-user] Dual or Quad CPU complications?

[Grant]

[-- Attachment #1: Type: text/plain, Size: 2097 bytes --]

> > >> > So if I have 2 physical CPU's with 4 cores each and I enable SMP,
I'm
> > >> > using
> > >> > 8 cores?  Can NUMA be either enabled or disabled when using more
than
> > >> > one
> > >> > physical CPU, or is it required?
> > >>
> > >>
> > >> NUMA is a hardware architecture. It's how you access memory on a
> > >> hardware level: NUMA = Non Uniform Memory Access vs a UMA
architecture
> > >> of typical (old/legacy) SMP systems (UMA = Uniform Memory Access).
> > >>
> > >> In a UMA system, all the memory belongs to all the sockets. In a NUMA
> > >> system, each socket has it's "own" local memory. In modern (x86-64)
> > >> processors, each socket has it's own memory controller so each socket
> > >> controls its own local memory. If one socket runs out of memory it
can
> > >> ask another socket to lend him some memory. In a UMA system, no
socket
> > >> has to ask since memory is global and belongs to all sockets so if
one
> > >> socket uses up all the memory ... the rest "starve". In NUMA, there's
> > >> more control over who uses what (be it cores or RAM).
> > >>
> > >> If you have a modern dual or quad (or higher #) socket system ...
> > >> you've got NUMA architecture and you can't get rid of it, it's
> > >> hardware, not software.
> > >
> > > So I must enable CONFIG_NUMA for more than one physical CPU, and
disable it
> > > for only one physical CPU?
> >
> >
> > Yup. But ... Why would you want to disable a socket (CPU)? If you
> > disable a socket (CPU) ... you lose the memory attached to that socket
> > (CPU) not to mention you lose those cores ;)
>
> Sure but it sounds like if my system only has one CPU socket, CONFIG_NUMA
should be disabled.

I read this in make menuconfig:

"The kernel will try to allocate memory used by a CPU on the local memory
controller of the CPU and add some more NUMA awareness to the kernel. For
64-bit this is recommended if the system is Intel Core i7 (or later), AMD
Opteron, or EM64T NUMA."

To be sure I have this right, I should disable CONFIG_NUMA on any system
with a single physical CPU, even if it's an AMD Opteron?

- Grant

[-- Attachment #2: Type: text/html, Size: 2718 bytes --]

Re: [gentoo-user] Dual or Quad CPU complications?

[Michael Mol]

On Fri, Dec 14, 2012 at 10:16 PM, Grant <emailgrant@gmail.com> wrote:
>> > >> > So if I have 2 physical CPU's with 4 cores each and I enable SMP,
>> > >> > I'm
>> > >> > using
>> > >> > 8 cores?  Can NUMA be either enabled or disabled when using more
>> > >> > than
>> > >> > one
>> > >> > physical CPU, or is it required?
>> > >>
>> > >>
>> > >> NUMA is a hardware architecture. It's how you access memory on a
>> > >> hardware level: NUMA = Non Uniform Memory Access vs a UMA
>> > >> architecture
>> > >> of typical (old/legacy) SMP systems (UMA = Uniform Memory Access).
>> > >>
>> > >> In a UMA system, all the memory belongs to all the sockets. In a NUMA
>> > >> system, each socket has it's "own" local memory. In modern (x86-64)
>> > >> processors, each socket has it's own memory controller so each socket
>> > >> controls its own local memory. If one socket runs out of memory it
>> > >> can
>> > >> ask another socket to lend him some memory. In a UMA system, no
>> > >> socket
>> > >> has to ask since memory is global and belongs to all sockets so if
>> > >> one
>> > >> socket uses up all the memory ... the rest "starve". In NUMA, there's
>> > >> more control over who uses what (be it cores or RAM).
>> > >>
>> > >> If you have a modern dual or quad (or higher #) socket system ...
>> > >> you've got NUMA architecture and you can't get rid of it, it's
>> > >> hardware, not software.
>> > >
>> > > So I must enable CONFIG_NUMA for more than one physical CPU, and
>> > > disable it
>> > > for only one physical CPU?
>> >
>> >
>> > Yup. But ... Why would you want to disable a socket (CPU)? If you
>> > disable a socket (CPU) ... you lose the memory attached to that socket
>> > (CPU) not to mention you lose those cores ;)
>>
>> Sure but it sounds like if my system only has one CPU socket, CONFIG_NUMA
>> should be disabled.
>
> I read this in make menuconfig:
>
> "The kernel will try to allocate memory used by a CPU on the local memory
> controller of the CPU and add some more NUMA awareness to the kernel. For
> 64-bit this is recommended if the system is Intel Core i7 (or later), AMD
> Opteron, or EM64T NUMA."
>
> To be sure I have this right, I should disable CONFIG_NUMA on any system
> with a single physical CPU, even if it's an AMD Opteron?

No harm done if you enable NUMA on a system where it's not necessary.

--
:wq

Re: [gentoo-user] Dual or Quad CPU complications?

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 2396 bytes --]

Am 15.12.2012 04:16, schrieb Grant:
>> > >> > So if I have 2 physical CPU's with 4 cores each and I enable
> SMP, I'm
>> > >> > using
>> > >> > 8 cores?  Can NUMA be either enabled or disabled when using
> more than
>> > >> > one
>> > >> > physical CPU, or is it required?
>> > >>
>> > >>
>> > >> NUMA is a hardware architecture. It's how you access memory on a
>> > >> hardware level: NUMA = Non Uniform Memory Access vs a UMA
> architecture
>> > >> of typical (old/legacy) SMP systems (UMA = Uniform Memory Access).
>> > >>
>> > >> In a UMA system, all the memory belongs to all the sockets. In a NUMA
>> > >> system, each socket has it's "own" local memory. In modern (x86-64)
>> > >> processors, each socket has it's own memory controller so each socket
>> > >> controls its own local memory. If one socket runs out of memory
> it can
>> > >> ask another socket to lend him some memory. In a UMA system, no
> socket
>> > >> has to ask since memory is global and belongs to all sockets so
> if one
>> > >> socket uses up all the memory ... the rest "starve". In NUMA, there's
>> > >> more control over who uses what (be it cores or RAM).
>> > >>
>> > >> If you have a modern dual or quad (or higher #) socket system ...
>> > >> you've got NUMA architecture and you can't get rid of it, it's
>> > >> hardware, not software.
>> > >
>> > > So I must enable CONFIG_NUMA for more than one physical CPU, and
> disable it
>> > > for only one physical CPU?
>> >
>> >
>> > Yup. But ... Why would you want to disable a socket (CPU)? If you
>> > disable a socket (CPU) ... you lose the memory attached to that socket
>> > (CPU) not to mention you lose those cores ;)
>>
>> Sure but it sounds like if my system only has one CPU socket,
> CONFIG_NUMA should be disabled.
> 
> I read this in make menuconfig:
> 
> "The kernel will try to allocate memory used by a CPU on the local
> memory controller of the CPU and add some more NUMA awareness to the
> kernel. For 64-bit this is recommended if the system is Intel Core
> i7 (or later), AMD Opteron, or EM64T NUMA."
> 
> To be sure I have this right, I should disable CONFIG_NUMA on any system
> with a single physical CPU, even if it's an AMD Opteron?
> 
> - Grant

Disable it. You only have one memory controller. There is nothing the
kernel could do wrong without.

Regards,
Florian Philipp


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [gentoo-user] Dual or Quad CPU complications?

[Volker Armin Hemmann]

Am Freitag, 14. Dezember 2012, 01:44:26 schrieb Grant:
> > > So if I have 2 physical CPU's with 4 cores each and I enable SMP, I'm
> 
> using
> 
> > > 8 cores?  Can NUMA be either enabled or disabled when using more than
> 
> one
> 
> > > physical CPU, or is it required?
> > 
> > NUMA is a hardware architecture. It's how you access memory on a
> > hardware level: NUMA = Non Uniform Memory Access vs a UMA architecture
> > of typical (old/legacy) SMP systems (UMA = Uniform Memory Access).
> > 
> > In a UMA system, all the memory belongs to all the sockets. In a NUMA
> > system, each socket has it's "own" local memory. In modern (x86-64)
> > processors, each socket has it's own memory controller so each socket
> > controls its own local memory. If one socket runs out of memory it can
> > ask another socket to lend him some memory. In a UMA system, no socket
> > has to ask since memory is global and belongs to all sockets so if one
> > socket uses up all the memory ... the rest "starve". In NUMA, there's
> > more control over who uses what (be it cores or RAM).
> > 
> > If you have a modern dual or quad (or higher #) socket system ...
> > you've got NUMA architecture and you can't get rid of it, it's
> > hardware, not software.
> 
> So I must enable CONFIG_NUMA for more than one physical CPU, and disable it
> for only one physical CPU?

you never need numa for one cpu. Ok?

And even if you have several, you will probably never need it.

-- 
#163933

[gentoo-user] Re: Dual or Quad CPU complications?

[James]

Grant <emailgrant <at> gmail.com> writes:


> I've only ever used systems with a single CPU.  I'm looking for a new host for
a dedicated server (suggestions?) and it looks like I'll probably choose a
machine with two or four CPUs.  

NUMA is specialization, imho:
http://pubs.vmware.com/vsphere-4-esx-vcenter/index.jsp?topic=/com.vmware.vsphere.resourcemanagement.doc_41/using_numa_systems_with_esx_esxi/c_what_is_numa.html

The more cores the better. 6 and 8 are readily available.
The 6 core AMD near 4 GHz is the sweet spot, imho.
Here is a 4 core on sale at Newegg: AMD FX-4170 Zambezi 4.2GHz 

If you run a feature rich desktop (kde, gnome, etc) then the more cores the
better. Compiling code is much faster and you can still have a snappy
desk top. Most gentoo folks compile quite a bit of code, depending on your
updates and how often you experiment with new features or software.

I'm setting up some new FX-8350 machines, but fully flushed out, there
around a 1K (USD). Surely you can replace a mobo with a quad and as much ram as
will fit, and get a fine machine. CPU speed, for me, is the dominate feature,
when you are only doing a few things for a snappy workstation. Lots of cores and
low CPU speed and low ram, sucks, imho. Max amount and max speed of the RAM
is the killer performance edge for most workstations, imho.

It boils down to a personal decision. The world of software
is migrating to multi-threading, so the more cores, the
more future-proof, imho.

hth,
James

Re: [gentoo-user] Re: Dual or Quad CPU complications?

[Rafa Griman]

On Thu, Dec 13, 2012 at 4:55 PM, James <wireless@tampabay.rr.com> wrote:
> Grant <emailgrant <at> gmail.com> writes:
>
>
>> I've only ever used systems with a single CPU.  I'm looking for a new host for
> a dedicated server (suggestions?) and it looks like I'll probably choose a
> machine with two or four CPUs.
>
> NUMA is specialization, imho:
> http://pubs.vmware.com/vsphere-4-esx-vcenter/index.jsp?topic=/com.vmware.vsphere.resourcemanagement.doc_41/using_numa_systems_with_esx_esxi/c_what_is_numa.html
>
> The more cores the better. 6 and 8 are readily available.
> The 6 core AMD near 4 GHz is the sweet spot, imho.
> Here is a 4 core on sale at Newegg: AMD FX-4170 Zambezi 4.2GHz


That depends greatly on the applications he's running. If the
application(s) is(are) memory bound or I/O bound, more cores doesn't
necessarily mean better performance.


> If you run a feature rich desktop (kde, gnome, etc) then the more cores the
> better. Compiling code is much faster and you can still have a snappy
> desk top. Most gentoo folks compile quite a bit of code, depending on your
> updates and how often you experiment with new features or software.
>
> I'm setting up some new FX-8350 machines, but fully flushed out, there
> around a 1K (USD). Surely you can replace a mobo with a quad and as much ram as
> will fit, and get a fine machine. CPU speed, for me, is the dominate feature,
> when you are only doing a few things for a snappy workstation. Lots of cores and
> low CPU speed and low ram, sucks, imho. Max amount and max speed of the RAM
> is the killer performance edge for most workstations, imho.


MHO and experience is that you need a balanced system, IOW: if you
have a whole bunch of cores and GHz but crappy drive ... say bye to
performance, you'll be getting iowaits and your cores will be idle :(


> It boils down to a personal decision. The world of software
> is migrating to multi-threading, so the more cores, the
> more future-proof, imho.


MHO: it boils down to the software he's running ;)

   Rafa

Re: [gentoo-user] Dual or Quad CPU complications?

[Michael Hampicke]

Am 13.12.2012 07:12, schrieb Grant:
> I've only ever used systems with a single CPU.  I'm looking for a new host
> for a dedicated server (suggestions?) and it looks like I'll probably
> choose a machine with two or four CPUs.  What sort of complications does
> that add to set up and/or maintenance with Gentoo?
> 
> - Grant
> 

If you want to run mysql with high memory usage on that machine, you
might want to read
http://blog.jcole.us/2010/09/28/mysql-swap-insanity-and-the-numa-architecture/

Everything else that I can think of already has beed said.

Oh, tweak MAKEOPTS for a faster compile time, you also might want to
look at emerges --jobs and --load-average parameters

Re: [gentoo-user] Dual or Quad CPU complications?

[Volker Armin Hemmann]

Am Mittwoch, 12. Dezember 2012, 22:12:18 schrieb Grant:
> I've only ever used systems with a single CPU.  I'm looking for a new host
> for a dedicated server (suggestions?) and it looks like I'll probably
> choose a machine with two or four CPUs.  What sort of complications does
> that add to set up and/or maintenance with Gentoo?

none

also, forget numa. You won't deal with douzends of cores each using local 
memory and acccession the memory managed by the other cores. 

> 
> - Grant
-- 
#163933

Re: [gentoo-user] Dual or Quad CPU complications?

[Rafa Griman]

On Thu, Dec 13, 2012 at 7:22 PM, Volker Armin Hemmann
<volkerarmin@googlemail.com> wrote:
> Am Mittwoch, 12. Dezember 2012, 22:12:18 schrieb Grant:
>> I've only ever used systems with a single CPU.  I'm looking for a new host
>> for a dedicated server (suggestions?) and it looks like I'll probably
>> choose a machine with two or four CPUs.  What sort of complications does
>> that add to set up and/or maintenance with Gentoo?
>
> none
>
> also, forget numa. You won't deal with douzends of cores each using local
> memory and acccession the memory managed by the other cores.


It depends on his application, maybe his application does benefit on
NUMA architecture. Until we don't know what he's running, we can't
really say this or that architecture/technology is of no use ;)

So Volker, what applications are you running (and BTW: what volume of
data are you managing, how many users, ...)? This will helps us help
you :)

   Rafa

Re: [gentoo-user] Dual or Quad CPU complications?

[Grant]

[-- Attachment #1: Type: text/plain, Size: 897 bytes --]

> >> I've only ever used systems with a single CPU.  I'm looking for a new
host
> >> for a dedicated server (suggestions?) and it looks like I'll probably
> >> choose a machine with two or four CPUs.  What sort of complications
does
> >> that add to set up and/or maintenance with Gentoo?
> >
> > none
> >
> > also, forget numa. You won't deal with douzends of cores each using
local
> > memory and acccession the memory managed by the other cores.
>
>
> It depends on his application, maybe his application does benefit on
> NUMA architecture. Until we don't know what he's running, we can't
> really say this or that architecture/technology is of no use ;)
>
> So Volker, what applications are you running (and BTW: what volume of
> data are you managing, how many users, ...)? This will helps us help
> you :)
>
>    Rafa

It's an apache2/mysql server with a medium amount of traffic.

- Grant

[-- Attachment #2: Type: text/html, Size: 1135 bytes --]

Re: [gentoo-user] Dual or Quad CPU complications?

[Volker Armin Hemmann]

Am Freitag, 14. Dezember 2012, 08:55:08 schrieb Rafa Griman:
> On Thu, Dec 13, 2012 at 7:22 PM, Volker Armin Hemmann
> 
> <volkerarmin@googlemail.com> wrote:
> > Am Mittwoch, 12. Dezember 2012, 22:12:18 schrieb Grant:
> >> I've only ever used systems with a single CPU.  I'm looking for a new
> >> host
> >> for a dedicated server (suggestions?) and it looks like I'll probably
> >> choose a machine with two or four CPUs.  What sort of complications does
> >> that add to set up and/or maintenance with Gentoo?
> > 
> > none
> > 
> > also, forget numa. You won't deal with douzends of cores each using local
> > memory and acccession the memory managed by the other cores.
> 
> It depends on his application, maybe his application does benefit on
> NUMA architecture. Until we don't know what he's running, we can't
> really say this or that architecture/technology is of no use ;)
> 
> So Volker, what applications are you running (and BTW: what volume of
> data are you managing, how many users, ...)? This will helps us help
> you :)

you don't get NUMA just for free.

You have to buy NUMA hardware. If the hardware you buys does not scream NUMA 
at you, you don't have it. It is really that simple.

Multicore, multisocket systems MIGHT be NUMA systems - but that is not a 
guarantee. Now can this stupid thread please die away?

There are no caveats going from single to multicore on consumer hardware. and 
even on non consumer hardware there aren't many.

-- 
#163933

Re: [gentoo-user] Dual or Quad CPU complications?

[Grant]

[-- Attachment #1: Type: text/plain, Size: 344 bytes --]

> You have to buy NUMA hardware. If the hardware you buys does not scream
NUMA
> at you, you don't have it. It is really that simple.
>
> Multicore, multisocket systems MIGHT be NUMA systems - but that is not a
> guarantee. Now can this stupid thread please die away?

I guess the question seems stupid if you already know the answer.

- Grant

[-- Attachment #2: Type: text/html, Size: 417 bytes --]

Re: [gentoo-user] Dual or Quad CPU complications?

[J. Roeleveld]

On Saturday, December 15, 2012 11:46:36 AM Grant wrote:
> > You have to buy NUMA hardware. If the hardware you buys does not scream
> 
> NUMA
> 
> > at you, you don't have it. It is really that simple.
> > 
> > Multicore, multisocket systems MIGHT be NUMA systems - but that is not a
> > guarantee. Now can this stupid thread please die away?
> 
> I guess the question seems stupid if you already know the answer.

There are no stupid questions, only stupid answers...

Even on a system with only 2 sockets, it can be useful to have NUMA available.

Re: [gentoo-user] Dual or Quad CPU complications?

[Volker Armin Hemmann]

Am Samstag, 15. Dezember 2012, 20:57:24 schrieb J. Roeleveld:
> On Saturday, December 15, 2012 11:46:36 AM Grant wrote:
> > > You have to buy NUMA hardware. If the hardware you buys does not scream
> > 
> > NUMA
> > 
> > > at you, you don't have it. It is really that simple.
> > > 
> > > Multicore, multisocket systems MIGHT be NUMA systems - but that is not a
> > > guarantee. Now can this stupid thread please die away?
> > 
> > I guess the question seems stupid if you already know the answer.
> 
> There are no stupid questions, only stupid answers...
> 
> Even on a system with only 2 sockets, it can be useful to have NUMA
> available.

or not, because it costs you performance.

And while the starting questions were not stupid this thread is overflowing 
with stupid answers.

-- 
#163933

Re: [gentoo-user] Dual or Quad CPU complications?

[J. Roeleveld]

On Sunday, December 16, 2012 01:52:46 PM Volker Armin Hemmann wrote:
> Am Samstag, 15. Dezember 2012, 20:57:24 schrieb J. Roeleveld:
> > Even on a system with only 2 sockets, it can be useful to have NUMA
> > available.
> 
> or not, because it costs you performance.

When does it cost performance?
In all situations?

> And while the starting questions were not stupid this thread is overflowing
> with stupid answers.

Matter of opinion...

Re: [gentoo-user] Dual or Quad CPU complications?

[Michael Mol]

On Sun, Dec 16, 2012 at 10:39 AM, J. Roeleveld <joost@antarean.org> wrote:
> On Sunday, December 16, 2012 01:52:46 PM Volker Armin Hemmann wrote:
>> Am Samstag, 15. Dezember 2012, 20:57:24 schrieb J. Roeleveld:
>> > Even on a system with only 2 sockets, it can be useful to have NUMA
>> > available.
>>
>> or not, because it costs you performance.
>
> When does it cost performance?
> In all situations?

It adds some additional logic to memory allocation (put an allocation
near the process that uses it) and to process scheduling (keep the
process near its memory, but bump it to a more distant idle core if
necessary).

In all honestly, it's not a performance loss you're likely to notice,
unless you're so in need of squeezing out every spare cycle that you
most definitely _have_ hardware where there are disconnected memory
banks. I'm not convinced it's even measurable for us mundanes and our
hardware.

>
>> And while the starting questions were not stupid this thread is overflowing
>> with stupid answers.
>
> Matter of opinion...

Indeed.

--
:wq

Re: [gentoo-user] Dual or Quad CPU complications?

[J. Roeleveld]

> On Sun, Dec 16, 2012 at 10:39 AM, J. Roeleveld <joost@antarean.org> wrote:
>> On Sunday, December 16, 2012 01:52:46 PM Volker Armin Hemmann wrote:
>>> Am Samstag, 15. Dezember 2012, 20:57:24 schrieb J. Roeleveld:
>>> > Even on a system with only 2 sockets, it can be useful to have NUMA
>>> > available.
>>>
>>> or not, because it costs you performance.
>>
>> When does it cost performance?
>> In all situations?
>
> It adds some additional logic to memory allocation (put an allocation
> near the process that uses it) and to process scheduling (keep the
> process near its memory, but bump it to a more distant idle core if
> necessary).

That's the way it's supposed to work, yes :)

> In all honestly, it's not a performance loss you're likely to notice,
> unless you're so in need of squeezing out every spare cycle that you
> most definitely _have_ hardware where there are disconnected memory
> banks. I'm not convinced it's even measurable for us mundanes and our
> hardware.

I don't think I would notice it either, but as the system I have supports
it, I want to use it.
And then I want to be certain it actually supports it correctly.

The system I'm talking about is used for testing purposes. Running
multiple VMs. As far as I know, Xen has support for it, just need to
configure it properly.
And for this usecase, I think NUMA with only 2 physical CPUs should make a
positive difference.

--
Joost

Re: [gentoo-user] Dual or Quad CPU complications?

[Michael Mol]

Something _really_ weird happened to your quoting; you quoted my
email, but your email client said you wrote it.

On Mon, Dec 17, 2012 at 2:00 AM, J. Roeleveld <joost@antarean.org> wrote:
>> On Sun, Dec 16, 2012 at 10:39 AM, J. Roeleveld <joost@antarean.org> wrote:

^-- weird --^

>>> On Sunday, December 16, 2012 01:52:46 PM Volker Armin Hemmann wrote:
>>>> Am Samstag, 15. Dezember 2012, 20:57:24 schrieb J. Roeleveld:
>>>> > Even on a system with only 2 sockets, it can be useful to have NUMA
>>>> > available.
>>>>
>>>> or not, because it costs you performance.
>>>
>>> When does it cost performance?
>>> In all situations?
>>
>> It adds some additional logic to memory allocation (put an allocation
>> near the process that uses it) and to process scheduling (keep the
>> process near its memory, but bump it to a more distant idle core if
>> necessary).
>
> That's the way it's supposed to work, yes :)
>
>> In all honestly, it's not a performance loss you're likely to notice,
>> unless you're so in need of squeezing out every spare cycle that you
>> most definitely _have_ hardware where there are disconnected memory
>> banks. I'm not convinced it's even measurable for us mundanes and our
>> hardware.
>
> I don't think I would notice it either, but as the system I have supports
> it, I want to use it.
> And then I want to be certain it actually supports it correctly.
>
> The system I'm talking about is used for testing purposes. Running
> multiple VMs. As far as I know, Xen has support for it, just need to
> configure it properly.
> And for this usecase, I think NUMA with only 2 physical CPUs should make a
> positive difference.

Don't get me wrong; I was arguing that it shouldn't hurt to have it enabled. :)

--
:wq

Re: [gentoo-user] Dual or Quad CPU complications?

[J. Roeleveld]

> Something _really_ weird happened to your quoting; you quoted my
> email, but your email client said you wrote it.
>
> On Mon, Dec 17, 2012 at 2:00 AM, J. Roeleveld <joost@antarean.org> wrote:
>>> On Sun, Dec 16, 2012 at 10:39 AM, J. Roeleveld <joost@antarean.org>
>>> wrote:
>
> ^-- weird --^

Very weird, especially as I am using the same client now (squirrelmail)
and I don't see that line at all now.

>>>> On Sunday, December 16, 2012 01:52:46 PM Volker Armin Hemmann wrote:
>>>>> Am Samstag, 15. Dezember 2012, 20:57:24 schrieb J. Roeleveld:
>>>>> > Even on a system with only 2 sockets, it can be useful to have NUMA
>>>>> > available.
>>>>>
>>>>> or not, because it costs you performance.
>>>>
>>>> When does it cost performance?
>>>> In all situations?
>>>
>>> It adds some additional logic to memory allocation (put an allocation
>>> near the process that uses it) and to process scheduling (keep the
>>> process near its memory, but bump it to a more distant idle core if
>>> necessary).
>>
>> That's the way it's supposed to work, yes :)
>>
>>> In all honestly, it's not a performance loss you're likely to notice,
>>> unless you're so in need of squeezing out every spare cycle that you
>>> most definitely _have_ hardware where there are disconnected memory
>>> banks. I'm not convinced it's even measurable for us mundanes and our
>>> hardware.
>>
>> I don't think I would notice it either, but as the system I have
>> supports
>> it, I want to use it.
>> And then I want to be certain it actually supports it correctly.
>>
>> The system I'm talking about is used for testing purposes. Running
>> multiple VMs. As far as I know, Xen has support for it, just need to
>> configure it properly.
>> And for this usecase, I think NUMA with only 2 physical CPUs should make
>> a
>> positive difference.
>
> Don't get me wrong; I was arguing that it shouldn't hurt to have it
> enabled. :)

I know, just wanted to add the use-case for considering NUMA a usefull
option even with only 2 physical CPUs :)

--
Joost