From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) by finch.gentoo.org (Postfix) with ESMTP id 89DB41381F3 for ; Fri, 14 Dec 2012 22:28:29 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id A64AE21C03F; Fri, 14 Dec 2012 22:28:14 +0000 (UTC) Received: from mail-ee0-f53.google.com (mail-ee0-f53.google.com [74.125.83.53]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 6642A21C03F for ; Fri, 14 Dec 2012 22:26:56 +0000 (UTC) Received: by mail-ee0-f53.google.com with SMTP id c50so2321268eek.40 for ; Fri, 14 Dec 2012 14:26:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=0YcHvbHX3fV2MaRN3zhkaANYFzCqAPvkz5MbaFTDnak=; b=S4KpIqrcznI6vzZa9MttaPFgAACNzr1gkNjeD6BbQgqIU77o9yPI752D8LfHbAOilX zC6nyCJgKbpWxOv4nEj88wcjc14ldWiZKDYADziwesvwfPHscxut8On2i1HmkH/9OJoj M+rPG8e2sZQRnkfO8sMRznJb9fv6z0jZ3o3x/7LU0XCfHvsaKO+Jett6r7sMkk16wZPb ANZbmp2iBlTiMYwRAsXFuMvzk99+ka5C6ArPnx+MjdHvqvl4uZCkdipQZ0uxr0iP+nNI tPbZeDZYMycjY08tF9dlzkU/8AOWXSDNr9qdl6lfNzbegyM+/i+1gGqRIn6VSM3QqCyk qQjA== Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-user@lists.gentoo.org Reply-to: gentoo-user@lists.gentoo.org MIME-Version: 1.0 Received: by 10.14.3.195 with SMTP id 43mr18360064eeh.36.1355524015016; Fri, 14 Dec 2012 14:26:55 -0800 (PST) Received: by 10.223.158.68 with HTTP; Fri, 14 Dec 2012 14:26:54 -0800 (PST) In-Reply-To: References: <20121213082346.7b6bae37@khamul.example.com> <50C98588.9080409@binarywings.net> Date: Fri, 14 Dec 2012 14:26:54 -0800 Message-ID: Subject: Re: [gentoo-user] Dual or Quad CPU complications? From: Grant To: Gentoo mailing list Content-Type: multipart/alternative; boundary=047d7b66f239664b5904d0d787a0 X-Archives-Salt: 5888f9e0-1a4b-440a-8a95-7101ee816156 X-Archives-Hash: 39073bde93ec29be9366873b36832dd7 --047d7b66f239664b5904d0d787a0 Content-Type: text/plain; charset=ISO-8859-1 > >> > So if I have 2 physical CPU's with 4 cores each and I enable SMP, I'm > >> > using > >> > 8 cores? Can NUMA be either enabled or disabled when using more than > >> > one > >> > physical CPU, or is it required? > >> > >> > >> NUMA is a hardware architecture. It's how you access memory on a > >> hardware level: NUMA = Non Uniform Memory Access vs a UMA architecture > >> of typical (old/legacy) SMP systems (UMA = Uniform Memory Access). > >> > >> In a UMA system, all the memory belongs to all the sockets. In a NUMA > >> system, each socket has it's "own" local memory. In modern (x86-64) > >> processors, each socket has it's own memory controller so each socket > >> controls its own local memory. If one socket runs out of memory it can > >> ask another socket to lend him some memory. In a UMA system, no socket > >> has to ask since memory is global and belongs to all sockets so if one > >> socket uses up all the memory ... the rest "starve". In NUMA, there's > >> more control over who uses what (be it cores or RAM). > >> > >> If you have a modern dual or quad (or higher #) socket system ... > >> you've got NUMA architecture and you can't get rid of it, it's > >> hardware, not software. > > > > So I must enable CONFIG_NUMA for more than one physical CPU, and disable it > > for only one physical CPU? > > > Yup. But ... Why would you want to disable a socket (CPU)? If you > disable a socket (CPU) ... you lose the memory attached to that socket > (CPU) not to mention you lose those cores ;) Sure but it sounds like if my system only has one CPU socket, CONFIG_NUMA should be disabled. - Grant > A better solution would be to use cgroups or numactl tools to pin a > certain process to a set of cores and a memory region. > > If you really want to deactivate cores (but not the whole socket), you can type: > > echo 0 > /sys/devices/system/cpu/cpu1/online > > This would deactivate core #1. You can deactivate as many cores as you > wish, except for core #0. > > This can be done without rebooting your server (aka during run time). > Your memory will not be affected, but you will have less cores (and > theoretically more memory bandwidth). I say "theoretically" because > you always have to benchmark these things with YOUR application > (remember logic NEVER applies to real life ;) > > If you want to check the # of cores you've got: > > cat /proc/interrupts | grep CPU > > Other possibilities such as cat /proc/cpuinfo or dmesg, ... can be > useful too for this: your choice, FLOSS gives you options. > > If you want to activate the previously deactivated core, you can run: > > echo 1 > /sys/devices/system/cpu/cpu1/online > > Now ... be sure your core numbering is the expected core numbering. > IOW, not all server vendors follow the same numbering scheme so core > #1 in vendor A's server could be core #2 in vendor B's server. Never > trust logic ;) > > As I mentioned previously: test/benchmark YOUR software. DON'T trust > logic or generic benchmarks or web pages with results. Trust YOUR > results only. > > HTH > > Rafa --047d7b66f239664b5904d0d787a0 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable > >> > So if I have 2 physical CPU's with 4 cores each and = I enable SMP, I'm
> >> > using
> >> > 8 c= ores? =A0Can NUMA be either enabled or disabled when using more than
>= ; >> > one
> >> > physical CPU, or is it required?
> >>
>= ; >>
> >> NUMA is a hardware architecture. It's how y= ou access memory on a
> >> hardware level: NUMA =3D Non Uniform= Memory Access vs a UMA architecture
> >> of typical (old/legacy) SMP systems (UMA =3D Uniform Memory A= ccess).
> >>
> >> In a UMA system, all the memory b= elongs to all the sockets. In a NUMA
> >> system, each socket h= as it's "own" local memory. In modern (x86-64)
> >> processors, each socket has it's own memory controller so= each socket
> >> controls its own local memory. If one socket = runs out of memory it can
> >> ask another socket to lend him s= ome memory. In a UMA system, no socket
> >> has to ask since memory is global and belongs to all sockets = so if one
> >> socket uses up all the memory ... the rest "= ;starve". In NUMA, there's
> >> more control over who = uses what (be it cores or RAM).
> >>
> >> If you have a modern dual or quad (or higher= #) socket system ...
> >> you've got NUMA architecture and= you can't get rid of it, it's
> >> hardware, not softw= are.
> >
> > So I must enable CONFIG_NUMA for more than one physi= cal CPU, and disable it
> > for only one physical CPU?
>
= >
> Yup. But ... Why would you want to disable a socket (CPU)? If = you
> disable a socket (CPU) ... you lose the memory attached to that socket=
> (CPU) not to mention you lose those cores ;)

Sure but it so= unds like if my system only has one CPU socket, CONFIG_NUMA should be disab= led.

- Grant


> A better solution would be to= use cgroups or numactl tools to pin a
> certain process to a set of = cores and a memory region.
>
> If you really want to deactivate= cores (but not the whole socket), you can type:
>
> =A0 =A0 =A0 =A0 =A0 echo 0 > /sys/devices/system/cpu/cpu1/o= nline
>
> This would deactivate core #1. You can deactivate as = many cores as you
> wish, except for core #0.
>
> This ca= n be done without rebooting your server (aka during run time).
> Your memory will not be affected, but you will have less cores (and> theoretically more memory bandwidth). I say "theoretically"= because
> you always have to benchmark these things with YOUR applic= ation
> (remember logic NEVER applies to real life ;)
>
> If you w= ant to check the # of cores you've got:
>
> =A0 =A0 =A0cat = /proc/interrupts | grep CPU
>
> Other possibilities such as cat= /proc/cpuinfo or dmesg, ... can be
> useful too for this: your choice, FLOSS gives you options.
>
= > If you want to activate the previously deactivated core, you can run:<= br>>
> =A0 =A0 =A0 =A0 =A0 echo 1 > /sys/devices/system/cpu/cpu= 1/online
>
> Now ... be sure your core numbering is the expected core numbe= ring.
> IOW, not all server vendors follow the same numbering scheme = so core
> #1 in vendor A's server could be core #2 in vendor B= 9;s server. Never
> trust logic ;)
>
> As I mentioned previously: test/benchma= rk YOUR software. DON'T trust
> logic or generic benchmarks or we= b pages with results. Trust YOUR
> results only.
>
> HTH<= br> >
> =A0 =A0Rafa
--047d7b66f239664b5904d0d787a0--