[gentoo-cluster] Stats on HPC clusters running Gentoo

[gentoo-cluster] Stats on HPC clusters running Gentoo

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 1091 bytes --]

Hi Alexxy and all!

While preparing Gentoo flyer[1,2] on upcoming FOSDEM event it was
suggested to include some words about Gentoo in HPC. So I'm asking
community about recent stats of using Gentoo on HPC clusters.

I ran several clusters at a university on Gentoo, but they are weak
by modern standards (<10 TFlops CPU, no GPU for the best one, though
with IB QDR interconnect), so I'm reluctant to add them in the
flyer. If someone can provide us with better and actual numbers,
this will be nice.

IIRC Alexxy is running some HPC setups. Are they on Gentoo? If yes,
please provide some stats (Flops, number of cores, nodes, storage,
interconnect and so on).

If someone can provide any other relevant info, we'll be grateful.

Years ago there was a Gentoo cluster project page with list of
active setups, but I can't find it any more and it is likely to be
outdated anyway.

[1] https://download.sumptuouscapital.com/gentoo/fosdem2018/gentoo-fosdem2018-flyer.pdf
[2] git://git.sumptuouscapital.com/gentoo/pr-fosdem2018-flyer.git

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 833 bytes --]

[gentoo-cluster] Re: Stats on HPC clusters running Gentoo

[Alexey Shvetsov]

Hi all!

Sorry for delay.

Well we currently have few systems at SPbSTU (and yes some of them runs 
gentoo, but currently not that one that is in top500)

So about systems:

One old called Sigma

  8 nodes:
                 cpu:    4 x AMD Opteron 6174, 12 core, 2.2GHz
                 mem:    128G RAM
                 net:    40Gbps QDR Infiniband, 10Gbps Ethernet
                 Storage FS:     Lustre
                 Storage size:   13.5Tb

Second one is slightly exotic system called ccNUMA (actualy part of HPC 
complex installed in 2014)

64 nodes:
                 cpu:    3 x AMD Opteron 6380, 16 core, 2.5GHz
                 mem:    96G RAM
                 net:    56Gbps FDR Infiniband, NumaScale
                 Storage FS:     Lustre
                 Storage size:   1Pb

ccNUMA actuly uses NumaScale interconnect and bootloader so it can be 
used as Jumbo ccNUMA Single System Image
with ~3000 cores and 12Tb RAM that will be seen by single linux kernel 
(and yes it runs gentoo)


Other 3 partitions of our HPC systems currently runs CentOS 7 (but may 
be migrated to gentoo in near future). All this partitions shares same 
Lustre FS as ccNUMA

   * tornado     : nodes: 612
                     cpu: 2 x Intel Xeon CPU E5-2697 v3 @ 2.60GHz
         cores/hwthreads: 28 / 28
                     mem: 64G
                     net: 56Gbps FDR Infiniband
   * tornado-k40 : nodes: 56
                     cpu: 2 x Intel Xeon CPU E5-2697 v3 @ 2.60GHz
         cores/hwthreads: 28 / 28
            co-processor: 2 x Nvidia Tesla K40x
        co-porcessor mem: 12G
                     mem: 64G
                     net: 56Gbps FDR Infiniband
   * mic         : nodes: 288
                     cpu: 1 x Intel Xeon Phi 5120D @ 1.10GHz
         cores/hwthreads: 60 / 240
                     mem: 8G
                     net: 56Gbps FDR Infiniband

Storage: 1 PB Lustre FS



PS also we have some exotic storage system used for backup (400+ disks 
attached via SAS and runs single ZFS filesystem)

Andrew Savchenko писал 05-01-2018 02:18:
> Hi Alexxy and all!
> 
> While preparing Gentoo flyer[1,2] on upcoming FOSDEM event it was
> suggested to include some words about Gentoo in HPC. So I'm asking
> community about recent stats of using Gentoo on HPC clusters.
> 
> I ran several clusters at a university on Gentoo, but they are weak
> by modern standards (<10 TFlops CPU, no GPU for the best one, though
> with IB QDR interconnect), so I'm reluctant to add them in the
> flyer. If someone can provide us with better and actual numbers,
> this will be nice.
> 
> IIRC Alexxy is running some HPC setups. Are they on Gentoo? If yes,
> please provide some stats (Flops, number of cores, nodes, storage,
> interconnect and so on).
> 
> If someone can provide any other relevant info, we'll be grateful.
> 
> Years ago there was a Gentoo cluster project page with list of
> active setups, but I can't find it any more and it is likely to be
> outdated anyway.
> 
> [1]
> https://download.sumptuouscapital.com/gentoo/fosdem2018/gentoo-fosdem2018-flyer.pdf
> [2] git://git.sumptuouscapital.com/gentoo/pr-fosdem2018-flyer.git
> 
> Best regards,
> Andrew Savchenko

-- 
Best Regards,
Alexey 'Alexxy' Shvetsov
Best Regards,
Alexey 'Alexxy' Shvetsov, PhD
Department of Molecular and Radiation Biophysics
FSBI Petersburg Nuclear Physics Institute, NRC Kurchatov Institute,
Leningrad region, Gatchina, Russia
Gentoo Team Ru
Gentoo Linux Dev
mailto:alexxyum@gmail.com
mailto:alexxy@gentoo.org
mailto:alexxy@omrb.pnpi.spb.ru

Re: [gentoo-cluster] Re: Stats on HPC clusters running Gentoo

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 2410 bytes --]

On Thu, 11 Jan 2018 21:42:37 +0300 Alexey Shvetsov wrote:
> Hi all!
> 
> Sorry for delay.
> 
> Well we currently have few systems at SPbSTU (and yes some of them runs 
> gentoo, but currently not that one that is in top500)
> 
> So about systems:
> 
> One old called Sigma
> 
>   8 nodes:
>                  cpu:    4 x AMD Opteron 6174, 12 core, 2.2GHz
>                  mem:    128G RAM
>                  net:    40Gbps QDR Infiniband, 10Gbps Ethernet
>                  Storage FS:     Lustre
>                  Storage size:   13.5Tb
> 
> Second one is slightly exotic system called ccNUMA (actualy part of HPC 
> complex installed in 2014)
> 
> 64 nodes:
>                  cpu:    3 x AMD Opteron 6380, 16 core, 2.5GHz
>                  mem:    96G RAM
>                  net:    56Gbps FDR Infiniband, NumaScale
>                  Storage FS:     Lustre
>                  Storage size:   1Pb
> 
> ccNUMA actuly uses NumaScale interconnect and bootloader so it can be 
> used as Jumbo ccNUMA Single System Image
> with ~3000 cores and 12Tb RAM that will be seen by single linux kernel 
> (and yes it runs gentoo)
> 
> 
> Other 3 partitions of our HPC systems currently runs CentOS 7 (but may 
> be migrated to gentoo in near future). All this partitions shares same 
> Lustre FS as ccNUMA
> 
>    * tornado     : nodes: 612
>                      cpu: 2 x Intel Xeon CPU E5-2697 v3 @ 2.60GHz
>          cores/hwthreads: 28 / 28
>                      mem: 64G
>                      net: 56Gbps FDR Infiniband
>    * tornado-k40 : nodes: 56
>                      cpu: 2 x Intel Xeon CPU E5-2697 v3 @ 2.60GHz
>          cores/hwthreads: 28 / 28
>             co-processor: 2 x Nvidia Tesla K40x
>         co-porcessor mem: 12G
>                      mem: 64G
>                      net: 56Gbps FDR Infiniband
>    * mic         : nodes: 288
>                      cpu: 1 x Intel Xeon Phi 5120D @ 1.10GHz
>          cores/hwthreads: 60 / 240
>                      mem: 8G
>                      net: 56Gbps FDR Infiniband
> 
> Storage: 1 PB Lustre FS
> 
> 
> 
> PS also we have some exotic storage system used for backup (400+ disks 
> attached via SAS and runs single ZFS filesystem)

Thanks, Alexey!

This information will be very useful, we'll include parts of it in
the Gentoo's FOSDEM flyer.

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-cluster] Stats on HPC clusters running Gentoo

[Alexey Shvetsov]

Kristian Fiskerstrand писал 14-01-2018 18:31:
> On 01/14/2018 03:22 PM, Ultrabug wrote:
>> Excuse my delay
>> 
>> In case it is still of interest, at Numberly we run a lot bunch of
>> Gentoo production machines such as:
> 
> Thank you all, the current draft is available at
> 
> https://download.sumptuouscapital.com/gentoo/fosdem2018/gentoo-fosdem2018-flyer.pdf

Hi!

Seems like there is an error in flyer:

Numascale system has 64*192G ram ~ 12T (seems like it was my mistake in 
previos letter)


-- 
Best Regards,
Alexey 'Alexxy' Shvetsov
Best Regards,
Alexey 'Alexxy' Shvetsov, PhD
Department of Molecular and Radiation Biophysics
FSBI Petersburg Nuclear Physics Institute, NRC Kurchatov Institute,
Leningrad region, Gatchina, Russia
Gentoo Team Ru
Gentoo Linux Dev
mailto:alexxyum@gmail.com
mailto:alexxy@gentoo.org
mailto:alexxy@omrb.pnpi.spb.ru