[gentoo-dev] ICC Profile

[gentoo-dev] ICC Profile

[Adam Stylinski]

The intel C Compiler (icc) has an ebuild for gentoo and the wiki has a script to integrate it with portage.  This script works will in terms of building binaries, however when mixed with gcc environments there are massive linking issues.  I propose that an ICC profile is made which contains specific versions and default flags for people who want to build a mixed icc-gcc environment.  ICC is much faster than GCC and although not free, offers a free non-commercial license.  I would be very interested in this project and more than willing to help to the best of my abilities.  I've already been trying to maintain a mixed environment with some luck, while there have been a lot of problems using dynamically linked libraries (ld from intel and ld from gcc don't always get along), my system is substatially faster.  The kernel obviously will still be built under gcc as well as bash (unless intel helps submit patches to make the code work with their compiler).  There are many tools icc
  !
has to offer for vectorization.  If these were streamlined into Gentoo with a fetch restriction for ICC, a bootsrapping boot disk could be made and result in a very fast
distribution. 
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] ICC Profile

[Luca Barbato]

Adam Stylinski wrote:
> The intel C Compiler (icc)

icc, xlc, llvm, sunstudio could be interesting fields of discovery.

Which are the pitfalls of using icc?

lu

-- 

Luca Barbato
Gentoo Council Member
Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] ICC Profile

[Robert Bridge]

On Fri, 18 Jul 2008 11:34:11 +0900
Luca Barbato <lu_zero@gentoo.org> wrote:

> Adam Stylinski wrote:
> > The intel C Compiler (icc)
> 
> icc, xlc, llvm, sunstudio could be interesting fields of discovery.
> 
> Which are the pitfalls of using icc?
> 
> lu
> 

If I recall correctly, needing some packages compiled with ICC flags
set one way, and needing others compiled with the same flag set the
oter way.

Can it compile a kernel yet?

Rob.
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] ICC Profile

[Adam Stylinski]

There are very few pitfalls, none of which I see as real killers.  These include:

1.) Closed source compiler: Yes this stands against what we believe, and yes by closing their source they're protecting the trade secrets of their architecture.  It also could be more difficult to debug, although that's highly unlikely, they have the idb (intel debugger) which works very much like gdb.

2.) Linking issues: So far it's pretty versatile, but it doesn't always cooperate with gcc compiled apps.  It may be a good strategy to make the troublesome apps which won't compile with ICC compile with ICC.

Pro's:

1.) Bloody fast machine code.  Intel obfuscates their architecture but they give back to the community as much as possible to make their hardware marketable toward the open source sysadmin, developer, etc etc.  Their drivers are open and they develop for the kernel constantly.  This cooperation leads me to believe that they would assist a team of developers in making 100% icc compatible code.

2.) Bloody fast compilation time.  In my experience the compiler works much faster even with heavy optimization.

3.) Takes full advantage of SSE enabled hardware.  SIMD instructions are quite useful, code is extremely vectorized.

4.) will project gentoo toward the power user more, helps the gentoo image, and overall will make linux a more professional operating system (and a quite competitive alternative to something like a SPARC+Solaris configuration).  This would also make cluster farms and science application more respectful toward the gentoo community.  The academic and research world already uses ICC to compile their apps for the sake of speed.  The interprocedural optimizations for both the fortran and c/c++ compilers make it a must.

5.) It's free, albeit a commercial product.  As gentoo is entirely non-profit, there is no restriction when it comes to licensing.  The binaries won't be sold for the intel-compiled livecd, and the compiler itself with a fetch restriction allows the user to legally register for their free non-commercial license.  
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] ICC Profile

[Adam Stylinski]

There is some record of a version of the kernel being compiled with some patches involved.  It's probably possible, I'd imagine.  Though, this is not necessary for the first builds.
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] ICC Profile

[Adam Stylinski]

Oh yes, and we can also take advantage of the free support that intel offers for all users:

http://www.intel.com/support/performancetools/sb/CS-017156.htm

Not to mention they have forums filled with intel developers/experts which we can get involved.  I'm sure intel would benefit from this and be more than excited to collaborate:
http://softwarecommunity.intel.com/isn/Community/en-US/forums/default.aspx
 
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] ICC Profile

[Donnie Berkholz]

[-- Attachment #1: Type: text/plain, Size: 2066 bytes --]

On 14:23 Thu 17 Jul     , Adam Stylinski wrote:
> The intel C Compiler (icc) has an ebuild for gentoo and the wiki has a 
> script to integrate it with portage.  This script works will in terms 
> of building binaries, however when mixed with gcc environments there 
> are massive linking issues.  I propose that an ICC profile is made 
> which contains specific versions and default flags for people who want 
> to build a mixed icc-gcc environment.  ICC is much faster than GCC and 
> although not free, offers a free non-commercial license.  I would be 
> very interested in this project and more than willing to help to the 
> best of my abilities.  I've already been trying to maintain a mixed 
> environment with some luck, while there have been a lot of problems 
> using dynamically linked libraries (ld from intel and ld from gcc 
> don't always get along), my system is substatially faster.  The kernel 
> obviously will still be built under gcc as well as bash (unless intel 
> helps submit patches to make the code work with their compiler).  
> There are many tools icc has to offer for vectorization.  If these 
> were streamlined into Gentoo with a fetch restriction for ICC, a 
> bootsrapping boot disk could be made and result in a very fast 
> distribution. -- gentoo-dev@lists.gentoo.org mailing list

Adam,

I've already got a mixed system that works reasonably well. I use 
/etc/portage/env/ to control which packages build with icc and 
icc-specific CFLAGS by having /etc/portage/env/$CATEGORY/$PN be symlinks 
to /etc/portage/env/env.icc with these contents:

  echo " * Exporting: Intel compilers in $EBUILD_PHASE"
  CC="icc"
  CXX="icc"
  FC="ifort"
  F77="ifort"
  FFLAGS="-openmp -parallel -ipo -xO -O2 -no-prec-div"
  FCFLAGS="${FFLAGS}"
  
  export CC CXX FC F77 FFLAGS FCFLAGS

Is what you're proposing here an improvement over something like the 
above setup? If so, could you explain how?

-- 
Thanks,
Donnie

Donnie Berkholz
Developer, Gentoo Linux
Blog: http://dberkholz.wordpress.com

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] ICC Profile

[Donnie Berkholz]

[-- Attachment #1: Type: text/plain, Size: 2958 bytes --]

On 23:24 Thu 17 Jul     , Adam Stylinski wrote:
> There are very few pitfalls, none of which I see as real killers.  
> These include:
> 
> 1.) Closed source compiler: Yes this stands against what we believe, 
> and yes by closing their source they're protecting the trade secrets 
> of their architecture.  It also could be more difficult to debug, 
> although that's highly unlikely, they have the idb (intel debugger) 
> which works very much like gdb.

Doesn't really matter, although we could never move to it as the default 
without violating our social contract.

> 2.) Linking issues: So far it's pretty versatile, but it doesn't 
> always cooperate with gcc compiled apps.  It may be a good strategy to 
> make the troublesome apps which won't compile with ICC compile with 
> ICC.

Pretty sure you didn't mean ICC twice here, but sure.

> Pro's:
> 
> 1.) Bloody fast machine code.  Intel obfuscates their architecture but 
> they give back to the community as much as possible to make their 
> hardware marketable toward the open source sysadmin, developer, etc 
> etc.  Their drivers are open and they develop for the kernel 
> constantly.  This cooperation leads me to believe that they would 
> assist a team of developers in making 100% icc compatible code.

OK. This involves upstream projects more than us, though.

> 2.) Bloody fast compilation time.  In my experience the compiler works 
> much faster even with heavy optimization.
> 
> 3.) Takes full advantage of SSE enabled hardware.  SIMD instructions 
> are quite useful, code is extremely vectorized.

Sure, sure, speed is nice.

> 4.) will project gentoo toward the power user more, helps the gentoo 
> image, and overall will make linux a more professional operating 
> system (and a quite competitive alternative to something like a 
> SPARC+Solaris configuration).

I don't buy any parts of this argument, although the rest of your email 
is pretty good.

> This would also make cluster farms and science application more 
> respectful toward the gentoo community.  The academic and research 
> world already uses ICC to compile their apps for the sake of speed.  
> The interprocedural optimizations for both the fortran and c/c++ 
> compilers make it a must.

Yes, ICC is nice. And people can already use it easily within Gentoo for 
performance-critical apps.

> 5.) It's free, albeit a commercial product.  As gentoo is entirely 
> non-profit, there is no restriction when it comes to licensing.  The 
> binaries won't be sold for the intel-compiled livecd, and the compiler 
> itself with a fetch restriction allows the user to legally register 
> for their free non-commercial license.

OK, so we can't sell icc-compiled software in our Gentoo store, so the 
releases would always remain built with gcc.

-- 
Thanks,
Donnie

Donnie Berkholz
Developer, Gentoo Linux
Blog: http://dberkholz.wordpress.com

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] ICC Profile

[Adam Stylinski]

I'm not suggesting that it be sold. Gentoo is
non-profit anyway, the livecd could be available for
download only. The binaries don't have to be
licensed if you're not selling them, however the
compiler does. This is where the non-commercial free
license comes in (with a fetch restriction requiring
the user to register for it). This is completely
free and I don't see it much more of a pain than it
is to go sign up for the IBM developers work to
extract their PPC version of java.
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] ICC Profile

[Doug Goldstein]

Robert Bridge wrote:
> On Fri, 18 Jul 2008 11:34:11 +0900
> Luca Barbato <lu_zero@gentoo.org> wrote:
>
>   
>> Adam Stylinski wrote:
>>     
>>> The intel C Compiler (icc)
>>>       
>> icc, xlc, llvm, sunstudio could be interesting fields of discovery.
>>
>> Which are the pitfalls of using icc?
>>
>> lu
>>
>>     
>
> If I recall correctly, needing some packages compiled with ICC flags
> set one way, and needing others compiled with the same flag set the
> oter way.
>
> Can it compile a kernel yet?
>
> Rob.
>   
No. It doesn't implement all of GCCs extensions and all GNUisms that the 
kernel relies on.
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] ICC Profile

[Sébastien Fabbro]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Apologies if you received this mail already, I'm having problems with my
smtp.

On Thursday July 17 Adam Stylinski wrote:

> The intel C Compiler (icc) has an ebuild for gentoo and the wiki has
> a script to integrate it with portage.  This script works will in
> terms of building binaries, however when mixed with gcc environments
> there are massive linking issues.  I propose that an ICC profile is
> made which contains specific versions and default flags for people
> who want to build a mixed icc-gcc environment.  ICC is much faster
> than GCC and although not free, offers a free non-commercial
> license.  I would be very interested in this project and more than
> willing to help to the best of my abilities.  I've already been
> trying to maintain a mixed environment with some luck, while there
> have been a lot of problems using dynamically linked libraries (ld
> from intel and ld from gcc don't always get along), my system is
> substatially faster.  The kernel obviously will still be built under
> gcc as well as bash (unless intel helps submit patches to make the
> code work with their compiler).  There are many tools icc ! has to
> offer for vectorization.  If these were streamlined into Gentoo with
> a fetch restriction for ICC, a bootsrapping boot disk could be made
> and result in a very fast distribution. 


An icc profile would be welcome. I've been the maintainer of icc (and
other Intel tools) for the last year or so more by default than
real interest. I would welcome any input from the Gentoo community.
Re-adding slots and an icc profile was on my mind, but never found the
time to invest in it, and got at the tail of my priority list. So don't
hesitate to contact me (email, irc, bugs) and others.

There was some attempts a few years ago for rolling up a full Gentoo
with icc, but it hit several problems if I recall. Now both icc and gcc
have improved since then.

Also, if you haven't already, check also some of the old bugs [1,2] we
have, and a recurring one [3].

I would like to recall one important issue with the Intel license
concerning the "free for non-commercial use" [4]. It means
you can't use it for free if you're paid to use it. Yes,
beer is not free for academic scientists too.

[1] http://bugs.gentoo.org/show_bug.cgi?id=26757
[2] http://bugs.gentoo.org/show_bug.cgi?id=53808
[3] http://bugs.gentoo.org/show_bug.cgi?id=201596
[4] http://www.intel.com/cd/software/products/asmo-na/eng/219692.htm#0

- - -- 
Sébastien
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)

iEYEARECAAYFAkiApboACgkQ1ycZbhPLE2BMHgCggfzfS4SakVyw42r+JnnxYNpL
E9gAoJvRrocinIDlInF6kbeSGF2kvX9t
=M4XK
-----END PGP SIGNATURE-----

Re: [gentoo-dev] ICC Profile

[Sébastien Fabbro]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=UTF-8, Size: 3259 bytes --]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


On Thursday July 17, Adam Stylinski wrote:

> Pro's:
> 
> 1.) Bloody fast machine code.  Intel obfuscates their architecture
> but they give back to the community as much as possible to make their
> hardware marketable toward the open source sysadmin, developer, etc
> etc.  Their drivers are open and they develop for the kernel
> constantly.  This cooperation leads me to believe that they would
> assist a team of developers in making 100% icc compatible code.  

Gentoo is not supported from Intel, and they had not plans doing so. As
of October 2007, I asked their Software channel whether Gentoo
users have similar support as RedHat or SUSE users and the answer was:

"No, we have no current plans to support Gentoo. Also, Gentoo is NOT a
derivative of a Linux we do support.  My understanding is that it is
independently derived from Kernel.org.  Thus it is less likely to work
than a distro which is a derivative of a supported distribution.

Meanwhile, Debian/Ubuntu got support, so things might change if
Gentoo re-becomes/remains popular. Any Intel dev reading this list,
please contact us.

And as Luca mentioned, having sunstudio, xlc (is this one free?) or llvm
would not make Intel a privileged case for Gentoo.


> 2.) Bloody fast compilation time.  In my experience the compiler
> works much faster even with heavy optimization.  

I don't experience this that much, but I really don't use it much
either. Would be nice to have benchmarks here.

 
> 4.) will project gentoo toward the power user more, helps the gentoo
> image, and overall will make linux a more professional operating
> system (and a quite competitive alternative to something like a
> SPARC+Solaris configuration).  This would also make cluster farms and
> science application more respectful toward the gentoo community.  The
> academic and research world already uses ICC to compile their apps
> for the sake of speed.  The interprocedural optimizations for both
> the fortran and c/c++ compilers make it a must.  

I would be careful about this, and this needs benchmarks, especially
with gcc > 4.3. By default icc flags are fairly agressive. For example,
for many scientific applications, you don't want a simple -O2 where you
loose floating point precision. Add -mp or -mp1 to your icc flags, add
some decent gcc flags, and improvement over gcc is much smaller.
 
> 5.) It's free, albeit a commercial product.  As gentoo is entirely
> non-profit, there is no restriction when it comes to licensing.  The
> binaries won't be sold for the intel-compiled livecd, and the
> compiler itself with a fetch restriction allows the user to legally
> register for their free non-commercial license.    

Again, as long as you're not being compensated for doing it (for Gentoo
I'm not).

In summary, I'm completely in favor of trying projects like this, but
first, this needs a few benchmarks before going further.


- -- 
Sébastien
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)

iEYEARECAAYFAkiApcoACgkQ1ycZbhPLE2C0TQCfYLD2+mazHKjosK7wiHFU6FGb
d3EAnR1FWZ92O2B9xBJerCj4dj2GRUZ4
=YhT2
-----END PGP SIGNATURE-----
éí¢‡^¾X¬¶È\x1ežÚ(¢¸&j)bž	b²

Re: [gentoo-dev] ICC Profile

[Robert Bridge]

On Fri, 18 Jul 2008 15:16:18 +0100
Sébastien Fabbro <bicatali@gentoo.org> wrote:

> There was some attempts a few years ago for rolling up a full Gentoo
> with icc, but it hit several problems if I recall. Now both icc and
> gcc have improved since then.

Including needing package specific CFLAGS because some packages in
@system needed mutually exclusive flag settings. 

I'll see if I can dig up the link for an old blog on the topic later.
--
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] ICC Profile

[Adam Stylinski]

Also, in the academic environment the grad student/university can pay for the license that the student slipstreams into their gentoo installation, making it 100% legal depending on how many seats he or she buys.
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] ICC Profile

[Michael Hammer]

[-- Attachment #1: Type: text/plain, Size: 224 bytes --]

Cold you please fix your reply?

-- 
------------------------------------------------------------
Michael Hammer    |    <mueli@gentoo.org>     |     Graz, AT
Gentoo Developer (Kerberos)  |  http://www.michael-hammer.at

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] ICC Profile

[Adam Stylinski]

Well, it looks like we're not alone on this project.  This email was just sent to me:


Adam,
I managed to track down someone who can probably help you with your ICC work, if you're nice to him. :)
See the forwarded message:

Have your friend contact bill dot hilliard at intel dot com.  He's on the ICC team and can either answer any questions or works next to someone who can.

Scott
-- 
gentoo-dev@lists.gentoo.org mailing list