[gentoo-dev] Add rust eclass to support multi-target compilation

[gentoo-dev] Add rust eclass to support multi-target compilation

[gibix]

[-- Attachment #1: Type: text/plain, Size: 981 bytes --]

Hello,

I opened a PR here: https://github.com/gentoo/gentoo/pull/9388

Here a copy of the message:

# add support for rust multi-target

- add rust.eclass (with rust-utils.class) to support rust multi-target with
  multibuild
- modify cargo.class to support rust multi-target
- change dev-lang/rust slot system

This will allows projects like rustfmt, clippy, bindgen that need runtime
linking with the proper rust version to work correctly. Beyond this while
rust is getting older as project we will see more projects that will
require a specific rust version for compilation.

This PR replaces the need for rustup in gentoo for toolchain handling and
components.

requires:
- [features in
  cargo-ebuild](https://github.com/cardoe/cargo-ebuild/pull/14/commits/1aefd302f9430c0b628923da23e2a8d74b83d1ec)
- [binaries support into
  eselect-rust](https://github.com/jauhien/eselect-rust/pull/4)

see first discussion on
[gentoo-rust](https://github.com/gentoo/gentoo-rust/pull/362)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Add rust eclass to support multi-target compilation

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 1625 bytes --]

Hi!

On Mon, 30 Jul 2018 17:00:20 +0200 gibix wrote:
> Hello,
> 
> I opened a PR here: https://github.com/gentoo/gentoo/pull/9388
> 
> Here a copy of the message:
> 
> # add support for rust multi-target
> 
> - add rust.eclass (with rust-utils.class) to support rust multi-target with
>   multibuild
> - modify cargo.class to support rust multi-target
> - change dev-lang/rust slot system
> 
> This will allows projects like rustfmt, clippy, bindgen that need runtime
> linking with the proper rust version to work correctly. Beyond this while
> rust is getting older as project we will see more projects that will
> require a specific rust version for compilation.
> 
> This PR replaces the need for rustup in gentoo for toolchain handling and
> components.
> 
> requires:
> - [features in
>   cargo-ebuild](https://github.com/cardoe/cargo-ebuild/pull/14/commits/1aefd302f9430c0b628923da23e2a8d74b83d1ec)
> - [binaries support into
>   eselect-rust](https://github.com/jauhien/eselect-rust/pull/4)
> 
> see first discussion on
> [gentoo-rust](https://github.com/gentoo/gentoo-rust/pull/362)

If you are submitting code to the main tree, please submit all
requirements to the tree first.

Your PR brokes tree:
https://github.com/gentoo/gentoo/pull/9388#issuecomment-408914903

Please fix these issues as well, for both QA and CI checks. (These
problems may be a result of §1 (not yet submitted changes to the
tree).

It would be easier if your will split your improvements into series
of atomic changes and submit them individually for review.

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Add rust eclass to support multi-target compilation

[Dirkjan Ochtman]

[-- Attachment #1: Type: text/plain, Size: 2292 bytes --]

On Mon, Jul 30, 2018 at 5:02 PM gibix <gibix@riseup.net> wrote:

> This will allows projects like rustfmt, clippy, bindgen that need runtime
> linking with the proper rust version to work correctly. Beyond this while
> rust is getting older as project we will see more projects that will
> require a specific rust version for compilation.
>

As I mentioned in the PR already, I'm not a fan of this change. (Context: I
have been writing Rust code for 2 years, I'm on the Rust team, and I've
done most of the maintenance on the dev-lang/rust ebuilds for the past year
or so.)

As far as I know, the Rust ecosystem is largely bimodal: stuff is either
compatible with stable and later, or it works only on nightly. It seems
very rare that code is actually tied to a particular Rust release and does
not compile against latest Rust stable. (Upstream actually promises they
won't break code except in case they need to fix a soundness issue in the
compiler.) So instead of building this whole target infrastructure (which
is definitely needed for something like Python or Ruby), we should be able
to just work with stable and nightly slots, and ebuilds can depend on those.

Upstream is also very clearly packaging their core tooling as a single
package made up of a number of components, and these are distributed and
compiled together in common usage on other platforms (with the rustup
tooling). That also means they are tightly coupled -- for example, rustfmt
can change formats over time, and CI that checks formatting will need to
align with whatever the current stable Rust version of rustfmt is.
Installing the latest (nightly or beta) version of rustfmt while you have a
stable Rust toolchain installed is thus not a good idea. As another
example, cargo is now tagged as 0.28, but when passing --version it will
report as 1.27.0 -- actually the Rust toolchain version.

For these reasons, I think it would be better to align our Rust ebuilds
with upstream and work with single ebuilds (dev-lang/rust and
dev-lang/rust-bin) that install all the tools belonging to a particular
version of the Rust toolchain. What tools are installed can be customized
with USE flags, and the default install can be minimal (just rustc and
cargo, which you need to build packages anyway).

Regards,

Dirkjan

[-- Attachment #2: Type: text/html, Size: 2699 bytes --]

Re: [gentoo-dev] Add rust eclass to support multi-target compilation

[Luca Barbato]

On 31/07/2018 09:35, Dirkjan Ochtman wrote:
> On Mon, Jul 30, 2018 at 5:02 PM gibix <gibix@riseup.net> wrote:
> 

> As far as I know, the Rust ecosystem is largely bimodal: stuff is either
> compatible with stable and later, or it works only on nightly. It seems
> very rare that code is actually tied to a particular Rust release and does
> not compile against latest Rust stable. (Upstream actually promises they
> won't break code except in case they need to fix a soundness issue in the
> compiler.) So instead of building this whole target infrastructure (which
> is definitely needed for something like Python or Ruby), we should be able
> to just work with stable and nightly slots, and ebuilds can depend on those.

This is true until you do not want to use libsyntax and other
compiler-specific libraries.

Because of the enforced ABI-randomness, what you built against stable
must be rebuilt on stable.1, making those libraries non-shared might be
a topic.

> Upstream is also very clearly packaging their core tooling as a single
> package made up of a number of components, and these are distributed and
> compiled together in common usage on other platforms (with the rustup
> tooling). That also means they are tightly coupled -- for example, rustfmt
> can change formats over time, and CI that checks formatting will need to
> align with whatever the current stable Rust version of rustfmt is.

Actually rustfmt is interesting since it does use libsyntax and
depending on which project you may have to use the nightly rustfmt while
targeting stable.

> Installing the latest (nightly or beta) version of rustfmt while you have a
> stable Rust toolchain installed is thus not a good idea. As another
> example, cargo is now tagged as 0.28, but when passing --version it will
> report as 1.27.0 -- actually the Rust toolchain version.

Well it might be surprising, but for at least one project is actually
required :)

> For these reasons, I think it would be better to align our Rust ebuilds
> with upstream and work with single ebuilds (dev-lang/rust and
> dev-lang/rust-bin) that install all the tools belonging to a particular
> version of the Rust toolchain.

> What tools are installed can be customized
> with USE flags, and the default install can be minimal (just rustc and
> cargo, which you need to build packages anyway).

So once you need bindgen you have to rebuild rustc and then you need
clippy you build again rustc?

And what happens once you have yet-another-tool using libsyntax, you add
it to the rustc ebuild and unleash a -r1 to the users?

Sorry, does not sound good.

lu

Re: [gentoo-dev] Add rust eclass to support multi-target compilation

[gibix]

[-- Attachment #1: Type: text/plain, Size: 504 bytes --]

> Your PR brokes tree:
> https://github.com/gentoo/gentoo/pull/9388#issuecomment-408914903
> 
> Please fix these issues as well, for both QA and CI checks. (These
> problems may be a result of §1 (not yet submitted changes to the
> tree).

Thanks, done.

> 
> It would be easier if your will split your improvements into series
> of atomic changes and submit them individually for review.

I split the commits and now are GLEP 66 compatible. They are enough atomic
to be cherry-picked.


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Add rust eclass to support multi-target compilation

[Dirkjan Ochtman]

[-- Attachment #1: Type: text/plain, Size: 3359 bytes --]

On Tue, Jul 31, 2018 at 12:03 PM Luca Barbato <lu_zero@gentoo.org> wrote:

> > As far as I know, the Rust ecosystem is largely bimodal: stuff is either
> > compatible with stable and later, or it works only on nightly. It seems
> > very rare that code is actually tied to a particular Rust release and
> does
> > not compile against latest Rust stable. (Upstream actually promises they
> > won't break code except in case they need to fix a soundness issue in the
> > compiler.) So instead of building this whole target infrastructure (which
> > is definitely needed for something like Python or Ruby), we should be
> able
> > to just work with stable and nightly slots, and ebuilds can depend on
> those.
>
> This is true until you do not want to use libsyntax and other
> compiler-specific libraries.
>
> Because of the enforced ABI-randomness, what you built against stable
> must be rebuilt on stable.1, making those libraries non-shared might be
> a topic.
>

I'm not sure what you mean here, can you give an example?


> > Upstream is also very clearly packaging their core tooling as a single
> > package made up of a number of components, and these are distributed and
> > compiled together in common usage on other platforms (with the rustup
> > tooling). That also means they are tightly coupled -- for example,
> rustfmt
> > can change formats over time, and CI that checks formatting will need to
> > align with whatever the current stable Rust version of rustfmt is.
>
> Actually rustfmt is interesting since it does use libsyntax and
> depending on which project you may have to use the nightly rustfmt while
> targeting stable.
>

Yeah, but in my suggested approach we would have slotted rustfmt.


> > Installing the latest (nightly or beta) version of rustfmt while you
> have a
> > stable Rust toolchain installed is thus not a good idea. As another
> > example, cargo is now tagged as 0.28, but when passing --version it will
> > report as 1.27.0 -- actually the Rust toolchain version.
>
> Well it might be surprising, but for at least one project is actually
> required :)
>

So what project is that, and what exactly does it require? At some point if
your project requires very custom stuff maybe you should just build your
own Rust.


> > For these reasons, I think it would be better to align our Rust ebuilds
> > with upstream and work with single ebuilds (dev-lang/rust and
> > dev-lang/rust-bin) that install all the tools belonging to a particular
> > version of the Rust toolchain.
>
> > What tools are installed can be customized
> > with USE flags, and the default install can be minimal (just rustc and
> > cargo, which you need to build packages anyway).
>
> So once you need bindgen you have to rebuild rustc and then you need
> clippy you build again rustc?
>

Yes, it can be a pain, but I don't think "switching what auxiliary tools
are installed" is the common scenario that we should optimize for
necessarily.


> And what happens once you have yet-another-tool using libsyntax, you add
> it to the rustc ebuild and unleash a -r1 to the users?
>

If upstream includes another tool in their distribution, we add that to the
rust ebuild, yes.

I think there's a lot of value in aligning with upstream and with how
something is distributed on other platforms, because this reduces friction
for our users.

Regards,

Dirkjan

[-- Attachment #2: Type: text/html, Size: 4520 bytes --]