[gentoo-dev] dev-python/cryptography to use rust, effectively killing alpha, hppa, ia64, m68k, s390

[gentoo-dev] dev-python/cryptography to use rust, effectively killing alpha, hppa, ia64, m68k, s390

[Michał Górny]

Hi,

FYI the developers of dev-python/cryptography decided that Rust is going
to be mandatory for 1.5+ versions.  It's unlikely that they're going to
provide LTS support or security fixes for the old versions.

Since cryptography is a very important package in the Python ecosystem,
and it is an indirect dependency of Portage, this means that we will
probably have to entirely drop support for architectures that are not
supported by Rust.

According to upstream platform support information [1], this probably
means (eventually) entirely removing the following architectures:
- alpha (stable)
- hppa (stable)
- ia64 (stable)
- m68k (exp)
- s390 (except for s390x, exp)

Furthermore, the Gentoo Rust packages are missing support
for the following platforms, apparently supported upstream:
- mips (exp)
- ppc (32) (stable)
- sparc (stable)
- s390x (exp)
- riscv (stable)

Apparently it's non-trivial to bootstrap Rust on these platforms,
so it's unclear when Gentoo is going to start providing Rust on them.

I've raised a protest on the cryptography bug tracker [2] but apparently
upstream considers Rust's 'memory safety' more important than ability to
actually use the package.

Honestly, I don't think it likely that Rust will gain support for these
platforms.  This involves a lot of work, starting with writing a new
LLVM backend and getting it accepted (getting new code into LLVM is very
hard unless you're doing that on behalf one of the big companies).  You
can imagine how much effort that involves compared to rewriting the new
code from Cryptography into C.

If we can't convince upstream, I'm afraid we'll either have to drop
these architectures entirely or fork Cryptography.


[1] https://doc.rust-lang.org/nightly/rustc/platform-support.html
[2] https://github.com/pyca/cryptography/issues/5771

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] dev-python/cryptography to use rust, effectively killing alpha, hppa, ia64, m68k, s390

[Pacho Ramos]

[-- Attachment #1: Type: text/plain, Size: 2169 bytes --]

El lun, 08-02-2021 a las 12:19 +0100, Michał Górny escribió:
> Hi,
> 
> FYI the developers of dev-python/cryptography decided that Rust is going
> to be mandatory for 1.5+ versions.  It's unlikely that they're going to
> provide LTS support or security fixes for the old versions.
> 
> Since cryptography is a very important package in the Python ecosystem,
> and it is an indirect dependency of Portage, this means that we will
> probably have to entirely drop support for architectures that are not
> supported by Rust.
> 
> According to upstream platform support information [1], this probably
> means (eventually) entirely removing the following architectures:
> - alpha (stable)
> - hppa (stable)
> - ia64 (stable)
> - m68k (exp)
> - s390 (except for s390x, exp)
> 
> Furthermore, the Gentoo Rust packages are missing support
> for the following platforms, apparently supported upstream:
> - mips (exp)
> - ppc (32) (stable)
> - sparc (stable)
> - s390x (exp)
> - riscv (stable)
> 
> Apparently it's non-trivial to bootstrap Rust on these platforms,
> so it's unclear when Gentoo is going to start providing Rust on them.
> 
> I've raised a protest on the cryptography bug tracker [2] but apparently
> upstream considers Rust's 'memory safety' more important than ability to
> actually use the package.
> 
> Honestly, I don't think it likely that Rust will gain support for these
> platforms.  This involves a lot of work, starting with writing a new
> LLVM backend and getting it accepted (getting new code into LLVM is very
> hard unless you're doing that on behalf one of the big companies).  You
> can imagine how much effort that involves compared to rewriting the new
> code from Cryptography into C.
> 
> If we can't convince upstream, I'm afraid we'll either have to drop
> these architectures entirely or fork Cryptography.
> 
> 
> [1] https://doc.rust-lang.org/nightly/rustc/platform-support.html
> [2] https://github.com/pyca/cryptography/issues/5771
> 

It will also affect non-SSE2 x86 setups as upstream is neither providing
prebuilt package for them :/

https://bugs.gentoo.org/741708



[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] dev-python/cryptography to use rust, effectively killing alpha, hppa, ia64, m68k, s390

[James Le Cuirot]

[-- Attachment #1: Type: text/plain, Size: 977 bytes --]

On Mon, 08 Feb 2021 12:19:13 +0100
Michał Górny <mgorny@gentoo.org> wrote:

> Hi,
> 
> FYI the developers of dev-python/cryptography decided that Rust is going
> to be mandatory for 1.5+ versions.  It's unlikely that they're going to
> provide LTS support or security fixes for the old versions.
> 
> <snip>
>
> Honestly, I don't think it likely that Rust will gain support for these
> platforms.  This involves a lot of work, starting with writing a new
> LLVM backend and getting it accepted (getting new code into LLVM is very
> hard unless you're doing that on behalf one of the big companies).  You
> can imagine how much effort that involves compared to rewriting the new
> code from Cryptography into C.

I do know that there is ongoing work to get Rust going on m68k but it's
very slow and I'm not entirely confident they'll get there. At least I
don't have this package on my m68k system!

-- 
James Le Cuirot (chewi)
Gentoo Linux Developer

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] dev-python/cryptography to use rust, effectively killing alpha, hppa, ia64, m68k, s390

[Sam James]

[-- Attachment #1: Type: text/plain, Size: 1445 bytes --]


> On 8 Feb 2021, at 11:19, Michał Górny <mgorny@gentoo.org> wrote:
> 
> Hi,
> 
> FYI the developers of dev-python/cryptography decided that Rust is going
> to be mandatory for 1.5+ versions.  It's unlikely that they're going to
> provide LTS support or security fixes for the old versions.
> 
> Since cryptography is a very important package in the Python ecosystem,
> and it is an indirect dependency of Portage, this means that we will
> probably have to entirely drop support for architectures that are not
> supported by Rust.
> 

It seems that there was no mention of this in release notes or the changelog -
neither to warn packagers or consult with them for their views on this, which would’ve
been the best place to do so.

> [snip]
> 
> Furthermore, the Gentoo Rust packages are missing support
> for the following platforms, apparently supported upstream:
> - mips (exp)
> - ppc (32) (stable)
> - sparc (stable)
> - s390x (exp)
> - riscv (stable)

We also need to add support for armv4/armv5.

> 
> Apparently it's non-trivial to bootstrap Rust on these platforms,
> so it's unclear when Gentoo is going to start providing Rust on them.
> 
> [snip]

Filed a Gentoo bug too: https://bugs.gentoo.org/769482.

> [1] https://doc.rust-lang.org/nightly/rustc/platform-support.html
> [2] https://github.com/pyca/cryptography/issues/5771
> 
> --
> Best regards,
> Michał Górny
> 
> 
> 


[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] dev-python/cryptography to use rust, effectively killing alpha, hppa, ia64, m68k, s390

[Michael Orlitzky]

On Mon, 2021-02-08 at 12:19 +0100, Michał Górny wrote:
> 
> Since cryptography is a very important package in the Python ecosystem,
> and it is an indirect dependency of Portage, this means that we will
> probably have to entirely drop support for architectures that are not
> supported by Rust.
> 

Not only that, but you will be dropping support for at least two of my
machines that are literally incapable of building the 10+ GiB bundled
rust package due to the amount of disk space and RAM required.

Re: [gentoo-dev] dev-python/cryptography to use rust, effectively killing alpha, hppa, ia64, m68k, s390

[Joonas Niilola]

[-- Attachment #1.1: Type: text/plain, Size: 417 bytes --]



On 2/8/21 2:27 PM, Michael Orlitzky wrote:
> Not only that, but you will be dropping support for at least two of my
> machines that are literally incapable of building the 10+ GiB bundled
> rust package due to the amount of disk space and RAM required.
>
Pardon my intervention, but there is rust-bin available at least for
amd64, x86, arm, arm64 and ppc64. Just a PSA to anyone not aware.

-- juippis


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] dev-python/cryptography to use rust, effectively killing alpha, hppa, ia64, m68k, s390

[Brian Evans]

[-- Attachment #1.1: Type: text/plain, Size: 835 bytes --]

On 2/8/2021 6:19 AM, Michał Górny wrote:
> Hi,
> 
> FYI the developers of dev-python/cryptography decided that Rust is going
> to be mandatory for 1.5+ versions.  It's unlikely that they're going to
> provide LTS support or security fixes for the old versions.
> 
> Since cryptography is a very important package in the Python ecosystem,
> and it is an indirect dependency of Portage, this means that we will
> probably have to entirely drop support for architectures that are not
> supported by Rust.
> 

For the portage indirect dependency, can it be swapped for pycurl?

AFAICT, it is just used to pull GPG sigs in gemato via dev-python/requests.

This at least would at least keep the arches in question with some 
support but not necessarily all of python world until a clearer plan can 
be made.

Brian


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

Re: [gentoo-dev] dev-python/cryptography to use rust, effectively killing alpha, hppa, ia64, m68k, s390

[Joonas Niilola]

[-- Attachment #1.1: Type: text/plain, Size: 265 bytes --]



On 2/8/21 7:53 PM, Brian Evans wrote:
>
> AFAICT, it is just used to pull GPG sigs in gemato via
> dev-python/requests.
>
And it that's true, isn't this only relevant when syncing the tree using
rsync? So using git would still be viable?

-- juippis


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] dev-python/cryptography to use rust, effectively killing alpha, hppa, ia64, m68k, s390

[Michał Górny]

On Mon, 2021-02-08 at 12:53 -0500, Brian Evans wrote:
> On 2/8/2021 6:19 AM, Michał Górny wrote:
> > Hi,
> > 
> > FYI the developers of dev-python/cryptography decided that Rust is going
> > to be mandatory for 1.5+ versions.  It's unlikely that they're going to
> > provide LTS support or security fixes for the old versions.
> > 
> > Since cryptography is a very important package in the Python ecosystem,
> > and it is an indirect dependency of Portage, this means that we will
> > probably have to entirely drop support for architectures that are not
> > supported by Rust.
> > 
> 
> For the portage indirect dependency, can it be swapped for pycurl?
> 
> AFAICT, it is just used to pull GPG sigs in gemato via dev-python/requests.

I'll probably switch it to built-in urllib.request.  The py3 versions
should be good enough (previously urllib3 had better TLS support than
old urllib AFAIR).

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] dev-python/cryptography to use rust, effectively killing alpha, hppa, ia64, m68k, s390

[Brian Evans]

[-- Attachment #1.1: Type: text/plain, Size: 1270 bytes --]

On 2/8/2021 12:53 PM, Brian Evans wrote:
> On 2/8/2021 6:19 AM, Michał Górny wrote:
>> Hi,
>>
>> FYI the developers of dev-python/cryptography decided that Rust is going
>> to be mandatory for 1.5+ versions.  It's unlikely that they're going to
>> provide LTS support or security fixes for the old versions.
>>
>> Since cryptography is a very important package in the Python ecosystem,
>> and it is an indirect dependency of Portage, this means that we will
>> probably have to entirely drop support for architectures that are not
>> supported by Rust.
>>
> 
> For the portage indirect dependency, can it be swapped for pycurl?
> 
> AFAICT, it is just used to pull GPG sigs in gemato via dev-python/requests.
> 
> This at least would at least keep the arches in question with some 
> support but not necessarily all of python world until a clearer plan can 
> be made.
> 
> Brian
> 

After discussion in #gentoo-dev and simple chroot testing, it seems like 
dev-python/requests nor dev-python/urlllib3 needs 
dev-python/cryptography at all (when run with stable Python, tested with 
3.8).   So unless there's a really good reason outside of gemato sync, 
this looks to be a non-issue for portage and more of a dependency fix.

Brian


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

Re: [gentoo-dev] dev-python/cryptography to use rust, effectively killing alpha, hppa, ia64, m68k, s390

[Mike Gilbert]

On Mon, Feb 8, 2021 at 1:35 PM Brian Evans <grknight@gentoo.org> wrote:
>
> On 2/8/2021 12:53 PM, Brian Evans wrote:
> > On 2/8/2021 6:19 AM, Michał Górny wrote:
> >> Hi,
> >>
> >> FYI the developers of dev-python/cryptography decided that Rust is going
> >> to be mandatory for 1.5+ versions.  It's unlikely that they're going to
> >> provide LTS support or security fixes for the old versions.
> >>
> >> Since cryptography is a very important package in the Python ecosystem,
> >> and it is an indirect dependency of Portage, this means that we will
> >> probably have to entirely drop support for architectures that are not
> >> supported by Rust.
> >>
> >
> > For the portage indirect dependency, can it be swapped for pycurl?
> >
> > AFAICT, it is just used to pull GPG sigs in gemato via dev-python/requests.
> >
> > This at least would at least keep the arches in question with some
> > support but not necessarily all of python world until a clearer plan can
> > be made.
> >
> > Brian
> >
>
> After discussion in #gentoo-dev and simple chroot testing, it seems like
> dev-python/requests nor dev-python/urlllib3 needs
> dev-python/cryptography at all (when run with stable Python, tested with
> 3.8).   So unless there's a really good reason outside of gemato sync,
> this looks to be a non-issue for portage and more of a dependency fix.

I created a PR to drop this and related deps from dev-python/urllib3
and dev-python/requests. Please review.

https://github.com/gentoo/gentoo/pull/19383

Re: [gentoo-dev] dev-python/cryptography to use rust, effectively killing alpha, hppa, ia64, m68k, s390

[Alessandro Barbieri]

[-- Attachment #1: Type: text/plain, Size: 2186 bytes --]

Il Lun 8 Feb 2021, 12:19 Michał Górny <mgorny@gentoo.org> ha scritto:

> Hi,
>
> FYI the developers of dev-python/cryptography decided that Rust is going
> to be mandatory for 1.5+ versions.  It's unlikely that they're going to
> provide LTS support or security fixes for the old versions.
>
> Since cryptography is a very important package in the Python ecosystem,
> and it is an indirect dependency of Portage, this means that we will
> probably have to entirely drop support for architectures that are not
> supported by Rust.
>
> According to upstream platform support information [1], this probably
> means (eventually) entirely removing the following architectures:
> - alpha (stable)
> - hppa (stable)
> - ia64 (stable)
> - m68k (exp)
> - s390 (except for s390x, exp)
>
> Furthermore, the Gentoo Rust packages are missing support
> for the following platforms, apparently supported upstream:
> - mips (exp)
> - ppc (32) (stable)
> - sparc (stable)
> - s390x (exp)
> - riscv (stable)
>
> Apparently it's non-trivial to bootstrap Rust on these platforms,
> so it's unclear when Gentoo is going to start providing Rust on them.
>
> I've raised a protest on the cryptography bug tracker [2] but apparently
> upstream considers Rust's 'memory safety' more important than ability to
> actually use the package.
>
> Honestly, I don't think it likely that Rust will gain support for these
> platforms.  This involves a lot of work, starting with writing a new
> LLVM backend and getting it accepted (getting new code into LLVM is very
> hard unless you're doing that on behalf one of the big companies).  You
> can imagine how much effort that involves compared to rewriting the new
> code from Cryptography into C.
>
> If we can't convince upstream, I'm afraid we'll either have to drop
> these architectures entirely or fork Cryptography.
>
>
> [1] https://doc.rust-lang.org/nightly/rustc/platform-support.html
> [2] https://github.com/pyca/cryptography/issues/5771
>
> --
> Best regards,
> Michał Górny
>

Should we shed tears for those legacy architectures or move forward? Does
anyone really use them in production?

>

[-- Attachment #2: Type: text/html, Size: 3051 bytes --]

Re: [gentoo-dev] dev-python/cryptography to use rust, effectively killing alpha, hppa, ia64, m68k, s390

[Alec Warner]

On Mon, Feb 8, 2021 at 9:56 AM Alessandro Barbieri
<lssndrbarbieri@gmail.com> wrote:
>
> Il Lun 8 Feb 2021, 12:19 Michał Górny <mgorny@gentoo.org> ha scritto:
>>
>> Hi,
>>
>> FYI the developers of dev-python/cryptography decided that Rust is going
>> to be mandatory for 1.5+ versions.  It's unlikely that they're going to
>> provide LTS support or security fixes for the old versions.
>>
>> Since cryptography is a very important package in the Python ecosystem,
>> and it is an indirect dependency of Portage, this means that we will
>> probably have to entirely drop support for architectures that are not
>> supported by Rust.
>>
>> According to upstream platform support information [1], this probably
>> means (eventually) entirely removing the following architectures:
>> - alpha (stable)
>> - hppa (stable)
>> - ia64 (stable)
>> - m68k (exp)
>> - s390 (except for s390x, exp)
>>
>> Furthermore, the Gentoo Rust packages are missing support
>> for the following platforms, apparently supported upstream:
>> - mips (exp)
>> - ppc (32) (stable)
>> - sparc (stable)
>> - s390x (exp)
>> - riscv (stable)
>>
>> Apparently it's non-trivial to bootstrap Rust on these platforms,
>> so it's unclear when Gentoo is going to start providing Rust on them.
>>
>> I've raised a protest on the cryptography bug tracker [2] but apparently
>> upstream considers Rust's 'memory safety' more important than ability to
>> actually use the package.
>>
>> Honestly, I don't think it likely that Rust will gain support for these
>> platforms.  This involves a lot of work, starting with writing a new
>> LLVM backend and getting it accepted (getting new code into LLVM is very
>> hard unless you're doing that on behalf one of the big companies).  You
>> can imagine how much effort that involves compared to rewriting the new
>> code from Cryptography into C.
>>
>> If we can't convince upstream, I'm afraid we'll either have to drop
>> these architectures entirely or fork Cryptography.
>>
>>
>> [1] https://doc.rust-lang.org/nightly/rustc/platform-support.html
>> [2] https://github.com/pyca/cryptography/issues/5771
>>
>> --
>> Best regards,
>> Michał Górny
>
>
> Should we shed tears for those legacy architectures or move forward? Does anyone really use them in production?

Many users don't use gentoo in 'production' so I'm not sure how that
matters in our decision making. I'm not sure the trade off here
(taking rust as a core dep) is reasonable and it looks like we can
avoid it.

-A

Re: [gentoo-dev] dev-python/cryptography to use rust, effectively killing alpha, hppa, ia64, m68k, s390

[Aaron Bauman]

[-- Attachment #1: Type: text/plain, Size: 660 bytes --]

On Mon, Feb 08, 2021 at 10:02:43AM -0800, Alec Warner wrote:
> On Mon, Feb 8, 2021 at 9:56 AM Alessandro Barbieri
> <lssndrbarbieri@gmail.com> wrote:
> >
> > Il Lun 8 Feb 2021, 12:19 Michał Górny <mgorny@gentoo.org> ha scritto:
> >>

<snip>

> >
> > Should we shed tears for those legacy architectures or move forward? Does anyone really use them in production?
> 
> Many users don't use gentoo in 'production' so I'm not sure how that
> matters in our decision making. I'm not sure the trade off here
> (taking rust as a core dep) is reasonable and it looks like we can
> avoid it.
>

Do we have anything to support this claim?

-Aaron

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] dev-python/cryptography to use rust, effectively killing alpha, hppa, ia64, m68k, s390

[Michał Górny]

On Mon, 2021-02-08 at 18:56 +0100, Alessandro Barbieri wrote:
> Il Lun 8 Feb 2021, 12:19 Michał Górny <mgorny@gentoo.org> ha scritto:
> 
> > Hi,
> > 
> > FYI the developers of dev-python/cryptography decided that Rust is going
> > to be mandatory for 1.5+ versions.  It's unlikely that they're going to
> > provide LTS support or security fixes for the old versions.
> > 
> > Since cryptography is a very important package in the Python ecosystem,
> > and it is an indirect dependency of Portage, this means that we will
> > probably have to entirely drop support for architectures that are not
> > supported by Rust.
> > 
> > According to upstream platform support information [1], this probably
> > means (eventually) entirely removing the following architectures:
> > - alpha (stable)
> > - hppa (stable)
> > - ia64 (stable)
> > - m68k (exp)
> > - s390 (except for s390x, exp)
> > 
> > Furthermore, the Gentoo Rust packages are missing support
> > for the following platforms, apparently supported upstream:
> > - mips (exp)
> > - ppc (32) (stable)
> > - sparc (stable)
> > - s390x (exp)
> > - riscv (stable)
> > 
> > Apparently it's non-trivial to bootstrap Rust on these platforms,
> > so it's unclear when Gentoo is going to start providing Rust on them.
> > 
> > I've raised a protest on the cryptography bug tracker [2] but apparently
> > upstream considers Rust's 'memory safety' more important than ability to
> > actually use the package.
> > 
> > Honestly, I don't think it likely that Rust will gain support for these
> > platforms.  This involves a lot of work, starting with writing a new
> > LLVM backend and getting it accepted (getting new code into LLVM is very
> > hard unless you're doing that on behalf one of the big companies).  You
> > can imagine how much effort that involves compared to rewriting the new
> > code from Cryptography into C.
> > 
> > If we can't convince upstream, I'm afraid we'll either have to drop
> > these architectures entirely or fork Cryptography.
> > 
> > 
> > [1] https://doc.rust-lang.org/nightly/rustc/platform-support.html
> > [2] https://github.com/pyca/cryptography/issues/5771
> > 
> > --
> > Best regards,
> > Michał Górny
> > 
> 
> Should we shed tears for those legacy architectures or move forward? Does
> anyone really use them in production?

I'm pretty sure people do use them in production.  Whether we want to
really continue supporting them is a separate topic.  But the last thing
I'd like to do is drop support for arches because of a single upstream
maintainer being a Rust lover.

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] dev-python/cryptography to use rust, effectively killing alpha, hppa, ia64, m68k, s390

[Michał Górny]

On Mon, 2021-02-08 at 12:19 +0100, Michał Górny wrote:
> FYI the developers of dev-python/cryptography decided that Rust is going
> to be mandatory for 1.5+ versions.  It's unlikely that they're going to
> provide LTS support or security fixes for the old versions.
> 
> Since cryptography is a very important package in the Python ecosystem,
> and it is an indirect dependency of Portage, this means that we will
> probably have to entirely drop support for architectures that are not
> supported by Rust.
> 
> I...]
> 
> I've raised a protest on the cryptography bug tracker [2] but apparently
> upstream considers Rust's 'memory safety' more important than ability to
> actually use the package.
> 
> Honestly, I don't think it likely that Rust will gain support for these
> platforms.  This involves a lot of work, starting with writing a new
> LLVM backend and getting it accepted (getting new code into LLVM is very
> hard unless you're doing that on behalf one of the big companies).  You
> can imagine how much effort that involves compared to rewriting the new
> code from Cryptography into C.
> 
> If we can't convince upstream, I'm afraid we'll either have to drop
> these architectures entirely or fork Cryptography.
> 
> 
> [1] https://doc.rust-lang.org/nightly/rustc/platform-support.html
> [2] https://github.com/pyca/cryptography/issues/5771

So it seems that upstream has practically closed the discussion,
and the short summary is that they only care for the 'majority' of
users, they don't care for minor platforms (but we're free to port
LLVM/Rust to them) and -- unsurprisingly -- this is a part of crusade
towards promoting Rust.

Given the aggressive opinions of a number of Python core devs
participating in the discussion, I'm afraid that it is quite probable
that a future version of CPython may require Rust.  In fact, they've
already started having knee-jerk reactions to the problem at hand [1]. 
To be honest, I've never thought I'd be this disappointed in Python
upstream.

Good news is that they've promised to keep a LTS branch with security
fixes to the non-Rust version.  Until end-of-year.  And they've pretty
aggressively stated that they won't fix anything except security bugs
with a CVE assigned.  So if it stops building for whatever reason, we're
on our own.

I've reached out to Debian and they're planning to remove support for
minor architectures for this package in the next release.  However,
Python is not as central to them as it is to us.  Alpine is also
affected but seems intent on pushing Rust forward, so they'll probably
drop these architectures as well.

Mike's submitted a PR to remove (unnecessary) cryptography dep from our
urllib3/requests packages [2].  This should make it possible to avoid
cryptography at least on some systems.  However, it is still an indirect
test dependency of these packages, so we're going to have a hard time
keeping them properly tested.

At this point, I'm really depressed about this and I'm seriously
wondering why I'm wasting so much effort on open source.  I don't see
a good way out of it.  Rust could be a nice language -- but it won't if
it continues to be surround by arrogant zealots who want to destroy
everything in their path towards promoting it.

The first big blocker we're going to hit is trustme [3] package that
relies on cryptography API pretty heavily to generate TLS certs for
testing.  If we managed to convince upstream to support an alternate
crypto backend, we'd be able to retain minor keywords a lot of packages
without too much pain.

[1] https://bugs.python.org/issue43179
[2] https://github.com/gentoo/gentoo/pull/19383
[3] https://github.com/python-trio/trustme

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] dev-python/cryptography to use rust, effectively killing alpha, hppa, ia64, m68k, s390

[Benda Xu]

Hi Michał,

Michał Górny <mgorny@gentoo.org> writes:

> So it seems that upstream has practically closed the discussion,
> and the short summary is that they only care for the 'majority' of
> users, they don't care for minor platforms (but we're free to port
> LLVM/Rust to them) and -- unsurprisingly -- this is a part of crusade
> towards promoting Rust.
>
> Given the aggressive opinions of a number of Python core devs
> participating in the discussion, I'm afraid that it is quite probable
> that a future version of CPython may require Rust.  In fact, they've
> already started having knee-jerk reactions to the problem at hand [1]. 
> To be honest, I've never thought I'd be this disappointed in Python
> upstream.
>
> Good news is that they've promised to keep a LTS branch with security
> fixes to the non-Rust version.  Until end-of-year.  And they've pretty
> aggressively stated that they won't fix anything except security bugs
> with a CVE assigned.  So if it stops building for whatever reason, we're
> on our own.
>
> I've reached out to Debian and they're planning to remove support for
> minor architectures for this package in the next release.  However,
> Python is not as central to them as it is to us.  Alpine is also
> affected but seems intent on pushing Rust forward, so they'll probably
> drop these architectures as well.
>
> Mike's submitted a PR to remove (unnecessary) cryptography dep from our
> urllib3/requests packages [2].  This should make it possible to avoid
> cryptography at least on some systems.  However, it is still an indirect
> test dependency of these packages, so we're going to have a hard time
> keeping them properly tested.
>
> At this point, I'm really depressed about this and I'm seriously
> wondering why I'm wasting so much effort on open source.  I don't see
> a good way out of it.  Rust could be a nice language -- but it won't if
> it continues to be surround by arrogant zealots who want to destroy
> everything in their path towards promoting it.
>
> The first big blocker we're going to hit is trustme [3] package that
> relies on cryptography API pretty heavily to generate TLS certs for
> testing.  If we managed to convince upstream to support an alternate
> crypto backend, we'd be able to retain minor keywords a lot of packages
> without too much pain.

I could feel the pain.  

Bootstraping Rust on Prefix is somewhere between alpha, hppa, ia64,
m68k, s390 and amd64[1].  The problem was exposed by
gnome-base/librsvg[2].

I am wondering how useable pkgcore is on alpha, hppa, etc.  Maybe it's
time for us to plan for a Gentoo without essential Python dependency.

Looking forward to gcc-rust for saving the world in the end.

Benda

1. https://bugs.gentoo.org/689160
2. https://bugs.gentoo.org/739574

Re: [gentoo-dev] dev-python/cryptography to use rust, effectively killing alpha, hppa, ia64, m68k, s390

[Tim Harder]

On 2021-02-09 Tue 17:51, Benda Xu wrote:
> I am wondering how useable pkgcore is on alpha, hppa, etc.  Maybe it's
> time for us to plan for a Gentoo without essential Python dependency.

Just to keep misinformation down, pkgcore currently has nothing to do
with rust as it's implemented in python due to basically being deemed
portage-ng when it forked ~15 years ago. It previously did have some
extensions written in C which have mostly been removed in the current
day from CPython catching up in a number of areas.

That being said, alternative languages and related support has been
looked at for a number of reasons/features and development could move in
that direction, but hasn't yet in a public fashion.

Tim

Re: [gentoo-dev] dev-python/cryptography to use rust, effectively killing alpha, hppa, ia64, m68k, s390

[Alexey Sokolov]

ср, 10 февр. 2021 г. в 01:16, Tim Harder <radhermit@gentoo.org>:
>
> On 2021-02-09 Tue 17:51, Benda Xu wrote:
> > I am wondering how useable pkgcore is on alpha, hppa, etc.  Maybe it's
> > time for us to plan for a Gentoo without essential Python dependency.
>
> Just to keep misinformation down, pkgcore currently has nothing to do
> with rust as it's implemented in python due to basically being deemed
> portage-ng when it forked ~15 years ago. It previously did have some
> extensions written in C which have mostly been removed in the current
> day from CPython catching up in a number of areas.
>
> That being said, alternative languages and related support has been
> looked at for a number of reasons/features and development could move in
> that direction, but hasn't yet in a public fashion.
>
> Tim
>

Another far-fetched option is to compile CPython to WebAssembly+WASI
[1] (even if CPython itself will require Rust in future), and run it
on the non-Rust-supported architecture via an interpreter [2].

If this works at all, bootstrap will probably be not trivial. And such
version of interpreted python may need to be added to PYTHON_TARGETS
to enable testing of this on more usual architectures like amd64.

[1] https://wasi.dev/
[2] https://github.com/wasm3/wasm3

Re: [gentoo-dev] dev-python/cryptography to use rust, effectively killing alpha, hppa, ia64, m68k, s390

[Michael Orlitzky]

On Wed, 2021-02-10 at 08:51 +0800, Benda Xu wrote:
> > 
> > The first big blocker we're going to hit is trustme [3] package that
> > relies on cryptography API pretty heavily to generate TLS certs for
> > testing.  If we managed to convince upstream to support an alternate
> > crypto backend, we'd be able to retain minor keywords a lot of packages
> > without too much pain.
> 
> I could feel the pain.  
> 
> Bootstraping Rust on Prefix is somewhere between alpha, hppa, ia64,
> m68k, s390 and amd64[1].  The problem was exposed by
> gnome-base/librsvg[2].
> 
> I am wondering how useable pkgcore is on alpha, hppa, etc.  Maybe it's
> time for us to plan for a Gentoo without essential Python dependency.

It's not usable anywhere. We keep updating the PMS, and the council
keeps voting to approve the new versions, and we teach all new
developers that they need to respect both the PMS and council
decisions... and then from that day on, everyone completely, publicly,
ignores it, adding thousands of packages across entire ecosystems that
don't work properly without portage. I'd like to say it's one of the
craziest things I've ever seen, but, there was 2020.

We "started" with three package managers, and now we're down to one.
The council needs to grow some balls and enforce the PMS before that
can change. Pkgcore could be salvaged if you could actually update your
system with it.

For my "me too," I've been told by upstream that the next major version
of clamav will require rust. There are a lot of "real" UNIX machines
relying on clamav for e.g. PCI compliance that will be screwed by that,
not to mention all of the small business routers and mail gateways on
obscure or limited hardware. Personally, I just haven't spent the last
20 years contributing to free software to be casually migrated to a
system of bundled binary blobs; nor am I able to set aside 8GB of RAM
for hours to rebuild the latest version of rust and its myriad
unofficial dependencies every day on a production mail server.
Eventually clamav will disappear, and we'll likely move a few big
customers to Microsoft O365 as a result.

Re: [gentoo-dev] dev-python/cryptography to use rust, effectively killing alpha, hppa, ia64, m68k, s390

[Peter Stuge]

[-- Attachment #1: Type: text/plain, Size: 2089 bytes --]

Michał Górny wrote:
> I'm seriously wondering why I'm wasting so much effort on open source.

Open source only ever works when taking responsibility for one's problems.


> I don't see a good way out of it.

I see a couple. Of course all require some effort.

One was already mentioned; move Gentoo package management from Python
to some compiled language. High effort but maximum independence/reward.

Another option, less effort and less reward, is to investigate how much
CPython portage in fact requires, and make that a special package in Gentoo.

This essentially means a special-purpose fork of CPython, only for running
portage. Obviously portage development must then be comfortable without
using the latest shiny Python language stuff that only future RustPython
will offer. I guess that's not a problem.

Yet another is a variant on the previous, but even less effort and
much less reward; freeze what language stuff is allowed in portage code
and always run portage with some chosen existing/later CPython version.


Like libressl and gtk2 this thread also converges on the common point in
my argumentation: it's not per se bad, and sometimes supremely wise, to
quit chasing the latest version, and rest on a known platform.

Coupled with independent efforts to place security-relevant parts in
isolated environments (see sandbox) - ongoing effort regardless of Python -
I don't see why portage couldn't depend on a CPython interpreter of its own,
some last version that works well and is then copied and renamed.

It seems like that would be rather straightforward.

It might also be a good thing to take portage out of the overall Gentoo
Python picture? I don't know here - this bit is just a guess.


> arrogant zealots who want to destroy everything in their path

LOL, priceless.


> The first big blocker we're going to hit is trustme [3] package that
> relies on cryptography API pretty heavily to generate TLS certs for
> testing.

For which testing? Could it be changed to generate certs differently?

CAs aren't magic. Attached is a basic script of mine.


//Peter

[-- Attachment #2: mkca.sh --]
[-- Type: application/x-sh, Size: 2535 bytes --]

Re: [gentoo-dev] dev-python/cryptography to use rust, effectively killing alpha, hppa, ia64, m68k, s390

[Mike Gilbert]

On Wed, Feb 10, 2021 at 4:57 PM Peter Stuge <peter@stuge.se> wrote:
> > The first big blocker we're going to hit is trustme [3] package that
> > relies on cryptography API pretty heavily to generate TLS certs for
> > testing.
>
> For which testing? Could it be changed to generate certs differently?

He is probably referring to the test suite in dev-python/urllib3,
which uses the python "trustme" package in several places.

https://github.com/urllib3/urllib3/search?q=trustme

Someone would need to convince the urllib3 developers to use something
else to manage certificates in their unit tests.

Re: [gentoo-dev] dev-python/cryptography to use rust, effectively killing alpha, hppa, ia64, m68k, s390

[Peter Stuge]

Mike Gilbert wrote:
> > > The first big blocker we're going to hit is trustme [3] package that
> > > relies on cryptography API pretty heavily to generate TLS certs for
> > > testing.
> >
> > For which testing? Could it be changed to generate certs differently?
> 
> He is probably referring to the test suite in dev-python/urllib3,
> which uses the python "trustme" package in several places.
> 
> https://github.com/urllib3/urllib3/search?q=trustme
> 
> Someone would need to convince the urllib3 developers to use something
> else to manage certificates in their unit tests.

Ah - non-Gentoo packages - thanks, I understand.

The trustme API isn't very large, so it seems doable to create a
different implementation of it without rust dependencies, add
virtual/trustme and be done?


//Peter