[gentoo-dev] [PATCH 0/3] net-misc/curl: add curl_quic USE_EXPAND

[gentoo-dev] [PATCH 0/3] net-misc/curl: add curl_quic USE_EXPAND

[kangie]

From: Matt Jolly <kangie@gentoo.org>

Hi all,

Thanks to dev-libs/openssl exposing an RFC 9000 (QUIC) API from 3.2,
cURL is able to use OpenSSL as a QUIC backend in addition
to its current suppport via net-libs/ngtcp2.

Considering the current state of QUIC implementations, the popularity
of cURL, its support for a multitude of TLS implementations (in Gentoo
we support OpenSSL, GnuTLS, Mbed TLS, and Rustls), I think it makes sense
to implement this by introducing a new USE_EXPAND variable.

This will make it easier to manage the ebuild logic for QUIC backends,
enables us to easily add new backends as they are supported upstream,
and lets users control the selection of QUIC implementations using a
familiar mechanism; the same way that they currently do (or do not)
select a TLS implementation.

The relevant patches are attached for consideration, and I've raised a
Pull Request[1] to enable easy review and comment. The PR also includes
the required configure and USE changes to dev-libs/openssl which aren't
relevant to the discussion of a new USE_EXPAND.

Cheers,

Matt

[1]: https://github.com/gentoo/gentoo/pull/37238

Matt Jolly (3):
  profiles/desc: add curl_quic
  profiles/base: make.defaults: add CURL_QUIC
  net-misc/curl: wire up live ebuild for openssl-quic

 net-misc/curl/curl-9999.ebuild | 70 +++++++++++++---------------------
 net-misc/curl/metadata.xml     |  4 +-
 profiles/base/make.defaults    |  2 +-
 profiles/desc/curl_quic.desc   |  7 ++++
 4 files changed, 38 insertions(+), 45 deletions(-)
 create mode 100644 profiles/desc/curl_quic.desc

-- 
2.45.2

[gentoo-dev] [PATCH 1/3] profiles/desc: add curl_quic

[kangie]

From: Matt Jolly <kangie@gentoo.org>

The CURL_QUIC USE_EXPAND enables us to sanely manage QUIC (RFC 9000)
backends as they are added to cURL in the future: currently there are
two supported implementations, OpenSSL and ngtcp2,  however it's likely
that other popular TLS libraries will expose QUIC APIs over time,
and that these will be eventually be supported by cURL (see CURL_SSL
for examples of TLS libraries that we support) - we may as well
get ahead of the curve here.

There are already a number of other small players (i.e. OpenSSL Forks)
exposing QUIC support for quite a while, however these have not been
available in ::gentoo and we've only needed the one USE to enable
for HTTP/3 and QUIC to this point.

Signed-off-by: Matt Jolly <kangie@gentoo.org>
---
 profiles/desc/curl_quic.desc | 7 +++++++
 1 file changed, 7 insertions(+)
 create mode 100644 profiles/desc/curl_quic.desc

diff --git a/profiles/desc/curl_quic.desc b/profiles/desc/curl_quic.desc
new file mode 100644
index 000000000000..372bb9ce8f83
--- /dev/null
+++ b/profiles/desc/curl_quic.desc
@@ -0,0 +1,7 @@
+# Copyright 1999-2024 Gentoo Authors
+# Distributed under the terms of the GNU General Public License v2
+
+# This file contains descriptions of CURL_QUIC USE_EXPAND flags for net-misc/curl
+
+openssl - Use OpenSSL
+ngtcp2 - Use ngtcp2
-- 
2.45.2

[gentoo-dev] [PATCH 2/3] profiles/base: make.defaults: add CURL_QUIC

[kangie]

From: Matt Jolly <kangie@gentoo.org>

Signed-off-by: Matt Jolly <kangie@gentoo.org>
---
 profiles/base/make.defaults | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/profiles/base/make.defaults b/profiles/base/make.defaults
index c70c5d198531..957af187bda2 100644
--- a/profiles/base/make.defaults
+++ b/profiles/base/make.defaults
@@ -12,7 +12,7 @@ USE_EXPAND_VALUES_KERNEL="Darwin linux SunOS"
 
 # Env vars to expand into USE vars.  Modifying this requires prior
 # discussion on gentoo-dev@lists.gentoo.org.
-USE_EXPAND="ABI_MIPS ABI_S390 ABI_X86 ADA_TARGET ALSA_CARDS AMDGPU_TARGETS APACHE2_MODULES APACHE2_MPMS CALLIGRA_FEATURES CAMERAS COLLECTD_PLUGINS CPU_FLAGS_ARM CPU_FLAGS_PPC CPU_FLAGS_X86 CURL_SSL ELIBC FFTOOLS GPSD_PROTOCOLS GRUB_PLATFORMS INPUT_DEVICES KERNEL L10N LCD_DEVICES LIBREOFFICE_EXTENSIONS LLVM_SLOT LLVM_TARGETS LUA_SINGLE_TARGET LUA_TARGETS MONKEYD_PLUGINS NGINX_MODULES_HTTP NGINX_MODULES_MAIL NGINX_MODULES_STREAM OFFICE_IMPLEMENTATION OPENMPI_FABRICS OPENMPI_OFED_FEATURES OPENMPI_RM PERL_FEATURES PHP_TARGETS POSTGRES_TARGETS PYTHON_SINGLE_TARGET PYTHON_TARGETS QEMU_SOFTMMU_TARGETS QEMU_USER_TARGETS RUBY_TARGETS SANE_BACKENDS UWSGI_PLUGINS VIDEO_CARDS VOICEMAIL_STORAGE XTABLES_ADDONS"
+USE_EXPAND="ABI_MIPS ABI_S390 ABI_X86 ADA_TARGET ALSA_CARDS AMDGPU_TARGETS APACHE2_MODULES APACHE2_MPMS CALLIGRA_FEATURES CAMERAS COLLECTD_PLUGINS CPU_FLAGS_ARM CPU_FLAGS_PPC CPU_FLAGS_X86 CURL_SSL CURL_QUIC ELIBC FFTOOLS GPSD_PROTOCOLS GRUB_PLATFORMS INPUT_DEVICES KERNEL L10N LCD_DEVICES LIBREOFFICE_EXTENSIONS LLVM_SLOT LLVM_TARGETS LUA_SINGLE_TARGET LUA_TARGETS MONKEYD_PLUGINS NGINX_MODULES_HTTP NGINX_MODULES_MAIL NGINX_MODULES_STREAM OFFICE_IMPLEMENTATION OPENMPI_FABRICS OPENMPI_OFED_FEATURES OPENMPI_RM PERL_FEATURES PHP_TARGETS POSTGRES_TARGETS PYTHON_SINGLE_TARGET PYTHON_TARGETS QEMU_SOFTMMU_TARGETS QEMU_USER_TARGETS RUBY_TARGETS SANE_BACKENDS UWSGI_PLUGINS VIDEO_CARDS VOICEMAIL_STORAGE XTABLES_ADDONS"
 
 # USE_EXPAND variables whose contents are not shown in package manager
 # output. Changes need discussion on gentoo-dev.
-- 
2.45.2

[gentoo-dev] [PATCH 3/3] net-misc/curl: wire up live ebuild for openssl-quic

[kangie]

From: Matt Jolly <kangie@gentoo.org>

Thanks to dev-libs/openssl exposing a QUIC API from
3.2, cURL is able to use OpenSSL as a QUIC backend.

This commit enables users to select between the
`curl_quic_ngtcp2` and `curl_quic_openssl` backends
via their associated USE_EXPAND and provides a mechanism
to easily add more backends as they are implemented.

It also renames the live ebuild's `nghttp3` USE to
`http3` to support its use as a generic feature
flag, and to align with net-analyzer/wireshark
and www-serners/nginx.

Finally, we get to drop a disgusting pkg-config
workaround thanks to some upstream efforts
that render it irrelevant (and detrimental
to a successful build).

Signed-off-by: Matt Jolly <kangie@gentoo.org>
---
 net-misc/curl/curl-9999.ebuild | 70 +++++++++++++---------------------
 net-misc/curl/metadata.xml     |  4 +-
 2 files changed, 30 insertions(+), 44 deletions(-)

diff --git a/net-misc/curl/curl-9999.ebuild b/net-misc/curl/curl-9999.ebuild
index 7f560fb74a9b..b42cca9c5152 100644
--- a/net-misc/curl/curl-9999.ebuild
+++ b/net-misc/curl/curl-9999.ebuild
@@ -26,16 +26,22 @@ fi
 
 LICENSE="BSD curl ISC test? ( BSD-4 )"
 SLOT="0"
-IUSE="+adns +alt-svc brotli debug +ftp gnutls gopher +hsts +http2 idn +imap kerberos ldap mbedtls nghttp3 +openssl +pop3"
-IUSE+=" +psl +progress-meter rtmp rustls samba +smtp ssh ssl sslv3 static-libs test telnet +tftp websockets zstd"
-# These select the default SSL implementation
-IUSE+=" curl_ssl_gnutls curl_ssl_mbedtls +curl_ssl_openssl curl_ssl_rustls"
+IUSE="+adns +alt-svc brotli debug +ftp gnutls gopher +hsts +http2 http3 idn +imap kerberos ldap mbedtls +openssl +pop3"
+IUSE+=" +psl +progress-meter quic rtmp rustls samba +smtp ssh ssl sslv3 static-libs test telnet +tftp websockets zstd"
+# These select the default tls implementation / which quic impl to use
+IUSE+=" curl_quic_openssl curl_quic_ngtcp2 curl_ssl_gnutls curl_ssl_mbedtls +curl_ssl_openssl curl_ssl_rustls"
 RESTRICT="!test? ( test )"
 
-# Only one default ssl provider can be enabled
-# The default ssl provider needs its USE satisfied
-# nghttp3 = https://bugs.gentoo.org/912029
+# Only one default ssl / quic provider can be enabled
+# The default provider needs its USE satisfied
 REQUIRED_USE="
+	quic? (
+		^^ (
+			curl_quic_openssl
+			curl_quic_ngtcp2
+		)
+		http3
+	)
 	ssl? (
 		^^ (
 			curl_ssl_gnutls
@@ -44,13 +50,13 @@ REQUIRED_USE="
 			curl_ssl_rustls
 		)
 	)
+	curl_quic_openssl? ( openssl )
+	curl_quic_ngtcp2? ( gnutls )
 	curl_ssl_gnutls? ( gnutls )
 	curl_ssl_mbedtls? ( mbedtls )
 	curl_ssl_openssl? ( openssl )
 	curl_ssl_rustls? ( rustls )
-	nghttp3? (
-		!openssl
-		alt-svc )
+	http3? ( alt-svc quic )
 "
 
 # cURL's docs and CI/CD are great resources for confirming supported versions
@@ -60,20 +66,21 @@ REQUIRED_USE="
 # - https://github.com/curl/curl/blob/master/.github/workflows/quiche-linux.yml (CI/CD for TCP/2)
 # However 'supported' vs 'works' are two entirely different things; be sane but
 # don't be afraid to require a later version.
-
+# ngtcp2 = https://bugs.gentoo.org/912029 - can only build with one tls backend at a time.
 RDEPEND="
 	>=sys-libs/zlib-1.1.4[${MULTILIB_USEDEP}]
 	adns? ( >=net-dns/c-ares-1.16.0:=[${MULTILIB_USEDEP}] )
 	brotli? ( app-arch/brotli:=[${MULTILIB_USEDEP}] )
 	http2? ( >=net-libs/nghttp2-1.12.0:=[${MULTILIB_USEDEP}] )
+	http3? ( >=net-libs/nghttp3-1.1.0[${MULTILIB_USEDEP}] )
 	idn? ( net-dns/libidn2:=[static-libs?,${MULTILIB_USEDEP}] )
 	kerberos? ( >=virtual/krb5-0-r1[${MULTILIB_USEDEP}] )
 	ldap? ( >=net-nds/openldap-2.0.0:=[static-libs?,${MULTILIB_USEDEP}] )
-	nghttp3? (
-		>=net-libs/nghttp3-1.1.0[${MULTILIB_USEDEP}]
-		>=net-libs/ngtcp2-1.2.0[gnutls,ssl,-openssl,${MULTILIB_USEDEP}]
-	)
 	psl? ( net-libs/libpsl[${MULTILIB_USEDEP}] )
+	quic? (
+		curl_quic_openssl? ( >=dev-libs/openssl-3.3.0:=[quic,${MULTILIB_USEDEP}] )
+		curl_quic_ngtcp2? ( >=net-libs/ngtcp2-1.2.0[gnutls,ssl,-openssl,${MULTILIB_USEDEP}] )
+	)
 	rtmp? ( media-video/rtmpdump[${MULTILIB_USEDEP}] )
 	ssh? ( >=net-libs/libssh2-1.0.0[${MULTILIB_USEDEP}] )
 	ssl? (
@@ -104,7 +111,7 @@ BDEPEND="
 	test? (
 		sys-apps/diffutils
 		http2? ( >=net-libs/nghttp2-1.15.0:=[utils,${MULTILIB_USEDEP}] )
-		nghttp3? ( net-libs/nghttp2:=[utils,${MULTILIB_USEDEP}] )
+		http3? ( net-libs/nghttp2:=[utils,${MULTILIB_USEDEP}] )
 	)
 	verify-sig? ( sec-keys/openpgp-keys-danielstenberg )
 "
@@ -262,8 +269,9 @@ multilib_src_configure() {
 		--without-libgsasl
 		$(use_with psl libpsl)
 		--without-msh3
-		$(use_with nghttp3)
-		$(use_with nghttp3 ngtcp2)
+		$(use_with http3 nghttp3)
+		$(use_with curl_quic_ngtcp2 ngtcp2)
+		$(use_with curl_quic_openssl openssl-quic)
 		--without-quiche
 		$(use_with rtmp librtmp)
 		--without-schannel
@@ -285,7 +293,7 @@ multilib_src_configure() {
 		)
 	fi
 
-	if use test && multilib_is_native_abi && ( use http2 || use nghttp3 ); then
+	if use test && multilib_is_native_abi && ( use http2 || use http3 ); then
 		myconf+=(
 			--with-test-nghttpx="${BROOT}/usr/bin/nghttpx"
 		)
@@ -305,30 +313,6 @@ multilib_src_configure() {
 		sed -i -e '/SUBDIRS/s:scripts::' Makefile || die
 	fi
 
-	# Fix up the pkg-config file to be more robust.
-	# https://github.com/curl/curl/issues/864
-	local priv=() libs=()
-	# We always enable zlib.
-	libs+=( "-lz" )
-	priv+=( "zlib" )
-	if use http2; then
-		libs+=( "-lnghttp2" )
-		priv+=( "libnghttp2" )
-	fi
-	if use nghttp3; then
-		libs+=( "-lnghttp3" "-lngtcp2" )
-		priv+=( "libnghttp3" "libngtcp2" )
-	fi
-	if use ssl && use curl_ssl_openssl; then
-		libs+=( "-lssl" "-lcrypto" )
-		priv+=( "openssl" )
-	fi
-	grep -q Requires.private libcurl.pc && die "need to update ebuild"
-	libs=$(printf '|%s' "${libs[@]}")
-	sed -i -r \
-		-e "/^Libs.private/s:(${libs#|})( |$)::g" \
-		libcurl.pc || die
-	echo "Requires.private: ${priv[*]}" >> libcurl.pc || die
 }
 
 multilib_src_compile() {
diff --git a/net-misc/curl/metadata.xml b/net-misc/curl/metadata.xml
index d79a1ed4affd..884608c7c1f5 100644
--- a/net-misc/curl/metadata.xml
+++ b/net-misc/curl/metadata.xml
@@ -17,11 +17,13 @@
 		<flag name="hsts">Enable HTTP Strict Transport Security</flag>
 		<flag name="imap">Enable Internet Message Access Protocol support</flag>
 		<flag name="mbedtls">Enable mbedtls ssl backend</flag>
-		<flag name="nghttp3">Enable HTTP/3.0 support using <pkg>net-libs/nghttp3</pkg> and <pkg>net-libs/ngtcp2</pkg></flag>
+		<flag name="nghttp3">Enable HTTP/3 support using <pkg>net-libs/nghttp3</pkg></flag>
+		<flag name="http3">Enable HTTP/3 support</flag>
 		<flag name="openssl">Enable openssl ssl backend</flag>
 		<flag name="pop3">Enable Post Office Protocol 3 support</flag>
 		<flag name="progress-meter">Enable the progress meter</flag>
 		<flag name="psl">Enable Public Suffix List (PSL) support. See https://daniel.haxx.se/blog/2024/01/10/psl-in-curl/.</flag>
+		<flag name="quic">Enable support for QUIC (RFC 9000); a UDP-based protocol intended to replace TCP</flag>
 		<flag name="rtmp">Enable RTMP Streaming Media support</flag>
 		<flag name="rustls">Enable Rustls ssl backend</flag>
 		<flag name="smtp">Enable Simple Mail Transfer Protocol support</flag>
-- 
2.45.2

Re: [gentoo-dev] [PATCH 2/3] profiles/base: make.defaults: add CURL_QUIC

[Mike Gilbert]

On Fri, Jun 21, 2024 at 9:15 AM <kangie@gentoo.org> wrote:

I would combine this with the previous commit which adds the values to
curl_quic.desc, and has a much better commit message.

For the summary, something like "profiles: add CURL_QUIC USE_EXPAND"
should do the trick.

Re: [gentoo-dev] [PATCH 1/3] profiles/desc: add curl_quic

[Jaco Kroon]

Hi,

On 2024/06/21 15:15, kangie@gentoo.org wrote:

> From: Matt Jolly <kangie@gentoo.org>
>
> The CURL_QUIC USE_EXPAND enables us to sanely manage QUIC (RFC 9000)
> backends as they are added to cURL in the future: currently there are
> two supported implementations, OpenSSL and ngtcp2,  however it's likely
> that other popular TLS libraries will expose QUIC APIs over time,
> and that these will be eventually be supported by cURL (see CURL_SSL
> for examples of TLS libraries that we support) - we may as well
> get ahead of the curve here.
>
> There are already a number of other small players (i.e. OpenSSL Forks)
> exposing QUIC support for quite a while, however these have not been
> available in ::gentoo and we've only needed the one USE to enable
> for HTTP/3 and QUIC to this point.
>
> Signed-off-by: Matt Jolly <kangie@gentoo.org>
> ---
>   profiles/desc/curl_quic.desc | 7 +++++++
>   1 file changed, 7 insertions(+)
>   create mode 100644 profiles/desc/curl_quic.desc
>
> diff --git a/profiles/desc/curl_quic.desc b/profiles/desc/curl_quic.desc
> new file mode 100644
> index 000000000000..372bb9ce8f83
> --- /dev/null
> +++ b/profiles/desc/curl_quic.desc
> @@ -0,0 +1,7 @@
> +# Copyright 1999-2024 Gentoo Authors
> +# Distributed under the terms of the GNU General Public License v2
> +
> +# This file contains descriptions of CURL_QUIC USE_EXPAND flags for net-misc/curl
> +
> +openssl - Use OpenSSL
> +ngtcp2 - Use ngtcp2

May I suggest simply calling this USE_EXPAND QUIC_IMPL so that other 
packages can potentially re-use as well?

looking through ::gentoo at least net-dns/dnsdist and net-dns/knot also 
has a quic support, using ngtcp2 and/or net-libs/quiche.

With openssl 3.2 hopefully approaching stable at some point I suspect 
the number of projects that will be adding quic support via one or 
another channel (possibly with alternative implementations) will only 
increase, thus pinning the USE_EXPAND on a single package seems 
potentially short-sighted.

Kind regards,
Jaco

Re: [gentoo-dev] [PATCH 1/3] profiles/desc: add curl_quic

[Matt Jolly]

Hi Jaco,

> May I suggest simply calling this USE_EXPAND QUIC_IMPL so that other 
> packages can potentially re-use as well?
> 
> looking through ::gentoo at least net-dns/dnsdist and net-dns/knot also 
> has a quic support, using ngtcp2 and/or net-libs/quiche.
>
> With openssl 3.2 hopefully approaching stable at some point I suspect 
> the number of projects that will be adding quic support via one or 
> another channel (possibly with alternative implementations) will only 
> increase, thus pinning the USE_EXPAND on a single package seems 
> potentially short-sighted.

My knee-jerk response was to claim that cURL is unique in the
number of backends supported (and way that it tends to support
configuring for multiple implementations at once), but then I took the
time to look at the various TLS USE flags for things like web servers 
and I've warmed up to the suggestion.

I can certainly see some benefit to having a generic USE_EXPAND that
covers QUIC implementations (and maybe one for TLS impls?). We could
probably replace CURL_SSL and CURL_QUIC with the generics, though I'd
still need to retain the existing global USE that this would deprecate
(at least as local USE in net-misc/curl) as the current ebuild logic
relies on both USE and USE_EXPAND for TLS implementation selection.

I'm interested in hearing some other opinions though - is there some 
reason this hasn't already been done?

The alternative (doing nothing) still seems appealing given that OpenSSL
seems likely to remain the 'default' implementation as QUIC adoption
rises, and the existing USE (and profile) settings have proven
sufficient (and not too confusing) so far.

Ideally, if the generic USE_EXPAND option is pursued I imagine that
we would want to hit all of the ebuilds (etc) at once and ensure that
an appropriate news item concerning the migration has been distributed.
There's nothing stopping us from implementing this solution as a
separate change that doesn't block the cURL updates while we decide
whether one (or more) generic USE_EXPAND variables make sense.

Thanks,

Matt