[gentoo-user] net-libs/gnutls-3.7.2 fails to verify some certificates (duplicate server certificate?)

[gentoo-user] net-libs/gnutls-3.7.2 fails to verify some certificates (duplicate server certificate?)

[Branko Grubić]

Hi,

I have few applications which use webkit-gtk and gnutls behind as far
as I know, recently I noticed that RSS feeds for some distrowatch.com
subscriptions I had started to fail, initially I did ignore them I
thought something is wrong on the server side and it was not critical.

But since it wasn't fixed I started to investigate a little bit more.

So, in the end it seems to be related to gnutls on Gentoo (I'm running
~amd64)

net-libs/gnutls-3.7.2 abi_x86_64 cxx idn nls openssl seccomp tls-
heartbeat tools

Important note, websites using Let's Encrypt certificates work fine,
except this one (only example known to me). Based on the output of
`gnutls-cli` it seems that server certificate is served twice compared
to other working ones (I could be wrong).

Example output:
$ gnutls-cli  distrowatch.com:443  
Processed 130 CA certificate(s).
Resolving 'distrowatch.com:443'...
Connecting to '82.103.129.71:443'...
- Certificate type: X.509
- Got a certificate list of 4 certificates.
- Certificate[0] info:
 - subject `CN=distrowatch.com', issuer `CN=R3,O=Let's Encrypt,C=US',
serial 0x0408fd5a5ae26286bed92e97da0c830f623c, RSA key 2048 bits,
signed using RSA-SHA256, activated `2021-09-15 03:49:15 UTC', expires
`2021-12-14 03:49:14 UTC', pin-
sha256="QoW1tiDGE8S3FLukw86yRL8IfevROPxnx0qwVuu/rUI="
	Public Key ID:
		sha1:fcd2b25ac6ffd73fce3ef65211defd25331dc151
		sha256:4285b5b620c613c4b714bba4c3ceb244bf087debd138fc6
7c74ab056ebbfad42
	Public Key PIN:
		pin-
sha256:QoW1tiDGE8S3FLukw86yRL8IfevROPxnx0qwVuu/rUI=

- Certificate[1] info:
 - subject `CN=distrowatch.com', issuer `CN=R3,O=Let's Encrypt,C=US',
serial 0x0408fd5a5ae26286bed92e97da0c830f623c, RSA key 2048 bits,
signed using RSA-SHA256, activated `2021-09-15 03:49:15 UTC', expires
`2021-12-14 03:49:14 UTC', pin-
sha256="QoW1tiDGE8S3FLukw86yRL8IfevROPxnx0qwVuu/rUI="
- Certificate[2] info:
 - subject `CN=R3,O=Let's Encrypt,C=US', issuer `CN=ISRG Root
X1,O=Internet Security Research Group,C=US', serial
0x00912b084acf0c18a753f6d62e25a75f5a, RSA key 2048 bits, signed using
RSA-SHA256, activated `2020-09-04 00:00:00 UTC', expires `2025-09-15
16:00:00 UTC', pin-
sha256="jQJTbIh0grw0/1TkHSumWb+Fs0Ggogr621gT3PvPKG0="
- Certificate[3] info:
 - subject `CN=ISRG Root X1,O=Internet Security Research Group,C=US',
issuer `CN=DST Root CA X3,O=Digital Signature Trust Co.', serial
0x4001772137d4e942b8ee76aa3c640ab7, RSA key 4096 bits, signed using
RSA-SHA256, activated `2021-01-20 19:14:03 UTC', expires `2024-09-30
18:14:03 UTC', pin-
sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="
- Status: The certificate is NOT trusted. The certificate issuer is
unknown. 
*** PKI verification of server certificate failed...
*** Fatal error: Error in the certificate.


Firefox and Chrome open website just fine, no complains. Also openssl
client doesn't complain if I read the output right.


I have tested this on Fedora 35 as well using gnutls-cli, it comes with
same gnutls release, and has no issues connecting to problematic host.
So I suspect it's something to do with my system, Gentoo ebuild, or
combination of libraries used for gnutls on my Gentoo system.

I have found an interesting (similar) bug[1] which was fixed in the
current release (fix is included in 3.7.2 based on the NEWS/Release
notes) where gnutls would fail if Root CA certificate is present twice
in the chain.

Can anyone confirm it happening on their system as well, I was not sure
should I open a Gentoo bug.

Regards,
Branko


[1] https://gitlab.com/gnutls/gnutls/-/issues/1131

Re: [gentoo-user] net-libs/gnutls-3.7.2 fails to verify some certificates (duplicate server certificate?)

[Jack]

On 2021.11.23 14:43, Branko Grubić wrote:
> Hi,
> 
> I have few applications which use webkit-gtk and gnutls behind as far
> as I know, recently I noticed that RSS feeds for some distrowatch.com
> subscriptions I had started to fail, initially I did ignore them I
> thought something is wrong on the server side and it was not critical.
> 
> But since it wasn't fixed I started to investigate a little bit more.
> 
> So, in the end it seems to be related to gnutls on Gentoo (I'm running
> ~amd64)
> 
> net-libs/gnutls-3.7.2 abi_x86_64 cxx idn nls openssl seccomp tls-
> heartbeat tools
> 
> Important note, websites using Let's Encrypt certificates work fine,
> except this one (only example known to me). Based on the output of
> `gnutls-cli` it seems that server certificate is served twice compared
> to other working ones (I could be wrong).
> 
> Example output:
> $ gnutls-cli  distrowatch.com:443
> Processed 130 CA certificate(s).
> Resolving 'distrowatch.com:443'...
> Connecting to '82.103.129.71:443'...
> - Certificate type: X.509
> - Got a certificate list of 4 certificates.
> - Certificate[0] info:
>  - subject `CN=distrowatch.com', issuer `CN=R3,O=Let's Encrypt,C=US',
> serial 0x0408fd5a5ae26286bed92e97da0c830f623c, RSA key 2048 bits,
> signed using RSA-SHA256, activated `2021-09-15 03:49:15 UTC', expires
> `2021-12-14 03:49:14 UTC', pin-
> sha256="QoW1tiDGE8S3FLukw86yRL8IfevROPxnx0qwVuu/rUI="
> 	Public Key ID:
> 		sha1:fcd2b25ac6ffd73fce3ef65211defd25331dc151
> 		sha256:4285b5b620c613c4b714bba4c3ceb244bf087debd138fc6
> 7c74ab056ebbfad42
> 	Public Key PIN:
> 		pin-
> sha256:QoW1tiDGE8S3FLukw86yRL8IfevROPxnx0qwVuu/rUI=
> 
> - Certificate[1] info:
>  - subject `CN=distrowatch.com', issuer `CN=R3,O=Let's Encrypt,C=US',
> serial 0x0408fd5a5ae26286bed92e97da0c830f623c, RSA key 2048 bits,
> signed using RSA-SHA256, activated `2021-09-15 03:49:15 UTC', expires
> `2021-12-14 03:49:14 UTC', pin-
> sha256="QoW1tiDGE8S3FLukw86yRL8IfevROPxnx0qwVuu/rUI="
> - Certificate[2] info:
>  - subject `CN=R3,O=Let's Encrypt,C=US', issuer `CN=ISRG Root
> X1,O=Internet Security Research Group,C=US', serial
> 0x00912b084acf0c18a753f6d62e25a75f5a, RSA key 2048 bits, signed using
> RSA-SHA256, activated `2020-09-04 00:00:00 UTC', expires `2025-09-15
> 16:00:00 UTC', pin-
> sha256="jQJTbIh0grw0/1TkHSumWb+Fs0Ggogr621gT3PvPKG0="
> - Certificate[3] info:
>  - subject `CN=ISRG Root X1,O=Internet Security Research Group,C=US',
> issuer `CN=DST Root CA X3,O=Digital Signature Trust Co.', serial
> 0x4001772137d4e942b8ee76aa3c640ab7, RSA key 4096 bits, signed using
> RSA-SHA256, activated `2021-01-20 19:14:03 UTC', expires `2024-09-30
> 18:14:03 UTC', pin-
> sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="
> - Status: The certificate is NOT trusted. The certificate issuer is
> unknown.
> *** PKI verification of server certificate failed...
> *** Fatal error: Error in the certificate.
> 
> 
> Firefox and Chrome open website just fine, no complains. Also openssl
> client doesn't complain if I read the output right.
> 
> 
> I have tested this on Fedora 35 as well using gnutls-cli, it comes  
> with
> same gnutls release, and has no issues connecting to problematic host.
> So I suspect it's something to do with my system, Gentoo ebuild, or
> combination of libraries used for gnutls on my Gentoo system.
> 
> I have found an interesting (similar) bug[1] which was fixed in the
> current release (fix is included in 3.7.2 based on the NEWS/Release
> notes) where gnutls would fail if Root CA certificate is present twice
> in the chain.
> 
> Can anyone confirm it happening on their system as well, I was not  
> sure
> should I open a Gentoo bug.
> 
> Regards,
> Branko
> 
> 
> [1] https://gitlab.com/gnutls/gnutls/-/issues/1131
Works fine for me, after recompiling gnutls to add nls and tools.  I  
haven't compared the outputs completely, but first pass I don't see any  
differences, although I could easily have missed one.

$gnutls-cli  distrowatch.com:443
Processed 128 CA certificate(s).
Resolving 'distrowatch.com:443'...
Connecting to '82.103.129.71:443'...
- Certificate type: X.509
- Got a certificate list of 4 certificates.
- Certificate[0] info:
  - subject `CN=distrowatch.com', issuer `CN=R3,O=Let's Encrypt,C=US',  
serial 0x0408fd5a5ae26286bed92e97da0c830f623c, RSA key 2048 bits,  
signed using RSA-SHA256, activated `2021-09-15 03:49:15 UTC', expires  
`2021-12-14 03:49:14 UTC',  
pin-sha256="QoW1tiDGE8S3FLukw86yRL8IfevROPxnx0qwVuu/rUI="
         Public Key ID:
                 sha1:fcd2b25ac6ffd73fce3ef65211defd25331dc151
                  
sha256:4285b5b620c613c4b714bba4c3ceb244bf087debd138fc67c74ab056ebbfad42
         Public Key PIN:
                 pin-sha256:QoW1tiDGE8S3FLukw86yRL8IfevROPxnx0qwVuu/rUI=

- Certificate[1] info:
  - subject `CN=distrowatch.com', issuer `CN=R3,O=Let's Encrypt,C=US',  
serial 0x0408fd5a5ae26286bed92e97da0c830f623c, RSA key 2048 bits,  
signed using RSA-SHA256, activated `2021-09-15 03:49:15 UTC', expires  
`2021-12-14 03:49:14 UTC',  
pin-sha256="QoW1tiDGE8S3FLukw86yRL8IfevROPxnx0qwVuu/rUI="
- Certificate[2] info:
  - subject `CN=R3,O=Let's Encrypt,C=US', issuer `CN=ISRG Root  
X1,O=Internet Security Research Group,C=US', serial  
0x00912b084acf0c18a753f6d62e25a75f5a, RSA key 2048 bits, signed using  
RSA-SHA256, activated `2020-09-04 00:00:00 UTC', expires `2025-09-15  
16:00:00 UTC', pin-sha256="jQJTbIh0grw0/1TkHSumWb+Fs0Ggogr621gT3PvPKG0="
- Certificate[3] info:
  - subject `CN=ISRG Root X1,O=Internet Security Research Group,C=US',  
issuer `CN=DST Root CA X3,O=Digital Signature Trust Co.', serial  
0x4001772137d4e942b8ee76aa3c640ab7, RSA key 4096 bits, signed using  
RSA-SHA256, activated `2021-01-20 19:14:03 UTC', expires `2024-09-30  
18:14:03 UTC', pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="
- Status: The certificate is trusted.
- Description:  
(TLS1.3-X.509)-(ECDHE-SECP256R1)-(RSA-PSS-RSAE-SHA256)-(AES-256-GCM)
- Session ID:  
83:62:97:D1:C0:77:19:76:F8:2F:41:7E:DD:CD:C5:A6:35:2A:5D:4C:39:B4:F5:12:CA:09:0F:07:26:BA:83:5F
- Options:
- Handshake was completed

- Simple Client Mode:

- Peer has closed the GnuTLS connection

Re: [gentoo-user] net-libs/gnutls-3.7.2 fails to verify some certificates (duplicate server certificate?)

[Jack]

On 2021.11.23 14:43, Branko Grubić wrote:
> Hi,
> 
> I have few applications which use webkit-gtk and gnutls behind as far
> as I know, recently I noticed that RSS feeds for some distrowatch.com
> subscriptions I had started to fail, initially I did ignore them I
> thought something is wrong on the server side and it was not critical.
> 
> But since it wasn't fixed I started to investigate a little bit more.
> 
> So, in the end it seems to be related to gnutls on Gentoo (I'm running
> ~amd64)
> 
> net-libs/gnutls-3.7.2 abi_x86_64 cxx idn nls openssl seccomp tls-
> heartbeat tools
> 
> Important note, websites using Let's Encrypt certificates work fine,
> except this one (only example known to me). Based on the output of
> `gnutls-cli` it seems that server certificate is served twice compared
> to other working ones (I could be wrong).
> 
> Example output:
> $ gnutls-cli  distrowatch.com:443
> Processed 130 CA certificate(s).
> Resolving 'distrowatch.com:443'...
> Connecting to '82.103.129.71:443'...
> - Certificate type: X.509
> - Got a certificate list of 4 certificates.
> - Certificate[0] info:
>  - subject `CN=distrowatch.com', issuer `CN=R3,O=Let's Encrypt,C=US',
> serial 0x0408fd5a5ae26286bed92e97da0c830f623c, RSA key 2048 bits,
> signed using RSA-SHA256, activated `2021-09-15 03:49:15 UTC', expires
> `2021-12-14 03:49:14 UTC', pin-
> sha256="QoW1tiDGE8S3FLukw86yRL8IfevROPxnx0qwVuu/rUI="
> 	Public Key ID:
> 		sha1:fcd2b25ac6ffd73fce3ef65211defd25331dc151
> 		sha256:4285b5b620c613c4b714bba4c3ceb244bf087debd138fc6
> 7c74ab056ebbfad42
> 	Public Key PIN:
> 		pin-
> sha256:QoW1tiDGE8S3FLukw86yRL8IfevROPxnx0qwVuu/rUI=
> 
> - Certificate[1] info:
>  - subject `CN=distrowatch.com', issuer `CN=R3,O=Let's Encrypt,C=US',
> serial 0x0408fd5a5ae26286bed92e97da0c830f623c, RSA key 2048 bits,
> signed using RSA-SHA256, activated `2021-09-15 03:49:15 UTC', expires
> `2021-12-14 03:49:14 UTC', pin-
> sha256="QoW1tiDGE8S3FLukw86yRL8IfevROPxnx0qwVuu/rUI="
> - Certificate[2] info:
>  - subject `CN=R3,O=Let's Encrypt,C=US', issuer `CN=ISRG Root
> X1,O=Internet Security Research Group,C=US', serial
> 0x00912b084acf0c18a753f6d62e25a75f5a, RSA key 2048 bits, signed using
> RSA-SHA256, activated `2020-09-04 00:00:00 UTC', expires `2025-09-15
> 16:00:00 UTC', pin-
> sha256="jQJTbIh0grw0/1TkHSumWb+Fs0Ggogr621gT3PvPKG0="
> - Certificate[3] info:
>  - subject `CN=ISRG Root X1,O=Internet Security Research Group,C=US',
> issuer `CN=DST Root CA X3,O=Digital Signature Trust Co.', serial
> 0x4001772137d4e942b8ee76aa3c640ab7, RSA key 4096 bits, signed using
> RSA-SHA256, activated `2021-01-20 19:14:03 UTC', expires `2024-09-30
> 18:14:03 UTC', pin-
> sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="
> - Status: The certificate is NOT trusted. The certificate issuer is
> unknown.
> *** PKI verification of server certificate failed...
> *** Fatal error: Error in the certificate.
> 
> 
> Firefox and Chrome open website just fine, no complains. Also openssl
> client doesn't complain if I read the output right.
> 
> 
> I have tested this on Fedora 35 as well using gnutls-cli, it comes  
> with
> same gnutls release, and has no issues connecting to problematic host.
> So I suspect it's something to do with my system, Gentoo ebuild, or
> combination of libraries used for gnutls on my Gentoo system.
> 
> I have found an interesting (similar) bug[1] which was fixed in the
> current release (fix is included in 3.7.2 based on the NEWS/Release
> notes) where gnutls would fail if Root CA certificate is present twice
> in the chain.
> 
> Can anyone confirm it happening on their system as well, I was not  
> sure
> should I open a Gentoo bug.
> 
> Regards,
> Branko
> 
> 
> [1] https://gitlab.com/gnutls/gnutls/-/issues/1131
OK, diff of your output to mine:
1,2c1,2
< $ gnutls-cli  distrowatch.com:443
< Processed 130 CA certificate(s).
---
> $gnutls-cli  distrowatch.com:443
> Processed 128 CA certificate(s).
21,23c21,29
< - Status: The certificate is NOT trusted. The certificate issuer is  
unknown.
< *** PKI verification of server certificate failed...
< *** Fatal error: Error in the certificate.
---
> - Status: The certificate is trusted.
> - Description:  
> (TLS1.3-X.509)-(ECDHE-SECP256R1)-(RSA-PSS-RSAE-SHA256)-(AES-256-GCM)
> - Session ID:  
> 83:62:97:D1:C0:77:19:76:F8:2F:41:7E:DD:CD:C5:A6:35:2A:5D:4C:39:B4:F5:12:CA:09:0F:07:26:BA:83:5F
> - Options:
> - Handshake was completed
> 
> - Simple Client Mode:
> 
> - Peer has closed the GnuTLS connection

So I have two fewer CA Certs than you do, but somehow I know the issuer  
and you do not.
I have app-misc/ca-certificates 20210119.3.66, but there are two ~  
versions.  I wonder if something got dropped (whether intentionally or  
not.)

Re: [gentoo-user] net-libs/gnutls-3.7.2 fails to verify some certificates (duplicate server certificate?)

[Branko Grubić]

On Tue, 2021-11-23 at 17:26 -0500, Jack wrote:
> On 2021.11.23 14:43, Branko Grubić wrote:
> > Hi,
> > [1] https://gitlab.com/gnutls/gnutls/-/issues/1131
> OK, diff of your output to mine:
> 1,2c1,2
> < $ gnutls-cli  distrowatch.com:443
> < Processed 130 CA certificate(s).
> ---
> > $gnutls-cli  distrowatch.com:443
> > Processed 128 CA certificate(s).
> 21,23c21,29
> < - Status: The certificate is NOT trusted. The certificate issuer
> is  
> unknown.
> < *** PKI verification of server certificate failed...
> < *** Fatal error: Error in the certificate.
> ---
> > - Status: The certificate is trusted.
> > - Description:  
> > (TLS1.3-X.509)-(ECDHE-SECP256R1)-(RSA-PSS-RSAE-SHA256)-(AES-256-
> > GCM)
> > - Session ID:  
> > 83:62:97:D1:C0:77:19:76:F8:2F:41:7E:DD:CD:C5:A6:35:2A:5D:4C:39:B4:F
> > 5:12:CA:09:0F:07:26:BA:83:5F
> > - Options:
> > - Handshake was completed
> > 
> > - Simple Client Mode:
> > 
> > - Peer has closed the GnuTLS connection
> 
> So I have two fewer CA Certs than you do, but somehow I know the
> issuer  
> and you do not.
> I have app-misc/ca-certificates 20210119.3.66, but there are two ~  
> versions.  I wonder if something got dropped (whether intentionally
> or  
> not.)
> 

Hi,

Thanks for testing. Not happy that it's only me :D

I don't think my issue is related to ca-certificates, btw. I'm "fully"
on ~amd64 (unstable/testing).
Here is what's installed here:

app-misc/ca-certificates-20211016.3.72 -cacert

Also, as I said Let's Encrypt certificates from other sites work fine
with gnutls-cli (and other clients using gnutls), which have similar
certificate chain's (same, except the server certificate, same
intermediate and same "Root" certificate), so I don't think it's
related to a "missing trust".

Could be something specific to my system or ~arch (unstable), that some
library gnutls uses is causing issues (but I cannot remember now when
it started happening to pinpoint after which update it stopped
working).

Regards,
Branko

Re: [gentoo-user] net-libs/gnutls-3.7.2 fails to verify some certificates (duplicate server certificate?)

[Jack]

OK, here's something.

I changed my stable version of ca-certificates from -cacert to cacert,  
and now I get the same failure you do.  So - it's due to either  
something in nss-cacert-class1-class3-r2.patch which only gets applied  
if that USE flag is set, or to something else only done when that USE  
flag is set.

I don't understand it, but it's a place to start - and note the note in  
the ebuild:

# When triaging user reports, refer to our wiki for tips:
# https://wiki.gentoo.org/wiki/Certificates#Debugging_certificate_issues

Re: [gentoo-user] net-libs/gnutls-3.7.2 fails to verify some certificates (duplicate server certificate?)

[Branko Grubić]

On Tue, 2021-11-23 at 18:14 -0500, Jack wrote:
> OK, here's something.
> 
> I changed my stable version of ca-certificates from -cacert to
> cacert,  
> and now I get the same failure you do.  So - it's due to either  
> something in nss-cacert-class1-class3-r2.patch which only gets
> applied  
> if that USE flag is set, or to something else only done when that
> USE  
> flag is set.
> 
> I don't understand it, but it's a place to start - and note the note
> in  
> the ebuild:
> 
> # When triaging user reports, refer to our wiki for tips:
> #
> https://wiki.gentoo.org/wiki/Certificates#Debugging_certificate_issues
> 

Thanks Jack.

Interesting that it breaks for you as well now. Btw I haven't used
USE="cacert" (could be that I copied it wrong here), it's a additional
Certificate Authority which is not included by default in Mozilla
database as far as I understand. I have tried to change USE="cacert"
and rebuild app-misc/ca-certificates but no change when tested, which
to me is expected, but who knows what could be an issue. 

I'll try to dig deeper into this. Initially I was hoping that someone
more familiar with the topic could jump in and suggest what to do next.

I did look at the wiki, but wiki uses openssl tools for debugging, and
I have no issues with openssl client connecting to this server :/ (so I
don't think it's useful in this case)

Regards,
Branko

Re: [gentoo-user] net-libs/gnutls-3.7.2 fails to verify some certificates (duplicate server certificate?)

[Branko Grubić]

On Tue, 2021-11-23 at 18:14 -0500, Jack wrote:
> OK, here's something.
> 
> I changed my stable version of ca-certificates from -cacert to
> cacert,  
> and now I get the same failure you do.  So - it's due to either  
> something in nss-cacert-class1-class3-r2.patch which only gets
> applied  
> if that USE flag is set, or to something else only done when that
> USE  
> flag is set.
> 
> I don't understand it, but it's a place to start - and note the note
> in  
> the ebuild:
> 
> # When triaging user reports, refer to our wiki for tips:
> #
> https://wiki.gentoo.org/wiki/Certificates#Debugging_certificate_issues
> 


Another update, I have masked ~arch ca-certificates:
>app-misc/ca-certificates-20210119.3.66

Downgraded to stable one, and now certificate verification is
successful with gnutls-cli on my test example. Weird since it didn't
fail to verify similar chains with newer app-misc/ca-certificates. I
will file a bug report, but still not sure which component app-mist/ca-
certificates or net-libs/gnutls. 

Regards,
Branko

[gentoo-user] Re: net-libs/gnutls-3.7.2 fails to verify some certificates (duplicate server certificate?)

[Branko Grubić]

On Tue, 2021-11-23 at 20:43 +0100, Branko Grubić wrote:
> Hi,
> 
> I have few applications which use webkit-gtk and gnutls behind as far
> ...


Filed a bug report[1]


[1] https://bugs.gentoo.org/827718