[gentoo-dev]

[gentoo-dev]

[Tobias Klausmann]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=utf-8, Size: 1899 bytes --]

Hi, 

glibc 2.9 uses a different way to implement getaddrinfo() which
triggers a race condition in most (if not all) Netfilter
firewalls that use connection tracking. glibc does nothing wrong
per se, it just triggers the condition. (technical details here:
http://marc.info/?l=linux-netdev&m=123304473331445)

Since glibc 2.9 fires off two udp packets (a query for the A
record and one for the AAAA record), a race condition is
triggered in Netfilter (see URL). This has been acknowledged by
several people and I can reproduce it (relatively) reliably in
our LAN with all Gentoo boxes that have 2.9.

Why am I bringing this up here? Well, I figure that eventually,
2.9 (or some other version with equivalent code) will become
stable and we'll get lots of bug reports from people who run into
this. Since they can not simply backdate to 2.7 for various
reasons *and* they might be unable to fix a packetfilter (because
it might not be their own), this might become very ugly.

The Kernel/Netfilter devs (probably) are aware now of the issue
since I mailed them - but it's not all that easy to fix. On top
of that, even if it was fixed in (say) 2.6.28.3 and 2.6.29, this
does not mean that it's deployed out there and it might be very
hard for our users to get some firewall guy to update their
kernel when this is perceived as glibc's or our fault (plus the
widespread "ricer" cliché about Gentoo users; I've gotten an
idiotic reply to that effect already).

I don't have any experience with glibc upstream but pestering
them about this out of the blue might only cause a flame war
between kernel and glibc folks. Thus, I'm asking you, my fellow
devs (and the glibc and kernel teams specifically), what you
think is the best idea/course of action.

Regards,
Tobias
(Blackb|rd)
-- 
printk("Cool stuff's happening!\n")
        linux-2.4.3/fs/jffs/intrep.c

Re: [gentoo-dev]

[Peter Alfredsen]

[Mike: This looks like your field of expertise]
On Tue, 27 Jan 2009 16:47:50 +0100
Tobias Klausmann <klausman@gentoo.org> wrote:

> Hi, 
> 
> glibc 2.9 uses a different way to implement getaddrinfo() which
> triggers a race condition in most (if not all) Netfilter
> firewalls that use connection tracking. glibc does nothing wrong
> per se, it just triggers the condition. (technical details here:
> http://marc.info/?l=linux-netdev&m=123304473331445)
[...]
> I don't have any experience with glibc upstream but pestering
> them about this out of the blue might only cause a flame war
> between kernel and glibc folks. Thus, I'm asking you, my fellow
> devs (and the glibc and kernel teams specifically), what you
> think is the best idea/course of action.

The connection with IPv6 leads me to believe that this is
http://bugs.gentoo.org/250468
http://sourceware.org/bugzilla/show_bug.cgi?id=7060

Mike has added a patch to Gentoo's patchset but hasn't bumped the
revision yet. It does look spectacularly hacky, though :-)

Anyway, if this is your problem, it looks like upstream is already
working on it and that we just need to *prod* Mike a bit to get a fix
into the tarball.

/PA

Re: [gentoo-dev] Race condition in Netfilter triggered by glibc 2.9

[Tobias Klausmann]

Hi! 

(fixed the subject)

On Tue, 27 Jan 2009, Peter Alfredsen wrote:
> [Mike: This looks like your field of expertise]
> On Tue, 27 Jan 2009 16:47:50 +0100
> Tobias Klausmann <klausman@gentoo.org> wrote:
> 
> > glibc 2.9 uses a different way to implement getaddrinfo() which
> > triggers a race condition in most (if not all) Netfilter
> > firewalls that use connection tracking. glibc does nothing wrong
> > per se, it just triggers the condition. (technical details here:
> > http://marc.info/?l=linux-netdev&m=123304473331445)
> [...]
> > I don't have any experience with glibc upstream but pestering
> > them about this out of the blue might only cause a flame war
> > between kernel and glibc folks. Thus, I'm asking you, my fellow
> > devs (and the glibc and kernel teams specifically), what you
> > think is the best idea/course of action.
> 
> The connection with IPv6 leads me to believe that this is
> http://bugs.gentoo.org/250468
> http://sourceware.org/bugzilla/show_bug.cgi?id=7060

I doubt it: sometimes the lookups work, as this race is very
timing-critical. When experimenting, I had to go below 50
microseconds between the two packets to actually trigger it plus
the forwarding machines always were multicore. Also, the content
is irrelevant. I implemented this myself sending the payloads
with sendto() and it didn't matter if I sent the A or AAAA query
first.

That said, without seeing a tcpdump from those people with the
error described in those two bugs, I can not rule it out.

On the wire between the client and the firewall, this happens:

a packet 1 is sent
b packet 2 is sent
c answer 1 is received
d answer 2 is received

Sometimes d doesn't happen because b is lost in the firewall
along the way (where the race condition happens). 

> Mike has added a patch to Gentoo's patchset but hasn't bumped the
> revision yet. It does look spectacularly hacky, though :-)
> 
> Anyway, if this is your problem, it looks like upstream is already
> working on it and that we just need to *prod* Mike a bit to get a fix
> into the tarball.

The bug is in the kernel, not glibc. The latter just triggers it
because the newer resolver has a more aggressive timing. Note
that I think that what the glibc guys did is a *good* idea. It
just happens to rub Netfilter the wrong way.

Regards,
Tobias

-- 
printk("Cool stuff's happening!\n")
        linux-2.4.3/fs/jffs/intrep.c

Re: [gentoo-dev] Race condition in Netfilter triggered by glibc 2.9

[Mike Frysinger]

On Tuesday 27 January 2009 11:59:46 Tobias Klausmann wrote:
> On Tue, 27 Jan 2009, Peter Alfredsen wrote:
> > On Tue, 27 Jan 2009 16:47:50 +0100 Tobias Klausmann wrote:
> > > glibc 2.9 uses a different way to implement getaddrinfo() which
> > > triggers a race condition in most (if not all) Netfilter
> > > firewalls that use connection tracking. glibc does nothing wrong
> > > per se, it just triggers the condition. (technical details here:
> > > http://marc.info/?l=linux-netdev&m=123304473331445)
> >
> > [...]
> >
> > > I don't have any experience with glibc upstream but pestering
> > > them about this out of the blue might only cause a flame war
> > > between kernel and glibc folks. Thus, I'm asking you, my fellow
> > > devs (and the glibc and kernel teams specifically), what you
> > > think is the best idea/course of action.
> >
> > The connection with IPv6 leads me to believe that this is
> > http://bugs.gentoo.org/250468
> > http://sourceware.org/bugzilla/show_bug.cgi?id=7060
>
> I doubt it: sometimes the lookups work, as this race is very
> timing-critical. When experimenting, I had to go below 50
> microseconds between the two packets to actually trigger it plus
> the forwarding machines always were multicore. Also, the content
> is irrelevant. I implemented this myself sending the payloads
> with sendto() and it didn't matter if I sent the A or AAAA query
> first.
>
> That said, without seeing a tcpdump from those people with the
> error described in those two bugs, I can not rule it out.

the referenced bug generally deals with broken nameservers that cant handle 
the type of requests that glibc sends out (the requests are correct according 
to the relevant standards/RFCs, but apparently many DNS servers out there 
screw up with it due to the ipv4/ipv6 combo).

the referenced thread seems to indicate even more the issue is in the 
netfilter code.

> On the wire between the client and the firewall, this happens:
>
> a packet 1 is sent
> b packet 2 is sent
> c answer 1 is received
> d answer 2 is received
>
> Sometimes d doesn't happen because b is lost in the firewall
> along the way (where the race condition happens).

does this affect actual userspace behavior ?  in other words, does this lead 
to lost lookups and errors from the resolver ?
-mike

[gentoo-dev] Re: Race condition in Netfilter triggered by glibc 2.9

[Duncan]

Mike Frysinger <vapier@gentoo.org> posted
200901282125.52845.vapier@gentoo.org, excerpted below, on  Wed, 28 Jan
2009 21:25:50 -0500:

>> On the wire between the client and the firewall, this happens:
>>
>> a packet 1 is sent
>> b packet 2 is sent
>> c answer 1 is received
>> d answer 2 is received
>>
>> Sometimes d doesn't happen because b is lost in the firewall along the
>> way (where the race condition happens).
> 
> does this affect actual userspace behavior ?  in other words, does this
> lead to lost lookups and errors from the resolver ?

Some of this is beyond my comprehension level, but I've seen interesting 
lookup behavior that is at minimum, rather nicely coincidental.

Specifically, from my machine (running a local caching bind, with 
netfilter on both the machine itself and on my OpenWRT based router), 
doing host lookups on second level domains (cox.com in my case) with MX 
entries works fine, while lookups on third level domains unlikely to have 
MX entries (www.cox.com) return the A record right away, then timeout on 
the MX entry.  AFAIK this is fairly new behavior, apparently quite 
coincident with my installation of glibc-2.9 (_p20081201_r1, currently), 
as IIRC, it formerly returned fine, without waiting for the timeout.

dig -tMX has the same behavior, while a simple dig (A record only) does 
not.

I stumbled across this while investigating after someone (running another 
distribution, no local DNS server) on the local Cox Unix newsgroup 
complained about the response time to www.cox.com.  We traced it down to 
long resolve times and checking them I noted this issue.  I initially 
chalked it up to DNS weirdness on their part and that may indeed be part 
or all of it, but reading this, it sure looks coincidentally similar and 
the timing seems right, at least here (I've no idea what his glibc 
version is or whether he's running netfilter based firewalls either on 
his machine or router, I asked, but don't have a reply yet).

I have not noted any particular delays other than with host/dig -tMX 
myself, but I suspect that may be because I'm running a local bind and it 
mitigates the issue under normal operating conditions.

As I said, it's enough above my head to have no real idea whether this is 
connected or not, but it sure seems coincidental if not.  I'm posting 
because it seems it might help answer the "Does this affect actual 
userspace behavior?" bit.

I can't help feeling a bit uncomfortable with the discussion here as it's 
too much like a normally discouraged bug discussion on the main dev 
list.  So if people want to take the discussion to a bug, post the bug 
link and I'll be happy to CC myself. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Race condition in Netfilter triggered by glibc 2.9

[Tobias Klausmann]

Hi! 

On Wed, 28 Jan 2009, Mike Frysinger wrote:
> > On the wire between the client and the firewall, this happens:
> >
> > a packet 1 is sent
> > b packet 2 is sent
> > c answer 1 is received
> > d answer 2 is received
> >
> > Sometimes d doesn't happen because b is lost in the firewall
> > along the way (where the race condition happens).
> 
> does this affect actual userspace behavior ?  in other words,
> does this lead to lost lookups and errors from the resolver ?

The most visible effect (and the way we found out about it first)
is a 5s hang on ssh connects. Thing is: how long that timeout is
is program dependant (whatever they use in select()). A recvfrom() 
simply hangs. I wrote a simple C program to do what glibc does
(simplified for brevity):

sockfd = socket(AF_INET, SOCK_DGRAM, IPPROTO_IP);
connect(sockfd, tgt->ai_addr, tgt->ai_addrlen);
sendto(sockfd, payload1, sizeof(payload1), 0, tgt->ai_addr, tgt->ai_addrlen); 
sendto(sockfd, payload2, sizeof(payload2), 0, tgt->ai_addr, tgt->ai_addrlen); 
recvfrom(sockfd, buf, sizeof(buf), 0, &addr, &fromlen);
recvfrom(sockfd, buf, sizeof(buf), 0, &addr, &fromlen);

payload1 and 2 are an A and a AAAA request for the same name,
respectively. That second recvfrom() hangs indefinitely in the
error case. Here's the full program for those interested:

http://eric.schwarzvogel.de/~klausman/dnstest2.c.txt

It'd be easy to put in a call to select and make the program
timeout as glibc does instead of simply hanging. Note that for an
actual test in your environment, you'll probably have to change
the payloads and line 44.

Here's the tcpdump of the error case:
09:42:53.614905 IP 192.168.0.2.39355 > 192.168.22.9.53: 64583+[|domain]
09:42:53.614920 IP 192.168.0.2.39355 > 192.168.22.9.53: 61812+[|domain]
09:42:53.615623 IP 192.168.22.9.53 > 192.168.0.2.39355: 64583[|domain]

Or, if you prefer tshark:

0.000000 192.168.0.2 -> 192.168.22.9  DNS Standard query A eric.schwarzvogel.de
0.000015 192.168.0.2 -> 192.168.22.9  DNS Standard query AAAA eric.schwarzvogel.de
0.000667  192.168.22.9 -> 192.168.0.2 DNS Standard query response A 194.97.4.250

As you can see, timing on the two queries is very close. glibc
usually is in the 20-50 microsecond range on this machine, my
little program can get as low as 5 microseconds. "Correct" timing
of course depends on a myriad of variables including load on the
packetfilter, kernel version there etc etc.

A "quickfix" would indeed be using two different ports for the
queries - but the bug in Netfilter would still be there.

Regards,
Tobias

Re: [gentoo-dev] Race condition in Netfilter triggered by glibc 2.9

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 1781 bytes --]

On Thursday 29 January 2009 03:47:48 Tobias Klausmann wrote:
> Hi!
>
> On Wed, 28 Jan 2009, Mike Frysinger wrote:
> > > On the wire between the client and the firewall, this happens:
> > >
> > > a packet 1 is sent
> > > b packet 2 is sent
> > > c answer 1 is received
> > > d answer 2 is received
> > >
> > > Sometimes d doesn't happen because b is lost in the firewall
> > > along the way (where the race condition happens).
> >
> > does this affect actual userspace behavior ?  in other words,
> > does this lead to lost lookups and errors from the resolver ?
>
> The most visible effect (and the way we found out about it first)
> is a 5s hang on ssh connects.

this is why i turn off dns lookup in all my sshd_config's (well, not because 
of this bug, but because DNS lookup on ssh can cause annoying delays).  plus, 
that info is largely useless: for the logged attempts from "bad" people, the 
dns is usually screwed up / wrong / unavailable anyways.

> Thing is: how long that timeout is
> is program dependant (whatever they use in select()). A recvfrom()
> simply hangs. I wrote a simple C program to do what glibc does
> (simplified for brevity):
> ...

so glibc will not trigger hangs, just delays in some cases.

> A "quickfix" would indeed be using two different ports for the
> queries - but the bug in Netfilter would still be there.

sure, the bug still exists in netfilter (kernel).  but if we can easily 
mitigate the effects seen by applications using glibc's resolver code, that 
seems sane to me.  i havent perused the glibc resolver code in a while ... do 
you know if it can easily be tweaked to use different ports, or would such a 
change be invasive ?  if the latter, well i guess we'll have to suck it up.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 835 bytes --]

Re: [gentoo-dev] Race condition in Netfilter triggered by glibc 2.9

[Tobias Klausmann]

Hi! 

On Thu, 29 Jan 2009, Mike Frysinger wrote:
> > The most visible effect (and the way we found out about it
> > first) is a 5s hang on ssh connects.
> 
> this is why i turn off dns lookup in all my sshd_config's
> (well, not because of this bug, but because DNS lookup on ssh
> can cause annoying delays).  plus, that info is largely
> useless: for the logged attempts from "bad" people, the dns is
> usually screwed up / wrong / unavailable anyways.

It's not on the daemon side but the client side. If you don't
want to remember the IPs of all the hosts you might want to ssh
into (at close to 3000, I don't), the client will have to make
DNS lookups. Those are what delays the connection.

> > Thing is: how long that timeout is is program dependant
> > (whatever they use in select()). A recvfrom() simply hangs. I
> > wrote a simple C program to do what glibc does (simplified
> > for brevity): ...
> 
> so glibc will not trigger hangs, just delays in some cases.

Yup. Still: write a wrapper around ssh that will delay connects
by five seconds on 50% of the cases. Use it for two or more weeks
at work. That's how annoying it really is.

> > A "quickfix" would indeed be using two different ports for the
> > queries - but the bug in Netfilter would still be there.
> 
> sure, the bug still exists in netfilter (kernel).  but if we
> can easily mitigate the effects seen by applications using
> glibc's resolver code, that seems sane to me.  i havent perused
> the glibc resolver code in a while ... do you know if it can
> easily be tweaked to use different ports, or would such a
> change be invasive ?  if the latter, well i guess we'll have to
> suck it up.

I tried understanding what glibc 2.9 does regarding dns lookups,
but since it involves a rather complex (and probably quite
clever) queueing mechanism, I'm not quite sure I wouldn't break
more than I fix in doing so.

Regards,
Tobias

Re: [gentoo-dev] Race condition in Netfilter triggered by glibc 2.9

[Tobias Klausmann]

Hi! 

On Thu, 29 Jan 2009, Tobias Klausmann wrote:
> I tried understanding what glibc 2.9 does regarding dns lookups,
> but since it involves a rather complex (and probably quite
> clever) queueing mechanism, I'm not quite sure I wouldn't break
> more than I fix in doing so.

Apparently, it's enough to not export gethostbyname4() to fix
this.

I'll try building a glibc-2.9_p20081201-r1 plus this patch:
http://pasky.or.cz/~pasky/dev/glibc/glibc-2.10-dns-no-gethostbyname4.diff

If it works, I'll test drive it for a while and report back.

Regards,
Tobias

-- 
        if(ct<0)
                ct=2;        /* Shit happens.. */
        linux-2.6.6/drivers/net/wan/z85230.c

Re: [gentoo-dev] Race condition in Netfilter triggered by glibc 2.9

[Tobias Klausmann]

Hi! 

On Mon, 02 Feb 2009, Tobias Klausmann wrote:
> If it works, I'll test drive it for a while and report back.

I've been running a patched glibc-2.9_p20081201-r1 for a week
now. Nothing broke and I've been unable to trigger the lost
packet syndrome by using getaddrinfo(). I was still able to
produce it by sending UDP packets myself, but that was to be
expected.

So in essence, the patch I mentioned "works". We could use it
should we want to have a glibc 2.9. On the bug mentioned earlier
(where I got the patch from), one of the glibc guys mentions that
more work is planned for the resolver part of glibc, so this
whole ordeal may pass us.

Regards,
Tobias

-- 
printk("NONONONOO!!!!\n");
        linux-2.6.6/drivers/atm/zatm.c

Re: [gentoo-dev] Race condition in Netfilter triggered by glibc 2.9

[Mike Frysinger]

On Monday 09 February 2009 04:33:54 Tobias Klausmann wrote:
> On Mon, 02 Feb 2009, Tobias Klausmann wrote:
> > If it works, I'll test drive it for a while and report back.
>
> I've been running a patched glibc-2.9_p20081201-r1 for a week
> now. Nothing broke and I've been unable to trigger the lost
> packet syndrome by using getaddrinfo(). I was still able to
> produce it by sending UDP packets myself, but that was to be
> expected.
>
> So in essence, the patch I mentioned "works". We could use it
> should we want to have a glibc 2.9. On the bug mentioned earlier
> (where I got the patch from), one of the glibc guys mentions that
> more work is planned for the resolver part of glibc, so this
> whole ordeal may pass us.

that patch is already in the glibc patchset, i just havent released it yet.  
guess i could do that this weekend.
-mike