[gentoo-user] IPv6 troubles

[gentoo-user] IPv6 troubles

[Mike Williams]

Hey all,

I was hoping we've got some IPv6 experts around, as I've got some "issues"
I've been banging my head against for 2 days.

Very briefly our network is a gentoo firewall box with 5 interfaces, 1 to the internet,
and 4 to private networks (192.168.xxx.0/24). What I would like to do is
assign a /64 to each "internal" network.

Our host has assigned us a /48, and added dead:beef:2::1/48 to their router as
our gateway.
I can add dead:beef:2::11/64 (yes, /64) to the internet side of router/firewall, a 
default route via dead:beef:2::1 and then happily ping ipv6 things on the internet.
Starting on one of the "internal" networks I add dead:beef:2:136::11/64, run
radvd on that interface, and the hosts on that network get v6 addresses. All
of them can ping the firewall, but cannot ping our ISPs router.
OK, so I figured I try another "internal" network, 137. Same process as above,
but this time radvd won't work:

# radvd -d5 -mstderr
[Jul 19 12:02:30] radvd: version 1.0 started
[Jul 19 12:02:30] radvd: inet_pton returned 1
[Jul 19 12:02:30] radvd: mtu for bond4 is 1500
[Jul 19 12:02:30] radvd: hardware type for bond4 is 1
[Jul 19 12:02:30] radvd: link layer token length for bond4 is 48
[Jul 19 12:02:30] radvd: prefix length for bond4 is 64
[Jul 19 12:02:30] radvd: interface definition for bond4 is ok
[Jul 19 12:02:30] radvd: sending RA on bond4
[Jul 19 12:02:30] radvd: sendmsg: Invalid argument
[Jul 19 12:02:30] radvd: setting timer: 16.00 secs
[Jul 19 12:02:30] radvd: setting timer: 16 secs 0 usecs
[Jul 19 12:02:30] radvd: calling schedule_timer from set_timer context
[Jul 19 12:02:30] radvd: calling alarm: 15 secs, 999929 usecs

sendmsg: Invalid argument ??
It's the same definition as for bond2 (136), with the interface and prefix
changed. Does the same with or without any other definitions. All but bond2
fail, but I've no idea what's so special about bond2.
The machine is amd64, and using radvd-1.0-r1.

Anyway, I can add one or two addresses manually. I do so using iproute2 
and CIDR notation, so the local route is added for me, and hosts on the 137
network can ping each other, and hosts on the 136 network after I give them 
a default route via the v6 address on the firewall interface on their network, so 
the firewall is properly forwarding traffic.
However, none of the hosts on the "internal" networks can ping any of the
hosts the firewall can ping. 
I caught the following traffic with tcpdump on the firewall:

# tcpdump -i bond2 ip6
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bond2, link-type EN10MB (Ethernet), capture size 96 bytes
12:24:02.204882 IP6 dead:beef:2:136:204:23ff:fed7:e86a > beef:dead:1f0:1:20f:3dff:feae:74c1: ICMP6, echo request, seq 1, length 64
12:24:03.208737 IP6 dead:beef:2:136:204:23ff:fed7:e86a > beef:dead:1f0:1:20f:3dff:feae:74c1: ICMP6, echo request, seq 2, length 64

# tcpdump -i bond0 ip6
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bond0, link-type EN10MB (Ethernet), capture size 96 bytes
12:24:02.205409 IP6 dead:beef:2:136:204:23ff:fed7:e86a > beef:dead:1f0:1:20f:3dff:feae:74c1: ICMP6, echo request, seq 1, length 64
12:24:02.516433 IP6 fe80::214:f600:b67e:b4db > ff02::1:ffd7:e86a: ICMP6, neighbor solicitation, who has dead:beef:2:136:204:23ff:fed7:e86a, length 32
12:24:03.208748 IP6 dead:beef:2:136:204:23ff:fed7:e86a > beef:dead:1f0:1:20f:3dff:feae:74c1: ICMP6, echo request, seq 2, length 64
12:24:03.517294 IP6 fe80::214:f600:b67e:b4db > ff02::1:ffd7:e86a: ICMP6, neighbor solicitation, who has dead:beef:2:136:204:23ff:fed7:e86a, length 32
12:24:04.517504 IP6 fe80::214:f600:b67e:b4db > ff02::1:ffd7:e86a: ICMP6, neighbor solicitation, who has dead:beef:2:136:204:23ff:fed7:e86a, length 32

bond0 and beef:dead:1f0:1::/64 are the internet side, bond2 and dead:beef:2:136::/64 
the "internal" side.
I can't understand why the firewall isn't answering/forwarding the solicitation, it knows
who dead:beef:2:136:204:23ff:fed7:e86a is.
The firewall has no netfilter rules at all, everything is default accept.

Am I just doing something stupid, or have I asked our host to set it up wrong?
Would really like to know what radvd is up to too...

Cheers

-- 
Mike Williams
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] IPv6 troubles

[Etaoin Shrdlu]

On Thursday 19 July 2007 13:45, Mike Williams wrote:

> I can add dead:beef:2::11/64 (yes, /64) to the internet side of
> router/firewall, a default route via dead:beef:2::1 and then happily
> ping ipv6 things on the internet.

Ok, so your ipv6 link to your provider (and to the ipv6 Internet) is 
working.

> Starting on one of the "internal" networks I add
> dead:beef:2:136::11/64, run radvd on that interface, and the hosts on
> that network get v6 addresses. All of them can ping the firewall, but
> cannot ping our ISPs router.

Ok, just some shots in the dark:

- Do the hosts also get the default router, along with the ipv6 address? 
You can check with "ip -6 route". You should get, among the others, a 
default route pointing to the ipv6 link local (fe80:) address of the 
router's interface on the link.

- Also, although I don't think this is the source of your problems, every 
internal router interface should recognize (and be configured to use) 
the "subnet router anycast address" for that subnet, that is, usually, 
the plain /64 subnet address (eg, dead:beef:2:136::/64). This anycast 
address has to be manually configured on the interface ("ip addr add 
dead:beef:2:136::/64 dev bond2").
Is this the address that internal hosts are able to ping on the firewall, 
or did you assign another, or are you referring to the link local 
address?

- Are you using native ipv6 connectivity with your provider or through a 
(SIT/6to4) tunnel? This is important because it affects the MTU of the 
Internet-facing interface.

Seeing the actual radvd.conf file could help better here.

> sendmsg: Invalid argument ??
> It's the same definition as for bond2 (136), with the interface and
> prefix changed. Does the same with or without any other definitions.
> All but bond2 fail, but I've no idea what's so special about bond2.
> The machine is amd64, and using radvd-1.0-r1.

Are these bondX regular single ethernet interfaces or are they of some 
other kind?

> Anyway, I can add one or two addresses manually. I do so using
> iproute2 and CIDR notation, so the local route is added for me, and
> hosts on the 137 network can ping each other, and hosts on the 136
> network after I give them a default route via the v6 address on the
> firewall interface on their network, so the firewall is properly
> forwarding traffic. 

Ok, it seems forwarding is enabled then. Are you giving default routes 
pointing to global addresses? You should try using link-local addresses 
instead.

> However, none of the hosts on the "internal" networks can ping any of
> the hosts the firewall can ping.
> I caught the following traffic with tcpdump on the firewall:
>
> # tcpdump -i bond2 ip6
> tcpdump: verbose output suppressed, use -v or -vv for full protocol
> decode listening on bond2, link-type EN10MB (Ethernet), capture size
> 96 bytes 12:24:02.204882 IP6 dead:beef:2:136:204:23ff:fed7:e86a >
> beef:dead:1f0:1:20f:3dff:feae:74c1: ICMP6, echo request, seq 1, length
> 64 12:24:03.208737 IP6 dead:beef:2:136:204:23ff:fed7:e86a >
> beef:dead:1f0:1:20f:3dff:feae:74c1: ICMP6, echo request, seq 2, length
> 64
>
> # tcpdump -i bond0 ip6
> tcpdump: verbose output suppressed, use -v or -vv for full protocol
> decode listening on bond0, link-type EN10MB (Ethernet), capture size
> 96 bytes 12:24:02.205409 IP6 dead:beef:2:136:204:23ff:fed7:e86a >
> beef:dead:1f0:1:20f:3dff:feae:74c1: ICMP6, echo request, seq 1, length
> 64 12:24:02.516433 IP6 fe80::214:f600:b67e:b4db > ff02::1:ffd7:e86a:
> ICMP6, neighbor solicitation, who has
> dead:beef:2:136:204:23ff:fed7:e86a, length 32 12:24:03.208748 IP6
> dead:beef:2:136:204:23ff:fed7:e86a >
> beef:dead:1f0:1:20f:3dff:feae:74c1: ICMP6, echo request, seq 2, length
> 64 12:24:03.517294 IP6 fe80::214:f600:b67e:b4db > ff02::1:ffd7:e86a:
> ICMP6, neighbor solicitation, who has
> dead:beef:2:136:204:23ff:fed7:e86a, length 32 12:24:04.517504 IP6
> fe80::214:f600:b67e:b4db > ff02::1:ffd7:e86a: ICMP6, neighbor
> solicitation, who has dead:beef:2:136:204:23ff:fed7:e86a, length 32

IIUC, icmpv6 echo request packets enter the router/firewall from the 
bond2 interface, and leave the box using the bond0 interface (confirming 
that forwarding works). But, the router/firewall is trying to get the 
link-layer address of the interface whose ipv6 global address is 
dead:beef:2:136:204:23ff:fed7:e86a (thus an internal host), but for some 
reason it sends these neighbor solicitation messages out of the Internet 
interface. Not surprisingly, it gets no answers.

> The firewall has no netfilter rules at all, everything is default
> accept.

Are the internal hosts using ip6tables? They might be blocking icmpv6 
messages.

> Am I just doing something stupid, or have I asked our host to set it
> up wrong? Would really like to know what radvd is up to too...

Try posting more config info (radvd), debug info (ip -6 route and ip -6 
neigh on the internal hosts and on the router) and the scripts (if any) 
you use to handle the connection (Internet side and internal side).
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] IPv6 troubles

[Mike Williams]

On Thursday 19 July 2007 13:56:23 Etaoin Shrdlu wrote:
> Ok, just some shots in the dark:
>
> - Do the hosts also get the default router, along with the ipv6 address?
> You can check with "ip -6 route". You should get, among the others, a
> default route pointing to the ipv6 link local (fe80:) address of the
> router's interface on the link.

Yep, they get a default route via the link local address of the firewall
interface local to themselves. Same happens if I subsitute it for the global
address (blah:blah:blah:137::11).

> - Also, although I don't think this is the source of your problems, every
> internal router interface should recognize (and be configured to use)
> the "subnet router anycast address" for that subnet, that is, usually,
> the plain /64 subnet address (eg, dead:beef:2:136::/64). This anycast
> address has to be manually configured on the interface ("ip addr add
> dead:beef:2:136::/64 dev bond2").
> Is this the address that internal hosts are able to ping on the firewall,
> or did you assign another, or are you referring to the link local
> address?

I hadn't configured the "subnet router anycast address", but I can still ping
it. Again makes no difference if it's specified or not.

> - Are you using native ipv6 connectivity with your provider or through a
> (SIT/6to4) tunnel? This is important because it affects the MTU of the
> Internet-facing interface.

It's native IPv6.

> Seeing the actual radvd.conf file could help better here.

interface bond2
{
        AdvSendAdvert on;
        prefix dead:beef:2:136::/64
        {
                AdvOnLink on;
                AdvAutonomous on;
        };
};

interface bond4
{
       AdvSendAdvert on;
       prefix dead:beef:2:137::/64
       {
               AdvOnLink on;
               AdvAutonomous on;
       };
};

interface bond1
{
       AdvSendAdvert on;
       prefix dead:beef:2:131::/64
       {
               AdvOnLink on;
               AdvAutonomous on;
       };
};

The order makes no difference.

> > sendmsg: Invalid argument ??
> > It's the same definition as for bond2 (136), with the interface and
> > prefix changed. Does the same with or without any other definitions.
> > All but bond2 fail, but I've no idea what's so special about bond2.
> > The machine is amd64, and using radvd-1.0-r1.
>
> Are these bondX regular single ethernet interfaces or are they of some
> other kind?

It's an ethernet link, just not a single one :)

> Ok, it seems forwarding is enabled then. Are you giving default routes
> pointing to global addresses? You should try using link-local addresses
> instead.

As above, no difference. I've even tried using the link-local address of the
upstream router for the default route out of the firewall.

> IIUC, icmpv6 echo request packets enter the router/firewall from the
> bond2 interface, and leave the box using the bond0 interface (confirming
> that forwarding works). But, the router/firewall is trying to get the
> link-layer address of the interface whose ipv6 global address is
> dead:beef:2:136:204:23ff:fed7:e86a (thus an internal host), but for some
> reason it sends these neighbor solicitation messages out of the Internet
> interface. Not surprisingly, it gets no answers.

Ahh, so I was understanding the output right.

> Are the internal hosts using ip6tables? They might be blocking icmpv6
> messages.

Nope, no ip6tables rules anywhere.

> Try posting more config info (radvd), debug info (ip -6 route and ip -6
> neigh on the internal hosts and on the router) and the scripts (if any)
> you use to handle the connection (Internet side and internal side).

radvd config above, routing and neighbour info here:

relevant routing info
# ip -6 route show 
dead:beef:2::/64 dev bond0  metric 1  expires 21253232sec mtu 1500 advmss 1440 hoplimit 4294967295
dead:beef:2::/64 dev bond0  metric 256  expires 21254724sec mtu 1500 advmss 1440 hoplimit 4294967295
dead:beef:2:131::/64 dev bond1  metric 256  expires 21256488sec mtu 1500 advmss 1440 hoplimit 4294967295
dead:beef:2:136::/64 dev bond2  metric 256  expires 21252676sec mtu 1500 advmss 1440 hoplimit 4294967295
dead:beef:2:137::/64 dev bond4  metric 256  expires 21255086sec mtu 1500 advmss 1440 hoplimit 4294967295
default via fe80::214:f600:b67e:b4db dev bond0  metric 1  expires 21334235sec mtu 1500 advmss 1440 hoplimit 4294967295

firewall # ip -6 neigh
fe80::214:f600:b67e:b4db dev bond0 lladdr 00:14:f6:7e:b4:db router STALE
dead:beef:2:136:204:23ff:fed7:e86a dev bond2 lladdr 00:04:23:d7:e8:6a REACHABLE
fe80::204:23ff:fed7:e86a dev bond2 lladdr 00:04:23:d7:e8:6a STALE

host # ip -6 neigh
dead:beef:2:136::11 dev bond0 lladdr 00:04:23:d7:f3:32 router REACHABLE
fe80::204:23ff:fed7:f332 dev bond0 lladdr 00:04:23:d7:f3:32 router REACHABLE

The host has bonded ethernet connections too.

Thanks

-- 
Mike Williams
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] IPv6 troubles

[Etaoin Shrdlu]

On Thursday 19 July 2007 14:56, Etaoin Shrdlu wrote:

> IIUC, icmpv6 echo request packets enter the router/firewall from the
> bond2 interface, and leave the box using the bond0 interface
> (confirming that forwarding works). But, the router/firewall is trying
> to get the link-layer address of the interface whose ipv6 global
> address is dead:beef:2:136:204:23ff:fed7:e86a (thus an internal host),
> but for some reason it sends these neighbor solicitation messages out
> of the Internet interface. Not surprisingly, it gets no answers.

More precisely, it seems that these neighbor solicitation messages come 
from the far end router, like it somehow believes that your internal 
host is on its same subnet, and is trying to resolve its ipv6 address to 
its link layer address (similar to what ARP does in ipv4).
Is fe80::214:f600:b67e:b4db the address of your provider's router?
There could be a misconfiguration somewhere.
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] IPv6 troubles

[Mike Williams]

On Thursday 19 July 2007 16:12:18 Etaoin Shrdlu wrote:
> More precisely, it seems that these neighbor solicitation messages come
> from the far end router, like it somehow believes that your internal
> host is on its same subnet, and is trying to resolve its ipv6 address to
> its link layer address (similar to what ARP does in ipv4).
> Is fe80::214:f600:b67e:b4db the address of your provider's router?
> There could be a misconfiguration somewhere.

fe80::214:f600:b67e:b4db is the link local address of the upstream router, 
which is also configured as dead:beef:2::1/48.
It is required that all hosts are access via, and get access though, the 
firewall we control. The upstream router can have changes made to it if 
required, but it's not good to keep bothering the ISP.

Now I think I understand what's wrong. The upstream router needs a route to 
dead:beef:2:1::/49 (or similar to cover any and all of our "internal" 
networks) via dead:beef:2::11, and be configured as dead:beef:2::1/64 instead 
of /48. Then it would route packets for dead:beef:2:136:204:23ff:fed7:e86a to 
dead:beef:2::11, rather than soliciting a link-local address for it.
Have I got that right?

Cheers

-- 
Mike Williams
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] IPv6 troubles

[Etaoin Shrdlu]

On Thursday 19 July 2007 17:02, Mike Williams wrote:

> I hadn't configured the "subnet router anycast address", but I can
> still ping it. Again makes no difference if it's specified or not.

Ok, then probably the linux implementation recognizes it anyway. But it 
seems this is not the issue here.

> interface bond2
> {
>         AdvSendAdvert on;
>         prefix dead:beef:2:136::/64
>         {
>                 AdvOnLink on;
>                 AdvAutonomous on;
>         };
> };

Looks OK.

> As above, no difference. I've even tried using the link-local address
> of the upstream router for the default route out of the firewall.

> # ip -6 route show
> dead:beef:2::/64 dev bond0  metric 1  expires 21253232sec mtu 1500
> advmss 1440 hoplimit 4294967295 

> dead:beef:2::/64 dev bond0  metric 256 expires 21254724sec mtu 1500
> advmss 1440 hoplimit 4294967295 

> dead:beef:2:131::/64 dev bond1  metric 256  expires 21256488sec mtu
> 1500 advmss 1440 hoplimit 4294967295 

> dead:beef:2:136::/64 dev bond2  metric 256  expires 21252676sec mtu
> 1500 advmss 1440 hoplimit 4294967295 

> dead:beef:2:137::/64 dev bond4  metric 256  expires 21255086sec mtu
> 1500 advmss 1440 hoplimit 4294967295 

> default via fe80::214:f600:b67e:b4db dev bond0 metric 1  expires
> 21334235sec mtu 1500 advmss 1440 hoplimit 4294967295

Ok, so fe80::214:f600:b67e:b4db *is* your provider's router interface.

You should also have, for each interface, a route to fe80::/64 and one to 
ff00::/8, eg something like

fe80::/64 dev bond1 ...
ff00::/8 dev bond1 ...

for each interface.

And at each internal host you should have a route to 
dead:beef:2:xxx::/64, one to fe80::/64, one to ff00::/8, and a default 
route pointing to the link-local address of the router interface on that 
link.

> firewall # ip -6 neigh
> fe80::214:f600:b67e:b4db dev bond0 lladdr 00:14:f6:7e:b4:db router
> STALE 
> dead:beef:2:136:204:23ff:fed7:e86a dev bond2 lladdr 00:04:23:d7:e8:6a
> REACHABLE 
> fe80::204:23ff:fed7:e86a dev bond2 lladdr 00:04:23:d7:e8:6a STALE
>
> host # ip -6 neigh
> dead:beef:2:136::11 dev bond0 lladdr 00:04:23:d7:f3:32 router
> REACHABLE 
> fe80::204:23ff:fed7:f332 dev bond0 lladdr 00:04:23:d7:f3:32 
> router REACHABLE

I assume you took this immediately after pinging the global firewall 
address (manually assigned, I supposed), otherwise the entry 
dead:beef:2:136::11 would have no reason to be there.

Everything seems ok so far.
The thing left to understand is why your provider's router, when it 
receives a packet coming from an internal host of yours, tries to do a 
neighbor discovery (similar to ARP) on that address.

I'd make sure that everything is configured correctly at the far side of 
the link. If you don't do dynamic routing with them (and if you are 
single-homed you probably don't), they should have a static route 
pointing to dead:beef:2::/48 via your firewall's bond0 link-local 
address. 

Their end of the link should definitely have a /64 (or longer) address, 
as should your end; the behaviour we are seeing /could/ mean that they 
have configured their end with a /48 address from the same pool, thus 
erroneously thinking that all the addresses belonging to your /48 block 
(ie, everything starting with dead:beef:2) belong to the link between 
you and them.

If neither of you needs said link to be pingable or otherwise reachable, 
you can also (like I do) avoid assigning global addresses to the link 
altogether and only use link-local addresses, thereby saving global 
addresses (not a concern with ipv6, but anyway...).
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] IPv6 troubles

[Etaoin Shrdlu]

On Thursday 19 July 2007 18:00, Mike Williams wrote:

> fe80::214:f600:b67e:b4db is the link local address of the upstream
> router, which is also configured as dead:beef:2::1/48.

Strictly speaking, if it's taken from the same block, it should be at 
least /49; otherwise, they would uncorrectly believe that the single 
(giant) dead:beef:2::/48 subnet is attached *directly* to the interface 
between you and them, and try to map ipv6 addresses to link-layer 
addresses accordingly.

In practice, if reachability of the transit network is needed, most 
providers give you a /64 or longer (even /126 in some cases), taken from 
a dedicated pool reserved for point-to-point and transit networks. 

In many cases then, the use of global addresses can be avoided altogether 
(see my other reply).

> It is required that all hosts are access via, and get access though,
> the firewall we control. The upstream router can have changes made to
> it if required, but it's not good to keep bothering the ISP.

Of course, but configuring their end of the link correctly is something 
that should be done by them, and /if/ they got it wrong, they should 
correct it (imho).

> Now I think I understand what's wrong. The upstream router needs a
> route to dead:beef:2:1::/49 (or similar to cover any and all of our
> "internal" networks) via dead:beef:2::11, and be configured as
> dead:beef:2::1/64 instead of /48. Then it would route packets for
> dead:beef:2:136:204:23ff:fed7:e86a to dead:beef:2::11, rather than
> soliciting a link-local address for it. Have I got that right?

Almost (see what I wrote above). With ipv6, using link-local addresses 
for routing is not in any way wrong, and in fact is usually the 
preferred way of doing things (this is different from ipv4, if nothing 
else because ipv4 does not have the notion of link-local address!). 

So, to summarize:

- you can use only link-local addresses, and manually configure static 
routes pointing to the correct local interfaces and to the other end's 
link-local address; your static route is a default one (::/0), and their 
static route is for the dead:beef:2::/48 block. If you have a single 
link to the provider and don't do dynamic routing, this is usually the 
easiest setup;

- optionally, you can also agree on assigning global addresses to the 
link; in this case, you must make sure that the subnet address assigned 
to the link is more specific (ie, has a longer prefix) than the block 
assigned to you, or is taken from a different pool.

When things are set up correctly, packets arriving at the ISP and 
addressed to dead:beef:2::/48 would be routed out their interface to 
you. To find the next hop, they would perform a neighbor discovery on 
your end's ipv6 link-local address (which they know) and, once they get 
your end's link-layer (MAC) address, they would send the traffic to your 
firewall's Internet interface. From there, your firewall has specific 
routes for the internal networks, and can delivery the packets to the 
intended recipients. 
If the physical layer betwwen you and the ISP is not ethernet, things are 
similar (there are well defined procedures for discovering the 
link-layer next-hop on almost every kind of media); the important thing 
is that each end knows the other end's link-local address.
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] IPv6 troubles

[Mike Williams]

On Thursday 19 July 2007 17:51:32 Etaoin Shrdlu wrote:
> On Thursday 19 July 2007 18:00, Mike Williams wrote:

Etaoin, thanks for you time.
I fear I would fail basic routing, which is surprising, seeing how I do 
similar things with IPv4 networks!

These addresses are all supposed to be properly internet routable, as I'm in 
the very earliest stages of making our services IPv6 available.
We're also the first customer to ask the ISP for an IPv6 transit, so this is 
new to them too. Luckily I've got a direct line to their main man.
I think the best way to go is to get a single address from a separate 
allocation, and have them route our /48 to that.

Thanks

-- 
Mike Williams
-- 
gentoo-user@gentoo.org mailing list