[gentoo-embedded] dhcp across segments with dnsmasq?

[gentoo-embedded] dhcp across segments with dnsmasq?

[mjinks]

We have a big old Dell running RedHat, which does nothing (any more) but
serve DHCP for a number of labs around our campus.  We thought it would
be an ideal candidate for replacement with a home-rolled GNAP machine
using dnsmasq's DHCP service.  It seemed to work well in testing, but
when we dropped in the replacement system we found that machines on
remote VLANs could no longer get addresses.

The campus network runs on Cisco switches, controlled by another group.
I'm pretty much Cisco-ignorant myself, but my undertanding is that the
VLANs which serve our DHCP clients have been configured to propagate
DHCP requests beyond the local segment and on to our centrally-located
DHCP server, and this works fine with the old machine.

A little tcpdumping shows some differences between the old (RedHat) box
and the new (GNAP) system.  In this example, "msfc-ri-v17.uchicago.edu"
is the router which serves our DHCP client; "dell-test-03.uchicago.edu"
is the DHCP client; and "bonzai.uchicago.edu" houses the DHCP server.
tcpdump is running on bonzai:

14:42:27.985373 msfc-ri-v17.uchicago.edu.bootps > bonzai.uchicago.edu.bootps:  (request) hops:1 xid:0x20dcf2e1 secs:4 flags:0x8000 G:msfc-ri-v17.uchicago.edu ether 0:f:1f:dc:f2:e1 [|bootp]
14:42:27.986066 bonzai.uchicago.edu.bootps > msfc-ri-v17.uchicago.edu.bootps:  (reply) hops:1 xid:0x20dcf2e1 secs:4 flags:0x8000 Y:dell-test-03.uchicago.edu S:bonzai.uchicago.edu G:msfc-ri-v17.uchicago.edu ether 0:f:1f:dc:f2:e1 [|bootp] (DF)

Now here's an example of the same case, except that the hardware acting
as bonzai.uchicago.edu is now our GNAP machine:

01:06:27.601181 IP msfc-ri-v17.uchicago.edu.bootps > bonzai.uchicago.edu.bootps: BOOTP/DHCP, Request [|bootp]

The GNAP box doesn't issue a response, apparently because it can't see
that the request is coming from one of its listed DHCP clients.

So, does anybody know what might be going on here?  I don't think that
dnsmasq is (necessarily) the culprit, since tcpdump shows more
information in the case of the packets dumped on the RedHat machine.  Is
this a difference, maybe, in the uclibc network handling?  Do I need to
tweak my kernel?  Am I just crazy?

Thanks for any thoughts.

--Michael
-- 
gentoo-embedded@gentoo.org mailing list

Re: [gentoo-embedded] dhcp across segments with dnsmasq?

[Natanael Copa]

On Mon, 2006-06-26 at 16:16 -0500, mjinks@uchicago.edu wrote:
> We have a big old Dell running RedHat, which does nothing (any more) but
> serve DHCP for a number of labs around our campus.  We thought it would
> be an ideal candidate for replacement with a home-rolled GNAP machine
> using dnsmasq's DHCP service.

[ snip ]

> A little tcpdumping shows some differences between the old (RedHat) box
> and the new (GNAP) system.  In this example, "msfc-ri-v17.uchicago.edu"
> is the router which serves our DHCP client; "dell-test-03.uchicago.edu"
> is the DHCP client; and "bonzai.uchicago.edu" houses the DHCP server.
> tcpdump is running on bonzai:
> 
> 14:42:27.985373 msfc-ri-v17.uchicago.edu.bootps > bonzai.uchicago.edu.bootps:  (request) hops:1 xid:0x20dcf2e1 secs:4 flags:0x8000 G:msfc-ri-v17.uchicago.edu ether 0:f:1f:dc:f2:e1 [|bootp]
> 14:42:27.986066 bonzai.uchicago.edu.bootps > msfc-ri-v17.uchicago.edu.bootps:  (reply) hops:1 xid:0x20dcf2e1 secs:4 flags:0x8000 Y:dell-test-03.uchicago.edu S:bonzai.uchicago.edu G:msfc-ri-v17.uchicago.edu ether 0:f:1f:dc:f2:e1 [|bootp] (DF)
> 
> Now here's an example of the same case, except that the hardware acting
> as bonzai.uchicago.edu is now our GNAP machine:
> 
> 01:06:27.601181 IP msfc-ri-v17.uchicago.edu.bootps > bonzai.uchicago.edu.bootps: BOOTP/DHCP, Request [|bootp]
> 
> The GNAP box doesn't issue a response, apparently because it can't see
> that the request is coming from one of its listed DHCP clients.

Have you tried running dhcpd on the GNAP box?

> So, does anybody know what might be going on here?  I don't think that
> dnsmasq is (necessarily) the culprit, since tcpdump shows more
> information in the case of the packets dumped on the RedHat machine.  

To me it looks like its dnsmasq. The request packet arrives to your GNAP
box but dnsmasq is never responding.

> Is
> this a difference, maybe, in the uclibc network handling?  Do I need to
> tweak my kernel?  Am I just crazy?
> 
> Thanks for any thoughts.

Try this one:
http://thekelleys.org.uk/dnsmasq/docs/FAQ

Look for this part:
Q: This new DHCP server is well and good, but it doesn't work for me.
   What's the problem?


--
Natanael Copa

-- 
gentoo-embedded@gentoo.org mailing list

Re: [gentoo-embedded] dhcp across segments with dnsmasq?

[mjinks]

On Tue, Jun 27, 2006 at 12:56:38AM +0200, Natanael Copa wrote:
> On Mon, 2006-06-26 at 16:16 -0500, mjinks@uchicago.edu wrote:
> > 
> > The GNAP box doesn't issue a response, apparently because it can't see
> > that the request is coming from one of its listed DHCP clients.
> 
> Have you tried running dhcpd on the GNAP box?

I tried building ISC dhcpd for GNAP, but it required a huge raft of
dependencies, some of which were failing to compile, so I gave up.  I
could try again, I suppose, but read on...

> > So, does anybody know what might be going on here?  I don't think that
> > dnsmasq is (necessarily) the culprit, since tcpdump shows more
> > information in the case of the packets dumped on the RedHat machine.  
> 
> To me it looks like its dnsmasq. The request packet arrives to your GNAP
> box but dnsmasq is never responding.

Right, but how would dnsmasq know that it should respond?  Compare this
packet dump on our old machine:

14:42:27.985373 msfc-ri-v17.uchicago.edu.bootps > bonzai.uchicago.edu.bootps:  (request) hops:1 xid:0x20dcf2e1 secs:4 flags:0x8000 G:msfc-ri-v17.uchicago.edu ether 0:f:1f:dc:f2:e1 [|bootp]

...to this one on the GNAP box:

01:06:27.601181 IP msfc-ri-v17.uchicago.edu.bootps > bonzai.uchicago.edu.bootps: BOOTP/DHCP, Request [|bootp]

Those packets come from the same client, but in the former case, there's
a lot more information in the packet dump, including the MAC address of
the originating client.  If tcpdump sees the two packets differently,
then either tcpdump isn't as smart on GNAP (possible), or information
isn't being preserved as the packet makes its way up the stack, in which
case dnsmasq is never going to see the MAC information it needs to match
to the client's address.  (Right?)

> Try this one:
> http://thekelleys.org.uk/dnsmasq/docs/FAQ
> 
> Look for this part:
> Q: This new DHCP server is well and good, but it doesn't work for me.
>    What's the problem?

I've got a good broadcast address, and I can serve DHCP to clients on
the local LAN segment.

The GNAP machine doesn't currently have any firewalling in place.

-mrj
-- 
gentoo-embedded@gentoo.org mailing list

Re: [gentoo-embedded] dhcp across segments with dnsmasq?

[Natanael Copa]

On Mon, 2006-06-26 at 18:17 -0500, mjinks@uchicago.edu wrote:
> On Tue, Jun 27, 2006 at 12:56:38AM +0200, Natanael Copa wrote:
> > On Mon, 2006-06-26 at 16:16 -0500, mjinks@uchicago.edu wrote:
> > >
> > > The GNAP box doesn't issue a response, apparently because it can't see
> > > that the request is coming from one of its listed DHCP clients.
> >
> > Have you tried running dhcpd on the GNAP box?
> 
> I tried building ISC dhcpd for GNAP, but it required a huge raft of
> dependencies, some of which were failing to compile, so I gave up.  I
> could try again, I suppose, but read on...

You could try my little pet, alpinelinux (a distro built on gentoo). at
least it has ISC dhcp. Still at the alpha stage though, but it could be
worth a try.

http://alpinelinux.org

--
Natanael Copa

-- 
gentoo-embedded@gentoo.org mailing list

Re: [gentoo-embedded] dhcp across segments with dnsmasq?

[wireless]

mjinks@uchicago.edu wrote:
> We have a big old Dell running RedHat, which does nothing (any more) but
> serve DHCP for a number of labs around our campus.  We thought it would
> be an ideal candidate for replacement with a home-rolled GNAP machine
> using dnsmasq's DHCP service.  It seemed to work well in testing, but
> when we dropped in the replacement system we found that machines on
> remote VLANs could no longer get addresses.
> 
> The campus network runs on Cisco switches, controlled by another group.
> I'm pretty much Cisco-ignorant myself, but my undertanding is that the
> VLANs which serve our DHCP clients have been configured to propagate
> DHCP requests beyond the local segment and on to our centrally-located
> DHCP server, and this works fine with the old machine.


This cisco equipment could be filtering based on MAC addresses.
You could put one of the dhcp clients on the same LAN as the
server and see if they work together. If they do, then your
problem is your campus network security. If not, then get the two
(DHCP server and DHCP client working on the same network segment;
then move the client back to it's proper segment and debut the
network.  Use Ethereal to watch the new DHCP server and one of
the DHCP clients on a quite test network. Get those 2 working
first, then distribute/debug the new server's DHCP services...

hth,

James

-- 
gentoo-embedded@gentoo.org mailing list

Re: [gentoo-embedded] dhcp across segments with dnsmasq?

[mjinks]

On Mon, Jun 26, 2006 at 09:22:17PM -0400, wireless wrote:
> 
> This cisco equipment could be filtering based on MAC addresses.

Good idea, but, nope.

> You could put one of the dhcp clients on the same LAN as the
> server and see if they work together.

They do.  That was the "testing" to which I had (vaguely) referred.

> If they do, then your
> problem is your campus network security.

I really don't think so.  Again, note the two packet dumps I posted
earlier.  In both cases, we're on the same IP address, same physical
network cable; only difference is the switch from old RedHat/Dell to new
GNAP/WRAP.  In both cases, networking in general works fine, inbound and
outbound connections work as expected, and the DHCP request packet gets
through.  But, a dump of the DHCP packet on the RedHat box shows a lot
more data, including an ethernet address for the client, which isn't
present in a packet dump on the GNAP box.

While I've been in this business too long to say that anything is
impossible, it strikes me as highly unlikely that a network switch would
make a decision to edit packets like this based on the receiving end's
MAC address, unless the switch operater had some very specific measures
in place to make that happen.  That's unlikely since the networking
folks are working with us on this, and saying that everything on their
end should be allowing what we want.

So, I'm still assuming the following:

- Both the RedHat and the GNAP box receive the same data from the wire;

- tcpdump functions equivalently on both GNAP and RedHat;[0] 

- Somewhere between the wire and tcpdump, something in GNAP is
  truncating the DHCP request packet, such that:

  a) tcpdump shows different output, and
  b) dnsmasq never sees the client MAC address, so (correctly) ignores
  the request.


[0] This is the point that I currently see as the weakest in my
hypothesis.  We had to do some semi-emergency backpedaling in order to
restore client services, so at the moment I don't have a GNAP machine
that I can use to run comparisons, so I don't know whether tcpdump
defaults to a shorter buffer size (or something) on GNAP than it does on
our old RedHat machine.  I should be able to test again tomorrow.


Knowing that DHCP conversations are often confined to a single segment,
it wouldn't surprise me to learn that there's some kind of tunneling
operation necessary to get the broadcast DHCP request off the local
segment to where it needs to go, and it also wouldn't surprise me to
learn that the receiving end of that operation would be present in
glibc, but left out of uclibc as too rarely used to deserve the code
space.  All this by way of explaining why I think it's more likely to
look at GNAP as the source of the packet differences, rather than some
Cisco arcana.

Maybe I'll go ask the uclibc folks if this is plausible, or if I'm just
confused.
-- 
gentoo-embedded@gentoo.org mailing list

Re: [gentoo-embedded] dhcp across segments with dnsmasq?

[wireless]

mjinks@uchicago.edu wrote:
> On Mon, Jun 26, 2006 at 09:22:17PM -0400, wireless wrote:
> 
>>This cisco equipment could be filtering based on MAC addresses.
> Good idea, but, nope.

MAC address filtering is but one of a myriad of things that cisco
IOS can do to manage data traffic.
> 
>>You could put one of the dhcp clients on the same LAN as the
>>server and see if they work together.
> 
> They do.  That was the "testing" to which I had (vaguely) referred.
>

Not sure what you are exactly saying here. But if they work on
the same (local) segment, then proper routing should do the rest
unless the data traffic is being managed by other devices......

>>If they do, then your
>>problem is your campus network security.
> 
> 
> I really don't think so.  Again, note the two packet dumps I posted
> earlier.  In both cases, we're on the same IP address, same physical
> network cable; only difference is the switch from old RedHat/Dell to new
> GNAP/WRAP.  In both cases, networking in general works fine, inbound and
> outbound connections work as expected, and the DHCP request packet gets
> through.  But, a dump of the DHCP packet on the RedHat box shows a lot
> more data, including an ethernet address for the client, which isn't
> present in a packet dump on the GNAP box.
> 

If the cisco switches are 'managed' that means they have IOS and
many, many things you can do to the configurations on the
switches and the resulting traffic shaping. I do not know your
network, so these suggestions  are 'sanity checks'.

Check the arp (cache) tables on the cisco equipment....

Also, many folks use Windows based software to configure
cisco equipent, (in lieu of comman line). This often leads
to multiple things happening inside the configurations
of cisco routers and managed cisco switches and pix firewalls
that are undesirable. Often the admins  are unaware of these
nuances because they do not know IOS. I'm not saying this is
your problem, just sharing a little bit of info......



> While I've been in this business too long to say that anything is
> impossible, it strikes me as highly unlikely that a network switch would
> make a decision to edit packets like this based on the receiving end's
> MAC address, unless the switch operater had some very specific measures
> in place to make that happen.  That's unlikely since the networking
> folks are working with us on this, and saying that everything on their
> end should be allowing what we want.

Still, putting a dhcp client on a 'hub with the new server and a
portable running ethereal will tell you much more that going
through tcpdump, in my opinion. You can capture packets and watch
the hanshaking (protocol negotiations) in detail and find eactly
what's missing or incompatible.


> Knowing that DHCP conversations are often confined to a single segment,
> it wouldn't surprise me to learn that there's some kind of tunneling
> operation necessary to get the broadcast DHCP request off the local
> segment to where it needs to go, and it also wouldn't surprise me to
> learn that the receiving end of that operation would be present in
> glibc, but left out of uclibc as too rarely used to deserve the code
> space.  

Hey, it's only a suggestion, but I always get things working on a
simple flat hub (that I can sniff real time traffic with
ethereal) then move to the network or Internet..... It just
simplifies debugging and tracking down problems.


hth,

James
-- 
gentoo-embedded@gentoo.org mailing list