[gentoo-user] Cant connect to local webserver - ICMP admin prohibited

[gentoo-user] Cant connect to local webserver - ICMP admin prohibited

[Adam Carter]

# netstat -antp | grep apach
tcp        0      0 192.168.1.250:80        0.0.0.0:*
LISTEN      25577/apache2

# tcpdump -n -i eth0 host 192.168.1.6 and port not 22
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
21:10:57.011994 IP 192.168.1.6.46161 > 192.168.1.250.80: S
4279617058:4279617058(0) win 14600 <mss 1460,sackOK,timestamp 7007662
0,nop,wscale 6>
21:10:57.037227 IP 192.168.1.250 > 192.168.1.6: ICMP host
192.168.1.250 unreachable - admin prohibited filter, length 36
21:11:06.157052 IP 192.168.1.6.46162 > 192.168.1.250.80: S
3082744432:3082744432(0) win 14600 <mss 1460,sackOK,timestamp 7016807
0,nop,wscale 6>
21:11:06.182781 IP 192.168.1.250 > 192.168.1.6: ICMP host
192.168.1.250 unreachable - admin prohibited filter, length 36

ssh works. Connection from the same client to a third gentoo box
running a webserver works.

Anyone seen this behavior? There's no iptables, the hosts are gentoo
and on the same subnet. I've only seen admin prohibited ICMP from
filtering by cisco ACLs - what could be the problem?

Re: [gentoo-user] Cant connect to local webserver - ICMP admin prohibited

[Jonas de Buhr]

>ssh works. 

routing should be ok then.

>Connection from the same client to a third gentoo box
>running a webserver works.

what about connecting to the webserver from that third gentoo box?

>Anyone seen this behavior? There's no iptables,

you did check that on both machines, didn't you?
what about tcp-wrappers?

> the hosts are gentoo
>and on the same subnet. I've only seen admin prohibited ICMP from
>filtering by cisco ACLs - what could be the problem?
>

Re: [gentoo-user] Cant connect to local webserver - ICMP admin prohibited

[Adam Carter]

On Thu, Sep 22, 2011 at 10:12 PM, Jonas de Buhr <jonas.de.buhr@gmx.net> wrote:
>>ssh works.
>
> routing should be ok then.
>
>>Connection from the same client to a third gentoo box
>>running a webserver works.
>
> what about connecting to the webserver from that third gentoo box?

Same ICMP response, so its not a client side issue.

>>Anyone seen this behavior? There's no iptables,
>
> you did check that on both machines, didn't you?
> what about tcp-wrappers?

No iptables on any of the boxes. No tcpwrappers.

Re: [gentoo-user] Cant connect to local webserver - ICMP admin prohibited

[Michael Mol]

On Thu, Sep 22, 2011 at 8:25 AM, Adam Carter <adamcarter3@gmail.com> wrote:
> On Thu, Sep 22, 2011 at 10:12 PM, Jonas de Buhr <jonas.de.buhr@gmx.net> wrote:
>>>ssh works.
>>
>> routing should be ok then.
>>
>>>Connection from the same client to a third gentoo box
>>>running a webserver works.
>>
>> what about connecting to the webserver from that third gentoo box?
>
> Same ICMP response, so its not a client side issue.
>
>>>Anyone seen this behavior? There's no iptables,
>>
>> you did check that on both machines, didn't you?
>> what about tcp-wrappers?
>
> No iptables on any of the boxes. No tcpwrappers.

strace output? Which syscall is actually failing, and with what error code?

-- 
:wq

Re: [gentoo-user] Cant connect to local webserver - ICMP admin prohibited

[Richard Gration]

On 22 September 2011 12:39, Adam Carter <adamcarter3@gmail.com> wrote:
> # tcpdump -n -i eth0 host 192.168.1.6 and port not 22
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
> 21:10:57.011994 IP 192.168.1.6.46161 > 192.168.1.250.80: S
> 4279617058:4279617058(0) win 14600 <mss 1460,sackOK,timestamp 7007662
> 0,nop,wscale 6>
> 21:10:57.037227 IP 192.168.1.250 > 192.168.1.6: ICMP host
> 192.168.1.250 unreachable - admin prohibited filter, length 36

> Anyone seen this behavior? There's no iptables, the hosts are gentoo
> and on the same subnet. I've only seen admin prohibited ICMP from
> filtering by cisco ACLs - what could be the problem?

It's not the ICMP that is being prohibited. This packet:

> 21:10:57.037227 IP 192.168.1.250 > 192.168.1.6: ICMP host
> 192.168.1.250 unreachable - admin prohibited filter, length 36

is an ICMP "host unreachable" response from .250.  The extended reason
for the unreachability is that there is an administrative policy
preventing the traffic. It almost certainly *is* a firewall that's
preventing this, one with a REJECT target, as REJECT specifies to
return an ICMP unreachable packet. I suggest that you look more
closely at the firewalling on .250. If there is definitely no
firewalling going on (ie iptables -nvL shows only default policies and
the default is ACCEPT for INPUT and OUTPUT chains) then could there be
an intervening network device?

Rich

Re: [gentoo-user] Cant connect to local webserver - ICMP admin prohibited

[Jonas de Buhr]

>> what about connecting to the webserver from that third gentoo box?
>
>Same ICMP response, so its not a client side issue.

yep.

>No iptables on any of the boxes. No tcpwrappers.

is there anything in between on the network? 

does traceroute show anything unusual?

what happens if you try to connect to a closed port (81 for example)?

[gentoo-user] Re: Cant connect to local webserver - ICMP admin prohibited

[James]

Adam Carter <adamcarter3 <at> gmail.com> writes:


> ssh works. Connection from the same client to a third gentoo box
> running a webserver works.

KISS may be at work here.

go and delete the ".ssh/known_hosts"
files just to be certain it not something
really simple....

hth,
James

Re: [gentoo-user] Cant connect to local webserver - ICMP admin prohibited

[Adam Carter]

> It's not the ICMP that is being prohibited.

Understood, that's clear from the packet trace.

> is an ICMP "host unreachable" response from .250.  The extended reason
> for the unreachability is that there is an administrative policy
> preventing the traffic. It almost certainly *is* a firewall that's
> preventing this, one with a REJECT target, as REJECT specifies to
> return an ICMP unreachable packet.

Most firewalls i've seen send a spoofed TCP reset, not an ICMP when
rejecting TCP. However, iptables can do either. I have run iptables -F
and the tables are shown as clear with iptables -L.

proxy vhosts.d # iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain fail2ban-SSH (0 references)
target     prot opt source               destination

Chain fail2ban-apache (0 references)
target     prot opt source               destination
proxy vhosts.d #

> I suggest that you look more
> closely at the firewalling on .250. If there is definitely no
> firewalling going on (ie iptables -nvL shows only default policies and
> the default is ACCEPT for INPUT and OUTPUT chains) then could there be
> an intervening network device?

The devices are connected, there's only a switch between them (a
billion ADSL router).

Re: [gentoo-user] Cant connect to local webserver - ICMP admin prohibited

[Adam Carter]

> is there anything in between on the network?

Just a billion switch

> does traceroute show anything unusual?

Nothing to trace - same subnet

> what happens if you try to connect to a closed port (81 for example)?

I get the expected TCP reset.

Re: [gentoo-user] Re: Cant connect to local webserver - ICMP admin prohibited

[Adam Carter]

>> ssh works. Connection from the same client to a third gentoo box
>> running a webserver works.
>
> KISS may be at work here.
>
> go and delete the ".ssh/known_hosts"
> files just to be certain it not something
> really simple....

That file just contains the cached ssh host keys - nothing to do with
a webserver.

Re: [gentoo-user] Cant connect to local webserver - ICMP admin prohibited

[Adam Carter]

> strace output? Which syscall is actually failing, and with what error code?

What would I trace? Why do you think the information provided suggests
a syscall failing?

Re: [gentoo-user] Cant connect to local webserver - ICMP admin prohibited

[Michael Mol]

On Thu, Sep 22, 2011 at 7:14 PM, Adam Carter <adamcarter3@gmail.com> wrote:
>> strace output? Which syscall is actually failing, and with what error code?
>
> What would I trace? Why do you think the information provided suggests
> a syscall failing?

Misread your original post.



-- 
:wq

Re: [gentoo-user] Cant connect to local webserver - ICMP admin prohibited

[Mick]

[-- Attachment #1: Type: Text/Plain, Size: 253 bytes --]

On Friday 23 Sep 2011 00:10:31 Adam Carter wrote:
> > is there anything in between on the network?
> 
> Just a billion switch

  .... which is running a firewall?

(The tcpdump shows a firewall is in play on 192.168.1.250).
-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Cant connect to local webserver - ICMP admin prohibited

[Adam Carter]

>> Just a billion switch
>
>  .... which is running a firewall?
>
> (The tcpdump shows a firewall is in play on 192.168.1.250).

The firewall's disabled, and should only be in play if the packet gets
routed from what I understand. These packets are being switched. I
guess I should connect the hosts directly with a crossover to rule it
out.

Anyway, i've run up apache on port 81 and it works fine.

I also tried disabling apache and connecting to port 80 and got the
ICMP message instead of the expected TCP reset.

Re: [gentoo-user] Cant connect to local webserver - ICMP admin prohibited

[Jonas de Buhr]

>The devices are connected, there's only a switch between them (a
>billion ADSL router).

wait... billion as in "billion the company"? and
you are using your router as a switch?

please connect the two computers without any switch (crossover cable if
they aren't 1000mbit) and try again. maybe the router is doing
something funny with port 80? most routers DO run firewalls.

[gentoo-user] Re: Cant connect to local webserver - ICMP admin prohibited

[James]

Adam Carter <adamcarter3 <at> gmail.com> writes:


> > go and delete the ".ssh/known_hosts"

> That file just contains the cached ssh host keys - nothing to do with

My bad, I though I had read where you cannot ssh into the
server....

so sorry

Re: [gentoo-user] Cant connect to local webserver - ICMP admin prohibited

[Pandu Poluan]

[-- Attachment #1: Type: text/plain, Size: 1274 bytes --]

On Sep 23, 2011 6:11 AM, "Adam Carter" <adamcarter3@gmail.com> wrote:
>
> > It's not the ICMP that is being prohibited.
>
> Understood, that's clear from the packet trace.
>
> > is an ICMP "host unreachable" response from .250.  The extended reason
> > for the unreachability is that there is an administrative policy
> > preventing the traffic. It almost certainly *is* a firewall that's
> > preventing this, one with a REJECT target, as REJECT specifies to
> > return an ICMP unreachable packet.
>
> Most firewalls i've seen send a spoofed TCP reset, not an ICMP when
> rejecting TCP. However, iptables can do either. I have run iptables -F
> and the tables are shown as clear with iptables -L.
>
> proxy vhosts.d # iptables -L
> Chain INPUT (policy ACCEPT)
> target     prot opt source               destination
>
> Chain FORWARD (policy ACCEPT)
> target     prot opt source               destination
>
> Chain OUTPUT (policy ACCEPT)
> target     prot opt source               destination
>
> Chain fail2ban-SSH (0 references)
> target     prot opt source               destination
>
> Chain fail2ban-apache (0 references)
> target     prot opt source               destination
> proxy vhosts.d #
>

Can you post the outputs of 'iptables-save' and 'ip rule show'?

Rgds,

[-- Attachment #2: Type: text/html, Size: 1729 bytes --]

Re: [gentoo-user] Cant connect to local webserver - ICMP admin prohibited

[Adam Carter]

> Can you post the outputs of 'iptables-save' and 'ip rule show'?

# iptables-save
# Generated by iptables-save v1.4.12.1 on Sat Sep 24 02:57:42 2011
*nat
:PREROUTING ACCEPT [239188:15840835]
:INPUT ACCEPT [230129:15089630]
:OUTPUT ACCEPT [265028:20043915]
:POSTROUTING ACCEPT [0:0]
-A PREROUTING -s 10.0.0.254/32 -p tcp -m tcp --dport 80 -j ACCEPT
-A PREROUTING -s 10.0.0.254/32 -p tcp -m tcp --dport 8081 -j ACCEPT
-A PREROUTING -p tcp -m tcp --dport 80 -j DNAT --to-destination 10.0.0.254:3129
-A POSTROUTING -j MASQUERADE
COMMIT
# Completed on Sat Sep 24 02:57:42 2011
# Generated by iptables-save v1.4.12.1 on Sat Sep 24 02:57:42 2011
*mangle
:PREROUTING ACCEPT [63823853:97394042876]
:INPUT ACCEPT [62454740:96723050843]
:FORWARD ACCEPT [1367064:670686100]
:OUTPUT ACCEPT [47954138:21176280811]
:POSTROUTING ACCEPT [49321180:21846964975]
COMMIT
# Completed on Sat Sep 24 02:57:42 2011
# Generated by iptables-save v1.4.12.1 on Sat Sep 24 02:57:42 2011
*filter
:INPUT ACCEPT [683278:162916016]
:FORWARD ACCEPT [18:1044]
:OUTPUT ACCEPT [750201:170843065]
:fail2ban-SSH - [0:0]
:fail2ban-apache - [0:0]
COMMIT
# Completed on Sat Sep 24 02:57:42 2011

The wlan interface that uses 10. addressing is not in use at the moment.

I'm using ifconfig so i dont have the ip binary on this system.

Re: [gentoo-user] Cant connect to local webserver - ICMP admin prohibited

[Adam Carter]

On Fri, Sep 23, 2011 at 9:24 PM, Jonas de Buhr <jonas.de.buhr@gmx.net> wrote:
>>The devices are connected, there's only a switch between them (a
>>billion ADSL router).
>
> wait... billion as in "billion the company"? and
> you are using your router as a switch?

Yeah - this is just at home. The router has a 4 port switch built in.

> please connect the two computers without any switch (crossover cable if
> they aren't 1000mbit) and try again. maybe the router is doing
> something funny with port 80? most routers DO run firewalls.

Its disabled, but I will try a crossover to eliminate any possibility
its the billion.

Will gig negotiate auto cross over on a straight cable? I have a cross
over i can use, but since you mentioned gig....

Re: [gentoo-user] Cant connect to local webserver - ICMP admin prohibited

[Bill Longman]

On 09/23/2011 10:06 AM, Adam Carter wrote:
> Will gig negotiate auto cross over on a straight cable? I have a cross
> over i can use, but since you mentioned gig....

Yes. GigE is always auto-mdi by definition.

Re: [gentoo-user] Cant connect to local webserver - ICMP admin prohibited

[Pandu Poluan]

[-- Attachment #1: Type: text/plain, Size: 802 bytes --]

On Sep 24, 2011 12:05 AM, "Adam Carter" <adamcarter3@gmail.com> wrote:
>
> > Can you post the outputs of 'iptables-save' and 'ip rule show'?
>
> # iptables-save
> # Generated by iptables-save v1.4.12.1 on Sat Sep 24 02:57:42 2011
> *nat

[snip]

> -A PREROUTING -p tcp -m tcp --dport 80 -j DNAT --to-destination
10.0.0.254:3129

This line looks suspicious.

What's living at 10.0.0.254:3129 ?

Try inserting an ACCEPT target above that line, e.g.:

iptables -t nat -I PREROUTING 3 -j ACCEPT

and test again. (Use iptables-save after the above command to ensure that
the newly inserted rule indeed slips before the suspicious line).

> I'm using ifconfig so i dont have the ip binary on this system.
>

No problem. If my hunch is correct, it's that suspicious line that's been
causing you grief.

Rgds,

[-- Attachment #2: Type: text/html, Size: 1102 bytes --]