[gentoo-hardened] Kernel panic on openvpn connection

[gentoo-hardened] Kernel panic on openvpn connection

[Jean-Pierre Schwickerath]

Hello, 

I've been experiencing really strange behaviours with the 2 latest
hardened kernels (2.6.14-r6 and r7) on 2 different machines that are
both used as OpenVPN concentrators. 
2.6.14-r5 is working fine on these machines. 

The phenomenon is the following. When I connect to the openvpn server
from remote with openvpn, a connection is established (from the view of
the client) but in the same moment the server crashes. 

This is what I managed to capture with netconsole:

Unable to handle kernel paging request at virtual address 00695052
 printing eip:
*pgd =    0
*pmd =    0
Oops: 0000 [#1]
Modules linked in: netconsole w83781d hwmon_vid hwmon i2c_isa
ip6table_filter cls_fw sch_sfq sch_htb cls_u32 bsd_comp ppp_synctty
ppp_async crc_ccitt ppp_generic slhc ipt_DSCP ipt_tos ipt_length
ipt_TCPMSS i2c_viapro i2c_core parport_pc loop
CPU:    0
EIP: 0060:[<00000001>]    Not tainted VLI 
EFLAGS: 00010286
(2.6.14-hardened-r7) EIP is at checkCPUtype+0xfffffefc/0x81 
eax: c0c25a20   ebx: 00000000   ecx: f793e4e0   edx: f5b89640 
esi: 0028d4f0   edi: c0c95240   ebp: 00289370   esp: c0c56e70 
ds: 007b   es: 007b   ss: 0068 
Process openvpn (pid: 24564, threadinfo=c0c56000 task=f730e070) 
Stack: 0028d5c1 f5b89640 00000001 c0c95300 00000000 
0028d4f0 c0c95240 00289370 0028de4b f5b89640 00000000 c0c56ec0 f6c3c000


I first though it was machine related (hardware) but as it now happened on a
second machine, I'm not sure anymore...
I tried to recompile openvpn and the kernel (one machine has gcc 3.3.6
- the other one 3.4.5): it didn't help.

I'm unsure where to start looking for the problem. Is it a flaw in OpenVPN or could it be caused by one of the patches included in 2.4.15-r6?

Thanks for your advice. Regards, 


Jean-Pierre
-- 
Powered by Linux From Scratch - http://schwicky.net/
PGP Key ID: 0xEE6F49B4 - AIM/Jabber: Schwicky - ICQ: 4690141

Nothing is impossible... Everything is relative!
-- 
gentoo-hardened@gentoo.org mailing list

Re: [gentoo-hardened] Kernel panic on openvpn connection

[pageexec]

On 24 Apr 2006 at 19:02, Jean-Pierre Schwickerath wrote:
> I've been experiencing really strange behaviours with the 2 latest
> hardened kernels (2.6.14-r6 and r7) on 2 different machines that are
> both used as OpenVPN concentrators. 
> 2.6.14-r5 is working fine on these machines. 

looking at the diff between r5 and r6 i only see grsec related
changes, so that could be the culprit. would it be possible to
try the latest grsec patch alone (it's in grsecurity.net/~spender )?

> The phenomenon is the following. When I connect to the openvpn server
> from remote with openvpn, a connection is established (from the view of
> the client) but in the same moment the server crashes. 
> 
> This is what I managed to capture with netconsole:

is this the full oops report? also posting your kernel .config
and corresponding System.map would be useful (probably not to
the list as they are quite big).

-- 
gentoo-hardened@gentoo.org mailing list

Re: [gentoo-hardened] Kernel panic on openvpn connection

[Jean-Pierre Schwickerath]

Hello,

> > I've been experiencing really strange behaviours with the 2 latest
> > hardened kernels (2.6.14-r6 and r7) on 2 different machines that are
> > both used as OpenVPN concentrators. 
> > 2.6.14-r5 is working fine on these machines. 
> 
> looking at the diff between r5 and r6 i only see grsec related
> changes, so that could be the culprit. would it be possible to
> try the latest grsec patch alone (it's in grsecurity.net/~spender )?

I compile gentoo-sources-2.6.14-r7 and added
grsecurity-2.1.9-2.6.14.7-200602141849.patch
I couldn't make that kernel crash... 
I'll try this afternoon to add various other patches from the
hardened-patches-2.6.14-7.extras.tar.bz2 series to see which one is
responsible. 
I also tried to compile hardened-2.6.16-r4 but the make process failed with 

  LD      arch/i386/lib/built-in.o
  CC      arch/i386/lib/bitops.o
  AS      arch/i386/lib/checksum.o
  CC      arch/i386/lib/delay.o
  AS      arch/i386/lib/getuser.o
  CC      arch/i386/lib/memcpy.o
  AS      arch/i386/lib/putuser.o
  CC      arch/i386/lib/strstr.o
  CC      arch/i386/lib/usercopy.o
  AR      arch/i386/lib/lib.a
  GEN     .version
  CHK     include/linux/compile.h
  UPD     include/linux/compile.h
  CC      init/version.o
  LD      init/built-in.o
  LD      .tmp_vmlinux1
arch/i386/kernel/vmlinux.lds:1681 cannot move location counter
backwards (from 0 00000000102e388 to 000000000102e387)
make: *** [.tmp_vmlinux1] Error 1


> > The phenomenon is the following. When I connect to the openvpn
> > server from remote with openvpn, a connection is established (from
> > the view of the client) but in the same moment the server crashes. 
> > 
> > This is what I managed to capture with netconsole:
> 
> is this the full oops report? also posting your kernel .config
> and corresponding System.map would be useful (probably not to
> the list as they are quite big).

Yes, this is everything I get. 


You can find the .config file at
http://schwicky.net/linux/download/config
and the system.map at
http://schwicky.net/linux/download/System.map-2.6.14-hardened-r7
 
Regards. 
Jean-Pierre

  
-- 
Powered by Linux From Scratch - http://schwicky.net/
PGP Key ID: 0xEE6F49B4 - AIM/Jabber: Schwicky - ICQ: 4690141

Nothing is impossible... Everything is relative!
-- 
gentoo-hardened@gentoo.org mailing list

Re: [gentoo-hardened] Kernel panic on openvpn connection

[pageexec]

On 25 Apr 2006 at 13:06, Jean-Pierre Schwickerath wrote:
> > looking at the diff between r5 and r6 i only see grsec related
> > changes, so that could be the culprit. would it be possible to
> > try the latest grsec patch alone (it's in grsecurity.net/~spender )?
> 
> I compile gentoo-sources-2.6.14-r7 and added
> grsecurity-2.1.9-2.6.14.7-200602141849.patch
> I couldn't make that kernel crash... 

note that the grsec patch applies on top of vanilla sources, gentoo
sources may interfere (so you may have just gotten lucky here).

> I'll try this afternoon to add various other patches from the
> hardened-patches-2.6.14-7.extras.tar.bz2 series to see which one is
> responsible. 

i decoded the oops stack trace and it seems that the code where the
problem triggered (not necessarily the culprit) has something to do
with netfilter/bridging/ipv6. are there known problems in that area?

-- 
gentoo-hardened@gentoo.org mailing list

Re: [gentoo-hardened] Kernel panic on openvpn connection

[B.J. Orvis]

On 25 Apr 2006 at 7:27, pageexec@freemail.hu wrote:
>> I'll try this afternoon to add various other patches from the
>> hardened-patches-2.6.14-7.extras.tar.bz2 series to see which one is
>> responsible.
>>
>
> i decoded the oops stack trace and it seems that the code where the
> problem triggered (not necessarily the culprit) has something to do
> with netfilter/bridging/ipv6. are there known problems in that area?

I encountered a problem like this that I resolved a few weeks ago  
when I decided to get 2.6.14-hardened-r7 to work (r6 had the same  
problem, but I stuck to r5 until r7 came out). I have a bridge set up  
for use with openvpn.

One of the patches (1431_15.4_bridge-netfilter-race.patch) that r6  
and r7 apply to the vanilla 2.6.14 modifies the function  
br_nf_pre_routing_finish_ipv6() in net/bridge/br_netfilter.c in a way  
that made my hardened server crash whenever I attempted to ssh to it  
(over IPv6). Looking at the upstream source for the kernel (2.6.16.9  
from kernel.org), the patch appears to have been reverted back or  
never applied.
I changed the patched part to look like the upstream sources (which  
also looks like 2.6.14-hardened-r5), and that stopped the kernel  
panic. The patch calls skb_pull() rather than skb_push(), which I  
suspect filled up a buffer rather than empty it.

The following diff shows how I reverted the patch, and my server  
hasn't panicked since then.

-B.J. Orvis

diff -urd linux-2.6.14-hardened-r7/net/bridge/br_netfilter.c  
linux-2.6.14-hardened-r7-bridgemod/net/bridge/br_netfilter.c
--- linux-2.6.14-hardened-r7/net/bridge/br_netfilter.c  2006-05-01  
16:25:54.000000000 -0700
+++ linux-2.6.14-hardened-r7-bridgemod/net/bridge/ 
br_netfilter.c        2006-05-01 16:35:07.000000000 -0700
@@ -116,17 +116,30 @@
         dst_hold(skb->dst);
         skb->dev = nf_bridge->physindev;
-       if (!skb->dev)
-               kfree_skb(skb);
-       else {
-               if (skb->protocol == __constant_htons(ETH_P_8021Q)) {
-                       skb_pull(skb, VLAN_HLEN);
-                       skb->nh.raw += VLAN_HLEN;
-               }
-               skb->dst->output(skb);
+       /* the following has been shifted back to how it is in  
hardened-sources
+        * 2.6.14-r5. r6 and r7 cause a crash that i think happens  
here. In the
+        * 2.6.16.9 official linux kernel, this part is switched  
back, and the
+        * patch that applied the change is supposed to fix a race  
condition
+        * that doesnt quite look like this. maybe the if (!skb-dev)  
check is
+        * ok, but i'm trying out looking like upstream first.
+        * 1431_15.4_bridge-netfilter-race.patch
+        */
+/*     if (!skb->dev)
+ *             kfree_skb(skb);
+ *     else {
+ *             if (skb->protocol == __constant_htons(ETH_P_8021Q)) {
+ *                     skb_pull(skb, VLAN_HLEN);
+ *                     skb->nh.raw += VLAN_HLEN;
+ *             }
+ *             skb->dst->output(skb);
+ */
+       if (skb->protocol == __constant_htons(ETH_P_8021Q)) {
+               skb_push(skb, VLAN_HLEN);
+               skb->nh.raw -= VLAN_HLEN;
+               /* end of change */
         }
         NF_HOOK_THRESH(PF_BRIDGE, NF_BR_PRE_ROUTING, skb, skb->dev,  
NULL,
-                      br_handle_frame_finish, 1);
+                       br_handle_frame_finish, 1);
         return 0;
}

-- 
gentoo-hardened@gentoo.org mailing list

Re: [gentoo-hardened] Kernel panic on openvpn connection

[Jean-Pierre Schwickerath]

Hi B.J. 


> I encountered a problem like this that I resolved a few weeks ago  
> when I decided to get 2.6.14-hardened-r7 to work (r6 had the same  
> problem, but I stuck to r5 until r7 came out). I have a bridge set
> up for use with openvpn.
> 
> One of the patches (1431_15.4_bridge-netfilter-race.patch) that r6  
> and r7 apply to the vanilla 2.6.14 modifies the function  
> br_nf_pre_routing_finish_ipv6() in net/bridge/br_netfilter.c in a
> way that made my hardened server crash whenever I attempted to ssh to
> it (over IPv6). Looking at the upstream source for the kernel
> (2.6.16.9 from kernel.org), the patch appears to have been reverted
> back or never applied.
> I changed the patched part to look like the upstream sources (which  
> also looks like 2.6.14-hardened-r5), and that stopped the kernel  
> panic. The patch calls skb_pull() rather than skb_push(), which I  
> suspect filled up a buffer rather than empty it.
> 
> The following diff shows how I reverted the patch, and my server  
> hasn't panicked since then.


It took me some time before I could test this (both servers I could
test it on are production servers and it's not always easy to find a
timeframe where you can "play" with them). 
But I can confirm that your patch applied to 2.6.14-hardened-r7 does
indeed remove the panic I encountered when connecting with OpenVPN. 

Thanks.


Jean-Pierre


-- 
Powered by Linux From Scratch - http://schwicky.net/
PGP Key ID: 0xEE6F49B4 - AIM/Jabber: Schwicky - ICQ: 4690141

Nothing is impossible... Everything is relative!
-- 
gentoo-hardened@gentoo.org mailing list