* [gentoo-user-ru] Kernel Panic при высокой сетевой загрузке
@ 2009-02-26 21:20 Konstantin Astafjev
2009-02-26 21:22 ` Алексей Белкин
` (2 more replies)
0 siblings, 3 replies; 5+ messages in thread
From: Konstantin Astafjev @ 2009-02-26 21:20 UTC (permalink / raw
To: gentoo-user-ru
[-- Attachment #1: Type: text/plain, Size: 2158 bytes --]
Приветствую!
Вероятнее всего эта проблема не относится напрямую к Gentoo, но я
хотел бы сначала попросить совета у единственного известного мне
русскоязычного комьюнити. :)
Есть несколько маршрутизаторов на базе Intel Serverboard Aspen 3000AH,
основной функцией которых является шейпинг. Дерево HTB имеет не более
2 тысяч классов. Для фильтров используются hash tables. При суммарной
загрузке свыше 400 Mbps машинка начинает случайным образом
поругиваться и убивать процессы. В конечном итоге это всё приводит к
kernel panic. Пример ругани привожу в приложенном текстовом файле.
wr-up в примере это скрипт, который посредством запуска tc формиурет
классы и фильтры для шейпера. В данном случае он спровоцировал
убийство других процессов, но реально даже если ничего не делать
начинается: swapper: page allocation failure. order:0, mode:0x20
Налицо проблема в том, что кому-то не хватает памяти, судя по top'у
свободная память есть. На более сложные проверки у меня не хватает
знаний.
~ # top
top - 22:49:11 up 3 days, 3:06, 1 user, load average: 0.06, 0.06, 0.01
Tasks: 85 total, 1 running, 84 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.1%us, 0.1%sy, 0.0%ni, 92.2%id, 0.1%wa, 1.3%hi, 6.1%si, 0.0%st
Mem: 4148644k total, 832628k used, 3316016k free, 852k buffers
Swap: 4000056k total, 160k used, 3999896k free, 13100k cached
Чем больше трафик - тем выше вероятность ошибки. Иногда проходит 3
дня, а иногда достаточно и 5 минут. Вариант сбойного железа исключаю,
так как машинка не одна и на всех наблюдаются одинаковые симптомы тест
и проверку памяти они проходят исправно. Версии ядра менял начиная с
2.6.20, сейчас стоит 2.6.25-hardened-r10. Использование не hardened, а
gentoo ядра не помогло. Для сетевых интерфейсов использую модули e1000
и e1000e. Кажется в модуле e1000e видел совет увеличить значение
vm.min_free_kbytes. Увеличил до 65536, но в моём случае не помогло.
Может быть кто-нибудь сталкивался с такой проблемой и/или знает её
решение? В крайнем случае, подскажите как правильно дебагить такую
проблему и куда лучше обратиться. Спасибо.
--
Konstantin Astafjev
[-- Attachment #2: kern_log_error_parts.txt --]
[-- Type: text/plain, Size: 42679 bytes --]
wr-up invoked oom-killer: gfp_mask=0xd0, order=1, oomkilladj=0
Pid: 1874, comm: wr-up Not tainted 2.6.25-hardened-r10 #1
[<c0440ce7>] oom_kill_process+0x4f/0x194
[<c0441141>] out_of_memory+0x150/0x183
[<c0442be6>] __alloc_pages+0x237/0x2bd
[<c0442ca5>] __get_free_pages+0x39/0x47
[<c041c0e0>] copy_process+0x97/0x108c
[<c045c590>] alloc_file+0x24/0x2d
[<c041d230>] do_fork+0xab/0x1a1
[<c0420068>] do_wait+0x55c/0x8fb
[<c0401db4>] sys_clone+0x28/0x2d
[<c040352d>] sysenter_past_esp+0x66/0x99
=======================
Mem-info:
DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
CPU 1: hi: 0, btch: 1 usd: 0
CPU 2: hi: 0, btch: 1 usd: 0
CPU 3: hi: 0, btch: 1 usd: 0
Normal per-cpu:
CPU 0: hi: 186, btch: 31 usd: 0
CPU 1: hi: 186, btch: 31 usd: 0
CPU 2: hi: 186, btch: 31 usd: 0
CPU 3: hi: 186, btch: 31 usd: 0
HighMem per-cpu:
CPU 0: hi: 186, btch: 31 usd: 0
CPU 1: hi: 186, btch: 31 usd: 0
CPU 2: hi: 186, btch: 31 usd: 0
CPU 3: hi: 186, btch: 31 usd: 0
Active:8118 inactive:3847 dirty:0 writeback:0 unstable:0
free:825642 slab:193769 mapped:2290 pagetables:232 bounce:0
DMA free:4648kB min:1168kB low:1460kB high:1752kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 873 4556 4556
Normal free:75300kB min:64364kB low:80452kB high:96544kB active:300kB inactive:220kB present:894080kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 29464 29464
HighMem free:3222620kB min:512kB low:68388kB high:136264kB active:32172kB inactive:15168kB present:3771392kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 298*4kB 0*8kB 26*16kB 15*32kB 6*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 4648kB
Normal: 10814*4kB 0*8kB 49*16kB 186*32kB 21*64kB 2*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 5*4096kB = 75400kB
HighMem: 494*4kB 834*8kB 647*16kB 312*32kB 163*64kB 104*128kB 43*256kB 22*512kB 4*1024kB 3*2048kB 766*4096kB = 3222776kB
6705 total pagecache pages
Swap cache: add 40, delete 0, find 0/0
Free swap = 3999896kB
Total swap = 4000056kB
Free swap: 3999896kB
1179648 pages of RAM
950272 pages of HIGHMEM
142487 reserved pages
14575 pages shared
40 pages swap cached
0 pages dirty
0 pages writeback
2290 pages mapped
193769 pages slab
232 pages pagetables
Out of memory: kill process 26430 (apache2) score 30872 or a child
Killed process 26430 (apache2)
wr-up invoked oom-killer: gfp_mask=0x84d0, order=0, oomkilladj=0
Pid: 16400, comm: wr-up Not tainted 2.6.25-hardened-r10 #1
[<c0440ce7>] oom_kill_process+0x4f/0x194
[<c0441141>] out_of_memory+0x150/0x183
[<c0442be6>] __alloc_pages+0x237/0x2bd
[<c0412399>] pte_alloc_one+0x12/0x2d
[<c044a162>] __pte_alloc+0x14/0xc3
[<c044a781>] copy_page_range+0x1be/0x456
[<c041c9fd>] copy_process+0x9b4/0x108c
[<c041d230>] do_fork+0xab/0x1a1
[<c0420068>] do_wait+0x55c/0x8fb
[<c0401db4>] sys_clone+0x28/0x2d
[<c040352d>] sysenter_past_esp+0x66/0x99
[<c0680000>] i8042_create_aux_port+0x48/0xc6
=======================
Mem-info:
DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
CPU 1: hi: 0, btch: 1 usd: 0
CPU 2: hi: 0, btch: 1 usd: 0
CPU 3: hi: 0, btch: 1 usd: 0
Normal per-cpu:
CPU 0: hi: 186, btch: 31 usd: 181
CPU 1: hi: 186, btch: 31 usd: 172
CPU 2: hi: 186, btch: 31 usd: 208
CPU 3: hi: 186, btch: 31 usd: 142
HighMem per-cpu:
CPU 0: hi: 186, btch: 31 usd: 74
CPU 1: hi: 186, btch: 31 usd: 161
CPU 2: hi: 186, btch: 31 usd: 123
CPU 3: hi: 186, btch: 31 usd: 115
Active:7599 inactive:3866 dirty:0 writeback:0 unstable:0
free:822956 slab:197269 mapped:2279 pagetables:177 bounce:0
DMA free:4816kB min:1168kB low:1460kB high:1752kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 873 4556 4556
Normal free:64308kB min:64364kB low:80452kB high:96544kB active:204kB inactive:296kB present:894080kB pages_scanned:845 all_unreclaimable? yes
lowmem_reserve[]: 0 0 29464 29464
HighMem free:3222700kB min:512kB low:68388kB high:136264kB active:30192kB inactive:15168kB present:3771392kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 286*4kB 25*8kB 28*16kB 15*32kB 6*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 4832kB
Normal: 6553*4kB 39*8kB 401*16kB 186*32kB 21*64kB 2*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 5*4096kB = 64300kB
HighMem: 395*4kB 859*8kB 639*16kB 314*32kB 163*64kB 107*128kB 43*256kB 22*512kB 4*1024kB 3*2048kB 766*4096kB = 3222900kB
6700 total pagecache pages
Swap cache: add 40, delete 0, find 0/0
Free swap = 3999896kB
Total swap = 4000056kB
Free swap: 3999896kB
1179648 pages of RAM
950272 pages of HIGHMEM
142487 reserved pages
12685 pages shared
40 pages swap cached
0 pages dirty
0 pages writeback
2279 pages mapped
197269 pages slab
177 pages pagetables
Out of memory: kill process 26431 (apache2) score 30872 or a child
Killed process 26431 (apache2)
wr-up invoked oom-killer: gfp_mask=0x84d0, order=0, oomkilladj=0
Pid: 17072, comm: wr-up Not tainted 2.6.25-hardened-r10 #1
[<c0440ce7>] oom_kill_process+0x4f/0x194
[<c0441141>] out_of_memory+0x150/0x183
[<c0442be6>] __alloc_pages+0x237/0x2bd
[<c0442cef>] get_zeroed_page+0x3c/0x4a
[<c0412715>] pgd_alloc+0x61/0xcc
[<c041befd>] mm_init+0xa1/0xc3
[<c041c844>] copy_process+0x7fb/0x108c
[<c041d230>] do_fork+0xab/0x1a1
[<c0420068>] do_wait+0x55c/0x8fb
[<c0401db4>] sys_clone+0x28/0x2d
[<c040352d>] sysenter_past_esp+0x66/0x99
[<c0680000>] i8042_create_aux_port+0x48/0xc6
=======================
Mem-info:
DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
CPU 1: hi: 0, btch: 1 usd: 0
CPU 2: hi: 0, btch: 1 usd: 0
CPU 3: hi: 0, btch: 1 usd: 0
Normal per-cpu:
CPU 0: hi: 186, btch: 31 usd: 161
CPU 1: hi: 186, btch: 31 usd: 140
CPU 2: hi: 186, btch: 31 usd: 164
CPU 3: hi: 186, btch: 31 usd: 176
HighMem per-cpu:
CPU 0: hi: 186, btch: 31 usd: 50
CPU 1: hi: 186, btch: 31 usd: 158
CPU 2: hi: 186, btch: 31 usd: 49
CPU 3: hi: 186, btch: 31 usd: 138
Active:7011 inactive:3859 dirty:0 writeback:0 unstable:0
free:823623 slab:197381 mapped:2302 pagetables:154 bounce:0
DMA free:4752kB min:1168kB low:1460kB high:1752kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 873 4556 4556
Normal free:64272kB min:64364kB low:80452kB high:96544kB active:340kB inactive:268kB present:894080kB pages_scanned:1088 all_unreclaimable? yes
lowmem_reserve[]: 0 0 29464 29464
HighMem free:3225468kB min:512kB low:68388kB high:136264kB active:27704kB inactive:15168kB present:3771392kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 284*4kB 25*8kB 29*16kB 16*32kB 7*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 4808kB
Normal: 6572*4kB 40*8kB 401*16kB 186*32kB 21*64kB 2*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 5*4096kB = 64384kB
HighMem: 668*4kB 890*8kB 616*16kB 323*32kB 174*64kB 108*128kB 45*256kB 22*512kB 4*1024kB 3*2048kB 766*4096kB = 3225504kB
6702 total pagecache pages
Swap cache: add 40, delete 0, find 0/0
Free swap = 3999896kB
Total swap = 4000056kB
Free swap: 3999896kB
1179648 pages of RAM
950272 pages of HIGHMEM
142487 reserved pages
10550 pages shared
40 pages swap cached
0 pages dirty
0 pages writeback
2302 pages mapped
197381 pages slab
154 pages pagetables
Out of memory: kill process 26426 (apache2) score 3819 or a child
Killed process 26426 (apache2)
wr-up invoked oom-killer: gfp_mask=0x84d0, order=0, oomkilladj=0
Pid: 18061, comm: wr-up Not tainted 2.6.25-hardened-r10 #1
[<c0440ce7>] oom_kill_process+0x4f/0x194
[<c0441141>] out_of_memory+0x150/0x183
[<c0442be6>] __alloc_pages+0x237/0x2bd
[<c0442cef>] get_zeroed_page+0x3c/0x4a
[<c0412715>] pgd_alloc+0x61/0xcc
[<c041befd>] mm_init+0xa1/0xc3
[<c045ec9b>] bprm_mm_init+0xf/0x143
[<c045faf3>] do_execve+0xfa/0x350
[<c044b3ae>] handle_mm_fault+0x603/0x66e
[<c0684502>] do_page_fault+0x33c/0x769
[<c0684569>] do_page_fault+0x3a3/0x769
[<c0401d4b>] sys_execve+0x2a/0x4f
[<c040352d>] sysenter_past_esp+0x66/0x99
[<c0680000>] i8042_create_aux_port+0x48/0xc6
=======================
Mem-info:
DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
CPU 1: hi: 0, btch: 1 usd: 0
CPU 2: hi: 0, btch: 1 usd: 0
CPU 3: hi: 0, btch: 1 usd: 0
Normal per-cpu:
CPU 0: hi: 186, btch: 31 usd: 70
CPU 1: hi: 186, btch: 31 usd: 114
CPU 2: hi: 186, btch: 31 usd: 42
CPU 3: hi: 186, btch: 31 usd: 181
HighMem per-cpu:
CPU 0: hi: 186, btch: 31 usd: 98
CPU 1: hi: 186, btch: 31 usd: 168
CPU 2: hi: 186, btch: 31 usd: 141
CPU 3: hi: 186, btch: 31 usd: 75
Active:6678 inactive:3879 dirty:37 writeback:0 unstable:0
free:823851 slab:197598 mapped:2270 pagetables:211 bounce:0
DMA free:4660kB min:1168kB low:1460kB high:1752kB active:0kB inactive:36kB present:16256kB pages_scanned:66 all_unreclaimable? yes
lowmem_reserve[]: 0 873 4556 4556
Normal free:64272kB min:64364kB low:80452kB high:96544kB active:228kB inactive:308kB present:894080kB pages_scanned:1344 all_unreclaimable? yes
lowmem_reserve[]: 0 0 29464 29464
HighMem free:3226472kB min:512kB low:68388kB high:136264kB active:26508kB inactive:15172kB present:3771392kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 258*4kB 26*8kB 31*16kB 19*32kB 4*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 4648kB
Normal: 6574*4kB 20*8kB 401*16kB 186*32kB 21*64kB 2*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 5*4096kB = 64232kB
HighMem: 634*4kB 951*8kB 620*16kB 319*32kB 178*64kB 106*128kB 45*256kB 23*512kB 4*1024kB 3*2048kB 766*4096kB = 3226304kB
6746 total pagecache pages
Swap cache: add 40, delete 0, find 0/0
Free swap = 3999896kB
Total swap = 4000056kB
Free swap: 3999896kB
1179648 pages of RAM
950272 pages of HIGHMEM
142487 reserved pages
13250 pages shared
40 pages swap cached
37 pages dirty
0 pages writeback
2270 pages mapped
197598 pages slab
211 pages pagetables
Out of memory: kill process 17945 (apache2) score 61039 or a child
Killed process 17945 (apache2)
wr-up invoked oom-killer: gfp_mask=0x84d0, order=0, oomkilladj=0
Pid: 27365, comm: wr-up Not tainted 2.6.25-hardened-r10 #1
[<c0440ce7>] oom_kill_process+0x4f/0x194
[<c0441141>] out_of_memory+0x150/0x183
[<c0442be6>] __alloc_pages+0x237/0x2bd
[<c0412399>] pte_alloc_one+0x12/0x2d
[<c044a162>] __pte_alloc+0x14/0xc3
[<c044a781>] copy_page_range+0x1be/0x456
[<c041c9fd>] copy_process+0x9b4/0x108c
[<c041d230>] do_fork+0xab/0x1a1
[<c0425839>] recalc_sigpending+0xb/0x1d
[<c0425924>] sigprocmask+0xa1/0xbb
[<c0401db4>] sys_clone+0x28/0x2d
[<c040352d>] sysenter_past_esp+0x66/0x99
[<c0680000>] i8042_create_aux_port+0x48/0xc6
=======================
Mem-info:
DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
CPU 1: hi: 0, btch: 1 usd: 0
CPU 2: hi: 0, btch: 1 usd: 0
CPU 3: hi: 0, btch: 1 usd: 0
Normal per-cpu:
CPU 0: hi: 186, btch: 31 usd: 20
CPU 1: hi: 186, btch: 31 usd: 4
CPU 2: hi: 186, btch: 31 usd: 107
CPU 3: hi: 186, btch: 31 usd: 6
HighMem per-cpu:
CPU 0: hi: 186, btch: 31 usd: 110
CPU 1: hi: 186, btch: 31 usd: 153
CPU 2: hi: 186, btch: 31 usd: 138
CPU 3: hi: 186, btch: 31 usd: 81
Active:6757 inactive:3783 dirty:0 writeback:0 unstable:0
free:823831 slab:197916 mapped:2277 pagetables:211 bounce:0
DMA free:4652kB min:1168kB low:1460kB high:1752kB active:0kB inactive:0kB present:16256kB pages_scanned:4 all_unreclaimable? yes
lowmem_reserve[]: 0 873 4556 4556
Normal free:63980kB min:64364kB low:80452kB high:96544kB active:688kB inactive:0kB present:894080kB pages_scanned:1777 all_unreclaimable? yes
lowmem_reserve[]: 0 0 29464 29464
HighMem free:3226692kB min:512kB low:68388kB high:136264kB active:26344kB inactive:15168kB present:3771392kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 288*4kB 8*8kB 30*16kB 18*32kB 5*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 4640kB
Normal: 6494*4kB 7*8kB 405*16kB 186*32kB 21*64kB 2*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 5*4096kB = 63872kB
HighMem: 639*4kB 951*8kB 620*16kB 320*32kB 178*64kB 107*128kB 45*256kB 23*512kB 4*1024kB 3*2048kB 766*4096kB = 3226484kB
6711 total pagecache pages
Swap cache: add 40, delete 0, find 0/0
Free swap = 3999896kB
Total swap = 4000056kB
Free swap: 3999896kB
1179648 pages of RAM
950272 pages of HIGHMEM
142487 reserved pages
13039 pages shared
40 pages swap cached
0 pages dirty
0 pages writeback
2277 pages mapped
197916 pages slab
211 pages pagetables
Out of memory: kill process 18742 (apache2) score 61039 or a child
Killed process 18742 (apache2)
..........[skipped]
sshd invoked oom-killer: gfp_mask=0xd0, order=0, oomkilladj=0
Pid: 19486, comm: sshd Not tainted 2.6.25-hardened-r10 #1
[<c0440ce7>] oom_kill_process+0x4f/0x194
[<c0441141>] out_of_memory+0x150/0x183
[<c0442be6>] __alloc_pages+0x237/0x2bd
[<c063b9d0>] tcp_sendmsg+0x4af/0x997
[<c04342e2>] tick_dev_program_event+0x1e/0x82
[<c060f160>] sock_aio_write+0xb3/0xc0
[<c042f24a>] hrtimer_reprogram+0x83/0x9b
[<c045b4c8>] do_sync_write+0xbf/0x100
[<c0626a4c>] qdisc_watchdog_schedule+0x16/0x19
[<c042cf60>] autoremove_wake_function+0x0/0x2d
[<c0505920>] security_file_permission+0xc/0xd
[<c045bbe0>] vfs_write+0x94/0xf6
[<c045c105>] sys_write+0x3c/0x63
[<c040352d>] sysenter_past_esp+0x66/0x99
=======================
Mem-info:
DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
CPU 1: hi: 0, btch: 1 usd: 0
CPU 2: hi: 0, btch: 1 usd: 0
CPU 3: hi: 0, btch: 1 usd: 0
Normal per-cpu:
CPU 0: hi: 186, btch: 31 usd: 132
CPU 1: hi: 186, btch: 31 usd: 179
CPU 2: hi: 186, btch: 31 usd: 143
CPU 3: hi: 186, btch: 31 usd: 128
HighMem per-cpu:
CPU 0: hi: 186, btch: 31 usd: 61
CPU 1: hi: 186, btch: 31 usd: 33
CPU 2: hi: 186, btch: 31 usd: 108
CPU 3: hi: 186, btch: 31 usd: 118
Active:17244 inactive:2522 dirty:1 writeback:0 unstable:0
free:813843 slab:198617 mapped:1358 pagetables:115 bounce:0
DMA free:4648kB min:1168kB low:1460kB high:1752kB active:0kB inactive:100kB present:16256kB pages_scanned:177 all_unreclaimable? yes
lowmem_reserve[]: 0 873 4556 4556
Normal free:60360kB min:64364kB low:80452kB high:96544kB active:468kB inactive:136kB present:894080kB pages_scanned:942 all_unreclaimable? yes
lowmem_reserve[]: 0 0 29464 29464
HighMem free:3190364kB min:512kB low:68388kB high:136264kB active:68528kB inactive:9852kB present:3771392kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 110*4kB 85*8kB 36*16kB 17*32kB 6*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 4672kB
Normal: 5274*4kB 140*8kB 424*16kB 186*32kB 21*64kB 2*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 5*4096kB = 60360kB
HighMem: 1*4kB 1*8kB 1*16kB 2*32kB 0*64kB 0*128kB 52*256kB 35*512kB 15*1024kB 3*2048kB 766*4096kB = 3190364kB
4144 total pagecache pages
Swap cache: add 40, delete 0, find 0/0
Free swap = 3999896kB
Total swap = 4000056kB
Free swap: 3999896kB
1179648 pages of RAM
950272 pages of HIGHMEM
142487 reserved pages
4961 pages shared
40 pages swap cached
1 pages dirty
0 pages writeback
1358 pages mapped
198617 pages slab
115 pages pagetables
Out of memory: kill process 19486 (sshd) score 2256 or a child
Killed process 19489 (bash)
....................................[skipped]...........
Feb 4 19:18:37 kernsrc@router swapper: page allocation failure. order:0, mode:0x20
Feb 4 19:18:37 kernsrc@router Pid: 0, comm: swapper Not tainted 2.6.25-hardened-r10 #1
Feb 4 19:18:37 kernsrc@router [<c0442c58>] __alloc_pages+0x2a9/0x2bd
Feb 4 19:18:37 kernsrc@router [<c0458d8b>] cache_alloc_refill+0x26b/0x454
Feb 4 19:18:37 kernsrc@router [<c0458fd6>] __kmalloc+0x62/0x9c
Feb 4 19:18:37 kernsrc@router [<c0614cb5>] __alloc_skb+0x49/0xf5
Feb 4 19:18:37 kernsrc@router [<c06157f0>] __netdev_alloc_skb+0x14/0x2d
Feb 4 19:18:37 kernsrc@router [<f8965945>] e1000_alloc_rx_buffers+0xa0/0x258 [e1000]
Feb 4 19:18:37 kernsrc@router [<f89663d5>] e1000_clean_rx_irq+0x427/0x460 [e1000]
Feb 4 19:18:37 kernsrc@router [<f8963cea>] e1000_clean+0x51/0x1c8 [e1000]
Feb 4 21:21:07 kernsrc@router [<c042fa86>] ktime_get_ts+0x11/0x3a
Feb 4 21:21:07 kernsrc@router [<c061aa9a>] net_rx_action+0x8a/0x157
Feb 4 21:21:07 kernsrc@router [<c0421be0>] __do_softirq+0x63/0xcc
Feb 4 21:21:07 kernsrc@router [<c0421c7e>] do_softirq+0x35/0x39
Feb 4 21:21:07 kernsrc@router [<c0421e8e>] irq_exit+0x38/0x6a
Feb 4 21:21:07 kernsrc@router [<c04059d1>] do_IRQ+0x73/0x82
Feb 4 21:21:07 kernsrc@router [<c0404503>] common_interrupt+0x23/0x30
Feb 4 21:21:07 kernsrc@router [<c0401e0b>] mwait_idle_with_hints+0x36/0x39
Feb 4 21:21:07 kernsrc@router [<c0401e0e>] mwait_idle+0x0/0xa
Feb 4 21:21:07 kernsrc@router [<c0402139>] cpu_idle+0x6d/0x86
Feb 4 21:21:07 kernsrc@router =======================
Feb 4 21:21:07 kernsrc@router Mem-info:
Feb 4 21:21:07 kernsrc@router DMA per-cpu:
Feb 4 21:21:07 kernsrc@router CPU 0: hi: 0, btch: 1 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 1: hi: 0, btch: 1 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 2: hi: 0, btch: 1 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 3: hi: 0, btch: 1 usd: 0
Feb 4 21:21:07 kernsrc@router Normal per-cpu:
Feb 4 21:21:07 kernsrc@router CPU 0: hi: 186, btch: 31 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 1: hi: 186, btch: 31 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 2: hi: 186, btch: 31 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 3: hi: 186, btch: 31 usd: 0
Feb 4 21:21:07 kernsrc@router HighMem per-cpu:
Feb 4 21:21:07 kernsrc@router CPU 0: hi: 186, btch: 31 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 1: hi: 186, btch: 31 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 2: hi: 186, btch: 31 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 3: hi: 186, btch: 31 usd: 0
Feb 4 21:21:07 kernsrc@router Active:5522 inactive:3048 dirty:0 writeback:0 unstable:0
Feb 4 21:21:07 kernsrc@router free:816292 slab:208254 mapped:1614 pagetables:112 bounce:0
Feb 4 21:21:07 kernsrc@router DMA free:3920kB min:1168kB low:1460kB high:1752kB active:24kB inactive:0kB present:16256kB pages_scanned:36 all_unreclaimable? no
Feb 4 21:21:07 kernsrc@router lowmem_reserve[]: 0 873 4556 4556
Feb 4 21:21:07 kernsrc@router Normal free:24048kB min:64364kB low:80452kB high:96544kB active:640kB inactive:748kB present:894080kB pages_scanned:1291 all_unreclaimable? no
Feb 4 21:21:07 kernsrc@router lowmem_reserve[]: 0 0 29464 29464
Feb 4 21:21:07 kernsrc@router HighMem free:3237200kB min:512kB low:68388kB high:136264kB active:21424kB inactive:11444kB present:3771392kB pages_scanned:0 all_unreclaimable? no
Feb 4 21:21:07 kernsrc@router lowmem_reserve[]: 0 0 0 0
Feb 4 21:21:07 kernsrc@router DMA: 1*4kB 2*8kB 2*16kB 4*32kB 4*64kB 3*128kB 2*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 3892kB
Feb 4 21:21:07 kernsrc@router Normal: 2*4kB 2*8kB 1*16kB 3*32kB 1*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 5*4096kB = 24264kB
Feb 4 21:21:07 kernsrc@router HighMem: 1038*4kB 505*8kB 243*16kB 83*32kB 39*64kB 12*128kB 6*256kB 5*512kB 5*1024kB 3*2048kB 782*4096kB = 3237200kB
Feb 4 21:21:07 kernsrc@router 6029 total pagecache pages
Feb 4 21:21:07 kernsrc@router Swap cache: add 0, delete 0, find 0/0
Feb 4 21:21:07 kernsrc@router Free swap = 4000056kB
Feb 4 21:21:07 kernsrc@router Total swap = 4000056kB
Feb 4 21:21:07 kernsrc@router Free swap: 4000056kB
Feb 4 21:21:07 kernsrc@router swapper: page allocation failure. order:0, mode:0x20
Feb 4 21:21:07 kernsrc@router Pid: 0, comm: swapper Not tainted 2.6.25-hardened-r10 #1
Feb 4 21:21:07 kernsrc@router [<c0442c58>] __alloc_pages+0x2a9/0x2bd
Feb 4 21:21:07 kernsrc@router [<c0458d8b>] cache_alloc_refill+0x26b/0x454
Feb 4 21:21:07 kernsrc@router [<c0458fd6>] __kmalloc+0x62/0x9c
Feb 4 21:21:07 kernsrc@router [<f8cb00c3>] match+0x328/0x501 [xt_layer7]
Feb 4 21:21:07 kernsrc@router [<f8ca80c3>] tos_tg_v0+0x6c/0x91 [xt_DSCP]
Feb 4 21:21:07 kernsrc@router [<f8c56278>] ipt_do_table+0x1b7/0x427 [ip_tables]
Feb 4 21:21:07 kernsrc@router [<c062d268>] nf_iterate+0x30/0x63
Feb 4 21:21:07 kernsrc@router [<c062d2f9>] nf_hook_slow+0x3a/0x90
Feb 4 21:21:07 kernsrc@router [<f89a59cc>] br_nf_pre_routing_finish+0x0/0x291 [bridge]
Feb 4 21:21:07 kernsrc@router [<f89a6661>] br_nf_pre_routing+0x520/0x547 [bridge]
Feb 4 21:21:07 kernsrc@router [<f89a59cc>] br_nf_pre_routing_finish+0x0/0x291 [bridge]
Feb 4 21:21:07 kernsrc@router [<c062d268>] nf_iterate+0x30/0x63
Feb 4 21:21:07 kernsrc@router [<f89a2596>] br_handle_frame_finish+0x0/0x104 [bridge]
Feb 4 21:21:07 kernsrc@router [<c062d2f9>] nf_hook_slow+0x3a/0x90
Feb 4 21:21:07 kernsrc@router [<f89a2596>] br_handle_frame_finish+0x0/0x104 [bridge]
Feb 4 21:21:07 kernsrc@router [<f89a283f>] br_handle_frame+0x1a5/0x1ca [bridge]
Feb 4 21:21:07 kernsrc@router [<f89a2596>] br_handle_frame_finish+0x0/0x104 [bridge]
Feb 4 21:21:07 kernsrc@router [<c0618a1f>] netif_receive_skb+0x268/0x391
Feb 4 21:21:07 kernsrc@router [<f89112d7>] e1000_receive_skb+0x12f/0x149 [e1000e]
Feb 4 21:21:07 kernsrc@router [<f8913291>] e1000_clean_rx_irq+0x20e/0x2b1 [e1000e]
Feb 4 21:21:07 kernsrc@router [<f8910429>] e1000_clean+0x4c/0x1b1 [e1000e]
Feb 4 21:21:07 kernsrc@router [<c061aa9a>] net_rx_action+0x8a/0x157
Feb 4 21:21:07 kernsrc@router [<c0421be0>] __do_softirq+0x63/0xcc
Feb 4 21:21:07 kernsrc@router [<c0421c7e>] do_softirq+0x35/0x39
Feb 4 21:21:07 kernsrc@router [<c0421e8e>] irq_exit+0x38/0x6a
Feb 4 21:21:07 kernsrc@router [<c04059d1>] do_IRQ+0x73/0x82
Feb 4 21:21:07 kernsrc@router [<c0404503>] common_interrupt+0x23/0x30
Feb 4 21:21:07 kernsrc@router [<c0401e0b>] mwait_idle_with_hints+0x36/0x39
Feb 4 21:21:07 kernsrc@router [<c0401e0e>] mwait_idle+0x0/0xa
Feb 4 21:21:07 kernsrc@router [<c0402139>] cpu_idle+0x6d/0x86
Feb 4 21:21:07 kernsrc@router =======================
Feb 4 21:21:07 kernsrc@router Mem-info:
Feb 4 21:21:07 kernsrc@router DMA per-cpu:
Feb 4 21:21:07 kernsrc@router CPU 0: hi: 0, btch: 1 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 1: hi: 0, btch: 1 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 2: hi: 0, btch: 1 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 3: hi: 0, btch: 1 usd: 0
Feb 4 21:21:07 kernsrc@router Normal per-cpu:
Feb 4 21:21:07 kernsrc@router CPU 0: hi: 186, btch: 31 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 1: hi: 186, btch: 31 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 2: hi: 186, btch: 31 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 3: hi: 186, btch: 31 usd: 0
Feb 4 21:21:07 kernsrc@router HighMem per-cpu:
Feb 4 21:21:07 kernsrc@router CPU 0: hi: 186, btch: 31 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 1: hi: 186, btch: 31 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 2: hi: 186, btch: 31 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 3: hi: 186, btch: 31 usd: 0
Feb 4 21:21:07 kernsrc@router Active:5522 inactive:3048 dirty:0 writeback:0 unstable:0
Feb 4 21:21:07 kernsrc@router free:816292 slab:208254 mapped:1614 pagetables:112 bounce:0
Feb 4 21:21:07 kernsrc@router DMA free:3920kB min:1168kB low:1460kB high:1752kB active:24kB inactive:0kB present:16256kB pages_scanned:36 all_unreclaimable? no
Feb 4 21:21:07 kernsrc@router lowmem_reserve[]: 0 873 4556 4556
Feb 4 21:21:07 kernsrc@router Normal free:24048kB min:64364kB low:80452kB high:96544kB active:640kB inactive:748kB present:894080kB pages_scanned:1291 all_unreclaimable? no
Feb 4 21:21:07 kernsrc@router lowmem_reserve[]: 0 0 29464 29464
Feb 4 21:21:07 kernsrc@router HighMem free:3237200kB min:512kB low:68388kB high:136264kB active:21424kB inactive:11444kB present:3771392kB pages_scanned:0 all_unreclaimable? no
Feb 4 21:21:07 kernsrc@router lowmem_reserve[]: 0 0 0 0
Feb 4 21:21:07 kernsrc@router DMA: 1*4kB 2*8kB 2*16kB 4*32kB 4*64kB 3*128kB 2*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 3892kB
Feb 4 21:21:07 kernsrc@router Normal: 2*4kB 2*8kB 1*16kB 3*32kB 1*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 5*4096kB = 24264kB
Feb 4 21:21:07 kernsrc@router HighMem: 1038*4kB 505*8kB 243*16kB 83*32kB 39*64kB 12*128kB 6*256kB 5*512kB 5*1024kB 3*2048kB 782*4096kB = 3237200kB
Feb 4 21:21:07 kernsrc@router 6029 total pagecache pages
Feb 4 21:21:07 kernsrc@router Swap cache: add 0, delete 0, find 0/0
Feb 4 21:21:07 kernsrc@router Free swap = 4000056kB
Feb 4 21:21:07 kernsrc@router Total swap = 4000056kB
Feb 4 21:21:07 kernsrc@router Free swap: 4000056kB
Feb 4 21:21:07 kernsrc@router 1179648 pages of RAM
Feb 4 21:21:07 kernsrc@router 1179648 pages of RAM
Feb 4 21:21:07 kernsrc@router 950272 pages of HIGHMEM
Feb 4 21:21:07 kernsrc@router 142487 reserved pages
Feb 4 21:21:07 kernsrc@router 9788 pages shared
Feb 4 21:21:07 kernsrc@router 0 pages swap cached
Feb 4 21:21:07 kernsrc@router 0 pages dirty
Feb 4 21:21:07 kernsrc@router 950272 pages of HIGHMEM
Feb 4 21:21:07 kernsrc@router 0 pages writeback
Feb 4 21:21:07 kernsrc@router 1614 pages mapped
Feb 4 21:21:07 kernsrc@router 208254 pages slab
Feb 4 21:21:07 kernsrc@router 112 pages pagetables
Feb 4 21:21:07 kernsrc@router 142487 reserved pages
Feb 4 21:21:07 kernsrc@router 9788 pages shared
Feb 4 21:21:07 kernsrc@router 0 pages swap cached
Feb 4 21:21:07 kernsrc@router 0 pages dirty
Feb 4 21:21:07 kernsrc@router 0 pages writeback
Feb 4 21:21:07 kernsrc@router 1614 pages mapped
Feb 4 21:21:07 kernsrc@router 208254 pages slab
Feb 4 21:21:07 kernsrc@router 112 pages pagetables
Feb 4 21:21:07 kernsrc@router layer7: out of memory in match, bailing.
Feb 4 21:21:07 kernsrc@router swapper: page allocation failure. order:0, mode:0x20
Feb 4 21:21:07 kernsrc@router Pid: 0, comm: swapper Not tainted 2.6.25-hardened-r10 #1
Feb 4 21:21:07 kernsrc@router [<c0442c58>] __alloc_pages+0x2a9/0x2bd
Feb 4 21:21:07 kernsrc@router [<c0458d8b>] cache_alloc_refill+0x26b/0x454
Feb 4 21:21:07 kernsrc@router [<c04045c8>] apic_timer_interrupt+0x28/0x30
Feb 4 21:21:07 kernsrc@router [<c0458fd6>] __kmalloc+0x62/0x9c
Feb 4 21:21:07 kernsrc@router [<f8cb00c3>] match+0x328/0x501 [xt_layer7]
Feb 4 21:21:07 kernsrc@router [<f8ca80c3>] tos_tg_v0+0x6c/0x91 [xt_DSCP]
Feb 4 21:21:07 kernsrc@router [<f8c56278>] ipt_do_table+0x1b7/0x427 [ip_tables]
Feb 4 21:21:07 kernsrc@router [<c062d268>] nf_iterate+0x30/0x63
Feb 4 21:21:07 kernsrc@router [<c062d2f9>] nf_hook_slow+0x3a/0x90
Feb 4 21:21:07 kernsrc@router [<f89a59cc>] br_nf_pre_routing_finish+0x0/0x291 [bridge]
Feb 4 21:21:07 kernsrc@router [<f89a6661>] br_nf_pre_routing+0x520/0x547 [bridge]
Feb 4 21:21:07 kernsrc@router [<f89a59cc>] br_nf_pre_routing_finish+0x0/0x291 [bridge]
Feb 4 21:21:07 kernsrc@router [<c062d268>] nf_iterate+0x30/0x63
Feb 4 21:21:07 kernsrc@router [<f89a2596>] br_handle_frame_finish+0x0/0x104 [bridge]
Feb 4 21:21:07 kernsrc@router [<c062d2f9>] nf_hook_slow+0x3a/0x90
Feb 4 21:21:07 kernsrc@router [<f89a2596>] br_handle_frame_finish+0x0/0x104 [bridge]
Feb 4 21:21:07 kernsrc@router [<f89a283f>] br_handle_frame+0x1a5/0x1ca [bridge]
Feb 4 21:21:07 kernsrc@router [<f89a2596>] br_handle_frame_finish+0x0/0x104 [bridge]
Feb 4 21:21:07 kernsrc@router [<c0618a1f>] netif_receive_skb+0x268/0x391
Feb 4 21:21:07 kernsrc@router [<f89112d7>] e1000_receive_skb+0x12f/0x149 [e1000e]
Feb 4 21:21:07 kernsrc@router [<f8913291>] e1000_clean_rx_irq+0x20e/0x2b1 [e1000e]
Feb 4 21:21:07 kernsrc@router [<f8910429>] e1000_clean+0x4c/0x1b1 [e1000e]
Feb 4 21:21:07 kernsrc@router [<c061aa9a>] net_rx_action+0x8a/0x157
Feb 4 21:21:07 kernsrc@router [<c0421be0>] __do_softirq+0x63/0xcc
Feb 4 21:21:07 kernsrc@router [<c0421c7e>] do_softirq+0x35/0x39
Feb 4 21:21:07 kernsrc@router [<c0421e8e>] irq_exit+0x38/0x6a
Feb 4 21:21:07 kernsrc@router [<c04059d1>] do_IRQ+0x73/0x82
Feb 4 21:21:07 kernsrc@router [<c0404503>] common_interrupt+0x23/0x30
Feb 4 21:21:07 kernsrc@router [<c0401e0b>] mwait_idle_with_hints+0x36/0x39
Feb 4 21:21:07 kernsrc@router [<c0401e0e>] mwait_idle+0x0/0xa
Feb 4 21:21:07 kernsrc@router [<c0402139>] cpu_idle+0x6d/0x86
Feb 4 21:21:07 kernsrc@router =======================
Feb 4 21:21:07 kernsrc@router Mem-info:
Feb 4 21:21:07 kernsrc@router DMA per-cpu:
Feb 4 21:21:07 kernsrc@router CPU 0: hi: 0, btch: 1 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 1: hi: 0, btch: 1 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 2: hi: 0, btch: 1 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 3: hi: 0, btch: 1 usd: 0
Feb 4 21:21:07 kernsrc@router Normal per-cpu:
Feb 4 21:21:07 kernsrc@router CPU swap = 4000056kB
Feb 4 21:21:07 kernsrc@router Free swap: 4000056kB
Feb 4 21:21:07 kernsrc@router 1179648 pages of RAM
Feb 4 21:21:07 kernsrc@router 950272 pages of HIGHMEM
Feb 4 21:21:07 kernsrc@router 142487 reserved pages
Feb 4 21:21:07 kernsrc@router 9793 pages shared
Feb 4 21:21:07 kernsrc@router 0 pages swap cached
Feb 4 21:21:07 kernsrc@router 0 pages dirty
Feb 4 21:21:07 kernsrc@router 0 pages writeback
Feb 4 21:21:07 kernsrc@router 1619 pages mapped
Feb 4 21:21:07 kernsrc@router 208296 pages slab
Feb 4 21:21:07 kernsrc@router 112 pages pagetables
Feb 4 21:21:07 kernsrc@router printk: 102551 messages suppressed.
Feb 4 21:21:07 kernsrc@router swapper: page allocation failure. order:0, mode:0x20
Feb 4 21:21:07 kernsrc@router Pid: 0, comm: swapper Not tainted 2.6.25-hardened-r10 #1
Feb 4 21:21:07 kernsrc@router [<c0442c58>] __alloc_pages+0x2a9/0x2bd
Feb 4 21:21:07 kernsrc@router [<c0458d8b>] cache_alloc_refill+0x26b/0x454
Feb 4 21:21:07 kernsrc@router [<f89a283f>] br_handle_frame+0x1a5/0x1ca [bridge]
Feb 4 21:21:07 kernsrc@router [<c0458fd6>] __kmalloc+0x62/0x9c
Feb 4 21:21:07 kernsrc@router [<c0614cb5>] __alloc_skb+0x49/0xf5
Feb 4 21:21:07 kernsrc@router [<c06157f0>] __netdev_alloc_skb+0x14/0x2d
Feb 4 21:21:07 kernsrc@router [<f8966178>] e1000_clean_rx_irq+0x1ca/0x460 [e1000]
Feb 4 21:21:07 kernsrc@router [<f8963cea>] e1000_clean+0x51/0x1c8 [e1000]
Feb 4 21:21:07 kernsrc@router [<c061aa9a>] net_rx_action+0x8a/0x157
Feb 4 21:21:07 kernsrc@router [<c0421be0>] __do_softirq+0x63/0xcc
Feb 4 21:21:07 kernsrc@router [<c0421c7e>] do_softirq+0x35/0x39
Feb 4 21:21:07 kernsrc@router [<c0421e8e>] irq_exit+0x38/0x6a
Feb 4 21:21:07 kernsrc@router [<c040ed01>] smp_apic_timer_interrupt+0x75/0x80
Feb 4 21:21:07 kernsrc@router [<c04045c8>] apic_timer_interrupt+0x28/0x30
Feb 4 21:21:07 kernsrc@router [<c0401e0b>] mwait_idle_with_hints+0x36/0x39
Feb 4 21:21:07 kernsrc@router [<c0401e0e>] mwait_idle+0x0/0xa
Feb 4 21:21:07 kernsrc@router [<c0402139>] cpu_idle+0x6d/0x86
Feb 4 21:21:07 kernsrc@router =======================
Feb 4 21:21:07 kernsrc@router Mem-info:
Feb 4 21:21:07 kernsrc@router DMA per-cpu:
Feb 4 21:21:07 kernsrc@router CPU 0: hi: 0, btch: 1 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 1: hi: 0, btch: 1 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 2: hi: 0, btch: 1 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 3: hi: 0, btch: 1 usd: 0
Feb 4 21:21:07 kernsrc@router Normal per-cpu:
Feb 4 21:21:07 kernsrc@router CPU 0: hi: 186, btch: 31 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 1: hi: 186, btch: 31 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 2: hi: 186, btch: 31 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 3: hi: 186, btch: 31 usd: 0
Feb 4 21:21:07 kernsrc@router HighMem per-cpu:
Feb 4 21:21:07 kernsrc@router CPU 0: hi: 186, btch: 31 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 1: hi: 186, btch: 31 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 2: hi: 186, btch: 31 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 3: hi: 186, btch: 31 usd: 0
Feb 4 21:21:07 kernsrc@router Active:5590 inactive:3009 dirty:0 writeback:0 unstable:0
Feb 4 21:21:07 kernsrc@router free:816313 slab:208296 mapped:1619 pagetables:112 bounce:0
Feb 4 21:21:07 kernsrc@router DMA free:3912kB min:1168kB low:1460kB high:1752kB active:12kB inactive:36kB present:16256kB pages_scanned:99 all_unreclaimable? no
Feb 4 21:21:07 kernsrc@router lowmem_reserve[]: 0 873 4556 4556
Feb 4 21:21:07 kernsrc@router Normal free:24140kB min:64364kB low:80452kB high:96544kB active:904kB inactive:576kB present:894080kB pages_scanned:2415 all_unreclaimable? yes
Feb 4 21:21:07 kernsrc@router lowmem_reserve[]: 0 0 29464 29464
Feb 4 21:21:07 kernsrc@router HighMem free:3237200kB min:512kB low:68388kB high:136264kB active:21444kB inactive:11424kB present:3771392kB pages_scanned:0 all_unreclaimable? no
Feb 4 21:21:07 kernsrc@router lowmem_reserve[]: 0 0 0 0
Feb 4 21:21:07 kernsrc@router DMA: 6*4kB 2*8kB 2*16kB 4*32kB 4*64kB 3*128kB 2*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 3912kB
Feb 4 21:21:07 kernsrc@router Normal: 31*4kB 0*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 5*4096kB = 24140kB
Feb 4 21:21:07 kernsrc@router HighMem: 1038*4kB 505*8kB 243*16kB 83*32kB 39*64kB 12*128kB 6*256kB 5*512kB 5*1024kB 3*2048kB 782*4096kB = 3237200kB
Feb 4 21:21:07 kernsrc@router 6029 total pagecache pages
Feb 4 21:21:07 kernsrc@router Swap cache: add 0, delete 0, find 0/0
Feb 4 21:21:07 kernsrc@router Free swap = 4000056kB
Feb 4 21:21:07 kernsrc@router Total swap = 4000056kB
Feb 4 21:21:07 kernsrc@router Free swap: 4000056kB
Feb 4 21:21:07 kernsrc@router 1179648 pages of RAM
Feb 4 21:21:07 kernsrc@router 950272 pages of HIGHMEM
Feb 4 21:21:07 kernsrc@router 142487 reserved pages
Feb 4 21:21:07 kernsrc@router 9793 pages shared
Feb 4 21:21:07 kernsrc@router 0 pages swap cached
Feb 4 21:21:07 kernsrc@router 0 pages dirty
Feb 4 21:21:07 kernsrc@router 0 pages writeback
Feb 4 21:21:07 kernsrc@router 1619 pages mapped
Feb 4 21:21:07 kernsrc@router 208296 pages slab
Feb 4 21:21:07 kernsrc@router 112 pages pagetables
Feb 4 21:21:07 kernsrc@router printk: 95995 messages suppressed.
Feb 4 21:21:07 kernsrc@router swapper: page allocation failure. order:0, mode:0x20
Feb 4 21:21:07 kernsrc@router Pid: 0, comm: swapper Not tainted 2.6.25-hardened-r10 #1
Feb 4 21:21:07 kernsrc@router [<c0442c58>] __alloc_pages+0x2a9/0x2bd
Feb 4 21:21:07 kernsrc@router [<c0458d8b>] cache_alloc_refill+0x26b/0x454
Feb 4 21:21:07 kernsrc@router [<f89a283f>] br_handle_frame+0x1a5/0x1ca [bridge]
Feb 4 21:21:07 kernsrc@router [<c0458fd6>] __kmalloc+0x62/0x9c
Feb 4 21:21:07 kernsrc@router [<c0614cb5>] __alloc_skb+0x49/0xf5
Feb 4 21:21:07 kernsrc@router [<c06157f0>] __netdev_alloc_skb+0x14/0x2d
Feb 4 21:21:07 kernsrc@router [<f8966178>] e1000_clean_rx_irq+0x1ca/0x460 [e1000]
Feb 4 21:21:07 kernsrc@router [<f8963cea>] e1000_clean+0x51/0x1c8 [e1000]
Feb 4 21:21:07 kernsrc@router [<c061aa9a>] net_rx_action+0x8a/0x157
Feb 4 21:21:07 kernsrc@router [<c0421be0>] __do_softirq+0x63/0xcc
Feb 4 21:21:07 kernsrc@router [<c0421c7e>] do_softirq+0x35/0x39
Feb 4 21:21:07 kernsrc@router [<c0421e8e>] irq_exit+0x38/0x6a
Feb 4 21:21:07 kernsrc@router [<c040ed01>] smp_apic_timer_interrupt+0x75/0x80
Feb 4 21:21:07 kernsrc@router [<c04045c8>] apic_timer_interrupt+0x28/0x30
Feb 4 21:21:07 kernsrc@router [<c0401e0b>] mwait_idle_with_hints+0x36/0x39
Feb 4 21:21:07 kernsrc@router [<c0401e0e>] mwait_idle+0x0/0xa
Feb 4 21:21:07 kernsrc@router [<c0402139>] cpu_idle+0x6d/0x86
Feb 4 21:21:07 kernsrc@router =======================
Feb 4 21:21:07 kernsrc@router Mem-info:
Feb 4 21:21:07 kernsrc@router DMA per-cpu:
Feb 4 21:21:07 kernsrc@router CPU 0: hi: 0, btch: 1 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 1: hi: 0, btch: 1 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 2: hi: 0, btch: 1 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 3: hi: 0, btch: 1 usd: 0
Feb 4 21:21:07 kernsrc@router Normal per-cpu:
Feb 4 21:21:07 kernsrc@router CPU 0: hi: 186, btch: 31 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 1: hi: 186, btch: 31 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 2: hi: 186, btch: 31 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 3: hi: 186, btch: 31 usd: 0
Feb 4 21:21:07 kernsrc@router HighMem per-cpu:
Feb 4 21:21:07 kernsrc@router CPU 0: hi: 186, btch: 31 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 1: hi: 186, btch: 31 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 2: hi: 186, btch: 31 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 3: hi: 186, btch: 31 usd: 0
Feb 4 21:21:07 kernsrc@router Active:5581 inactive:3000 dirty:0 writeback:0 unstable:0
Feb 4 21:21:07 kernsrc@router free:816289 slab:208295 mapped:1619 pagetables:112 bounce:0
Feb 4 21:21:07 kernsrc@router DMA free:3912kB min:1168kB low:1460kB high:1752kB active:4kB inactive:36kB present:16256kB pages_scanned:72 all_unreclaimable? yes
Feb 4 21:21:07 kernsrc@router lowmem_reserve[]: 0 873 4556 4556
Feb 4 21:21:07 kernsrc@router Normal free:24044kB min:64364kB low:80452kB high:96544kB active:876kB inactive:540kB present:894080kB pages_scanned:2535 all_unreclaimable? yes
Feb 4 21:21:07 kernsrc@router lowmem_reserve[]: 0 0 29464 29464
Feb 4 21:21:07 kernsrc@router HighMem free:3237200kB min:512kB low:68388kB high:136264kB active:21444kB inactive:11424kB present:3771392kB pages_scanned:0 all_unreclaimable? no
Feb 4 21:21:07 kernsrc@router lowmem_reserve[]: 0 0 0 0
Feb 4 21:21:07 kernsrc@router DMA: 6*4kB 2*8kB 2*16kB 4*32kB 4*64kB 3*128kB 2*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 3912kB
Feb 4 21:21:07 kernsrc@router Normal: 31*4kB 0*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 5*4096kB = 24140kB
Feb 4 21:21:07 kernsrc@router HighMem: 1038*4kB 505*8kB 243*16kB 83*32kB 39*64kB 12*128kB 6*256kB 5*512kB 5*1024kB 3*2048kB 782*4096kB = 3237200kB
Feb 4 21:21:07 kernsrc@router 6029 total pagecache pages
Feb 4 21:21:07 kernsrc@router Swap cache: add 0, delete 0, find 0/0
Feb 4 21:21:07 kernsrc@router Free swap = 4000056kB
Feb 4 21:21:07 kernsrc@router Total swap = 4000056kB
Feb 4 21:21:07 kernsrc@router Free swap: 4000056kB
Feb 4 21:21:07 kernsrc@router 1179648 pages of RAM
Feb 4 21:21:07 kernsrc@router 950272 pages of HIGHMEM
Feb 4 21:21:07 kernsrc@router 142487 reserved pages
Feb 4 21:21:07 kernsrc@router 9793 pages shared
Feb 4 21:21:07 kernsrc@router 0 pages swap cached
Feb 4 21:21:07 kernsrc@router 0 pages dirty
Feb 4 21:21:07 kernsrc@router 0 pages writeback
Feb 4 21:21:07 kernsrc@router 1619 pages mapped
Feb 4 21:21:07 kernsrc@router 208295 pages slab
Feb 4 21:21:07 kernsrc@router 112 pages pagetables
Feb 4 21:21:07 kernsrc@router printk: 53439 messages suppressed.
Feb 4 21:21:07 kernsrc@router swapper: page allocation failure. order:0, mode:0x20
Feb 4 21:21:07 kernsrc@router Pid: 0, comm: swapper Not tainted 2.6.25-hardened-r10 #1
Feb 4 21:21:07 kernsrc@router [<c0442c58>] __alloc_pages+0x2a9/0x2bd
Feb 4 21:21:07 kernsrc@router [<c04342e2>] tick_dev_program_event+0x1e/0x82
Feb 4 21:21:07 kernsrc@router [<c0458d8b>] cache_alloc_refill+0x26b/0x454
Feb 4 21:21:07 kernsrc@router [<c0434395>] tick_program_event+0x1f/0x23
Feb 4 21:21:07 kernsrc@router [<c0458fd6>] __kmalloc+0x62/0x9c
Feb 4 21:21:07 kernsrc@router [<c0614cb5>] __alloc_skb+0x49/0xf5
Feb 4 21:21:07 kernsrc@router [<c06157f0>] __netdev_alloc_skb+0x14/0x2d
Feb 4 21:21:07 kernsrc@router [<f8966178>] e1000_clean_rx_irq+0x1ca/0x460 [e1000]
Feb 4 21:21:07 kernsrc@router [<f8963cea>] e1000_clean+0x51/0x1c8 [e1000]
Feb 4 21:21:07 kernsrc@router [<c042fa86>] ktime_get_ts+0x11/0x3a
Feb 4 21:21:07 kernsrc@router [<c061aa9a>] net_rx_action+0x8a/0x157
Feb 4 21:21:07 kernsrc@router [<c0421be0>] __do_softirq+0x63/0xcc
Feb 4 21:21:07 kernsrc@router [<c0421c7e>] do_softirq+0x35/0x39
Feb 4 21:21:07 kernsrc@router [<c0421e8e>] irq_exit+0x38/0x6a
Feb 4 21:21:07 kernsrc@router [<c04059d1>] do_IRQ+0x73/0x82
Feb 4 21:21:07 kernsrc@router [<c0404503>] common_interrupt+0x23/0x30
Feb 4 21:21:07 kernsrc@router [<c0401e0b>] mwait_idle_with_hints+0x36/0x39
Feb 4 21:21:07 kernsrc@router [<c0401e0e>] mwait_idle+0x0/0xa
Feb 4 21:21:07 kernsrc@router [<c0402139>] cpu_idle+0x6d/0x86
Feb 4 21:21:07 kernsrc@router =======================
Feb 4 21:21:07 kernsrc@router Mem-info:
Feb 4 21:21:07 kernsrc@router DMA per-cpu:
Feb 4 21:21:07 kernsrc@router CPU 0: hi: 0, btch: 1 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 1: hi: 0, btch: 1 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 2: hi: 0, btch: 1 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 3: hi: 0, btch: 1 usd: 0
Feb 4 21:21:07 kernsrc@router Normal per-cpu:
Feb 4 21:21:07 kernsrc@router CPU 0: hi: 186, btch: 31 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 1: hi: 186, btch: 31 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 2: hi: 186, btch: 31 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 3: hi: 186, btch: 31 usd: 0
Feb 4 21:21:07 kernsrc@router HighMem per-cpu:
Feb 4 21:21:07 kernsrc@router CPU 0: hi: 186, btch: 31 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 1: hi: 186, btch: 31 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 2: hi: 186, btch: 31 usd: 0
Feb 4 21:21:07 kernsrc@router CPU 3: hi: 186, btch: 31 usd: 0
Feb 4 21:21:07 kernsrc@router Active:5606 inactive:2936 dirty:0 writeback:0 unstable:0
Feb 4 21:21:07 kernsrc@router free:816313 slab:208296 mapped:1619 pagetables:112 bounce:0
Feb 4 21:21:07 kernsrc@router DMA free:3912kB min:1168kB low:1460kB high:1752kB active:12kB inactive:0kB present:16256kB pages_scanned:45 all_unreclaimable? yes
Feb 4 21:21:07 kernsrc@router lowmem_reserve[]: 0 873 4556 4556
Feb 4 21:21:07 kernsrc@router Normal free:24140kB min:64364kB low:80452kB high:96544kB active:968kB inactive:320kB present:894080kB pages_scanned:2514 all_unreclaimable? no
Feb 4 21:21:07 kernsrc@router lowmem_reserve[]: 0 0 29464 29464
Feb 4 21:21:07 kernsrc@router HighMem free:3237200kB min:512kB low:68388kB high:136264kB active:21444kB inactive:11424kB present:3771392kB pages_scanned:0 all_unreclaimable? no
Feb 4 21:21:07 kernsrc@router lowmem_reserve[]: 0 0 0 0
...............................[skipped]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [gentoo-user-ru] Kernel Panic при высокой сетевой загрузке
2009-02-26 21:20 [gentoo-user-ru] Kernel Panic при высокой сетевой загрузке Konstantin Astafjev
@ 2009-02-26 21:22 ` Алексей Белкин
2009-02-27 15:40 ` [gentoo-user-ru] " alexey conovalov
2009-02-28 12:12 ` Peter Volkov
2 siblings, 0 replies; 5+ messages in thread
From: Алексей Белкин @ 2009-02-26 21:22 UTC (permalink / raw
To: gentoo-user-ru
[-- Attachment #1: Type: text/plain, Size: 2284 bytes --]
On Пятница 27 февраля 2009 01:20:58 Konstantin Astafjev wrote:
> Приветствую!
>
> Вероятнее всего эта проблема не относится напрямую к Gentoo, но я
> хотел бы сначала попросить совета у единственного известного мне
> русскоязычного комьюнити. :)
>
> Есть несколько маршрутизаторов на базе Intel Serverboard Aspen 3000AH,
> основной функцией которых является шейпинг. Дерево HTB имеет не более
> 2 тысяч классов. Для фильтров используются hash tables. При суммарной
> загрузке свыше 400 Mbps машинка начинает случайным образом
> поругиваться и убивать процессы. В конечном итоге это всё приводит к
> kernel panic. Пример ругани привожу в приложенном текстовом файле.
> wr-up в примере это скрипт, который посредством запуска tc формиурет
> классы и фильтры для шейпера. В данном случае он спровоцировал
> убийство других процессов, но реально даже если ничего не делать
> начинается: swapper: page allocation failure. order:0, mode:0x20
> Налицо проблема в том, что кому-то не хватает памяти, судя по top'у
> свободная память есть. На более сложные проверки у меня не хватает
> знаний.
>
> ~ # top
> top - 22:49:11 up 3 days, 3:06, 1 user, load average: 0.06, 0.06, 0.01
> Tasks: 85 total, 1 running, 84 sleeping, 0 stopped, 0 zombie
> Cpu(s): 0.1%us, 0.1%sy, 0.0%ni, 92.2%id, 0.1%wa, 1.3%hi, 6.1%si,
> 0.0%st Mem: 4148644k total, 832628k used, 3316016k free, 852k
> buffers Swap: 4000056k total, 160k used, 3999896k free, 13100k
> cached
>
> Чем больше трафик - тем выше вероятность ошибки. Иногда проходит 3
> дня, а иногда достаточно и 5 минут. Вариант сбойного железа исключаю,
> так как машинка не одна и на всех наблюдаются одинаковые симптомы тест
> и проверку памяти они проходят исправно. Версии ядра менял начиная с
> 2.6.20, сейчас стоит 2.6.25-hardened-r10. Использование не hardened, а
> gentoo ядра не помогло. Для сетевых интерфейсов использую модули e1000
> и e1000e. Кажется в модуле e1000e видел совет увеличить значение
> vm.min_free_kbytes. Увеличил до 65536, но в моём случае не помогло.
>
> Может быть кто-нибудь сталкивался с такой проблемой и/или знает её
> решение? В крайнем случае, подскажите как правильно дебагить такую
> проблему и куда лучше обратиться. Спасибо.
Ого, русская рассылка живая О_о
Простите за оффтоп)
[-- Attachment #2: Type: text/html, Size: 3066 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* [gentoo-user-ru] Re: [gentoo-user-ru] Kernel Panic при высокой сетевой загрузке
2009-02-26 21:20 [gentoo-user-ru] Kernel Panic при высокой сетевой загрузке Konstantin Astafjev
2009-02-26 21:22 ` Алексей Белкин
@ 2009-02-27 15:40 ` alexey conovalov
2009-02-28 12:12 ` Peter Volkov
2 siblings, 0 replies; 5+ messages in thread
From: alexey conovalov @ 2009-02-27 15:40 UTC (permalink / raw
To: gentoo-user-ru
> Может быть кто-нибудь сталкивался с такой проблемой и/или знает её
> решение? В крайнем случае, подскажите как правильно дебагить такую
> проблему и куда лучше обратиться. Спасибо.
С таким не сталкивался, погуглил
на LKML.org писали о похожих проблемах
например http://lkml.org/lkml/2008/10/26/188
есть советы по sysctrl
http://www.cyberciti.biz/faq/linux-page-allocation-failure-erro/
vm.lower_zone_protection
vm.vfs_cache_pressure
полумера на мой взгляд.
А все ли у вас хорошо с конфигами ядра в части vm?
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [gentoo-user-ru] Kernel Panic при высокой сетевой загрузке
2009-02-26 21:20 [gentoo-user-ru] Kernel Panic при высокой сетевой загрузке Konstantin Astafjev
2009-02-26 21:22 ` Алексей Белкин
2009-02-27 15:40 ` [gentoo-user-ru] " alexey conovalov
@ 2009-02-28 12:12 ` Peter Volkov
2009-02-28 12:34 ` Dmitry Pisarev
2 siblings, 1 reply; 5+ messages in thread
From: Peter Volkov @ 2009-02-28 12:12 UTC (permalink / raw
To: gentoo-user-ru
[-- Attachment #1: Type: text/plain, Size: 1585 bytes --]
В Чтв, 26/02/2009 в 23:20 +0200, Konstantin Astafjev пишет:
> swapper: page allocation failure. order:0, mode:0x20
>
> Для сетевых интерфейсов использую модули e1000 и e1000e.
А что у вас за железо? Кроме того:
http://kerneltrap.org/index.php?q=mailarchive/linux-kernel/2008/2/24/969714
Так что, если e1000e у вас работают, вы точно видите те же самые
сообщения без e1000?
> Может быть кто-нибудь сталкивался с такой проблемой и/или знает её
> решение? В крайнем случае, подскажите как правильно дебагить такую
> проблему и куда лучше обратиться.
Не сталкивался. Так как у вас хорошо воспроизводится проблема, высокая
нагрузка драйвера e1000 -> паника, то можно начать писать напрямую
разработчикам intel (из файла /usr/src/linux/MAINTAINERS):
INTEL ETHERNET DRIVERS (e100/e1000/e1000e/igb/ixgb/ixgbe)
L: e1000-devel@lists.sourceforge.net
W: http://e1000.sourceforge.net/
И в любом случае интересно, чем закончится это разбирательство. Так что
если найдёте workaround или решение, напишите, пожалуйста, в эту
рассылку.
--
Peter.
[-- Attachment #2: Эта часть сообщения подписана цифровой подписью --]
[-- Type: application/pgp-signature, Size: 197 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [gentoo-user-ru] Kernel Panic при высокой сетевой загрузке
2009-02-28 12:12 ` Peter Volkov
@ 2009-02-28 12:34 ` Dmitry Pisarev
0 siblings, 0 replies; 5+ messages in thread
From: Dmitry Pisarev @ 2009-02-28 12:34 UTC (permalink / raw
To: gentoo-user-ru
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
>> Может быть кто-нибудь сталкивался с такой проблемой и/или знает её
>> решение? В крайнем случае, подскажите как правильно дебагить такую
>> проблему и куда лучше обратиться.
>
> Не сталкивался. Так как у вас хорошо воспроизводится проблема, высокая
> нагрузка драйвера e1000 -> паника, то можно начать писать напрямую
> разработчикам intel (из файла /usr/src/linux/MAINTAINERS):
>
> INTEL ETHERNET DRIVERS (e100/e1000/e1000e/igb/ixgb/ixgbe)
> L: e1000-devel@lists.sourceforge.net
> W: http://e1000.sourceforge.net/
>
ну вообще не факт, что из-за драйвера сетевой карты..
вроде по его дампам проблема с памятью.. можно например покурить /proc/sys/kernel/shmmax или около
того... или с числом симофоров в /proc/sys/kernel/sem
можно посмотреть сколько памяти выделяется на страницу памяти, а может я сейчас и бред несу ).
при такой нагрузке не удивительно, что драйвер сетевой карты потребляет много ЦПУ... но там же еще
куча шейперов (хотя как писал человек, такое количество должно быть как два пальца об асфальт),
которые как раз и жрут память.. ( своими очередями? или чем нить еще)..
можно посмотреть ulimit юзера, от которого запускается wr-up... хотя есть от рута, то хм. )..
вообще я с какими масштабами не сталкивался... )
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iQEcBAEBAgAGBQJJqS9OAAoJEFTFN2idlx7w/JwIAK2ydQI6NoCbxF3eotVBrK+n
QpIPWaUQs5T+Z8V/J7p0qK8RcEYzMpjcX3Hgxm/U4UtfNaeF9uqkjlCYeIhFSAzy
ylL7zQODOQkBVK98EHBxN0nc15LX2trGSYF1Fq9qGPq+EZ/XukTpXVW5fmOSuO9K
YCT6bkLbieYjVg7uNlEyC9nQGngAsTEYt6nAxJurPT7QGwL0d1JPW9xq6VBTeW3j
wzieqbHIrrPsyTiQKB2Tz95NUMYxP8EeVnaudyJd0oIdzsV+Fu4D/Ve2EVWD+7Gx
SoPnnEbDOZKNpUjfGu8hZMiuqiy0XBatnNjrkJpfwJFWNgS8PaNSBOrcYGJfCtg=
=FUbW
-----END PGP SIGNATURE-----
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2009-02-28 12:34 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2009-02-26 21:20 [gentoo-user-ru] Kernel Panic при высокой сетевой загрузке Konstantin Astafjev
2009-02-26 21:22 ` Алексей Белкин
2009-02-27 15:40 ` [gentoo-user-ru] " alexey conovalov
2009-02-28 12:12 ` Peter Volkov
2009-02-28 12:34 ` Dmitry Pisarev
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox