[gentoo-user] xfs recovery + kernel panic

[gentoo-user] xfs recovery + kernel panic

[Daniel Iliev]

Hello, guys!

Situation:
1) Unclean shutdown
2) Boot
3) Kernel tries to mount /
4) xfs recovery starts
5) In the same time (before recovery is finished) kernel tries to start
"init", fails and panics.

If I take the HDD and mount it (not as "/") on another machine, there is
no problem. The recovery finishes OK and afterward xfs_rapair doesn't
find any errors. The next unclean shutdown everything repeats.

Is there a way to make the kernel wait for the recovery to finish before
trying to start init or any other workaround?.

Thank you in advance!

-- 
Best regards,
Daniel

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] xfs recovery + kernel panic

[David Miller]

[-- Attachment #1: Type: text/plain, Size: 888 bytes --]

I would try booting off a live cd that has the xfs utils on it.  That way
you can get your drive sorted out without worrying about your OS getting in
the way.
--
David

On 7/7/06, Daniel Iliev <danny@ilievnet.com> wrote:
>
> Hello, guys!
>
> Situation:
> 1) Unclean shutdown
> 2) Boot
> 3) Kernel tries to mount /
> 4) xfs recovery starts
> 5) In the same time (before recovery is finished) kernel tries to start
> "init", fails and panics.
>
> If I take the HDD and mount it (not as "/") on another machine, there is
> no problem. The recovery finishes OK and afterward xfs_rapair doesn't
> find any errors. The next unclean shutdown everything repeats.
>
> Is there a way to make the kernel wait for the recovery to finish before
> trying to start init or any other workaround?.
>
> Thank you in advance!
>
> --
> Best regards,
> Daniel
>
> --
> gentoo-user@gentoo.org mailing list
>
>

[-- Attachment #2: Type: text/html, Size: 1271 bytes --]

Re: [gentoo-user] xfs recovery + kernel panic

[Daniel Iliev]

David Miller wrote:
> I would try booting off a live cd that has the xfs utils on it.  That way
> you can get your drive sorted out without worrying about your OS getting in
> the way.

Interesting approach. This is exactly a workaround. If there is no
straight solution I may finally find myself forced to go this way.

Thank, you for the idea!

-- 
Best regards,
Daniel

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] xfs recovery + kernel panic

[Richard Fish]

On 7/7/06, Daniel Iliev <danny@ilievnet.com> wrote:
> Hello, guys!
>
> Situation:
> 1) Unclean shutdown
> 2) Boot
> 3) Kernel tries to mount /
> 4) xfs recovery starts
> 5) In the same time (before recovery is finished) kernel tries to start
> "init", fails and panics.

Hmm, I don't see this with my system.  I'm pretty sure that the
mount() system call should not be returning before the recovery is
finished.  What kernel and baselayout versions are you using?

-Richard
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] xfs recovery + kernel panic

[Daniel Iliev]

Richard Fish wrote:

> 
> Hmm, I don't see this with my system.  I'm pretty sure that the
> mount() system call should not be returning before the recovery is
> finished.  What kernel and baselayout versions are you using?
> 
> -Richard

===
kernel: 2.6.16-ck12 #3 PREEMPT  (ck-sources)
sys-apps/baselayout-1.11.15-r3  USE="static unicode -bootstrap -build"
===

I use the same kernel on my desktop and never had this problem.However I
have to say there's a big difference between the desktop and the
problematic PC. While the PC has a plain setup - only 1 hdd with only 1
partition (hda1), the desktop has its root on a 2-disk software raid0.
So the desktop mounts /dev/md0 preliminary, then xfs recovery takes
place if needed, and "switchroot /sysroot" is made at the end.

Hmmmm...witting this email I think an idea occurred to me...:)

What stops me make the same way of booting on the problematic PC? Well,
this would be a workaround again while I prefer a straight solution.

-- 
Best regards,
Daniel

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] xfs recovery + kernel panic

[Richard Fish]

On 7/7/06, Daniel Iliev <danny@ilievnet.com> wrote:
> I use the same kernel on my desktop and never had this problem.However I
> have to say there's a big difference between the desktop and the
> problematic PC. While the PC has a plain setup - only 1 hdd with only 1
> partition (hda1), the desktop has its root on a 2-disk software raid0.
> So the desktop mounts /dev/md0 preliminary, then xfs recovery takes
> place if needed, and "switchroot /sysroot" is made at the end.

Ahh, this could very likely be the difference.  I use dm-crypt on all
my filesystems, so a very similar situation: my initramfs first mounts
my root filesystem, and then chroot's into it.  The mount call from
userspace is likely the difference.

I would suggest to try and duplicate it with an initrd/initramfs, and
if it goes away, file a bug at bugs.kernel.org.  They might refuse a
bug report with -ck sources, so you might have to report it with the
maintainer of the -ck sources, or switch to vanilla for the testing.
But either way it seems like a kernel bug to me...

-Richard
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] [OT] xfs recovery + kernel panic

[Daniel Iliev]

Richard Fish wrote:

--snip
> I use dm-crypt on all
> my filesystems, so a very similar situation: my initramfs first mounts
> my root filesystem, and then chroot's into it.  The mount call from
> userspace is likely the difference.

A way out of the topic, but its a question that I want to ask.
What is the performance hit of using encrypted file system?
I hate laptops, but you never know ;-)

> I would suggest to try and duplicate it with an initrd/initramfs, and
> if it goes away, file a bug at bugs.kernel.org.  They might refuse a
> bug report with -ck sources, so you might have to report it with the
> maintainer of the -ck sources, or switch to vanilla for the testing.
> But either way it seems like a kernel bug to me...
> 
> -Richard

Yep, I will try initramfs on the problematic PC tomorrow.

Bug-report...Well I'm very confused here. Isn't it Gentoo the right
place to file
a bug-report at? After all these sources get patched with gentoo
patches. I haven't
directly downloaded the sources from the maintainer or used vanilla with
ck-patches.
On the other hand gentoo people have masked these sources with "~" so they
could also refuse to take the report (unlikely,but..). Additionally I
remember I've
read somewhere that Gentoo maintainer(s) are not happy that people "jump
over"
them and send bug-reports directly to the mainstream developers who of
course
refuse to accept the report because Gentoo has patched their sources and
so on...

I'm really not sure what is the proper thing to do about this.

-- 
Best regards,
Daniel

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] [OT] xfs recovery + kernel panic

[Richard Fish]

On 7/7/06, Daniel Iliev <danny@ilievnet.com> wrote:
> A way out of the topic, but its a question that I want to ask.
> What is the performance hit of using encrypted file system?
> I hate laptops, but you never know ;-)

Not so bad.  I really don't notice any real performance problem using
it, certainly not much worse than a typical laptop HD:

carcharias rjf # hdparm -Tt /dev/sda /dev/sys/swap

/dev/sda:
 Timing cached reads:   4096 MB in  1.99 seconds = 2053.20 MB/sec
 Timing buffered disk reads:  142 MB in  3.01 seconds =  47.17 MB/sec

/dev/sys/swap:
 Timing cached reads:   4848 MB in  1.99 seconds = 2432.13 MB/sec
HDIO_DRIVE_CMD(null) (wait for flush complete) failed: Inappropriate
ioctl for device
 Timing buffered disk reads:  138 MB in  3.01 seconds =  45.79 MB/sec
HDIO_DRIVE_CMD(null) (wait for flush complete) failed: Inappropriate
ioctl for device

The biggest hit is in CPU, and having a dual-core system helps *a lot*
here.  Running  a "dd if=/dev/sys/root of=/dev/null bs=64k" while
running top at the same time shows that kcryptd consumes about 75% of
the processor time on a *single* CPU.

BTW, I also use dm-crypt on my AMD X2 desktop at home.  There I have a
raid0 array which can read at almost 130MB/sec total bandwidth.  If
memory serves, using dm-crypt there cut my read bandwidth down to
about 105MB/s, and writes to 90MB/s.  The issue there seems mostly
that dm-crypt is a single-thread, even when encrypting multiple
devices, so cannot really take advantage of my dual-core processor in
that system.

I think loop-AES gives higher performance on that box due to having
one thread per encrypted device, but it isn't enough for me to worry
about.

Side note: I think it is appalling that government and business
laptops are generally so insecure.  Every week brings news of yet
another laptop theft that contained sensitive data for hundreds of
thousands to millions of people, and oh, btw, we didn't encrypt it
"because it's hard".  Maybe I'm paranoid, but I like knowing that if
my house is ever robbed and my computer stolen, I don't have to worry
that the crooks have all my financial records!

> Bug-report...Well I'm very confused here. Isn't it Gentoo the right
> place to file
> a bug-report at? After all these sources get patched with gentoo
> patches. I haven't

Well you can certainly file on bugs.gentoo.org, and let the gentoo
devs work with upstream to find a fix.  And in many cases that is the
right thing to do, so I'll leave it up to you.

My view is that Gentoo devs are all volunteers, and very busy, so if I
come across an issue that is clearly a problem with $upstream, and not
with any gentoo patches or compile options, and I feel confident that
I can communicate properly with $upstream, then I will file the bug
there instead.  The fix can then get filtered down to Gentoo.  In
fact, if it might be awhile before the fix filters down, I would file
a bug both places, with the gentoo one being "please apply patch
referenced in http://bugs.upstream.org/#12345".

> On the other hand gentoo people have masked these sources with "~" so they
> could also refuse to take the report (unlikely,but..).

Yeah, unlikely, considering that ~arch is supposed to be for testing!!

> them and send bug-reports directly to the mainstream developers who of
> course refuse to accept the report because Gentoo has patched their sources

Well, like I said, for me this depends on the nature of the bug.  So
far I've never had $upstream reject a bug report because I was using
Gentoo.  Well, ok, I have, but that was a commercial software company
that just said "try {RedHat,SuSe}".

-Richard
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] [OT] xfs recovery + kernel panic

[Daniel Iliev]

Richard Fish wrote:
> On 7/7/06, Daniel Iliev <danny@ilievnet.com> wrote:
>> A way out of the topic, but its a question that I want to ask.
>> What is the performance hit of using encrypted file system?
>> I hate laptops, but you never know ;-)
> 
> Not so bad.  I really don't notice any real performance problem using
> it, certainly not much worse than a typical laptop HD:
> 
> carcharias rjf # hdparm -Tt /dev/sda /dev/sys/swap
> 
> /dev/sda:
> Timing cached reads:   4096 MB in  1.99 seconds = 2053.20 MB/sec
> Timing buffered disk reads:  142 MB in  3.01 seconds =  47.17 MB/sec
> 
> /dev/sys/swap:
> Timing cached reads:   4848 MB in  1.99 seconds = 2432.13 MB/sec
> HDIO_DRIVE_CMD(null) (wait for flush complete) failed: Inappropriate
> ioctl for device
> Timing buffered disk reads:  138 MB in  3.01 seconds =  45.79 MB/sec
> HDIO_DRIVE_CMD(null) (wait for flush complete) failed: Inappropriate
> ioctl for device
> 
> The biggest hit is in CPU, and having a dual-core system helps *a lot*
> here.  Running  a "dd if=/dev/sys/root of=/dev/null bs=64k" while
> running top at the same time shows that kcryptd consumes about 75% of
> the processor time on a *single* CPU.
> 
> BTW, I also use dm-crypt on my AMD X2 desktop at home.  There I have a
> raid0 array which can read at almost 130MB/sec total bandwidth.  If
> memory serves, using dm-crypt there cut my read bandwidth down to
> about 105MB/s, and writes to 90MB/s.  The issue there seems mostly
> that dm-crypt is a single-thread, even when encrypting multiple
> devices, so cannot really take advantage of my dual-core processor in
> that system.
> 
> I think loop-AES gives higher performance on that box due to having
> one thread per encrypted device, but it isn't enough for me to worry
> about.
> 
> Side note: I think it is appalling that government and business
> laptops are generally so insecure.  Every week brings news of yet
> another laptop theft that contained sensitive data for hundreds of
> thousands to millions of people, and oh, btw, we didn't encrypt it
> "because it's hard".  Maybe I'm paranoid, but I like knowing that if
> my house is ever robbed and my computer stolen, I don't have to worry
> that the crooks have all my financial records!
> 
>> Bug-report...Well I'm very confused here. Isn't it Gentoo the right
>> place to file
>> a bug-report at? After all these sources get patched with gentoo
>> patches. I haven't
> 
> Well you can certainly file on bugs.gentoo.org, and let the gentoo
> devs work with upstream to find a fix.  And in many cases that is the
> right thing to do, so I'll leave it up to you.
> 
> My view is that Gentoo devs are all volunteers, and very busy, so if I
> come across an issue that is clearly a problem with $upstream, and not
> with any gentoo patches or compile options, and I feel confident that
> I can communicate properly with $upstream, then I will file the bug
> there instead.  The fix can then get filtered down to Gentoo.  In
> fact, if it might be awhile before the fix filters down, I would file
> a bug both places, with the gentoo one being "please apply patch
> referenced in http://bugs.upstream.org/#12345".
> 
>> On the other hand gentoo people have masked these sources with "~" so
>> they
>> could also refuse to take the report (unlikely,but..).
> 
> Yeah, unlikely, considering that ~arch is supposed to be for testing!!
> 
>> them and send bug-reports directly to the mainstream developers who of
>> course refuse to accept the report because Gentoo has patched their
>> sources
> 
> Well, like I said, for me this depends on the nature of the bug.  So
> far I've never had $upstream reject a bug report because I was using
> Gentoo.  Well, ok, I have, but that was a commercial software company
> that just said "try {RedHat,SuSe}".
> 
> -Richard

Richard, thank you very much for this detailed answer.

I'm very impressed by the results of your dual core CPU. May be the time
has really come for me to upgrade the box.
On my AthlonXP 1700+ the results show that the CPU-intensive *cached
reads* are far beyond slow compared to yours. I had *buffered disk read*
results about 104MB/s only when the disks were new and empty. Now the
file system is about 90% full and its normal to get decreased speeds.
Here are my results for comparison.

 hdparm -Tt /dev/sd{a,b} /dev/md{0,1}

/dev/sda:
 Timing cached reads:   1796 MB in  2.00 seconds = 897.19 MB/sec
 Timing buffered disk reads:  140 MB in  3.04 seconds =  46.05 MB/sec

/dev/sdb:
 Timing cached reads:   1740 MB in  2.00 seconds = 868.99 MB/sec
 Timing buffered disk reads:  160 MB in  3.10 seconds =  51.58 MB/sec

/dev/md0:
 Timing cached reads:   1756 MB in  2.00 seconds = 877.73 MB/sec
 Timing buffered disk reads:  298 MB in  3.01 seconds =  99.13 MB/sec

/dev/md1:
 Timing cached reads:   1724 MB in  2.00 seconds = 861.35 MB/sec
 Timing buffered disk reads:  272 MB in  3.01 seconds =  90.30 MB/sec

The conclusion is that I am not going to encrypt the disks. Not at least
until I upgrade ;-)

---------

Laptops...I hate them, I never had one and I think I'll never have. But
you know "never say never" :)
One thing is for sure - I absolutely agree with you that encrypting the
disks of any laptop is "a must".

--------

Bug-report. Well, I think I'm going to send one. As you suggested - to
ck-sources developer(s) and one to gentoo maintainers with reference to
the upstream report.

Thank you again.

-- 
Best regards,
Daniel

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] [SOLVED] xfs recovery + kernel panic

[Daniel Iliev]

The workaround I applied is to put an initfs image into kernel.
The init script there "pre-mounts" the root file system and afterwards
switches root. Nothing else.
I tested twice by booting the machine and intentionally turning its
power off.
When stared again xfs recovery starts, finishes, then the switch_root
takes place followed by the real init routine. In other words everything
seems to work as supposed to.
If the problem doesn't appear again after several unclean shutdowns I am
going to send a bug report.

So this should hopefully be the last message in this thread.

Thanks everyone!

-- 
Best regards,
Daniel

-- 
gentoo-user@gentoo.org mailing list