[gentoo-user] syslog-ng segfaults

[gentoo-user] syslog-ng segfaults

[Randy Barlow]

I saw that there were a couple of syslog-ng threads recently, but
neither of them mentioned any segfaulting which I am currently
experiencing. My log file (dmesg and /var/log/messages) is full of lines
like this:

syslog-ng[32015]: segfault at 44d8 ip 00007f4f3fa23c83 sp
00007fffb233b940 error 4 in libsyslog-ng.so.0.0.0 (deleted)[7f4f3f9eb000
+67000]

Has anybody else seen anything like that?

-- 
R

Re: [gentoo-user] syslog-ng segfaults

[Randy Barlow]

On Thu, 2013-07-18 at 18:41 -0400, Randy Barlow wrote:
> I saw that there were a couple of syslog-ng threads recently, but
> neither of them mentioned any segfaulting which I am currently
> experiencing. My log file (dmesg and /var/log/messages) is full of lines
> like this:
> 
> syslog-ng[32015]: segfault at 44d8 ip 00007f4f3fa23c83 sp
> 00007fffb233b940 error 4 in libsyslog-ng.so.0.0.0 (deleted)[7f4f3f9eb000
> +67000]

I forgot to mention my version:

$ equery list syslog-ng
 * Searching for syslog-ng ...
[IP-] [  ] app-admin/syslog-ng-3.4.2:0

-- 
R

Re: [gentoo-user] syslog-ng segfaults

[Bruce Hill]

On Thu, Jul 18, 2013 at 06:41:14PM -0400, Randy Barlow wrote:
> I saw that there were a couple of syslog-ng threads recently, but
> neither of them mentioned any segfaulting which I am currently
> experiencing. My log file (dmesg and /var/log/messages) is full of lines
> like this:
> 
> syslog-ng[32015]: segfault at 44d8 ip 00007f4f3fa23c83 sp
> 00007fffb233b940 error 4 in libsyslog-ng.so.0.0.0 (deleted)[7f4f3f9eb000
> +67000]
> 
> Has anybody else seen anything like that?

Try changing the version at the top of the config file to the present version.
-- 
Happy Penguin Computers               >')
126 Fenco Drive                       ( \
Tupelo, MS 38801                       ^^
support@happypenguincomputers.com
662-269-2706 662-205-6424
http://happypenguincomputers.com/

A: Because it messes up the order in which people normally read text.                                                                                                                                                          
Q: Why is top-posting such a bad thing?                                                                                                                                                                                        
A: Top-posting.                                                                                                                                                                                                                
Q: What is the most annoying thing in e-mail?

Don't top-post: http://en.wikipedia.org/wiki/Top_post#Top-posting

Re: [gentoo-user] syslog-ng segfaults

[Bruce Hill]

On Thu, Jul 18, 2013 at 06:41:14PM -0400, Randy Barlow wrote:
> I saw that there were a couple of syslog-ng threads recently, but
> neither of them mentioned any segfaulting which I am currently
> experiencing. My log file (dmesg and /var/log/messages) is full of lines
> like this:
> 
> syslog-ng[32015]: segfault at 44d8 ip 00007f4f3fa23c83 sp
> 00007fffb233b940 error 4 in libsyslog-ng.so.0.0.0 (deleted)[7f4f3f9eb000
> +67000]
> 
> Has anybody else seen anything like that?

If it wasn't clear before:

router ~ # diff /etc/syslog-ng/syslog-ng.conf /etc/syslog-ng/syslog-ng.conf~
1c1
< @version: 3.4
---
> @version: 3.3

Cheers,
Bruce
-- 
Happy Penguin Computers               >')
126 Fenco Drive                       ( \
Tupelo, MS 38801                       ^^
support@happypenguincomputers.com
662-269-2706 662-205-6424
http://happypenguincomputers.com/

A: Because it messes up the order in which people normally read text.                                                                                                                                                          
Q: Why is top-posting such a bad thing?                                                                                                                                                                                        
A: Top-posting.                                                                                                                                                                                                                
Q: What is the most annoying thing in e-mail?

Don't top-post: http://en.wikipedia.org/wiki/Top_post#Top-posting

Re: [gentoo-user] syslog-ng segfaults

[Randy Barlow]

On Thu, 2013-07-18 at 17:49 -0500, Bruce Hill wrote:
> Try changing the version at the top of the config file to the present version.

It's at 3.4, and I have the gentoo default config. Thanks for the
suggestion!

-- 
R

Re: [gentoo-user] syslog-ng segfaults

[Randy Barlow]

On Thu, 2013-07-18 at 18:41 -0400, Randy Barlow wrote:
> Has anybody else seen anything like that?

I'll add in that this seems to be happening to all three of my x86_64
hosts. It does not appear to happen on my x86 host.

-- 
R

Re: [gentoo-user] [Solved] syslog-ng segfaults

[Randy Barlow]

Randy Barlow wrote:
> I'll add in that this seems to be happening to all three of my x86_64
> hosts. It does not appear to happen on my x86 host.

I rebooted one of the hosts that was experiencing this issue, and it did 
not return afterwards. This surprises me, as restarting the service did 
not solve the problem. I don't really know what was the cause, but it 
does appear to have gone away. I know that I have not rebooted since I 
upgraded to that particular version, so perhaps there is something odd 
that requires more than just a restart of syslog-ng.

Thanks to Bruce for help!

-- 
R

Re: [gentoo-user] [Solved] syslog-ng segfaults

[Dan Johansson]

[-- Attachment #1.1: Type: text/plain, Size: 1201 bytes --]

On 19.07.2013 02:53, Randy Barlow wrote:
> Randy Barlow wrote:
>> I'll add in that this seems to be happening to all three of my x86_64
>> hosts. It does not appear to happen on my x86 host.
> 
> I rebooted one of the hosts that was experiencing this issue, and it did
> not return afterwards. This surprises me, as restarting the service did
> not solve the problem. I don't really know what was the cause, but it
> does appear to have gone away. I know that I have not rebooted since I
> upgraded to that particular version, so perhaps there is something odd
> that requires more than just a restart of syslog-ng.

Question: Is this a "physical" host or is it a virtual host running
under qemu?
Reason for my question ist that recently the CPU-Id presented from quemu
in the guest has changed and if you have CFLAGS="-march=native" then
some newly compiles SW could fail. One way to solve this is to change
the CFLAGS to "-mtune=native" and recompile the affected SW (or world).

-- 
Dan Johansson, <http://www.dmj.nu>
***************************************************
This message is printed on 100% recycled electrons!
***************************************************

[-- Attachment #1.2: 0x2FB894AD.asc --]
[-- Type: application/pgp-keys, Size: 3477 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: [gentoo-user] syslog-ng segfaults

[Алексей Мишустин]

2013/7/19 Randy Barlow <randy@electronsweatshop.com>:
> I saw that there were a couple of syslog-ng threads recently, but
> neither of them mentioned any segfaulting which I am currently
> experiencing. My log file (dmesg and /var/log/messages) is full of lines
> like this:
>
> syslog-ng[32015]: segfault at 44d8 ip 00007f4f3fa23c83 sp
> 00007fffb233b940 error 4 in libsyslog-ng.so.0.0.0 (deleted)[7f4f3f9eb000
> +67000]
>
> Has anybody else seen anything like that?

Yes, I had the same issue last Sunday.

At x86_64 XEN-based VPS. It's gone after the reboot too. What is
CFLAGS there, I don't remember. I'll check it.

-- 
Regards,
Alex

Re: [gentoo-user] syslog-ng segfaults

[Adam Carter]

[-- Attachment #1: Type: text/plain, Size: 859 bytes --]

>
> > syslog-ng[32015]: segfault at 44d8 ip 00007f4f3fa23c83 sp
> > 00007fffb233b940 error 4 in libsyslog-ng.so.0.0.0 (deleted)[7f4f3f9eb000
> > +67000]
> >
> > Has anybody else seen anything like that?
>

Perhaps you forgot to restart syslog-ng after the update?

# ldd /usr/sbin/syslog-ng | grep libsyslog-ng
    libsyslog-ng-3.4.2.so =>
/usr/lib64/libsyslog-ng-3.4.2.so(0x00007fe7dfe5b000)

If the library was called libsyslog-ng.so.0.0.0 in the previous version,
and if syslog-ng tried to reload it, it would fail after the update which
could cause the segfault.

So, restarting syslog-ng should be all that's required to fix it - reboot
is overkill.

Also, any errors in the config file should not cause segfault. If it does,
that indicates a bug in the input validation/config parsing code. Errors in
the config should result in a graceful termination.

[-- Attachment #2: Type: text/html, Size: 1362 bytes --]

Re: [gentoo-user] syslog-ng segfaults

[Alexey Mishustin]

2013/7/19 Adam Carter <adamcarter3@gmail.com>:
>> > syslog-ng[32015]: segfault at 44d8 ip 00007f4f3fa23c83 sp
>> > 00007fffb233b940 error 4 in libsyslog-ng.so.0.0.0 (deleted)[7f4f3f9eb000
>> > +67000]
>> >
>> > Has anybody else seen anything like that?
>
>
> Perhaps you forgot to restart syslog-ng after the update?
>
> # ldd /usr/sbin/syslog-ng | grep libsyslog-ng
>     libsyslog-ng-3.4.2.so => /usr/lib64/libsyslog-ng-3.4.2.so
> (0x00007fe7dfe5b000)
>
> If the library was called libsyslog-ng.so.0.0.0 in the previous version, and
> if syslog-ng tried to reload it, it would fail after the update which could
> cause the segfault.
>
> So, restarting syslog-ng should be all that's required to fix it - reboot is
> overkill.

As for me, first I updated syslog-ng, then I issued
'/etc/init.d/syslog-ng reload' (by mistake, instead of 'restart'), and
then 'restart' as I should. Then, just when syslog-ng was restarting,
the segfault happened.

> Also, any errors in the config file should not cause segfault. If it does,
> that indicates a bug in the input validation/config parsing code. Errors in
> the config should result in a graceful termination.

-- 
Regards,
Alex

Re: [gentoo-user] [Solved] syslog-ng segfaults

[Randy Barlow]

Dan Johansson wrote:
> Question: Is this a "physical" host or is it a virtual host running
> under qemu?
> Reason for my question ist that recently the CPU-Id presented from quemu
> in the guest has changed and if you have CFLAGS="-march=native" then
> some newly compiles SW could fail. One way to solve this is to change
> the CFLAGS to "-mtune=native" and recompile the affected SW (or world).

Hi Dan!

Out of the three machines that were experiencing this issue, one was 
physical and two were virtual. The one I rebooted that solved the 
problem was virtual. I haven't yet rebooted the other two.

Thanks for the tip, I believe I am using -march=native, so perhaps I 
should address that issue as well. Do you know which version of qemu 
this change was introduced in?

-- 
R

Re: [gentoo-user] [Solved] syslog-ng segfaults

[Alexey Mishustin]

2013/7/19 Dan Johansson <Dan.Johansson@dmj.nu>:

> Question: Is this a "physical" host or is it a virtual host running
> under qemu?
> Reason for my question ist that recently the CPU-Id presented from quemu
> in the guest has changed and if you have CFLAGS="-march=native" then
> some newly compiles SW could fail. One way to solve this is to change
> the CFLAGS to "-mtune=native" and recompile the affected SW (or world).

My virtual guest is compiled with "-mtune=generic". It's an universal
solution for all changes in virtual hosts, isn't it?

-- 
Regards,
Alex

Re: [gentoo-user] syslog-ng segfaults

[Randy Barlow]

Alexey Mishustin wrote:
>> So, restarting syslog-ng should be all that's required to fix it - reboot is
>> >overkill.

> As for me, first I updated syslog-ng, then I issued
> '/etc/init.d/syslog-ng reload' (by mistake, instead of 'restart'), and
> then 'restart' as I should. Then, just when syslog-ng was restarting,
> the segfault happened.

I also noted that restarting syslog didn't seem to solve the problem. I 
do think Adam's reasoning makes sense, but there must be something else 
that needed to be restarted as well.

-- 
R

Re: [gentoo-user] syslog-ng segfaults

[kwkhui]

[-- Attachment #1: Type: text/plain, Size: 919 bytes --]

On Fri, 19 Jul 2013 17:03:36 -0400
Randy Barlow <randy@electronsweatshop.com> wrote:

> Alexey Mishustin wrote:
> >> So, restarting syslog-ng should be all that's required to fix it -
> >> reboot is
> >> >overkill.
> 
> > As for me, first I updated syslog-ng, then I issued
> > '/etc/init.d/syslog-ng reload' (by mistake, instead of 'restart'),
> > and then 'restart' as I should. Then, just when syslog-ng was
> > restarting, the segfault happened.
> 
> I also noted that restarting syslog didn't seem to solve the problem.
> I do think Adam's reasoning makes sense, but there must be something
> else that needed to be restarted as well.
> 

Same behaviour here.  In my case with an "lsof | grep libsyslog-ng" I
see in the physical host hp-systray from hplip was still
using the old libsyslog-ng.so, so killing that and a restart of
syslog-ng service stops the segfault lines.  YMMV,

Kerwin.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-user] syslog-ng segfaults

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 482 bytes --]

On Sat, 20 Jul 2013 20:27:51 +0800, kwkhui@hkbn.net wrote:

> Same behaviour here.  In my case with an "lsof | grep libsyslog-ng" I
> see in the physical host hp-systray from hplip was still
> using the old libsyslog-ng.so, so killing that and a restart of
> syslog-ng service stops the segfault lines.  YMMV,

Try app-admin/checkrestart, I generally run this after updating any
daemons or libraries.


-- 
Neil Bothwick

"RAM DISK is NOT an installation procedure!"

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] syslog-ng segfaults

[Randy Barlow]

Neil Bothwick wrote:
> Try app-admin/checkrestart, I generally run this after updating any
> daemons or libraries.

This sounds very helpful, thanks for the suggestion Neil!

-- 
R