[gentoo-user] chrony-3.3 hangs at boot

[gentoo-user] chrony-3.3 hangs at boot

[Mick]

[-- Attachment #1: Type: text/plain, Size: 1030 bytes --]

I just noticed today chronyd hangs during boot for a minute and a half.  The 
logs do not reveal anything amiss.  I suspect it waits for a network 
connection, which is not yet up when chronyd launches.

Sep  3 17:38:26 asus syslog-ng[1858]: syslog-ng starting up; version='3.13.2'
Sep  3 17:38:26 asus chronyd[1930]: chronyd version 3.3 starting (+CMDMON +NTP 
+REFCLOCK +RTC -PRIVDROP +SCFILTER -SIGND +ASYNCDNS -SECHASH +IPV6 -DEBUG)
Sep  3 17:38:26 asus chronyd[1930]: Frequency -13.776 +/- 1.023 ppm read from 
/var/lib/chrony/drift
Sep  3 17:38:26 asus acpid[1894]: starting up with netlink and the input layer
Sep  3 17:38:26 asus acpid[1894]: 1 rule loaded
Sep  3 17:38:26 asus acpid[1894]: waiting for events: event logging is off
Sep  3 17:40:57 asus kernel: random: crng init done
Sep  3 17:40:57 asus kernel: random: 3 urandom warning(s) missed due to 
ratelimiting
Sep  3 17:40:57 asus chronyd[1930]: Loaded dump file for 10.10.10.1


Have you noticed the same?

Is there a fix/workaround for this?

-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] chrony-3.3 hangs at boot

[James Stevenson]

I have also experienced this since upgrading chrony.

James

Re: [gentoo-user] chrony-3.3 hangs at boot

[Dale]

James Stevenson wrote:
> I have also experienced this since upgrading chrony.
>
> James
>
>


I finally synced today but have not updated yet.  I did notice the USE
flag seccomp is changed/added for chrony tho.  Could that be the cause? 
Maybe it needs adjustment or some additional configuration before running. 

Hope that helps.

Dale

:-)  :-) 

[gentoo-user] Re: chrony-3.3 hangs at boot

[Holger Hoffstätte]

On Mon, 03 Sep 2018 17:48:53 +0100, Mick wrote:

> I just noticed today chronyd hangs during boot for a minute and a half.  The 
> logs do not reveal anything amiss.  I suspect it waits for a network 
> connection, which is not yet up when chronyd launches.
> 
> Sep  3 17:38:26 asus syslog-ng[1858]: syslog-ng starting up; version='3.13.2'
> Sep  3 17:38:26 asus chronyd[1930]: chronyd version 3.3 starting (+CMDMON +NTP 
> +REFCLOCK +RTC -PRIVDROP +SCFILTER -SIGND +ASYNCDNS -SECHASH +IPV6 -DEBUG)
> Sep  3 17:38:26 asus chronyd[1930]: Frequency -13.776 +/- 1.023 ppm read from 
> /var/lib/chrony/drift
> Sep  3 17:38:26 asus acpid[1894]: starting up with netlink and the input layer
> Sep  3 17:38:26 asus acpid[1894]: 1 rule loaded
> Sep  3 17:38:26 asus acpid[1894]: waiting for events: event logging is off
> Sep  3 17:40:57 asus kernel: random: crng init done
> Sep  3 17:40:57 asus kernel: random: 3 urandom warning(s) missed due to 
> ratelimiting
> Sep  3 17:40:57 asus chronyd[1930]: Loaded dump file for 10.10.10.1
> 
> 
> Have you noticed the same?
> 
> Is there a fix/workaround for this?

Yes and yes; countless people have been hit by this, I was probably the first
to notice/debug it. It happens due to changes in Linux' random number
generator, which is now slowly making it into older kernels as well.

If you're interested in the gory details you can read about them
in the thread "Chrony vs. Linux RNG" at [1] or just update to chrony-3.4rc1
which contains the "fix". Despite being an rc1 it works fine as usual,
so short of applying the patch to your local build this is the easiest
option.

cheers,
Holger

[1] https://listengine.tuxfamily.org/chrony.tuxfamily.org/chrony-users/2018/04/threads.html#00036

Re: [gentoo-user] Re: chrony-3.3 hangs at boot

[Mick]

[-- Attachment #1: Type: text/plain, Size: 2486 bytes --]

On Monday, 3 September 2018 19:10:52 BST Holger Hoffstätte wrote:
> On Mon, 03 Sep 2018 17:48:53 +0100, Mick wrote:
> > I just noticed today chronyd hangs during boot for a minute and a half. 
> > The logs do not reveal anything amiss.  I suspect it waits for a network
> > connection, which is not yet up when chronyd launches.
> > 
> > Sep  3 17:38:26 asus syslog-ng[1858]: syslog-ng starting up;
> > version='3.13.2' Sep  3 17:38:26 asus chronyd[1930]: chronyd version 3.3
> > starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP +SCFILTER -SIGND
> > +ASYNCDNS -SECHASH +IPV6 -DEBUG) Sep  3 17:38:26 asus chronyd[1930]:
> > Frequency -13.776 +/- 1.023 ppm read from /var/lib/chrony/drift
> > Sep  3 17:38:26 asus acpid[1894]: starting up with netlink and the input
> > layer Sep  3 17:38:26 asus acpid[1894]: 1 rule loaded
> > Sep  3 17:38:26 asus acpid[1894]: waiting for events: event logging is off
> > Sep  3 17:40:57 asus kernel: random: crng init done
> > Sep  3 17:40:57 asus kernel: random: 3 urandom warning(s) missed due to
> > ratelimiting
> > Sep  3 17:40:57 asus chronyd[1930]: Loaded dump file for 10.10.10.1
> > 
> > 
> > Have you noticed the same?
> > 
> > Is there a fix/workaround for this?
> 
> Yes and yes; countless people have been hit by this, I was probably the
> first to notice/debug it. It happens due to changes in Linux' random number
> generator, which is now slowly making it into older kernels as well.
> 
> If you're interested in the gory details you can read about them
> in the thread "Chrony vs. Linux RNG" at [1] or just update to chrony-3.4rc1
> which contains the "fix". Despite being an rc1 it works fine as usual,
> so short of applying the patch to your local build this is the easiest
> option.
> 
> cheers,
> Holger
> 
> [1]
> https://listengine.tuxfamily.org/chrony.tuxfamily.org/chrony-users/2018/04/
> threads.html#00036

Thanks for this Holger, I had noticed in the logs a message just before 
chronyd started the kernel reporting "3 urandom warning(s) missed due to 
ratelimiting" but as this has been happening since the upgrade to 4.14.65-
gentoo I didn't think it was kernel related.

Anyway, I can confirm 3.4_pre1 available in the tree starts up fine with no 
delays.

However, while shutting down I noticed:

* Stopping chronyd ...
* Error: chronyd failed to stop

This is repeatable if I try to stop it manually.  Version 3.3 didn't have this 
problem.

-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[gentoo-user] Re: chrony-3.3 hangs at boot

[Holger Hoffstätte]

On Mon, 03 Sep 2018 22:23:18 +0100, Mick wrote:

> On Monday, 3 September 2018 19:10:52 BST Holger Hoffstätte wrote:
>> On Mon, 03 Sep 2018 17:48:53 +0100, Mick wrote:
>> > I just noticed today chronyd hangs during boot for a minute and a half. 
>> > The logs do not reveal anything amiss.  I suspect it waits for a network
>> > connection, which is not yet up when chronyd launches.
>> > 
>> > Sep  3 17:38:26 asus syslog-ng[1858]: syslog-ng starting up;
>> > version='3.13.2' Sep  3 17:38:26 asus chronyd[1930]: chronyd version 3.3
>> > starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP +SCFILTER -SIGND
>> > +ASYNCDNS -SECHASH +IPV6 -DEBUG) Sep  3 17:38:26 asus chronyd[1930]:
>> > Frequency -13.776 +/- 1.023 ppm read from /var/lib/chrony/drift
>> > Sep  3 17:38:26 asus acpid[1894]: starting up with netlink and the input
>> > layer Sep  3 17:38:26 asus acpid[1894]: 1 rule loaded
>> > Sep  3 17:38:26 asus acpid[1894]: waiting for events: event logging is off
>> > Sep  3 17:40:57 asus kernel: random: crng init done
>> > Sep  3 17:40:57 asus kernel: random: 3 urandom warning(s) missed due to
>> > ratelimiting
>> > Sep  3 17:40:57 asus chronyd[1930]: Loaded dump file for 10.10.10.1
>> > 
>> > 
>> > Have you noticed the same?
>> > 
>> > Is there a fix/workaround for this?
>> 
>> Yes and yes; countless people have been hit by this, I was probably the
>> first to notice/debug it. It happens due to changes in Linux' random number
>> generator, which is now slowly making it into older kernels as well.
>> 
>> If you're interested in the gory details you can read about them
>> in the thread "Chrony vs. Linux RNG" at [1] or just update to chrony-3.4rc1
>> which contains the "fix". Despite being an rc1 it works fine as usual,
>> so short of applying the patch to your local build this is the easiest
>> option.
>> 
>> cheers,
>> Holger
>> 
>> [1]
>> https://listengine.tuxfamily.org/chrony.tuxfamily.org/chrony-users/2018/04/
>> threads.html#00036
> 
> Thanks for this Holger, I had noticed in the logs a message just before 
> chronyd started the kernel reporting "3 urandom warning(s) missed due to 
> ratelimiting" but as this has been happening since the upgrade to 4.14.65-
> gentoo I didn't think it was kernel related.
> 
> Anyway, I can confirm 3.4_pre1 available in the tree starts up fine with no 
> delays.
> 
> However, while shutting down I noticed:
> 
> * Stopping chronyd ...
> * Error: chronyd failed to stop
> 
> This is repeatable if I try to stop it manually.  Version 3.3 didn't have this 
> problem.

Ah yes, that's from upgrading. Apologies for the misleading advice,
I fixed that locally and forgot. :}

1) Edit /etc/init.d/chronyd and change occurrences of /run/chronyd.pid to
/run/chrony/chronyd.pid (i.e. add the subdirectory).

2) file a bug so that the init file gets fixed in portage as well. :)

cheers
Holger

Re: [gentoo-user] Re: chrony-3.3 hangs at boot

[Mick]

[-- Attachment #1: Type: text/plain, Size: 798 bytes --]

On Monday, 3 September 2018 22:47:50 BST Holger Hoffstätte wrote:
> On Mon, 03 Sep 2018 22:23:18 +0100, Mick wrote:

> > This is repeatable if I try to stop it manually.  Version 3.3 didn't have
> > this problem.
> 
> Ah yes, that's from upgrading. Apologies for the misleading advice,
> I fixed that locally and forgot. :}
> 
> 1) Edit /etc/init.d/chronyd and change occurrences of /run/chronyd.pid to
> /run/chrony/chronyd.pid (i.e. add the subdirectory).
> 
> 2) file a bug so that the init file gets fixed in portage as well. :)
> 
> cheers
> Holger

Done!

https://bugs.gentoo.org/665210
https://bugs.gentoo.org/665212

Interestingly, two Intel systems running 3.3 are booting normally.  It was 
only an AMD system here which was hanging at boot.

-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] chrony-3.3 hangs at boot

[Stroller]

[-- Attachment #1: Type: text/plain, Size: 870 bytes --]



> On 3 Sep 2018, at 19:10, Holger Hoffstätte <holger@applied-asynchrony.com> wrote:
> 
> On Mon, 03 Sep 2018 17:48:53 +0100, Mick wrote:
> 
>> I just noticed today chronyd hangs during boot for a minute and a half.  The 
>> logs do not reveal anything amiss.  I suspect it waits for a network 
>> connection, which is not yet up when chronyd launches.
>> … 
> 
> Yes and yes; countless people have been hit by this, I was probably the first
> to notice/debug it. It happens due to changes in Linux' random number
> generator, which is now slowly making it into older kernels as well.

Perhaps I'm experiencing this because I'm using a Linode VM - it seems to hang indefinitely, not just for 90 seconds - but it looks a bit like the devs have passed the report upstream and left the affected versions in the tree to bite stable users.

Stroller.


[-- Attachment #2: Type: text/html, Size: 5552 bytes --]

Re: [gentoo-user] chrony-3.3 hangs at boot

[Peter Humphrey]

On Friday, 2 November 2018 16:38:14 GMT Stroller wrote:
> > On 3 Sep 2018, at 19:10, Holger Hoffstätte <holger@applied-asynchrony.com>
> > wrote:> 
> > On Mon, 03 Sep 2018 17:48:53 +0100, Mick wrote:
> >> I just noticed today chronyd hangs during boot for a minute and a half. 
> >> The logs do not reveal anything amiss.  I suspect it waits for a network
> >> connection, which is not yet up when chronyd launches.
> >> …
> > 
> > Yes and yes; countless people have been hit by this, I was probably the
> > first to notice/debug it. It happens due to changes in Linux' random
> > number generator, which is now slowly making it into older kernels as
> > well.
> Perhaps I'm experiencing this because I'm using a Linode VM - it seems to
> hang indefinitely, not just for 90 seconds - but it looks a bit like the
> devs have passed the report upstream and left the affected versions in the
> tree to bite stable users.

It just happens on one machine here, but I shorten the wait by rattling keys 
to supply entropy.

-- 
Regards,
Peter.

Re: [gentoo-user] chrony-3.3 hangs at boot

[Mick]

[-- Attachment #1: Type: text/plain, Size: 1291 bytes --]

On Friday, 2 November 2018 16:59:32 GMT Peter Humphrey wrote:
> On Friday, 2 November 2018 16:38:14 GMT Stroller wrote:
> > > On 3 Sep 2018, at 19:10, Holger Hoffstätte
> > > <holger@applied-asynchrony.com>
> > > wrote:>
> > > 
> > > On Mon, 03 Sep 2018 17:48:53 +0100, Mick wrote:
> > >> I just noticed today chronyd hangs during boot for a minute and a half.
> > >> The logs do not reveal anything amiss.  I suspect it waits for a
> > >> network
> > >> connection, which is not yet up when chronyd launches.
> > >> …
> > > 
> > > Yes and yes; countless people have been hit by this, I was probably the
> > > first to notice/debug it. It happens due to changes in Linux' random
> > > number generator, which is now slowly making it into older kernels as
> > > well.
> > 
> > Perhaps I'm experiencing this because I'm using a Linode VM - it seems to
> > hang indefinitely, not just for 90 seconds - but it looks a bit like the
> > devs have passed the report upstream and left the affected versions in the
> > tree to bite stable users.
> 
> It just happens on one machine here, but I shorten the wait by rattling keys
> to supply entropy.


I'm on chrony-3.3 and gentoo kernel-4.14.78.  I have not noticed this problem 
for a while now.

-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]