[gentoo-dev] Signed push & clock drift rejection

[gentoo-dev] Signed push & clock drift rejection

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 1332 bytes --]

Hi all,

In tracing down problems with the git->rsync path, it has been noticed
that some developers have significant clock drift on their local systems
(up to one case of 14 days wrong), and it's potentially contributing to
problems in generating the rsync tree.

I have implemented a check as part of the hook that validates Git push
certificates (require-signed-push). It looks for clock drift or an
overly long push, and aborts if needed.

The tolerances are presently set to:
- 5 seconds of clock drift.
- 'git push' must be completed in 60 seconds.

The two possible errors you will get during push are:
===
Your system clock is off by 119 seconds (limit 5).
Run NTP, rebase your commits as needed, and push again.
===
Try again! Your push took 80 seconds (limit 60).
===

If you do get the first error, please strongly look at running ntpd,
tlsdate or some other time keeping solution.

This applies to all Git repos that require signed pushes presently:
- data/*
- foundation/*
- infra/*
- proj/portage
- repo/gentoo
- repo/proj/gen-b0rk
(and a few more)

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Trustee & Treasurer
E-Mail   : robbat2@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 445 bytes --]

Re: [gentoo-dev] Signed push & clock drift rejection

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 1803 bytes --]

Hi,

On Fri, 15 Jul 2016 18:03:30 +0000 Robin H. Johnson wrote:
> Hi all,
> 
> In tracing down problems with the git->rsync path, it has been noticed
> that some developers have significant clock drift on their local systems
> (up to one case of 14 days wrong), and it's potentially contributing to
> problems in generating the rsync tree.
> 
> I have implemented a check as part of the hook that validates Git push
> certificates (require-signed-push). It looks for clock drift or an
> overly long push, and aborts if needed.
> 
> The tolerances are presently set to:
> - 5 seconds of clock drift.

Why such tight requirement? Why not a minute, which will not hurt
git, but will help with system _temporarily_ out-of-sync.

Some hardware clocks are real mess and can drift more that for 5
seconds in a few days (e.g. when system was shut down). And for NTP
it will take time to correct system clock _properly_. While stuff
like running ntpdate before ntp server if system is out of sync is
possible, but it is not recommended nor possible on some workloads.
So IRL NTP may take several hours to sync system properly.

Set it for a minute or two. This will protect from commits from
really out-of-sync systems (like 14 days mentioned above) and will
keep usablity hight for others.

> - 'git push' must be completed in 60 seconds.

Why?! What is wrong if push will take 120 seconds? I often commit
from quite an old box and git push takes 20-40 seconds, while this
is within your limits, the margin is not safe.

What if someone needs to commit via 2G GPRS or similar slow network
link? Afaik we have developers on quite slow and unstable links.

Just set this limit to 5 minutes to make it a sane protection of a
stale push.

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] Signed push & clock drift rejection

[Rafael Goncalves Martins]

On Sat, Jul 16, 2016 at 11:33 AM, Andrew Savchenko <bircoph@gentoo.org> wrote:
> Hi,
>
> On Fri, 15 Jul 2016 18:03:30 +0000 Robin H. Johnson wrote:
>> Hi all,
>>
>> In tracing down problems with the git->rsync path, it has been noticed
>> that some developers have significant clock drift on their local systems
>> (up to one case of 14 days wrong), and it's potentially contributing to
>> problems in generating the rsync tree.
>>
>> I have implemented a check as part of the hook that validates Git push
>> certificates (require-signed-push). It looks for clock drift or an
>> overly long push, and aborts if needed.
>>
>> The tolerances are presently set to:
>> - 5 seconds of clock drift.
>
> Why such tight requirement? Why not a minute, which will not hurt
> git, but will help with system _temporarily_ out-of-sync.
>
> Some hardware clocks are real mess and can drift more that for 5
> seconds in a few days (e.g. when system was shut down). And for NTP
> it will take time to correct system clock _properly_. While stuff
> like running ntpdate before ntp server if system is out of sync is
> possible, but it is not recommended nor possible on some workloads.
> So IRL NTP may take several hours to sync system properly.
>
> Set it for a minute or two. This will protect from commits from
> really out-of-sync systems (like 14 days mentioned above) and will
> keep usablity hight for others.

I second this "request" :)

    remote: Your system clock is off by 6 seconds (limit 5)

Regards,
Rafael

>> - 'git push' must be completed in 60 seconds.
>
> Why?! What is wrong if push will take 120 seconds? I often commit
> from quite an old box and git push takes 20-40 seconds, while this
> is within your limits, the margin is not safe.
>
> What if someone needs to commit via 2G GPRS or similar slow network
> link? Afaik we have developers on quite slow and unstable links.
>
> Just set this limit to 5 minutes to make it a sane protection of a
> stale push.
>
> Best regards,
> Andrew Savchenko



-- 
Rafael Goncalves Martins
Gentoo Linux developer
http://rafaelmartins.eng.br/

Re: [gentoo-dev] Signed push & clock drift rejection

[Marc Schiffbauer]

[-- Attachment #1: Type: text/plain, Size: 604 bytes --]

* Rafael Goncalves Martins schrieb am 18.07.16 um 03:12 Uhr:
> On Sat, Jul 16, 2016 at 11:33 AM, Andrew Savchenko <bircoph@gentoo.org> wrote:
> > Set it for a minute or two. This will protect from commits from
> > really out-of-sync systems (like 14 days mentioned above) and will
> > keep usablity hight for others.
> 
> I second this "request" :)
> 
>     remote: Your system clock is off by 6 seconds (limit 5)

Why not fix your system clock? No ntpd running?

Check 'ntpq -pn'

-Marc

-- 
0xCA3E7BF67F979BE5 - F7FB 78F7 7CC3 79F6 DF07
                     6E9E CA3E 7BF6 7F97 9BE5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] Signed push & clock drift rejection

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 879 bytes --]

On Mon, 18 Jul 2016 22:03:35 +0200 Marc Schiffbauer wrote:
> * Rafael Goncalves Martins schrieb am 18.07.16 um 03:12 Uhr:
> > On Sat, Jul 16, 2016 at 11:33 AM, Andrew Savchenko <bircoph@gentoo.org> wrote:
> > > Set it for a minute or two. This will protect from commits from
> > > really out-of-sync systems (like 14 days mentioned above) and will
> > > keep usablity hight for others.
> > 
> > I second this "request" :)
> > 
> >     remote: Your system clock is off by 6 seconds (limit 5)
> 
> Why not fix your system clock? No ntpd running?

As I wrote earlier in this thread, ntp server is not a guarantee
that such problems will not happen. If hardware clocked was
significantly offset during boot, it may take several _hours_ for
ntp to fix this via clock skew. Apparantly commit may be made
during these several hours.

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] Signed push & clock drift rejection

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 1125 bytes --]

>>>>> On Mon, 18 Jul 2016, Rafael Goncalves Martins wrote:

> On Sat, Jul 16, 2016 at 11:33 AM, Andrew Savchenko <bircoph@gentoo.org> wrote:
>> Why such tight requirement? Why not a minute, which will not hurt
>> git, but will help with system _temporarily_ out-of-sync.
>> 
>> Some hardware clocks are real mess and can drift more that for 5
>> seconds in a few days (e.g. when system was shut down). And for NTP
>> it will take time to correct system clock _properly_. While stuff
>> like running ntpdate before ntp server if system is out of sync is
>> possible, but it is not recommended nor possible on some workloads.
>> So IRL NTP may take several hours to sync system properly.
>> 
>> Set it for a minute or two. This will protect from commits from
>> really out-of-sync systems (like 14 days mentioned above) and will
>> keep usablity hight for others.

> I second this "request" :)

>     remote: Your system clock is off by 6 seconds (limit 5)

+1

Same here, push was rejected because of 8 seconds clock offset.
This happened shortly after resuming from suspend, so apparently ntpd
had not caught up yet.

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Signed push & clock drift rejection

[Rich Freeman]

On Sat, Jul 16, 2016 at 5:33 AM, Andrew Savchenko <bircoph@gentoo.org> wrote:
>
> On Fri, 15 Jul 2016 18:03:30 +0000 Robin H. Johnson wrote:
>>
>> The tolerances are presently set to:
>> - 5 seconds of clock drift.
>
> Set it for a minute or two. This will protect from commits from
> really out-of-sync systems (like 14 days mentioned above) and will
> keep usablity hight for others.

I'll defer to infra on how much they can accept, but I tend to think
that we can afford to be a bit more liberal.  However, I don't think
we want to accept things like systems coming out of suspend that are
off by hours.

>
>> - 'git push' must be completed in 60 seconds.
>
> Why?! What is wrong if push will take 120 seconds? I often commit
> from quite an old box and git push takes 20-40 seconds, while this
> is within your limits, the margin is not safe.
>
> What if someone needs to commit via 2G GPRS or similar slow network
> link? Afaik we have developers on quite slow and unstable links.
>
> Just set this limit to 5 minutes to make it a sane protection of a
> stale push.
>

Somebody can correct me if I'm wrong, but I'm pretty sure that only
one person can be pushing anything at time.  So, regardless of any
rsync limitations, I'm not sure we really want developers to be
spending 5 minutes doing a push.  That means that if anybody else does
a commit during that 5 minutes they're going to have to rebase it.
For repos that don't get heavy use I think we could be more liberal.


-- 
Rich

Re: [gentoo-dev] Signed push & clock drift rejection

[james]

On 07/18/2016 03:03 PM, Marc Schiffbauer wrote:
> * Rafael Goncalves Martins schrieb am 18.07.16 um 03:12 Uhr:
>> On Sat, Jul 16, 2016 at 11:33 AM, Andrew Savchenko <bircoph@gentoo.org> wrote:
>>> Set it for a minute or two. This will protect from commits from
>>> really out-of-sync systems (like 14 days mentioned above) and will
>>> keep usablity hight for others.
>>
>> I second this "request" :)
>>
>>     remote: Your system clock is off by 6 seconds (limit 5)
>
> Why not fix your system clock? No ntpd running?
>
> Check 'ntpq -pn'
>
> -Marc
>

net-misc/openntpd is simple and might do the job well enough, or is
net-misc/ntp a hard requirement ?

I just use the  default (gentoo) time servers, for now, but perhaps 
using specified servers in different regions might work too?


hth,
James

Re: [gentoo-dev] Signed push & clock drift rejection

[Mart Raudsepp]

Ühel kenal päeval, E, 18.07.2016 kell 16:25, kirjutas james:
> On 07/18/2016 03:03 PM, Marc Schiffbauer wrote:
> > * Rafael Goncalves Martins schrieb am 18.07.16 um 03:12 Uhr:
> > > On Sat, Jul 16, 2016 at 11:33 AM, Andrew Savchenko <bircoph@gento
> > > o.org> wrote:
> > > > Set it for a minute or two. This will protect from commits from
> > > > really out-of-sync systems (like 14 days mentioned above) and
> > > > will
> > > > keep usablity hight for others.
> > > 
> > > I second this "request" :)
> > > 
> > >     remote: Your system clock is off by 6 seconds (limit 5)
> > 
> > Why not fix your system clock? No ntpd running?
> > 
> > Check 'ntpq -pn'
> > 
> > -Marc
> > 
> 
> net-misc/openntpd is simple and might do the job well enough, or is
> net-misc/ntp a hard requirement ?
> 

or just

systemctl enable systemd-timesyncd.service
systemctl start systemd-timesyncd.service

if you are fortunate enough to run systemd.
If not, well, some other ntp client indeed, no need to debate fortunes
further :)

If there's no RTC clock ticking and syncing during/after suspend,
something seems off in my experience. Disabled ACPI? :D

I didn't find any indications of why this is actually a problem in the
announcement or replies, and I couldn't read the backlog for the
explanation that I saw might have skimmed through. I think if we want
to nitpick what the actual drift allowed is, we need to know the
reasons this restriction is needed and what could be the difference to
that unspecified potential rsync issue between a seconds of drifts and
a couple minutes or an hour or so of drift.
I'm sure infra will adjust if possible and choose what's best :)


Mart

Re: [gentoo-dev] Signed push & clock drift rejection

[Bill Kenworthy]

On 19/07/16 07:06, Mart Raudsepp wrote:
> Ühel kenal päeval, E, 18.07.2016 kell 16:25, kirjutas james:
>> On 07/18/2016 03:03 PM, Marc Schiffbauer wrote:
>>> * Rafael Goncalves Martins schrieb am 18.07.16 um 03:12 Uhr:
>>>> On Sat, Jul 16, 2016 at 11:33 AM, Andrew Savchenko <bircoph@gento
>>>> o.org> wrote:
>>>>> Set it for a minute or two. This will protect from commits from
>>>>> really out-of-sync systems (like 14 days mentioned above) and
>>>>> will
>>>>> keep usablity hight for others.
>>>>
>>>> I second this "request" :)
>>>>
>>>>     remote: Your system clock is off by 6 seconds (limit 5)
>>>
>>> Why not fix your system clock? No ntpd running?
>>>
>>> Check 'ntpq -pn'
>>>
>>> -Marc
>>>
>>
>> net-misc/openntpd is simple and might do the job well enough, or is
>> net-misc/ntp a hard requirement ?
>>
> 
> or just
> 
> systemctl enable systemd-timesyncd.service
> systemctl start systemd-timesyncd.service
> 
> if you are fortunate enough to run systemd.
> If not, well, some other ntp client indeed, no need to debate fortunes
> further :)
> 
> If there's no RTC clock ticking and syncing during/after suspend,
> something seems off in my experience. Disabled ACPI? :D
> 
> I didn't find any indications of why this is actually a problem in the
> announcement or replies, and I couldn't read the backlog for the
> explanation that I saw might have skimmed through. I think if we want
> to nitpick what the actual drift allowed is, we need to know the
> reasons this restriction is needed and what could be the difference to
> that unspecified potential rsync issue between a seconds of drifts and
> a couple minutes or an hour or so of drift.
> I'm sure infra will adjust if possible and choose what's best :)
> 
> 
> Mart
> 

or configure ntpd appropriately - look into the -g argument to ntpd, or
"tinker panic 0", burst and iburst  in the config file to quickly step
if the error is greater than 1000s.  Chrony has worked better than ntp
for me in libvirt managed qemu instances but may still not sync for
quite awhile.

If you have a raspberry pi or similar (no hwclock) use swclock instead
to prime the system time with a close value - just ran into this on a pi
where the default date (1st Jan 1970) was outside a networks certificate
range so it couldn't get a network connection to set the time so it
could access the network :)  Systemd can do this too - but I have read
(cant find the reference) that its time mechanism is non-standard (who
would have thought!) and is very primitive compared to alternatives -
good enough for non-critical applications though.

BillK

Re: [gentoo-dev] Signed push & clock drift rejection

[waltdnes]

On Mon, Jul 18, 2016 at 11:27:09PM +0300, Andrew Savchenko wrote
> 
> As I wrote earlier in this thread, ntp server is not a guarantee
> that such problems will not happen. If hardware clocked was
> significantly offset during boot, it may take several _hours_ for
> ntp to fix this via clock skew. Apparantly commit may be made
> during these several hours.

  I'm amazed that "robust linux servers" are deathly afraid of simply
setting the time, and being done with it.  And while we're at it, if a
developer is doing development on a server machine, he may have other
problems to worry about.  At home I occasionally manually run a script
that includes the 2 lines...

/usr/bin/sudo /usr/bin/openrdate -n -s ca.pool.ntp.org
/usr/bin/sudo /sbin/hwclock --systohc

-- 
Walter Dnes <waltdnes@waltdnes.org>
I don't run "desktop environments"; I run useful applications

Re: [gentoo-dev] Signed push & clock drift rejection

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 572 bytes --]

On Mon, 18 Jul 2016 22:21:22 -0400
waltdnes@waltdnes.org wrote:

>   I'm amazed that "robust linux servers" are deathly afraid of simply
> setting the time, and being done with it.

There's problems at the software level everywhere that are not so
simply solved.

A more obvious example is in the event your system time gets *ahead* of
real-time.

Then its likely that some actions have been performed that have to only
happen once at a given time, and a time going back in time results in
"double spending" of some seconds, and that's obviously not great.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] Signed push & clock drift rejection

[Chí-Thanh Christopher Nguyễn]

[-- Attachment #1: Type: text/plain, Size: 852 bytes --]

Kent Fredric schrieb:
> On Mon, 18 Jul 2016 22:21:22 -0400
> waltdnes@waltdnes.org wrote:
>
>>    I'm amazed that "robust linux servers" are deathly afraid of simply
>> setting the time, and being done with it.
>
> There's problems at the software level everywhere that are not so
> simply solved.
>
> A more obvious example is in the event your system time gets *ahead* of
> real-time.

And even if the system is behind time, it can cause problems. cronjobs 
running unexpectedly close to each other (or missed cronjobs in extreme 
cases). User sessions expiring early, etc.

And even if there is only one second, and that is known well ahead (e.g. leap 
seconds): Unless you know that there isn't going to be a problem, a great 
deal of care needs to go into handling that.


Best regards,
Chí-Thanh Christopher Nguyễn



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] Signed push & clock drift rejection

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 1294 bytes --]

On Mon, 18 Jul 2016 16:25:34 -0500 james wrote:
> On 07/18/2016 03:03 PM, Marc Schiffbauer wrote:
> > * Rafael Goncalves Martins schrieb am 18.07.16 um 03:12 Uhr:
> >> On Sat, Jul 16, 2016 at 11:33 AM, Andrew Savchenko <bircoph@gentoo.org> wrote:
> >>> Set it for a minute or two. This will protect from commits from
> >>> really out-of-sync systems (like 14 days mentioned above) and will
> >>> keep usablity hight for others.
> >>
> >> I second this "request" :)
> >>
> >>     remote: Your system clock is off by 6 seconds (limit 5)
> >
> > Why not fix your system clock? No ntpd running?
> >
> > Check 'ntpq -pn'
> >
> > -Marc
> >
> 
> net-misc/openntpd is simple and might do the job well enough, or is
> net-misc/ntp a hard requirement ?
> 
> I just use the  default (gentoo) time servers, for now, but perhaps 
> using specified servers in different regions might work too?

Any ntp sync daemon will be fine.

I prefer net-misc/chrony: it is simpler and more secure than ntpd
(e.g. look at CVE counts) and handles well situations when upstream
servers go offline (e.g. continues to apply calculated drift to rtc
clock). Though it doesn't support all features of ntpd (e.g. ssl),
it has all subset of them sufficient IRL.

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] Signed push & clock drift rejection

[waltdnes]

On Tue, Jul 19, 2016 at 10:07:12AM +0200, Chí-Thanh Christopher Nguy???n wrote
> Kent Fredric schrieb:
> > On Mon, 18 Jul 2016 22:21:22 -0400
> > waltdnes@waltdnes.org wrote:
> >
> >>    I'm amazed that "robust linux servers" are deathly afraid of simply
> >> setting the time, and being done with it.
> >
> > There's problems at the software level everywhere that are not so
> > simply solved.
> >
> > A more obvious example is in the event your system time gets *ahead* of
> > real-time.
> 
> And even if the system is behind time, it can cause problems. cronjobs 
> running unexpectedly close to each other (or missed cronjobs in extreme 
> cases). User sessions expiring early, etc.
> 
> And even if there is only one second, and that is known well ahead
> (e.g. leap seconds): Unless you know that there isn't going to be
> a problem, a great deal of care needs to go into handling that.

  In that case, the dev machine should be a separate machine from the
server.  They don't even have to be separate physical machines.  Do the
dev work in a VM, and set the time in the VM just before doing the push.

-- 
Walter Dnes <waltdnes@waltdnes.org>
I don't run "desktop environments"; I run useful applications

Re: [gentoo-dev] Signed push & clock drift rejection

[Consus]

On 18:03 Fri 15 Jul, Robin H. Johnson wrote:
> Hi all,
> 
> In tracing down problems with the git->rsync path, it has been noticed
> that some developers have significant clock drift on their local systems
> (up to one case of 14 days wrong), and it's potentially contributing to
> problems in generating the rsync tree.
> 
> I have implemented a check as part of the hook that validates Git push
> certificates (require-signed-push). It looks for clock drift or an
> overly long push, and aborts if needed.
> 
> The tolerances are presently set to:
> - 5 seconds of clock drift.
> - 'git push' must be completed in 60 seconds.

AFAIR Windows AD servers allow 5 minutes clock drift. How about at least
a minute or so?

Re: [gentoo-dev] Signed push & clock drift rejection

[Chí-Thanh Christopher Nguyễn]

waltdnes@waltdnes.org schrieb:
>> And even if the system is behind time, it can cause problems. cronjobs
>> running unexpectedly close to each other (or missed cronjobs in extreme
>> cases). User sessions expiring early, etc.
>>
>> And even if there is only one second, and that is known well ahead
>> (e.g. leap seconds): Unless you know that there isn't going to be
>> a problem, a great deal of care needs to go into handling that.
>    In that case, the dev machine should be a separate machine from the
> server.  They don't even have to be separate physical machines.  Do the
> dev work in a VM, and set the time in the VM just before doing the push.

I am not talking about bircoph's dev machine, I am explaining why ntpd 
works the way it does. Which usually doesn't lead to problems as the 
drift will be corrected sooner or later. So there was never a problem 
nor reason for action in either case, until it was decided that 5 
seconds is the maximum drift for a push to gentoo.git.


Best regards,
Chí-Thanh Christopher Nguyễn

Re: [gentoo-dev] Signed push & clock drift rejection

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1008 bytes --]

On Mon, 18 Jul 2016 22:03:35 +0200
Marc Schiffbauer <mschiff@gentoo.org> wrote:

> * Rafael Goncalves Martins schrieb am 18.07.16 um 03:12 Uhr:
> > On Sat, Jul 16, 2016 at 11:33 AM, Andrew Savchenko <bircoph@gentoo.org> wrote:  
> > > Set it for a minute or two. This will protect from commits from
> > > really out-of-sync systems (like 14 days mentioned above) and will
> > > keep usablity hight for others.  
> > 
> > I second this "request" :)
> > 
> >     remote: Your system clock is off by 6 seconds (limit 5)  
> 
> Why not fix your system clock? No ntpd running?
> 
> Check 'ntpq -pn'

But Gentoo is about choice! Developers should have a choice! If they
want to use systems with completely broken clock, why do you want to
take away that choice from them? They have a right to push commits
with completely useless timestamps if they desire, or to break rsync
generation for our users every now and then!

-- 
Best regards,
Michał Górny
<http://dev.gentoo.org/~mgorny/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] Signed push & clock drift rejection

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 2520 bytes --]

Hi,

On Mon, 18 Jul 2016 17:20:50 -0400 Rich Freeman wrote:
> On Sat, Jul 16, 2016 at 5:33 AM, Andrew Savchenko <bircoph@gentoo.org> wrote:
> >
> > On Fri, 15 Jul 2016 18:03:30 +0000 Robin H. Johnson wrote:
> >>
> >> The tolerances are presently set to:
> >> - 5 seconds of clock drift.
> >
> > Set it for a minute or two. This will protect from commits from
> > really out-of-sync systems (like 14 days mentioned above) and will
> > keep usablity hight for others.
> 
> I'll defer to infra on how much they can accept, but I tend to think
> that we can afford to be a bit more liberal.  However, I don't think
> we want to accept things like systems coming out of suspend that are
> off by hours.

Nobody asks for hours :) 5 minutes should be fine and sane.

> >
> >> - 'git push' must be completed in 60 seconds.
> >
> > Why?! What is wrong if push will take 120 seconds? I often commit
> > from quite an old box and git push takes 20-40 seconds, while this
> > is within your limits, the margin is not safe.
> >
> > What if someone needs to commit via 2G GPRS or similar slow network
> > link? Afaik we have developers on quite slow and unstable links.
> >
> > Just set this limit to 5 minutes to make it a sane protection of a
> > stale push.
> >
> 
> Somebody can correct me if I'm wrong, but I'm pretty sure that only
> one person can be pushing anything at time.

Indeed so.

> So, regardless of any
> rsync limitations, I'm not sure we really want developers to be
> spending 5 minutes doing a push.  That means that if anybody else does
> a commit during that 5 minutes they're going to have to rebase it.

Let us consider which way we'll make more harm than good.

If git push takes longer than a minute due to a slow box or a bad
uplink, there is no way to fix this. If current limit will be
enforced further, we'll actually ban developers with slow
environment from contributing to Gentoo.

As for rebasing, this is simple and can be automated by
pull.rebase=true (it should be disabled during merge commits and
can be done so on the command line of course, but random merge
commits are not welcomed due to our git workflow policy, so this is
a rare event).

If I understand a reasoning behind the git push time constraint
correctly, it is needed as a safeguard from stale pushes, so events
where push takes > 1 minute should be very rare, thus a little harm
can be done by raising git push limit to 5 minutes.

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] Signed push & clock drift rejection

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 2119 bytes --]

On Mon, 18 Jul 2016 22:21:22 -0400 waltdnes@waltdnes.org wrote:
> On Mon, Jul 18, 2016 at 11:27:09PM +0300, Andrew Savchenko wrote
> > 
> > As I wrote earlier in this thread, ntp server is not a guarantee
> > that such problems will not happen. If hardware clocked was
> > significantly offset during boot, it may take several _hours_ for
> > ntp to fix this via clock skew. Apparantly commit may be made
> > during these several hours.
> 
>   I'm amazed that "robust linux servers" are deathly afraid of simply
> setting the time, and being done with it. And while we're at it, if a
> developer is doing development on a server machine, he may have other
> problems to worry about.  At home I occasionally manually run a script
> that includes the 2 lines...

I never said anything about "robust linux servers". If you think
than only servers need a gradual clock slew instead of stepping,
you are mistaken.

> /usr/bin/sudo /usr/bin/openrdate -n -s ca.pool.ntp.org
> /usr/bin/sudo /sbin/hwclock --systohc

And if time delta is significant, the system may become broken
this way. Congratulations.

The gradual NTP time slew was not invented because people were lazy
to run two simple commands. Actually it is just the opposite:
setting system time via a single huge step is much easier to
implement than a proper adjustment via a series of small time slews.
For servers it is indeed important in many ways, including
time stamp based cryptography as kerberos or database integrity.

But desktops also do need a proper time adjustment:
- Filesystems will not operate correctly with time stamps in
future, in the best keys they will be marked/reported as needed a
repair procedure.
- Cron jobs may go broken too as chithanh mentioned already.
- Video encoding is not happy with time shifts at all. While small
predictable slews can be tolerated, large jumps will just broke the
process.
- System may become *vulnerable* because of time stamp based attack.
Though it is not easy to use such behaviour, it is still possible.
- and many more...

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] Signed push & clock drift rejection

[R0b0t1]

On Tue, Jul 19, 2016 at 1:23 PM, Andrew Savchenko <bircoph@gentoo.org> wrote:
> - System may become *vulnerable* because of time stamp based attack.
> Though it is not easy to use such behaviour, it is still possible.

All offline attacks are harder, even ones which haven't been invented yet.

Re: [gentoo-dev] Signed push & clock drift rejection

[Patrick McLean]

On Tue, 19 Jul 2016 21:23:16 +0300
Andrew Savchenko <bircoph@gentoo.org> wrote:
> On Mon, 18 Jul 2016 22:21:22 -0400 waltdnes@waltdnes.org wrote:
> > On Mon, Jul 18, 2016 at 11:27:09PM +0300, Andrew Savchenko wrote  
> > > 
> > > As I wrote earlier in this thread, ntp server is not a guarantee
> > > that such problems will not happen. If hardware clocked was
> > > significantly offset during boot, it may take several _hours_ for
> > > ntp to fix this via clock skew. Apparantly commit may be made
> > > during these several hours.  
> > 
> >   I'm amazed that "robust linux servers" are deathly afraid of
> > simply setting the time, and being done with it. And while we're at
> > it, if a developer is doing development on a server machine, he may
> > have other problems to worry about.  At home I occasionally
> > manually run a script that includes the 2 lines...  
> 
> I never said anything about "robust linux servers". If you think
> than only servers need a gradual clock slew instead of stepping,
> you are mistaken.
> 
> > /usr/bin/sudo /usr/bin/openrdate -n -s ca.pool.ntp.org
> > /usr/bin/sudo /sbin/hwclock --systohc  
> 
> And if time delta is significant, the system may become broken
> this way. Congratulations.

Really? I have many times skewed time by weeks using ntpdate with no
issues.

> The gradual NTP time slew was not invented because people were lazy
> to run two simple commands. Actually it is just the opposite:
> setting system time via a single huge step is much easier to
> implement than a proper adjustment via a series of small time slews.
> For servers it is indeed important in many ways, including
> time stamp based cryptography as kerberos or database integrity.

Sure randomly skewing time can cause weird issues, which is why it is a
manual operation (and/or boot time operation).

> 
> But desktops also do need a proper time adjustment:
> - Filesystems will not operate correctly with time stamps in
> future, in the best keys they will be marked/reported as needed a
> repair procedure.

I have only ever seen ext4 complain about the superblock timestamp,
and fsck generally just corrects without issue, and it generally is
only an issue after a reboot.

> - Cron jobs may go broken too as chithanh mentioned already.

Get a better crond? Decent cron daemons can detect time skews and act
accordingly.

> - Video encoding is not happy with time shifts at all. While small
> predictable slews can be tolerated, large jumps will just broke the
> process.

Anything that cares about having a monotonic clock, and doesn't
actually care about the real time (like video encoding) should just use
the monotonic clock, not the system time.

> - System may become *vulnerable* because of time stamp based attack.
> Though it is not easy to use such behaviour, it is still possible.

Once again, monotonic clock exists for a reason. If you want to
talk about vulnerabilities, you do realize that doesn't work properly
unless the client and server have reasonably similar system times.

> - and many more...
> 
> Best regards,
> Andrew Savchenko

Re: [gentoo-dev] Signed push & clock drift rejection

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 857 bytes --]

On Tue, Jul 19, 2016 at 4:31 AM, Consus <consus@gmx.com> wrote:

> On 18:03 Fri 15 Jul, Robin H. Johnson wrote:
> > Hi all,
> >
> > In tracing down problems with the git->rsync path, it has been noticed
> > that some developers have significant clock drift on their local systems
> > (up to one case of 14 days wrong), and it's potentially contributing to
> > problems in generating the rsync tree.
> >
> > I have implemented a check as part of the hook that validates Git push
> > certificates (require-signed-push). It looks for clock drift or an
> > overly long push, and aborts if needed.
> >
> > The tolerances are presently set to:
> > - 5 seconds of clock drift.
> > - 'git push' must be completed in 60 seconds.
>
> AFAIR Windows AD servers allow 5 minutes clock drift. How about at least
> a minute or so?
>
>
I would also argue for 5 minutes.

-A

[-- Attachment #2: Type: text/html, Size: 1417 bytes --]

Re: [gentoo-dev] Signed push & clock drift rejection

[Patrick McLean]

On Tue, 19 Jul 2016 13:22:29 -0700
Patrick McLean <chutzpah@gentoo.org> wrote:

> On Tue, 19 Jul 2016 21:23:16 +0300
> Andrew Savchenko <bircoph@gentoo.org> wrote:
> > On Mon, 18 Jul 2016 22:21:22 -0400 waltdnes@waltdnes.org wrote:  
> > > On Mon, Jul 18, 2016 at 11:27:09PM +0300, Andrew Savchenko
> > > wrote    
> > > > 
> > > > As I wrote earlier in this thread, ntp server is not a guarantee
> > > > that such problems will not happen. If hardware clocked was
> > > > significantly offset during boot, it may take several _hours_
> > > > for ntp to fix this via clock skew. Apparantly commit may be
> > > > made during these several hours.    
> > > 
> > >   I'm amazed that "robust linux servers" are deathly afraid of
> > > simply setting the time, and being done with it. And while we're
> > > at it, if a developer is doing development on a server machine,
> > > he may have other problems to worry about.  At home I occasionally
> > > manually run a script that includes the 2 lines...    
> > 
> > I never said anything about "robust linux servers". If you think
> > than only servers need a gradual clock slew instead of stepping,
> > you are mistaken.
> >   
> > > /usr/bin/sudo /usr/bin/openrdate -n -s ca.pool.ntp.org
> > > /usr/bin/sudo /sbin/hwclock --systohc    
> > 
> > And if time delta is significant, the system may become broken
> > this way. Congratulations.  
> 
> Really? I have many times skewed time by weeks using ntpdate with no
> issues.
> 
> > The gradual NTP time slew was not invented because people were lazy
> > to run two simple commands. Actually it is just the opposite:
> > setting system time via a single huge step is much easier to
> > implement than a proper adjustment via a series of small time slews.
> > For servers it is indeed important in many ways, including
> > time stamp based cryptography as kerberos or database integrity.  
> 
> Sure randomly skewing time can cause weird issues, which is why it is
> a manual operation (and/or boot time operation).
> 
> > 
> > But desktops also do need a proper time adjustment:
> > - Filesystems will not operate correctly with time stamps in
> > future, in the best keys they will be marked/reported as needed a
> > repair procedure.  
> 
> I have only ever seen ext4 complain about the superblock timestamp,
> and fsck generally just corrects without issue, and it generally is
> only an issue after a reboot.
> 
> > - Cron jobs may go broken too as chithanh mentioned already.  
> 
> Get a better crond? Decent cron daemons can detect time skews and act
> accordingly.
> 
> > - Video encoding is not happy with time shifts at all. While small
> > predictable slews can be tolerated, large jumps will just broke the
> > process.  
> 
> Anything that cares about having a monotonic clock, and doesn't
> actually care about the real time (like video encoding) should just
> use the monotonic clock, not the system time.
> 
> > - System may become *vulnerable* because of time stamp based attack.
> > Though it is not easy to use such behaviour, it is still possible.  
> 
> Once again, monotonic clock exists for a reason. If you want to
> talk about vulnerabilities, you do realize that doesn't work properly
> unless the client and server have reasonably similar system times.
> 
Err, SSL doesn't work properly.

> > - and many more...
> > 
> > Best regards,
> > Andrew Savchenko  
> 
> 

Re: [gentoo-dev] Signed push & clock drift rejection

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 3124 bytes --]

On Tue, Jul 19, 2016 at 11:23 AM, Andrew Savchenko <bircoph@gentoo.org>
wrote:

> On Mon, 18 Jul 2016 22:21:22 -0400 waltdnes@waltdnes.org wrote:
> > On Mon, Jul 18, 2016 at 11:27:09PM +0300, Andrew Savchenko wrote
> > >
> > > As I wrote earlier in this thread, ntp server is not a guarantee
> > > that such problems will not happen. If hardware clocked was
> > > significantly offset during boot, it may take several _hours_ for
> > > ntp to fix this via clock skew. Apparantly commit may be made
> > > during these several hours.
> >
> >   I'm amazed that "robust linux servers" are deathly afraid of simply
> > setting the time, and being done with it. And while we're at it, if a
> > developer is doing development on a server machine, he may have other
> > problems to worry about.  At home I occasionally manually run a script
> > that includes the 2 lines...
>

Its actually really typical for programmers to do bad things with time.
Some examples include:

1) Assuming time is monotonically increasing.
2) Assuming time changes at a constant rate.

Doing 1 leads to bugs where your local clock is faster than 'real time' and
your time sync daemon sets your time to a time in the past. This is a very
bad thing for a lot of workloads. Generally callers should use
CLOCK_MONOTONIC (or CLOCK_MONOTONIC_RAW) to properly use a clock, but many
do not.

Doing 2 leads to synchronization issues because you expected APIs like:

DoThing()
WaitFiveSeconds() # Actually expect 5 seconds to pass here?, haha!
DoOtherThing()

Of course this is also a bad pattern, but you'd be surprised where you see
this crop up.

Not to mention other workloads where time is part the security stack
(generating randomness, GUIDs, nonces, etc, avoiding replay, etc.)

-A


> I never said anything about "robust linux servers". If you think
> than only servers need a gradual clock slew instead of stepping,
> you are mistaken.
>
> > /usr/bin/sudo /usr/bin/openrdate -n -s ca.pool.ntp.org
> > /usr/bin/sudo /sbin/hwclock --systohc
>
> And if time delta is significant, the system may become broken
> this way. Congratulations.
>
> The gradual NTP time slew was not invented because people were lazy
> to run two simple commands. Actually it is just the opposite:
> setting system time via a single huge step is much easier to
> implement than a proper adjustment via a series of small time slews.
> For servers it is indeed important in many ways, including
> time stamp based cryptography as kerberos or database integrity.
>
> But desktops also do need a proper time adjustment:
> - Filesystems will not operate correctly with time stamps in
> future, in the best keys they will be marked/reported as needed a
> repair procedure.
> - Cron jobs may go broken too as chithanh mentioned already.
> - Video encoding is not happy with time shifts at all. While small
> predictable slews can be tolerated, large jumps will just broke the
> process.
> - System may become *vulnerable* because of time stamp based attack.
> Though it is not easy to use such behaviour, it is still possible.
> - and many more...
>
> Best regards,
> Andrew Savchenko
>

[-- Attachment #2: Type: text/html, Size: 4222 bytes --]