[gentoo-dev] mtime preservation

[gentoo-dev] mtime preservation

[Denis Dupeyron]

I'm going to try and sum up the situation of the thread started in
[1]. Feel free to correct me or add to the summary in replies.

There seems to be two main ideas. I have removed the authors' name in
quotes below in order to make sure we talk about the ideas and not who
proposed them.

1- All packages are treated equally. Some files have their mtime
preserved, some don't. We need to agree on what files have their mtime
preserved and at what phase the mtime is frozen. What we seemed to
have agreed on is that the wording in PMS needs to be unambiguous
enough that it doesn't defeat the purpose. Here's a first proposition:

"The package manager must preserve modification times of regular files.
This includes files being compressed before merging. Exceptions to
this are:

(Now we need to enumerate the exceptions:)

* files newly created by the package manager,
(This will cover splitdebug, for example. (And please don't tell me
that the wording is flawed because the PM could save a file's contents
in some buffer, then delete the file and create it newly. This would
be as unreasonable as the rot-13 example.) )

* binary object files being stripped of symbols.

(Anything else missing from above list?)"


2- Here's the second idea, shamelessly pasted (note that it says EAPI4
below instead of EAPI3 but this is irrelevant to the idea):

"Thus, I would let EAPI 4 ebuilds call dopreservemtimes (with an API
similar to docompress) in both src_install and pkg_preinst. Doing so
would instruct the package manager that it must preserve mtimes
(including subsecond, if supported on the filesystem) for a particular
set of paths, even if doing so means no stripping etc. All other mtimes
may be rewritten as the package manager sees fit, and from EAPI 4
onwards must be rewritten at merge time for anything predating the
start of the build."

In both cases nanosecond resolution may be required and is a problem
due to python. The following workaround can be used until the
nanosecond issue is fixed in python:

"Alternatively, we could simply make portage spawn the mv binary
whenever rename fails (it fails when the source and destination are
on different devices). Although it's relatively slow, it should
solve the problem."

My intention is to ask the council to vote on which method is
preferable in two weeks. I will also ask the council on whether we
still want mtime preservation for EAPI3 or if we now think it's better
to push it to EAPI4. Please discuss.

Denis.

[1] http://archives.gentoo.org/gentoo-council/msg_becc15a2882cc7e4f1079b2f9f4dcaad.xml

Re: [gentoo-dev] mtime preservation

[Zac Medico]

Denis Dupeyron wrote:
> 1- All packages are treated equally. Some files have their mtime
> preserved, some don't. We need to agree on what files have their mtime
> preserved and at what phase the mtime is frozen.

I'd vote for method 1.

> My intention is to ask the council to vote on which method is
> preferable in two weeks. I will also ask the council on whether we
> still want mtime preservation for EAPI3 or if we now think it's better
> to push it to EAPI4. Please discuss.

You can probably do method 1 retroactively for all EAPIs, since the
few existing packages which require mtime preservation are
presumably broken already anyway (any damage is already done), and
packages which don't require mtime preservation are not hurt by it.
-- 
Thanks,
Zac

Re: [gentoo-dev] mtime preservation

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 3913 bytes --]

On Mon, Nov 23, 2009 at 11:49:25AM -0700, Denis Dupeyron wrote:
> 2- Here's the second idea, shamelessly pasted (note that it says EAPI4
> below instead of EAPI3 but this is irrelevant to the idea):
> 
> "Thus, I would let EAPI 4 ebuilds call dopreservemtimes (with an API
> similar to docompress) in both src_install and pkg_preinst. Doing so
> would instruct the package manager that it must preserve mtimes
> (including subsecond, if supported on the filesystem) for a particular
> set of paths, even if doing so means no stripping etc. All other mtimes
> may be rewritten as the package manager sees fit, and from EAPI 4
> onwards must be rewritten at merge time for anything predating the
> start of the build."

Someone mind explaining to me why we're making mtime preservation so 
nasty?  Having to enumerate every pathway that requires mtime 
preservation is pretty arduous for the ebuild dev, meaning it's 
unlikely they'll get it right, leading to bugs.

The thing I'm not understanding here is that pkgcore since day one has 
preserved mtime- I've yet to see any complaints about that nor any 
issues caused by it.  Portage shifted over a year or two back, same 
thing, haven't heard complaints.

The thing that's bugging me about this whole discussion is that we're 
spending more time focusing on stupid things the manager could do to 
(basically intentionally) screw up the mtime preservation.

I know it won't fly, but realistically stating "the package manager 
must preserve mtime- if there are instances where it does not (due to 
some feature, perhaps splitdebug or some form of compression) it is 
the package managers responsibility to ensure this does not break the 
ebuilds resultant merge/runtime invocation".

Via such wording an exemption is created and it's made clear it's the 
managers responsibility to keep things playing nice... if the manager 
can't do that, then the feature/functionality that is changing the 
mtime (resulting in the pkg on disk failing) must be fixed or 
disabled.

The nice thing about this wording is that it basically matches what 
the case is now, and what has worked for quite a few years.


> In both cases nanosecond resolution may be required and is a problem
> due to python. The following workaround can be used until the
> nanosecond issue is fixed in python:

It'd be nice if someone enumerated merge scenarios where nanosecond 
resolution is actually required.  Seems like a white elephant to me, 
especially in combination w/ the fact that the target fs may not 
support nanosecond.

Basically, if it's required the manager support nanosecond resolution 
for merging, ebuilds must be able to rely on that- end result, any 
merging to FS's that do not support nanosecond (this includes the 
intermediate ${D}) are no longer supported.  Unless I'm missing 
something, iso-9660 doesn't look to support nanosecond resolution.  
Beyond that, does cramfs/squashfs?  If not, taking this to the 
logical conclusion the livecds aren't technically supported as a merge 
environment.

Essentially If we're going to require the manager support NS 
resolution w/out requiring all FS's involved support NS level 
resolution, in reality ebuilds can't rely on NS level resolution.  

Thus drop the requirement since we gain nothing from it- if it's not a 
gurantee ebuilds can rely on then they should be writing to the lowest 
common denominator- second level resolution.  That may sound heinous, 
but it's also being realistic.


> "Alternatively, we could simply make portage spawn the mv binary
> whenever rename fails (it fails when the source and destination are
> on different devices). Although it's relatively slow, it should
> solve the problem."

Yeah... no.  Slowing down the main manager for a thereotical edge case 
doesn't seem particularly useful to me ;)

~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-dev] Re: mtime preservation

[Duncan]

Brian Harring posted on Mon, 23 Nov 2009 15:19:00 -0800 as excerpted:

>> "Alternatively, we could simply make portage spawn the mv binary
>> whenever rename fails (it fails when the source and destination are on
>> different devices). Although it's relatively slow, it should solve the
>> problem."
> 
> Yeah... no.  Slowing down the main manager for a thereotical edge case
> doesn't seem particularly useful to me ;)

I say go for second resolution now, basically using the #1 proposal, and 
apply it to all eapis for the reasons zac outlined, but with a fixed 
effective date say 90 or 180 days out from the council resolution to give 
PMs time to come into compliance.  To try for nanosecond resolution now 
is simply letting the complex perfect be the enemy of the simple good, 
when what we have at present is the no-good.

Then make a donsmtimes like the #2 proposal, that individual ebuilds can 
call for specific paths if it's really necessary.  It's more work for the 
ebuilder, but that's in relative measure to the slowdown it'll cause on 
two out of three PMs including our core PM, so a bit of discouragement 
from using it could be considered a good thing, tho it would be there, if 
required.

If at some point in the future ns resolution becomes a much bigger issue 
and many ebuilds are using the donsmtimes call, there's nothing stopping 
us from looking at further tightening up the requirements for the general 
case, probably as part of an eapi, then.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] mtime preservation

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 3780 bytes --]

On Mon, 23 Nov 2009 15:19:00 -0800
Brian Harring <ferringb@gmail.com> wrote:
> Someone mind explaining to me why we're making mtime preservation so 
> nasty?  Having to enumerate every pathway that requires mtime 
> preservation is pretty arduous for the ebuild dev, meaning it's 
> unlikely they'll get it right, leading to bugs.

It's not in the least bit nasty. It's requiring people to be explicit
about special requirements.

> The thing I'm not understanding here is that pkgcore since day one
> has preserved mtime- I've yet to see any complaints about that nor
> any issues caused by it.  Portage shifted over a year or two back,
> same thing, haven't heard complaints.

You can't have been listening very hard, then. The complaint is that it
results in files with dumb mtimes being merged to /.

> I know it won't fly, but realistically stating "the package manager 
> must preserve mtime- if there are instances where it does not (due to 
> some feature, perhaps splitdebug or some form of compression) it is 
> the package managers responsibility to ensure this does not break the 
> ebuilds resultant merge/runtime invocation".
> 
> Via such wording an exemption is created and it's made clear it's the 
> managers responsibility to keep things playing nice... if the manager 
> can't do that, then the feature/functionality that is changing the 
> mtime (resulting in the pkg on disk failing) must be fixed or 
> disabled.
> 
> The nice thing about this wording is that it basically matches what 
> the case is now, and what has worked for quite a few years.

Wording such as that just isn't suitable for a specification. It
requires implementers to guess what future ebuilds are going to
rely upon (and ebuilds regularly do rely upon weird quirks), and makes
it impossible to produce a package manager that can be shown to be
compliant.

> > In both cases nanosecond resolution may be required and is a problem
> > due to python. The following workaround can be used until the
> > nanosecond issue is fixed in python:
> 
> It'd be nice if someone enumerated merge scenarios where nanosecond 
> resolution is actually required.  Seems like a white elephant to me, 
> especially in combination w/ the fact that the target fs may not 
> support nanosecond.

POSIX considers several of the non-nanosecond resolution calls to be
deprecated. Going "la la la I can't hear you!" because Python happens
to have utterly screwed this up is just going to lead to problems when
programs start using sub-second validity checks -- 'make' already does,
and it's given rise to various build-directory-related issues.

> Basically, if it's required the manager support nanosecond resolution 
> for merging, ebuilds must be able to rely on that- end result, any 
> merging to FS's that do not support nanosecond (this includes the 
> intermediate ${D}) are no longer supported.  Unless I'm missing 
> something, iso-9660 doesn't look to support nanosecond resolution.  
> Beyond that, does cramfs/squashfs?  If not, taking this to the 
> logical conclusion the livecds aren't technically supported as a
> merge environment.

No, we're after a requirement that the package manager must not screw up
nanosecond-resolution timestamps, and must not partially preserve and
partially not preserve them.

> > "Alternatively, we could simply make portage spawn the mv binary
> > whenever rename fails (it fails when the source and destination are
> > on different devices). Although it's relatively slow, it should
> > solve the problem."
> 
> Yeah... no.  Slowing down the main manager for a thereotical edge
> case doesn't seem particularly useful to me ;)

...or you could just fix Python.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-dev] Re: mtime preservation

[Denis Dupeyron]

A quick note to tell you that I have tried to look for examples of
breakage due to how mtime preservation is currently implemented in
portage. Unfortunately I didn't find anything, maybe because I'm not
knowledgeable enough or because I can't afford to spend any more time
on this. If any of you can provide an example then please do so.

In case nobody could show any sign of such breakage I would have to
add to the list of two propositions to vote on:
3- Do nothing.

We can always point to code that shows the implementation is
suboptimal but we have to understand that some of us who are less
proactive at fixing issues may want to procrastinate until breakage
actually happens.

Denis.

Re: [gentoo-dev] Re: mtime preservation

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1186 bytes --]

On Wed, 25 Nov 2009 14:13:58 -0700
Denis Dupeyron <calchan@gentoo.org> wrote:
> A quick note to tell you that I have tried to look for examples of
> breakage due to how mtime preservation is currently implemented in
> portage. Unfortunately I didn't find anything, maybe because I'm not
> knowledgeable enough or because I can't afford to spend any more time
> on this. If any of you can provide an example then please do so.

Portage uses two ways of merging a file: os.rename() and the manual
way.

os.rename() correctly preserves mtimes.

Python's os.utime call, which is what Portage uses to preserve mtimes
for files that it installs the hard way, and which is not a wrapper for
the Unix utime call, uses an IEEE 754 double to handle timestamps. But
a double only gives sixteen accurate decimal digits, and you've got ten
to the left of the decimal point, leaving only six reliably preserved,
with the remaining three digits being corrupted.

Thus, packages can end up being installed with some files having
accurately-preserved mtimes, and some having corrupted mtimes, which
will lead to mtime comparisons giving incorrect results.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-dev] Re: mtime preservation

[Duncan]

Ciaran McCreesh posted on Wed, 25 Nov 2009 21:27:18 +0000 as excerpted:

> Portage uses two ways of merging a file: os.rename() and the manual way.
> 
> os.rename() correctly preserves mtimes.
> 
> Python's os.utime call, which is what Portage uses to preserve mtimes
> for files that it installs the hard way, and which is not a wrapper for
> the Unix utime call, uses an IEEE 754 double to handle timestamps. But a
> double only gives sixteen accurate decimal digits, and you've got ten to
> the left of the decimal point, leaving only six reliably preserved, with
> the remaining three digits being corrupted.
> 
> Thus, packages can end up being installed with some files having
> accurately-preserved mtimes, and some having corrupted mtimes, which
> will lead to mtime comparisons giving incorrect results.

That's a great explanation (thanks, I now know the details to the degree 
I'd be interested), but what was asked for was examples of breakage, aka 
actual bugs.  I seem to remember either you or someone else mentioning 
that there had been a couple issues already, and I'll take the claim at 
face value, but actual bug numbers would be nice. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: mtime preservation

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 655 bytes --]

On Wed, 25 Nov 2009 21:52:00 +0000 (UTC)
Duncan <1i5t5.duncan@cox.net> wrote:
> That's a great explanation (thanks, I now know the details to the
> degree I'd be interested), but what was asked for was examples of
> breakage, aka actual bugs.

Why? You can easily look and see that it's broken. Examples will merely
be dismissed as one-off cases that can be worked around, or as relying
upon a string of coincidences that will "obviously" never really
happen, right up until they do, at which point they'll be dismissed
with a WORKSFORME. What you have is a proof that it's broken, which is
far better than an example.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] mtime preservation

[Ulrich Mueller]

>>>>> On Wed, 25 Nov 2009, Ciaran McCreesh wrote:

> On Wed, 25 Nov 2009 21:52:00 +0000 (UTC)
> Duncan <1i5t5.duncan@cox.net> wrote:
>> That's a great explanation (thanks, I now know the details to the
>> degree I'd be interested), but what was asked for was examples of
>> breakage, aka actual bugs.

> Why? You can easily look and see that it's broken.

Only for a suitable definition of "broken".

> Examples will merely be dismissed as one-off cases that can be
> worked around, or as relying upon a string of coincidences that will
> "obviously" never really happen, right up until they do, at which
> point they'll be dismissed with a WORKSFORME.

Real examples would be issues like bugs 83877 [1] or 263387 [2].
Nothing that could be easily dismissed or worked around. Both issues
are fixed with Portage since a long time.

I don't know of any example where non-preservation of nanosecond
timestamps would cause problems.

> What you have is a proof that it's broken, which is far better than
> an example.

So we have a proven theorem, but unfortunately the cases where it is
applicable form an empty set. ;-)

Ulrich

[1] <http://bugs.gentoo.org/83877#c36>
[2] <http://bugs.gentoo.org/263387>

Re: [gentoo-dev] mtime preservation

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1118 bytes --]

On Wed, 25 Nov 2009 23:59:45 +0100
Ulrich Mueller <ulm@gentoo.org> wrote:
> Real examples would be issues like bugs 83877 [1] or 263387 [2].
> Nothing that could be easily dismissed or worked around. Both issues
> are fixed with Portage since a long time.

Yes, those are examples of packages relying upon something that is
undefined behaviour, and that behaves differently depending upon the
Portage version you use.

> I don't know of any example where non-preservation of nanosecond
> timestamps would cause problems.

Not non-preservation. Partial and inconsistent corruption.

srcfile gets installed with its mtime preserved, because it is installed
using rename.

objfile, whose mtime should be a bit later than srctime's, gets
installed with its mtime corrupted to be slightly less than it should
be, and less than srctime's, because it is installed the slow way.

A program checks that objfile's mtime is greater than or equal to
srcfile's mtime. That check will fail, and reinstalls will randomly fix
and break it depending upon which way the corruption goes.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] mtime preservation

[Zac Medico]

Ciaran McCreesh wrote:
> On Wed, 25 Nov 2009 23:59:45 +0100
> Ulrich Mueller <ulm@gentoo.org> wrote:
>> Real examples would be issues like bugs 83877 [1] or 263387 [2].
>> Nothing that could be easily dismissed or worked around. Both issues
>> are fixed with Portage since a long time.
> 
> Yes, those are examples of packages relying upon something that is
> undefined behaviour, and that behaves differently depending upon the
> Portage version you use.
> 
>> I don't know of any example where non-preservation of nanosecond
>> timestamps would cause problems.
> 
> Not non-preservation. Partial and inconsistent corruption.

Wouldn't "loss of precision" be a more accurate description? Of the
known packages which require timestamp preservation, do any of them
use sub-second precision in their timestamp comparisons?
-- 
Thanks,
Zac

Re: [gentoo-dev] mtime preservation

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 1660 bytes --]

On Wed, Nov 25, 2009 at 04:49:17PM -0800, Zac Medico wrote:
> Ciaran McCreesh wrote:
> > On Wed, 25 Nov 2009 23:59:45 +0100
> > Ulrich Mueller <ulm@gentoo.org> wrote:
> >> Real examples would be issues like bugs 83877 [1] or 263387 [2].
> >> Nothing that could be easily dismissed or worked around. Both issues
> >> are fixed with Portage since a long time.
> > 
> > Yes, those are examples of packages relying upon something that is
> > undefined behaviour, and that behaves differently depending upon the
> > Portage version you use.
> > 
> >> I don't know of any example where non-preservation of nanosecond
> >> timestamps would cause problems.
> > 
> > Not non-preservation. Partial and inconsistent corruption.
> 
> Wouldn't "loss of precision" be a more accurate description? Of the
> known packages which require timestamp preservation, do any of them
> use sub-second precision in their timestamp comparisons?

This discussion in generall is daft.  No package can rely on 
nanonsecond resolution for installation because the most common FS out 
there (ext3) does *second* level resolution only.  As such, I can 
pretty much gurantee there is *zero* packages out there that require 
nanosecond resolution for installation.

It's an academic discussion, and pointless.  We don't mandate the 
filesystems PMS implementations are run on- as such we cannot make a 
gurantee to ebuilds that nanosecond resolution is available.  It's 
daft to encode in the spec NS resolution when it's essentially 
impossible to gurantee it- I'm honestly not sure why we're having this 
discussion beyond the "python sucks" angle.

~brian

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] mtime preservation

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 3694 bytes --]

On Tue, Nov 24, 2009 at 10:21:06PM +0000, Ciaran McCreesh wrote:
> On Mon, 23 Nov 2009 15:19:00 -0800
> Brian Harring <ferringb@gmail.com> wrote:
> > Someone mind explaining to me why we're making mtime preservation so 
> > nasty?  Having to enumerate every pathway that requires mtime 
> > preservation is pretty arduous for the ebuild dev, meaning it's 
> > unlikely they'll get it right, leading to bugs.
> 
> It's not in the least bit nasty. It's requiring people to be explicit
> about special requirements.

I honestly wish that explicitness you're pushing for mtime were 
applied to all parts of mtime.

Why is this one special?  Two out of three do this already, and it 
works.  Paludis doesn't preserve mtime- and it's approach to randomly 
resetting mtimes is known to cause issues (bug 263387, python eclass 
having to do pyc/pyo compilation in postinst, etc).


> > The thing I'm not understanding here is that pkgcore since day one
> > has preserved mtime- I've yet to see any complaints about that nor
> > any issues caused by it.  Portage shifted over a year or two back,
> > same thing, haven't heard complaints.
> 
> You can't have been listening very hard, then. The complaint is that it
> results in files with dumb mtimes being merged to /.

As far as I can tell, no one but you is complaining about the mtimes.  

Your complaints primarily have been that tarball'd mtimes can pass 
through to the fs- I've not seen any comments from you in bug 264130 
that actually enumerate spots were this is anything but an aesthetic 
complaint.  Specifically your aesthetics- the pkgs in question have 
functioned fine w/ odd mtimes passing through.


> > I know it won't fly, but realistically stating "the package manager 
> > must preserve mtime- if there are instances where it does not (due to 
> > some feature, perhaps splitdebug or some form of compression) it is 
> > the package managers responsibility to ensure this does not break the 
> > ebuilds resultant merge/runtime invocation".
> > 
> > Via such wording an exemption is created and it's made clear it's the 
> > managers responsibility to keep things playing nice... if the manager 
> > can't do that, then the feature/functionality that is changing the 
> > mtime (resulting in the pkg on disk failing) must be fixed or 
> > disabled.
> > 
> > The nice thing about this wording is that it basically matches what 
> > the case is now, and what has worked for quite a few years.
> 
> Wording such as that just isn't suitable for a specification. It
> requires implementers to guess what future ebuilds are going to
> rely upon (and ebuilds regularly do rely upon weird quirks), and makes
> it impossible to produce a package manager that can be shown to be
> compliant.

That wording is explicit that PMS cannot lock down all potentials.  
This is no different then PMS's stance on VDB (and similar in 
reasoning)- ebuilds are reliant on vdb internals yet PMS explicitly 
ducked documenting it.  Double standards suck.

Further, drop the specificity BS.  You've intentionally left parts of 
PMS vague- trailing '/' for ebuild vars is a good example (ssmtp 
breaks if ${D} vs ${D}/, perl likely has the same breakage).  These 
lead to actual incompatibilities, and because PMS isn't explicit, 
there is no 'right' implementation for it.

The daft thing about you picking at this wording is that the scenarios 
you can come up with are all hypothetical.  Yes, a manager could 
randomize mtimes when it's doing splitdebug.

The manager wouldn't however because the only reason to do things like 
that is to intentionally cause issues.

~brian

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-dev] Re: mtime preservation

[Duncan]

Ciaran McCreesh posted on Wed, 25 Nov 2009 22:13:27 +0000 as excerpted:

> Examples will merely be
> dismissed as one-off cases that can be worked around, or as relying upon
> a string of coincidences that will "obviously" never really happen,
> right up until they do, at which point they'll be dismissed with a
> WORKSFORME. What you have is a proof that it's broken, which is far
> better than an example.

Actually, that "dismissed with WORKSFORME" strikes a chord, here.  There 
was a very strange parallel make bug that I filed that was closed with 
that.  I'd have really liked to see someone with some skill tackle it, as 
that was the only one I've ever seen that had striped fail and working 
zones, and I've have loved to see some logic as to why...  (If -j10 
failed, -j3 and -j15 might succeed, -l24 fail again, and -j33 succeed 
again...)  Unfortunately, flameeyes, the only one I know who really gets 
into such things, was fresh out of the hospital at the time, and I think 
it was beyond the maintainer's abilities, so WORKSFORME was about the 
best that could be done.  I've long since changed and changed again my 
makeopts, and don't remember the pkg now, tho I could probably find it in 
my old bug mail if I needed to.

So I gotta admit you have a point, with that one.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

[gentoo-dev] Re: mtime preservation

[Duncan]

Ulrich Mueller posted on Wed, 25 Nov 2009 23:59:45 +0100 as excerpted:

> Real examples would be issues like bugs 83877 [1] or 263387 [2]. Nothing
> that could be easily dismissed or worked around. Both issues are fixed
> with Portage since a long time.

Thanks.

> I don't know of any example where non-preservation of nanosecond
> timestamps would cause problems.

That's really what I was hoping for, a bug where the nanosecond times 
thing made a difference.  But Ciaran does have a point about such things 
being potentially closed as WORKSFORME or the like, as I've seen that in 
other cases, see my just earlier post.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] mtime preservation

[Zac Medico]

Brian Harring wrote:
> This discussion in generall is daft.  No package can rely on 
> nanonsecond resolution for installation because the most common FS out 
> there (ext3) does *second* level resolution only.  As such, I can 
> pretty much gurantee there is *zero* packages out there that require 
> nanosecond resolution for installation.

Your "guarantee" is filesystem-specific. However, if we can
establish that all known packages with timestamp preservation
requirements do their timestamp comparisons with 1-second
granularity, then we'll have a much safer (filesystem-independent)
assumption.
-- 
Thanks,
Zac

[gentoo-dev] Re: mtime preservation

[Duncan]

Brian Harring posted on Wed, 25 Nov 2009 17:14:27 -0800 as excerpted:

> On Wed, Nov 25, 2009 at 04:49:17PM -0800, Zac Medico wrote:
>> Ciaran McCreesh wrote:
>> > On Wed, 25 Nov 2009 23:59:45 +0100
>> > Ulrich Mueller <ulm@gentoo.org> wrote:
>> >> Real examples would be issues like bugs 83877 [1] or 263387 [2].
>> >> Nothing that could be easily dismissed or worked around. Both issues
>> >> are fixed with Portage since a long time.
>> > 
>> > Yes, those are examples of packages relying upon something that is
>> > undefined behaviour, and that behaves differently depending upon the
>> > Portage version you use.
>> > 
>> >> I don't know of any example where non-preservation of nanosecond
>> >> timestamps would cause problems.
>> > 
>> > Not non-preservation. Partial and inconsistent corruption.
>> 
>> Wouldn't "loss of precision" be a more accurate description? Of the
>> known packages which require timestamp preservation, do any of them use
>> sub-second precision in their timestamp comparisons?
> 
> This discussion in generall is daft.  No package can rely on nanonsecond
> resolution for installation because the most common FS out there (ext3)
> does *second* level resolution only.  As such, I can pretty much
> gurantee there is *zero* packages out there that require nanosecond
> resolution for installation.

One of the reasons I was asking for a bug number, was because there's two 
possible failure modes, and I was hoping reading them would clue me (or 
at least those who are making the decisions here) in on which one 
actually occurs.

Your posts suggest a mode where subsecond times are always truncated.  In 
such a case, there should be no issue.

But it's also possible that times are compared as-is.  If that's the 
case, then there should be no issue on second-resolution filesystems 
(such as ext3, in your example), because the filesystem would effectively 
truncate those, reducing to case-one, above.  But on higher resolution 
filesystems, there might very well be issues, due to the subtle double 
resolution issue Ciaran pointed out, especially when compared against 
renames where the mtimes were accurately preserved.

Now it could well be that it's possible to turn case two into case one by 
adding a simple option to the the command line or the like.  Or maybe 
not.  Again, that's why I wanted specific examples, so people could see 
if that were tried or not, and I would have to write all this up and 
possibly look like a moron if I'm getting it wrong, somehow.  But since 
the examples don't seem to be forthcoming, and in view of the IMO 
legitimate point about such bugs very possibly being closed as 
WORKSFORME, I guess I post this and hope someone can either explain that 
I have it all wrong, or say definitively that such command-line time 
truncation option workarounds are generally practical, or not, thus 
potentially affecting the direction of the debate.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

[gentoo-dev] Re: mtime preservation

[Duncan]

Zac Medico posted on Wed, 25 Nov 2009 21:26:59 -0800 as excerpted:

> Brian Harring wrote:
>> This discussion in generall is daft.  No package can rely on
>> nanonsecond resolution for installation because the most common FS out
>> there (ext3) does *second* level resolution only.  As such, I can
>> pretty much gurantee there is *zero* packages out there that require
>> nanosecond resolution for installation.
> 
> Your "guarantee" is filesystem-specific. However, if we can establish
> that all known packages with timestamp preservation requirements do
> their timestamp comparisons with 1-second granularity, then we'll have a
> much safer (filesystem-independent) assumption.

Thanks.  That's basically what I'm wondering also, well, if they do it 
with one second granularity, or if it's possible to make them do it that 
with with a simple command-line sed, adding an option, but here it's 
stated in different (fewer) words.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] mtime preservation

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 1240 bytes --]

On Wed, Nov 25, 2009 at 09:26:59PM -0800, Zac Medico wrote:
> Brian Harring wrote:
> > This discussion in generall is daft.  No package can rely on 
> > nanonsecond resolution for installation because the most common FS out 
> > there (ext3) does *second* level resolution only.  As such, I can 
> > pretty much gurantee there is *zero* packages out there that require 
> > nanosecond resolution for installation.
> 
> Your "guarantee" is filesystem-specific. However, if we can
> establish that all known packages with timestamp preservation
> requirements do their timestamp comparisons with 1-second
> granularity, then we'll have a much safer (filesystem-independent)
> assumption.

I've no complaints with mandating that ebuilds can rely on second 
level resolution- it's a valid gurantee as far as I'm concerned.  
Further any fs that can't offer it involves the user doing something 
seriously wonky, thus their problem if the ebuild horks.

If/when the major filesystems out there all do NS level resolution, 
and are in common deployment, I'd have no arguement extending the 
spec to mandating NS level resolution.  I've serious problems w/ 
mandating NS resolution in PMS prior to that however.

~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] mtime preservation

[Ulrich Mueller]

>>>>> On Thu, 26 Nov 2009, Ciaran McCreesh wrote:

>> Real examples would be issues like bugs 83877 [1] or 263387 [2].
>> Nothing that could be easily dismissed or worked around. Both issues
>> are fixed with Portage since a long time.

> Yes, those are examples of packages relying upon something that is
> undefined behaviour,

That was the reason why I filed bug 264130, in the first place. ;-)

> and that behaves differently depending upon the Portage version you
> use.

Fact is that all versions of Portage in the tree get it right for both
packages, but Paludis fails for them.

> objfile, whose mtime should be a bit later than srctime's, gets
> installed with its mtime corrupted to be slightly less than it
> should be, and less than srctime's, because it is installed the slow
> way.

The major usage cases where mtimes play a role (*.py vs *.pyc, *.lisp
vs *.fasl, and *.el vs *.elc) install source and object files in the
same directory. So why would the PM use different methods for
installing them?

> A program checks that objfile's mtime is greater than or equal to
> srcfile's mtime. That check will fail, and reinstalls will randomly
> fix and break it depending upon which way the corruption goes.

Again, you don't come up with a concrete example where such breakage
would occur.

I'm starting to get tired of this whole discussion. Maybe we should
just follow Calchan's suggestion and do nothing. Portage works, after
all.

Ulrich

Re: [gentoo-dev] mtime preservation

[Ulrich Mueller]

>>>>> On Wed, 25 Nov 2009, Zac Medico wrote:

>> Not non-preservation. Partial and inconsistent corruption.

> Wouldn't "loss of precision" be a more accurate description?

Yes. Or even "rounding".

> Of the known packages which require timestamp preservation, do any
> of them use sub-second precision in their timestamp comparisons?

I can speak for Emacs only, where the comparison code (in fileio.c) is
as follows:

  if (stat (SDATA (absname1), &st) < 0)
    return Qnil;

  mtime1 = st.st_mtime;

  if (stat (SDATA (absname2), &st) < 0)
    return Qt;

  return (mtime1 > st.st_mtime) ? Qt : Qnil;

It uses stat(2), therefore nanoseconds are ignored.

Ulrich

Re: [gentoo-dev] mtime preservation

[Ulrich Mueller]

>>>>> On Thu, 26 Nov 2009, Ulrich Mueller wrote:

>>>>> On Wed, 25 Nov 2009, Zac Medico wrote:
>> Of the known packages which require timestamp preservation, do any
>> of them use sub-second precision in their timestamp comparisons?

> I can speak for Emacs only, where the comparison code (in fileio.c) is
> as follows: [...]

> It uses stat(2), therefore nanoseconds are ignored.

I've also checked with dev-lisp/sbcl and dev-lisp/cmucl. They use
stat(2) or fstat(2), i.e. compare timestamps on the seconds level.

Ulrich

Re: [gentoo-dev] mtime preservation

[Łukasz Michalik]

[-- Attachment #1: Type: text/plain, Size: 381 bytes --]

On 08:59 2009-11-26 +0100, Ulrich Mueller wrote:
> 
> I can speak for Emacs only, where the comparison code (in fileio.c) is
> as follows:
> 
[snip]
> 
> It uses stat(2), therefore nanoseconds are ignored.
> 

So, when that is updated to require nanosecond resolution where
supported, we're off to having this discussion again?

-- 
Pozdrawiam,
Łukasz P. Michalik

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] mtime preservation

[Ulrich Mueller]

>>>>> On Thu, 26 Nov 2009, Łukasz Michalik wrote:

>> I can speak for Emacs only, where the comparison code (in fileio.c) is
>> as follows:
>> 
> [snip]
>> 
>> It uses stat(2), therefore nanoseconds are ignored.

> So, when that is updated to require nanosecond resolution where
> supported, we're off to having this discussion again?

No, then my second argument still holds, since *.el and *.elc are
installed into the same directory.

Ulrich

Re: [gentoo-dev] mtime preservation

[David Leverton]

2009/11/26 Brian Harring <ferringb@gmail.com>:
> This discussion in generall is daft.  No package can rely on
> nanonsecond resolution for installation because the most common FS out
> there (ext3) does *second* level resolution only.  As such, I can
> pretty much gurantee there is *zero* packages out there that require
> nanosecond resolution for installation.

It's possible that a package might assume that *if* nanosecond
timestamps were available on the build filesystem, then they can do
nanosecond comparisons on the installed filesystem.  That would be a
rather questionable assumption, and I'm not sure what we could do
about packages that do require that, but that's why we discuss things,
right?  To find solutions?

> It's an academic discussion, and pointless.  We don't mandate the
> filesystems PMS implementations are run on- as such we cannot make a
> gurantee to ebuilds that nanosecond resolution is available.  It's
> daft to encode in the spec NS resolution when it's essentially
> impossible to gurantee it

If we're not going to insist on preserving nanoseconds as far as
possible, then package managers should be required to explcitly clear
the nanoseconds part.  Otherwise we'll get situations where a package
works on the maintainer's machine, because it happens to use a package
manager, filesystem setup, configuration etc that happens to cause
nanoseconds to be preserved accurately, but it randomly and
mysteriously fails for users.  This is especially a concern in this
case because a maintainer can easily have no idea that he's
accidentally relying on nanoseconds being preserved - it's not
something the ebuild requests explicitly, and if the timestamps happen
to be set on his machine as the package expects, it'll just work
without any indication that there was a potential issue.

> I'm honestly not sure why we're having this discussion beyond the "python sucks" angle.

Because some of us want to find a correct solution, not just one that
no-one's complained about yet in the context of the few filetypes that
are currently known to benefit from preservation.  Shocking, I know.

[gentoo-dev] Re: mtime preservation

[Duncan]

David Leverton posted on Thu, 26 Nov 2009 12:35:53 +0000 as excerpted:

> If we're not going to insist on preserving nanoseconds as far as
> possible, then package managers should be required to explcitly clear
> the nanoseconds part.

While I'm not sure what it's going to do to install-times and thus 
whether it's practical, this /does/ at least in theory make quite some 
sense to me.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] mtime preservation

[David Leverton]

2009/11/26 Brian Harring <ferringb@gmail.com>:
> Why is this one special?  Two out of three do this already, and it
> works.

You mean "two out of three blatently ignored long-standing behaviour
and added a new feature without discussion or an EAPI bump".

> Paludis doesn't preserve mtime

You mean "Paludis carefully emulated Portage behaviour, and is now
somehow being blamed for the whole matter, to the extent that people
are trying to use threats (to the effect of 'I'm going to deliberately
break packages for Paludis users") to try to get their way in the
discussion".

> and it's approach to randomly resetting mtimes

There's nothing random about it.  Files' mtimes are reset to the
current time while being merged, just as Portage did for years.

Re: [gentoo-dev] mtime preservation

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 3158 bytes --]

Potentially just being a tool and taking the bait..

On Thu, Nov 26, 2009 at 12:41:55PM +0000, David Leverton wrote:
> 2009/11/26 Brian Harring <ferringb@gmail.com>:
> > Why is this one special?  Two out of three do this already, and it
> > works.
> 
> You mean "two out of three blatently ignored long-standing behaviour
> and added a new feature without discussion or an EAPI bump".

It was always on the todo to convert portage over to preserving mtime- 
this long predates PMS and even EAPI.

This stretches all the way back to '03/'04.  I'll also note that 
pkgcore from the getgo preserved mtime, back when it was called 
'portage'.  Ahh, the good old days before svn pissed me off enough to 
move it outside of gentoo.

Beyond that, I presume your intention is to stir things up- PMS went 
out of it's way to explicitly leave it up to the manager if they 
preserve mtimes.  Meaning portage hasn't done anything wrong in 
changing it's behaviour.

It's a bit ironic really.  Y'all didn't want mtime in there so it was 
left unspecified.  Now you're complaining that portage changed it's 
behaviour (2+ years after the fact) as an arguement against adding 
mtime preservation into the next eapi.


> > Paludis doesn't preserve mtime
> 
> You mean "Paludis carefully emulated Portage behaviour, and is now
> somehow being blamed for the whole matter, to the extent that people
> are trying to use threats (to the effect of 'I'm going to deliberately
> break packages for Paludis users") to try to get their way in the
> discussion".

I mean paludis doesn't preserve mtimes.  People aren't going out of 
their way to break paludis (and claiming so is just trolling).

Breakages happen because the common sense assumption devs have 
(preservation of mtime) aren't actually encoded as a format standard.  
Further, there are quite a few postinst hacks (and quite a few that 
don't work all that well) working around the lack of mtime 
preservation.


> > and it's approach to randomly resetting mtimes
> 
> There's nothing random about it.  Files' mtimes are reset to the
> current time while being merged, just as Portage did for years.

Just because portage did something for a few years, does not make it 
right (this is something the PMS folk have been claiming since day 
one).  So... that arguement is invalidated by your own statements.

I label it as 'randomly' due to the fact it's contrary to what ebuild 
developers expect- it was a flaw when portage was doing it, it's a 
flaw that paludis does it due to it unnecessarily creating a gotcha 
for ebuild developers.  Most importantly, no one has brought up a 
single instance of mtime preservation *breaking* things- the only 
criticism leveled has been ciaranms aesthetic complaints about 
ca-certificates and older timestamps making their way to the FS.
Hence labeling it unnecessarily.


Basically, y'all need to provide actual examples of this causing 
issues- right now the responses are either quite flammable, or lacking 
in technical arguements.  More obstructionist then useful for a 
technical discourse.

~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] mtime preservation

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 1185 bytes --]

On Thu, Nov 26, 2009 at 12:35:53PM +0000, David Leverton wrote:
> 2009/11/26 Brian Harring <ferringb@gmail.com>:
> > It's an academic discussion, and pointless.  We don't mandate the
> > filesystems PMS implementations are run on- as such we cannot make a
> > gurantee to ebuilds that nanosecond resolution is available.  It's
> > daft to encode in the spec NS resolution when it's essentially
> > impossible to gurantee it
> 
> If we're not going to insist on preserving nanoseconds as far as
> possible, then package managers should be required to explcitly clear
> the nanoseconds part.

I'd had the same thought actually.  If folks really consider it to be 
needed, then jam it in as far as I'm concerned.  It has the potential 
to remove a heisenbug or two.

Presuming I'm reading posixmodule correctly from the python source, 
pkgcore already does second truncation (reliance on utime) and portage 
does it whenever movefile is invoked- any direct mv invocations 
obviously do not however.

Basically, if stat_float_mtimes is False (the case for all portage 
context and most of pkgcore context), os.utime auto truncates to 
second precision.

~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] mtime preservation

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 3378 bytes --]

On Wed, 25 Nov 2009 17:28:33 -0800
Brian Harring <ferringb@gmail.com> wrote:
> > It's not in the least bit nasty. It's requiring people to be
> > explicit about special requirements.
> 
> I honestly wish that explicitness you're pushing for mtime were 
> applied to all parts of mtime.
> 
> Why is this one special?  Two out of three do this already, and it 
> works.  Paludis doesn't preserve mtime- and it's approach to randomly 
> resetting mtimes is known to cause issues (bug 263387, python eclass 
> having to do pyc/pyo compilation in postinst, etc).

No, two out of three do a half-arsed job about it. If we're going to
make it behaviour upon which packages can rely, we should be doing so
properly to avoid the whole mess cropping up again as people start
following the newer POSIX standard.

> > You can't have been listening very hard, then. The complaint is
> > that it results in files with dumb mtimes being merged to /.
> 
> As far as I can tell, no one but you is complaining about the
> mtimes.  
> 
> Your complaints primarily have been that tarball'd mtimes can pass 
> through to the fs- I've not seen any comments from you in bug 264130 
> that actually enumerate spots were this is anything but an aesthetic 
> complaint.  Specifically your aesthetics- the pkgs in question have 
> functioned fine w/ odd mtimes passing through.

Yes, again, it's about doing it properly.

> > Wording such as that just isn't suitable for a specification. It
> > requires implementers to guess what future ebuilds are going to
> > rely upon (and ebuilds regularly do rely upon weird quirks), and
> > makes it impossible to produce a package manager that can be shown
> > to be compliant.
> 
> That wording is explicit that PMS cannot lock down all potentials.  

Ok, so PMS would effectively say "ebuilds must not rely upon any
particular behaviour", which in turn means any ebuild that breaks with
older Portage / Paludis is broken. That's not what you're after, is it?

> This is no different then PMS's stance on VDB (and similar in 
> reasoning)- ebuilds are reliant on vdb internals yet PMS explicitly 
> ducked documenting it.  Double standards suck.

That one's being fixed, slowly...

> Further, drop the specificity BS.  You've intentionally left parts of 
> PMS vague- trailing '/' for ebuild vars is a good example (ssmtp 
> breaks if ${D} vs ${D}/, perl likely has the same breakage).  These 
> lead to actual incompatibilities, and because PMS isn't explicit, 
> there is no 'right' implementation for it.

These are entirely unrelated. Any ebuild that relies upon the slash
either way is broken. But we can't realistically do that for mtimes,
since it would involve modifying *the package that gets installed*, and
doing so by patching sources to use things POSIX considers deprecated.
It's not just an ebuild tweak.

> The daft thing about you picking at this wording is that the
> scenarios you can come up with are all hypothetical.  Yes, a manager
> could randomize mtimes when it's doing splitdebug.
> 
> The manager wouldn't however because the only reason to do things
> like that is to intentionally cause issues.

It's about doing it right, so we don't get bitten by it later on when
everyone's using nanosecond-resolution timestamps like POSIX suggests.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] mtime preservation

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 708 bytes --]

On Wed, 25 Nov 2009 16:49:17 -0800
Zac Medico <zmedico@gentoo.org> wrote:
> > Not non-preservation. Partial and inconsistent corruption.
> 
> Wouldn't "loss of precision" be a more accurate description? Of the
> known packages which require timestamp preservation, do any of them
> use sub-second precision in their timestamp comparisons?

No. When you're dealing with decimals, floating point rounding isn't a
simple truncation:

>>> 1234567890.999999999
1234567891.0

nor is it a rounding:

>>> 1234567890.111111111
1234567890.1111112

The former is particularly bad, since POSIX deals with legacy functions
by ignoring the nanosecond part, not by rounding it.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] mtime preservation

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 742 bytes --]

On Wed, 25 Nov 2009 17:14:27 -0800
Brian Harring <ferringb@gmail.com> wrote:
> This discussion in generall is daft.  No package can rely on 
> nanonsecond resolution for installation because the most common FS
> out there (ext3) does *second* level resolution only.  As such, I can 
> pretty much gurantee there is *zero* packages out there that require 
> nanosecond resolution for installation.

We aren't discussing requiring nanosecond resolution. We're discussing
not getting corrupted timestamps, which is what happens.

If you're building on a filesystem which does support nanosecond
resolution timestamps, Portage will screw up the *second* part of the
timestamp as well as the nanosecond part.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] mtime preservation

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1576 bytes --]

On Thu, 26 Nov 2009 08:59:44 +0100
Ulrich Mueller <ulm@gentoo.org> wrote:
> >> Not non-preservation. Partial and inconsistent corruption.
> 
> > Wouldn't "loss of precision" be a more accurate description?
> 
> Yes. Or even "rounding".

No, corruption, including of the seconds part, is the right way to
describe it.

> > Of the known packages which require timestamp preservation, do any
> > of them use sub-second precision in their timestamp comparisons?
> 
> I can speak for Emacs only, where the comparison code (in fileio.c) is
> as follows:
> 
>   if (stat (SDATA (absname1), &st) < 0)
>     return Qnil;
> 
>   mtime1 = st.st_mtime;
> 
>   if (stat (SDATA (absname2), &st) < 0)
>     return Qt;
> 
>   return (mtime1 > st.st_mtime) ? Qt : Qnil;
> 
> It uses stat(2), therefore nanoseconds are ignored.

Uh, two things.

First, Portage sometimes corrupts the second part too.

Second, [1]:

> For compatibility with earlier versions of this standard, the st_atime
> macro shall be defined with the value st_atim.tv_sec. Similarly,
> st_ctime and st_mtime shall be defined as macros with the values
> st_ctim.tv_sec and st_mtim.tv_sec, respectively.

Note that POSIX considers st_mtime to be legacy, and thinks you should
be using st_mtim instead.

Do you want this to break when Emacs starts following the newer
standard, or do you want to get this right now so we won't have to have
the discussion again a year later?

[1]: http://www.opengroup.org/onlinepubs/9699919799/basedefs/sys_stat.h.html

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] mtime preservation

[David Leverton]

On Thursday 26 November 2009 13:21:43 Brian Harring wrote:
> It was always on the todo to convert portage over to preserving mtime-
> this long predates PMS and even EAPI.

Like, for example, use deps?  Yet somehow we managed to introduce those in a 
new EAPI, instead of retroactively adding them to all EAPIs.  Why should 
mtimes be different?

> Beyond that, I presume your intention is to stir things up

I suppose you have the right to presume whatever you want.

> It's a bit ironic really.  Y'all didn't want mtime in there so it was
> left unspecified.  Now you're complaining that portage changed it's
> behaviour (2+ years after the fact) as an arguement against adding
> mtime preservation into the next eapi.

I'm certainly not arguing against adding it, I just want it to be done 
properly, and I'm expressing distaste at people trying to blame Paludis for 
the fact that it's not as easy as some people want it to be.

> I mean paludis doesn't preserve mtimes.  People aren't going out of
> their way to break paludis (and claiming so is just trolling).

I don't think anyone's talking about changing packages purely for the sake of 
break Paludis and for no other reason, but people have been talking about 
making changes that they know will break Paludis.  (Whether they've actually 
done so is a different question, but the talk, and people blaming Paludis 
both when it behaves differently from Portage and when we've taken care to 
make it behave the same as Portage only for Portage to randomly change, are 
quite irritating.)

> Just because portage did something for a few years, does not make it
> right (this is something the PMS folk have been claiming since day
> one).  So... that arguement is invalidated by your own statements.

PMS tries to document Portage behaviour as long as it's not clearly 
unreasonable and unspecifiable.  Discarding mtimes is suboptimal behaviour, 
yes, but it's coherent enough that it can't be considered a blatent bug.  
Much like the lack of use deps in older EAPIs.

Re: [gentoo-dev] Re: mtime preservation

[Zac Medico]

Duncan wrote:
> David Leverton posted on Thu, 26 Nov 2009 12:35:53 +0000 as excerpted:
> 
>> If we're not going to insist on preserving nanoseconds as far as
>> possible, then package managers should be required to explcitly clear
>> the nanoseconds part.
> 
> While I'm not sure what it's going to do to install-times and thus 
> whether it's practical, this /does/ at least in theory make quite some 
> sense to me.

This seems like a pretty reasonable (and easy) thing to do. All that
we have to do is round the mtime value before it's passed into the
os.utime() call.
-- 
Thanks,
Zac

[gentoo-dev] Re: mtime preservation

[Peter Hjalmarsson]

tor 2009-11-26 klockan 05:04 +0000 skrev Duncan:
> Ciaran McCreesh posted on Wed, 25 Nov 2009 22:13:27 +0000 as excerpted:
> 
> > Examples will merely be
> > dismissed as one-off cases that can be worked around, or as relying upon
> > a string of coincidences that will "obviously" never really happen,
> > right up until they do, at which point they'll be dismissed with a
> > WORKSFORME. What you have is a proof that it's broken, which is far
> > better than an example.
> 
> Actually, that "dismissed with WORKSFORME" strikes a chord, here.  There 
> was a very strange parallel make bug that I filed that was closed with 
> that.  I'd have really liked to see someone with some skill tackle it, as 
> that was the only one I've ever seen that had striped fail and working 
> zones, and I've have loved to see some logic as to why...  (If -j10 
> failed, -j3 and -j15 might succeed, -l24 fail again, and -j33 succeed 
> again...)  Unfortunately, flameeyes, the only one I know who really gets 
> into such things, was fresh out of the hospital at the time, and I think 
> it was beyond the maintainer's abilities, so WORKSFORME was about the 
> best that could be done.  I've long since changed and changed again my 
> makeopts, and don't remember the pkg now, tho I could probably find it in 
> my old bug mail if I needed to.
> 
> So I gotta admit you have a point, with that one.
> 

That is just a *really* bad maintainer. Or a *really* bad bugreport. Or
something that showed you already had a *really* screwed up system.
What is the bug number?
Did you post your emerge --info and a *full* build.log?

Also information about what makeopts works and not gives nothing, that
just becomes noise. Becouse what makeopts works and not depends most of
the time on a race condition in the makefiles and can because of that
also depend on stuff like your current system load.