[gentoo-council] Agenda (draft) for November meeting 2009-11-09

[gentoo-council] Agenda (draft) for November meeting 2009-11-09

[Ulrich Mueller]

Find below a proposed agenda for our next meeting.

I'll post it to gentoo-council and gentoo-dev-announce this Friday,
unless there are objections.

Ulrich


1. Intro (10 minutes, including late arrivals)
   1.1. Make sure that at least a couple people are logging the meeting.
   1.2. Roll call.
   1.3. Any volunteers to chair this meeting?

2. Approve summary of October meeting (5 minutes)
   (in case it will be ready then)

3. Follow-ups from previous meeting (15 minutes)
   3.1. EAPI 3 status (zmedico, ulm)
   3.2. Upgrade path for old systems (leio, solar)
        Should important system packages (e.g. Python) be reverted to
        EAPI 0, in order to provide a defined upgrade path?

4. Prefix support in main Portage tree (grobian) (20 minutes)
   Fabian Groffen proposes that package managers should define the
   EPREFIX, EROOT, and ED variables (see [1] and replies).
   4.1. Does the council generally support this idea?
   4.2. Can it still be included in EAPI 3, or even in a fast EAPI 2.1?

5. AOB / open floor (10 minutes)

[1] http://archives.gentoo.org/gentoo-dev/msg_2a62689c71f95e4de5699a330b8b5524.xml

Re: [gentoo-council] Agenda (draft) for November meeting 2009-11-09

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1144 bytes --]

On Tue, 3 Nov 2009 18:03:20 +0100
Ulrich Mueller <ulm@gentoo.org> wrote:
> Find below a proposed agenda for our next meeting.

Could you add in something like:

"Agree upon a wording for PMS for the mtime modification change
introduced to EAPI 3 last time"

please? Due to insufficient clarity in the proposal, the PMS team hasn't
been able to come up with a wording that wouldn't either require
changes to Portage (which appears to be against the Council's intent)
or that would permit behaviour currently seen as undesirable. As I
understand it, the issues are:

* What's to be done about sub-second timestamps? What about cases where
  the build filesystem supports them but the root filesystem doesn't?

* For which files must mtimes be preserved, and which can be modified?

* Is it the intent of this proposal to prevent package managers from
  automatically rewriting, say, #!/usr/bin/python to
  #!/opt/gentoo/bin/python if prefix is being used?

Or, a solution before the meeting would be fine too. It's just I don't
think this is something the PMS team is able to resolve on its own.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-council] mtime preservation (was: Agenda (draft) for November meeting 2009-11-09)

[Ulrich Mueller]

>>>>> On Tue, 3 Nov 2009, Ciaran McCreesh wrote:

> Due to insufficient clarity in the proposal, the PMS team hasn't
> been able to come up with a wording that wouldn't either require
> changes to Portage (which appears to be against the Council's
> intent) or that would permit behaviour currently seen as
> undesirable. As I understand it, the issues are:

> * What's to be done about sub-second timestamps? What about cases
>   where the build filesystem supports them but the root filesystem
>   doesn't?

Obviously we cannot guarantee anything below the seconds level because
of limitations in the underlying filesystems or software (e.g., tar
for binpkgs). But is there a reason for limiting it further, i.e. not
preserving sub-second timestamps if they are supported by both
filesystems?

> * For which files must mtimes be preserved, and which can be modified?

> * Is it the intent of this proposal to prevent package managers from
>   automatically rewriting, say, #!/usr/bin/python to
>   #!/opt/gentoo/bin/python if prefix is being used?

Part of the problem (what you call "insufficient clarity") is that the
proposal's original intention was to cover only the merge process,
i.e. what takes place after pkg_preinst. Whereas you want to extend it
to include everything that is taking place after src_install (for
Portage, prepstrip and whatnot).

If you limit it to the final merge process from D to ROOT, then the
answer is easy, namely mtimes of all regular files must be preserved.

Ulrich

[gentoo-council] Re: mtime preservation (was: Agenda (draft) for November meeting 2009-11-09)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 2379 bytes --]

On Wed, 4 Nov 2009 16:07:30 +0100
Ulrich Mueller <ulm@gentoo.org> wrote:
> Obviously we cannot guarantee anything below the seconds level because
> of limitations in the underlying filesystems or software (e.g., tar
> for binpkgs). But is there a reason for limiting it further, i.e. not
> preserving sub-second timestamps if they are supported by both
> filesystems?

So far as I can see, if they're fully supported on both filesystems,
Portage sometimes preserves nanosecond-resolution timestamps and
sometimes doesn't. So, requiring nanosecond-resolution timestamp
preservation where possible will need Portage changes. But not
requiring it means we'll go through this whole thing again as packages
start using more accurate timestamps for cache validation things (and
yes, this will happen -- we've already hit make-related issues on this
front).

> > * For which files must mtimes be preserved, and which can be
> > modified?
> 
> > * Is it the intent of this proposal to prevent package managers from
> >   automatically rewriting, say, #!/usr/bin/python to
> >   #!/opt/gentoo/bin/python if prefix is being used?
> 
> Part of the problem (what you call "insufficient clarity") is that the
> proposal's original intention was to cover only the merge process,
> i.e. what takes place after pkg_preinst. Whereas you want to extend it
> to include everything that is taking place after src_install (for
> Portage, prepstrip and whatnot).
> 
> If you limit it to the final merge process from D to ROOT, then the
> answer is easy, namely mtimes of all regular files must be preserved.

What I want is for the proposal to be sufficiently specific that it
covers exactly what the package manager can and cannot do, and what
ebuilds can and cannot rely upon happening. If you require mtime
preservation between pkg_preinst and the merge to /, the package
manager can just screw things up (by implementing reasonable features)
elsewhere. It is by no means clear to me that merely requiring mtime
preservation from after pkg_preinst to before pkg_postinst, and
allowing arbitrary mtime tinkering elsewhere, is what is desired.

As an example for the above, is it legal for a package manager to
rewrite any mtime that is before the start of the build process if it
does it after src_install but before pkg_preinst?

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-council] Re: mtime preservation

[Zac Medico]

Ciaran McCreesh wrote:
> On Wed, 4 Nov 2009 16:07:30 +0100
> Ulrich Mueller <ulm@gentoo.org> wrote:
>> Obviously we cannot guarantee anything below the seconds level because
>> of limitations in the underlying filesystems or software (e.g., tar
>> for binpkgs). But is there a reason for limiting it further, i.e. not
>> preserving sub-second timestamps if they are supported by both
>> filesystems?
> 
> So far as I can see, if they're fully supported on both filesystems,
> Portage sometimes preserves nanosecond-resolution timestamps and
> sometimes doesn't. So, requiring nanosecond-resolution timestamp
> preservation where possible will need Portage changes.

I think it always preserves them, as long as you have at least
python-2.5 since that is required for floating-point mtime support.
-- 
Thanks,
Zac

Re: [gentoo-council] Re: mtime preservation

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 751 bytes --]

On Wed, 04 Nov 2009 13:12:37 -0800
Zac Medico <zmedico@gentoo.org> wrote:
> > So far as I can see, if they're fully supported on both filesystems,
> > Portage sometimes preserves nanosecond-resolution timestamps and
> > sometimes doesn't. So, requiring nanosecond-resolution timestamp
> > preservation where possible will need Portage changes.
> 
> I think it always preserves them, as long as you have at least
> python-2.5 since that is required for floating-point mtime support.

Mm, I can't see the code for that. So far as I can see, for the
non-fast case you're using stat.st_mtime and os.utime, which assuming
they correspond to the POSIX things of the same name, are
second-resolution. What am I missing?

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-council] Re: mtime preservation

[Zac Medico]

[-- Attachment #1: Type: text/plain, Size: 967 bytes --]

Ciaran McCreesh wrote:
> On Wed, 04 Nov 2009 13:12:37 -0800
> Zac Medico <zmedico@gentoo.org> wrote:
>>> So far as I can see, if they're fully supported on both filesystems,
>>> Portage sometimes preserves nanosecond-resolution timestamps and
>>> sometimes doesn't. So, requiring nanosecond-resolution timestamp
>>> preservation where possible will need Portage changes.
>> I think it always preserves them, as long as you have at least
>> python-2.5 since that is required for floating-point mtime support.
> 
> Mm, I can't see the code for that. So far as I can see, for the
> non-fast case you're using stat.st_mtime and os.utime, which assuming
> they correspond to the POSIX things of the same name, are
> second-resolution. What am I missing?

Ah, I guess you're right. The documentation led me to believe that
os.utime would provide nanosecond-resolution on platforms that
support it, but a simple test case seems to indicate that it does not.
-- 
Thanks,
Zac

[-- Attachment #2: utime_test.py --]
[-- Type: text/x-python, Size: 193 bytes --]

#!/usr/bin/env python

import os
import sys
import tempfile

f = tempfile.NamedTemporaryFile()
t = 0.1
os.utime(f.name, (t, t))
sys.stdout.write(str(os.stat(f.name).st_mtime) + '\n')
f.close()

Re: [gentoo-council] Re: mtime preservation

[Ulrich Mueller]

>>>>> On Wed, 04 Nov 2009, Zac Medico wrote:

> Ah, I guess you're right. The documentation led me to believe that
> os.utime would provide nanosecond-resolution on platforms that
> support it, but a simple test case seems to indicate that it does not.

Is there an os.utimes in Python? utime(2) is marked as obsolete in
POSIX.1-2008 (says the man page).

Ulrich

Re: [gentoo-council] Re: mtime preservation

[Zac Medico]

Ulrich Mueller wrote:
>>>>>> On Wed, 04 Nov 2009, Zac Medico wrote:
> 
>> Ah, I guess you're right. The documentation led me to believe that
>> os.utime would provide nanosecond-resolution on platforms that
>> support it, but a simple test case seems to indicate that it does not.
> 
> Is there an os.utimes in Python? utime(2) is marked as obsolete in
> POSIX.1-2008 (says the man page).

There is no separate os.utimes function, but it looks like the patch
on this bug might make os.utime behave as desired:

  http://bugs.python.org/issue3425
-- 
Thanks,
Zac

Re: [gentoo-council] Re: mtime preservation

[Ulrich Mueller]

>>>>> On Wed, 04 Nov 2009, Zac Medico wrote:

>>> Ah, I guess you're right. The documentation led me to believe that
>>> os.utime would provide nanosecond-resolution on platforms that
>>> support it, but a simple test case seems to indicate that it does not.
>> 
>> Is there an os.utimes in Python? utime(2) is marked as obsolete in
>> POSIX.1-2008 (says the man page).

> There is no separate os.utimes function, but it looks like the patch
> on this bug might make os.utime behave as desired:

>   http://bugs.python.org/issue3425

So let's declare it as a Python bug and include sub-second timestamps
(where possible with the filesystem) in the spec?

Anyway, I see this as a very minor issue.

Ulrich

[gentoo-council] Re: mtime preservation

[Ulrich Mueller]

>>>>> On Wed, 4 Nov 2009, Ciaran McCreesh wrote:

>> Part of the problem (what you call "insufficient clarity") is that
>> the proposal's original intention was to cover only the merge
>> process, i.e. what takes place after pkg_preinst. Whereas you want
>> to extend it to include everything that is taking place after
>> src_install (for Portage, prepstrip and whatnot).
>> 
>> If you limit it to the final merge process from D to ROOT, then the
>> answer is easy, namely mtimes of all regular files must be
>> preserved.

> What I want is for the proposal to be sufficiently specific that it
> covers exactly what the package manager can and cannot do, and what
> ebuilds can and cannot rely upon happening. If you require mtime
> preservation between pkg_preinst and the merge to /, the package
> manager can just screw things up (by implementing reasonable
> features) elsewhere. It is by no means clear to me that merely
> requiring mtime preservation from after pkg_preinst to before
> pkg_postinst, and allowing arbitrary mtime tinkering elsewhere, is
> what is desired.

Can you try to find a suitable wording? Otherwise, it's not clear to
me how the council could resolve the issue during the next meeting.

(And as my suggested wording [1] caused some unfortunate discussion,
I don't feel like I should come up with a new one myself.)

> As an example for the above, is it legal for a package manager to
> rewrite any mtime that is before the start of the build process if
> it does it after src_install but before pkg_preinst?

So you really want this? ;-) My personal opinion is that it wouldn't
break anything and we could therefore declare it as an allowed QA
measure. And if it takes place before pkg_preinst then the ebuild
could override it in special cases.

But please be aware that the council (October meeting) has voted
against this sort of mtime fixup.

Ulrich

[1] <http://bugs.gentoo.org/264130#c39> and following comments

Re: [gentoo-council] Re: mtime preservation

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1028 bytes --]

On Thu, 5 Nov 2009 11:24:05 +0100
Ulrich Mueller <ulm@gentoo.org> wrote:
> > What I want is for the proposal to be sufficiently specific that it
> > covers exactly what the package manager can and cannot do, and what
> > ebuilds can and cannot rely upon happening. If you require mtime
> > preservation between pkg_preinst and the merge to /, the package
> > manager can just screw things up (by implementing reasonable
> > features) elsewhere. It is by no means clear to me that merely
> > requiring mtime preservation from after pkg_preinst to before
> > pkg_postinst, and allowing arbitrary mtime tinkering elsewhere, is
> > what is desired.
> 
> Can you try to find a suitable wording? Otherwise, it's not clear to
> me how the council could resolve the issue during the next meeting.

I've been thinking about this, and I honestly don't think it's
achievable without losing at least one of the aims. So far as I can
see, the Council's goals are mutually contradictory on this one.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-council] Re: mtime preservation

[Denis Dupeyron]

Let the clueless dev that I am summarize the situation. Please correct
me if I'm wrong, I think that will help more people than just me. In
the end I'll propose 2 wordings for PMS.

I understand that the issue with requiring mtime preservation is that
after unpacking the tarball(s) some files can have absurd dates, like
January 1st 1970 or else. One easy solution to this would be to reset
mtimes at merge time which will take care of everything. However, I
know of a package which requires mtime preservation from src_compile
to the moment it's merged to ${ROOT}. And if there's a package like
this in my very small corner of Gentoo there must be more elsewhere,
and there probably will be more in the future.

So here are the 2 wordings I propose.

1 - All mtimes from the package's sources are preserved until files
are merged to ${ROOT}. In case some mtimes are absurd and/or
unsuitable, they're considered a bug and it's the maintainer's
responsibility to fix them and report back upstream.

2 - All mtimes from the package's sources are reset to current time
between the src_unpack phase and the next one. That is before
src_configure when EAPI<2 and before src_prepare when EAPI>=2. mtimes
are then preserved until files are merged to ${ROOT}.

How's that? Am I missing something?

Denis.

Re: [gentoo-council] Re: mtime preservation

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 2400 bytes --]

On Mon, 9 Nov 2009 08:18:15 -0700
Denis Dupeyron <calchan@gentoo.org> wrote:
> So here are the 2 wordings I propose.
> 
> 1 - All mtimes from the package's sources are preserved until files
> are merged to ${ROOT}. In case some mtimes are absurd and/or
> unsuitable, they're considered a bug and it's the maintainer's
> responsibility to fix them and report back upstream.
> 
> 2 - All mtimes from the package's sources are reset to current time
> between the src_unpack phase and the next one. That is before
> src_configure when EAPI<2 and before src_prepare when EAPI>=2. mtimes
> are then preserved until files are merged to ${ROOT}.
> 
> How's that? Am I missing something?

Neither of those are what Portage currently does. It is my
understanding that the Council doesn't want to require any changes to
Portage behaviour for this.

What Portage currently does is, more or less, to preserve mtimes (but
not necessarily sub-second mtimes, even where supported) from every
part of the build and merge processes, except for an arbitrary set of
files that it modifies outside of the ebuild's control. Examples of
such files include, but are not limited to, executables and shared
objects.

The problem lies in the exceptions. Either we word PMS so vaguely that
it's legal for the package manager to clobber any mtime (thus defeating
the point of guaranteeing preservation at all), or we include long,
convoluted wording describing exactly the files Portage currently
alters (thus preventing reasonable-looking future changes), or we make
the mtime preservation guarantee only cover a small portion of time
such as between pkg_preinst and the merge phase (which disallows some
of the things people want to be able to do with mtime preservation).

None of these strike me as reasonable solutions, and I am still of the
opinion that this isn't something that should have been rushed into
EAPI 3 without people having spent time taking a good look at what all
mtime preservation would be needed to do, what Portage currently does
(since it does not, as some people have claimed, preserve mtimes) and
what should be done about packages that end up installing files with
absurd mtimes. If guaranteed mtime preservation is a desired feature, we
should be designing it around what developers need, not around what
Portage currently does.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-council] Re: mtime preservation

[Petteri Räty]

[-- Attachment #1: Type: text/plain, Size: 613 bytes --]

Ciaran McCreesh wrote:

> 
> Neither of those are what Portage currently does. It is my
> understanding that the Council doesn't want to require any changes to
> Portage behaviour for this.
> 

Ideally we would get by without changes to Portage but the goal isn't
avoiding to change Portage but to make sure the packages that require
certain behavior for mtimes work if it requires changes to Portage in
order to work then so be it. They are not likely to be huge so they can
be done for EAPI 3 or if they are just postpone this to EAPI 4 like I
suggested in the first place.

Regards,
Petteri


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [gentoo-council] Re: mtime preservation

[Ulrich Mueller]

>>>>> On Mon, 9 Nov 2009, Ciaran McCreesh wrote:

> What Portage currently does is, more or less, to preserve mtimes
> [...] from every part of the build and merge processes, except for
> an arbitrary set of files that it modifies outside of the ebuild's
> control.

And exactly this should be the guideline: A modified file may have its
mtime updated, an unmodified file has its mtime preserved. But for
some reason this is not wanted and was countered with arguments such
as "the PM could do two passes of rot-13 encryption".

> None of these strike me as reasonable solutions, and I am still of
> the opinion that this isn't something that should have been rushed
> into EAPI 3

The respective bug was opened in March, more than 7 months ago. That's
not what I would call "rush".

> without people having spent time taking a good look at what all
> mtime preservation would be needed to do, what Portage currently
> does (since it does not, as some people have claimed, preserve
> mtimes) and what should be done about packages that end up
> installing files with absurd mtimes.

> If guaranteed mtime preservation is a desired feature, we should be
> designing it around what developers need, not around what Portage
> currently does.

What if the two coincide?

Ulrich

Re: [gentoo-council] Re: mtime preservation

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 3231 bytes --]

On Mon, 9 Nov 2009 17:37:26 +0100
Ulrich Mueller <ulm@gentoo.org> wrote:
> > What Portage currently does is, more or less, to preserve mtimes
> > [...] from every part of the build and merge processes, except for
> > an arbitrary set of files that it modifies outside of the ebuild's
> > control.
> 
> And exactly this should be the guideline: A modified file may have its
> mtime updated, an unmodified file has its mtime preserved. But for
> some reason this is not wanted and was countered with arguments such
> as "the PM could do two passes of rot-13 encryption".

It's not wanted because it's utterly useless as a specification. It
means ebuilds can't rely upon any file having its mtime preserved,
because a package manager can decide, reasonably or otherwise, to
modify any file it likes.

The double-rot-13 argument is merely a compact illustration of why
leaving it open is unworkable. If you prefer a more subtle argument,
consider "the package manager might want to rewrite #!/usr/bin/python
lines automatically in Prefix environments", but be aware that any
example of that nature is prone to being sidetracked into arguments
about "but Prefix won't do it that way", entirely missing the wider
point.

Now, you could claim, as you do, that the proposal should only matter
for pkg_preinst to merge mtime preservation, and so rewriting of this
kind is fine if it's done straight after src_install. But we have yet
to establish whether every ebuild mtime-related operation is best done
exclusively in pkg_preinst (and this is far from a straight-forward
question, when you have to start thinking about binaries and about
cleanup), and it's not safe to jump to the conclusion that allowing the
package manager to modify things straight after src_install is safe and
in line with what best suits developers.

> > None of these strike me as reasonable solutions, and I am still of
> > the opinion that this isn't something that should have been rushed
> > into EAPI 3
> 
> The respective bug was opened in March, more than 7 months ago. That's
> not what I would call "rush".

And it was left out of EAPI 3 originally because no appropriate solution
had been found. As you're aware, there's a finite amount of time that
can be spent working on these things, and thus-far not enough of that
time has been devoted to turning this into a solvable problem.

> > without people having spent time taking a good look at what all
> > mtime preservation would be needed to do, what Portage currently
> > does (since it does not, as some people have claimed, preserve
> > mtimes) and what should be done about packages that end up
> > installing files with absurd mtimes.
> 
> > If guaranteed mtime preservation is a desired feature, we should be
> > designing it around what developers need, not around what Portage
> > currently does.
> 
> What if the two coincide?

They don't. I have yet to see any argument in favour of random and
inconsistent clobbering of sub-second mtimes, and I have yet to see any
argument in favour of files with a timestamp of 1 Jan 1970 being merged
to /, other than that Portage currently does both of those things.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-council] mtime preservation

[Ulrich Mueller]

Coming back again to your previous posting.

>>>>> On Mon, 9 Nov 2009, Ciaran McCreesh wrote:

> The problem lies in the exceptions. Either we word PMS so vaguely
> that it's legal for the package manager to clobber any mtime (thus
> defeating the point of guaranteeing preservation at all),

Agreed. This is not what is wanted.

> or we include long, convoluted wording describing exactly the files
> Portage currently alters

Hm, maybe this isn't as bad as it seems:

,----
| The package manager must preserve modification times of regular files.
| This includes files being compressed before merging. Exceptions to
| this are:
`----

Now we need to enumerate the exceptions:

,----
| * files newly created by the package manager,
`----

This will cover splitdebug, for example. (And please don't tell me
that the wording is flawed because the PM could save a file's contents
in some buffer, then delete the file and create it newly. This would
be as unreasonable as the rot-13 example.)

,----
| * binary object files being stripped of symbols.
`----

Anything else missing from above list?

> (thus preventing reasonable-looking future changes), [...]

I don't get the point here. For any future change not covered by the
list of exceptions, the PM would have to preserve mtime, in spite of
modifying the file. Why would this prevent doing the change?

Ulrich

Re: [gentoo-council] mtime preservation

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 2396 bytes --]

On Tue, 10 Nov 2009 08:07:17 +0100
Ulrich Mueller <ulm@gentoo.org> wrote:
> ,----
> | The package manager must preserve modification times of regular
> files. | This includes files being compressed before merging.
> Exceptions to | this are:
> `----
> 
> Now we need to enumerate the exceptions:
> 
> ,----
> | * files newly created by the package manager,
> `----
> 
> This will cover splitdebug, for example. (And please don't tell me
> that the wording is flawed because the PM could save a file's contents
> in some buffer, then delete the file and create it newly. This would
> be as unreasonable as the rot-13 example.)

Funnily enough... objcopy does its own fancy things with utimes. Last
time I looked it wasn't sub-second aware, though.

> ,----
> | * binary object files being stripped of symbols.
> `----
>
> Anything else missing from above list?

man pages, info pages, things covered by ecompress, arbitrary
documentation, any source code file, probably some others.

> > (thus preventing reasonable-looking future changes), [...]
> 
> I don't get the point here. For any future change not covered by the
> list of exceptions, the PM would have to preserve mtime, in spite of
> modifying the file. Why would this prevent doing the change?

Preserving mtime isn't always the right thing to do, though. If the
package manager does something that would invalidate caches, you don't
want it to do so in a way that won't be noticed.

Also, back on the subject of sub-second resolution mtimes... Python
doesn't support them properly thanks to some utter brain damage from
its designers [1]. Files will end up with corrupted sub-second mtimes if
Python is used to set those times, which can again lead to broken
caches. But since not everything Portage merges does use Python to set
sub-second mtimes, this will result in any cache that starts looking at
sub-second resolution mtimes (which will happen -- POSIX considers the
non-sub-second functions deprecated) seeing a mix of valid and subtly
corrupted timestamps.

I honestly don't think we should be going anywhere with this until
Portage is using a mechanism that guarantees correct sub-second
resolution preservation in all cases.

[1]: http://ciaranm.wordpress.com/2009/11/15/this-week-in-python-stupidity-os-stat-os-utime-and-sub-second-timestamps/

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-council] mtime preservation

[Denis Dupeyron]

On Thu, Nov 19, 2009 at 3:12 PM, Ciaran McCreesh
<ciaran.mccreesh@googlemail.com> wrote:
> I honestly don't think we should be going anywhere with this until
> Portage is using a mechanism that guarantees correct sub-second
> resolution preservation in all cases.

Any idea, or even the beginning of one, of how to make that happen?

Denis.

Re: [gentoo-council] mtime preservation

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 645 bytes --]

On Thu, 19 Nov 2009 17:14:20 -0700
Denis Dupeyron <calchan@gentoo.org> wrote:
> On Thu, Nov 19, 2009 at 3:12 PM, Ciaran McCreesh
> <ciaran.mccreesh@googlemail.com> wrote:
> > I honestly don't think we should be going anywhere with this until
> > Portage is using a mechanism that guarantees correct sub-second
> > resolution preservation in all cases.
> 
> Any idea, or even the beginning of one, of how to make that happen?

My understanding is that someone with Python knowledge would have to
write an external C library that provides a replacement for stat and a
wrapper for either utimensat or futimens.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-council] mtime preservation

[Zac Medico]

Ciaran McCreesh wrote:
> On Thu, 19 Nov 2009 17:14:20 -0700
> Denis Dupeyron <calchan@gentoo.org> wrote:
>> On Thu, Nov 19, 2009 at 3:12 PM, Ciaran McCreesh
>> <ciaran.mccreesh@googlemail.com> wrote:
>>> I honestly don't think we should be going anywhere with this until
>>> Portage is using a mechanism that guarantees correct sub-second
>>> resolution preservation in all cases.
>> Any idea, or even the beginning of one, of how to make that happen?
> 
> My understanding is that someone with Python knowledge would have to
> write an external C library that provides a replacement for stat and a
> wrapper for either utimensat or futimens.

Alternatively, we could simply make portage spawn the mv binary
whenever rename fails (it fails when the source and destination are
on different devices). Although it's relatively slow, it should
solve the problem.
-- 
Thanks,
Zac

Re: [gentoo-council] mtime preservation

[Ulrich Mueller]

>>>>> On Thu, 19 Nov 2009, Ciaran McCreesh wrote:

>> Now we need to enumerate the exceptions:
>> 
>> ,----
>> | * files newly created by the package manager,
>> `----

>> ,----
>> | * binary object files being stripped of symbols.
>> `----
>> 
>> Anything else missing from above list?

> man pages, info pages, things covered by ecompress, arbitrary
> documentation,

Last time I checked, mtimes of all of these were preserved.

> any source code file,

Do you mean sources installed with FEATURES="installsources"?

> I honestly don't think we should be going anywhere with this until
> Portage is using a mechanism that guarantees correct sub-second
> resolution preservation in all cases.

I disagree. Sub-second resolution is currently a non-issue, and I'm
not aware of any problems caused by it (or its absence). And it's not
even supported by all filesystems. (Is it supported by tar? That might
also be an interesting question with respect to binpkgs.)

It's much better to have a spec that requires second resolution *now*,
instead of not preserving mtimes at all (as Paludis current does).
There's always the possibility to refine the spec later, in case any
issues with missing sub-second resolution should arise.

Ulrich

Re: [gentoo-council] Re: mtime preservation

[Ulrich Mueller]

>>>>> On Mon, 9 Nov 2009, Denis Dupeyron wrote:

> Let the clueless dev that I am summarize the situation. Please
> correct me if I'm wrong, I think that will help more people than
> just me. In the end I'll propose 2 wordings for PMS.

> I understand that the issue with requiring mtime preservation is
> that after unpacking the tarball(s) some files can have absurd
> dates, like January 1st 1970 or else. One easy solution to this
> would be to reset mtimes at merge time which will take care of
> everything. However, I know of a package which requires mtime
> preservation from src_compile to the moment it's merged to ${ROOT}.

Which package? sci-electronics/ghdl?

> And if there's a package like this in my very small corner of Gentoo
> there must be more elsewhere, and there probably will be more in the
> future.

The "absurd dates" issue would have been addressed by option B in
<http://bugs.gentoo.org/264130#c26>. This was up for vote at the last
council meeting, but wasn't chosen.

> So here are the 2 wordings I propose.

> 1 - All mtimes from the package's sources are preserved until files
> are merged to ${ROOT}. In case some mtimes are absurd and/or
> unsuitable, they're considered a bug and it's the maintainer's
> responsibility to fix them and report back upstream.

> 2 - All mtimes from the package's sources are reset to current time
> between the src_unpack phase and the next one. That is before
> src_configure when EAPI<2 and before src_prepare when EAPI>=2.
> mtimes are then preserved until files are merged to ${ROOT}.

> How's that? Am I missing something?

Hm, the idea was something like wording 1, but more details should be
added since there are also files that are modified by Portage (e.g.
binaries being stripped).

Ulrich

Re: [gentoo-council] Re: mtime preservation

[Denis Dupeyron]

On Mon, Nov 9, 2009 at 8:34 AM, Ciaran McCreesh
<ciaran.mccreesh@googlemail.com> wrote:
> If guaranteed mtime preservation is a desired feature, we
> should be designing it around what developers need, not around what
> Portage currently does.

I totally agree with you here. Let's focus on what we need and we'll
see about the implementation later. The idea of not doing something
because it requires changes to Portage's code is silly. Manpower can
and will always be an issue, but it's not a reason for freezing
development or not fixing what we think is broken.

On Mon, Nov 9, 2009 at 9:36 AM, Ulrich Mueller <ulm@gentoo.org> wrote:
> Which package? sci-electronics/ghdl?

Yes, but it doesn't really matter.

> Hm, the idea was something like wording 1, but more details should be
> added since there are also files that are modified by Portage (e.g.
> binaries being stripped).

If we decide we want to preserve mtimes I'm in favor of something
simple such as: *all* mtimes are being preserved, period, whether the
package manager modifies them or not. If the binaries are being
stripped their original mtimes can certainly be restored, can't they?
This would make for a much more reliable and manageable behavior.

On Mon, Nov 9, 2009 at 9:53 AM, Ciaran McCreesh
<ciaran.mccreesh@googlemail.com> wrote:
> And it was left out of EAPI 3 originally because no appropriate solution
> had been found.

Any chance you can tell us what you would propose disregarding what
the council would think of it or want to do and whether Portage would
have the feature or not already? It doesn't have to be fully fleshed
out and there can be blanks where you don't have the right answers.
And don't bother about implementation but just about the need. I would
encourage anybody who's interested to do the same.

If it were me only I'd do what I proposed in my previous email with
the precision above about all mtimes being preserved. This gives:

All mtimes from the package's sources are reset to current time
between the src_unpack phase and the next one. That is before
src_configure when EAPI<2 and before src_prepare when EAPI>=2. mtimes
are then preserved until files are merged to ${ROOT}. If the package
manager modifies a file on its own then the original mtime needs to be
restored.

About nanosecond resolution I think we should enforce it where
possible and progressively deprecate those versions of package
managers and their supporting libraries which do not allow for
nanosecond resolution. I have no idea how technically feasible that is
though. My fear is that in case we want to preserve mtimes and that a
given package relies on nanosecond resolution then not supporting the
latter is like not preserving mtimes.

Denis.

Re: [gentoo-council] Re: mtime preservation

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 3491 bytes --]

On Mon, 9 Nov 2009 10:58:43 -0700
Denis Dupeyron <calchan@gentoo.org> wrote:
> > Hm, the idea was something like wording 1, but more details should
> > be added since there are also files that are modified by Portage
> > (e.g. binaries being stripped).
> 
> If we decide we want to preserve mtimes I'm in favor of something
> simple such as: *all* mtimes are being preserved, period, whether the
> package manager modifies them or not. If the binaries are being
> stripped their original mtimes can certainly be restored, can't they?
> This would make for a much more reliable and manageable behavior.

It's doable. It's just that all mtimes being preserved is clearly wrong
for the 1 Jan 1970 cases.

> > And it was left out of EAPI 3 originally because no appropriate
> > solution had been found.
> 
> Any chance you can tell us what you would propose disregarding what
> the council would think of it or want to do and whether Portage would
> have the feature or not already? It doesn't have to be fully fleshed
> out and there can be blanks where you don't have the right answers.
> And don't bother about implementation but just about the need. I would
> encourage anybody who's interested to do the same.

My understanding is that:

* Most ebuilds don't care what happens, but a small number want to be
  able to guarantee mtime preservation for a subset of what they
  install. Those that care may prefer to handle preservation in various
  different functions.

* Changing mtimes on source trees is likely problematic with autotooled
  packages.

* Preserving mtimes unconditionally leads to silly mtimes being
  preserved.

Thus, I would let EAPI 4 ebuilds call dopreservemtimes (with an API
similar to docompress) in both src_install and pkg_preinst. Doing so
would instruct the package manager that it must preserve mtimes
(including subsecond, if supported on the filesystem) for a particular
set of paths, even if doing so means no stripping etc. All other mtimes
may be rewritten as the package manager sees fit, and from EAPI 4
onwards must be rewritten at merge time for anything predating the
start of the build.

> All mtimes from the package's sources are reset to current time
> between the src_unpack phase and the next one. That is before
> src_configure when EAPI<2 and before src_prepare when EAPI>=2. mtimes
> are then preserved until files are merged to ${ROOT}. If the package
> manager modifies a file on its own then the original mtime needs to be
> restored.

I have a sneaking suspicion that this will cause horrible complications
for generated Makefiles and the like...

> About nanosecond resolution I think we should enforce it where
> possible and progressively deprecate those versions of package
> managers and their supporting libraries which do not allow for
> nanosecond resolution. I have no idea how technically feasible that is
> though. My fear is that in case we want to preserve mtimes and that a
> given package relies on nanosecond resolution then not supporting the
> latter is like not preserving mtimes.

My understanding of the Python documentation is that they've royally
buggered the whole thing up by overloading stat.st_mtime and utimes on
floating point values for nanosecond-resolution timestamps, which
doesn't work because a float isn't precise enough to hold ten decimal
digits to the left of the decimal point and nine to the right...

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-council] Re: mtime preservation

[Denis Dupeyron]

On Mon, Nov 9, 2009 at 11:14 AM, Ciaran McCreesh
<ciaran.mccreesh@googlemail.com> wrote:
> It's doable. It's just that all mtimes being preserved is clearly wrong
> for the 1 Jan 1970 cases.

Not if you reset mtimes at unpacking, but:

> * Changing mtimes on source trees is likely problematic with autotooled
>  packages.

Good point, I wasn't thinking of that.

> Thus, I would let EAPI 4 ebuilds call dopreservemtimes (with an API
> similar to docompress) in both src_install and pkg_preinst. Doing so
> would instruct the package manager that it must preserve mtimes
> (including subsecond, if supported on the filesystem) for a particular
> set of paths, even if doing so means no stripping etc. All other mtimes
> may be rewritten as the package manager sees fit, and from EAPI 4
> onwards must be rewritten at merge time for anything predating the
> start of the build.

Thanks for your input. That would work for me although I would prefer
if the stripping and such things were kept automatic in order to try
and maintain QA at a reasonable level.

Denis.

Re: [gentoo-council] Agenda (draft) for November meeting 2009-11-09

[Patrick Lauer]

On Tuesday 03 November 2009 18:03:20 Ulrich Mueller wrote:
> Find below a proposed agenda for our next meeting.
> 
I'd like council to discuss what I consider a major bug in PMS - 
see the discussion at
http://archives.gentoo.org/gentoo-pms/msg_95a13d880eb521b13d7090f30350c26a.xml

Since the PMS / QA ppl seemingly don't have the authority to decide it needs 
to be deferred to council as far as I can tell.

Short version, PMS mandates bash 3.0, eclasses use 3.2 features (e.g. += 
assignment). No 3.0 ebuild is in the tree anymore, so the PMS requirement is 
quite silly.
Practically the tree has grown beyond what PMS defines (and done so for about 
a year now with eclasses and ebuilds "violating" PMS with noone caring).

I would appreciate a resolution to this issue, preferably one that doesn't 
kill half our eclasses.

Thanks,

Patrick

Re: [gentoo-council] Agenda (draft) for November meeting 2009-11-09

[Denis Dupeyron]

On Tue, Nov 3, 2009 at 11:06 AM, Ciaran McCreesh
<ciaran.mccreesh@googlemail.com> wrote:
> Could you add in something like:
> "Agree upon a wording for PMS for the mtime modification change
> introduced to EAPI 3 last time"

On Tue, Nov 3, 2009 at 1:17 PM, Patrick Lauer <patrick@gentoo.org> wrote:
> I'd like council to discuss what I consider a major bug in PMS -
> see the discussion at
> http://archives.gentoo.org/gentoo-pms/msg_95a13d880eb521b13d7090f30350c26a.xml

Could the both of you please flesh out a proposal on how you'd expect
the council to solve these issues? It would best if, on top of telling
what should be done, you explained why it should be done this way.
Raising the questions is already interesting but proposing answers is
even better. You may have done that elsewhere before but summarizing
it here would help tremendously. And by the way those who know they
will disagree with the above posters are welcome to make proposals of
their own. It would be nice if we'd all get an opportunity to discuss
it here before the council meeting.

Denis.

Re: [gentoo-council] Agenda (draft) for November meeting 2009-11-09

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 2277 bytes --]

On Tue, 3 Nov 2009 15:56:39 -0700
Denis Dupeyron <calchan@gentoo.org> wrote:
> Could the both of you please flesh out a proposal on how you'd expect
> the council to solve these issues? It would best if, on top of telling
> what should be done, you explained why it should be done this way.
> Raising the questions is already interesting but proposing answers is
> even better. You may have done that elsewhere before but summarizing
> it here would help tremendously.

Ok. For this:

> > "Agree upon a wording for PMS for the mtime modification change
> > introduced to EAPI 3 last time"

I honestly don't have an answer to this, nor a preferred solution. So
far as I can see, any possible solution is hit by one of:

* requiring changes to Portage.

* preventing future changes to Portage that would clearly be useful
  (for example, shebang rewriting for Prefix).

* being so vague and unspecific that it is entirely legal for package
  managers not to implement mtime preservation at all.

Obviously the Council had something else in mind when they voted the
proposal in, but the combined wisdom of the PMS team hasn't managed to
work out what that is. Thus, I'd like the Council to explain their
decision in sufficiently precise language that I can convert it to
LaTeX for PMS, and implement it in Paludis.

> > I'd like council to discuss what I consider a major bug in PMS -
> > see the discussion at
> > http://archives.gentoo.org/gentoo-pms/msg_95a13d880eb521b13d7090f30350c26a.xml  

As for this one, I was hoping Patrick would, as asked, start a
discussion on the gentoo-dev list regarding backwards compatibility and
how much developers care about having a clean upgrade path. I gather
from the previous Council meeting and from other comments that at least
some developers consider not being able to upgrade from an older Gentoo
install as being a problem.

The question, really, is whether there not being an upgrade path is
deliberate, or whether it's an accident that needs fixing. I don't
think that's a decision that should be made without a general
discussion amongst all Gentoo developers, and I certainly don't think
it's a decision that should be made arbitrarily by the PMS editors.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-council] Agenda (draft) for November meeting 2009-11-09

[Patrick Lauer]

> On Tue, Nov 3, 2009 at 1:17 PM, Patrick Lauer <patrick@gentoo.org> wrote:
> > I'd like council to discuss what I consider a major bug in PMS -
> > see the discussion at
> > http://archives.gentoo.org/gentoo-pms/msg_95a13d880eb521b13d7090f30350c26
> >a.xml
> 
> Could the both of you please flesh out a proposal on how you'd expect
> the council to solve these issues? 

Problem:
Usage of bash 3.2 features in ebuilds and eclasses where PMS forces bash 3.0
Apart from potentially breaking backwards compatibility etc. this is an 
inconsistency between specification and product.

Possible solutions:
1) Forbid bash 3.2 features.
Impact: Cleanup of many eclasses, lots of work for maintainers, removes 
actively used and useful functionality. Makes many people unhappy.

2) Fix PMS to require bash 3.2
Impact: one-line patch to PMS, small reduction in backwards compatibility
As bash 3.2 was added Oct 2006 and stabled May 2007 this would only affect 
systems not updated for over 2 years, which is far beyond our usual support 
horizon. Thus compatibility impact is negligible.

3) Ignore it
Impact: Well, it's what we've been doing for a year now. Seems to work out ok, 
but it's slightly unsatisfying.

4) something else?

I strongly suggest (2) since it has a very low impact, comes at no cost to 
maintainers and removes the need for endless further discussions of the topic.


> It would best if, on top of telling
> what should be done, you explained why it should be done this way.
> Raising the questions is already interesting but proposing answers is
> even better. You may have done that elsewhere before but summarizing
> it here would help tremendously. And by the way those who know they
> will disagree with the above posters are welcome to make proposals of
> their own. It would be nice if we'd all get an opportunity to discuss
> it here before the council meeting.
> 
> Denis.
> 

Re: [gentoo-council] Agenda (draft) for November meeting 2009-11-09

[Ulrich Mueller]

I can add both points suggested by Ciaran and Patrick to the agenda.
But the amount of topics is becoming a bit unrealistic for an one-hour
meeting.

Can the meeting be extended to 90 minutes, exceptionally?

Ulrich

Re: [gentoo-council] Agenda (draft) for November meeting 2009-11-09

[Denis Dupeyron]

On Wed, Nov 4, 2009 at 8:19 AM, Ulrich Mueller <ulm@gentoo.org> wrote:
> I can add both points suggested by Ciaran and Patrick to the agenda.
> But the amount of topics is becoming a bit unrealistic for an one-hour
> meeting.
>
> Can the meeting be extended to 90 minutes, exceptionally?

I don't mind at all. But if we discussed it on lists before the
meeting we would mostly have to vote and that would allow us to gain a
lot of time. Should we post something to gentoo-dev-announce to tell
people about this thread?

Denis.

Re: [gentoo-council] Agenda (draft) for November meeting 2009-11-09

[Ulrich Mueller]

>>>>> On Tue, 3 Nov 2009, I wrote:

> Find below a proposed agenda for our next meeting.

Find an updated version below. I'll post it to gentoo-council and
gentoo-dev-announce tomorrow, unless there are objections.

Ulrich


1. Intro (10 minutes, including late arrivals)
   1.1. Make sure that at least a couple people are logging the meeting.
   1.2. Roll call.
   1.3. Does everyone agree to extend today's meeting to 90 minutes?

2. Approve summary of October meeting (5 minutes)
   (in case it will be ready then)

3. Follow-ups from previous meeting (15 minutes)
   3.1. EAPI 3 status (zmedico, ulm)
   3.2. Upgrade path for old systems (leio, solar)
        Should important system packages (e.g. Python) be reverted to
        EAPI 0, in order to provide a defined upgrade path?

4. Prefix support in main Portage tree (grobian) (20 minutes)
   Fabian Groffen proposes that package managers should define the
   EPREFIX, EROOT, and ED variables (see [1] and replies).
   4.1. Does the council generally support this idea?
   4.2. Can it still be included in EAPI 3, or even in a fast EAPI 2.1?

5. Usage of bash 3.2 features in Portage tree (patrick) (15 minutes)
   Should usage of bash 3.2 features in the Portage tree be forbidden,
   allowed, or should the issue be ignored? See [2] for details.

6. Preservation of file modification times in EAPI 3 (ciaranm) (15 minutes)
   Agree upon a wording for PMS for the mtime modification change
   introduced to EAPI 3 last time.

7. AOB / open floor (10 minutes)

[1] http://archives.gentoo.org/gentoo-dev/msg_2a62689c71f95e4de5699a330b8b5524.xml
[2] http://archives.gentoo.org/gentoo-council/msg_ed3cba3e0ded55c4e497451af46ea55b.xml