[gentoo-portage-dev] [PATCH] egencache: add --stable-mtime option

[gentoo-portage-dev] [PATCH] egencache: add --stable-mtime option

[Zac Medico]

Disable Manifest "stable mtime" behavior by default, and add
a corresponding egencache option.

Suggested-by: Michał Górny <mgorny@gentoo.org>
---
 bin/egencache                                              |  6 +++++-
 man/egencache.1                                            |  3 +++
 pym/portage/manifest.py                                    | 14 +++++++++-----
 .../package/ebuild/_parallel_manifest/ManifestProcess.py   |  6 ++++--
 .../package/ebuild/_parallel_manifest/ManifestScheduler.py |  7 +++++--
 .../package/ebuild/_parallel_manifest/ManifestTask.py      |  8 +++++---
 6 files changed, 31 insertions(+), 13 deletions(-)

diff --git a/bin/egencache b/bin/egencache
index 7e3387e..07665e8 100755
--- a/bin/egencache
+++ b/bin/egencache
@@ -120,6 +120,9 @@ def parse_args(args):
 		choices=('y', 'n'),
 		metavar="<y|n>",
 		help="manually override layout.conf sign-manifests setting")
+	common.add_argument("--stable-mtime",
+		action="store_true",
+		help="apply stable mtime to generated manifests (for rsync)")
 	common.add_argument("--strict-manifests",
 		choices=('y', 'n'),
 		metavar="<y|n>",
@@ -1151,7 +1154,8 @@ def egencache_main(args):
 			force_sign_key=force_sign_key,
 			max_jobs=options.jobs,
 			max_load=options.load_average,
-			event_loop=event_loop)
+			event_loop=event_loop,
+			manifest_kwargs=dict(stable_mtime=options.stable_mtime))
 
 		signum = run_main_scheduler(scheduler)
 		if signum is not None:
diff --git a/man/egencache.1 b/man/egencache.1
index 7fd17c2..081e8c1 100644
--- a/man/egencache.1
+++ b/man/egencache.1
@@ -100,6 +100,9 @@ Manually override layout.conf sign-manifests setting.
 .BR "\-\-strict\-manifests< y | n >"
 Manually override "strict" FEATURES setting.
 .TP
+.BR "\-\-stable\-mtime"
+Apply stable mtime to generated manifests (for rsync).
+.TP
 .BR "\-\-thin\-manifests< y | n >"
 Manually override layout.conf thin-manifests setting.
 .TP
diff --git a/pym/portage/manifest.py b/pym/portage/manifest.py
index 818515f..0724272 100644
--- a/pym/portage/manifest.py
+++ b/pym/portage/manifest.py
@@ -128,7 +128,7 @@ class Manifest(object):
 	def __init__(self, pkgdir, distdir=None, fetchlist_dict=None,
 		manifest1_compat=DeprecationWarning, from_scratch=False, thin=False,
 		allow_missing=False, allow_create=True, hashes=None,
-		find_invalid_path_char=None):
+		find_invalid_path_char=None, stable_mtime=False):
 		""" Create new Manifest instance for package in pkgdir.
 		    Do not parse Manifest file if from_scratch == True (only for internal use)
 			The fetchlist_dict parameter is required only for generation of
@@ -145,6 +145,7 @@ class Manifest(object):
 			find_invalid_path_char = _find_invalid_path_char
 		self._find_invalid_path_char = find_invalid_path_char
 		self.pkgdir = _unicode_decode(pkgdir).rstrip(os.sep) + os.sep
+		self.stable_mtime = stable_mtime
 		self.fhashdict = {}
 		self.hashes = set()
 
@@ -283,7 +284,6 @@ class Manifest(object):
 			myentries = list(self._createManifestEntries())
 			update_manifest = True
 			preserved_stats = {}
-			preserved_stats[self.pkgdir.rstrip(os.sep)] = os.stat(self.pkgdir)
 			if myentries and not force:
 				try:
 					f = io.open(_unicode_encode(self.getFullname(),
@@ -291,7 +291,9 @@ class Manifest(object):
 						mode='r', encoding=_encodings['repo.content'],
 						errors='replace')
 					oldentries = list(self._parseManifestLines(f))
-					preserved_stats[self.getFullname()] = os.fstat(f.fileno())
+					if self.stable_mtime:
+						preserved_stats[self.getFullname()] = os.fstat(f.fileno())
+						preserved_stats[self.pkgdir.rstrip(os.sep)] = os.stat(self.pkgdir)
 					f.close()
 					if len(oldentries) == len(myentries):
 						update_manifest = False
@@ -313,7 +315,8 @@ class Manifest(object):
 					# non-empty for all currently known use cases.
 					write_atomic(self.getFullname(), "".join("%s\n" %
 						_unicode(myentry) for myentry in myentries))
-					self._apply_max_mtime(preserved_stats, myentries)
+					if self.stable_mtime:
+						self._apply_max_mtime(preserved_stats, myentries)
 					rval = True
 				else:
 					# With thin manifest, there's no need to have
@@ -442,7 +445,8 @@ class Manifest(object):
 			fetchlist_dict=self.fetchlist_dict, from_scratch=True,
 			thin=self.thin, allow_missing=self.allow_missing,
 			allow_create=self.allow_create, hashes=self.hashes,
-			find_invalid_path_char=self._find_invalid_path_char)
+			find_invalid_path_char=self._find_invalid_path_char,
+			stable_mtime=self.stable_mtime)
 		pn = os.path.basename(self.pkgdir.rstrip(os.path.sep))
 		cat = self._pkgdir_category()
 
diff --git a/pym/portage/package/ebuild/_parallel_manifest/ManifestProcess.py b/pym/portage/package/ebuild/_parallel_manifest/ManifestProcess.py
index 44e2576..01595a3 100644
--- a/pym/portage/package/ebuild/_parallel_manifest/ManifestProcess.py
+++ b/pym/portage/package/ebuild/_parallel_manifest/ManifestProcess.py
@@ -10,14 +10,16 @@ from portage.util._async.ForkProcess import ForkProcess
 
 class ManifestProcess(ForkProcess):
 
-	__slots__ = ("cp", "distdir", "fetchlist_dict", "repo_config")
+	__slots__ = ("cp", "distdir", "fetchlist_dict", "manifest_kwargs",
+		"repo_config")
 
 	MODIFIED = 16
 
 	def _run(self):
 		mf = self.repo_config.load_manifest(
 			os.path.join(self.repo_config.location, self.cp),
-			self.distdir, fetchlist_dict=self.fetchlist_dict)
+			self.distdir, fetchlist_dict=self.fetchlist_dict,
+			**(self.manifest_kwargs or {}))
 
 		try:
 			mf.create(assumeDistHashesAlways=True)
diff --git a/pym/portage/package/ebuild/_parallel_manifest/ManifestScheduler.py b/pym/portage/package/ebuild/_parallel_manifest/ManifestScheduler.py
index 38ac482..8a1c1d0 100644
--- a/pym/portage/package/ebuild/_parallel_manifest/ManifestScheduler.py
+++ b/pym/portage/package/ebuild/_parallel_manifest/ManifestScheduler.py
@@ -12,11 +12,13 @@ from .ManifestTask import ManifestTask
 class ManifestScheduler(AsyncScheduler):
 
 	def __init__(self, portdb, cp_iter=None,
-		gpg_cmd=None, gpg_vars=None, force_sign_key=None, **kwargs):
+		gpg_cmd=None, gpg_vars=None, force_sign_key=None,
+		manifest_kwargs=None, **kwargs):
 
 		AsyncScheduler.__init__(self, **kwargs)
 
 		self._portdb = portdb
+		self._manifest_kwargs = manifest_kwargs
 
 		if cp_iter is None:
 			cp_iter = self._iter_every_cp()
@@ -79,7 +81,8 @@ class ManifestScheduler(AsyncScheduler):
 				yield ManifestTask(cp=cp, distdir=distdir,
 					fetchlist_dict=fetchlist_dict, repo_config=repo_config,
 					gpg_cmd=self._gpg_cmd, gpg_vars=self._gpg_vars,
-					force_sign_key=self._force_sign_key)
+					force_sign_key=self._force_sign_key,
+					manifest_kwargs=self._manifest_kwargs)
 
 	def _task_exit(self, task):
 
diff --git a/pym/portage/package/ebuild/_parallel_manifest/ManifestTask.py b/pym/portage/package/ebuild/_parallel_manifest/ManifestTask.py
index 0ee2b91..fb5e16e 100644
--- a/pym/portage/package/ebuild/_parallel_manifest/ManifestTask.py
+++ b/pym/portage/package/ebuild/_parallel_manifest/ManifestTask.py
@@ -18,8 +18,8 @@ from .ManifestProcess import ManifestProcess
 
 class ManifestTask(CompositeTask):
 
-	__slots__ = ("cp", "distdir", "fetchlist_dict", "gpg_cmd",
-		"gpg_vars", "repo_config", "force_sign_key", "_manifest_path")
+	__slots__ = ("cp", "distdir", "fetchlist_dict", "force_sign_key",
+		"gpg_cmd", "gpg_vars", "manifest_kwargs", "repo_config", "_manifest_path")
 
 	_PGP_HEADER = b"BEGIN PGP SIGNED MESSAGE"
 	_manifest_line_re = re.compile(r'^(%s) ' % "|".join(MANIFEST2_IDENTIFIERS))
@@ -30,7 +30,9 @@ class ManifestTask(CompositeTask):
 		self._manifest_path = os.path.join(self.repo_config.location,
 			self.cp, "Manifest")
 		manifest_proc = ManifestProcess(cp=self.cp, distdir=self.distdir,
-			fetchlist_dict=self.fetchlist_dict, repo_config=self.repo_config,
+			fetchlist_dict=self.fetchlist_dict,
+			manifest_kwargs=self.manifest_kwargs,
+			repo_config=self.repo_config,
 			scheduler=self.scheduler)
 		self._start_task(manifest_proc, self._manifest_proc_exit)
 
-- 
2.4.10

Re: [gentoo-portage-dev] [PATCH] egencache: add --stable-mtime option

[Alexander Berntsen]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 16/12/15 06:33, Zac Medico wrote:
> Disable Manifest "stable mtime" behavior by default, and add a
> corresponding egencache option.
This message tells me nothing about why we need to do this.
- -- 
Alexander
bernalex@gentoo.org
https://secure.plaimi.net/~alexander
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCgAGBQJWcWk7AAoJENQqWdRUGk8B+vAQAIM0vppqXsIfCs3IlP7pmCme
VXJx++RS85r7+81Jx8eWzoVoiT5liX2obkqIhob+4gaZisF9WEOmO71plojbtDKe
DYGYsV97lRr0DEM52eQp51cbl8Le5Q4RujJm3qX3JQkH4CyvKdB05NNTAMjBMb3c
j7hk4CxFFqsD5CDWbE4c+3VyClEuB2pAO6rB4nuF8YKkwAip4B0TCWD9hyeiLe+V
5bubknSDDzA2McAaiYdlb7SF0Y6Hr2DOkogzhzf4R+xSEYcVT/RiDiG7oxkAZ1JA
GWNxnGYKni+TynwqYkueL1j9+rGLOvAVVzXKWB6R7qO9QBfGw8Zx3QJoCGbkx0Qj
Nhrq/u9F/sbFCyE50CnNqqcxxAEyxsnCk4SemO61tkG92c4sAQDrD4SGGUY41JHM
Hf1VnZsg+LBpH/5oz6n+ZsVHid1T8ISel6V/NSUEeQO+JDspEUZyFMrUKOMcGWGw
A2DJS2UGufWRxe9Q8cZkvtP4Q8meE+8gHW1m2iKZt+ghZn7Zos7zgfT6lZw6w69F
KtWd++JmTAGnH4Gyx857eGgzY7gxUFJTHSsMdUzps7nI+4NppyP3AqNZxDv/ETBx
d1jtwjKW7wFmglPXjK+vEMGj0rFJQbJJAjg9sxMQF1oUXgp7jFRmlHKSLOhHvXQc
R2l8HvdysLCUK1/pxAYf
=yW/c
-----END PGP SIGNATURE-----

Re: [gentoo-portage-dev] [PATCH] egencache: add --stable-mtime option

[Zac Medico]

On 12/16/2015 05:38 AM, Alexander Berntsen wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
> 
> On 16/12/15 06:33, Zac Medico wrote:
>> Disable Manifest "stable mtime" behavior by default, and add a
>> corresponding egencache option.
> This message tells me nothing about why we need to do this.

Yeah, I'm not so sure that we need to do this, so let's hold off for now.
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [PATCH] egencache: add --stable-mtime option

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1055 bytes --]

On Wed, 16 Dec 2015 14:38:03 +0100
Alexander Berntsen <bernalex@gentoo.org> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
> 
> On 16/12/15 06:33, Zac Medico wrote:
> > Disable Manifest "stable mtime" behavior by default, and add a
> > corresponding egencache option.  
> This message tells me nothing about why we need to do this.

We need do this because we changed the behavior and the new behavior is
counter-intuitive. We already had a number of bugs caused by it,
and while it's used by Infra, it's at least unexpected when someone
manually runs 'repoman manifest'.

I mean, before this all started 'repoman manifest' just updated
the Manifest which meant it's mtime changed. Nowadays, it also sets
mtime to some value in the past, which means running 'repoman manifest'
may result in updated Manifest having mtime older than the old
Manifest. As a result, people using rsync are in trouble. And this has
been reported too by overlay owners.

-- 
Best regards,
Michał Górny
<http://dev.gentoo.org/~mgorny/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-portage-dev] [PATCH] egencache: add --stable-mtime option

[Zac Medico]

On 12/16/2015 09:49 AM, Michał Górny wrote:
> On Wed, 16 Dec 2015 14:38:03 +0100
> Alexander Berntsen <bernalex@gentoo.org> wrote:
> 
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA512
>>
>> On 16/12/15 06:33, Zac Medico wrote:
>>> Disable Manifest "stable mtime" behavior by default, and add a
>>> corresponding egencache option.  
>> This message tells me nothing about why we need to do this.
> 
> We need do this because we changed the behavior and the new behavior is
> counter-intuitive. We already had a number of bugs caused by it,
> and while it's used by Infra, it's at least unexpected when someone
> manually runs 'repoman manifest'.

Just because it was historically buggy does not mean that it will always
be that way. I believe that it will be very safe once we've fixed it to
include the mtimes of all relevant directories in the max mtime calculation.

> I mean, before this all started 'repoman manifest' just updated
> the Manifest which meant it's mtime changed. Nowadays, it also sets
> mtime to some value in the past, which means running 'repoman manifest'
> may result in updated Manifest having mtime older than the old
> Manifest. As a result, people using rsync are in trouble. And this has
> been reported too by overlay owners.

We should get our facts straight. It's not possible for the updated
Manifest to have an older mtime than the old manifest, because the mtime
of the old Manfiest is included in the max mtime calculation.
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [PATCH] egencache: add --stable-mtime option

[Brian Dolbec]

On Sat, 19 Dec 2015 15:51:29 -0800
Zac Medico <zmedico@gentoo.org> wrote:

> On 12/16/2015 09:49 AM, Michał Górny wrote:
> > On Wed, 16 Dec 2015 14:38:03 +0100
> > Alexander Berntsen <bernalex@gentoo.org> wrote:
> >   
> >> -----BEGIN PGP SIGNED MESSAGE-----
> >> Hash: SHA512
> >>
> >> On 16/12/15 06:33, Zac Medico wrote:  
> >>> Disable Manifest "stable mtime" behavior by default, and add a
> >>> corresponding egencache option.    
> >> This message tells me nothing about why we need to do this.  
> > 
> > We need do this because we changed the behavior and the new
> > behavior is counter-intuitive. We already had a number of bugs
> > caused by it, and while it's used by Infra, it's at least
> > unexpected when someone manually runs 'repoman manifest'.  
> 
> Just because it was historically buggy does not mean that it will
> always be that way. I believe that it will be very safe once we've
> fixed it to include the mtimes of all relevant directories in the max
> mtime calculation.
> 
> > I mean, before this all started 'repoman manifest' just updated
> > the Manifest which meant it's mtime changed. Nowadays, it also sets
> > mtime to some value in the past, which means running 'repoman
> > manifest' may result in updated Manifest having mtime older than
> > the old Manifest. As a result, people using rsync are in trouble.
> > And this has been reported too by overlay owners.  
> 
> We should get our facts straight. It's not possible for the updated
> Manifest to have an older mtime than the old manifest, because the
> mtime of the old Manfiest is included in the max mtime calculation.


I'm generally in favour of this one, infra is changing how they
generate the Changelogs, etc.  So, making this an option makes sense
to me.  
-- 
Brian Dolbec <dolsen>

Re: [gentoo-portage-dev] [PATCH] egencache: add --stable-mtime option

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1834 bytes --]

On Sat, 19 Dec 2015 15:51:29 -0800
Zac Medico <zmedico@gentoo.org> wrote:

> On 12/16/2015 09:49 AM, Michał Górny wrote:
> > On Wed, 16 Dec 2015 14:38:03 +0100
> > Alexander Berntsen <bernalex@gentoo.org> wrote:
> >   
> >> -----BEGIN PGP SIGNED MESSAGE-----
> >> Hash: SHA512
> >>
> >> On 16/12/15 06:33, Zac Medico wrote:  
> >>> Disable Manifest "stable mtime" behavior by default, and add a
> >>> corresponding egencache option.    
> >> This message tells me nothing about why we need to do this.  
> > 
> > We need do this because we changed the behavior and the new behavior is
> > counter-intuitive. We already had a number of bugs caused by it,
> > and while it's used by Infra, it's at least unexpected when someone
> > manually runs 'repoman manifest'.  
> 
> Just because it was historically buggy does not mean that it will always
> be that way. I believe that it will be very safe once we've fixed it to
> include the mtimes of all relevant directories in the max mtime calculation.
> 
> > I mean, before this all started 'repoman manifest' just updated
> > the Manifest which meant it's mtime changed. Nowadays, it also sets
> > mtime to some value in the past, which means running 'repoman manifest'
> > may result in updated Manifest having mtime older than the old
> > Manifest. As a result, people using rsync are in trouble. And this has
> > been reported too by overlay owners.  
> 
> We should get our facts straight. It's not possible for the updated
> Manifest to have an older mtime than the old manifest, because the mtime
> of the old Manfiest is included in the max mtime calculation.

Unless you remove the old Manifest (and distfiles) to have Portage
refetch the files and recalc the digests.

-- 
Best regards,
Michał Górny
<http://dev.gentoo.org/~mgorny/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-portage-dev] [PATCH] egencache: add --stable-mtime option

[Zac Medico]

On 12/20/2015 01:24 AM, Michał Górny wrote:
> On Sat, 19 Dec 2015 15:51:29 -0800
> Zac Medico <zmedico@gentoo.org> wrote:
> 
>> On 12/16/2015 09:49 AM, Michał Górny wrote:
>>> On Wed, 16 Dec 2015 14:38:03 +0100
>>> Alexander Berntsen <bernalex@gentoo.org> wrote:
>>>   
>>>> -----BEGIN PGP SIGNED MESSAGE-----
>>>> Hash: SHA512
>>>>
>>>> On 16/12/15 06:33, Zac Medico wrote:  
>>>>> Disable Manifest "stable mtime" behavior by default, and add a
>>>>> corresponding egencache option.    
>>>> This message tells me nothing about why we need to do this.  
>>>
>>> We need do this because we changed the behavior and the new behavior is
>>> counter-intuitive. We already had a number of bugs caused by it,
>>> and while it's used by Infra, it's at least unexpected when someone
>>> manually runs 'repoman manifest'.  
>>
>> Just because it was historically buggy does not mean that it will always
>> be that way. I believe that it will be very safe once we've fixed it to
>> include the mtimes of all relevant directories in the max mtime calculation.
>>
>>> I mean, before this all started 'repoman manifest' just updated
>>> the Manifest which meant it's mtime changed. Nowadays, it also sets
>>> mtime to some value in the past, which means running 'repoman manifest'
>>> may result in updated Manifest having mtime older than the old
>>> Manifest. As a result, people using rsync are in trouble. And this has
>>> been reported too by overlay owners.  
>>
>> We should get our facts straight. It's not possible for the updated
>> Manifest to have an older mtime than the old manifest, because the mtime
>> of the old Manfiest is included in the max mtime calculation.
> 
> Unless you remove the old Manifest (and distfiles) to have Portage
> refetch the files and recalc the digests.

Removing the Manifest will bump the mtime of its parent directory, and
that mtime is included in the max mtime calculation.
-- 
Thanks,
Zac

[gentoo-portage-dev] [PATCH v2] egencache: add --stable-mtime option

[Zac Medico]

Since the Manifest "stable mtime" behavior could be have undiscovered
bugs, disable it by default, and add a corresponding egencache option.

Suggested-by: Michał Górny <mgorny@gentoo.org>
---
[PATCH v2] fixes preserved_stats to contain stat results for self.pkgdir
even if the Manifest has been removed, and also explains the reasoning
for disabling the stable mtime behavior by default.

 bin/egencache                                             |  6 +++++-
 man/egencache.1                                           |  3 +++
 pym/portage/manifest.py                                   | 15 ++++++++++-----
 .../package/ebuild/_parallel_manifest/ManifestProcess.py  |  6 ++++--
 .../ebuild/_parallel_manifest/ManifestScheduler.py        |  7 +++++--
 .../package/ebuild/_parallel_manifest/ManifestTask.py     |  8 +++++---
 6 files changed, 32 insertions(+), 13 deletions(-)

diff --git a/bin/egencache b/bin/egencache
index 7e3387e..07665e8 100755
--- a/bin/egencache
+++ b/bin/egencache
@@ -120,6 +120,9 @@ def parse_args(args):
 		choices=('y', 'n'),
 		metavar="<y|n>",
 		help="manually override layout.conf sign-manifests setting")
+	common.add_argument("--stable-mtime",
+		action="store_true",
+		help="apply stable mtime to generated manifests (for rsync)")
 	common.add_argument("--strict-manifests",
 		choices=('y', 'n'),
 		metavar="<y|n>",
@@ -1151,7 +1154,8 @@ def egencache_main(args):
 			force_sign_key=force_sign_key,
 			max_jobs=options.jobs,
 			max_load=options.load_average,
-			event_loop=event_loop)
+			event_loop=event_loop,
+			manifest_kwargs=dict(stable_mtime=options.stable_mtime))
 
 		signum = run_main_scheduler(scheduler)
 		if signum is not None:
diff --git a/man/egencache.1 b/man/egencache.1
index 7fd17c2..081e8c1 100644
--- a/man/egencache.1
+++ b/man/egencache.1
@@ -100,6 +100,9 @@ Manually override layout.conf sign-manifests setting.
 .BR "\-\-strict\-manifests< y | n >"
 Manually override "strict" FEATURES setting.
 .TP
+.BR "\-\-stable\-mtime"
+Apply stable mtime to generated manifests (for rsync).
+.TP
 .BR "\-\-thin\-manifests< y | n >"
 Manually override layout.conf thin-manifests setting.
 .TP
diff --git a/pym/portage/manifest.py b/pym/portage/manifest.py
index 3a6bc7e..aba9ad6 100644
--- a/pym/portage/manifest.py
+++ b/pym/portage/manifest.py
@@ -128,7 +128,7 @@ class Manifest(object):
 	def __init__(self, pkgdir, distdir=None, fetchlist_dict=None,
 		manifest1_compat=DeprecationWarning, from_scratch=False, thin=False,
 		allow_missing=False, allow_create=True, hashes=None,
-		find_invalid_path_char=None):
+		find_invalid_path_char=None, stable_mtime=False):
 		""" Create new Manifest instance for package in pkgdir.
 		    Do not parse Manifest file if from_scratch == True (only for internal use)
 			The fetchlist_dict parameter is required only for generation of
@@ -145,6 +145,7 @@ class Manifest(object):
 			find_invalid_path_char = _find_invalid_path_char
 		self._find_invalid_path_char = find_invalid_path_char
 		self.pkgdir = _unicode_decode(pkgdir).rstrip(os.sep) + os.sep
+		self.stable_mtime = stable_mtime
 		self.fhashdict = {}
 		self.hashes = set()
 
@@ -283,7 +284,8 @@ class Manifest(object):
 			myentries = list(self._createManifestEntries())
 			update_manifest = True
 			preserved_stats = {}
-			preserved_stats[self.pkgdir.rstrip(os.sep)] = os.stat(self.pkgdir)
+			if self.stable_mtime:
+				preserved_stats[self.pkgdir.rstrip(os.sep)] = os.stat(self.pkgdir)
 			if myentries and not force:
 				try:
 					f = io.open(_unicode_encode(self.getFullname(),
@@ -291,7 +293,8 @@ class Manifest(object):
 						mode='r', encoding=_encodings['repo.content'],
 						errors='replace')
 					oldentries = list(self._parseManifestLines(f))
-					preserved_stats[self.getFullname()] = os.fstat(f.fileno())
+					if self.stable_mtime:
+						preserved_stats[self.getFullname()] = os.fstat(f.fileno())
 					f.close()
 					if len(oldentries) == len(myentries):
 						update_manifest = False
@@ -313,7 +316,8 @@ class Manifest(object):
 					# non-empty for all currently known use cases.
 					write_atomic(self.getFullname(), "".join("%s\n" %
 						_unicode(myentry) for myentry in myentries))
-					self._apply_max_mtime(preserved_stats, myentries)
+					if self.stable_mtime:
+						self._apply_max_mtime(preserved_stats, myentries)
 					rval = True
 				else:
 					# With thin manifest, there's no need to have
@@ -440,7 +444,8 @@ class Manifest(object):
 			fetchlist_dict=self.fetchlist_dict, from_scratch=True,
 			thin=self.thin, allow_missing=self.allow_missing,
 			allow_create=self.allow_create, hashes=self.hashes,
-			find_invalid_path_char=self._find_invalid_path_char)
+			find_invalid_path_char=self._find_invalid_path_char,
+			stable_mtime=self.stable_mtime)
 		pn = os.path.basename(self.pkgdir.rstrip(os.path.sep))
 		cat = self._pkgdir_category()
 
diff --git a/pym/portage/package/ebuild/_parallel_manifest/ManifestProcess.py b/pym/portage/package/ebuild/_parallel_manifest/ManifestProcess.py
index 44e2576..01595a3 100644
--- a/pym/portage/package/ebuild/_parallel_manifest/ManifestProcess.py
+++ b/pym/portage/package/ebuild/_parallel_manifest/ManifestProcess.py
@@ -10,14 +10,16 @@ from portage.util._async.ForkProcess import ForkProcess
 
 class ManifestProcess(ForkProcess):
 
-	__slots__ = ("cp", "distdir", "fetchlist_dict", "repo_config")
+	__slots__ = ("cp", "distdir", "fetchlist_dict", "manifest_kwargs",
+		"repo_config")
 
 	MODIFIED = 16
 
 	def _run(self):
 		mf = self.repo_config.load_manifest(
 			os.path.join(self.repo_config.location, self.cp),
-			self.distdir, fetchlist_dict=self.fetchlist_dict)
+			self.distdir, fetchlist_dict=self.fetchlist_dict,
+			**(self.manifest_kwargs or {}))
 
 		try:
 			mf.create(assumeDistHashesAlways=True)
diff --git a/pym/portage/package/ebuild/_parallel_manifest/ManifestScheduler.py b/pym/portage/package/ebuild/_parallel_manifest/ManifestScheduler.py
index 38ac482..8a1c1d0 100644
--- a/pym/portage/package/ebuild/_parallel_manifest/ManifestScheduler.py
+++ b/pym/portage/package/ebuild/_parallel_manifest/ManifestScheduler.py
@@ -12,11 +12,13 @@ from .ManifestTask import ManifestTask
 class ManifestScheduler(AsyncScheduler):
 
 	def __init__(self, portdb, cp_iter=None,
-		gpg_cmd=None, gpg_vars=None, force_sign_key=None, **kwargs):
+		gpg_cmd=None, gpg_vars=None, force_sign_key=None,
+		manifest_kwargs=None, **kwargs):
 
 		AsyncScheduler.__init__(self, **kwargs)
 
 		self._portdb = portdb
+		self._manifest_kwargs = manifest_kwargs
 
 		if cp_iter is None:
 			cp_iter = self._iter_every_cp()
@@ -79,7 +81,8 @@ class ManifestScheduler(AsyncScheduler):
 				yield ManifestTask(cp=cp, distdir=distdir,
 					fetchlist_dict=fetchlist_dict, repo_config=repo_config,
 					gpg_cmd=self._gpg_cmd, gpg_vars=self._gpg_vars,
-					force_sign_key=self._force_sign_key)
+					force_sign_key=self._force_sign_key,
+					manifest_kwargs=self._manifest_kwargs)
 
 	def _task_exit(self, task):
 
diff --git a/pym/portage/package/ebuild/_parallel_manifest/ManifestTask.py b/pym/portage/package/ebuild/_parallel_manifest/ManifestTask.py
index 0ee2b91..fb5e16e 100644
--- a/pym/portage/package/ebuild/_parallel_manifest/ManifestTask.py
+++ b/pym/portage/package/ebuild/_parallel_manifest/ManifestTask.py
@@ -18,8 +18,8 @@ from .ManifestProcess import ManifestProcess
 
 class ManifestTask(CompositeTask):
 
-	__slots__ = ("cp", "distdir", "fetchlist_dict", "gpg_cmd",
-		"gpg_vars", "repo_config", "force_sign_key", "_manifest_path")
+	__slots__ = ("cp", "distdir", "fetchlist_dict", "force_sign_key",
+		"gpg_cmd", "gpg_vars", "manifest_kwargs", "repo_config", "_manifest_path")
 
 	_PGP_HEADER = b"BEGIN PGP SIGNED MESSAGE"
 	_manifest_line_re = re.compile(r'^(%s) ' % "|".join(MANIFEST2_IDENTIFIERS))
@@ -30,7 +30,9 @@ class ManifestTask(CompositeTask):
 		self._manifest_path = os.path.join(self.repo_config.location,
 			self.cp, "Manifest")
 		manifest_proc = ManifestProcess(cp=self.cp, distdir=self.distdir,
-			fetchlist_dict=self.fetchlist_dict, repo_config=self.repo_config,
+			fetchlist_dict=self.fetchlist_dict,
+			manifest_kwargs=self.manifest_kwargs,
+			repo_config=self.repo_config,
 			scheduler=self.scheduler)
 		self._start_task(manifest_proc, self._manifest_proc_exit)
 
-- 
2.4.10

Re: [gentoo-portage-dev] [PATCH v2] egencache: add --stable-mtime option

[Alexander Berntsen]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

The patch looks OK.

On 20/12/15 23:32, Zac Medico wrote:
> Since the Manifest "stable mtime" behavior could be have
> undiscovered bugs, disable it by default, and add a corresponding
> egencache option.
You have an English børk here -- "could be have".

> [PATCH v2] fixes preserved_stats to contain stat results for
> self.pkgdir even if the Manifest has been removed, and also
> explains the reasoning for disabling the stable mtime behavior by
> default.
Maybe this could go into the commit message? I'd not mind a more
verbose message for this.

- -- 
Alexander
bernalex@gentoo.org
https://secure.plaimi.net/~alexander
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCgAGBQJWeBUaAAoJENQqWdRUGk8Bc8wQALHRzKQUdHzl8oGgzCvhvvCn
YSX3JT+R7OcDbIxy+JjDQ88Mh2DQf2JfEcAmUcL/lA0grAJ+xV5dYwOnRIEbqYZM
sTK+oIxIs0awOQa/qKvUCwpP8hBLB4CM95ECwFWMb1JvOWnP3+czuXz74DIrhmRC
3P6JVPOzF7oaNDx/WdHPYyp1un6jQRCAsnwWmSMA1DFzbpE9S94C9uFzWQDha/Kk
0pOZJmqYciCxUa0v7rcQCuJqdYmyHgN5wfaYN/bUC2YO+7IkTvM7EkI3avg95fK5
BsTGKZfse4x5VYBZ+urEl7w/rNLG6BvpWnpz3BELVE792It/+vazq1e/ceC582Zy
I69jidlP7xH5+0VpcUx0LVaPJ/x5qZ/18Jhwt4JAGgehKdneOp+JefhfhZ+im6n7
KTH/D6gtVmg+MkL51JTcVLXsWeCpwVbm1fPzi31+T3zqngiLKtNxUaIhlLVyJQNZ
JMl/GBGpXiyA7Ig9l7g2cBqUWAj3GcYk20XikEY3oTnxT3rQzIMP8GGpTFA4wYUf
88aSo+VFyr02nOHXtmyTugpvPlXxwjMxFhQ04+j57NR39Y4zfpteiNRb9o8yYxKu
3rOoexmqY3U7z49qPLaA+Gnf810v+Jw6Hwv0CNPjhCSbB705acTDpaAGLhj/hvXv
Dg7KrpzMqt5mYsPYIzLB
=iK0o
-----END PGP SIGNATURE-----

[gentoo-portage-dev] [PATCH v3] egencache: add --stable-mtime option

[Zac Medico]

Since the Manifest "stable mtime" behavior could have undiscovered
bugs, disable it by default, and add a corresponding egencache option.

Suggested-by: Michał Górny <mgorny@gentoo.org>
---
[PATCH v3] fixes the commit message and adds a comment block to
Manifest.write in order to document subtle issues involving the mtime
of self.pkgdir that must be accounted for.

 bin/egencache                                      |  6 +++++-
 man/egencache.1                                    |  3 +++
 pym/portage/manifest.py                            | 23 +++++++++++++++++-----
 .../ebuild/_parallel_manifest/ManifestProcess.py   |  6 ++++--
 .../ebuild/_parallel_manifest/ManifestScheduler.py |  7 +++++--
 .../ebuild/_parallel_manifest/ManifestTask.py      |  8 +++++---
 6 files changed, 40 insertions(+), 13 deletions(-)

diff --git a/bin/egencache b/bin/egencache
index 7e3387e..07665e8 100755
--- a/bin/egencache
+++ b/bin/egencache
@@ -120,6 +120,9 @@ def parse_args(args):
 		choices=('y', 'n'),
 		metavar="<y|n>",
 		help="manually override layout.conf sign-manifests setting")
+	common.add_argument("--stable-mtime",
+		action="store_true",
+		help="apply stable mtime to generated manifests (for rsync)")
 	common.add_argument("--strict-manifests",
 		choices=('y', 'n'),
 		metavar="<y|n>",
@@ -1151,7 +1154,8 @@ def egencache_main(args):
 			force_sign_key=force_sign_key,
 			max_jobs=options.jobs,
 			max_load=options.load_average,
-			event_loop=event_loop)
+			event_loop=event_loop,
+			manifest_kwargs=dict(stable_mtime=options.stable_mtime))
 
 		signum = run_main_scheduler(scheduler)
 		if signum is not None:
diff --git a/man/egencache.1 b/man/egencache.1
index 7fd17c2..081e8c1 100644
--- a/man/egencache.1
+++ b/man/egencache.1
@@ -100,6 +100,9 @@ Manually override layout.conf sign-manifests setting.
 .BR "\-\-strict\-manifests< y | n >"
 Manually override "strict" FEATURES setting.
 .TP
+.BR "\-\-stable\-mtime"
+Apply stable mtime to generated manifests (for rsync).
+.TP
 .BR "\-\-thin\-manifests< y | n >"
 Manually override layout.conf thin-manifests setting.
 .TP
diff --git a/pym/portage/manifest.py b/pym/portage/manifest.py
index f696f84..6efd6e5 100644
--- a/pym/portage/manifest.py
+++ b/pym/portage/manifest.py
@@ -128,7 +128,7 @@ class Manifest(object):
 	def __init__(self, pkgdir, distdir=None, fetchlist_dict=None,
 		manifest1_compat=DeprecationWarning, from_scratch=False, thin=False,
 		allow_missing=False, allow_create=True, hashes=None,
-		find_invalid_path_char=None):
+		find_invalid_path_char=None, stable_mtime=False):
 		""" Create new Manifest instance for package in pkgdir.
 		    Do not parse Manifest file if from_scratch == True (only for internal use)
 			The fetchlist_dict parameter is required only for generation of
@@ -145,6 +145,7 @@ class Manifest(object):
 			find_invalid_path_char = _find_invalid_path_char
 		self._find_invalid_path_char = find_invalid_path_char
 		self.pkgdir = _unicode_decode(pkgdir).rstrip(os.sep) + os.sep
+		self.stable_mtime = stable_mtime
 		self.fhashdict = {}
 		self.hashes = set()
 
@@ -283,7 +284,16 @@ class Manifest(object):
 			myentries = list(self._createManifestEntries())
 			update_manifest = True
 			preserved_stats = {}
-			preserved_stats[self.pkgdir.rstrip(os.sep)] = os.stat(self.pkgdir)
+			if self.stable_mtime:
+				# The pre-existing mtime of self.pkgdir is included in the
+				# max mtime calculation in order to account for anything
+				# that may have been renamed or removed in this directory
+				# (including the Manifest itself). Note that the mtime of
+				# this directory will be always be bumped as a side-effect
+				# of writing the Manifest (since write_atomic uses a rename
+				# operation for atomicity), therefore it must be preserved
+				# before the Manifest is written.
+				preserved_stats[self.pkgdir.rstrip(os.sep)] = os.stat(self.pkgdir)
 			if myentries and not force:
 				try:
 					f = io.open(_unicode_encode(self.getFullname(),
@@ -291,7 +301,8 @@ class Manifest(object):
 						mode='r', encoding=_encodings['repo.content'],
 						errors='replace')
 					oldentries = list(self._parseManifestLines(f))
-					preserved_stats[self.getFullname()] = os.fstat(f.fileno())
+					if self.stable_mtime:
+						preserved_stats[self.getFullname()] = os.fstat(f.fileno())
 					f.close()
 					if len(oldentries) == len(myentries):
 						update_manifest = False
@@ -313,7 +324,8 @@ class Manifest(object):
 					# non-empty for all currently known use cases.
 					write_atomic(self.getFullname(), "".join("%s\n" %
 						_unicode(myentry) for myentry in myentries))
-					self._apply_max_mtime(preserved_stats, myentries)
+					if self.stable_mtime:
+						self._apply_max_mtime(preserved_stats, myentries)
 					rval = True
 				else:
 					# With thin manifest, there's no need to have
@@ -450,7 +462,8 @@ class Manifest(object):
 			fetchlist_dict=self.fetchlist_dict, from_scratch=True,
 			thin=self.thin, allow_missing=self.allow_missing,
 			allow_create=self.allow_create, hashes=self.hashes,
-			find_invalid_path_char=self._find_invalid_path_char)
+			find_invalid_path_char=self._find_invalid_path_char,
+			stable_mtime=self.stable_mtime)
 		pn = os.path.basename(self.pkgdir.rstrip(os.path.sep))
 		cat = self._pkgdir_category()
 
diff --git a/pym/portage/package/ebuild/_parallel_manifest/ManifestProcess.py b/pym/portage/package/ebuild/_parallel_manifest/ManifestProcess.py
index 44e2576..01595a3 100644
--- a/pym/portage/package/ebuild/_parallel_manifest/ManifestProcess.py
+++ b/pym/portage/package/ebuild/_parallel_manifest/ManifestProcess.py
@@ -10,14 +10,16 @@ from portage.util._async.ForkProcess import ForkProcess
 
 class ManifestProcess(ForkProcess):
 
-	__slots__ = ("cp", "distdir", "fetchlist_dict", "repo_config")
+	__slots__ = ("cp", "distdir", "fetchlist_dict", "manifest_kwargs",
+		"repo_config")
 
 	MODIFIED = 16
 
 	def _run(self):
 		mf = self.repo_config.load_manifest(
 			os.path.join(self.repo_config.location, self.cp),
-			self.distdir, fetchlist_dict=self.fetchlist_dict)
+			self.distdir, fetchlist_dict=self.fetchlist_dict,
+			**(self.manifest_kwargs or {}))
 
 		try:
 			mf.create(assumeDistHashesAlways=True)
diff --git a/pym/portage/package/ebuild/_parallel_manifest/ManifestScheduler.py b/pym/portage/package/ebuild/_parallel_manifest/ManifestScheduler.py
index 38ac482..8a1c1d0 100644
--- a/pym/portage/package/ebuild/_parallel_manifest/ManifestScheduler.py
+++ b/pym/portage/package/ebuild/_parallel_manifest/ManifestScheduler.py
@@ -12,11 +12,13 @@ from .ManifestTask import ManifestTask
 class ManifestScheduler(AsyncScheduler):
 
 	def __init__(self, portdb, cp_iter=None,
-		gpg_cmd=None, gpg_vars=None, force_sign_key=None, **kwargs):
+		gpg_cmd=None, gpg_vars=None, force_sign_key=None,
+		manifest_kwargs=None, **kwargs):
 
 		AsyncScheduler.__init__(self, **kwargs)
 
 		self._portdb = portdb
+		self._manifest_kwargs = manifest_kwargs
 
 		if cp_iter is None:
 			cp_iter = self._iter_every_cp()
@@ -79,7 +81,8 @@ class ManifestScheduler(AsyncScheduler):
 				yield ManifestTask(cp=cp, distdir=distdir,
 					fetchlist_dict=fetchlist_dict, repo_config=repo_config,
 					gpg_cmd=self._gpg_cmd, gpg_vars=self._gpg_vars,
-					force_sign_key=self._force_sign_key)
+					force_sign_key=self._force_sign_key,
+					manifest_kwargs=self._manifest_kwargs)
 
 	def _task_exit(self, task):
 
diff --git a/pym/portage/package/ebuild/_parallel_manifest/ManifestTask.py b/pym/portage/package/ebuild/_parallel_manifest/ManifestTask.py
index 0ee2b91..fb5e16e 100644
--- a/pym/portage/package/ebuild/_parallel_manifest/ManifestTask.py
+++ b/pym/portage/package/ebuild/_parallel_manifest/ManifestTask.py
@@ -18,8 +18,8 @@ from .ManifestProcess import ManifestProcess
 
 class ManifestTask(CompositeTask):
 
-	__slots__ = ("cp", "distdir", "fetchlist_dict", "gpg_cmd",
-		"gpg_vars", "repo_config", "force_sign_key", "_manifest_path")
+	__slots__ = ("cp", "distdir", "fetchlist_dict", "force_sign_key",
+		"gpg_cmd", "gpg_vars", "manifest_kwargs", "repo_config", "_manifest_path")
 
 	_PGP_HEADER = b"BEGIN PGP SIGNED MESSAGE"
 	_manifest_line_re = re.compile(r'^(%s) ' % "|".join(MANIFEST2_IDENTIFIERS))
@@ -30,7 +30,9 @@ class ManifestTask(CompositeTask):
 		self._manifest_path = os.path.join(self.repo_config.location,
 			self.cp, "Manifest")
 		manifest_proc = ManifestProcess(cp=self.cp, distdir=self.distdir,
-			fetchlist_dict=self.fetchlist_dict, repo_config=self.repo_config,
+			fetchlist_dict=self.fetchlist_dict,
+			manifest_kwargs=self.manifest_kwargs,
+			repo_config=self.repo_config,
 			scheduler=self.scheduler)
 		self._start_task(manifest_proc, self._manifest_proc_exit)
 
-- 
2.4.10

Re: [gentoo-portage-dev] [PATCH v2] egencache: add --stable-mtime option

[Zac Medico]

On 12/21/2015 07:04 AM, Alexander Berntsen wrote:
> The patch looks OK.
> 
> On 20/12/15 23:32, Zac Medico wrote:
>> Since the Manifest "stable mtime" behavior could be have
>> undiscovered bugs, disable it by default, and add a corresponding
>> egencache option.
> You have an English børk here -- "could be have".

Thanks, fixed.

>> [PATCH v2] fixes preserved_stats to contain stat results for
>> self.pkgdir even if the Manifest has been removed, and also
>> explains the reasoning for disabling the stable mtime behavior by
>> default.
> Maybe this could go into the commit message? I'd not mind a more
> verbose message for this.

In v3 I've added a comment block to Manifest.write in order to document
subtle issues involving the mtime of self.pkgdir that must be accounted for.
-- 
Thanks,
Zac

[gentoo-portage-dev] [PATCH v4] egencache: add --stable-mtime option

[Zac Medico]

Since the Manifest "stable mtime" behavior could have undiscovered
bugs, disable it by default, and add a corresponding egencache option.

Suggested-by: Michał Górny <mgorny@gentoo.org>
---
[PATCH v4] fixes english børk in the Manifest.write comment block.

 bin/egencache                                      |  6 +++++-
 man/egencache.1                                    |  3 +++
 pym/portage/manifest.py                            | 23 +++++++++++++++++-----
 .../ebuild/_parallel_manifest/ManifestProcess.py   |  6 ++++--
 .../ebuild/_parallel_manifest/ManifestScheduler.py |  7 +++++--
 .../ebuild/_parallel_manifest/ManifestTask.py      |  8 +++++---
 6 files changed, 40 insertions(+), 13 deletions(-)

diff --git a/bin/egencache b/bin/egencache
index 7e3387e..07665e8 100755
--- a/bin/egencache
+++ b/bin/egencache
@@ -120,6 +120,9 @@ def parse_args(args):
 		choices=('y', 'n'),
 		metavar="<y|n>",
 		help="manually override layout.conf sign-manifests setting")
+	common.add_argument("--stable-mtime",
+		action="store_true",
+		help="apply stable mtime to generated manifests (for rsync)")
 	common.add_argument("--strict-manifests",
 		choices=('y', 'n'),
 		metavar="<y|n>",
@@ -1151,7 +1154,8 @@ def egencache_main(args):
 			force_sign_key=force_sign_key,
 			max_jobs=options.jobs,
 			max_load=options.load_average,
-			event_loop=event_loop)
+			event_loop=event_loop,
+			manifest_kwargs=dict(stable_mtime=options.stable_mtime))
 
 		signum = run_main_scheduler(scheduler)
 		if signum is not None:
diff --git a/man/egencache.1 b/man/egencache.1
index 7fd17c2..081e8c1 100644
--- a/man/egencache.1
+++ b/man/egencache.1
@@ -100,6 +100,9 @@ Manually override layout.conf sign-manifests setting.
 .BR "\-\-strict\-manifests< y | n >"
 Manually override "strict" FEATURES setting.
 .TP
+.BR "\-\-stable\-mtime"
+Apply stable mtime to generated manifests (for rsync).
+.TP
 .BR "\-\-thin\-manifests< y | n >"
 Manually override layout.conf thin-manifests setting.
 .TP
diff --git a/pym/portage/manifest.py b/pym/portage/manifest.py
index f696f84..eaeecc5 100644
--- a/pym/portage/manifest.py
+++ b/pym/portage/manifest.py
@@ -128,7 +128,7 @@ class Manifest(object):
 	def __init__(self, pkgdir, distdir=None, fetchlist_dict=None,
 		manifest1_compat=DeprecationWarning, from_scratch=False, thin=False,
 		allow_missing=False, allow_create=True, hashes=None,
-		find_invalid_path_char=None):
+		find_invalid_path_char=None, stable_mtime=False):
 		""" Create new Manifest instance for package in pkgdir.
 		    Do not parse Manifest file if from_scratch == True (only for internal use)
 			The fetchlist_dict parameter is required only for generation of
@@ -145,6 +145,7 @@ class Manifest(object):
 			find_invalid_path_char = _find_invalid_path_char
 		self._find_invalid_path_char = find_invalid_path_char
 		self.pkgdir = _unicode_decode(pkgdir).rstrip(os.sep) + os.sep
+		self.stable_mtime = stable_mtime
 		self.fhashdict = {}
 		self.hashes = set()
 
@@ -283,7 +284,16 @@ class Manifest(object):
 			myentries = list(self._createManifestEntries())
 			update_manifest = True
 			preserved_stats = {}
-			preserved_stats[self.pkgdir.rstrip(os.sep)] = os.stat(self.pkgdir)
+			if self.stable_mtime:
+				# The pre-existing mtime of self.pkgdir is included in the
+				# max mtime calculation in order to account for anything
+				# that may have been renamed or removed in this directory
+				# (including the Manifest itself). Note that the mtime of
+				# this directory will always be bumped as a side-effect of
+				# writing the Manifest (since write_atomic uses a rename
+				# operation for atomicity), therefore it must be preserved
+				# before writing the Manifest.
+				preserved_stats[self.pkgdir.rstrip(os.sep)] = os.stat(self.pkgdir)
 			if myentries and not force:
 				try:
 					f = io.open(_unicode_encode(self.getFullname(),
@@ -291,7 +301,8 @@ class Manifest(object):
 						mode='r', encoding=_encodings['repo.content'],
 						errors='replace')
 					oldentries = list(self._parseManifestLines(f))
-					preserved_stats[self.getFullname()] = os.fstat(f.fileno())
+					if self.stable_mtime:
+						preserved_stats[self.getFullname()] = os.fstat(f.fileno())
 					f.close()
 					if len(oldentries) == len(myentries):
 						update_manifest = False
@@ -313,7 +324,8 @@ class Manifest(object):
 					# non-empty for all currently known use cases.
 					write_atomic(self.getFullname(), "".join("%s\n" %
 						_unicode(myentry) for myentry in myentries))
-					self._apply_max_mtime(preserved_stats, myentries)
+					if self.stable_mtime:
+						self._apply_max_mtime(preserved_stats, myentries)
 					rval = True
 				else:
 					# With thin manifest, there's no need to have
@@ -450,7 +462,8 @@ class Manifest(object):
 			fetchlist_dict=self.fetchlist_dict, from_scratch=True,
 			thin=self.thin, allow_missing=self.allow_missing,
 			allow_create=self.allow_create, hashes=self.hashes,
-			find_invalid_path_char=self._find_invalid_path_char)
+			find_invalid_path_char=self._find_invalid_path_char,
+			stable_mtime=self.stable_mtime)
 		pn = os.path.basename(self.pkgdir.rstrip(os.path.sep))
 		cat = self._pkgdir_category()
 
diff --git a/pym/portage/package/ebuild/_parallel_manifest/ManifestProcess.py b/pym/portage/package/ebuild/_parallel_manifest/ManifestProcess.py
index 44e2576..01595a3 100644
--- a/pym/portage/package/ebuild/_parallel_manifest/ManifestProcess.py
+++ b/pym/portage/package/ebuild/_parallel_manifest/ManifestProcess.py
@@ -10,14 +10,16 @@ from portage.util._async.ForkProcess import ForkProcess
 
 class ManifestProcess(ForkProcess):
 
-	__slots__ = ("cp", "distdir", "fetchlist_dict", "repo_config")
+	__slots__ = ("cp", "distdir", "fetchlist_dict", "manifest_kwargs",
+		"repo_config")
 
 	MODIFIED = 16
 
 	def _run(self):
 		mf = self.repo_config.load_manifest(
 			os.path.join(self.repo_config.location, self.cp),
-			self.distdir, fetchlist_dict=self.fetchlist_dict)
+			self.distdir, fetchlist_dict=self.fetchlist_dict,
+			**(self.manifest_kwargs or {}))
 
 		try:
 			mf.create(assumeDistHashesAlways=True)
diff --git a/pym/portage/package/ebuild/_parallel_manifest/ManifestScheduler.py b/pym/portage/package/ebuild/_parallel_manifest/ManifestScheduler.py
index 38ac482..8a1c1d0 100644
--- a/pym/portage/package/ebuild/_parallel_manifest/ManifestScheduler.py
+++ b/pym/portage/package/ebuild/_parallel_manifest/ManifestScheduler.py
@@ -12,11 +12,13 @@ from .ManifestTask import ManifestTask
 class ManifestScheduler(AsyncScheduler):
 
 	def __init__(self, portdb, cp_iter=None,
-		gpg_cmd=None, gpg_vars=None, force_sign_key=None, **kwargs):
+		gpg_cmd=None, gpg_vars=None, force_sign_key=None,
+		manifest_kwargs=None, **kwargs):
 
 		AsyncScheduler.__init__(self, **kwargs)
 
 		self._portdb = portdb
+		self._manifest_kwargs = manifest_kwargs
 
 		if cp_iter is None:
 			cp_iter = self._iter_every_cp()
@@ -79,7 +81,8 @@ class ManifestScheduler(AsyncScheduler):
 				yield ManifestTask(cp=cp, distdir=distdir,
 					fetchlist_dict=fetchlist_dict, repo_config=repo_config,
 					gpg_cmd=self._gpg_cmd, gpg_vars=self._gpg_vars,
-					force_sign_key=self._force_sign_key)
+					force_sign_key=self._force_sign_key,
+					manifest_kwargs=self._manifest_kwargs)
 
 	def _task_exit(self, task):
 
diff --git a/pym/portage/package/ebuild/_parallel_manifest/ManifestTask.py b/pym/portage/package/ebuild/_parallel_manifest/ManifestTask.py
index 0ee2b91..fb5e16e 100644
--- a/pym/portage/package/ebuild/_parallel_manifest/ManifestTask.py
+++ b/pym/portage/package/ebuild/_parallel_manifest/ManifestTask.py
@@ -18,8 +18,8 @@ from .ManifestProcess import ManifestProcess
 
 class ManifestTask(CompositeTask):
 
-	__slots__ = ("cp", "distdir", "fetchlist_dict", "gpg_cmd",
-		"gpg_vars", "repo_config", "force_sign_key", "_manifest_path")
+	__slots__ = ("cp", "distdir", "fetchlist_dict", "force_sign_key",
+		"gpg_cmd", "gpg_vars", "manifest_kwargs", "repo_config", "_manifest_path")
 
 	_PGP_HEADER = b"BEGIN PGP SIGNED MESSAGE"
 	_manifest_line_re = re.compile(r'^(%s) ' % "|".join(MANIFEST2_IDENTIFIERS))
@@ -30,7 +30,9 @@ class ManifestTask(CompositeTask):
 		self._manifest_path = os.path.join(self.repo_config.location,
 			self.cp, "Manifest")
 		manifest_proc = ManifestProcess(cp=self.cp, distdir=self.distdir,
-			fetchlist_dict=self.fetchlist_dict, repo_config=self.repo_config,
+			fetchlist_dict=self.fetchlist_dict,
+			manifest_kwargs=self.manifest_kwargs,
+			repo_config=self.repo_config,
 			scheduler=self.scheduler)
 		self._start_task(manifest_proc, self._manifest_proc_exit)
 
-- 
2.4.10

Re: [gentoo-portage-dev] [PATCH v4] egencache: add --stable-mtime option

[Alexander Berntsen]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 21/12/15 20:57, Zac Medico wrote:
> Since the Manifest "stable mtime" behavior could have undiscovered 
> bugs
This isn't exactly very motivating. "Since Foo could have bugs"
applies to pretty much any Python code >10 lines.

The patch itself looks OK though.
- -- 
Alexander
bernalex@gentoo.org
https://secure.plaimi.net/~alexander
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCgAGBQJWeQkaAAoJENQqWdRUGk8BAS8P/0NnyweZ173smkUoJIKbPu76
y5j04EX80cgtT/wjLrtXeCChJxjRVzX99al/LVTvPD8eq/SAO3uFVUOvvQaWu/Vm
H5OyRixUFvvS73Sd1MJZBstJKumnfPHI9/6qP4cp8jARLo4zwYs764Ms6DPOMV+L
W39yCm+I1KPM6XCoh77c8l8Jg/OoZYkv+Wm3R5o5NswHkHc/BKPnLeRAka3er093
MW6xiPGerkrH1InsOZCUp6c1wN3hpA6cV0f7iy+A7bwV0ObrC0cm5VrXWYbvoTiZ
lRnKgYtV1gEmjQKEVRIEt2r1CqS7VImrpjUwzIOTNW/1hwsVIDar6CC9uIctnz5i
u4oPnySOKQQK7gcmT+LTQ3FqHe5y9J6lMeprIbTsr1SX9BYk52kB8KH6c0InMUtH
Ysbck6rLcvCKMZkwliHjonrNtM7ifuMsJrpwkPELupERjxhHi3rQYRS9eERzFZbP
++79/ph/tMdX+/YfruzWG+dmBD0/C+RahDBefQZcaLWQkayb3MlJSd5jXBkuN4Cc
bkYyCY2lutv+xRLcbGcpdsWn441/aio2yOONU4wDY4JvOw2PvyDq0LnvjMU+du0S
t9a4VeGz5ieh4CiWYeTuXk6sf9KPEUfw+iPmFN4ugGBtLgPh6G5Pvq5ThOh/k6BD
AdLeeNnvzqz8Z2ZFoteT
=Lgxj
-----END PGP SIGNATURE-----

Re: [gentoo-portage-dev] [PATCH v4] egencache: add --stable-mtime option

[Zac Medico]

On 12/22/2015 12:26 AM, Alexander Berntsen wrote:
> On 21/12/15 20:57, Zac Medico wrote:
>> Since the Manifest "stable mtime" behavior could have undiscovered 
>> bugs
> This isn't exactly very motivating. "Since Foo could have bugs"
> applies to pretty much any Python code >10 lines.

We could include micro-optimization as a motivation, since this option
will eliminate some stat, listdir, and utime calls that should be pretty
negligible in comparison to the overall cost of generating the manifest.

I'm confident enough in the reliability of the stable mtime code, so
from my perspective, micro-optimization is the most valid motivation.
However, I'm not so sure that it's a good idea to disable stable mtime
by default, since it seems like a reasonable behavior with negligible cost.

> The patch itself looks OK though.

Thank you!
-- 
Thanks,
Zac