[gentoo-portage-dev] [PATCH] lib/portage/dbapi/vartree: use dynamic report interval in _collision_protect

[gentoo-portage-dev] [PATCH] lib/portage/dbapi/vartree: use dynamic report interval in _collision_protect

[Fabian Groffen]

The reporting of files remaining can look somewhat odd since the report
interval is hardcoded to be per 1000 objects.  Adjust this interval to
be regular towards the end.  While at it, report percentage done.

Output before this patch:

 * checking 6111 files for package collisions
5111 files remaining ...
4111 files remaining ...
3111 files remaining ...
2111 files remaining ...
1111 files remaining ...
111 files remaining ...

After:

 * checking 6158 files for package collisions
16% done, 5131 files remaining ...
33% done, 4104 files remaining ...
50% done, 3077 files remaining ...
66% done, 2050 files remaining ...
83% done, 1023 files remaining ...

Signed-off-by: Fabian Groffen <grobian@gentoo.org>
---
 lib/portage/dbapi/vartree.py | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
index 4b91caea8..4d0bf2789 100644
--- a/lib/portage/dbapi/vartree.py
+++ b/lib/portage/dbapi/vartree.py
@@ -3475,13 +3475,19 @@ class dblink(object):
 			symlink_collisions = []
 			destroot = self.settings['ROOT']
 			totfiles = len(file_list) + len(symlink_list)
+			bucksize = 1000
+			buckcnt = int(totfiles / bucksize)
+			if buckcnt == 0 or totfiles % bucksize > int(bucksize / 2):
+				buckcnt = buckcnt + 1
+			bucksize = int(totfiles / buckcnt) + 1
 			showMessage(_(" %s checking %d files for package collisions\n") % \
 				(colorize("GOOD", "*"), totfiles))
 			for i, (f, f_type) in enumerate(chain(
 				((f, "reg") for f in file_list),
 				((f, "sym") for f in symlink_list))):
-				if i % 1000 == 0 and i != 0:
-					showMessage(_("%d files remaining ...\n") % (totfiles - i))
+				if i % bucksize == 0 and i != 0:
+					showMessage(_("%2d%% done, %d files remaining ...\n") %
+							(i * 100 / totfiles, totfiles - i))
 
 				dest_path = normalize_path(
 					os.path.join(destroot, f.lstrip(os.path.sep)))
-- 
2.20.1

Re: [gentoo-portage-dev] [PATCH] lib/portage/dbapi/vartree: use dynamic report interval in _collision_protect

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 2171 bytes --]

On 1/7/19 6:31 AM, Fabian Groffen wrote:
> The reporting of files remaining can look somewhat odd since the report
> interval is hardcoded to be per 1000 objects.  Adjust this interval to
> be regular towards the end.  While at it, report percentage done.
> 
> Output before this patch:
> 
>  * checking 6111 files for package collisions
> 5111 files remaining ...
> 4111 files remaining ...
> 3111 files remaining ...
> 2111 files remaining ...
> 1111 files remaining ...
> 111 files remaining ...
> 
> After:
> 
>  * checking 6158 files for package collisions
> 16% done, 5131 files remaining ...
> 33% done, 4104 files remaining ...
> 50% done, 3077 files remaining ...
> 66% done, 2050 files remaining ...
> 83% done, 1023 files remaining ...
> 
> Signed-off-by: Fabian Groffen <grobian@gentoo.org>
> ---
>  lib/portage/dbapi/vartree.py | 10 ++++++++--
>  1 file changed, 8 insertions(+), 2 deletions(-)
> 
> diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
> index 4b91caea8..4d0bf2789 100644
> --- a/lib/portage/dbapi/vartree.py
> +++ b/lib/portage/dbapi/vartree.py
> @@ -3475,13 +3475,19 @@ class dblink(object):
>  			symlink_collisions = []
>  			destroot = self.settings['ROOT']
>  			totfiles = len(file_list) + len(symlink_list)
> +			bucksize = 1000
> +			buckcnt = int(totfiles / bucksize)
> +			if buckcnt == 0 or totfiles % bucksize > int(bucksize / 2):
> +				buckcnt = buckcnt + 1
> +			bucksize = int(totfiles / buckcnt) + 1
>  			showMessage(_(" %s checking %d files for package collisions\n") % \
>  				(colorize("GOOD", "*"), totfiles))
>  			for i, (f, f_type) in enumerate(chain(
>  				((f, "reg") for f in file_list),
>  				((f, "sym") for f in symlink_list))):
> -				if i % 1000 == 0 and i != 0:
> -					showMessage(_("%d files remaining ...\n") % (totfiles - i))
> +				if i % bucksize == 0 and i != 0:
> +					showMessage(_("%2d%% done, %d files remaining ...\n") %
> +							(i * 100 / totfiles, totfiles - i))
>  
>  				dest_path = normalize_path(
>  					os.path.join(destroot, f.lstrip(os.path.sep)))
> 

Looks good, please merge.
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-portage-dev] [PATCH] lib/portage/dbapi/vartree: use dynamic report interval in _collision_protect

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 938 bytes --]

On Mon, 2019-01-07 at 15:31 +0100, Fabian Groffen wrote:
> The reporting of files remaining can look somewhat odd since the report
> interval is hardcoded to be per 1000 objects.  Adjust this interval to
> be regular towards the end.  While at it, report percentage done.
> 
> Output before this patch:
> 
>  * checking 6111 files for package collisions
> 5111 files remaining ...
> 4111 files remaining ...
> 3111 files remaining ...
> 2111 files remaining ...
> 1111 files remaining ...
> 111 files remaining ...
> 
> After:
> 
>  * checking 6158 files for package collisions
> 16% done, 5131 files remaining ...
> 33% done, 4104 files remaining ...
> 50% done, 3077 files remaining ...
> 66% done, 2050 files remaining ...
> 83% done, 1023 files remaining ...
> 

Not saying it's necessary but could we have some final message here?
I find it a bit weird to end at 83%.

-- 
Best regards,
Michał Górny

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

[gentoo-portage-dev] [PATCH] collision_protect: use dynamic report interval

[Fabian Groffen]

The reporting of files remaining can look somewhat odd since the report
interval is hardcoded to be per 1000 objects.  Adjust this interval to
be regular towards the end.  While at it, report percentage done.

Output before this patch:

 * checking 6111 files for package collisions
5111 files remaining ...
4111 files remaining ...
3111 files remaining ...
2111 files remaining ...
1111 files remaining ...
111 files remaining ...

After:

 * checking 6158 files for package collisions
 16% done, 5131 files remaining ...
 33% done, 4104 files remaining ...
 50% done, 3077 files remaining ...
 66% done, 2050 files remaining ...
 83% done, 1023 files remaining ...
100% done

Signed-off-by: Fabian Groffen <grobian@gentoo.org>
---
 lib/portage/dbapi/vartree.py | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
index 4b91caea8..78f2b37f2 100644
--- a/lib/portage/dbapi/vartree.py
+++ b/lib/portage/dbapi/vartree.py
@@ -3475,13 +3475,19 @@ class dblink(object):
 			symlink_collisions = []
 			destroot = self.settings['ROOT']
 			totfiles = len(file_list) + len(symlink_list)
+			bucksize = 1000
+			buckcnt = int(totfiles / bucksize)
+			if buckcnt == 0 or totfiles % bucksize > int(bucksize / 2):
+				buckcnt = buckcnt + 1
+			bucksize = int(totfiles / buckcnt) + 1
 			showMessage(_(" %s checking %d files for package collisions\n") % \
-				(colorize("GOOD", "*"), totfiles))
+					(colorize("GOOD", "*"), totfiles))
 			for i, (f, f_type) in enumerate(chain(
 				((f, "reg") for f in file_list),
 				((f, "sym") for f in symlink_list))):
-				if i % 1000 == 0 and i != 0:
-					showMessage(_("%d files remaining ...\n") % (totfiles - i))
+				if i % bucksize == 0 and i != 0:
+					showMessage(_("%3d%% done, %d files remaining ...\n") %
+							(i * 100 / totfiles, totfiles - i))
 
 				dest_path = normalize_path(
 					os.path.join(destroot, f.lstrip(os.path.sep)))
@@ -3570,6 +3576,8 @@ class dblink(object):
 							break
 					if stopmerge:
 						collisions.append(f)
+			if bucksize < totfiles:
+				showMessage(_("100% done\n"))
 			return collisions, dirs_ro, symlink_collisions, plib_collisions
 
 	def _lstat_inode_map(self, path_iter):
-- 
2.20.1

Re: [gentoo-portage-dev] [PATCH] collision_protect: use dynamic report interval

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 256 bytes --]

>>>>> On Tue, 08 Jan 2019, Fabian Groffen wrote:

> Output before this patch:

>  * checking 6111 files for package collisions
>  [...]

> After:

>  * checking 6158 files for package collisions
>  [...]

Why does the total number of files change?

Ulrich

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: [gentoo-portage-dev] [PATCH] collision_protect: use dynamic report interval

[Fabian Groffen]

[-- Attachment #1: Type: text/plain, Size: 541 bytes --]

On 08-01-2019 09:17:04 +0100, Ulrich Mueller wrote:
> >>>>> On Tue, 08 Jan 2019, Fabian Groffen wrote:
> 
> > Output before this patch:
> 
> >  * checking 6111 files for package collisions
> >  [...]
> 
> > After:
> 
> >  * checking 6158 files for package collisions
> >  [...]
> 
> Why does the total number of files change?

First was python2, second python3.  The result was indicative only, the
number doesn't change of course.

Sorry for the confusion.

Fabian


-- 
Fabian Groffen
Gentoo on a different level

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[gentoo-portage-dev] [PATCH v3] collision_protect: use dynamic report interval

[Fabian Groffen]

The reporting of files remaining can look somewhat odd since the report
interval is hardcoded to be per 1000 objects.  Adjust this interval to
be time based.  This means that modern (fast) machines likely will never
see the countdown messages at all.  On slow setups the message will be
informative that there is progress, albeit rather slowly.  While at it,
report percentage done.

Output before this patch:

 * checking 6158 files for package collisions
5158 files remaining ...
4158 files remaining ...
3158 files remaining ...
2158 files remaining ...
1158 files remaining ...
158 files remaining ...

Possible output after this patch on a slower machine:

 * checking 6158 files for package collisions
 48% done, 3145 files remaining ...
 96% done, 192 files remaining ...
100% done

Signed-off-by: Fabian Groffen <grobian@gentoo.org>
---
 lib/portage/dbapi/vartree.py | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
index 4b91caea8..244195fad 100644
--- a/lib/portage/dbapi/vartree.py
+++ b/lib/portage/dbapi/vartree.py
@@ -3475,13 +3475,18 @@ class dblink(object):
 			symlink_collisions = []
 			destroot = self.settings['ROOT']
 			totfiles = len(file_list) + len(symlink_list)
+			tnow = time.time()
+			tinterv = 2  # seconds
+			ninterv = tnow + tinterv
 			showMessage(_(" %s checking %d files for package collisions\n") % \
 				(colorize("GOOD", "*"), totfiles))
 			for i, (f, f_type) in enumerate(chain(
 				((f, "reg") for f in file_list),
 				((f, "sym") for f in symlink_list))):
-				if i % 1000 == 0 and i != 0:
-					showMessage(_("%d files remaining ...\n") % (totfiles - i))
+				if time.time() > ninterv:
+					showMessage(_("%3d%% done, %d files remaining ...\n") %
+							(i * 100 / totfiles, totfiles - i))
+					ninterv = time.time() + tinterv
 
 				dest_path = normalize_path(
 					os.path.join(destroot, f.lstrip(os.path.sep)))
@@ -3570,6 +3575,8 @@ class dblink(object):
 							break
 					if stopmerge:
 						collisions.append(f)
+			if tnow + tinterv < ninterv:
+				showMessage(_("100% done\n"))
 			return collisions, dirs_ro, symlink_collisions, plib_collisions
 
 	def _lstat_inode_map(self, path_iter):
-- 
2.20.1

Re: [gentoo-portage-dev] [PATCH v3] collision_protect: use dynamic report interval

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 2570 bytes --]

On 1/8/19 5:42 AM, Fabian Groffen wrote:
> The reporting of files remaining can look somewhat odd since the report
> interval is hardcoded to be per 1000 objects.  Adjust this interval to
> be time based.  This means that modern (fast) machines likely will never
> see the countdown messages at all.  On slow setups the message will be
> informative that there is progress, albeit rather slowly.  While at it,
> report percentage done.
> 
> Output before this patch:
> 
>  * checking 6158 files for package collisions
> 5158 files remaining ...
> 4158 files remaining ...
> 3158 files remaining ...
> 2158 files remaining ...
> 1158 files remaining ...
> 158 files remaining ...
> 
> Possible output after this patch on a slower machine:
> 
>  * checking 6158 files for package collisions
>  48% done, 3145 files remaining ...
>  96% done, 192 files remaining ...
> 100% done
> 
> Signed-off-by: Fabian Groffen <grobian@gentoo.org>
> ---
>  lib/portage/dbapi/vartree.py | 11 +++++++++--
>  1 file changed, 9 insertions(+), 2 deletions(-)
> 
> diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
> index 4b91caea8..244195fad 100644
> --- a/lib/portage/dbapi/vartree.py
> +++ b/lib/portage/dbapi/vartree.py
> @@ -3475,13 +3475,18 @@ class dblink(object):
>  			symlink_collisions = []
>  			destroot = self.settings['ROOT']
>  			totfiles = len(file_list) + len(symlink_list)
> +			tnow = time.time()
> +			tinterv = 2  # seconds
> +			ninterv = tnow + tinterv
>  			showMessage(_(" %s checking %d files for package collisions\n") % \
>  				(colorize("GOOD", "*"), totfiles))
>  			for i, (f, f_type) in enumerate(chain(
>  				((f, "reg") for f in file_list),
>  				((f, "sym") for f in symlink_list))):
> -				if i % 1000 == 0 and i != 0:
> -					showMessage(_("%d files remaining ...\n") % (totfiles - i))
> +				if time.time() > ninterv:
> +					showMessage(_("%3d%% done, %d files remaining ...\n") %
> +							(i * 100 / totfiles, totfiles - i))
> +					ninterv = time.time() + tinterv
>  
>  				dest_path = normalize_path(
>  					os.path.join(destroot, f.lstrip(os.path.sep)))
> @@ -3570,6 +3575,8 @@ class dblink(object):
>  							break
>  					if stopmerge:
>  						collisions.append(f)
> +			if tnow + tinterv < ninterv:
> +				showMessage(_("100% done\n"))
>  			return collisions, dirs_ro, symlink_collisions, plib_collisions
>  
>  	def _lstat_inode_map(self, path_iter):
> 

Please replace time.time() with portage.util.monotonic.monotonic().
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-portage-dev] [PATCH v3] collision_protect: use dynamic report interval

[M. J. Everitt]

[-- Attachment #1.1: Type: text/plain, Size: 3047 bytes --]

On 08/01/19 19:15, Zac Medico wrote:
> On 1/8/19 5:42 AM, Fabian Groffen wrote:
>> The reporting of files remaining can look somewhat odd since the report
>> interval is hardcoded to be per 1000 objects.  Adjust this interval to
>> be time based.  This means that modern (fast) machines likely will never
>> see the countdown messages at all.  On slow setups the message will be
>> informative that there is progress, albeit rather slowly.  While at it,
>> report percentage done.
>>
>> Output before this patch:
>>
>>  * checking 6158 files for package collisions
>> 5158 files remaining ...
>> 4158 files remaining ...
>> 3158 files remaining ...
>> 2158 files remaining ...
>> 1158 files remaining ...
>> 158 files remaining ...
>>
>> Possible output after this patch on a slower machine:
>>
>>  * checking 6158 files for package collisions
>>  48% done, 3145 files remaining ...
>>  96% done, 192 files remaining ...
>> 100% done
>>
>> Signed-off-by: Fabian Groffen <grobian@gentoo.org>
>> ---
>>  lib/portage/dbapi/vartree.py | 11 +++++++++--
>>  1 file changed, 9 insertions(+), 2 deletions(-)
>>
>> diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
>> index 4b91caea8..244195fad 100644
>> --- a/lib/portage/dbapi/vartree.py
>> +++ b/lib/portage/dbapi/vartree.py
>> @@ -3475,13 +3475,18 @@ class dblink(object):
>>  			symlink_collisions = []
>>  			destroot = self.settings['ROOT']
>>  			totfiles = len(file_list) + len(symlink_list)
>> +			tnow = time.time()
>> +			tinterv = 2  # seconds
>> +			ninterv = tnow + tinterv
>>  			showMessage(_(" %s checking %d files for package collisions\n") % \
>>  				(colorize("GOOD", "*"), totfiles))
>>  			for i, (f, f_type) in enumerate(chain(
>>  				((f, "reg") for f in file_list),
>>  				((f, "sym") for f in symlink_list))):
>> -				if i % 1000 == 0 and i != 0:
>> -					showMessage(_("%d files remaining ...\n") % (totfiles - i))
>> +				if time.time() > ninterv:
>> +					showMessage(_("%3d%% done, %d files remaining ...\n") %
>> +							(i * 100 / totfiles, totfiles - i))
>> +					ninterv = time.time() + tinterv
>>  
>>  				dest_path = normalize_path(
>>  					os.path.join(destroot, f.lstrip(os.path.sep)))
>> @@ -3570,6 +3575,8 @@ class dblink(object):
>>  							break
>>  					if stopmerge:
>>  						collisions.append(f)
>> +			if tnow + tinterv < ninterv:
>> +				showMessage(_("100% done\n"))
>>  			return collisions, dirs_ro, symlink_collisions, plib_collisions
>>  
>>  	def _lstat_inode_map(self, path_iter):
>>
> Please replace time.time() with portage.util.monotonic.monotonic().
It's a bit of a nit-pick, granted, but can we ensure that the count-down's
remain padded/justified such that the numbers line up for easy at-a-glance
inspection ? The optimal standard looks somewhat like the pre-merge Sizes:

 * Final size of build directory: 2696 KiB (2.6 MiB)
 * Final size of installed tree:  5372 KiB (5.2 MiB)

Otherwise, I think this will be quite helpful. Thanks.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-portage-dev] [PATCH v3] collision_protect: use dynamic report interval

[Fabian Groffen]

[-- Attachment #1: Type: text/plain, Size: 3434 bytes --]

On 08-01-2019 20:59:34 +0000, M. J. Everitt wrote:
> On 08/01/19 19:15, Zac Medico wrote:
> > On 1/8/19 5:42 AM, Fabian Groffen wrote:
> >> The reporting of files remaining can look somewhat odd since the report
> >> interval is hardcoded to be per 1000 objects.  Adjust this interval to
> >> be time based.  This means that modern (fast) machines likely will never
> >> see the countdown messages at all.  On slow setups the message will be
> >> informative that there is progress, albeit rather slowly.  While at it,
> >> report percentage done.
> >>
> >> Output before this patch:
> >>
> >>  * checking 6158 files for package collisions
> >> 5158 files remaining ...
> >> 4158 files remaining ...
> >> 3158 files remaining ...
> >> 2158 files remaining ...
> >> 1158 files remaining ...
> >> 158 files remaining ...
> >>
> >> Possible output after this patch on a slower machine:
> >>
> >>  * checking 6158 files for package collisions
> >>  48% done, 3145 files remaining ...
> >>  96% done, 192 files remaining ...
> >> 100% done
> >>
> >> Signed-off-by: Fabian Groffen <grobian@gentoo.org>
> >> ---
> >>  lib/portage/dbapi/vartree.py | 11 +++++++++--
> >>  1 file changed, 9 insertions(+), 2 deletions(-)
> >>
> >> diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
> >> index 4b91caea8..244195fad 100644
> >> --- a/lib/portage/dbapi/vartree.py
> >> +++ b/lib/portage/dbapi/vartree.py
> >> @@ -3475,13 +3475,18 @@ class dblink(object):
> >>  			symlink_collisions = []
> >>  			destroot = self.settings['ROOT']
> >>  			totfiles = len(file_list) + len(symlink_list)
> >> +			tnow = time.time()
> >> +			tinterv = 2  # seconds
> >> +			ninterv = tnow + tinterv
> >>  			showMessage(_(" %s checking %d files for package collisions\n") % \
> >>  				(colorize("GOOD", "*"), totfiles))
> >>  			for i, (f, f_type) in enumerate(chain(
> >>  				((f, "reg") for f in file_list),
> >>  				((f, "sym") for f in symlink_list))):
> >> -				if i % 1000 == 0 and i != 0:
> >> -					showMessage(_("%d files remaining ...\n") % (totfiles - i))
> >> +				if time.time() > ninterv:
> >> +					showMessage(_("%3d%% done, %d files remaining ...\n") %
> >> +							(i * 100 / totfiles, totfiles - i))
> >> +					ninterv = time.time() + tinterv
> >>  
> >>  				dest_path = normalize_path(
> >>  					os.path.join(destroot, f.lstrip(os.path.sep)))
> >> @@ -3570,6 +3575,8 @@ class dblink(object):
> >>  							break
> >>  					if stopmerge:
> >>  						collisions.append(f)
> >> +			if tnow + tinterv < ninterv:
> >> +				showMessage(_("100% done\n"))
> >>  			return collisions, dirs_ro, symlink_collisions, plib_collisions
> >>  
> >>  	def _lstat_inode_map(self, path_iter):
> >>
> > Please replace time.time() with portage.util.monotonic.monotonic().

Will do.

> It's a bit of a nit-pick, granted, but can we ensure that the count-down's
> remain padded/justified such that the numbers line up for easy at-a-glance
> inspection ? The optimal standard looks somewhat like the pre-merge Sizes:
> 
>  * Final size of build directory: 2696 KiB (2.6 MiB)
>  * Final size of installed tree:  5372 KiB (5.2 MiB)
> 
> Otherwise, I think this will be quite helpful. Thanks.

I think I understand what you mean, let me see if I can do something
without too much complexity.

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[gentoo-portage-dev] [PATCH v4] collision_protect: use dynamic report interval

[Fabian Groffen]

The reporting of files remaining can look somewhat odd since the report
interval is hardcoded to be per 1000 objects.  Adjust this interval to
be time based.  This means that modern (fast) machines likely will never
see the countdown messages at all.  On slow setups the message will be
informative that there is progress, albeit rather slowly.  While at it,
report percentage done.

Output before this patch:

 * checking 6158 files for package collisions
5158 files remaining ...
4158 files remaining ...
3158 files remaining ...
2158 files remaining ...
1158 files remaining ...
158 files remaining ...

Possible output after this patch on a slower machine:

 * checking 6158 files for package collisions
 48% done,  3145 files remaining ...
 96% done,   192 files remaining ...
100% done

Signed-off-by: Fabian Groffen <grobian@gentoo.org>
---
 lib/portage/dbapi/vartree.py | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
index 4b91caea8..df192e6fc 100644
--- a/lib/portage/dbapi/vartree.py
+++ b/lib/portage/dbapi/vartree.py
@@ -35,6 +35,7 @@ portage.proxy.lazyimport.lazyimport(globals(),
 	'portage.util.install_mask:install_mask_dir,InstallMask',
 	'portage.util.listdir:dircache,listdir',
 	'portage.util.movefile:movefile',
+	'portage.util.monotonic:monotonic',
 	'portage.util.path:first_existing,iter_parents',
 	'portage.util.writeable_check:get_ro_checker',
 	'portage.util._xattr:xattr',
@@ -3475,13 +3476,19 @@ class dblink(object):
 			symlink_collisions = []
 			destroot = self.settings['ROOT']
 			totfiles = len(file_list) + len(symlink_list)
+			tnow = monotonic()
+			tinterv = 2  # seconds
+			ninterv = tnow + tinterv
+			falign = len("%d" % totfiles)
 			showMessage(_(" %s checking %d files for package collisions\n") % \
 				(colorize("GOOD", "*"), totfiles))
 			for i, (f, f_type) in enumerate(chain(
 				((f, "reg") for f in file_list),
 				((f, "sym") for f in symlink_list))):
-				if i % 1000 == 0 and i != 0:
-					showMessage(_("%d files remaining ...\n") % (totfiles - i))
+				if monotonic() > ninterv:
+					showMessage(_("%3d%% done,  %*d files remaining ...\n") %
+							(falign, i * 100 / totfiles, totfiles - i))
+					ninterv = monotonic() + tinterv
 
 				dest_path = normalize_path(
 					os.path.join(destroot, f.lstrip(os.path.sep)))
@@ -3570,6 +3577,8 @@ class dblink(object):
 							break
 					if stopmerge:
 						collisions.append(f)
+			if tnow + tinterv < ninterv:
+				showMessage(_("100% done\n"))
 			return collisions, dirs_ro, symlink_collisions, plib_collisions
 
 	def _lstat_inode_map(self, path_iter):
-- 
2.20.1

Re: [gentoo-portage-dev] [PATCH v4] collision_protect: use dynamic report interval

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 3328 bytes --]

On 1/9/19 12:33 AM, Fabian Groffen wrote:
> The reporting of files remaining can look somewhat odd since the report
> interval is hardcoded to be per 1000 objects.  Adjust this interval to
> be time based.  This means that modern (fast) machines likely will never
> see the countdown messages at all.  On slow setups the message will be
> informative that there is progress, albeit rather slowly.  While at it,
> report percentage done.
> 
> Output before this patch:
> 
>  * checking 6158 files for package collisions
> 5158 files remaining ...
> 4158 files remaining ...
> 3158 files remaining ...
> 2158 files remaining ...
> 1158 files remaining ...
> 158 files remaining ...
> 
> Possible output after this patch on a slower machine:
> 
>  * checking 6158 files for package collisions
>  48% done,  3145 files remaining ...
>  96% done,   192 files remaining ...
> 100% done
> 
> Signed-off-by: Fabian Groffen <grobian@gentoo.org>
> ---
>  lib/portage/dbapi/vartree.py | 13 +++++++++++--
>  1 file changed, 11 insertions(+), 2 deletions(-)
> 
> diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
> index 4b91caea8..df192e6fc 100644
> --- a/lib/portage/dbapi/vartree.py
> +++ b/lib/portage/dbapi/vartree.py
> @@ -35,6 +35,7 @@ portage.proxy.lazyimport.lazyimport(globals(),
>  	'portage.util.install_mask:install_mask_dir,InstallMask',
>  	'portage.util.listdir:dircache,listdir',
>  	'portage.util.movefile:movefile',
> +	'portage.util.monotonic:monotonic',
>  	'portage.util.path:first_existing,iter_parents',
>  	'portage.util.writeable_check:get_ro_checker',
>  	'portage.util._xattr:xattr',
> @@ -3475,13 +3476,19 @@ class dblink(object):
>  			symlink_collisions = []
>  			destroot = self.settings['ROOT']
>  			totfiles = len(file_list) + len(symlink_list)
> +			tnow = monotonic()
> +			tinterv = 2  # seconds
> +			ninterv = tnow + tinterv
> +			falign = len("%d" % totfiles)
>  			showMessage(_(" %s checking %d files for package collisions\n") % \
>  				(colorize("GOOD", "*"), totfiles))
>  			for i, (f, f_type) in enumerate(chain(
>  				((f, "reg") for f in file_list),
>  				((f, "sym") for f in symlink_list))):
> -				if i % 1000 == 0 and i != 0:
> -					showMessage(_("%d files remaining ...\n") % (totfiles - i))
> +				if monotonic() > ninterv:
> +					showMessage(_("%3d%% done,  %*d files remaining ...\n") %
> +							(falign, i * 100 / totfiles, totfiles - i))
> +					ninterv = monotonic() + tinterv
>  
>  				dest_path = normalize_path(
>  					os.path.join(destroot, f.lstrip(os.path.sep)))
> @@ -3570,6 +3577,8 @@ class dblink(object):
>  							break
>  					if stopmerge:
>  						collisions.append(f)
> +			if tnow + tinterv < ninterv:
> +				showMessage(_("100% done\n"))

It took me a moment to understand the calculation here. How about if we
do something like this instead:


previous = monotonic()
progress_shown = False

for f in files:
	current = monotonic()
	if current - previous > tinterv:
	    showMessage(...)
	    previous = current
	    progress_shown = True

if progress_shown:
	showMessage(_("100% done\n"))



>  			return collisions, dirs_ro, symlink_collisions, plib_collisions
>  
>  	def _lstat_inode_map(self, path_iter):
> 


-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-portage-dev] [PATCH v4] collision_protect: use dynamic report interval

[Fabian Groffen]

[-- Attachment #1: Type: text/plain, Size: 908 bytes --]

On 09-01-2019 21:22:26 -0800, Zac Medico wrote:
> > +			if tnow + tinterv < ninterv:
> > +				showMessage(_("100% done\n"))
> 
> It took me a moment to understand the calculation here. How about if we
> do something like this instead:
> 
> 
> previous = monotonic()
> progress_shown = False

You're right.  I should've gone for readability in the first place.
There is no point in trying to save anything here.

Thanks,
Fabian

> for f in files:
> 	current = monotonic()
> 	if current - previous > tinterv:
> 	    showMessage(...)
> 	    previous = current
> 	    progress_shown = True
> 
> if progress_shown:
> 	showMessage(_("100% done\n"))
> 
> 
> 
> >  			return collisions, dirs_ro, symlink_collisions, plib_collisions
> >  
> >  	def _lstat_inode_map(self, path_iter):
> > 
> 
> 
> -- 
> Thanks,
> Zac
> 




-- 
Fabian Groffen
Gentoo on a different level

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[gentoo-portage-dev] [PATCH v5] collision_protect: use dynamic report interval

[Fabian Groffen]

The reporting of files remaining can look somewhat odd since the report
interval is hardcoded to be per 1000 objects.  Adjust this interval to
be time based.  This means that modern (fast) machines likely will never
see the countdown messages at all.  On slow setups the message will be
informative that there is progress, albeit rather slowly.  While at it,
report percentage done.

Output before this patch:

 * checking 6158 files for package collisions
5158 files remaining ...
4158 files remaining ...
3158 files remaining ...
2158 files remaining ...
1158 files remaining ...
158 files remaining ...

Possible output after this patch on a slower machine:

 * checking 6158 files for package collisions
 48% done,  3145 files remaining ...
 96% done,   192 files remaining ...
100% done

Signed-off-by: Fabian Groffen <grobian@gentoo.org>
---
 lib/portage/dbapi/vartree.py | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
index 4b91caea8..909c0e473 100644
--- a/lib/portage/dbapi/vartree.py
+++ b/lib/portage/dbapi/vartree.py
@@ -35,6 +35,7 @@ portage.proxy.lazyimport.lazyimport(globals(),
 	'portage.util.install_mask:install_mask_dir,InstallMask',
 	'portage.util.listdir:dircache,listdir',
 	'portage.util.movefile:movefile',
+	'portage.util.monotonic:monotonic',
 	'portage.util.path:first_existing,iter_parents',
 	'portage.util.writeable_check:get_ro_checker',
 	'portage.util._xattr:xattr',
@@ -3475,13 +3476,21 @@ class dblink(object):
 			symlink_collisions = []
 			destroot = self.settings['ROOT']
 			totfiles = len(file_list) + len(symlink_list)
+			previous = monotonic()
+			progress_shown = False
+			report_interval = 1.7  # seconds
+			falign = len("%d" % totfiles)
 			showMessage(_(" %s checking %d files for package collisions\n") % \
 				(colorize("GOOD", "*"), totfiles))
 			for i, (f, f_type) in enumerate(chain(
 				((f, "reg") for f in file_list),
 				((f, "sym") for f in symlink_list))):
-				if i % 1000 == 0 and i != 0:
-					showMessage(_("%d files remaining ...\n") % (totfiles - i))
+				current = monotonic()
+				if current - previous > report_interval:
+					showMessage(_("%3d%% done,  %*d files remaining ...\n") %
+							(i * 100 / totfiles, falign, totfiles - i))
+					previous = current
+					progress_shown = True
 
 				dest_path = normalize_path(
 					os.path.join(destroot, f.lstrip(os.path.sep)))
@@ -3570,6 +3579,8 @@ class dblink(object):
 							break
 					if stopmerge:
 						collisions.append(f)
+			if progress_shown:
+				showMessage(_("100% done\n"))
 			return collisions, dirs_ro, symlink_collisions, plib_collisions
 
 	def _lstat_inode_map(self, path_iter):
-- 
2.20.1

Re: [gentoo-portage-dev] [PATCH v5] collision_protect: use dynamic report interval

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 3032 bytes --]

On 1/10/19 7:30 AM, Fabian Groffen wrote:
> The reporting of files remaining can look somewhat odd since the report
> interval is hardcoded to be per 1000 objects.  Adjust this interval to
> be time based.  This means that modern (fast) machines likely will never
> see the countdown messages at all.  On slow setups the message will be
> informative that there is progress, albeit rather slowly.  While at it,
> report percentage done.
> 
> Output before this patch:
> 
>  * checking 6158 files for package collisions
> 5158 files remaining ...
> 4158 files remaining ...
> 3158 files remaining ...
> 2158 files remaining ...
> 1158 files remaining ...
> 158 files remaining ...
> 
> Possible output after this patch on a slower machine:
> 
>  * checking 6158 files for package collisions
>  48% done,  3145 files remaining ...
>  96% done,   192 files remaining ...
> 100% done
> 
> Signed-off-by: Fabian Groffen <grobian@gentoo.org>
> ---
>  lib/portage/dbapi/vartree.py | 15 +++++++++++++--
>  1 file changed, 13 insertions(+), 2 deletions(-)
> 
> diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
> index 4b91caea8..909c0e473 100644
> --- a/lib/portage/dbapi/vartree.py
> +++ b/lib/portage/dbapi/vartree.py
> @@ -35,6 +35,7 @@ portage.proxy.lazyimport.lazyimport(globals(),
>  	'portage.util.install_mask:install_mask_dir,InstallMask',
>  	'portage.util.listdir:dircache,listdir',
>  	'portage.util.movefile:movefile',
> +	'portage.util.monotonic:monotonic',
>  	'portage.util.path:first_existing,iter_parents',
>  	'portage.util.writeable_check:get_ro_checker',
>  	'portage.util._xattr:xattr',
> @@ -3475,13 +3476,21 @@ class dblink(object):
>  			symlink_collisions = []
>  			destroot = self.settings['ROOT']
>  			totfiles = len(file_list) + len(symlink_list)
> +			previous = monotonic()
> +			progress_shown = False
> +			report_interval = 1.7  # seconds
> +			falign = len("%d" % totfiles)
>  			showMessage(_(" %s checking %d files for package collisions\n") % \
>  				(colorize("GOOD", "*"), totfiles))
>  			for i, (f, f_type) in enumerate(chain(
>  				((f, "reg") for f in file_list),
>  				((f, "sym") for f in symlink_list))):
> -				if i % 1000 == 0 and i != 0:
> -					showMessage(_("%d files remaining ...\n") % (totfiles - i))
> +				current = monotonic()
> +				if current - previous > report_interval:
> +					showMessage(_("%3d%% done,  %*d files remaining ...\n") %
> +							(i * 100 / totfiles, falign, totfiles - i))
> +					previous = current
> +					progress_shown = True
>  
>  				dest_path = normalize_path(
>  					os.path.join(destroot, f.lstrip(os.path.sep)))
> @@ -3570,6 +3579,8 @@ class dblink(object):
>  							break
>  					if stopmerge:
>  						collisions.append(f)
> +			if progress_shown:
> +				showMessage(_("100% done\n"))
>  			return collisions, dirs_ro, symlink_collisions, plib_collisions
>  
>  	def _lstat_inode_map(self, path_iter):
> 

Looks good!
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-portage-dev] [PATCH v5] collision_protect: use dynamic report interval

[Fabian Groffen]

[-- Attachment #1: Type: text/plain, Size: 3321 bytes --]

Pushed, thanks

On 10-01-2019 20:37:16 -0800, Zac Medico wrote:
> On 1/10/19 7:30 AM, Fabian Groffen wrote:
> > The reporting of files remaining can look somewhat odd since the report
> > interval is hardcoded to be per 1000 objects.  Adjust this interval to
> > be time based.  This means that modern (fast) machines likely will never
> > see the countdown messages at all.  On slow setups the message will be
> > informative that there is progress, albeit rather slowly.  While at it,
> > report percentage done.
> > 
> > Output before this patch:
> > 
> >  * checking 6158 files for package collisions
> > 5158 files remaining ...
> > 4158 files remaining ...
> > 3158 files remaining ...
> > 2158 files remaining ...
> > 1158 files remaining ...
> > 158 files remaining ...
> > 
> > Possible output after this patch on a slower machine:
> > 
> >  * checking 6158 files for package collisions
> >  48% done,  3145 files remaining ...
> >  96% done,   192 files remaining ...
> > 100% done
> > 
> > Signed-off-by: Fabian Groffen <grobian@gentoo.org>
> > ---
> >  lib/portage/dbapi/vartree.py | 15 +++++++++++++--
> >  1 file changed, 13 insertions(+), 2 deletions(-)
> > 
> > diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
> > index 4b91caea8..909c0e473 100644
> > --- a/lib/portage/dbapi/vartree.py
> > +++ b/lib/portage/dbapi/vartree.py
> > @@ -35,6 +35,7 @@ portage.proxy.lazyimport.lazyimport(globals(),
> >  	'portage.util.install_mask:install_mask_dir,InstallMask',
> >  	'portage.util.listdir:dircache,listdir',
> >  	'portage.util.movefile:movefile',
> > +	'portage.util.monotonic:monotonic',
> >  	'portage.util.path:first_existing,iter_parents',
> >  	'portage.util.writeable_check:get_ro_checker',
> >  	'portage.util._xattr:xattr',
> > @@ -3475,13 +3476,21 @@ class dblink(object):
> >  			symlink_collisions = []
> >  			destroot = self.settings['ROOT']
> >  			totfiles = len(file_list) + len(symlink_list)
> > +			previous = monotonic()
> > +			progress_shown = False
> > +			report_interval = 1.7  # seconds
> > +			falign = len("%d" % totfiles)
> >  			showMessage(_(" %s checking %d files for package collisions\n") % \
> >  				(colorize("GOOD", "*"), totfiles))
> >  			for i, (f, f_type) in enumerate(chain(
> >  				((f, "reg") for f in file_list),
> >  				((f, "sym") for f in symlink_list))):
> > -				if i % 1000 == 0 and i != 0:
> > -					showMessage(_("%d files remaining ...\n") % (totfiles - i))
> > +				current = monotonic()
> > +				if current - previous > report_interval:
> > +					showMessage(_("%3d%% done,  %*d files remaining ...\n") %
> > +							(i * 100 / totfiles, falign, totfiles - i))
> > +					previous = current
> > +					progress_shown = True
> >  
> >  				dest_path = normalize_path(
> >  					os.path.join(destroot, f.lstrip(os.path.sep)))
> > @@ -3570,6 +3579,8 @@ class dblink(object):
> >  							break
> >  					if stopmerge:
> >  						collisions.append(f)
> > +			if progress_shown:
> > +				showMessage(_("100% done\n"))
> >  			return collisions, dirs_ro, symlink_collisions, plib_collisions
> >  
> >  	def _lstat_inode_map(self, path_iter):
> > 
> 
> Looks good!
> -- 
> Thanks,
> Zac
> 




-- 
Fabian Groffen
Gentoo on a different level

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]