Re: [gentoo-user] Portage performance dropped considerably

Re: [gentoo-user] Portage performance dropped considerably

[Greg Turner]

On Sun, Jan 26, 2014 at 6:35 AM, Nikos Chantziaras <realnc@gmail.com> wrote:
> Anyone else noticed this yet? Some portage update seems to have made "emerge
> -uDN @world" perform about 10 times slower than before. It used to take
> seconds, now it takes about 4 minutes only to tell me that there's nothing
> to update. And it does that every time, even directly in succession and with
> the caches warm.
>
> Is it just me?

Usually this occurs when you have done something that causes portage
to stop using the metadata that rolls in via rsync.  The classic
example of something that will cause this is setting some
eclass-overrides in /etc/portage/repos.conf.

As of yesterday, portage still performed tolerably on a relatively
vanilla base system.

That stated, you're not hallucinating.  Portage is slow as tar.  I
suspect it's getting slower at least partially as a result of the
recent explosion of multilib-build-based ebuilds for multimedia and
x11 library ebuilds, and perhaps also due to those nasty
emul-linux-x86 blockers (a problem that will eventually be resolved,
Larry-The-Cow-God-willing, within a year or three) that are placing
higher and higher demands on portage's depsolving engine, as time
passes.

Having lots and lots of packages installed is a huge factor here.
This may be because, by default, portage uses installed packages to
calculate dependencies, resulting in a need to re-cache lots of
stuff... I'm a bit fuzzy on exactly what criteria are used to
determine when this happens, tbh.. that's something I've been meaning
to look into.

Adding something like:

EMERGE_DEFAULT_OPTS="${EMERGE_DEFAULT_OPTS} --backtrack=5"
EMERGE_DEFAULT_OPTS="${EMERGE_DEFAULT_OPTS} --complete-graph=n"
EMERGE_DEFAULT_OPTS="${EMERGE_DEFAULT_OPTS} --complete-graph-if-new-use=n"
EMERGE_DEFAULT_OPTS="${EMERGE_DEFAULT_OPTS} --complete-graph-if-new-ver=n"
EMERGE_DEFAULT_OPTS="${EMERGE_DEFAULT_OPTS} --with-bdeps=n"

to you /etc/portage/make.conf may help, as may running emerge
--metadata and/or egencache on your repos. after syncing... but...
these are no panacea.

Sometimes emerge -O is the only way to get some things done in any
reasonable time-frame.

It would help if there were some decent way to get some statistics
about where portage is spending all its time (especially, I suspect,
within the bash code, but the python level would also be interesting
to analyse).  Anyone know of a good way of doing that?

-gmt

[gentoo-user] Re: Portage performance dropped considerably

[eroen]

[-- Attachment #1.1: Type: text/plain, Size: 657 bytes --]

On Sun, 26 Jan 2014 07:09:49 -0800, Greg Turner <gmt@malth.us> wrote:
>It would help if there were some decent way to get some statistics
>about where portage is spending all its time (especially, I suspect,
>within the bash code, but the python level would also be interesting
>to analyse).  Anyone know of a good way of doing that?

http://permalink.gmane.org/gmane.linux.gentoo.devel/89518

Also, the attached patch to portage (works with 2.2.7 and 2.2.8 at
least, that too by TomWij) makes portage print out some more
information on what part of dependency resolution currently takes
place, without all the noise from --debug.

-- 
eroen

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: emerg.resolv.time_print.patch --]
[-- Type: text/x-patch, Size: 5748 bytes --]

index 7b77edc..cfd7025 100644
--- a/pym/_emerge/depgraph.py
+++ b/pym/_emerge/depgraph.py
@@ -86,6 +86,10 @@ if sys.hexversion >= 0x3000000:
 else:
 	_unicode = unicode
 
+from time import gmtime, strftime
+def time_print(message):
+	print("\n%s: %s  " % (strftime("%Y-%m-%d %H:%M:%S", gmtime()), message), end="")
+
 class _scheduler_graph_config(object):
 	def __init__(self, trees, pkg_cache, graph, mergelist):
 		self.trees = trees
@@ -1585,6 +1589,10 @@ class depgraph(object):
 			self._spinner_update()
 			while dep_stack:
 				dep = dep_stack.pop()
+#				try:
+#					time_print("  Processing: %s" % dep.cpv)
+#				except AttributeError:
+#					pass
 				if isinstance(dep, Package):
 					if not self._add_pkg_deps(dep,
 						allow_unsatisfied=allow_unsatisfied):
@@ -2106,6 +2114,7 @@ class depgraph(object):
 		for dep_root, dep_string, dep_priority in deps:
 				if not dep_string:
 					continue
+#				time_print("    Reducing: %s" % dep_string)
 				if debug:
 					writemsg_level("\nParent:    %s\n" % (pkg,),
 						noiselevel=-1, level=logging.DEBUG)
@@ -2154,6 +2163,8 @@ class depgraph(object):
 				if not dep_string:
 					continue
 
+#				time_print("    Disjunct: %s" % dep_string)
+
 				if not self._add_pkg_dep_string(
 					pkg, dep_root, dep_priority, dep_string,
 					allow_unsatisfied):
@@ -3005,6 +3016,7 @@ class depgraph(object):
 		virtuals = pkgsettings.getvirtuals()
 		args = self._dynamic_config._initial_arg_list[:]
 
+		time_print("Adding root packages")
 		for arg in self._expand_set_args(args, add_to_digraph=True):
 			for atom in arg.pset.getAtoms():
 				self._spinner_update()
@@ -3116,20 +3128,24 @@ class depgraph(object):
 
 		# Now that the root packages have been added to the graph,
 		# process the dependencies.
+		time_print("Processing dependencies")
 		if not self._create_graph():
 			return 0, myfavorites
 
+		time_print("Checking for slot conflicts")
 		try:
 			self.altlist()
 		except self._unknown_internal_error:
 			return False, myfavorites
 
+		time_print("Checking for blocker conflicts")
 		if (self._dynamic_config._slot_collision_info and
 			not self._accept_blocker_conflicts()) or \
 			(self._dynamic_config._allow_backtracking and
 			"slot conflict" in self._dynamic_config._backtrack_infos):
 			return False, myfavorites
 
+		time_print("Checking for rebuild triggers")
 		if self._rebuild.trigger_rebuilds():
 			backtrack_infos = self._dynamic_config._backtrack_infos
 			config = backtrack_infos.setdefault("config", {})
@@ -3138,12 +3154,14 @@ class depgraph(object):
 			self._dynamic_config._need_restart = True
 			return False, myfavorites
 
+		time_print("Checking if restart is needed")
 		if "config" in self._dynamic_config._backtrack_infos and \
 			("slot_operator_mask_built" in self._dynamic_config._backtrack_infos["config"] or
 			"slot_operator_replace_installed" in self._dynamic_config._backtrack_infos["config"]) and \
 			self.need_restart():
 			return False, myfavorites
 
+		time_print("Checking if we have to prune rebuilds")
 		if not self._dynamic_config._prune_rebuilds and \
 			self._dynamic_config._slot_operator_replace_installed and \
 			self._get_missed_updates():
@@ -3161,10 +3179,12 @@ class depgraph(object):
 			self._dynamic_config._need_restart = True
 			return False, myfavorites
 
+		time_print("Checking if restart is needed")
 		if self.need_restart():
 			# want_restart_for_use_change triggers this
 			return False, myfavorites
 
+		time_print("Checking for parameters that change behavior")
 		if "--fetchonly" not in self._frozen_config.myopts and \
 			"--buildpkgonly" in self._frozen_config.myopts:
 			graph_copy = self._dynamic_config.digraph.copy()
@@ -3188,6 +3208,7 @@ class depgraph(object):
 
 		digraph_nodes = self._dynamic_config.digraph.nodes
 
+		time_print("Checking for changes that are needed")
 		if any(x in digraph_nodes for x in
 			self._dynamic_config._needed_unstable_keywords) or \
 			any(x in digraph_nodes for x in
@@ -3200,6 +3221,7 @@ class depgraph(object):
 			self._dynamic_config._success_without_autounmask = True
 			return False, myfavorites
 
+		time_print("Done resolving!")
 		# We're true here unless we are missing binaries.
 		return (True, myfavorites)
 
@@ -5851,21 +5873,25 @@ class depgraph(object):
 				root_config.root]["root_config"] = root_config
 
 	def _resolve_conflicts(self):
-
+		time_print ("Trying to accept blocker conflicts ")
 		if "complete" not in self._dynamic_config.myparams and \
 			self._dynamic_config._allow_backtracking and \
 			self._dynamic_config._slot_collision_nodes and \
 			not self._accept_blocker_conflicts():
 			self._dynamic_config.myparams["complete"] = True
 
+		time_print ("Resolving slot conflicts for complete graph ")
 		if not self._complete_graph():
 			raise self._unknown_internal_error()
 
+		time_print ("Processing slot conflicts ")
 		self._process_slot_conflicts()
 
 		if self._dynamic_config._allow_backtracking:
+			time_print ("Triggering slot operator reinstalls ")
 			self._slot_operator_trigger_reinstalls()
 
+		time_print ("Validating blockers ")
 		if not self._validate_blockers():
 			# Blockers don't trigger the _skip_restart flag, since
 			# backtracking may solve blockers when it solves slot
@@ -7883,7 +7909,7 @@ def _spinner_start(spinner, myopts):
 		spinner.update = spinner.update_quiet
 
 	if show_spinner:
-		portage.writemsg_stdout("Calculating dependencies  ")
+		portage.writemsg_stdout("%s: Calculating dependencies  " % strftime("%Y-%m-%d %H:%M:%S", gmtime()))
 
 def _spinner_stop(spinner):
 	if spinner is None or \

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

[gentoo-user] Portage performance dropped considerably

[Nikos Chantziaras]

Anyone else noticed this yet? Some portage update seems to have made 
"emerge -uDN @world" perform about 10 times slower than before. It used 
to take seconds, now it takes about 4 minutes only to tell me that 
there's nothing to update. And it does that every time, even directly in 
succession and with the caches warm.

Is it just me?

Re: [gentoo-user] Portage performance dropped considerably

[hasufell]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 01/26/2014 03:35 PM, Nikos Chantziaras wrote:
> Anyone else noticed this yet? Some portage update seems to have
> made "emerge -uDN @world" perform about 10 times slower than
> before. It used to take seconds, now it takes about 4 minutes only
> to tell me that there's nothing to update. And it does that every
> time, even directly in succession and with the caches warm.
> 
> Is it just me?
> 
> 

Some people don't follow PMS when writing ebuilds which could be one
reason.

https://bugs.gentoo.org/show_bug.cgi?id=174407
https://bugs.gentoo.org/show_bug.cgi?id=487846
https://bugs.gentoo.org/show_bug.cgi?id=393203
https://bugs.gentoo.org/show_bug.cgi?id=486566
https://bugs.gentoo.org/show_bug.cgi?id=434246
https://bugs.gentoo.org/show_bug.cgi?id=311267

toolchain has closed all relevant bugs as WONTFIX so far and even QA
does not seem to care enough (?), although there are alternative
approaches to fix the lack of USE-dynamic SLOTS.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJS5R9dAAoJEFpvPKfnPDWz6zkH/13KbO6s3d5GWe4yXJsL1C7G
xOx24vTwWkeEqmi7+kUxR5Y3LUmZ0Pl4E9n//q5KcgVouKtalwFmqNaVsxQSLG9h
VyRZgGNdvcvTYfdlch8VoiIhUiP+1ZaOsZFuHTOzzfw3qoc52LceJYSyV/lo/x/9
Qe6xT+TuTyUzRJunBFTEzsool8hEhu1UWPYfmjTbZ94wRgSuu68yuL/7hIr75sin
cjEWo25MGzXzf8IhgfM+nI37gurPX136aLuc+JGLTUnYqN9SC1Ad2wjtvHqWJ54O
/kfVlL0790v+l2HyV8CfBs3Z3LktaE7EOgEJBzogHhuO1tjDaoERYDGoE+30pK4=
=tCmP
-----END PGP SIGNATURE-----

[gentoo-user] Re: Portage performance dropped considerably

[eroen]

[-- Attachment #1: Type: text/plain, Size: 1319 bytes --]

On Sun, 26 Jan 2014 16:35:43 +0200, Nikos Chantziaras
<realnc@gmail.com> wrote:
>Anyone else noticed this yet? Some portage update seems to have made 
>"emerge -uDN @world" perform about 10 times slower than before. It
>used to take seconds, now it takes about 4 minutes only to tell me
>that there's nothing to update. And it does that every time, even
>directly in succession and with the caches warm.
>
>Is it just me?

You don't say when your baseline was, but the complexity of resolving
the package tree has increased quite a bit over the last year due to
new features like automatic rebuilds of consumers after library updates.

Another somewhat common cause of sudden slowdowns is how portage
resolves conflicts (like packageA requiring an old version of libraryB),
which is rather time-consuming. You can try adding --backtrack=0 to the
emerge command to make it stop and print an error message when
encountering a conflict rather than look for a solution. Then you can
'help' out by manually resolving any conflicts by adding package
versions to /etc/portage/package.mask . Preferably try this *after*
running an update, so your system is up-to-date against your local
version of the gentoo tree, otherwise "normal" simple-to-resolve
conflicts might cause confusion. ;-)

-- 
eroen

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-user] Re: Portage performance dropped considerably

[Alan McKinnon]

On 26/01/2014 17:24, eroen wrote:
> On Sun, 26 Jan 2014 16:35:43 +0200, Nikos Chantziaras
> <realnc@gmail.com> wrote:
>> Anyone else noticed this yet? Some portage update seems to have made 
>> "emerge -uDN @world" perform about 10 times slower than before. It
>> used to take seconds, now it takes about 4 minutes only to tell me
>> that there's nothing to update. And it does that every time, even
>> directly in succession and with the caches warm.
>>
>> Is it just me?
> 
> You don't say when your baseline was, but the complexity of resolving
> the package tree has increased quite a bit over the last year due to
> new features like automatic rebuilds of consumers after library updates.
> 
> Another somewhat common cause of sudden slowdowns is how portage
> resolves conflicts (like packageA requiring an old version of libraryB),
> which is rather time-consuming. You can try adding --backtrack=0 to the
> emerge command to make it stop and print an error message when
> encountering a conflict rather than look for a solution. Then you can
> 'help' out by manually resolving any conflicts by adding package
> versions to /etc/portage/package.mask . Preferably try this *after*
> running an update, so your system is up-to-date against your local
> version of the gentoo tree, otherwise "normal" simple-to-resolve
> conflicts might cause confusion. ;-)
> 

I've been noticing these slowdowns for a few months now too. I'm
somewhat conflicted in my head about them, as unresolved blockers is now
something that happens very rarely. How often did we do this in the past:

emerge -avuND world
  stare at output trying to figure out wtf?
emerge -C <some problem package>
emerge -avuND world
  emerge problem package back if world update didn't already do it

That used to happen A LOT, and it took much longer than 4 minutes.
Nowadays it happens seldom but the resolution does take 4 minutes.

So I dunno, it's annoying to have to wait, but it also prevents a lot of
wasted time by doing what software can do so well - detecting dependency
issues.



-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] Re: Portage performance dropped considerably

[hasufell]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 01/26/2014 06:42 PM, Alan McKinnon wrote:
> On 26/01/2014 17:24, eroen wrote:
>> On Sun, 26 Jan 2014 16:35:43 +0200, Nikos Chantziaras 
>> <realnc@gmail.com> wrote:
>>> Anyone else noticed this yet? Some portage update seems to have
>>> made "emerge -uDN @world" perform about 10 times slower than
>>> before. It used to take seconds, now it takes about 4 minutes
>>> only to tell me that there's nothing to update. And it does
>>> that every time, even directly in succession and with the
>>> caches warm.
>>> 
>>> Is it just me?
>> 
>> You don't say when your baseline was, but the complexity of
>> resolving the package tree has increased quite a bit over the
>> last year due to new features like automatic rebuilds of
>> consumers after library updates.
>> 
>> Another somewhat common cause of sudden slowdowns is how portage 
>> resolves conflicts (like packageA requiring an old version of
>> libraryB), which is rather time-consuming. You can try adding
>> --backtrack=0 to the emerge command to make it stop and print an
>> error message when encountering a conflict rather than look for a
>> solution. Then you can 'help' out by manually resolving any
>> conflicts by adding package versions to /etc/portage/package.mask
>> . Preferably try this *after* running an update, so your system
>> is up-to-date against your local version of the gentoo tree,
>> otherwise "normal" simple-to-resolve conflicts might cause
>> confusion. ;-)
>> 
> 
> I've been noticing these slowdowns for a few months now too. I'm 
> somewhat conflicted in my head about them, as unresolved blockers
> is now something that happens very rarely. How often did we do this
> in the past:
> 
> emerge -avuND world stare at output trying to figure out wtf? 
> emerge -C <some problem package> emerge -avuND world emerge problem
> package back if world update didn't already do it
> 
> That used to happen A LOT, and it took much longer than 4 minutes. 
> Nowadays it happens seldom but the resolution does take 4 minutes.
> 
> So I dunno, it's annoying to have to wait, but it also prevents a
> lot of wasted time by doing what software can do so well -
> detecting dependency issues.
> 
> 
> 

Dependency resolution is broken and incomplete. Portage is unable to
print useful autounmask and similar messages to solve conflicts
manually, is unable to solve circular deps at all and bails out on
simple things like only updating vim when gvim is installed as well.
Then we have dirty workarounds like PDEPEND which shouldn't be there
in the first place...

It's a miracle that it works at all, especially when people keep
breaking the cache and rely on undefined behavior. Also... we still
have binary files in the tree.

Each EAPI adds more complexity to the dependency calculation. We have
no performance regression tests. We don't have many people who want to
look into portage code (can't blame them and since ferringb is gone we
don't have any1 who works on the QA-side of the code). Afais, it will
get a lot worse.

So, not sure where your optimism comes from. But... some devs are
interested in starting from scratch or picking up pkgcore (which would
be the most sane thing to do IMO).
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJS5U40AAoJEFpvPKfnPDWziK0IAKwuPe4DOBvamSkhtYbipZOv
CdCmt4qjlYn/NjLMkyb8I5AO1m3rziHKuFnfMAksFodTZx9HJ8rbXh1H75bGNt+i
k1cJ4Z6eg9R6hHqsBXAdwBfl4eDouINYMs2Q2R85XFD2QdZKUE/8AcUU5s2YHkxR
NraYC/1n2LxUaXwA8D66KNHKSR31Gb5ISd+Slt+kvAGpXjRDJCDRAWD/QChPkVUL
06RA6qjIhmAooWo3x5lcBjpGUsVkiPk15sF3E0t1qyjp78eiOq8cZBMRYxnhUVN+
AtQlESyzVmYjaCI557fsPr6sZasD69U9luZM6UUToeDSoK7O81s99MilhqUGpKA=
=vPSH
-----END PGP SIGNATURE-----

Re: [gentoo-user] Re: Portage performance dropped considerably

[Tanstaafl]

On 2014-01-26 1:04 PM, hasufell <hasufell@gentoo.org> wrote:
> So, not sure where your optimism comes from. But... some devs are
> interested in starting from scratch or picking up pkgcore (which would
> be the most sane thing to do IMO).

?

If the problem is really this potentially serious, why start from 
scratch, when Paludis is already very mature? Is it pure politics 
(someone just doesn't like Ciaran)?

Re: [gentoo-user] Re: Portage performance dropped considerably

[Alan McKinnon]

On 27/01/2014 13:59, Tanstaafl wrote:
> On 2014-01-26 1:04 PM, hasufell <hasufell@gentoo.org> wrote:
>> So, not sure where your optimism comes from. But... some devs are
>> interested in starting from scratch or picking up pkgcore (which would
>> be the most sane thing to do IMO).
> 
> ?
> 
> If the problem is really this potentially serious, why start from
> scratch, when Paludis is already very mature? Is it pure politics
> (someone just doesn't like Ciaran)?
> 
> 
> 


No-one likes to admit it, but I think there's some NIH going on

-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] Re: Portage performance dropped considerably

[hasufell]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 01/27/2014 02:06 PM, Alan McKinnon wrote:
> On 27/01/2014 13:59, Tanstaafl wrote:
>> On 2014-01-26 1:04 PM, hasufell <hasufell@gentoo.org> wrote:
>>> So, not sure where your optimism comes from. But... some devs
>>> are interested in starting from scratch or picking up pkgcore
>>> (which would be the most sane thing to do IMO).
>> 
>> ?
>> 
>> If the problem is really this potentially serious, why start
>> from scratch, when Paludis is already very mature? Is it pure
>> politics (someone just doesn't like Ciaran)?
>> 
>> 
>> 
> 
> 
> No-one likes to admit it, but I think there's some NIH going on
> 

I just tried paludis again (after some time).

Things that make it currently impossible to properly migrate my
portage config:
* you cannot unmask USE flags at all, not without hackery... and that
is really non-trivial for unmasking abi_x86_32 globally, because those
masks are scattered across a lot of files in profiles/
The explanation from the paludis developer is simply wrong:
http://paludis.exherbo.org/trac/ticket/817
* seems there is no equivalent to --dynamic-deps=y (which is default
in emerge)... either I am missing something or this is a pretty good
reason to not use it
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJS6cwfAAoJEFpvPKfnPDWz+MoIAJvH9zDbsVQaRwyb+yMEowJ8
qXjaLmDTx5BFyyL7tSelFEYyNwh0DN1ypyOQu2VkScNOTNIbqfffWXsAPoe4GJrP
pngtb9xo4H4/IIdtr7i8fwRU937UK5V4Fq0Er/e56SGpdHG3G+emxrBeuB2Y6n0M
m+gdEI1xmSuB/YOd/byDc+t9qK1688qM23fHJd/SsW732FY9ooUlfSZuO39ltjpk
96ojLyGe4TAp5zkk2BNBbpLXyuAtszwsc8U5nPN89v87gbC7gH5pIrJDXbQkRfMF
EP0ouZ6nfB4cDqFM1/GVvJ2+V24jleWkpV3UQmCPDAd18T6Qa/fkujz0JuijXAg=
=FfQN
-----END PGP SIGNATURE-----

Re: [gentoo-user] Portage performance dropped considerably

[Stroller]

On 30 Jan 2014, at 03:50 am, hasufell <hasufell@gentoo.org> wrote:

> I just tried paludis again (after some time).
> ...
> * you cannot unmask USE flags at all, not without hackery... and that
> is really non-trivial for unmasking abi_x86_32 globally, because those
> masks are scattered across a lot of files in profiles/
> The explanation from the paludis developer is simply wrong:
> http://paludis.exherbo.org/trac/ticket/817


"WONTFIX: you can hack around it with your own profile if you need to deal with Gentoo not following its own policies correctly."

Yes, that's the Ciaran McCreesh I remember.

Stroller.

Re: [gentoo-user] Portage performance dropped considerably

[hasufell]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 01/30/2014 07:15 PM, Stroller wrote:
> 
> On 30 Jan 2014, at 03:50 am, hasufell <hasufell@gentoo.org> wrote:
> 
>> I just tried paludis again (after some time). ... * you cannot
>> unmask USE flags at all, not without hackery... and that is
>> really non-trivial for unmasking abi_x86_32 globally, because
>> those masks are scattered across a lot of files in profiles/ The
>> explanation from the paludis developer is simply wrong: 
>> http://paludis.exherbo.org/trac/ticket/817
> 
> 
> "WONTFIX: you can hack around it with your own profile if you need
> to deal with Gentoo not following its own policies correctly."
> 
> Yes, that's the Ciaran McCreesh I remember.
> 
> Stroller.
> 

The thread gets funny. I guess this is not so much about NIH, but
rather about the fact that no one wants to work with him or that no
one wants to be one of his users and gets his feature requests all
RESOLVED WONTFIX.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJS7AKzAAoJEFpvPKfnPDWzKAkIAKEIAx/4l690pHYvxvKkaypJ
XWPs+LRokNboyzXyeZLEgWhEIJ5LzflBMgcnn0KRRn3p81JYaERQ+Cnx3yBtL148
7ovlZug12dxLO+nWVajrOWP3YWcHV12Kla6q7qTWrTO4RxZbfNEncyyMc4uMzCyk
mQ13nBP7gooNdRx5pN61POKI23OPyK4Z/AnlJdMq6aForVuY788vOUZq8q/n96MU
tdkx7npzJVJ/OGgwIF5AqIn1G1NmzmkQ3R8hKnPN/0W+l6jlChoocq+9tELTnJ/r
UtXmVwdlsHCnG4rY+RxeVBIfLMi0f9xce1/ckENbLIiyoj5xMNkZ3/+6dyI/VhU=
=FIJW
-----END PGP SIGNATURE-----

Re: [gentoo-user] Portage performance dropped considerably

[Mariusz Ceier]

No, it's not just you, running "emerge -uNDvp world" takes slightly
over 18 minutes on my laptop (i5 M 430  @ 2.27GHz) - and when there
was lot to update I had to wait over 1hr to just get the list of
packages. I wonder if there's some profiling tool for portage.
Also:
time FEATURES=-xattr emerge owncloud -v
real    6m31.202s
user    2m42.057s
sys     2m1.541s

time FEATURES=xattr emerge owncloud -v
real    30m15.632s
user    22m44.369s
sys     5m30.129s

5 times longer - for something that's essentially copying files.


On 26 January 2014 15:35, Nikos Chantziaras <realnc@gmail.com> wrote:
> Anyone else noticed this yet? Some portage update seems to have made "emerge
> -uDN @world" perform about 10 times slower than before. It used to take
> seconds, now it takes about 4 minutes only to tell me that there's nothing
> to update. And it does that every time, even directly in succession and with
> the caches warm.
>
> Is it just me?
>
>

Re: [gentoo-user] Portage performance dropped considerably

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 959 bytes --]

On Sun, 26 Jan 2014 16:53:33 +0100 Mariusz Ceier wrote:
> No, it's not just you, running "emerge -uNDvp world" takes slightly
> over 18 minutes on my laptop (i5 M 430  @ 2.27GHz) - and when there
> was lot to update I had to wait over 1hr to just get the list of
> packages. I wonder if there's some profiling tool for portage.
> Also:
> time FEATURES=-xattr emerge owncloud -v
> real    6m31.202s
> user    2m42.057s
> sys     2m1.541s
> 
> time FEATURES=xattr emerge owncloud -v
> real    30m15.632s
> user    22m44.369s
> sys     5m30.129s
> 
> 5 times longer - for something that's essentially copying files.

Slow xattr is a known problem, see bugs 482290, 465000:
https://bugs.gentoo.org/show_bug.cgi?id=482290
https://bugs.gentoo.org/show_bug.cgi?id=465000

But xattr bug shows itself during install phase and has nothing to do
with dependency calculation time mentioned in the original mail.

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-user] Portage performance dropped considerably

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 1155 bytes --]

Am 26.01.2014 15:35, schrieb Nikos Chantziaras:
> Anyone else noticed this yet? Some portage update seems to have made
> "emerge -uDN @world" perform about 10 times slower than before. It used
> to take seconds, now it takes about 4 minutes only to tell me that
> there's nothing to update. And it does that every time, even directly in
> succession and with the caches warm.
> 
> Is it just me?
> 
> 

As a remedy, you can try to use portage with pypy. I've used this setup
for half a year and it works well.

make.conf:
PYTHON_TARGETS="python2_7 python3_3 pypy2_0"

/etc/portage/package.keywords:
dev-python/pypy
virtual/pypy

/etc/portage/package.use:
sys-apps/portage pypy2_0

/etc/portage/profile/package.use.mask
sys-apps/portage -python_targets_pypy2_0 -pypy2_0

I could give you figures for the performance improvement but they are
probably not very helpful as I have 289 overlays active. That means
portage regularly requires more than a Gig memory for dependency
calculation.

Downsides:
- emerging pypy takes forever and uses a lot of memory
- userfetch and userpriv don't work

Regards,
Florian Philipp


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: [gentoo-user] Portage performance dropped considerably

[hasufell]

On 01/26/2014 05:06 PM, Florian Philipp wrote:
> Downsides:
> - emerging pypy takes forever and uses a lot of memory
> - userfetch and userpriv don't work
> 


It also regularly causes segfaults.

<zmedico/#gentoo-portage>	hasufell: yeah, I get random segfaults with
pypy also (always seems to be in libc iirc)

Re: [gentoo-user] Portage performance dropped considerably

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 442 bytes --]

Am 26.01.2014 17:15, schrieb hasufell:
> On 01/26/2014 05:06 PM, Florian Philipp wrote:
>> Downsides:
>> - emerging pypy takes forever and uses a lot of memory
>> - userfetch and userpriv don't work
>>
> 
> 
> It also regularly causes segfaults.
> 
> <zmedico/#gentoo-portage>	hasufell: yeah, I get random segfaults with
> pypy also (always seems to be in libc iirc)
> 

Not on my machine. YMMV.

Regards,
Florian Philipp


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: [gentoo-user] Portage performance dropped considerably

[covici]

Nikos Chantziaras <realnc@gmail.com> wrote:

> Anyone else noticed this yet? Some portage update seems to have made
> "emerge -uDN @world" perform about 10 times slower than before. It
> used to take seconds, now it takes about 4 minutes only to tell me
> that there's nothing to update. And it does that every time, even
> directly in succession and with the caches warm.
> 
> Is it just me?

I have noticed that it did take a lot longer to do --update --deep
--backtrack=30  world update last time, where it took a while was just
to assemble the list of packages.

-- 
Your life is like a penny.  You're going to lose it.  The question is:
How do
you spend it?

         John Covici
         covici@ccs.covici.com

Re: [gentoo-user] Portage performance dropped considerably

[Tom Wijsman]

On Sun, 26 Jan 2014 16:35:43 +0200
Nikos Chantziaras <realnc@gmail.com> wrote:

> Anyone else noticed this yet? Some portage update seems to have made 
> "emerge -uDN @world" perform about 10 times slower than before. It
> used to take seconds, now it takes about 4 minutes only to tell me
> that there's nothing to update. And it does that every time, even
> directly in succession and with the caches warm.
> 
> Is it just me?

(TL;DR: It is a known problem and it will be improved.)

More people experience this, I've seen multiple reports in all the
communication channels I can think of so far; besides having been
bothered with it myself in the past, but now I'm fine with ~30 seconds.

=== What is going on and what is a cheap solution? ===

Often this is due to backtracking going mad; Portage keeps trying
again, and again, and ... until it has done it for a 10 extra times. And
if a single time perhaps takes like 30 seconds, doing it 10 times extra
gets you quickly to get to 6 minutes or so. Now imagine 30 times...

For this reason, I run --backtrack=0 myself; which finishes in half a
minute or so, but it needs some more experience to deal with the output
that comes after that as you'll need to fix up some things manually.
Not to forget --backtrack=0 has some side effects wrt package rebuilds;
I don't know the fine details, but was warned about this by others.

A solution in between would be to use --backtrack=0 and when that isn't
satisfied raise it a bit until satisfied; if you can get your
--backtrack=0 to work one or another time, you know you won't need
--backtrack=10 soon given that your Portage tree is pretty satisfied.
When you need to raise it, you will come around with --backtrack=5; and
in the case it doesn't you can raise it again, 3 minutes don't hurt.

So, it can be perceived as a trade-off between "I can quickly do this
myself" (given a good understanding of the output) and "Spent some time
doing it for me" (if you don't have the time to do it yourself).

=== Why is this becoming worse over time? ===

Simply put, it's a growth in complexity; we've moved to newer EAPIs
that bring newer features, we have packages improving and thus
providing more USE flags depending on more, we've got more overlays to
pull interesting packages from, as a consequence of the newer EAPIs
there are a lot of sub slot dependencies and the list goes on...

The more options Portage has to consider, the more time it will spend
on it; and if after going down a set of options Portage doesn't find a
satisfiable state, it has to backtrack with more and/or different
options to try again to find another satisfiable state. You can see
that the amount of options (= amount of overlays, packages, versions,
enabled USE flags, ...) takes a huge toll here (amongst other things).

To get an idea of what an @world emerge is doing by default, check:

https://gist.github.com/TomWij/65a61765ea03e6188a35

This is generated using the output additions from the attachment of:

http://article.gmane.org/gmane.linux.gentoo.user/272117

Those whom are interested can also find call tree graphs, although
these rather reveal ~2% improvements because huger improvements require
an actual understanding of how most code works together; they are here:

http://dev.gentoo.org/~tomwij/files/portage-2.2.7-python-2.7-backtrack-0.png
http://dev.gentoo.org/~tomwij/files/portage-2.2.7-python-2.7-backtrack-0-hot.png
http://dev.gentoo.org/~tomwij/files/portage-2.2.7-python-3.3-backtrack-0.png

(Generated with https://code.google.com/p/jrfonseca/wiki/Gprof2Dot)

From the output additions you can see that the slot operators (and thus
sub slots from the newer EAPIs) take up quite a few time. If you think
about what these slot operators are trying to solve, you'll remember
something along the lines of `revdep-rebuild` from a while ago. A lot
of time we've used to spend in that back in the days has moved towards
a cocktail of slot operators and preserved-rebuild.

So, part of the extra time we experience is just a move of time; but
that doesn't make it additional time (unless with lots of backtracking).

Tried to keep this layman explanation short; as I no longer intend to
make this more than a simple explanation, I used to share further
details in the past but I do no longer deem them as necessary today.

Why? Because Portage developer Sebastian Luther (few_) is working on
improving this; the results seem promising, so, let's await a Portage
release with that before we continue to look into this again.

=== What might be done to improve this? ===

Read this message (patch in second mail):

http://thread.gmane.org/gmane.linux.gentoo.portage.devel/4231

The results look promising; it goes from 9:28 to 1:59, whereas
--backtrack=0 is 1:38. In other words, it gets quite close to the time
of --backtrack=0 without bringing its disadvantages; as it still does
the conflict resolving that backtrack does, but somewhat different.

(There are some other smaller patches that also spare out time; if you
are interested, you can find them by searching for him on the mailing
list. However, I think those other patches will be in a release soon.)

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D