[gentoo-user] Portage performance dropped considerably

[gentoo-user] Portage performance dropped considerably

[Nikos Chantziaras]

Anyone else noticed this yet? Some portage update seems to have made 
"emerge -uDN @world" perform about 10 times slower than before. It used 
to take seconds, now it takes about 4 minutes only to tell me that 
there's nothing to update. And it does that every time, even directly in 
succession and with the caches warm.

Is it just me?

Re: [gentoo-user] Portage performance dropped considerably

[hasufell]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 01/26/2014 03:35 PM, Nikos Chantziaras wrote:
> Anyone else noticed this yet? Some portage update seems to have
> made "emerge -uDN @world" perform about 10 times slower than
> before. It used to take seconds, now it takes about 4 minutes only
> to tell me that there's nothing to update. And it does that every
> time, even directly in succession and with the caches warm.
> 
> Is it just me?
> 
> 

Some people don't follow PMS when writing ebuilds which could be one
reason.

https://bugs.gentoo.org/show_bug.cgi?id=174407
https://bugs.gentoo.org/show_bug.cgi?id=487846
https://bugs.gentoo.org/show_bug.cgi?id=393203
https://bugs.gentoo.org/show_bug.cgi?id=486566
https://bugs.gentoo.org/show_bug.cgi?id=434246
https://bugs.gentoo.org/show_bug.cgi?id=311267

toolchain has closed all relevant bugs as WONTFIX so far and even QA
does not seem to care enough (?), although there are alternative
approaches to fix the lack of USE-dynamic SLOTS.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJS5R9dAAoJEFpvPKfnPDWz6zkH/13KbO6s3d5GWe4yXJsL1C7G
xOx24vTwWkeEqmi7+kUxR5Y3LUmZ0Pl4E9n//q5KcgVouKtalwFmqNaVsxQSLG9h
VyRZgGNdvcvTYfdlch8VoiIhUiP+1ZaOsZFuHTOzzfw3qoc52LceJYSyV/lo/x/9
Qe6xT+TuTyUzRJunBFTEzsool8hEhu1UWPYfmjTbZ94wRgSuu68yuL/7hIr75sin
cjEWo25MGzXzf8IhgfM+nI37gurPX136aLuc+JGLTUnYqN9SC1Ad2wjtvHqWJ54O
/kfVlL0790v+l2HyV8CfBs3Z3LktaE7EOgEJBzogHhuO1tjDaoERYDGoE+30pK4=
=tCmP
-----END PGP SIGNATURE-----

[gentoo-user] Re: Portage performance dropped considerably

[Remy Blank]

[-- Attachment #1: Type: text/plain, Size: 98 bytes --]

Nikos Chantziaras wrote:
> Is it just me?

No, I observe the same symptoms here.

-- Remy


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-user] Re: Portage performance dropped considerably

[eroen]

[-- Attachment #1: Type: text/plain, Size: 1319 bytes --]

On Sun, 26 Jan 2014 16:35:43 +0200, Nikos Chantziaras
<realnc@gmail.com> wrote:
>Anyone else noticed this yet? Some portage update seems to have made 
>"emerge -uDN @world" perform about 10 times slower than before. It
>used to take seconds, now it takes about 4 minutes only to tell me
>that there's nothing to update. And it does that every time, even
>directly in succession and with the caches warm.
>
>Is it just me?

You don't say when your baseline was, but the complexity of resolving
the package tree has increased quite a bit over the last year due to
new features like automatic rebuilds of consumers after library updates.

Another somewhat common cause of sudden slowdowns is how portage
resolves conflicts (like packageA requiring an old version of libraryB),
which is rather time-consuming. You can try adding --backtrack=0 to the
emerge command to make it stop and print an error message when
encountering a conflict rather than look for a solution. Then you can
'help' out by manually resolving any conflicts by adding package
versions to /etc/portage/package.mask . Preferably try this *after*
running an update, so your system is up-to-date against your local
version of the gentoo tree, otherwise "normal" simple-to-resolve
conflicts might cause confusion. ;-)

-- 
eroen

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

[gentoo-user] Re: Portage performance dropped considerably

[eroen]

[-- Attachment #1.1: Type: text/plain, Size: 657 bytes --]

On Sun, 26 Jan 2014 07:09:49 -0800, Greg Turner <gmt@malth.us> wrote:
>It would help if there were some decent way to get some statistics
>about where portage is spending all its time (especially, I suspect,
>within the bash code, but the python level would also be interesting
>to analyse).  Anyone know of a good way of doing that?

http://permalink.gmane.org/gmane.linux.gentoo.devel/89518

Also, the attached patch to portage (works with 2.2.7 and 2.2.8 at
least, that too by TomWij) makes portage print out some more
information on what part of dependency resolution currently takes
place, without all the noise from --debug.

-- 
eroen

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: emerg.resolv.time_print.patch --]
[-- Type: text/x-patch, Size: 5748 bytes --]

index 7b77edc..cfd7025 100644
--- a/pym/_emerge/depgraph.py
+++ b/pym/_emerge/depgraph.py
@@ -86,6 +86,10 @@ if sys.hexversion >= 0x3000000:
 else:
 	_unicode = unicode
 
+from time import gmtime, strftime
+def time_print(message):
+	print("\n%s: %s  " % (strftime("%Y-%m-%d %H:%M:%S", gmtime()), message), end="")
+
 class _scheduler_graph_config(object):
 	def __init__(self, trees, pkg_cache, graph, mergelist):
 		self.trees = trees
@@ -1585,6 +1589,10 @@ class depgraph(object):
 			self._spinner_update()
 			while dep_stack:
 				dep = dep_stack.pop()
+#				try:
+#					time_print("  Processing: %s" % dep.cpv)
+#				except AttributeError:
+#					pass
 				if isinstance(dep, Package):
 					if not self._add_pkg_deps(dep,
 						allow_unsatisfied=allow_unsatisfied):
@@ -2106,6 +2114,7 @@ class depgraph(object):
 		for dep_root, dep_string, dep_priority in deps:
 				if not dep_string:
 					continue
+#				time_print("    Reducing: %s" % dep_string)
 				if debug:
 					writemsg_level("\nParent:    %s\n" % (pkg,),
 						noiselevel=-1, level=logging.DEBUG)
@@ -2154,6 +2163,8 @@ class depgraph(object):
 				if not dep_string:
 					continue
 
+#				time_print("    Disjunct: %s" % dep_string)
+
 				if not self._add_pkg_dep_string(
 					pkg, dep_root, dep_priority, dep_string,
 					allow_unsatisfied):
@@ -3005,6 +3016,7 @@ class depgraph(object):
 		virtuals = pkgsettings.getvirtuals()
 		args = self._dynamic_config._initial_arg_list[:]
 
+		time_print("Adding root packages")
 		for arg in self._expand_set_args(args, add_to_digraph=True):
 			for atom in arg.pset.getAtoms():
 				self._spinner_update()
@@ -3116,20 +3128,24 @@ class depgraph(object):
 
 		# Now that the root packages have been added to the graph,
 		# process the dependencies.
+		time_print("Processing dependencies")
 		if not self._create_graph():
 			return 0, myfavorites
 
+		time_print("Checking for slot conflicts")
 		try:
 			self.altlist()
 		except self._unknown_internal_error:
 			return False, myfavorites
 
+		time_print("Checking for blocker conflicts")
 		if (self._dynamic_config._slot_collision_info and
 			not self._accept_blocker_conflicts()) or \
 			(self._dynamic_config._allow_backtracking and
 			"slot conflict" in self._dynamic_config._backtrack_infos):
 			return False, myfavorites
 
+		time_print("Checking for rebuild triggers")
 		if self._rebuild.trigger_rebuilds():
 			backtrack_infos = self._dynamic_config._backtrack_infos
 			config = backtrack_infos.setdefault("config", {})
@@ -3138,12 +3154,14 @@ class depgraph(object):
 			self._dynamic_config._need_restart = True
 			return False, myfavorites
 
+		time_print("Checking if restart is needed")
 		if "config" in self._dynamic_config._backtrack_infos and \
 			("slot_operator_mask_built" in self._dynamic_config._backtrack_infos["config"] or
 			"slot_operator_replace_installed" in self._dynamic_config._backtrack_infos["config"]) and \
 			self.need_restart():
 			return False, myfavorites
 
+		time_print("Checking if we have to prune rebuilds")
 		if not self._dynamic_config._prune_rebuilds and \
 			self._dynamic_config._slot_operator_replace_installed and \
 			self._get_missed_updates():
@@ -3161,10 +3179,12 @@ class depgraph(object):
 			self._dynamic_config._need_restart = True
 			return False, myfavorites
 
+		time_print("Checking if restart is needed")
 		if self.need_restart():
 			# want_restart_for_use_change triggers this
 			return False, myfavorites
 
+		time_print("Checking for parameters that change behavior")
 		if "--fetchonly" not in self._frozen_config.myopts and \
 			"--buildpkgonly" in self._frozen_config.myopts:
 			graph_copy = self._dynamic_config.digraph.copy()
@@ -3188,6 +3208,7 @@ class depgraph(object):
 
 		digraph_nodes = self._dynamic_config.digraph.nodes
 
+		time_print("Checking for changes that are needed")
 		if any(x in digraph_nodes for x in
 			self._dynamic_config._needed_unstable_keywords) or \
 			any(x in digraph_nodes for x in
@@ -3200,6 +3221,7 @@ class depgraph(object):
 			self._dynamic_config._success_without_autounmask = True
 			return False, myfavorites
 
+		time_print("Done resolving!")
 		# We're true here unless we are missing binaries.
 		return (True, myfavorites)
 
@@ -5851,21 +5873,25 @@ class depgraph(object):
 				root_config.root]["root_config"] = root_config
 
 	def _resolve_conflicts(self):
-
+		time_print ("Trying to accept blocker conflicts ")
 		if "complete" not in self._dynamic_config.myparams and \
 			self._dynamic_config._allow_backtracking and \
 			self._dynamic_config._slot_collision_nodes and \
 			not self._accept_blocker_conflicts():
 			self._dynamic_config.myparams["complete"] = True
 
+		time_print ("Resolving slot conflicts for complete graph ")
 		if not self._complete_graph():
 			raise self._unknown_internal_error()
 
+		time_print ("Processing slot conflicts ")
 		self._process_slot_conflicts()
 
 		if self._dynamic_config._allow_backtracking:
+			time_print ("Triggering slot operator reinstalls ")
 			self._slot_operator_trigger_reinstalls()
 
+		time_print ("Validating blockers ")
 		if not self._validate_blockers():
 			# Blockers don't trigger the _skip_restart flag, since
 			# backtracking may solve blockers when it solves slot
@@ -7883,7 +7909,7 @@ def _spinner_start(spinner, myopts):
 		spinner.update = spinner.update_quiet
 
 	if show_spinner:
-		portage.writemsg_stdout("Calculating dependencies  ")
+		portage.writemsg_stdout("%s: Calculating dependencies  " % strftime("%Y-%m-%d %H:%M:%S", gmtime()))
 
 def _spinner_stop(spinner):
 	if spinner is None or \

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-user] Portage performance dropped considerably

[Mariusz Ceier]

No, it's not just you, running "emerge -uNDvp world" takes slightly
over 18 minutes on my laptop (i5 M 430  @ 2.27GHz) - and when there
was lot to update I had to wait over 1hr to just get the list of
packages. I wonder if there's some profiling tool for portage.
Also:
time FEATURES=-xattr emerge owncloud -v
real    6m31.202s
user    2m42.057s
sys     2m1.541s

time FEATURES=xattr emerge owncloud -v
real    30m15.632s
user    22m44.369s
sys     5m30.129s

5 times longer - for something that's essentially copying files.


On 26 January 2014 15:35, Nikos Chantziaras <realnc@gmail.com> wrote:
> Anyone else noticed this yet? Some portage update seems to have made "emerge
> -uDN @world" perform about 10 times slower than before. It used to take
> seconds, now it takes about 4 minutes only to tell me that there's nothing
> to update. And it does that every time, even directly in succession and with
> the caches warm.
>
> Is it just me?
>
>

Re: [gentoo-user] Portage performance dropped considerably

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 1155 bytes --]

Am 26.01.2014 15:35, schrieb Nikos Chantziaras:
> Anyone else noticed this yet? Some portage update seems to have made
> "emerge -uDN @world" perform about 10 times slower than before. It used
> to take seconds, now it takes about 4 minutes only to tell me that
> there's nothing to update. And it does that every time, even directly in
> succession and with the caches warm.
> 
> Is it just me?
> 
> 

As a remedy, you can try to use portage with pypy. I've used this setup
for half a year and it works well.

make.conf:
PYTHON_TARGETS="python2_7 python3_3 pypy2_0"

/etc/portage/package.keywords:
dev-python/pypy
virtual/pypy

/etc/portage/package.use:
sys-apps/portage pypy2_0

/etc/portage/profile/package.use.mask
sys-apps/portage -python_targets_pypy2_0 -pypy2_0

I could give you figures for the performance improvement but they are
probably not very helpful as I have 289 overlays active. That means
portage regularly requires more than a Gig memory for dependency
calculation.

Downsides:
- emerging pypy takes forever and uses a lot of memory
- userfetch and userpriv don't work

Regards,
Florian Philipp


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: [gentoo-user] Portage performance dropped considerably

[hasufell]

On 01/26/2014 05:06 PM, Florian Philipp wrote:
> Downsides:
> - emerging pypy takes forever and uses a lot of memory
> - userfetch and userpriv don't work
> 


It also regularly causes segfaults.

<zmedico/#gentoo-portage>	hasufell: yeah, I get random segfaults with
pypy also (always seems to be in libc iirc)

Re: [gentoo-user] Re: Portage performance dropped considerably

[Alan McKinnon]

On 26/01/2014 17:24, eroen wrote:
> On Sun, 26 Jan 2014 16:35:43 +0200, Nikos Chantziaras
> <realnc@gmail.com> wrote:
>> Anyone else noticed this yet? Some portage update seems to have made 
>> "emerge -uDN @world" perform about 10 times slower than before. It
>> used to take seconds, now it takes about 4 minutes only to tell me
>> that there's nothing to update. And it does that every time, even
>> directly in succession and with the caches warm.
>>
>> Is it just me?
> 
> You don't say when your baseline was, but the complexity of resolving
> the package tree has increased quite a bit over the last year due to
> new features like automatic rebuilds of consumers after library updates.
> 
> Another somewhat common cause of sudden slowdowns is how portage
> resolves conflicts (like packageA requiring an old version of libraryB),
> which is rather time-consuming. You can try adding --backtrack=0 to the
> emerge command to make it stop and print an error message when
> encountering a conflict rather than look for a solution. Then you can
> 'help' out by manually resolving any conflicts by adding package
> versions to /etc/portage/package.mask . Preferably try this *after*
> running an update, so your system is up-to-date against your local
> version of the gentoo tree, otherwise "normal" simple-to-resolve
> conflicts might cause confusion. ;-)
> 

I've been noticing these slowdowns for a few months now too. I'm
somewhat conflicted in my head about them, as unresolved blockers is now
something that happens very rarely. How often did we do this in the past:

emerge -avuND world
  stare at output trying to figure out wtf?
emerge -C <some problem package>
emerge -avuND world
  emerge problem package back if world update didn't already do it

That used to happen A LOT, and it took much longer than 4 minutes.
Nowadays it happens seldom but the resolution does take 4 minutes.

So I dunno, it's annoying to have to wait, but it also prevents a lot of
wasted time by doing what software can do so well - detecting dependency
issues.



-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] Portage performance dropped considerably

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 442 bytes --]

Am 26.01.2014 17:15, schrieb hasufell:
> On 01/26/2014 05:06 PM, Florian Philipp wrote:
>> Downsides:
>> - emerging pypy takes forever and uses a lot of memory
>> - userfetch and userpriv don't work
>>
> 
> 
> It also regularly causes segfaults.
> 
> <zmedico/#gentoo-portage>	hasufell: yeah, I get random segfaults with
> pypy also (always seems to be in libc iirc)
> 

Not on my machine. YMMV.

Regards,
Florian Philipp


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: [gentoo-user] Re: Portage performance dropped considerably

[hasufell]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 01/26/2014 06:42 PM, Alan McKinnon wrote:
> On 26/01/2014 17:24, eroen wrote:
>> On Sun, 26 Jan 2014 16:35:43 +0200, Nikos Chantziaras 
>> <realnc@gmail.com> wrote:
>>> Anyone else noticed this yet? Some portage update seems to have
>>> made "emerge -uDN @world" perform about 10 times slower than
>>> before. It used to take seconds, now it takes about 4 minutes
>>> only to tell me that there's nothing to update. And it does
>>> that every time, even directly in succession and with the
>>> caches warm.
>>> 
>>> Is it just me?
>> 
>> You don't say when your baseline was, but the complexity of
>> resolving the package tree has increased quite a bit over the
>> last year due to new features like automatic rebuilds of
>> consumers after library updates.
>> 
>> Another somewhat common cause of sudden slowdowns is how portage 
>> resolves conflicts (like packageA requiring an old version of
>> libraryB), which is rather time-consuming. You can try adding
>> --backtrack=0 to the emerge command to make it stop and print an
>> error message when encountering a conflict rather than look for a
>> solution. Then you can 'help' out by manually resolving any
>> conflicts by adding package versions to /etc/portage/package.mask
>> . Preferably try this *after* running an update, so your system
>> is up-to-date against your local version of the gentoo tree,
>> otherwise "normal" simple-to-resolve conflicts might cause
>> confusion. ;-)
>> 
> 
> I've been noticing these slowdowns for a few months now too. I'm 
> somewhat conflicted in my head about them, as unresolved blockers
> is now something that happens very rarely. How often did we do this
> in the past:
> 
> emerge -avuND world stare at output trying to figure out wtf? 
> emerge -C <some problem package> emerge -avuND world emerge problem
> package back if world update didn't already do it
> 
> That used to happen A LOT, and it took much longer than 4 minutes. 
> Nowadays it happens seldom but the resolution does take 4 minutes.
> 
> So I dunno, it's annoying to have to wait, but it also prevents a
> lot of wasted time by doing what software can do so well -
> detecting dependency issues.
> 
> 
> 

Dependency resolution is broken and incomplete. Portage is unable to
print useful autounmask and similar messages to solve conflicts
manually, is unable to solve circular deps at all and bails out on
simple things like only updating vim when gvim is installed as well.
Then we have dirty workarounds like PDEPEND which shouldn't be there
in the first place...

It's a miracle that it works at all, especially when people keep
breaking the cache and rely on undefined behavior. Also... we still
have binary files in the tree.

Each EAPI adds more complexity to the dependency calculation. We have
no performance regression tests. We don't have many people who want to
look into portage code (can't blame them and since ferringb is gone we
don't have any1 who works on the QA-side of the code). Afais, it will
get a lot worse.

So, not sure where your optimism comes from. But... some devs are
interested in starting from scratch or picking up pkgcore (which would
be the most sane thing to do IMO).
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJS5U40AAoJEFpvPKfnPDWziK0IAKwuPe4DOBvamSkhtYbipZOv
CdCmt4qjlYn/NjLMkyb8I5AO1m3rziHKuFnfMAksFodTZx9HJ8rbXh1H75bGNt+i
k1cJ4Z6eg9R6hHqsBXAdwBfl4eDouINYMs2Q2R85XFD2QdZKUE/8AcUU5s2YHkxR
NraYC/1n2LxUaXwA8D66KNHKSR31Gb5ISd+Slt+kvAGpXjRDJCDRAWD/QChPkVUL
06RA6qjIhmAooWo3x5lcBjpGUsVkiPk15sF3E0t1qyjp78eiOq8cZBMRYxnhUVN+
AtQlESyzVmYjaCI557fsPr6sZasD69U9luZM6UUToeDSoK7O81s99MilhqUGpKA=
=vPSH
-----END PGP SIGNATURE-----

Re: [gentoo-user] Portage performance dropped considerably

[covici]

Nikos Chantziaras <realnc@gmail.com> wrote:

> Anyone else noticed this yet? Some portage update seems to have made
> "emerge -uDN @world" perform about 10 times slower than before. It
> used to take seconds, now it takes about 4 minutes only to tell me
> that there's nothing to update. And it does that every time, even
> directly in succession and with the caches warm.
> 
> Is it just me?

I have noticed that it did take a lot longer to do --update --deep
--backtrack=30  world update last time, where it took a while was just
to assemble the list of packages.

-- 
Your life is like a penny.  You're going to lose it.  The question is:
How do
you spend it?

         John Covici
         covici@ccs.covici.com

Re: [gentoo-user] Re: Portage performance dropped considerably

[Alan McKinnon]

On 26/01/2014 20:04, hasufell wrote:
> So, not sure where your optimism comes from.


It comes from watching what happens at the end of running emerge, don't
read any more into it than that. Especially not optimism, I think you
might be projecting your own frustrations.

A couple of years ago I used to have to manually resolve blockers about
one world update in two. It started becoming a huge PITA especially as
the deps are usually easy to solve - if I can look at the screen for a
few seconds and figure it out, then software can do the same in
milliseconds. Recent portages now do this properly when viewed from a
results-only perspective.

On my machines, that is what I see happening. That is the ONLY set of
FACTS I have to work on; you may have more.

I'm willing to give up 4 minutes while emerge runs so I don't have to
spend many more minutes right afterwards doing manually the very shit
that software is very good at. Whether portage is a complete pile of
dogshit software or not is beside the point. Even if it is, my 4 minutes
still buys me lots <shrug>



-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] Re: Portage performance dropped considerably

[hasufell]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 01/26/2014 07:30 PM, Alan McKinnon wrote:
> On 26/01/2014 20:04, hasufell wrote:
>> So, not sure where your optimism comes from.
> 
> 
> It comes from watching what happens at the end of running emerge,
> don't read any more into it than that. Especially not optimism, I
> think you might be projecting your own frustrations.

That might be true. Mixing stable and unstable has become more and
more difficult these days.
Starting with USE="-*" on a server (which is a sane thing to do) has
become a lot more difficult as well.

> 
> A couple of years ago I used to have to manually resolve blockers
> about one world update in two. It started becoming a huge PITA
> especially as the deps are usually easy to solve - if I can look at
> the screen for a few seconds and figure it out, then software can
> do the same in milliseconds. Recent portages now do this properly
> when viewed from a results-only perspective.
> 
> On my machines, that is what I see happening. That is the ONLY set
> of FACTS I have to work on; you may have more.
> 
> I'm willing to give up 4 minutes while emerge runs so I don't have
> to spend many more minutes right afterwards doing manually the very
> shit that software is very good at. Whether portage is a complete
> pile of dogshit software or not is beside the point. Even if it is,
> my 4 minutes still buys me lots <shrug>
> 
> 
> 

My pessimism comes from the fact that I wasn't able to communicate to
any1 in real life that gentoo and especially portage have a positive
usability score. Especially to those who have tried it once. As
someone who knows the internals and doesn't read portage messages
about conflicts anymore, but digs into the ebuilds directly... I don't
have a lot of severe problems to maintain any gentoo system. But it is
sad that you need those skills.

Usually I tell people to use a desktop profile, never to use
autounmask, not to mix stable and unstable branch and not to play too
much with per-package useflags unless you are really missing something.

Portage does alienate new users a lot. These performance issues add to
that.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJS5VbwAAoJEFpvPKfnPDWz5NAH/i5viXuWexvbLgVRefuDWCFo
gmLekzmBpEQqvozdxXac1VNFHPWmmY8KVTkTe+JbtgGblzHsiQOug6n47Mpga+dt
tn7f60dLDrTBBpvahVADln9pha3OjtmKDI/JXGV1nZQbFFRBW0x01K6absoFYNs3
bdylCzcG5ZUOCKvBqVzpS8pKVeuBtrItadSOpJTDj6CQys2Q1RVQAl3aIIX96nAx
IcN8eyBeuhQbjACKOfUSgIV/q8XwLdzUHLu//SxJ4psMKL5ne/EQaph9aRI+si2h
bOP9MkKt/QTmj0dGSpbR5DJt6r0tlsmIa1ZIS/DnC7vbKvXVRESUn2tMDes9NEM=
=Hl9N
-----END PGP SIGNATURE-----

Re: [gentoo-user] Re: Portage performance dropped considerably

[Alan McKinnon]

On 26/01/2014 20:41, hasufell wrote:
> My pessimism comes from the fact that I wasn't able to communicate to
> any1 in real life that gentoo and especially portage have a positive
> usability score. Especially to those who have tried it once. As
> someone who knows the internals and doesn't read portage messages
> about conflicts anymore, but digs into the ebuilds directly... I don't
> have a lot of severe problems to maintain any gentoo system. But it is
> sad that you need those skills.
> 
> Usually I tell people to use a desktop profile, never to use
> autounmask, not to mix stable and unstable branch and not to play too
> much with per-package useflags unless you are really missing something.
> 
> Portage does alienate new users a lot. These performance issues add to
> that.


I agree with some points and not so much on others.

Gentoo has always targeted itself at a select bunch of users - those
with large amounts of clue who have tried and failed to get binary
distros to do what they want but can't see a good reason to do the LFS
thing. This is a VERY specific market and people who truly need Gentoo
already know what it will take to drive it. Thank $DEITY the ricer crowd
that were infesting the place 2004-ish have long since moved on and
taken their confirmation bias with them.

Those who want a shiny Linux that works out the box and can be admined
even on hangover days have Ubuntu and RHEL. So when people you talk to
about Gentoo then reject it, don't take it personal, they are not
usually rejecting the distro. What they are *really* saying more often
than not is that Gentoo solves a problem they do not have, and they
don't feel like investigating a complex solution they have no use for.
That's been my experience in this area.

People who need Gentoo and understand what it takes to build from source
usually find the idea of portage very attractive indeed. They might have
issue with the implementation of portage, but folks will put up with a
lot of quirks to get what Gentoo is actually selling.




-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] Re: Portage performance dropped considerably

[Volker Armin Hemmann]

Am 26.01.2014 18:42, schrieb Alan McKinnon:
> On 26/01/2014 17:24, eroen wrote:
>> On Sun, 26 Jan 2014 16:35:43 +0200, Nikos Chantziaras
>> <realnc@gmail.com> wrote:
>>> Anyone else noticed this yet? Some portage update seems to have made 
>>> "emerge -uDN @world" perform about 10 times slower than before. It
>>> used to take seconds, now it takes about 4 minutes only to tell me
>>> that there's nothing to update. And it does that every time, even
>>> directly in succession and with the caches warm.
>>>
>>> Is it just me?
>> You don't say when your baseline was, but the complexity of resolving
>> the package tree has increased quite a bit over the last year due to
>> new features like automatic rebuilds of consumers after library updates.
>>
>> Another somewhat common cause of sudden slowdowns is how portage
>> resolves conflicts (like packageA requiring an old version of libraryB),
>> which is rather time-consuming. You can try adding --backtrack=0 to the
>> emerge command to make it stop and print an error message when
>> encountering a conflict rather than look for a solution. Then you can
>> 'help' out by manually resolving any conflicts by adding package
>> versions to /etc/portage/package.mask . Preferably try this *after*
>> running an update, so your system is up-to-date against your local
>> version of the gentoo tree, otherwise "normal" simple-to-resolve
>> conflicts might cause confusion. ;-)
>>
> I've been noticing these slowdowns for a few months now too. I'm
> somewhat conflicted in my head about them, as unresolved blockers is now
> something that happens very rarely. How often did we do this in the past:
>
> emerge -avuND world
>   stare at output trying to figure out wtf?
> emerge -C <some problem package>
> emerge -avuND world
>   emerge problem package back if world update didn't already do it
>
> That used to happen A LOT, and it took much longer than 4 minutes.
> Nowadays it happens seldom but the resolution does take 4 minutes.
>
> So I dunno, it's annoying to have to wait, but it also prevents a lot of
> wasted time by doing what software can do so well - detecting dependency
> issues.
>
>
>

I disagree with you here. You still get a lot of unresolved blockers and
other problems you have to deal with manually AND portage is unbearable
slow now.

It never was really fast - back in the day pkgcore run cycles around it,
too bad it has died a slow death. 

Now you get a really slow portage, making updates an horrendous
experience plus most of the same old breakage.

The situations is really really bad.

Re: [gentoo-user] Re: Portage performance dropped considerably

[Volker Armin Hemmann]

Am 26.01.2014 19:04, schrieb hasufell:
> So, not sure where your optimism comes from. But... some devs are
> interested in starting from scratch or picking up pkgcore (which would
> be the most sane thing to do IMO).

please do. Please please pretty please.

Re: [gentoo-user] Re: Portage performance dropped considerably

[Alan McKinnon]

On 26/01/2014 21:29, Volker Armin Hemmann wrote:
> Am 26.01.2014 19:04, schrieb hasufell:
>> So, not sure where your optimism comes from. But... some devs are
>> interested in starting from scratch or picking up pkgcore (which would
>> be the most sane thing to do IMO).
> 
> please do. Please please pretty please.
> 
> 
> 
> 
> 


This topic came up on -dev two weeks ago, many people share your
sentiment. But TheRealWorld intrudes, so the general consensus that was
reached is

- pkgcore must first be gotten to EAPI-5; this is a non-trivial amount
of work
- portage is never[1] going to go away

[1] never is defined as "for the foreseeable future"

-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] Re: Portage performance dropped considerably

[Alan McKinnon]

On 26/01/2014 21:28, Volker Armin Hemmann wrote:
>> So I dunno, it's annoying to have to wait, but it also prevents a lot of
>> > wasted time by doing what software can do so well - detecting dependency
>> > issues.
>> >
>> >
>> >
> I disagree with you here. You still get a lot of unresolved blockers and
> other problems you have to deal with manually AND portage is unbearable
> slow now.

I don't see that here. Maybe I'm lucky, maybe every time I my config
deviate from as-shipped I hit the sweet spot. The only blockers I've
gotten from portage for months now has been incompatible USE flags
(totally not portage's fault).


> It never was really fast - back in the day pkgcore run cycles around it,
> too bad it has died a slow death. 


Hey, I never said portage was perfect or even fast :-)

I only said that for a given input I get the output I expect in a
timeframe that isn't intolerable. The process block in the middle is
exactly that - a black box that may be contain a pig's breakfast


> Now you get a really slow portage, making updates an horrendous
> experience plus most of the same old breakage.

Again, I think I'm just lucky. All my old slow machines and VMs run a
very lean Gentoo. It's only this laptop with i7, 16G and a new shiny SSD
that is loaded up with heaps of stuff. If portage is unbearably slow,
then that hardware is hiding it from me.



-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] Re: Portage performance dropped considerably

[Volker Armin Hemmann]

Am 26.01.2014 20:45, schrieb Alan McKinnon:
> On 26/01/2014 21:29, Volker Armin Hemmann wrote:
>> Am 26.01.2014 19:04, schrieb hasufell:
>>> So, not sure where your optimism comes from. But... some devs are
>>> interested in starting from scratch or picking up pkgcore (which would
>>> be the most sane thing to do IMO).
>> please do. Please please pretty please.
>>
>>
>>
>>
>>
>
> This topic came up on -dev two weeks ago, many people share your
> sentiment. But TheRealWorld intrudes, so the general consensus that was
> reached is
>
> - pkgcore must first be gotten to EAPI-5; this is a non-trivial amount
> of work
> - portage is never[1] going to go away
>
> [1] never is defined as "for the foreseeable future"
>
portage does not need to go away. But we do need an alternative.

Re: [gentoo-user] Re: Portage performance dropped considerably

[ny6p01]

[-- Attachment #1: Type: text/plain, Size: 2055 bytes --]

On Sun, Jan 26, 2014 at 09:22:08PM +0200, Alan McKinnon wrote:

> I agree with some points and not so much on others.
> 
> Gentoo has always targeted itself at a select bunch of users - those
> with large amounts of clue who have tried and failed to get binary
> distros to do what they want but can't see a good reason to do the LFS
> thing. This is a VERY specific market and people who truly need Gentoo
> already know what it will take to drive it. Thank $DEITY the ricer crowd
> that were infesting the place 2004-ish have long since moved on and
> taken their confirmation bias with them.
> 
> Those who want a shiny Linux that works out the box and can be admined
> even on hangover days have Ubuntu and RHEL. So when people you talk to
> about Gentoo then reject it, don't take it personal, they are not
> usually rejecting the distro. What they are *really* saying more often
> than not is that Gentoo solves a problem they do not have, and they
> don't feel like investigating a complex solution they have no use for.
> That's been my experience in this area.
> 
> People who need Gentoo and understand what it takes to build from source
> usually find the idea of portage very attractive indeed. They might have
> issue with the implementation of portage, but folks will put up with a
> lot of quirks to get what Gentoo is actually selling.
> 

Just my $.02:

I don't fit into the category of those who 'need' Gentoo. I simply find it
the most coherent distro out there. It doesn't hurt that the webspace is
filled with people like yourself who have hands on knowledge about how to do
stuff, and are willing to share it with others minus the 'tude' you see
elsewhere. 

Also, the documentation is the very highest quality. 90% of the time all I
need is a quick trip to the Gentoo Manual or Wiki to resolve something that
is beyond my paygrade.

It's been like 4 years since I started with Gentoo, and every day I am
grateful for this fabulous disto and the folks who understand it better than
I! :)



[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Re: Portage performance dropped considerably

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 506 bytes --]

On Sun, Jan 26, 2014 at 07:41:52PM +0100, hasufell wrote:
> Starting with USE="-*" on a server (which is a sane thing to do) has
> become a lot more difficult as well.

No, starting with USE="-*" is very dangerous. It is not recommended nor
supported, in any setup, by the dev community. If you do it, you are
solely responsible for your system and you get to keep the broken pieces
when things do not work.
A safer approach would be to turn off the specific use flags you do not
want to support.

William

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Re: Portage performance dropped considerably

[Andreas K. Huettel]

Am Montag, 27. Januar 2014, 00:26:19 schrieb William Hubbs:
> No, starting with USE="-*" is very dangerous. It is not recommended nor
> supported, in any setup, by the dev community. If you do it, you are
> solely responsible for your system and you get to keep the broken pieces
> when things do not work.
> A safer approach would be to turn off the specific use flags you do not
> want to support.

+1 +1 +1

PLEASE do NOT start with USE="-*"
You end up having to pick up the pieces on your own.

-- 

Andreas K. Huettel
Gentoo Linux developer 
KDE, Council, ComRel, Perl
dilfridge@gentoo.org
http://www.akhuettel.de/

Re: [gentoo-user] Re: Portage performance dropped considerably

[Andreas K. Huettel]

[-- Attachment #1: Type: Text/Plain, Size: 1037 bytes --]

PS.

> +1 +1 +1
> 
> PLEASE do NOT start with USE="-*"
> You end up having to pick up the pieces on your own.

If you want to have a sane but minimal set of useflags to start with, the 
recommendation is to use the main profile, e.g. for amd64

default/linux/amd64/13.0

The desktop profiles as e.g. default/linux/amd64/13.0/desktop are designed to 
provide a full-fledged desktop environment with all bells and whistles. The 
difference between desktop and desktop/kde or desktop/gnome is as far as I 
know rather small.

[In general, the main difference between the profiles is default settings for 
useflags. Sometimes a profile also hard-enables or hard-disables a useflag.]

If you intend to run a lightweight desktop environment, both the main profile 
and the desktop profile can be starting points; in one case you may want to 
enable single flags, in the other you may want to disable single flags...

-- 

Andreas K. Huettel
Gentoo Linux developer 
dilfridge@gentoo.org
http://www.akhuettel.de/


[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-user] Re: Portage performance dropped considerably

[Alan McKinnon]

On 26/01/2014 22:44, ny6p01@gmail.com wrote:
> Just my $.02:
> 
> I don't fit into the category of those who 'need' Gentoo. I simply find it
> the most coherent distro out there.


Ah, but you *are* one of those who need Gentoo. It floats your boat, it
satisfies your need to tinker and know what's going on, and using it
probably floods your brain with those fuzzy feel-good chemicals that
make life worthwhile (exactly like chocolate).

See what I did there? I did a shifty redefinition of "need"

:-)


-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] Re: Portage performance dropped considerably

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 386 bytes --]

On Sun, 26 Jan 2014 12:44:19 -0800, ny6p01@gmail.com wrote:

> It doesn't hurt that the webspace is
> filled with people like yourself who have hands on knowledge about how
> to do stuff, and are willing to share it with others minus the 'tude'
> you see elsewhere. 

Now you really have insulted Alan :)


-- 
Neil Bothwick

OPERATOR ERROR: Nyah, Nyah, Nyah, Nyah, Nyah!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Re: Portage performance dropped considerably

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 443 bytes --]

On Sun, 26 Jan 2014 20:29:47 +0100, Volker Armin Hemmann wrote:

> > So, not sure where your optimism comes from. But... some devs are
> > interested in starting from scratch or picking up pkgcore (which would
> > be the most sane thing to do IMO).  
> 
> please do. Please please pretty please.

Does anyone here still use paludis?


-- 
Neil Bothwick

Confucius says "He who posts with broken addresses gets no replies......"

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Re: Portage performance dropped considerably

[hasufell]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 01/27/2014 12:26 AM, William Hubbs wrote:
> On Sun, Jan 26, 2014 at 07:41:52PM +0100, hasufell wrote:
>> Starting with USE="-*" on a server (which is a sane thing to do)
>> has become a lot more difficult as well.
> 
> No, starting with USE="-*" is very dangerous. It is not recommended
> nor supported, in any setup, by the dev community. If you do it,
> you are solely responsible for your system and you get to keep the
> broken pieces when things do not work. A safer approach would be to
> turn off the specific use flags you do not want to support.
> 
> William
> 

That's nonsense imo and I use that setup on multiple servers/routers
without any issues.

It makes sense because you have the most minimal setup possible, the
most minimal codepaths possible which reduces exposure to bugs.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJS5kalAAoJEFpvPKfnPDWz/DcIAIZtMLZ907iMbqQs71Q2KGoY
kKhUavNCg/PxsxnASyRVahf9LBAoJ2ZOya5/V8fcyf5RZEh5M9Jhc05qmEh5FIPU
t7jTyRN1P0rCt3WLv/KMrIkszh/0iPWygu7BGio9KsdqxeUtsSdrQ4ylmjiJKyCJ
mszFnLmG57ovL/Uv4YB/QWyhRBbxf9Be1Vvv1XLHEKouJNzWeuVBoQtECozYpfp+
tLt5HVm9skvk2pnjlAuiIODT3bVlY0sgXlzBaz0EVHTrnId/EUqsn0U8JpVqOw05
HXRNt2GZtJBJuU6J/q9whvLt/oHl7yZsbilS9LbuWydO5ooAywO27qtuR24OVXc=
=L7hT
-----END PGP SIGNATURE-----

Re: [gentoo-user] Re: Portage performance dropped considerably

[Tanstaafl]

On 2014-01-26 1:04 PM, hasufell <hasufell@gentoo.org> wrote:
> So, not sure where your optimism comes from. But... some devs are
> interested in starting from scratch or picking up pkgcore (which would
> be the most sane thing to do IMO).

?

If the problem is really this potentially serious, why start from 
scratch, when Paludis is already very mature? Is it pure politics 
(someone just doesn't like Ciaran)?

Re: [gentoo-user] Re: Portage performance dropped considerably

[Helmut Jarausch]

On 01/26/2014 08:55:35 PM, Alan McKinnon wrote:
> Again, I think I'm just lucky.

I don't think it's luck.
A portage system likes to be updated very often (best: each day).
I have made the experience that updating a machine which hasn't been  
updated
for a long time (say one year), is just pain.

Normally, in those cases, I prefer to remove everything from / and  
/usr, copy an actively maintained machine to it
and do the necessary customizations for that machine.

Helmut

Re: [gentoo-user] Re: Portage performance dropped considerably

[Alan McKinnon]

On 27/01/2014 13:59, Tanstaafl wrote:
> On 2014-01-26 1:04 PM, hasufell <hasufell@gentoo.org> wrote:
>> So, not sure where your optimism comes from. But... some devs are
>> interested in starting from scratch or picking up pkgcore (which would
>> be the most sane thing to do IMO).
> 
> ?
> 
> If the problem is really this potentially serious, why start from
> scratch, when Paludis is already very mature? Is it pure politics
> (someone just doesn't like Ciaran)?
> 
> 
> 


No-one likes to admit it, but I think there's some NIH going on

-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] Re: Portage performance dropped considerably

[hasufell]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 01/27/2014 02:06 PM, Alan McKinnon wrote:
> On 27/01/2014 13:59, Tanstaafl wrote:
>> On 2014-01-26 1:04 PM, hasufell <hasufell@gentoo.org> wrote:
>>> So, not sure where your optimism comes from. But... some devs
>>> are interested in starting from scratch or picking up pkgcore
>>> (which would be the most sane thing to do IMO).
>> 
>> ?
>> 
>> If the problem is really this potentially serious, why start
>> from scratch, when Paludis is already very mature? Is it pure
>> politics (someone just doesn't like Ciaran)?
>> 
>> 
>> 
> 
> 
> No-one likes to admit it, but I think there's some NIH going on
> 

If it's about performance (in the sense of speed), then paludis is
worse, because dependency calculation is more complex/complete there.
Debatable if that's really a problem, though.

If it's about code quality... it's probably better, especially because
it's not that old. But from a few looks at the code, it's not properly
documented at class/method level (at least I could not find any comments).
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJS5mW2AAoJEFpvPKfnPDWzsTEH/jsxytMr2IQhNZcPdWhyNdu1
vCkiqV/kejjPtd9xDuRGMa6Adh3Jka1+I287J5ie61H+SU/4+mHYtkq9npohi9T8
YFgg8GsdrTfeC3o/d1qIBPHrKCAVs11D9IBYnFjNS4DkqM9chj8itnt7GTRWGZvx
0i5/nLQPq6fCW3nz9QzRfa6Mocx7m803ayWBpBSocr2xuIX8AsibG8YGTJzPLl64
IeZ31QPHJ5CqyIo5cidS2k4ZKnf0tEAJVoJUBWr412UHs+s2w1XaeyWPc1Faena7
L40VVjQp/jTjIz6GgMdbQrn/RGNe4rjxNQY2MuSezbqme8NDEtz1PnEZoQR1n9U=
=L3AQ
-----END PGP SIGNATURE-----

Re: [gentoo-user] Re: Portage performance dropped considerably

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 472 bytes --]

On Mon, 27 Jan 2014 14:57:10 +0100, hasufell wrote:

> If it's about performance (in the sense of speed), then paludis is
> worse, because dependency calculation is more complex/complete there.

That makes no sense at all. Paludis is written in a different language
using different algorithms. It's not about the amount of work it does so
much as how efficiently it does it.


-- 
Neil Bothwick

Earlier, I didn't have time to finish anything. This time I w

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Re: Portage performance dropped considerably

[hasufell]

On 01/27/2014 10:48 PM, Neil Bothwick wrote:
> On Mon, 27 Jan 2014 14:57:10 +0100, hasufell wrote:
> 
>> If it's about performance (in the sense of speed), then paludis
>> is worse, because dependency calculation is more complex/complete
>> there.
> 
> That makes no sense at all. Paludis is written in a different
> language using different algorithms. It's not about the amount of
> work it does so much as how efficiently it does it.
> 
> 

That's exactly what I was saying. I was talking about speed, not
efficiency.

Re: [gentoo-user] Re: Portage performance dropped considerably

[Stefan G. Weichinger]

Am 27.01.2014 13:06, schrieb Helmut Jarausch:
> On 01/26/2014 08:55:35 PM, Alan McKinnon wrote:
>> Again, I think I'm just lucky.
> 
> I don't think it's luck.
> A portage system likes to be updated very often (best: each day).
> I have made the experience that updating a machine which hasn't been
> updated
> for a long time (say one year), is just pain.

I agree ... I currently go through several such machines for customers
.. I am getting more experienced as I do the same tricky steps again and
again but my recommendation now is to get their boxes upgraded at least
once a year. Like the annual check for your car or something ...

I even consider to recommend them a quarterly review done by me and a
following offer to do the important/useful/easy/existing upgrades.

(this is from the view of a one-man-IT-consulter having X customers with
servers out there. I don't upgrade all around for free all the time)

Stefan

Re: [gentoo-user] Re: Portage performance dropped considerably

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 819 bytes --]

On Mon, 27 Jan 2014 22:54:28 +0100, hasufell wrote:

> >> If it's about performance (in the sense of speed), then paludis
> >> is worse, because dependency calculation is more complex/complete
> >> there.  
> > 
> > That makes no sense at all. Paludis is written in a different
> > language using different algorithms. It's not about the amount of
> > work it does so much as how efficiently it does it.

> That's exactly what I was saying. I was talking about speed, not
> efficiency.

But the efficiency of the algorithm, and the language, affects the speed.
You can't presume "it does more, therefore it takes longer" if the two
programs do things in very different ways.


-- 
Neil Bothwick

Law of Mechanical Repair: After your hands become coated with
grease, your nose will begin to itch.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Re: Portage performance dropped considerably

[hasufell]

On 01/27/2014 11:57 PM, Neil Bothwick wrote:
> On Mon, 27 Jan 2014 22:54:28 +0100, hasufell wrote:
> 
>>>> If it's about performance (in the sense of speed), then paludis
>>>> is worse, because dependency calculation is more complex/complete
>>>> there.  
>>>
>>> That makes no sense at all. Paludis is written in a different
>>> language using different algorithms. It's not about the amount of
>>> work it does so much as how efficiently it does it.
> 
>> That's exactly what I was saying. I was talking about speed, not
>> efficiency.
> 
> But the efficiency of the algorithm, and the language, affects the speed.
> You can't presume "it does more, therefore it takes longer" if the two
> programs do things in very different ways.
> 
> 

For people who are used to portage, paludis will be _slower_ in total,
although the dependency calculation will be more accurate.

Re: [gentoo-user] Re: Portage performance dropped considerably

[Walter Dnes]

On Sun, Jan 26, 2014 at 05:26:19PM -0600, William Hubbs wrote
> On Sun, Jan 26, 2014 at 07:41:52PM +0100, hasufell wrote:
> > Starting with USE="-*" on a server (which is a sane thing to do) has
> > become a lot more difficult as well.
> 
> No, starting with USE="-*" is very dangerous. It is not recommended nor
> supported, in any setup, by the dev community. If you do it, you are
> solely responsible for your system and you get to keep the broken pieces
> when things do not work.
> A safer approach would be to turn off the specific use flags you do not
> want to support.

  I run icewm desktop with the specific set of apps that I need.  In
make.conf I have...

USECPU="abi_x86_64 mmx mmxext sse sse2 sse3 ssse3"
USEOTHER=" X a52 aac bzip2 cxx dga dri exif ffmpeg flac fortran gallium gif intel jpeg mng mp3 mpeg ncurses nptl nptlonly nsplugin offensive ogg opengl openrc png posix readline ssl suid theora threads tiff tools truetype vim-syntax vorbis xcomposite webm x264 xpm xv xvid zlib"
USE="-* ${USECPU} ${USEOTHER}"

  Yes, you can do stuff like that in make.conf.  Plus in package.use I
have several additional package-specific entries.  Once a flag shows up
multiple times in package.use, I look at moving it into make.conf.

  If portage finds a package requires a missing dependancy, it lists it.
Then I can add the flag as required.  Or I may decide that I don't want
the package that badly.  If you want to look at it that way, what I've
really done is to replace the default USE flag set with my own defaults,
which are more appropriate for my machine and my usage.

-- 
Walter Dnes <waltdnes@waltdnes.org>
I don't run "desktop environments"; I run useful applications

[gentoo-user] Re: Portage performance dropped considerably

[Martin Vaeth]

hasufell <hasufell@gentoo.org> wrote:
>
> On 01/27/2014 12:26 AM, William Hubbs wrote:
>>
>> No, starting with USE="-*" is very dangerous.
>
> That's nonsense imo

No, William is completely right.

> and I use that setup on multiple servers/routers without any issues.

No one doubts that it is *possible* to add the correct USE for
every single package manually, but it is not a good idea to hide
the recommended defaults.

> It makes sense because you have the most minimal setup possible

This is not true, to start with: For instance, USE=minimal will
usually choose a more minimal setup.
With "-*" you will actually *disable* the default USE=minimal
for e.g. www-client/firefox, x11-apps/startx, sys-block/blocks,
dev-db/unixODBC, ... and thus get a setup which is even larger
than the recommended default.

> most minimal codepaths possible which reduces exposure to bugs.

No, you usually get less tested (and by upstream considered untypical)
codepaths which actually increases the probability to hit a bug
nobody did hit/test yet.

The USE="-*" approach was reasonable before EAPI=1 was introduced:
In these days, unusual codepaths would have been set by "negative"
USE-flags, e.g. IUSE="nocxx" for gcc.
Nowadays, the upstream-recommended codepaths are set by default-USE-Flags
in the ebuild, i.e. now the same is called IUSE="+cxx" in gcc.
Using -* you disable such defaults which are usually there for a
good reason.

Of course, if you know and care what every single USE-flags for every
single package does, it does not matter much which approach you take,
but I would guess that even in this case you need more exceptions
in /etc/portage/package.use with USE="-*" than with USE="".

Moreover, even for updates, it happens occassionally that a package
gets an additional USE-flag, whose default is then usually chosen in
such a way as the behaviour was before - so you risk dropping
crucial behaviour on updates if you are not very careful.

Re: [gentoo-user] Re: Portage performance dropped considerably

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 657 bytes --]

On Tue, 28 Jan 2014 00:35:24 +0100, hasufell wrote:

> > But the efficiency of the algorithm, and the language, affects the
> > speed. You can't presume "it does more, therefore it takes longer" if
> > the two programs do things in very different ways.

> For people who are used to portage, paludis will be _slower_ in total,
> although the dependency calculation will be more accurate.

Do you have any benchmarks or other figures for this?


-- 
Neil Bothwick

Octal: (n.) a base-8 counting system designed so that one hand may count
upon the fingers of the other. Thumbs are not used, and the index finger
is reserved for the 'carry.'

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-user] Re: Portage performance dropped considerably

[Martin Vaeth]

Walter Dnes <waltdnes@waltdnes.org> wrote:
> USE="-* ${USECPU} ${USEOTHER}"
>
> If you want to look at it that way, what I've
> really done is to replace the default USE flag set with my own defaults

... *including* the defaults specified in individual ebuilds.
About the default flags in profiles one may argue, but the
ebuild-enabled defaults are usually very decent and strongly
encouraged by upstream.
For this reason using "-Flag" in make.conf should better
be used sparringly: Unless you are sure that you want to disable
a particular feature really *globally*, it is better to turn it off
in /etc/portage/package.use for each package separately so that
packages which you might install in some future (or which add this
flag in some future) do not have already changed their default.

[gentoo-user] Re: Portage performance dropped considerably

[Martin Vaeth]

Alan McKinnon <alan.mckinnon@gmail.com> wrote:
> On 27/01/2014 13:59, Tanstaafl wrote:
>>
>> If the problem is really this potentially serious, why start from
>> scratch, when Paludis is already very mature? Is it pure politics
>> (someone just doesn't like Ciaran)?
>
> No-one likes to admit it, but I think there's some NIH going on

I don't think this is the reason - one could probably do a fork.
However, the approach taken by paludis is radically different
(and often not to the better - or at least, this is very disputable).
From the user perspective a change is not a good idea either:

The configuration is completely different, so more or less you have
to start from scratch, and if you want to switch back you have to
start from scratch again or update two setups in parallel
(and test in parallel that your changes were OK).

Moreover, you risk stability of your system if you change, since there
is no guarantee that the data written to /var/db/ is compatible:
There were times when this format was rather extended by portage,
so you could be missing some information after you switch back;
I do not know what is the current state of compatibility here.

Unfortunately, many of these arguments could hold for pkgcore as well.
At least, the compatibility of config and /var/db files is something
which has to be seriously considered by everybody who plans to change
on a production system.

Re: [gentoo-user] Re: Portage performance dropped considerably

[hasufell]

On 01/28/2014 02:34 AM, Martin Vaeth wrote:
> hasufell <hasufell@gentoo.org> wrote:
>>
>> On 01/27/2014 12:26 AM, William Hubbs wrote:
>>>
>>> No, starting with USE="-*" is very dangerous.
>>
>> That's nonsense imo
> 
> No, William is completely right.
> 
>> and I use that setup on multiple servers/routers without any issues.
> 
> No one doubts that it is *possible* to add the correct USE for
> every single package manually, but it is not a good idea to hide
> the recommended defaults.
> 

As someone who writes those recommendations, I disagree. That's why many
of my packages don't have a lot of them, because I don't like them in
the first place. Another nice thing you can do is mess with USE_ORDER.
And now don't tell me that is another bad idea. This is Gentoo.

>> It makes sense because you have the most minimal setup possible
> 
> This is not true, to start with: For instance, USE=minimal will
> usually choose a more minimal setup.
> With "-*" you will actually *disable* the default USE=minimal
> for e.g. www-client/firefox, x11-apps/startx, sys-block/blocks,
> dev-db/unixODBC, ... and thus get a setup which is even larger
> than the recommended default.
> 

USE="-* minimal"

>> most minimal codepaths possible which reduces exposure to bugs.
> 
> No, you usually get less tested (and by upstream considered untypical)
> codepaths which actually increases the probability to hit a bug
> nobody did hit/test yet.
> 

Many defaults gentoo sets do not have anything to do with default
codepaths upstream has tested. So this argument works both ways.
Especially after a profile is activated.

> The USE="-*" approach was reasonable before EAPI=1 was introduced:
> In these days, unusual codepaths would have been set by "negative"
> USE-flags, e.g. IUSE="nocxx" for gcc.
> Nowadays, the upstream-recommended codepaths are set by default-USE-Flags
> in the ebuild, i.e. now the same is called IUSE="+cxx" in gcc.
> Using -* you disable such defaults which are usually there for a
> good reason.

As above, our defaults are not necessarily following upstream
recommendations/defaults. Apache alone should make you think about that
claim.

> 
> Of course, if you know and care what every single USE-flags for every
> single package does, it does not matter much which approach you take,
> but I would guess that even in this case you need more exceptions
> in /etc/portage/package.use with USE="-*" than with USE="".
> 

I made the opposite experience.

> Moreover, even for updates, it happens occassionally that a package
> gets an additional USE-flag, whose default is then usually chosen in
> such a way as the behaviour was before - so you risk dropping
> crucial behaviour on updates if you are not very careful.
> 

I am careful. The amount of crucial packages on my servers are not that
big and I definitely watch _any_ update, unless I want a mysql update to
break hell.


Besides, if a useflag combination breaks something unexpectedly (e.g.
the build or unrelated features) then it's a bug (minimum is to
communicate the situation via elog). If disabling one useflag breaks the
whole package, then it's a bug. That's something the packager has to
care about and arch testers usually run all(or most?) useflag
permutations before stabilizing.
There is no excuse. Every other "breakage" is expected, because I have
disabled the features.
The power of useflags imply that I can mix them up any way I want. All
of those mixtures must be supported by the maintainer, unless he warns
the user about it through the ebuild, masks the useflag or sets an
appropriate REQUIRED_USE constraint. Otherwise... it's a bug.

Re: [gentoo-user] Re: Portage performance dropped considerably

[Walter Dnes]

On Tue, Jan 28, 2014 at 01:42:22AM +0000, Martin Vaeth wrote
> Walter Dnes <waltdnes@waltdnes.org> wrote:
> > USE="-* ${USECPU} ${USEOTHER}"
> >
> > If you want to look at it that way, what I've
> > really done is to replace the default USE flag set with my own defaults
> 
> ... *including* the defaults specified in individual ebuilds.
> About the default flags in profiles one may argue, but the
> ebuild-enabled defaults are usually very decent and strongly
> encouraged by upstream.
> For this reason using "-Flag" in make.conf should better
> be used sparringly: Unless you are sure that you want to disable
> a particular feature really *globally*, it is better to turn it off
> in /etc/portage/package.use for each package separately so that
> packages which you might install in some future (or which add this
> flag in some future) do not have already changed their default.

  I ran into that with the "suid" flag in xorg-server.  I enabled "suid"
for xorg-server in package.use.  Then a couple of other packages started
needing it.  I test ran...

USE="suid" emerge -pv --deep --changed-use --update @world

...and no additional packages (beyond the ones I wanted) needed to be
rebuilt.  That's when I moved "suid" into make.conf.

-- 
Walter Dnes <waltdnes@waltdnes.org>
I don't run "desktop environments"; I run useful applications

[gentoo-user] Re: Portage performance dropped considerably

[Martin Vaeth]

hasufell <hasufell@gentoo.org> wrote:
>
> Many defaults gentoo sets do not have anything to do with default
> codepaths upstream has tested.

I disagree: The USE-enabling in ebuilds usually follows upstream.
IIRC there was even a policy for gentoo developers which strongly
suggested this.

> As above, our defaults are not necessarily following upstream
> recommendations/defaults. Apache alone should make you think about that
> claim.

I never installed apache.
However, especially for packages for which the choice of algorithms
has to be selected (USE-flags thread, jit) or of protocols/interfaces
(openssl or gnutls, neon or other, sqlite or mysql, openvpn[lzo],
qtgui[exceptions], mesa, freetype, wine), the installation of tools
(utils, examples, tk, perl, python) or extensions (tls-heartbeat,
introspection, X, readline) the defaults usually follow the
upstream default or recommendation unless there is a severe reason
not to.

> If disabling one useflag breaks the whole package, then it's a bug.

Whether it breaks your machine/setup or not is independent of
whether it breaks a package.

> care about and arch testers usually run all(or most?) useflag
> permutations before stabilizing.

Simple mathematics shows that this cannot be even closely true.
Anyway, this has nothing to do with our discussion.

Re: [gentoo-user] Re: Portage performance dropped considerably

[hasufell]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 01/28/2014 06:45 PM, Martin Vaeth wrote:
> hasufell <hasufell@gentoo.org> wrote:
>> 
>> Many defaults gentoo sets do not have anything to do with
>> default codepaths upstream has tested.
> 
> I disagree: The USE-enabling in ebuilds usually follows upstream. 
> IIRC there was even a policy for gentoo developers which strongly 
> suggested this.
> 

I don't know of any and I strongly disagree with that concept.

>> As above, our defaults are not necessarily following upstream 
>> recommendations/defaults. Apache alone should make you think
>> about that claim.
> 
> I never installed apache. However, especially for packages for
> which the choice of algorithms has to be selected (USE-flags
> thread, jit) or of protocols/interfaces (openssl or gnutls, neon or
> other, sqlite or mysql, openvpn[lzo], qtgui[exceptions], mesa,
> freetype, wine), the installation of tools (utils, examples, tk,
> perl, python) or extensions (tls-heartbeat, introspection, X,
> readline) the defaults usually follow the upstream default or
> recommendation unless there is a severe reason not to.
> 

No, they don't necessarily. There is no consistency about this. It's
up to the maintainer to decide "what most users will want". You want
upstream defaults, others want different things. The decision is made
individually. And profiles totally mess up that concept anyway.

What I was trying to say is: if you allow useflag combinations that
break the package (both in terms of build, runtime or _unexpectedly_
missing features) or break reverse dependencies in those same ways,
then it's a bug, a missing REQUIRED_USE constraint, a missing elog or
whatever.

The whole line of argumentation does not work out anyway, imo.
Thinking that the defaults from e.g. "./configure --help" are what a)
developers have tested most thoroughly and b) users of other distros
like debian, ubuntu etc run... is simply an assumption. Debian rather
goes for enabling whatever they can enable.

Besides that... I run stable arch. And when I have a package that has
severely broken runtime behavior with many useflags disabled (except
for the features I expect to be disabled), then something went
horribly wrong during stabilization.

If we support disabling all useflags on package level (and we do),
then we support disabling all on global level as well. All
_unexpected_ breakage that occurs due to that are ebuild bugs that
have incorrect dependencies or missing REQUIRED_USE constraints.

Defaults are just a usability thing, nothing more.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJS5/H3AAoJEFpvPKfnPDWzGXEH/Aw68GvxkA98GoGfpYeD5jAB
TEc6BE7BXX+SjToZZd2LGvyo0gpzocTwYf0Y2OMkVvlrft1a4LJVPX1pHK8NSPdv
DIl7r+AosUcddBrSI45VuCC53sy66XxUDrsKnuXu1Qm9FlfIHhYTNcfxQM1v4UIx
/IP3X+MzH+kklPnYqzHDwxY+lpS1JB3lCPbYvKoJLvk22s+F9ZMg2zdserWRnSRB
EYKrw7ZbnornP71K7dQykQe0fh9f6d/s1fA56fvQ968Pfa1QIF/7eSd2270GF9Vq
5KTWATp8rThfo9O526+A4bwgceDFe04Ksbf6p1oOjxe6Hn4MIo020YFhVl7HQNg=
=NMPh
-----END PGP SIGNATURE-----

Re: [gentoo-user] Re: Portage performance dropped considerably

[Kerin Millar]

hasufell wrote:

<snip>

> If we support disabling all useflags on package level (and we do),
> then we support disabling all on global level as well. All
> _unexpected_  breakage that occurs due to that are ebuild bugs that
> have incorrect dependencies or missing REQUIRED_USE constraints.
>
> Defaults are just a usability thing, nothing more.

Amen. The notion that a particular USE flag combination is 'wrong' for a 
package is nonsensical. The notion that a user is culpable for QA issues 
that are solely the preserve of the maintainer even the more so. Any 
such issues should either be fixed or the options afforded to the user 
redacted if the maintainer is unwilling or incapable of supporting them. 
Scolding those users who have the audacity to configure Gentoo to serve 
their requirements is a distraction, to put it politely.

--Kerin

Re: [gentoo-user] Re: Portage performance dropped considerably

[hasufell]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 01/27/2014 02:06 PM, Alan McKinnon wrote:
> On 27/01/2014 13:59, Tanstaafl wrote:
>> On 2014-01-26 1:04 PM, hasufell <hasufell@gentoo.org> wrote:
>>> So, not sure where your optimism comes from. But... some devs
>>> are interested in starting from scratch or picking up pkgcore
>>> (which would be the most sane thing to do IMO).
>> 
>> ?
>> 
>> If the problem is really this potentially serious, why start
>> from scratch, when Paludis is already very mature? Is it pure
>> politics (someone just doesn't like Ciaran)?
>> 
>> 
>> 
> 
> 
> No-one likes to admit it, but I think there's some NIH going on
> 

I just tried paludis again (after some time).

Things that make it currently impossible to properly migrate my
portage config:
* you cannot unmask USE flags at all, not without hackery... and that
is really non-trivial for unmasking abi_x86_32 globally, because those
masks are scattered across a lot of files in profiles/
The explanation from the paludis developer is simply wrong:
http://paludis.exherbo.org/trac/ticket/817
* seems there is no equivalent to --dynamic-deps=y (which is default
in emerge)... either I am missing something or this is a pretty good
reason to not use it
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJS6cwfAAoJEFpvPKfnPDWz+MoIAJvH9zDbsVQaRwyb+yMEowJ8
qXjaLmDTx5BFyyL7tSelFEYyNwh0DN1ypyOQu2VkScNOTNIbqfffWXsAPoe4GJrP
pngtb9xo4H4/IIdtr7i8fwRU937UK5V4Fq0Er/e56SGpdHG3G+emxrBeuB2Y6n0M
m+gdEI1xmSuB/YOd/byDc+t9qK1688qM23fHJd/SsW732FY9ooUlfSZuO39ltjpk
96ojLyGe4TAp5zkk2BNBbpLXyuAtszwsc8U5nPN89v87gbC7gH5pIrJDXbQkRfMF
EP0ouZ6nfB4cDqFM1/GVvJ2+V24jleWkpV3UQmCPDAd18T6Qa/fkujz0JuijXAg=
=FfQN
-----END PGP SIGNATURE-----

Re: [gentoo-user] Portage performance dropped considerably

[Stroller]

On 30 Jan 2014, at 03:50 am, hasufell <hasufell@gentoo.org> wrote:

> I just tried paludis again (after some time).
> ...
> * you cannot unmask USE flags at all, not without hackery... and that
> is really non-trivial for unmasking abi_x86_32 globally, because those
> masks are scattered across a lot of files in profiles/
> The explanation from the paludis developer is simply wrong:
> http://paludis.exherbo.org/trac/ticket/817


"WONTFIX: you can hack around it with your own profile if you need to deal with Gentoo not following its own policies correctly."

Yes, that's the Ciaran McCreesh I remember.

Stroller.

Re: [gentoo-user] Portage performance dropped considerably

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 959 bytes --]

On Sun, 26 Jan 2014 16:53:33 +0100 Mariusz Ceier wrote:
> No, it's not just you, running "emerge -uNDvp world" takes slightly
> over 18 minutes on my laptop (i5 M 430  @ 2.27GHz) - and when there
> was lot to update I had to wait over 1hr to just get the list of
> packages. I wonder if there's some profiling tool for portage.
> Also:
> time FEATURES=-xattr emerge owncloud -v
> real    6m31.202s
> user    2m42.057s
> sys     2m1.541s
> 
> time FEATURES=xattr emerge owncloud -v
> real    30m15.632s
> user    22m44.369s
> sys     5m30.129s
> 
> 5 times longer - for something that's essentially copying files.

Slow xattr is a known problem, see bugs 482290, 465000:
https://bugs.gentoo.org/show_bug.cgi?id=482290
https://bugs.gentoo.org/show_bug.cgi?id=465000

But xattr bug shows itself during install phase and has nothing to do
with dependency calculation time mentioned in the original mail.

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-user] Re: Portage performance dropped considerably

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 1606 bytes --]

On Sun, 26 Jan 2014 20:30:19 +0200 Alan McKinnon wrote:
> It comes from watching what happens at the end of running emerge, don't
> read any more into it than that. Especially not optimism, I think you
> might be projecting your own frustrations.
> 
> A couple of years ago I used to have to manually resolve blockers about
> one world update in two. It started becoming a huge PITA especially as
> the deps are usually easy to solve - if I can look at the screen for a
> few seconds and figure it out, then software can do the same in
> milliseconds. Recent portages now do this properly when viewed from a
> results-only perspective.
> 
> On my machines, that is what I see happening. That is the ONLY set of
> FACTS I have to work on; you may have more.
> 
> I'm willing to give up 4 minutes while emerge runs so I don't have to
> spend many more minutes right afterwards doing manually the very shit
> that software is very good at. Whether portage is a complete pile of
> dogshit software or not is beside the point. Even if it is, my 4 minutes
> still buys me lots <shrug>

4 minutes are expendable but... on Atom N270 (my laptop) emerge
-DNuav world takes 40 (yes, forty) minutes to build dependency tree
with sqlite cache enabled and 60 minutes without sqlite. System was
pretty old (not updated aside from GLSA updates for a year). And this
40 minutes repeated many times since USE flag clashes and dependency
resolution failures. So I spent may day, damn whole day(!) for the
sake to just start compiling (distcc is my friend here).

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-user] Re: Portage performance dropped considerably

[Mick]

[-- Attachment #1: Type: Text/Plain, Size: 1848 bytes --]

On Friday 31 Jan 2014 19:03:05 Andrew Savchenko wrote:
> On Sun, 26 Jan 2014 20:30:19 +0200 Alan McKinnon wrote:
> > It comes from watching what happens at the end of running emerge, don't
> > read any more into it than that. Especially not optimism, I think you
> > might be projecting your own frustrations.
> > 
> > A couple of years ago I used to have to manually resolve blockers about
> > one world update in two. It started becoming a huge PITA especially as
> > the deps are usually easy to solve - if I can look at the screen for a
> > few seconds and figure it out, then software can do the same in
> > milliseconds. Recent portages now do this properly when viewed from a
> > results-only perspective.
> > 
> > On my machines, that is what I see happening. That is the ONLY set of
> > FACTS I have to work on; you may have more.
> > 
> > I'm willing to give up 4 minutes while emerge runs so I don't have to
> > spend many more minutes right afterwards doing manually the very shit
> > that software is very good at. Whether portage is a complete pile of
> > dogshit software or not is beside the point. Even if it is, my 4 minutes
> > still buys me lots <shrug>
> 
> 4 minutes are expendable but... on Atom N270 (my laptop) emerge
> -DNuav world takes 40 (yes, forty) minutes to build dependency tree
> with sqlite cache enabled and 60 minutes without sqlite. System was
> pretty old (not updated aside from GLSA updates for a year). And this
> 40 minutes repeated many times since USE flag clashes and dependency
> resolution failures. So I spent may day, damn whole day(!) for the
> sake to just start compiling (distcc is my friend here).

Out of interest what fs are you running portage on?  I changed an old box from 
reiserfs to ext4 and couldn't believe the speed up I got.

-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-user] Portage performance dropped considerably

[hasufell]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 01/30/2014 07:15 PM, Stroller wrote:
> 
> On 30 Jan 2014, at 03:50 am, hasufell <hasufell@gentoo.org> wrote:
> 
>> I just tried paludis again (after some time). ... * you cannot
>> unmask USE flags at all, not without hackery... and that is
>> really non-trivial for unmasking abi_x86_32 globally, because
>> those masks are scattered across a lot of files in profiles/ The
>> explanation from the paludis developer is simply wrong: 
>> http://paludis.exherbo.org/trac/ticket/817
> 
> 
> "WONTFIX: you can hack around it with your own profile if you need
> to deal with Gentoo not following its own policies correctly."
> 
> Yes, that's the Ciaran McCreesh I remember.
> 
> Stroller.
> 

The thread gets funny. I guess this is not so much about NIH, but
rather about the fact that no one wants to work with him or that no
one wants to be one of his users and gets his feature requests all
RESOLVED WONTFIX.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJS7AKzAAoJEFpvPKfnPDWzKAkIAKEIAx/4l690pHYvxvKkaypJ
XWPs+LRokNboyzXyeZLEgWhEIJ5LzflBMgcnn0KRRn3p81JYaERQ+Cnx3yBtL148
7ovlZug12dxLO+nWVajrOWP3YWcHV12Kla6q7qTWrTO4RxZbfNEncyyMc4uMzCyk
mQ13nBP7gooNdRx5pN61POKI23OPyK4Z/AnlJdMq6aForVuY788vOUZq8q/n96MU
tdkx7npzJVJ/OGgwIF5AqIn1G1NmzmkQ3R8hKnPN/0W+l6jlChoocq+9tELTnJ/r
UtXmVwdlsHCnG4rY+RxeVBIfLMi0f9xce1/ckENbLIiyoj5xMNkZ3/+6dyI/VhU=
=FIJW
-----END PGP SIGNATURE-----

Re: [gentoo-user] Re: Portage performance dropped considerably

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 1977 bytes --]

On Fri, 31 Jan 2014 19:13:21 +0000 Mick wrote:
> On Friday 31 Jan 2014 19:03:05 Andrew Savchenko wrote:
> > On Sun, 26 Jan 2014 20:30:19 +0200 Alan McKinnon wrote:
[...]
> > > I'm willing to give up 4 minutes while emerge runs so I don't have to
> > > spend many more minutes right afterwards doing manually the very shit
> > > that software is very good at. Whether portage is a complete pile of
> > > dogshit software or not is beside the point. Even if it is, my 4 minutes
> > > still buys me lots <shrug>
> > 
> > 4 minutes are expendable but... on Atom N270 (my laptop) emerge
> > -DNuav world takes 40 (yes, forty) minutes to build dependency tree
> > with sqlite cache enabled and 60 minutes without sqlite. System was
> > pretty old (not updated aside from GLSA updates for a year). And this
> > 40 minutes repeated many times since USE flag clashes and dependency
> > resolution failures. So I spent may day, damn whole day(!) for the
> > sake to just start compiling (distcc is my friend here).
> 
> Out of interest what fs are you running portage on?  I changed an old box from 
> reiserfs to ext4 and couldn't believe the speed up I got.

I use ext4. In my case the problem is in CPU usage and Atom N270 is
pretty weak these days. On this box HDD is fast and NCQ capable,
memory is enough (2GB for 32bit setup). During dependencies
calculation all 100% of time is spent in the python process. CPU is
slightly overclocked (as much as SHE allows). There is nothing more
that I can do here as an admin of this host. (Gentoo setup itself is
complicated with >2200 packages.)

IMO the only way to improve this issue (without throwing good working
hardware in the window) is to rewrite dependency resolution code in
some highly optimized pure C library (probably with some asm code)
and to use this library via some python binding from portage. But I
suppose algorithm itself should be reviewed first.

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-user] Re: Portage performance dropped considerably

[Alan McKinnon]

On 31/01/2014 23:18, Andrew Savchenko wrote:
> IMO the only way to improve this issue (without throwing good working
> hardware in the window) is to rewrite dependency resolution code in
> some highly optimized pure C library (probably with some asm code)

I very much doubt that will help.

There's nothing unusual about portage's dep resolution - it's just a
tree search. This is a well-understood problem and we've known how to
DoItRite for 30 years or more. Python is already optimized to do this
just fine, and it's bytecode, NOT classic interpreted. Rewriting it in C
probably won't do much as C isn't a magic bullet.

> and to use this library via some python binding from portage. But I
> suppose algorithm itself should be reviewed first.

^this is where the speedups will lie

4 minutes on this here i7 monster and 40 on your Atom is ridiculous
considering the problem that is being solved. Portage is probably
searching and re-searching dead paths in the tree or something equally
silly. The algorithm should be analysed and dead paths optimized away.
Not a language problem.

-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] Re: Portage performance dropped considerably

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 854 bytes --]

On Sat, 01 Feb 2014 00:12:58 +0200 Alan McKinnon wrote:
> > and to use this library via some python binding from portage. But I
> > suppose algorithm itself should be reviewed first.
> 
> ^this is where the speedups will lie
> 
> 4 minutes on this here i7 monster and 40 on your Atom is ridiculous
> considering the problem that is being solved. Portage is probably
> searching and re-searching dead paths in the tree or something equally
> silly. The algorithm should be analysed and dead paths optimized away.
> Not a language problem.
 
Another challenge is to make dependency resolution parallel — result
should be awesome on modern multi-core CPUs. And I'm sure this is a
doable task (on a first glance analyse subtrees first then join), but
this issue requires further and deeper investigation.

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

[gentoo-user] Re: Portage performance dropped considerably

[Martin Vaeth]

Andrew Savchenko <bircoph@gmail.com> wrote:
>
> Another challenge is to make dependency resolution parallel

It's a challange but won't solve the problem: On fast processors
portage's speed is not so much a big issue. Moreover, the factor
you can obtain this way is in the (unrealistic) best case at most
the number of cores. Realistically, on a dual core perhaps the
factor 1.3 (and at the price of blocking parallel running applications
correspondingly more).

Re: [gentoo-user] Re: Portage performance dropped considerably

[Greg Turner]

On Mon, Feb 3, 2014 at 2:55 AM, Martin Vaeth <martin@mvath.de> wrote:
> On fast processors
> portage's speed is not so much a big issue.

What kind of processor have you got, and where can I get one?

-gmt

[gentoo-user] Re: Portage performance dropped considerably

[Martin Vaeth]

Greg Turner <gmt@malth.us> wrote:
> On Mon, Feb 3, 2014 at 2:55 AM, Martin Vaeth <martin@mvath.de> wrote:
>> On fast processors
>> portage's speed is not so much a big issue.
>
> What kind of processor have you got, and where can I get one?

I run gentoo on i3 (double core), c2 (double core), athlon, and pentium3.
Only on athlon and pentium3 the speed is really annoying. So increasing
the speed on the two faster processors by a factor 1.3 by parallelization,
probably even at the cost of slightly decreasing the speed on the
slower processors (perhaps even dramatically if the algorithm then needs
more swap usage which already occurs now with 512MB RAM)
will altogether be perhaps even counterproductive for my use case.

Re: [gentoo-user] Re: Portage performance dropped considerably

[Pandu Poluan]

[-- Attachment #1: Type: text/plain, Size: 938 bytes --]

On Jan 28, 2014 5:57 AM, "Neil Bothwick" <neil@digimed.co.uk> wrote:
>
> On Mon, 27 Jan 2014 22:54:28 +0100, hasufell wrote:
>
> > >> If it's about performance (in the sense of speed), then paludis
> > >> is worse, because dependency calculation is more complex/complete
> > >> there.
> > >
> > > That makes no sense at all. Paludis is written in a different
> > > language using different algorithms. It's not about the amount of
> > > work it does so much as how efficiently it does it.
>
> > That's exactly what I was saying. I was talking about speed, not
> > efficiency.
>
> But the efficiency of the algorithm, and the language, affects the speed.
> You can't presume "it does more, therefore it takes longer" if the two
> programs do things in very different ways.
>

I was thinking: is it feasible, to "precalculate" the dependency tree? Or,
at least "preprocess" all the sane (and insane) dependencies to help
portage?

Rgds,
--

[-- Attachment #2: Type: text/html, Size: 1286 bytes --]

Re: [gentoo-user] Re: Portage performance dropped considerably

[Alan McKinnon]

On 03/02/2014 16:04, Pandu Poluan wrote:
> 
> On Jan 28, 2014 5:57 AM, "Neil Bothwick" <neil@digimed.co.uk
> <mailto:neil@digimed.co.uk>> wrote:
>>
>> On Mon, 27 Jan 2014 22:54:28 +0100, hasufell wrote:
>>
>> > >> If it's about performance (in the sense of speed), then paludis
>> > >> is worse, because dependency calculation is more complex/complete
>> > >> there.
>> > >
>> > > That makes no sense at all. Paludis is written in a different
>> > > language using different algorithms. It's not about the amount of
>> > > work it does so much as how efficiently it does it.
>>
>> > That's exactly what I was saying. I was talking about speed, not
>> > efficiency.
>>
>> But the efficiency of the algorithm, and the language, affects the speed.
>> You can't presume "it does more, therefore it takes longer" if the two
>> programs do things in very different ways.
>>
> 
> I was thinking: is it feasible, to "precalculate" the dependency tree?
> Or, at least "preprocess" all the sane (and insane) dependencies to help
> portage?


I thought that's what the portage cache does, as far as it can.

True, the cache reflects the state of the tree and not the parts of the
tree a given machine is using, so how big a diff does that give? And
don't forget overlays - they can slow things down immensely as more
often than not there's no cache for them unless the user knows to do it
manually.


-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] Re: Portage performance dropped considerably

[Pandu Poluan]

[-- Attachment #1: Type: text/plain, Size: 1758 bytes --]

On Feb 3, 2014 9:17 PM, "Alan McKinnon" <alan.mckinnon@gmail.com> wrote:
>
> On 03/02/2014 16:04, Pandu Poluan wrote:
> >
> > On Jan 28, 2014 5:57 AM, "Neil Bothwick" <neil@digimed.co.uk
> > <mailto:neil@digimed.co.uk>> wrote:
> >>
> >> On Mon, 27 Jan 2014 22:54:28 +0100, hasufell wrote:
> >>
> >> > >> If it's about performance (in the sense of speed), then paludis
> >> > >> is worse, because dependency calculation is more complex/complete
> >> > >> there.
> >> > >
> >> > > That makes no sense at all. Paludis is written in a different
> >> > > language using different algorithms. It's not about the amount of
> >> > > work it does so much as how efficiently it does it.
> >>
> >> > That's exactly what I was saying. I was talking about speed, not
> >> > efficiency.
> >>
> >> But the efficiency of the algorithm, and the language, affects the
speed.
> >> You can't presume "it does more, therefore it takes longer" if the two
> >> programs do things in very different ways.
> >>
> >
> > I was thinking: is it feasible, to "precalculate" the dependency tree?
> > Or, at least "preprocess" all the sane (and insane) dependencies to help
> > portage?
>
>
> I thought that's what the portage cache does, as far as it can.
>
> True, the cache reflects the state of the tree and not the parts of the
> tree a given machine is using, so how big a diff does that give? And
> don't forget overlays - they can slow things down immensely as more
> often than not there's no cache for them unless the user knows to do it
> manually.
>

Well, AFAIK, portage needs to kind of simulate everything going on in an
ebuild to get the list of dependencies/blockers... If this can be
'pre-simulated' resulting in a simpler to parse 'database' of
dependencies...

Rgds,
--

[-- Attachment #2: Type: text/html, Size: 2504 bytes --]

[gentoo-user] Re: Portage performance dropped considerably

[Martin Vaeth]

Pandu Poluan <pandu@poluan.info> wrote:
>> > I was thinking: is it feasible, to "precalculate" the dependency tree?
>>
>> I thought that's what the portage cache does, as far as it can.
>
> Well, AFAIK, portage needs to kind of simulate everything going on in an
> ebuild to get the list of dependencies/blockers... If this can be
> 'pre-simulated' resulting in a simpler to parse 'database' of
> dependencies...

This *is* the portage cache: Execute e.g.
cat /usr/portage/metadata/md5-cache/sys-apps/portage-9999
You see that DEPEND, RDEPEND, PDEPEND are readily available.
If metadata/md5-cache should not be up-to-date (e.g. if you
use an overlay without that data and without calling egencache),
portage generates a similar cache in /var/cache/edb/dep
(checksums and/or filestamps are used to verify that the
caches are up-to-date - this also takes somewhat time but
not very much and cannot be avoided to guarantee correct behaviour).

Re: [gentoo-user] Portage performance dropped considerably

[Tom Wijsman]

On Sun, 26 Jan 2014 16:35:43 +0200
Nikos Chantziaras <realnc@gmail.com> wrote:

> Anyone else noticed this yet? Some portage update seems to have made 
> "emerge -uDN @world" perform about 10 times slower than before. It
> used to take seconds, now it takes about 4 minutes only to tell me
> that there's nothing to update. And it does that every time, even
> directly in succession and with the caches warm.
> 
> Is it just me?

(TL;DR: It is a known problem and it will be improved.)

More people experience this, I've seen multiple reports in all the
communication channels I can think of so far; besides having been
bothered with it myself in the past, but now I'm fine with ~30 seconds.

=== What is going on and what is a cheap solution? ===

Often this is due to backtracking going mad; Portage keeps trying
again, and again, and ... until it has done it for a 10 extra times. And
if a single time perhaps takes like 30 seconds, doing it 10 times extra
gets you quickly to get to 6 minutes or so. Now imagine 30 times...

For this reason, I run --backtrack=0 myself; which finishes in half a
minute or so, but it needs some more experience to deal with the output
that comes after that as you'll need to fix up some things manually.
Not to forget --backtrack=0 has some side effects wrt package rebuilds;
I don't know the fine details, but was warned about this by others.

A solution in between would be to use --backtrack=0 and when that isn't
satisfied raise it a bit until satisfied; if you can get your
--backtrack=0 to work one or another time, you know you won't need
--backtrack=10 soon given that your Portage tree is pretty satisfied.
When you need to raise it, you will come around with --backtrack=5; and
in the case it doesn't you can raise it again, 3 minutes don't hurt.

So, it can be perceived as a trade-off between "I can quickly do this
myself" (given a good understanding of the output) and "Spent some time
doing it for me" (if you don't have the time to do it yourself).

=== Why is this becoming worse over time? ===

Simply put, it's a growth in complexity; we've moved to newer EAPIs
that bring newer features, we have packages improving and thus
providing more USE flags depending on more, we've got more overlays to
pull interesting packages from, as a consequence of the newer EAPIs
there are a lot of sub slot dependencies and the list goes on...

The more options Portage has to consider, the more time it will spend
on it; and if after going down a set of options Portage doesn't find a
satisfiable state, it has to backtrack with more and/or different
options to try again to find another satisfiable state. You can see
that the amount of options (= amount of overlays, packages, versions,
enabled USE flags, ...) takes a huge toll here (amongst other things).

To get an idea of what an @world emerge is doing by default, check:

https://gist.github.com/TomWij/65a61765ea03e6188a35

This is generated using the output additions from the attachment of:

http://article.gmane.org/gmane.linux.gentoo.user/272117

Those whom are interested can also find call tree graphs, although
these rather reveal ~2% improvements because huger improvements require
an actual understanding of how most code works together; they are here:

http://dev.gentoo.org/~tomwij/files/portage-2.2.7-python-2.7-backtrack-0.png
http://dev.gentoo.org/~tomwij/files/portage-2.2.7-python-2.7-backtrack-0-hot.png
http://dev.gentoo.org/~tomwij/files/portage-2.2.7-python-3.3-backtrack-0.png

(Generated with https://code.google.com/p/jrfonseca/wiki/Gprof2Dot)

From the output additions you can see that the slot operators (and thus
sub slots from the newer EAPIs) take up quite a few time. If you think
about what these slot operators are trying to solve, you'll remember
something along the lines of `revdep-rebuild` from a while ago. A lot
of time we've used to spend in that back in the days has moved towards
a cocktail of slot operators and preserved-rebuild.

So, part of the extra time we experience is just a move of time; but
that doesn't make it additional time (unless with lots of backtracking).

Tried to keep this layman explanation short; as I no longer intend to
make this more than a simple explanation, I used to share further
details in the past but I do no longer deem them as necessary today.

Why? Because Portage developer Sebastian Luther (few_) is working on
improving this; the results seem promising, so, let's await a Portage
release with that before we continue to look into this again.

=== What might be done to improve this? ===

Read this message (patch in second mail):

http://thread.gmane.org/gmane.linux.gentoo.portage.devel/4231

The results look promising; it goes from 9:28 to 1:59, whereas
--backtrack=0 is 1:38. In other words, it gets quite close to the time
of --backtrack=0 without bringing its disadvantages; as it still does
the conflict resolving that backtrack does, but somewhat different.

(There are some other smaller patches that also spare out time; if you
are interested, you can find them by searching for him on the mailing
list. However, I think those other patches will be in a release soon.)

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D