[gentoo-dev] Portage ebuild cruft

[gentoo-dev] Portage ebuild cruft

[Heinrich Wendel]

[-- Attachment #1: Type: text/plain, Size: 629 bytes --]

Hi,

Portage is slow? How to make it faster? By removing unused ebuilds!

I wrote a little script to check which ebuilds in portage aren't used 
anylonger, here the result:

Total packages checked: 9076
Total ebuilds checked: 18662
Total ebuilds to remove: 4643

Of course the script can't detect every ebuild situation, so take the numbers 
with care. But still it shows that 1/4 of all ebuilds could be removed. This 
would improve portage performance by at least 1/4, so developers go ahead. 
The script is attached, just run it as if it was repoman, it won't do 
anything, just show the orphaned packages.

mfg, heinrich :-)

[-- Attachment #2: cleanup.py --]
[-- Type: application/x-python, Size: 4728 bytes --]

Re: [gentoo-dev] Portage ebuild cruft

[Brian Harring]

On Thu, Apr 28, 2005 at 06:40:23PM +0200, Heinrich Wendel wrote:
> Hi,
> 
> Portage is slow? How to make it faster? By removing unused ebuilds!
Define "faster".  All this would do is cut down on a couple of stats 
per pkg; the # of ebuilds per pkg isn't a huge issue, the scanning of 
vdb and Config initialization is what is damned slow. :)

> I wrote a little script to check which ebuilds in portage aren't used 
> anylonger, here the result:
> 
> Total packages checked: 9076
> Total ebuilds checked: 18662
> Total ebuilds to remove: 4643

Dropping every ebuild that isn't the highest version for the mismash 
of arches isn't really valid.  Granted, people *could* stand to do 
cleansing of old versions in the tree, but the versions they choose to 
support/offer is completely up to the ebuild dev.

> Of course the script can't detect every ebuild situation, so take the numbers 
> with care. But still it shows that 1/4 of all ebuilds could be removed. This 
> would improve portage performance by at least 1/4
Again, define what aspect of portage performance.  The only thing less 
ebuilds cuts down on is the avg runtime of a cp_list call; cache is 
keyed, so lookup is *typically* constant.
~brian
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Portage ebuild cruft

[Tom Wesley]

[-- Attachment #1: Type: text/plain, Size: 1191 bytes --]

On Thu, 2005-04-28 at 18:40 +0200, Heinrich Wendel wrote:
> Hi,
> 
> Portage is slow? How to make it faster? By removing unused ebuilds!
> 
> I wrote a little script to check which ebuilds in portage aren't used 
> anylonger, here the result:
> 
> Total packages checked: 9076
> Total ebuilds checked: 18662
> Total ebuilds to remove: 4643
> 
> Of course the script can't detect every ebuild situation, so take the numbers 
> with care. But still it shows that 1/4 of all ebuilds could be removed. This 
> would improve portage performance by at least 1/4, so developers go ahead. 
> The script is attached, just run it as if it was repoman, it won't do 
> anything, just show the orphaned packages.
> 
> mfg, heinrich :-)

I feel that I should point out that "large style" corporations, and the
smaller PLC's whose Systems departments like to play at being larger
corporations don't like changing version numbers of packages, full stop.
Purging old versions for a few seconds speed increase in portage
operations would potentially cause some, if not many people to make
extensive use of the overlay, just to satisfy their PHB's.

-- 
Tom Wesley <tom@tomaw.org>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Portage ebuild cruft

[Athul Acharya]

> Purging old versions for a few seconds speed increase in portage [snip]

Few seconds? Try few miliseconds, if anything, at all, ever.  The
original email in this thread gave me the best laugh I've had in a
while, until I realized it came from a dev; then I was very sad.

Athul

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Portage ebuild cruft

[Jason Wever]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thu, 28 Apr 2005, Athul Acharya wrote:

>> Purging old versions for a few seconds speed increase in portage [snip]
>
> Few seconds? Try few miliseconds, if anything, at all, ever.  The
> original email in this thread gave me the best laugh I've had in a
> while, until I realized it came from a dev; then I was very sad.

Please don't assume everyone is running your latest and greatest PC 
hardware, or processors that measure in the GHz, regardless of 
architecture.  We have officially supported architectures where a few 
seconds may be a generous statement of the delay (low end SPARC64 systems 
for instance).  The initialization delay of portage is very much felt 
here, either via emerge or other tools like equery.

Cheers,
- -- 
Jason Wever
Gentoo/Sparc Co-Team Lead
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQFCcjardKvgdVioq28RApS3AJ9CsdeTu90RaRiZzK/4xPq8Y4B9KgCfVX0H
KkUrEoT37V7brPnZSW8oX0A=
=wbrO
-----END PGP SIGNATURE-----
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Portage ebuild cruft

[Jason Stubbs]

[-- Attachment #1: Type: text/plain, Size: 1154 bytes --]

On Friday 29 April 2005 22:29, Jason Wever wrote:
> On Thu, 28 Apr 2005, Athul Acharya wrote:
> >> Purging old versions for a few seconds speed increase in portage [snip]
> >
> > Few seconds? Try few miliseconds, if anything, at all, ever.  The
> > original email in this thread gave me the best laugh I've had in a
> > while, until I realized it came from a dev; then I was very sad.
>
> Please don't assume everyone is running your latest and greatest PC
> hardware, or processors that measure in the GHz, regardless of
> architecture.  We have officially supported architectures where a few
> seconds may be a generous statement of the delay (low end SPARC64 systems
> for instance).  The initialization delay of portage is very much felt
> here, either via emerge or other tools like equery.

The initialization time of portage is directly related to the number of 
packages installed. Cutting out excess ebuilds from the tree won't speed this 
up at all. Cutting out excess ebuilds won't have much effect on the general 
running of emerge at all, actually, except for updating the cache after 
syncing.

Regards,
Jason Stubbs

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Portage ebuild cruft

[Jason Wever]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Fri, 29 Apr 2005, Jason Stubbs wrote:

> The initialization time of portage is directly related to the number of
> packages installed. Cutting out excess ebuilds from the tree won't speed this
> up at all. Cutting out excess ebuilds won't have much effect on the general
> running of emerge at all, actually, except for updating the cache after
> syncing.

Does that include the about >=10 second delay when emerge is first run on 
a box after a reboot before you get any visible output on the screen, or 
is that an entirely different problem?

- -- 
Jason Wever
Gentoo/Sparc Co-Team Lead
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQFCckCCdKvgdVioq28RAlJ6AJ0bf2NHAFr6Otc7v0HvbyqOlb7DDgCfYrTR
6tFh38nSUwVw6XDXvYnWmxk=
=ODsL
-----END PGP SIGNATURE-----
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Portage ebuild cruft

[Elfyn McBratney]

[-- Attachment #1: Type: text/plain, Size: 1035 bytes --]

On Friday 29 Apr 2005 15:11, Jason Wever wrote:
> On Fri, 29 Apr 2005, Jason Stubbs wrote:
> > The initialization time of portage is directly related to the number of
> > packages installed. Cutting out excess ebuilds from the tree won't speed
> > this up at all. Cutting out excess ebuilds won't have much effect on the
> > general running of emerge at all, actually, except for updating the cache
> > after syncing.
>
> Does that include the about >=10 second delay when emerge is first run on
> a box after a reboot before you get any visible output on the screen, or
> is that an entirely different problem?

Heh, I get that after every invocation of emerge. :)

Best,
Elfyn

-- 
Elfyn McBratney                                     http://beu.merseine.nu/
beu/irc.freenode.net                            http://dev.gentoo.org/~beu/
+------------O.o--------------------- http://dev.gentoo.org/~beu/pubkey.asc

PGP Key ID: 0x69DF17AD
PGP Key Fingerprint:
  DBD3 B756 ED58 B1B4 47B9  B3BD 8D41 E597 69DF 17AD

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Portage ebuild cruft

[Jason Stubbs]

[-- Attachment #1: Type: text/plain, Size: 847 bytes --]

On Friday 29 April 2005 23:26, Elfyn McBratney wrote:
> On Friday 29 Apr 2005 15:11, Jason Wever wrote:
> > On Fri, 29 Apr 2005, Jason Stubbs wrote:
> > > The initialization time of portage is directly related to the number of
> > > packages installed. Cutting out excess ebuilds from the tree won't
> > > speed this up at all. Cutting out excess ebuilds won't have much effect
> > > on the general running of emerge at all, actually, except for updating
> > > the cache after syncing.
> >
> > Does that include the about >=10 second delay when emerge is first run on
> > a box after a reboot before you get any visible output on the screen, or
> > is that an entirely different problem?
>
> Heh, I get that after every invocation of emerge. :)

Yep. That's the scanning of all installed packages for any provided virtuals.

Regards,
Jason Stubbs

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Portage ebuild cruft

[Thomas de Grenier de Latour]

On Fri, 29 Apr 2005 23:38:58 +0900
Jason Stubbs <jstubbs@gentoo.org> wrote:

> Yep. That's the scanning of all installed packages for any
> provided virtuals.

Maybe that's a stupid idea, but I wonder whether removing empty
PROVIDE files from the vardb could save some time here.  I see
that in grabfile(), if the file can't be opened, then an empty
list is returned, so it would have the same behavior.  Does
reading an empty file takes more time than failing to open a
non-existing one?  (Yep, I don't know much about how filesystems
actually work...)

-- 
TGL.
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Portage ebuild cruft

[Jason Stubbs]

[-- Attachment #1: Type: text/plain, Size: 969 bytes --]

On Saturday 30 April 2005 00:59, Thomas de Grenier de Latour wrote:
> On Fri, 29 Apr 2005 23:38:58 +0900
>
> Jason Stubbs <jstubbs@gentoo.org> wrote:
> > Yep. That's the scanning of all installed packages for any
> > provided virtuals.
>
> Maybe that's a stupid idea, but I wonder whether removing empty
> PROVIDE files from the vardb could save some time here.  I see
> that in grabfile(), if the file can't be opened, then an empty
> list is returned, so it would have the same behavior.  Does
> reading an empty file takes more time than failing to open a
> non-existing one?  (Yep, I don't know much about how filesystems
> actually work...)

real    0m9.925s	# With
real    0m9.913s	# Without

That's timings for 20 imports of portage. Can't really confirm either way from 
those results. It might make a difference when the system cache is empty. My 
as yet unnumbered virtuals glep would negate this altogether. ;)

Regards,
Jason Stubbs

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Portage ebuild cruft

[Lina Pezzella]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

>  My
> as yet unnumbered virtuals glep would negate this altogether. ;)

Mmmmm - is there a draft of this somewhere?

- --Lina Pezzella
Ebuild/Porting Co-Lead
Gentoo for Mac OS X
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (Darwin)

iD8DBQFCcmL/NJ9STR9DbYERAnPIAKC4+P7cJAQjZoPp6pKNyH7iWgTSDQCfWwbc
g/xJtOy5uicmCw24GTmp4VY=
=idDB
-----END PGP SIGNATURE-----

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Portage ebuild cruft

[Jason Stubbs]

[-- Attachment #1: Type: text/plain, Size: 257 bytes --]

On Saturday 30 April 2005 01:38, Lina Pezzella wrote:
> >  My
> > as yet unnumbered virtuals glep would negate this altogether. ;)
>
> Mmmmm - is there a draft of this somewhere?

http://dev.gentoo.org/~jstubbs/docs/virtuals-glep.txt

Regards,
Jason Stubbs

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Portage ebuild cruft

[Thomas de Grenier de Latour]

On Sat, 30 Apr 2005 01:51:33 +0900
Jason Stubbs <jstubbs@gentoo.org> wrote:

> 
> http://dev.gentoo.org/~jstubbs/docs/virtuals-glep.txt
> 

Oh, i hadn't seen this version of the proposal yet, it's quite
good :)

-- 
TGL.
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Portage ebuild cruft

[Jan Kundrát]

[-- Attachment #1: Type: text/plain, Size: 486 bytes --]

Jason Stubbs wrote:
> http://dev.gentoo.org/~jstubbs/docs/virtuals-glep.txt

Please forgive me if this ml is a bad place for suggestions.

Ad "overrides" - what about "per-virtual" preferencies:
What happens if packages A and B provide functionality X, packages B & C
provide Y *and* you prefer B for X and C for Y? (This situation can be
actually solved by order "C > B > A", but that wouldn't be possible if C
provided X, too.)

TIA,
-jkt

-- 
cd /local/pub && more beer > /dev/mouth

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Portage ebuild cruft

[Jason Stubbs]

[-- Attachment #1: Type: text/plain, Size: 812 bytes --]

On Saturday 30 April 2005 05:25, Jan Kundrát wrote:
> Jason Stubbs wrote:
> > http://dev.gentoo.org/~jstubbs/docs/virtuals-glep.txt
>
> Please forgive me if this ml is a bad place for suggestions.
>
> Ad "overrides" - what about "per-virtual" preferencies:
> What happens if packages A and B provide functionality X, packages B & C
> provide Y *and* you prefer B for X and C for Y? (This situation can be
> actually solved by order "C > B > A", but that wouldn't be possible if C
> provided X, too.)

Manually emerge one or the other. If you're wanting to keep such a case in 
sync across a network, add them to system. These "preferences" are only for 
the purposes of dependency calculation. The ebuilds themselves are 
essentially free to build against whatever they want.

Regards,
Jason

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Portage ebuild cruft

[Sune Kloppenborg Jeppesen]

[-- Attachment #1: Type: text/plain, Size: 308 bytes --]

On Friday 29 April 2005 16:38, Jason Stubbs wrote:
> > Heh, I get that after every invocation of emerge. :)
>
> Yep. That's the scanning of all installed packages for any provided
> virtuals.
Why not let Portage print that before scanning?

-- 
Sune Kloppenborg Jeppesen
Gentoo Linux Security Team

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Portage ebuild cruft

[Marius Mauch]

[-- Attachment #1: Type: text/plain, Size: 675 bytes --]

On Sat, 30 Apr 2005 12:31:17 +0200
Sune Kloppenborg Jeppesen <jaervosz@gentoo.org> wrote:

> On Friday 29 April 2005 16:38, Jason Stubbs wrote:
> > > Heh, I get that after every invocation of emerge. :)
> >
> > Yep. That's the scanning of all installed packages for any provided
> > virtuals.
> Why not let Portage print that before scanning?

Print what?
The scanning is done on the general config parsing, and you can't really
do anything before that.

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Portage ebuild cruft

[Sune Kloppenborg Jeppesen]

[-- Attachment #1: Type: text/plain, Size: 792 bytes --]

On Saturday 30 April 2005 13:12, Marius Mauch wrote:
> On Sat, 30 Apr 2005 12:31:17 +0200
>
> Sune Kloppenborg Jeppesen <jaervosz@gentoo.org> wrote:
> > On Friday 29 April 2005 16:38, Jason Stubbs wrote:
> > > > Heh, I get that after every invocation of emerge. :)
> > >
> > > Yep. That's the scanning of all installed packages for any provided
> > > virtuals.
> >
> > Why not let Portage print that before scanning?
>
> Print what?
> The scanning is done on the general config parsing, and you can't really
> do anything before that.
Scanning configuration/installed packages or something. It just seems a bit 
more userfriendly to print something if it takes a long while to do anything.

Just my to 0.02€
 
-- 
Sune Kloppenborg Jeppesen
Gentoo Linux Security Team

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Portage ebuild cruft

[Elfyn McBratney]

[-- Attachment #1: Type: text/plain, Size: 1484 bytes --]

On Saturday 30 Apr 2005 13:07, Sune Kloppenborg Jeppesen wrote:
> On Saturday 30 April 2005 13:12, Marius Mauch wrote:
> > On Sat, 30 Apr 2005 12:31:17 +0200
> >
> > Sune Kloppenborg Jeppesen <jaervosz@gentoo.org> wrote:
> > > On Friday 29 April 2005 16:38, Jason Stubbs wrote:
> > > > > Heh, I get that after every invocation of emerge. :)
> > > >
> > > > Yep. That's the scanning of all installed packages for any provided
> > > > virtuals.
> > >
> > > Why not let Portage print that before scanning?
> >
> > Print what?
> > The scanning is done on the general config parsing, and you can't really
> > do anything before that.
>
> Scanning configuration/installed packages or something. It just seems a bit
> more userfriendly to print something if it takes a long while to do
> anything.
>
> Just my to 0.02€

Sounds good to me, if possible.  Could we maybe be more noisy here when people 
run with --verbose ?  You know, so your not left staring at nothingness while 
portage does it's imports, scans, whatever ..

  $emerge -pv whatever

  Loading..  checking X ..
  Scanning deps ..
  ..

Best,
Elfyn

-- 
Elfyn McBratney                                     http://beu.merseine.nu/
beu/irc.freenode.net                            http://dev.gentoo.org/~beu/
+------------O.o--------------------- http://dev.gentoo.org/~beu/pubkey.asc

PGP Key ID: 0x69DF17AD
PGP Key Fingerprint:
  DBD3 B756 ED58 B1B4 47B9  B3BD 8D41 E597 69DF 17AD

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Portage ebuild cruft

[Georgi Georgiev]

[-- Attachment #1: Type: text/plain, Size: 1535 bytes --]

maillog: 30/04/2005-13:27:44(+0100): Elfyn McBratney types
> On Saturday 30 Apr 2005 13:07, Sune Kloppenborg Jeppesen wrote:
> > On Saturday 30 April 2005 13:12, Marius Mauch wrote:
> > > On Sat, 30 Apr 2005 12:31:17 +0200
> > >
> > > Sune Kloppenborg Jeppesen <jaervosz@gentoo.org> wrote:
> > > > On Friday 29 April 2005 16:38, Jason Stubbs wrote:
> > > > > > Heh, I get that after every invocation of emerge. :)
> > > > >
> > > > > Yep. That's the scanning of all installed packages for any provided
> > > > > virtuals.
> > > >
> > > > Why not let Portage print that before scanning?
> > >
> > > Print what?
> > > The scanning is done on the general config parsing, and you can't really
> > > do anything before that.
> >
> > Scanning configuration/installed packages or something. It just seems a bit
> > more userfriendly to print something if it takes a long while to do
> > anything.
> >
> > Just my to 0.02€
> 
> Sounds good to me, if possible.  Could we maybe be more noisy here when people 
> run with --verbose ?  You know, so your not left staring at nothingness while 
> portage does it's imports, scans, whatever ..

Also consider that Control-C doesn't really do anything at that stage.

-- 
/    Georgi Georgiev   /  "Why can't we ever attempt to solve a        /
\     chutz@gg3.net    \  problem in this country without having a     \
/   +81(90)2877-8845   /  'War' on it?" -- Rich Thomson,               /
\  ------------------- \  talk.politics.misc                           \

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]