[gentoo-dev] emerge suggestions

[gentoo-dev] emerge suggestions

[Jerry A!]

Can we change the behavior of "emerge rsync" to not preserve perms?
Right now when I rsync, I get files and directories that are 777, 2777,
etc.  I don't think that people expect to update their portage tree and
have the entire thing be a security nightmare.  8(

I know that the more correct solution would be to ensure that the perms
are correct on the rsync server.  However, since those seem to be victim
to chmod-rot, I think we should make sure that at least the client-side
behaves as expected...

Also, any chance of having it ignore something like /usr/portage/local?
That way people using rsync could make out of tree changes and not have
them blown away each time they update their tree.

        --Jerry

name:  Jerry Alexandratos         ||  Open-Source software isn't a
phone: 703.599.6023               ||  matter of life or death...
email: jerry@thehutt.org          ||  ...It's much more important
                                  ||  than that!

Re: [gentoo-dev] emerge suggestions

[Daniel Robbins]

On Mon, Jul 09, 2001 at 06:02:09PM -0400, Jerry A! wrote:
> Can we change the behavior of "emerge rsync" to not preserve perms?
> Right now when I rsync, I get files and directories that are 777, 2777,
> etc.  I don't think that people expect to update their portage tree and
> have the entire thing be a security nightmare.  8(
> 
> I know that the more correct solution would be to ensure that the perms
> are correct on the rsync server.  However, since those seem to be victim
> to chmod-rot, I think we should make sure that at least the client-side
> behaves as expected...
> 
> Also, any chance of having it ignore something like /usr/portage/local?
> That way people using rsync could make out of tree changes and not have
> them blown away each time they update their tree.

Try remerging portage and still see if you get the 777 problem.  Also, type
"umask" and see if root has a umask set.  If not, add one to /etc/profiles
(fixed on CVS).

As for LOCAL_PORTDIR support, it'll be added soon.

-- 
Daniel Robbins					<drobbins@gentoo.org>
President/CEO					http://www.gentoo.org 
Gentoo Technologies, Inc.			

[gentoo-dev] emerge suggestions

[Philippe Trottier]

Hello,

          --time option would be nice as it could give a reference to 
how much time does it take for each packages
        
       put all the reports at the end of the long ebuilds, and put all 
the times there...

Phil

--
gentoo-dev@gentoo.org mailing list

[gentoo-dev] Re: emerge suggestions

[Sebastian Bergmann]

Philippe Trottier wrote:
> --time option would be nice as it could give a reference to how
> much time does it take for each packages

 On a related note I think it would be nice to store the time needed for
 each ebuild to merge and use this information in the output of "emerge
 -vp <package>":

   wopr-mobile ~ # emerge -vp openoffice-ximian

   These are the packages that I would merge, in order:

   Calculating dependencies ...done!
   [ebuild     U ] app-office/openoffice-ximian-1.3.3 [1.3.2]
                   -debug +gnome -kde -ooo-kde  993 kB  0d:06h:30m

   Total size of downloads: 993 kB
   Total time estimated: 0d:06h:30m

 Just a thought,
Sebastian

-- 
Sebastian Bergmann                      http://www.sebastian-bergmann.de/
GnuPG Key: 0xB85B5D69 / 27A7 2B14 09E4 98CD 6277 0E5B 6867 C514 B85B 5D69


--
gentoo-dev@gentoo.org mailing list

[gentoo-dev] Re: emerge suggestions

[Athul Acharya]

>  On a related note I think it would be nice to store the time needed for
>  each ebuild to merge and use this information in the output of "emerge
>  -vp <package>":
> 

Try genlop.

root@mbox ~ # emerge -p abiword | genlop --pretend
These are the pretended packages: (this may take a while; wait...)

 * app-office/abiword

Estimated update time: 15 minutes.

Does a lot of other neat stuff too.

Cheers,
Athul

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: emerge suggestions

[Chris White]

[-- Attachment #1: Type: text/plain, Size: 1335 bytes --]

Sebastian Bergmann wrote:

>Philippe Trottier wrote:
>  
>
>>--time option would be nice as it could give a reference to how
>>much time does it take for each packages
>>    
>>
>
> On a related note I think it would be nice to store the time needed for
> each ebuild to merge and use this information in the output of "emerge
> -vp <package>":
>
>   wopr-mobile ~ # emerge -vp openoffice-ximian
>
>   These are the packages that I would merge, in order:
>
>   Calculating dependencies ...done!
>   [ebuild     U ] app-office/openoffice-ximian-1.3.3 [1.3.2]
>                   -debug +gnome -kde -ooo-kde  993 kB  0d:06h:30m
>
>   Total size of downloads: 993 kB
>   Total time estimated: 0d:06h:30m
>
> Just a thought,
>Sebastian
>
>  
>
How are you going to effectively measure the times? CFLAGS can slow your 
emerge, or speed it up quite significantly if you don't/do set them 
correctly. I think this is probably why portage devs never took this 
into account (or maybe they did and I'm just not aware of it). As for 
genlop, that is best used as an "after the fact" log parser, so that you 
can give other users estimates of how long compilation takes based on 
what their compile time was. My 2 cents(r).

-- 
Chris White <chriswhite@gentoo.org>
------------------------
Sound | Video | Security
ChrisWhite @ irc.freenode.net


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 256 bytes --]

[gentoo-dev] Re: emerge suggestions

[Chris White]

[-- Attachment #1: Type: text/plain, Size: 1059 bytes --]

I think we're failing to see the point. As solar had already stated, 
there is no accuracy in this program. The core fundamentals of said 
program deals with information processing, or taking raw data (bash 
units?) and processing them to format information usable to someome. 
While some programs are fairly usable in estimates, it still lacks one 
of the core fundamentals of information processing, which is acuracy. 
Without this, the program would simply spew random data with no regards 
to any sort of useful correlation. In THEORY, it's a good idea, but the 
implementation would not go through due to the amount of variable 
fluxations in the data presented. Genlops usefullness in data processing 
lies in the non-variable data that is provided to it (log files 
containing times of ALREADY merged packages, something that cannot be 
changed since the action has already occured). I hope this clears things 
up. My 4 cents(r).

-- 
Chris White <chriswhite@gentoo.org>
------------------------
Sound | Video | Security
ChrisWhite @ irc.freenode.net


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 256 bytes --]

Re: [gentoo-dev] Re: emerge suggestions

[Heiko Wundram]

Am Mittwoch, 8. September 2004 18:43 schrieb Chris White:
> How are you going to effectively measure the times?

IIRC, there once was a proposal to do this using bash-units. Each product in 
the tree gets assigned a bash unit, which is a floating point number >0 which 
measures how long compilation takes relative to compiling some certain 
version of bash.

Now, all that needs to be done is to measure package compilation and merging 
time, divide by the number of bash units this package has, and you get an 
estimate on the time for a bash unit on this computer. The more packages you 
merge, the finer this number will become by simply averaging it out. Of 
course, this does not take into account changing the LDFLAGS (which should 
make up for the biggest part of different merge times), or CFLAGS (which 
might also change timing by varying optimization levels and swap 
requirement). But, anyway, these numbers don't change anything about the 
underlying unit, which should be to a large extent platform and machine 
independent.

I don't know when this proposal came up, I read about it on some forum, some 
time ago.

Heiko.

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: emerge suggestions

[Ned Ludd]

[-- Attachment #1: Type: text/plain, Size: 1763 bytes --]

On Wed, 2004-09-08 at 19:46, Heiko Wundram wrote:
> Am Mittwoch, 8. September 2004 18:43 schrieb Chris White:
> > How are you going to effectively measure the times?
> 
> IIRC, there once was a proposal to do this using bash-units. Each product in 
> the tree gets assigned a bash unit, which is a floating point number >0 which 
> measures how long compilation takes relative to compiling some certain 
> version of bash.
> 
> Now, all that needs to be done is to measure package compilation and merging 
> time, divide by the number of bash units this package has, and you get an 
> estimate on the time for a bash unit on this computer. The more packages you 
> merge, the finer this number will become by simply averaging it out. Of 
> course, this does not take into account changing the LDFLAGS (which should 
> make up for the biggest part of different merge times), or CFLAGS (which 
> might also change timing by varying optimization levels and swap 
> requirement). But, anyway, these numbers don't change anything about the 
> underlying unit, which should be to a large extent platform and machine 
> independent.

To complex and still wont account for various USE= & FEATURES= flags,
gcc versions and loads on said server that's building XYZ. It's
therefore sorta a waste of all of our time and bandwidth to continue
this thread when we already know we are going to all come to the same
conclusion in the end that said functionality will never be accurate.

> 
> I don't know when this proposal came up, I read about it on some forum, some 
> time ago.
> 
> Heiko.
> 
> --
> gentoo-dev@gentoo.org mailing list
-- 
Ned Ludd <solar@gentoo.org>
Gentoo (hardened,security,infrastructure,embedded,toolchain) Developer

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Re: emerge suggestions

[Thomas de Grenier de Latour]

On Thu, 9 Sep 2004 01:46:45 +0200
Heiko Wundram <heikowu@ceosg.de> wrote:

> I don't know when this proposal came up, I read about it on some
> forum, some time ago.

That's also an idea that I've seen several times (here and in
forums). I've found this in archives:
http://article.gmane.org/gmane.linux.gentoo.devel/10528
There are in this thread some valid points (USE flags mainly) that
show how difficult and time consuming this idea would be to
actually implement.  

-- 
TGL.

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: emerge suggestions

[Heiko Wundram]

Am Donnerstag, 9. September 2004 02:03 schrieb Ned Ludd:
> To complex and still wont account for various USE= & FEATURES= flags,
> gcc versions and loads on said server that's building XYZ. It's
> therefore sorta a waste of all of our time and bandwidth to continue
> this thread when we already know we are going to all come to the same
> conclusion in the end that said functionality will never be accurate.

The part about USE flags certainly is true, but still, I guess it wouldn't be 
all that hard to factor out the amount of bash units each use flag gives to 
the total amount of bash units for the package (simple, compile package 
without X, compile with X, subtract first from second, you have the amount 
that X adds to the package). And, calculating the bash units a program takes 
to compile only needs to be done once, so I don't think this is all the big 
hassle that you make out of it.

What, IMO, makes results rather unpredictable is the problem of using a lowest 
common denominator machine to create the measurements for bash units, as the 
main thing that has to be taken into account is not the speed of the 
processor, but rather the memory usage of the compile, as compilation starts 
to drop sharply in speed when the compiler has to swap (I've experienced this 
more than once with PyQt, which takes longer to compile on an Athlon XP 2400+ 
with 256MB of RAM than on an Athlon 1300 with 1024MB of RAM, with otherwise 
identical systems from the software side of view).

Anyway, I just wanted to tell the op that there were proposals to calculate 
the time, I didn't want to start a discussion about the methods... And I 
basically don't care if portage tells me how long a package takes to merge.

Heiko.

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: emerge suggestions

[Daniel Goller]

rather than --pretend, why not use simply 'genlop -t gcc' and see how 
much time gcc TOOK to compile
s/gcc/any_package/

isn't that and the pipe with --pretend more than enough info already?

Thomas de Grenier de Latour wrote:

>On Thu, 9 Sep 2004 01:46:45 +0200
>Heiko Wundram <heikowu@ceosg.de> wrote:
>
>  
>
>>I don't know when this proposal came up, I read about it on some
>>forum, some time ago.
>>    
>>
>
>That's also an idea that I've seen several times (here and in
>forums). I've found this in archives:
>http://article.gmane.org/gmane.linux.gentoo.devel/10528
>There are in this thread some valid points (USE flags mainly) that
>show how difficult and time consuming this idea would be to
>actually implement.  
>
>  
>

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: emerge suggestions

[Joseph Booker]

On Wed, September 8, 2004 10:47 am, Sebastian Bergmann said:
> Philippe Trottier wrote:
>> --time option would be nice as it could give a reference to how
>> much time does it take for each packages
>
>  On a related note I think it would be nice to store the time needed for
>  each ebuild to merge and use this information in the output of "emerge
>  -vp <package>":
>
>    wopr-mobile ~ # emerge -vp openoffice-ximian
>
>    These are the packages that I would merge, in order:
>
>    Calculating dependencies ...done!
>    [ebuild     U ] app-office/openoffice-ximian-1.3.3 [1.3.2]
>                    -debug +gnome -kde -ooo-kde  993 kB  0d:06h:30m
>
>    Total size of downloads: 993 kB
>    Total time estimated: 0d:06h:30m
>
>  Just a thought,
> Sebastian
>
> --
> Sebastian Bergmann                      http://www.sebastian-bergmann.de/
> GnuPG Key: 0xB85B5D69 / 27A7 2B14 09E4 98CD 6277 0E5B 6867 C514 B85B 5D69

gentoo-stats.org really really needs to advertise more. Its wildly
inaccurate, but makes good for statistical information for the devs if it
was used more. It gets around the USE/CFLAG problem by letting you adjust
the 'Gentoo Units'. also, whats the differance between a --time flag for
emerge and just running 'time emerge ....'


-- 
Joseph Booker
joe @ irc.neoturbine.net
jj110888 @ irc.freenode.net


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: emerge suggestions

[Alin Nastac]

[-- Attachment #1: Type: text/plain, Size: 192 bytes --]

As you stated, genlop compute the average time spent in package ebuilds. 
I think it serve its purpose, which is to inform user if it has to wait 
days,hours or minutes to get de update done.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 256 bytes --]

Re: [gentoo-dev] Re: emerge suggestions

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 1862 bytes --]

On Wed, 2004-09-08 at 19:46, Heiko Wundram wrote:
> Am Mittwoch, 8. September 2004 18:43 schrieb Chris White:
> > How are you going to effectively measure the times?
> 
> IIRC, there once was a proposal to do this using bash-units. Each product in 
> the tree gets assigned a bash unit, which is a floating point number >0 which 
> measures how long compilation takes relative to compiling some certain 
> version of bash.

Let's assume we'll just use bash-2.05 (or 3.0, doesn't matter).

> Now, all that needs to be done is to measure package compilation and merging 
> time, divide by the number of bash units this package has, and you get an 
> estimate on the time for a bash unit on this computer. The more packages you 
> merge, the finer this number will become by simply averaging it out. Of 
> course, this does not take into account changing the LDFLAGS (which should 
> make up for the biggest part of different merge times), or CFLAGS (which 
> might also change timing by varying optimization levels and swap 
> requirement). But, anyway, these numbers don't change anything about the 
> underlying unit, which should be to a large extent platform and machine 
> independent.

That's pretty much accurate.  Another thing that could be done is there
could be a way to "calibrate" the system... essentially, performing a
bash build using the current {C,CXX,LD}FLAGS to get an accurate time. 
This measurement could then be used by portage in giving time estimates.

> I don't know when this proposal came up, I read about it on some forum, some 
> time ago.

I know that I mentioned it a while back, but it had been said before
that, so I'm not taking credit (or blame ;p) for it.

-- 
Chris Gianelloni
Release Engineering - Operations/QA Manager
Games - Developer
Gentoo Linux

Is your power animal a penguin?

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Re: emerge suggestions

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 909 bytes --]

On Wed, 2004-09-08 at 20:03, Ned Ludd wrote:
> To complex and still wont account for various USE= & FEATURES= flags,
> gcc versions and loads on said server that's building XYZ. It's
> therefore sorta a waste of all of our time and bandwidth to continue
> this thread when we already know we are going to all come to the same
> conclusion in the end that said functionality will never be accurate.

I don't think the point was ever to be accurate, but to give a rough
idea.  If OpenOffice.org is 1047 bash units, and bash takes 1.0 minutes
on your machine, then you know *about* how long OpenOffice.org will take
to compile.  I don't think anyone is concerned with exact numbers so
much as ballpark figures to give them a reasonable estimate of needed
time.

-- 
Chris Gianelloni
Release Engineering - Operations/QA Manager
Games - Developer
Gentoo Linux

Is your power animal a penguin?

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Re: emerge suggestions

[Danny van Dyk]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

| I don't think the point was ever to be accurate, but to give a rough
| idea.  If OpenOffice.org is 1047 bash units, and bash takes 1.0 minutes
| on your machine, then you know *about* how long OpenOffice.org will take
| to compile.  I don't think anyone is concerned with exact numbers so
| much as ballpark figures to give them a reasonable estimate of needed
| time.
|
I don't share your optimism. I already see bug reports about wrong ETAs
... :-/

- --
Danny van Dyk
Gentoo/AMD64 Developer
kugelfang@gentoo.org
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBQIWsaVNL8NrtU6IRAvAaAJ0bvAgtGActc0e+OVgqUNJbvmb1RwCfQbmj
jTuggfceFM21bYOnlwcVaD0=
=C2Jy
-----END PGP SIGNATURE-----

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: emerge suggestions

[Marcus D. Hanwell]

Danny van Dyk wrote:

> | I don't think the point was ever to be accurate, but to give a rough
> | idea.  If OpenOffice.org is 1047 bash units, and bash takes 1.0 minutes
> | on your machine, then you know *about* how long OpenOffice.org will take
> | to compile.  I don't think anyone is concerned with exact numbers so
> | much as ballpark figures to give them a reasonable estimate of needed
> | time.
> |
> I don't share your optimism. I already see bug reports about wrong ETAs
> ... :-/

Surely these can be dismissed as WONTFIX, and move on. There will always 
be some users who expect everything to be perfect. Round up to the 
nearest minute, output Estimated/Approximate/Expected emerge time: 12 
minutes. This would be a great feature, and most users will not care if 
it takes 15 minutes when it said 12 - it's just a lot better to have a 
reasonable idea of emerge times.

If it was within 10 - 20% most of the time that is still useful to most, 
and I think it will do better than that most of the time. Even if we 
could make an exact estimate of emerge time issues such as other 
processes using processor time etc will always throw these estimates off.


--
gentoo-dev@gentoo.org mailing list