[gentoo-portage-dev] re: confcache

[gentoo-portage-dev] re: confcache

[Brian Harring]

Well... that's weird, attempted to send this directly to dan, having it
bail due to
'failed: relaying not allowed: danarmak@gentoo.org'
I assume I'm being an idiot and screwing up the address (or local
configuration)...

Either way, you might want to take a look at my implementation of
confcache in 
http://dev.gentoo.org/~ferringb/ebuild-daemon/51-pre20-cvs/

You can use SANDBOX_DEBUG_LOG and SANDBOX_DEBUG instead of modifying the
sandbox- works just the same.  Also, variables changing _will_ cause
configure to bail if they've been cached (cflags fex), so you might want
to either filter those entries (they start with ac_cv_env_), or rewrite
them (I went for rewrite, works although I worry about other cache
entries relying on the vars not changing).
Aside from that, there is also the md5'ing of /proc/cpuinfo- most users,
not an issue, as magnade pointed out, this is a killer for users that
have procs that adjust their frequency (laptops fex) for power saving-
this changes the md5 of /proc/cpuinfo pretty much continually,
invalidating the cache continually.
~brian


--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] re: confcache

[Dan Armak]

[-- Attachment #1: Type: text/plain, Size: 2353 bytes --]

On Friday 08 October 2004 22:08, Brian Harring wrote:
> Well... that's weird, attempted to send this directly to dan, having it
> bail due to
> 'failed: relaying not allowed: danarmak@gentoo.org'
> I assume I'm being an idiot and screwing up the address (or local
> configuration)...
All I can say is, I can receive at that address quite well...

>
> Either way, you might want to take a look at my implementation of
> confcache in
> http://dev.gentoo.org/~ferringb/ebuild-daemon/51-pre20-cvs/
I did, but it's tied to your ability to call back into python from ebuild.sh, 
so I couldn't use it with the main portage tree. Unless there's a way to do 
this in stock portage? I don't have any experience with the python side of 
portage, so please enlighten my ignorance...

>
> You can use SANDBOX_DEBUG_LOG and SANDBOX_DEBUG instead of modifying the
> sandbox- works just the same.  
I'll look at doing that, the less modifications the better.

> Also, variables changing _will_ cause 
> configure to bail if they've been cached (cflags fex), so you might want
> to either filter those entries (they start with ac_cv_env_), or rewrite
> them (I went for rewrite, works although I worry about other cache
> entries relying on the vars not changing).
I dump the cache when they change in the latest revision of my patch. That 
seems safest. The problem is that some ebuilds (eg fftw) modify CFLAGS, thus 
invalidating the cache.

How safe do you think it is to ignore these variables entirely? Some of them 
like LDFLAGS might affect the results of some configure tests. We could have 
separate cache data for different combinations of them...

> Aside from that, there is also the md5'ing of /proc/cpuinfo- most users,
> not an issue, as magnade pointed out, this is a killer for users that
> have procs that adjust their frequency (laptops fex) for power saving-
> this changes the md5 of /proc/cpuinfo pretty much continually,
> invalidating the cache continually.

We'll have to add special treatment for it then.

Unfortunately I've picked this time to fall sick, so my response time may be 
very high...

-- 
Dan Armak
Gentoo Linux developer (KDE)
Matan, Israel
Public GPG key: http://dev.gentoo.org/~danarmak/danarmak-gpg-public.key
Fingerprint: DD70 DBF9 E3D4 6CB9 2FDD  0069 508D 9143 8D5F 8951

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-portage-dev] re: confcache

[Brian Harring]

On Sat, 2004-10-09 at 02:45, Dan Armak wrote:
> On Friday 08 October 2004 22:08, Brian Harring wrote:
> > Well... that's weird, attempted to send this directly to dan, having it
> > bail due to
> > 'failed: relaying not allowed: danarmak@gentoo.org'
> > I assume I'm being an idiot and screwing up the address (or local
> > configuration)...
> All I can say is, I can receive at that address quite well...
Don't worry about it, I was being an idiot (usually a valid assumption).

> I did, but it's tied to your ability to call back into python from ebuild.sh, 
> so I couldn't use it with the main portage tree. Unless there's a way to do 
> this in stock portage? I don't have any experience with the python side of 
> portage, so please enlighten my ignorance...
Python side of it isn't really required, just how I wrote it
originally.  In hindsight, it's bound much too tightly to my daemon
code, should loosen sandbox's treatment of LD_PRELOAD and split
confcache into a seperate script/prog.  Course that introduces security
issues, although going userpriv + fakeroot ought to help that (playing
with that now).

> > Also, variables changing _will_ cause 
> > configure to bail if they've been cached (cflags fex), so you might want
> > to either filter those entries (they start with ac_cv_env_), or rewrite
> > them (I went for rewrite, works although I worry about other cache
> > entries relying on the vars not changing).
> I dump the cache when they change in the latest revision of my patch. That 
> seems safest. The problem is that some ebuilds (eg fftw) modify CFLAGS, thus 
> invalidating the cache.
> 
> How safe do you think it is to ignore these variables entirely? Some of them 
> like LDFLAGS might affect the results of some configure tests. We could have 
> separate cache data for different combinations of them...
If the variables change, autoconf catches it- causes the configure to
bail unfortunately.  They must be dealt with in some way, although I'm
starting to think filtering them from the cache is a better approach
them rewriting them.

> 
> > Aside from that, there is also the md5'ing of /proc/cpuinfo- most users,
> > not an issue, as magnade pointed out, this is a killer for users that
> > have procs that adjust their frequency (laptops fex) for power saving-
> > this changes the md5 of /proc/cpuinfo pretty much continually,
> > invalidating the cache continually.
> 
> We'll have to add special treatment for it then.
Haven't really figured out a sane way to do special treatment- a good
portion of /proc and /sys will likely require special handling. 
Thoughts?

Not much for just defining a bunch of special cases, but doesn't really
seem to be any other way.
~brian


--
gentoo-portage-dev@gentoo.org mailing list

[gentoo-portage-dev] confcache

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 694 bytes --]

Hola all.

Wrote another confcache implementation (this time not bound to ebd 
thank god), and stuck an ebuild and portage patch for it in 
http://dev.gentoo.org/~ferringb/confcache/ .

Should be a bit stricter then the 2.1 implementation; for those not 
aware of what it is, it's a global autoconf cache + staleness 
detection.  End result is a 2-4x faster configure run when the cache 
is fully primed, decreasing levels of gain if the cache isn't full.  
Worst case, cache is invalidated/missing, which is normal 
configure runtifme + minor overhead.

Portage integration is pretty minor; just an econf modification.  
Feedback/testing desired, so have fun with it.
~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

[gentoo-portage-dev] Re: confcache

[Thomas Kirchner]

[-- Attachment #1: Type: text/plain, Size: 252 bytes --]

* On Nov 14 19:43, Brian Harring (gentoo-portage-dev@lists.gentoo.org) wrote:
> Feedback/testing desired, so have fun with it.

So far it's working like a charm here; thanks for this!  I've been 
waiting for another confcache implementation.
Tom

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]