[gentoo-soc] Auto dependency builder progress report. Week 5.

[gentoo-soc] Auto dependency builder progress report. Week 5.

[Александр Берсенев]

What has been done this week:
- rewritten stage-logging algorithm. New one is quicker and more reliable.
- changed type of internal communication sockets from SOCK_STREAM to
SOCK_SEQPACKET. This gives no performance improvement but code became
simpler.
- move from select to epoll for socket events notifications in python
server part.
- fully rewritten ld_preload hooking library, here is a commit:
http://git.overlays.gentoo.org/gitweb/?p=proj/autodep.git;a=commitdiff;h=d8d0c8f39f95af3647f73b5781199899e9b9529d
.
- test of both approaches has been unified.

---
I've spent a lot of time debugging very interesting issue and I want
to describe it here.

One of approaches I use for logging filesystem events is ld_preload
approach. I set LD_PRELOAD environment variable to tell OS to load my
library before starting an application. My library has own functions
like open, read and write and they are launched before glibc's. This
functions tells main python process about an event through unix domain
sockets. Name of socket file is stored in LOG_SOCKET environment
variable. When proccess trying to launch another process LD_PRELOAD
and LOG_SOCKET are inherited so my library is loaded before this
another process too.

But I've met a problem: in "prerm" stage of building connects to
socket always were refused!

First I think it is a bug in python server part. I moved it back from
epoll to select method for socket events notifications. Bug not
disappeared. Then I double-checked the code. All seems to be good, but
bug remains. Then I reverted few commits and this not helps.

Then I begin to examine the logging library. I think it is some
threading issue. I forced library code execute in one thread. This
gives no effect. Then I wrote small unix-socket client. Connections of
prerm stage was refused but in same time my client was connecting
successfully. In netstat socket was "listening".

Then I began to log all environment variables and find then someone
changes my LOG_SOCKET variable during "prerm" stage. I renamed it to
LOG_SOCKET_<LONG_STRING> in all sources hoping this is a
name-collision. But variable continues changing its value. The most
interesting fact: value was changing to another valid socket name, for
example from "/tmp/tmpFSDhf/socket_for_logging" to
"/tmp/tmpHDvJ5/socket_for_logging". But nobody listened
"/tmp/tmpHDvJ5/socket_for_logging". I started to believe in the higher
powers in this moment because last socket file was created five days
ago and notebook has been rebooted several times these days.

But the explanation was simple: portage system remembers all
environment variables while installing a package and restores them
while removing this package!!

Best,

Alexander Bersenev

Re: [gentoo-soc] Auto dependency builder progress report. Week 5.

[Fabian Groffen]

On 29-06-2011 13:28:45 +0600, Александр Берсенев wrote:
> Then I began to log all environment variables and find then someone
> changes my LOG_SOCKET variable during "prerm" stage. I renamed it to
> LOG_SOCKET_<LONG_STRING> in all sources hoping this is a
> name-collision. But variable continues changing its value. The most
> interesting fact: value was changing to another valid socket name, for
> example from "/tmp/tmpFSDhf/socket_for_logging" to
> "/tmp/tmpHDvJ5/socket_for_logging". But nobody listened
> "/tmp/tmpHDvJ5/socket_for_logging". I started to believe in the higher
> powers in this moment because last socket file was created five days
> ago and notebook has been rebooted several times these days.
> 
> But the explanation was simple: portage system remembers all
> environment variables while installing a package and restores them
> while removing this package!!

You can add your variable(s) to a white/blacklist in the code that saves
the environment.
Check save_ebuild_env() in isolated-functions.sh.


-- 
Fabian Groffen
Gentoo on a different level

Re: [gentoo-soc] Auto dependency builder progress report. Week 5.

[Александр Берсенев]

I added a checking of environment variables after each fork call. The
checking was there before but it is not worked and I still not
understand why. I spent 3 days on this issue and it is seems to be
resolved now.

Can I add my variable to exclude list without patching the portage
part? In save_ebuild_env I see very big list of excluded variables and
functions names and this is very good that LD_PRELOAD is in it.

Thank you for advise.

2011/6/29 Fabian Groffen <grobian@gentoo.org>:
> You can add your variable(s) to a white/blacklist in the code that saves
> the environment.
> Check save_ebuild_env() in isolated-functions.sh.

Re: [gentoo-soc] Auto dependency builder progress report. Week 5.

[Zac Medico]

On 06/30/2011 01:06 PM, Александр Берсенев wrote:
> Can I add my variable to exclude list without patching the portage
> part?

No, there's currently no way to do that.

> In save_ebuild_env I see very big list of excluded variables and
> functions names and this is very good that LD_PRELOAD is in it.

Maybe you can write your variable settings to /etc/portage/bashrc. That
file is sourced by ebuild.sh and can be used to override saved
environment variables.
-- 
Thanks,
Zac