[gentoo-soc] Auto dependency builder progress report. Week 7.

[gentoo-soc] Auto dependency builder progress report. Week 7.

[Александр Берсенев]

What has been done this week:
- exec* call handling: execl, execlp, execle, execv, execvp, execvpe,
execve, system
- tests for all exec* calls
- fixed a bug with setenv and multithreading. I developed own setenv
function to fix it
- handling mmap call
- a function get_deps(pkg) has been developed. It returns a list of
installed dependencies for package, installed dependencies of
dependencies, etc.
- changes in file structure of my project

What I plan to do next:
- filter packages which are part of system profile in output
- filter packages which having files accessed only, but not readed or
writed or executed.
- highlight packages, used during package building but not listed in
dependency lists of any level
- think about python versions in dependencies.
- produce a log file with all events and data about environment to
send it on central storage.

Best,
Alexander Bersenev

Re: [gentoo-soc] Auto dependency builder progress report. Week 7.

[Donnie Berkholz]

[-- Attachment #1: Type: text/plain, Size: 981 bytes --]

On 06:22 Thu 14 Jul     , Александр Берсенев wrote:
> - filter packages which are part of system profile in output

Can you make this optional? If I'm working on a system-level package, I 
might want to keep track of some of its system dependencies.

> - filter packages which having files accessed only, but not readed or
> writed or executed.

Why does this happen, anyway?

> - highlight packages, used during package building but not listed in
> dependency lists of any level

For DEPEND rather than RDEPEND ones, we'll want to make sure they're 
definitely listed in dependencies of the ebuild being used, rather than 
any level.

> - think about python versions in dependencies.
> - produce a log file with all events and data about environment to
> send it on central storage.
> 
> Best,
> Alexander Bersenev
> 

-- 
Thanks,
Donnie

Donnie Berkholz
Admin, Summer of Code
Gentoo Linux and X.Org
Blog: http://dberkholz.com


[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-soc] Auto dependency builder progress report. Week 7.

[Александр Берсенев]

2011/7/15 Donnie Berkholz <dberkholz@gentoo.org>:
> On 06:22 Thu 14 Jul     , Александр Берсенев wrote:
>> - filter packages which are part of system profile in output
>
> Can you make this optional? If I'm working on a system-level package, I
> might want to keep track of some of its system dependencies.
Yes, this is already optional. I am trying to shrink an output, but
there is --verbose key to show all packages, including "unknown"
package and unknown stages and also there is --files key to show not
only packages and stages, but also files accessed(including not
founded and blocked).

I think in most cases filtering system packages from output is useful
because this packages almost never listed in dependencies and user has
them installed in general case.

>> - filter packages which having files accessed only, but not readed or
>> writed or executed.
>
> Why does this happen, anyway?
>
This is very rare case, for example when program scans a directory.
I've seen this for one or two times only.

I am trying to get rid from non-dependency packages in output. Here is
few examples.
All non-system packages when "emerge lsof":
dev-lang/python-3.1.3-r1
<many files here>
dev-libs/openssl-1.0.0d                 : ['unpack', 'compile',
'test', 'install']
  /etc/sandbox.d/10openssl                                 readed
media-libs/fontconfig-2.8.0-r1          : ['unpack', 'compile',
'test', 'install']
  /etc/sandbox.d/37fontconfig                              readed
net-zope/zope-fixers-1.0                : ['setup', 'unpack',
'compile', 'test', 'install', 'preinst', 'postinst', 'prerm',
'postrm']
  /usr/lib64/python3.1/site-packages/zope.fixers-1.0-py3.1-nspkg.pth readed
net-zope/zope-interface-3.6.2           : ['setup', 'unpack',
'compile', 'test', 'install', 'preinst', 'postinst', 'prerm',
'postrm']
  /usr/lib64/python3.1/site-packages/zope.interface-3.6.2-py3.1-nspkg.pth readed
sys-apps/sandbox-2.5                    : ['unpack', 'compile',
'test', 'install']
  /etc/sandbox.d/00default                                 readed
  /etc/sandbox.conf                                        readed
  /usr/share/sandbox/sandbox.bashrc                        readed

Openssl and fontconfig is here because /etc/sandbox.d/* is readed. I
think about filtering this directory from logging. Python is here
because it is not a part of system set. I don't know about zope-*
packages and why they have files readed.

While I do emerge bash I got more packages:
sys-devel/crossdev-20110310             : ['compile']
  /usr/share/config.site                                   readed
  /usr/share/crossdev/include/site/linux                   readed
sys-devel/gettext-0.18.1.1-r2           : ['compile']
  /usr/bin/msgmerge                                        readed
  /usr/bin/xgettext                                        readed
  /usr/bin/msgfmt                                          readed
  /usr/bin/gmsgfmt                                         readed

Gettext is a dependency of bash, but crossdev is unknown package.

>> - highlight packages, used during package building but not listed in
>> dependency lists of any level
>
> For DEPEND rather than RDEPEND ones, we'll want to make sure they're
> definitely listed in dependencies of the ebuild being used, rather than
> any level.

The main drawback of hooklib approach is an inability to track what
files have been loaded while exec call(runtime libraries). Most
RDEPENDS are naturally filtered here. Fusefs approach logs all file
system events.

This is a difficult problem to distinguish RDEPENDS from DEPENDS. If I
look on files accessed I can say if the dependency is runtime, but I
haven't general strategy. I should watch more building logs of various
packages to find a way.

Best,
Alexander Bersenev

Re: [gentoo-soc] Auto dependency builder progress report. Week 7.

[Donnie Berkholz]

[-- Attachment #1: Type: text/plain, Size: 1105 bytes --]

On 16:29 Fri 15 Jul     , Александр Берсенев wrote:
> 2011/7/15 Donnie Berkholz <dberkholz@gentoo.org>:
> > For DEPEND rather than RDEPEND ones, we'll want to make sure they're 
> > definitely listed in dependencies of the ebuild being used, rather 
> > than any level.
> 
> The main drawback of hooklib approach is an inability to track what
> files have been loaded while exec call(runtime libraries). Most
> RDEPENDS are naturally filtered here. Fusefs approach logs all file
> system events.
> 
> This is a difficult problem to distinguish RDEPENDS from DEPENDS. If I
> look on files accessed I can say if the dependency is runtime, but I
> haven't general strategy. I should watch more building logs of various
> packages to find a way.

You might be able to use src_test for this. If a package is accessed in 
src_test as well as src_configure/src_compile, it's RDEPEND. Clearly 
this won't work when src_test isn't defined, but it's a start.

-- 
Thanks,
Donnie

Donnie Berkholz
Admin, Summer of Code
Gentoo Linux and X.Org
Blog: http://dberkholz.com


[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-soc] Auto dependency builder progress report. Week 7.

[Jesus Rivero (Neurogeek)]

On Fri, Jul 15, 2011 at 2:54 PM, Donnie Berkholz <dberkholz@gentoo.org> wrote:
> On 16:29 Fri 15 Jul     , Александр Берсенев wrote:
>> 2011/7/15 Donnie Berkholz <dberkholz@gentoo.org>:
>> > For DEPEND rather than RDEPEND ones, we'll want to make sure they're
>> > definitely listed in dependencies of the ebuild being used, rather
>> > than any level.
>>
>> The main drawback of hooklib approach is an inability to track what
>> files have been loaded while exec call(runtime libraries). Most
>> RDEPENDS are naturally filtered here. Fusefs approach logs all file
>> system events.
>>
>> This is a difficult problem to distinguish RDEPENDS from DEPENDS. If I
>> look on files accessed I can say if the dependency is runtime, but I
>> haven't general strategy. I should watch more building logs of various
>> packages to find a way.
>
> You might be able to use src_test for this. If a package is accessed in
> src_test as well as src_configure/src_compile, it's RDEPEND. Clearly
> this won't work when src_test isn't defined, but it's a start.
>
RDEPEND will be tricky to get right, but I like this approach, could
work for those packages with tests.
We are going to discuss an approach for this and for the filtering of
@system, as we need to find a way to get rid of second-hand deps.

> --
> Thanks,
> Donnie
>
> Donnie Berkholz
> Admin, Summer of Code
> Gentoo Linux and X.Org
> Blog: http://dberkholz.com
>
>



-- 
Jesus Rivero (Neurogeek)
Gentoo Developer