[gentoo-soc] Auto dependency builder. Need some thoughts before start to code.

[gentoo-soc] Auto dependency builder. Need some thoughts before start to code.

[Александр Берсенев]

Hello,

I am doing auto dependency builder project for GSoC 2011. Here is the
description of my project:
http://www.google-melange.com/gsoc/project/google/gsoc2011/bay/25001 .
In short words, I intent to analyse the files accessed during package
building to generate a list of dependencies. Also I want to block
access to files that not belong to dependency packages to check
fullness of the dependency list .

Unfortunately, my mentor has not answered on my questions I sent two
weeks ago. Also, I got some emails from people trying to find him. I
hope he will appear but I should start to code on May, 23 and I need
some advise for starting from experienced developers because I will be
feel myself uncomfortable writing a code without it.

I plan to start coding with writing a tool on C for launching a
program and logging its file access events to the log file. The format
of log file record: seconds from launch, type of event, name of the
file, and all parent processes.

I want to try the this file logging approach first: to modify
LD_PRELOAD environment var of each child process and catch library
calls. If it will be not good I plan to try another approaches,
described in my proposal. Also I like using the clone of root
filesystem mounted through FUSE for logging.

After that I plan to write a tool on python that launch a logging tool
on C and parse logfile of it.

I have some questions for starting:
- Should I use a tool on C and parse logfile of it or write a logging
python module instead?
- Which logging and file access blocking approach do you think is the best?
- Does it make sense to log file read/write offsets?

I plan to address other questions to my mentor if I reach him.

Best,
Alexander Bersenev

Re: [gentoo-soc] Auto dependency builder. Need some thoughts before start to code.

[Brian Dolbec]

[-- Attachment #1: Type: text/plain, Size: 1440 bytes --]

On Sat, 2011-05-21 at 19:40 +0000, Александр Берсенев wrote:
> Hello,
> 
> I am doing auto dependency builder project for GSoC 2011. Here is the
> description of my project:

> I plan to start coding with writing a tool on C for launching a
> program and logging its file access events to the log file. The format
> of log file record: seconds from launch, type of event, name of the
> file, and all parent processes.
> 
> I want to try the this file logging approach first: to modify
> LD_PRELOAD environment var of each child process and catch library
> calls. If it will be not good I plan to try another approaches,
> described in my proposal. Also I like using the clone of root
> filesystem mounted through FUSE for logging.
> 
> After that I plan to write a tool on python that launch a logging tool
> on C and parse logfile of it.
> 
> I have some questions for starting:
> - Should I use a tool on C and parse logfile of it or write a logging
> python module instead?


What about writing the logging portion in cython so that it can have the
speed of c with the ease of integrating with python code.


> I plan to address other questions to my mentor if I reach him.
> 
> Best,
> Alexander Bersenev
> 

Alex, if you have any need to obtain info about the package using the
portage api, or gentoolkit api's I can help you with that.
-- 
Brian Dolbec <brian.dolbec@gmail.com>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-soc] Auto dependency builder. Need some thoughts before start to code.

[Александр Берсенев]

> What about writing the logging portion in cython so that it can have the
> speed of c with the ease of integrating with python code.
Interesting idea. I'll use it.

> Alex, if you have any need to obtain info about the package using the
> portage api, or gentoolkit api's I can help you with that.
Ok, I will ask you If I will be having a problem with api.
I wonder why depgraph command produce such huge output for any
packages: equery g --depth=0 grub.

Best,
Alexander Bersenev

Re: [gentoo-soc] Auto dependency builder. Need some thoughts before start to code.

[Kamil Kuduk]

> equery g --depth=0 grub

go with the --depth=1 and you might also want to specify the version expicitly.

Re: [gentoo-soc] Auto dependency builder. Need some thoughts before start to code.

[Александр Берсенев]

I am trying to list all packages that depends on a package. I want to
allow an access only to files belonging to one of dependency package
to block an access to other files. It seems depth=0, the unlimited
depth, is what I am should to do. But it gives me a lot of packages:
[ sys-boot/grub-1.98 stats: packages (1958), max depth (169) ]

I don't think that grub really depends on 1958 packages. What does the
output of depgraph with --depth=0 means?

Best,
Alexander Bersenev

Re: [gentoo-soc] Auto dependency builder. Need some thoughts before start to code.

[Brian Dolbec]

[-- Attachment #1: Type: text/plain, Size: 3842 bytes --]

On Sun, 2011-05-22 at 06:28 +0000, Александр Берсенев wrote:
> > What about writing the logging portion in cython so that it can have the
> > speed of c with the ease of integrating with python code.
> Interesting idea. I'll use it.
> 
> > Alex, if you have any need to obtain info about the package using the
> > portage api, or gentoolkit api's I can help you with that.
> Ok, I will ask you If I will be having a problem with api.
> I wonder why depgraph command produce such huge output for any
> packages: equery g --depth=0 grub.
> 
> Best,
> Alexander Bersenev
> 

Because nearly every pkg will depend on something else, so the list will
keep growing till nearly everything installed is listed.  So use
reasonable limits.

anyway for your use you will probably want to use the python api
directly.  

brian@big_daddy  $ python
Python 2.6.6 (r266:84292, Mar  5 2011, 10:27:05) 
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from gentoolkit.dependencies import Dependencies
>>> D=Dependencies('sys-boot/grub-0.97-r10')
>>> deps = D.graph_depends(max_depth=4)
>>> len(deps)
12
>>> deps
[(2, None), (3, <Package u'app-misc/pax-utils-0.2.3'>), (3, <Package
u'sys-apps/checkpolicy-2.0.21'>),
(2, <Package u'sec-policy/selinux-base-policy-20080525-r1'>), (1,
<Package u'sec-policy/selinux-gpm-20080525'>),
(0, <Package u'sys-libs/gpm-1.20.6'>), (3, <Package
u'app-xemacs/ruby-modes-1.04'>),
(3, <Package u'dev-libs/libpcre-8.12'>), (2, <Package
u'dev-lang/swig-2.0.4_pre12643'>),
(2, None), (1, <Package u'virtual/libintl-0'>), (0, <Package
u'sys-apps/findutils-4.5.10'>)]
>>> def printout(deps):
...     for depth, pkg in deps:
...             if pkg:
...                     print pkg.cpv, " depth =", depth
... 
>>> printout(deps)
app-misc/pax-utils-0.2.3  depth = 3
sys-apps/checkpolicy-2.0.21  depth = 3
sec-policy/selinux-base-policy-20080525-r1  depth = 2
sec-policy/selinux-gpm-20080525  depth = 1
sys-libs/gpm-1.20.6  depth = 0
app-xemacs/ruby-modes-1.04  depth = 3
dev-libs/libpcre-8.12  depth = 3
dev-lang/swig-2.0.4_pre12643  depth = 2
virtual/libintl-0  depth = 1
sys-apps/findutils-4.5.10  depth = 0
>>> deps = D.graph_depends(max_depth=1)
>>> printout(deps)
sys-libs/gpm-1.20.6  depth = 0
sys-apps/findutils-4.5.10  depth = 0
sys-apps/help2man-1.39.1  depth = 0
sys-devel/autoconf-wrapper-10-r1  depth = 0
app-arch/xz-utils-5.0.2  depth = 0
>>> deps = D.graph_depends(max_depth=6)
>>> printout(deps)
app-xemacs/ruby-modes-1.04  depth = 5
dev-libs/libpcre-8.12  depth = 5
sys-devel/gcc-4.5.2  depth = 5
dev-python/python-docs-2.7.1  depth = 5
dev-lang/python-2.7.1-r2  depth = 4
sys-apps/sed-4.2.1-r1  depth = 5
sys-devel/flex-2.5.35_p10  depth = 5
sys-libs/libcap-2.20  depth = 5
sys-freebsd/freebsd-lib-8.0  depth = 5
app-misc/pax-utils-0.2.3  depth = 5
sys-libs/timezone-data-2011g  depth = 4
sec-policy/selinux-openldap-20080525  depth = 5
sys-kernel/linux-headers-2.6.38  depth = 4
sys-auth/openpam-20071221  depth = 5
sys-kernel/linux-headers-2.6.38  depth = 4
dev-python/setuptools-0.6.16  depth = 5
sys-devel/binutils-2.21  depth = 5
sys-auth/pam_krb5-4.4  depth = 5
app-editors/gvim-7.3.189  depth = 5
app-vim/pam-syntax-20060424  depth = 4
dev-python/python-docs-3.2  depth = 5
dev-lang/python-3.2  depth = 4
dev-python/python-docs-2.6.6  depth = 5
dev-python/python-docs-2.5.4-r1  depth = 5
dev-lang/python-2.5.4-r4  depth = 4
app-misc/pax-utils-0.2.3  depth = 3
sys-devel/bison-2.5  depth = 4
sys-apps/checkpolicy-2.0.21  depth = 3
sec-policy/selinux-base-policy-20080525-r1  depth = 2
sec-policy/selinux-gpm-20080525  depth = 1
sys-libs/gpm-1.20.6  depth = 0
sys-apps/findutils-4.5.10  depth = 0
>>> 

-- 
Brian Dolbec <brian.dolbec@gmail.com>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-soc] Auto dependency builder. Need some thoughts before start to code.

[Александр Берсенев]

> Because nearly every pkg will depend on something else, so the list will
> keep growing till nearly everything installed is listed.  So use
> reasonable limits.

I understand: depgraph produce pessimistic graph - graph if all
conditional deps is satisfied.
For example I have "-gpm" USE flag. From ncurses ebuild: 'DEPEND="gpm?
( sys-libs/gpm )"'
If I launch equery, it shows the sys-libs/gpm:
# equery g =sys-libs/ncurses-5.9 --depth=1
 `--  sys-libs/ncurses-5.9  ~amd64
  `--  sys-libs/gpm-1.20.6  (sys-libs/gpm) ~amd64
[ sys-libs/ncurses-5.9 stats: packages (2), max depth (0) ]

Using API I got this:
from gentoolkit.dependencies import Dependencies
D=Dependencies('sys-libs/ncurses-5.9')
D.graph_depends(max_depth=1)
[(0, <Package u'sec-policy/selinux-gpm-20080525'>)]

I don't have any idea why selinux-gpm is here. There is no selinux-gpm
package in ebuild of ncurses nor in the inherited eclasses. Also, I
don't have it installed and it is hard-masked. I will try to
understand the logic of dependencies in Gentoo more deeply.

Best,
Alexander Bersenev

Re: [gentoo-soc] Auto dependency builder. Need some thoughts before start to code.

[Brian Dolbec]

[-- Attachment #1: Type: text/plain, Size: 1447 bytes --]

On Sun, 2011-05-22 at 17:08 +0000, Александр Берсенев wrote:
> > Because nearly every pkg will depend on something else, so the list will
> > keep growing till nearly everything installed is listed.  So use
> > reasonable limits.
> 
> I understand: depgraph produce pessimistic graph - graph if all
> conditional deps is satisfied.
> For example I have "-gpm" USE flag. From ncurses ebuild: 'DEPEND="gpm?
> ( sys-libs/gpm )"'
> If I launch equery, it shows the sys-libs/gpm:
> # equery g =sys-libs/ncurses-5.9 --depth=1
>  `--  sys-libs/ncurses-5.9  ~amd64
>   `--  sys-libs/gpm-1.20.6  (sys-libs/gpm) ~amd64
> [ sys-libs/ncurses-5.9 stats: packages (2), max depth (0) ]
> 
> Using API I got this:
> from gentoolkit.dependencies import Dependencies
> D=Dependencies('sys-libs/ncurses-5.9')
> D.graph_depends(max_depth=1)
> [(0, <Package u'sec-policy/selinux-gpm-20080525'>)]
> 
> I don't have any idea why selinux-gpm is here. There is no selinux-gpm
> package in ebuild of ncurses nor in the inherited eclasses. Also, I
> don't have it installed and it is hard-masked. I will try to
> understand the logic of dependencies in Gentoo more deeply.
> 
> Best,
> Alexander Bersenev
> 

OK, that is fixed in git now, so emerge gentoolkit-9999 for the fix.
Don't worry about it being 'unstable', it will likely be the -0.3.0.4
maintenance release anyway.


-- 
Brian Dolbec <brian.dolbec@gmail.com>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]