[gentoo-dev] Autodep project

[gentoo-dev] Autodep project

[Александр Берсенев]

Hi!

I am Alexander Bersenev, I was participating in GSoC this year.
I want to present you the tool I've developed during this program. I
hope that you'll find it useful.

The purpose of my project is to help ebuild developers to compose
accurate dependency list for a package.
The tool has many features in order to do it, most of them listed on
the documentation site: http://soc.dev.gentoo.org/~bay/autodep/

The killer-feature is an emulating the file system without
non-dependency packages installed. I call it dependency checking or
strict emerging. If program builds successfully in this environment
then check is passed. If no - check is failed and user likely will be
having a bad experience while trying to build this package if he
hasn't some packages installed.

It works fine, I've reported a few dozens of bugs about missing
dependencies here: https://bugs.gentoo.org/show_bug.cgi?id=autodep.

How to install and use it:
1) add neurogeek overlay in your overlay list
2) emerge autodep
3) use autodep and emerge_strict commands.

I want to tell you more about emerge_strict command. This is an emerge
command but with strict dependency checking. I've modified a portage
and add this feature into it. Actually, after "emerge autodep" you
will have two versions of portage: one is from your system(emerge) and
one is from modified portage(emerge_strict).

!!!ATTENTION!!!
I modified a last available portage version from git. It is about
Portage 2.2.0_alpha50. The you running portage 2.1.x.x it
theoretically can be unsafe to use them both. I used it together for
about a month and not found any problems, but, anyway, be careful.
!!!ATTENTION!!!

Here is an example of emerge_strict dev-libs/nss output:
https://381591.bugs.gentoo.org/attachment.cgi?id=285369

Missing dependency is founded here:
sdb.c:58:21: fatal error: sqlite3.h: No such file or directory
compilation terminated.
....

[NOT IN DEPS] dev-db/sqlite-3.7.7.1                   : [u'compile']
 /usr/include/sqlite3.h                                   blocked
....

I've set up a tinderbox to catch missing dependencies. And it works
right now. Also, I recently installed desktop gentoo linux on my new
notebook using only emerge_strict. It is also works.

Although GSoC is over, I want to support this tool in further. I will
be appreciate for any feedback.

Best,

Alexander Bersenev

Re: [gentoo-dev] Autodep project

[Marc Schiffbauer]

* Александр Берсенев schrieb am 04.09.11 um 09:13 Uhr:
> Hi!

Hi Александр,

seems to be a nice tool.

I am not a native english speaker, but I found at least some words
that need to be fixed:

* There is no "readed" its always "read"
* And: writed -> written

-Marc


> 
> I am Alexander Bersenev, I was participating in GSoC this year.
> I want to present you the tool I've developed during this program. I
> hope that you'll find it useful.
> 
> The purpose of my project is to help ebuild developers to compose
> accurate dependency list for a package.
> The tool has many features in order to do it, most of them listed on
> the documentation site: http://soc.dev.gentoo.org/~bay/autodep/
> 
> The killer-feature is an emulating the file system without
> non-dependency packages installed. I call it dependency checking or
> strict emerging. If program builds successfully in this environment
> then check is passed. If no - check is failed and user likely will be
> having a bad experience while trying to build this package if he
> hasn't some packages installed.
> 
> It works fine, I've reported a few dozens of bugs about missing
> dependencies here: https://bugs.gentoo.org/show_bug.cgi?id=autodep.
> 
> How to install and use it:
> 1) add neurogeek overlay in your overlay list
> 2) emerge autodep
> 3) use autodep and emerge_strict commands.
> 
> I want to tell you more about emerge_strict command. This is an emerge
> command but with strict dependency checking. I've modified a portage
> and add this feature into it. Actually, after "emerge autodep" you
> will have two versions of portage: one is from your system(emerge) and
> one is from modified portage(emerge_strict).
> 
> !!!ATTENTION!!!
> I modified a last available portage version from git. It is about
> Portage 2.2.0_alpha50. The you running portage 2.1.x.x it
> theoretically can be unsafe to use them both. I used it together for
> about a month and not found any problems, but, anyway, be careful.
> !!!ATTENTION!!!
> 
> Here is an example of emerge_strict dev-libs/nss output:
> https://381591.bugs.gentoo.org/attachment.cgi?id=285369
> 
> Missing dependency is founded here:
> sdb.c:58:21: fatal error: sqlite3.h: No such file or directory
> compilation terminated.
> ....
> 
> [NOT IN DEPS] dev-db/sqlite-3.7.7.1                   : [u'compile']
>  /usr/include/sqlite3.h                                   blocked
> ....
> 
> I've set up a tinderbox to catch missing dependencies. And it works
> right now. Also, I recently installed desktop gentoo linux on my new
> notebook using only emerge_strict. It is also works.
> 
> Although GSoC is over, I want to support this tool in further. I will
> be appreciate for any feedback.
> 
> Best,
> 
> Alexander Bersenev
> 

-- 
8AAC 5F46 83B4 DB70 8317  3723 296C 6CCA 35A6 4134

Re: [gentoo-dev] Autodep project

[justin]

[-- Attachment #1.1: Type: text/plain, Size: 2912 bytes --]

Hi,

if this tool works like it supposed, it will be very handy. But the
current ebuild/buildsystem suffers from disrespecting CC and failing
with forced as-needed, isn't strict multilib capable and
byte-compilation of python modules. Please find attached a hacky patch.

thanks justin

On 9/4/11 9:13 AM, Александр Берсенев wrote:
> Hi!
> 
> I am Alexander Bersenev, I was participating in GSoC this year.
> I want to present you the tool I've developed during this program. I
> hope that you'll find it useful.
> 
> The purpose of my project is to help ebuild developers to compose
> accurate dependency list for a package.
> The tool has many features in order to do it, most of them listed on
> the documentation site: http://soc.dev.gentoo.org/~bay/autodep/
> 
> The killer-feature is an emulating the file system without
> non-dependency packages installed. I call it dependency checking or
> strict emerging. If program builds successfully in this environment
> then check is passed. If no - check is failed and user likely will be
> having a bad experience while trying to build this package if he
> hasn't some packages installed.
> 
> It works fine, I've reported a few dozens of bugs about missing
> dependencies here: https://bugs.gentoo.org/show_bug.cgi?id=autodep.
> 
> How to install and use it:
> 1) add neurogeek overlay in your overlay list
> 2) emerge autodep
> 3) use autodep and emerge_strict commands.
> 
> I want to tell you more about emerge_strict command. This is an emerge
> command but with strict dependency checking. I've modified a portage
> and add this feature into it. Actually, after "emerge autodep" you
> will have two versions of portage: one is from your system(emerge) and
> one is from modified portage(emerge_strict).
> 
> !!!ATTENTION!!!
> I modified a last available portage version from git. It is about
> Portage 2.2.0_alpha50. The you running portage 2.1.x.x it
> theoretically can be unsafe to use them both. I used it together for
> about a month and not found any problems, but, anyway, be careful.
> !!!ATTENTION!!!
> 
> Here is an example of emerge_strict dev-libs/nss output:
> https://381591.bugs.gentoo.org/attachment.cgi?id=285369
> 
> Missing dependency is founded here:
> sdb.c:58:21: fatal error: sqlite3.h: No such file or directory
> compilation terminated.
> ....
> 
> [NOT IN DEPS] dev-db/sqlite-3.7.7.1                   : [u'compile']
>  /usr/include/sqlite3.h                                   blocked
> ....
> 
> I've set up a tinderbox to catch missing dependencies. And it works
> right now. Also, I recently installed desktop gentoo linux on my new
> notebook using only emerge_strict. It is also works.
> 
> Although GSoC is over, I want to support this tool in further. I will
> be appreciate for any feedback.
> 
> Best,
> 
> Alexander Bersenev
> 


[-- Attachment #1.2: autodep-0.1.patch --]
[-- Type: text/plain, Size: 2324 bytes --]

diff --git a/app-portage/autodep/autodep-0.1.ebuild b/app-portage/autodep/autodep-0.1.ebuild
index 62b4290..e4f494b 100644
--- a/app-portage/autodep/autodep-0.1.ebuild
+++ b/app-portage/autodep/autodep-0.1.ebuild
@@ -7,7 +7,7 @@ EAPI=3
 PYTHON_DEPEND="2:2.6"
 RESTRICT_PYTHON_ABIS='2.4 2.5 3.*'
 
-inherit python
+inherit autotools eutils flag-o-matic multilib python
 
 DESCRIPTION="Auto dependency analyser for Gentoo"
 HOMEPAGE="http://soc.dev.gentoo.org/~bay/autodep/"
@@ -18,15 +18,31 @@ SLOT="0"
 KEYWORDS="~x86 ~amd64"
 IUSE=""
 
-DEPEND="sys-fs/fuse
+RDEPEND="
+	app-portage/portage-utils
+	sys-fs/fuse"
+DEPEND="${RDEPEND}
 	dev-util/pkgconfig"
-RDEPEND="sys-fs/fuse
-	app-portage/portage-utils"
 
-src_compile() {
-    emake || die
+pkg_setup() {
+	python_set_active_version 2
+}
+
+src_prepare() {
+	epatch "${FILESDIR}"/${P}-build.patch
+	sed "s:usr/lib:/usr/$(get_libdir):g" -i Makefile || die
+	tc-export CC
+	export raw_LDFLAGS="$(raw-ldflags)"
 }
 
 src_install() {
     emake DESTDIR="${D}" install || die
 }
+
+pkg_postinst() {
+	python_mod_optimize /usr/$(get_libdir)/portage_with_autodep
+}
+
+pkg_postrm() {
+	python_mod_cleanup /usr/$(get_libdir)/portage_with_autodep
+}
diff --git a/app-portage/autodep/files/autodep-0.1-build.patch b/app-portage/autodep/files/autodep-0.1-build.patch
new file mode 100644
index 0000000..e49bd54
--- /dev/null
+++ b/app-portage/autodep/files/autodep-0.1-build.patch
@@ -0,0 +1,26 @@
+ Makefile |    8 ++++----
+ 1 files changed, 4 insertions(+), 4 deletions(-)
+
+diff --git a/Makefile b/Makefile
+index 01c92f1..1a26d16 100644
+--- a/Makefile
++++ b/Makefile
+@@ -1,14 +1,14 @@
+ all: hookfs file_hook.so
+ 
+ hookfs: src/hook_fusefs/hookfs.c
+-	$(CC) -std=c99 -Wall `pkg-config fuse --cflags --libs` -lulockmgr \
+-	$(CFLAGS) $(LDFLAGS) src/hook_fusefs/hookfs.c -o hookfs
++	$(CC) -std=c99 -Wall	$(CFLAGS) $(LDFLAGS) \
++	src/hook_fusefs/hookfs.c -o hookfs `pkg-config fuse --cflags --libs` -lulockmgr
+ 
+ file_hook.so: file_hook.o
+-	ld -shared -o file_hook.so -ldl -lc file_hook.o
++	ld $(raw_LDFLAGS) -shared -o file_hook.so file_hook.o -ldl -lc
+ 
+ file_hook.o: src/hook_lib/file_hook.c
+-	cc -Wall -fPIC -o file_hook.o -c src/hook_lib/file_hook.c
++	$(CC) -Wall -fPIC -o file_hook.o -c src/hook_lib/file_hook.c
+ 
+ install:
+ 	mkdir -p "${DESTDIR}/usr/lib/"

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 267 bytes --]

Re: [gentoo-dev] Autodep project

[justin]

[-- Attachment #1: Type: text/plain, Size: 3126 bytes --]

You need of course the toolchain-funcs.eclass and not the
autotools.eclass. Sorry for that mistake.


On 9/4/11 11:24 AM, justin wrote:
> Hi,
> 
> if this tool works like it supposed, it will be very handy. But the
> current ebuild/buildsystem suffers from disrespecting CC and failing
> with forced as-needed, isn't strict multilib capable and
> byte-compilation of python modules. Please find attached a hacky patch.
> 
> thanks justin
> 
> On 9/4/11 9:13 AM, Александр Берсенев wrote:
>> Hi!
>>
>> I am Alexander Bersenev, I was participating in GSoC this year.
>> I want to present you the tool I've developed during this program. I
>> hope that you'll find it useful.
>>
>> The purpose of my project is to help ebuild developers to compose
>> accurate dependency list for a package.
>> The tool has many features in order to do it, most of them listed on
>> the documentation site: http://soc.dev.gentoo.org/~bay/autodep/
>>
>> The killer-feature is an emulating the file system without
>> non-dependency packages installed. I call it dependency checking or
>> strict emerging. If program builds successfully in this environment
>> then check is passed. If no - check is failed and user likely will be
>> having a bad experience while trying to build this package if he
>> hasn't some packages installed.
>>
>> It works fine, I've reported a few dozens of bugs about missing
>> dependencies here: https://bugs.gentoo.org/show_bug.cgi?id=autodep.
>>
>> How to install and use it:
>> 1) add neurogeek overlay in your overlay list
>> 2) emerge autodep
>> 3) use autodep and emerge_strict commands.
>>
>> I want to tell you more about emerge_strict command. This is an emerge
>> command but with strict dependency checking. I've modified a portage
>> and add this feature into it. Actually, after "emerge autodep" you
>> will have two versions of portage: one is from your system(emerge) and
>> one is from modified portage(emerge_strict).
>>
>> !!!ATTENTION!!!
>> I modified a last available portage version from git. It is about
>> Portage 2.2.0_alpha50. The you running portage 2.1.x.x it
>> theoretically can be unsafe to use them both. I used it together for
>> about a month and not found any problems, but, anyway, be careful.
>> !!!ATTENTION!!!
>>
>> Here is an example of emerge_strict dev-libs/nss output:
>> https://381591.bugs.gentoo.org/attachment.cgi?id=285369
>>
>> Missing dependency is founded here:
>> sdb.c:58:21: fatal error: sqlite3.h: No such file or directory
>> compilation terminated.
>> ....
>>
>> [NOT IN DEPS] dev-db/sqlite-3.7.7.1                   : [u'compile']
>>  /usr/include/sqlite3.h                                   blocked
>> ....
>>
>> I've set up a tinderbox to catch missing dependencies. And it works
>> right now. Also, I recently installed desktop gentoo linux on my new
>> notebook using only emerge_strict. It is also works.
>>
>> Although GSoC is over, I want to support this tool in further. I will
>> be appreciate for any feedback.
>>
>> Best,
>>
>> Alexander Bersenev
>>
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 267 bytes --]

Re: [gentoo-dev] Autodep project

[Александр Берсенев]

Thanks, I'll try to fix it. And English too.

Best,

Alexander Bersenev

Re: [gentoo-dev] Autodep project

[Tomáš Chvátal]

Hi,
really cool thing you create :)

Would it be possible to move that package to main tree, and merge
or possibly add new FEATURES option to portage like "autobuildchecks" that
would be set by -dev profile?

Cheers
Tom

Re: [gentoo-dev] Autodep project

[Alexander Bersenev]

I already have depcheck and depcheckstrict FEATURES. Emerge_strict script launch modified version of emerge enabling depcheckstrict feature.

I have a 0.2 version of this utility with some bugs fixed, I testing it now.

I plan to move this package into main tree and I am going to talk about integration of this patch into portage but I think some things must be changed in it before.

Best,

Alexander Bersenev

On 07.09.2011, at 15:08, Tomáš Chvátal <scarabeus@gentoo.org> wrote:

> Hi,
> really cool thing you create :)
> 
> Would it be possible to move that package to main tree, and merge
> or possibly add new FEATURES option to portage like "autobuildchecks" that
> would be set by -dev profile?
> 
> Cheers
> Tom
> 

Re: [gentoo-dev] Autodep project

[Александр Берсенев]

Sorry for a long delay.

New version of autodep is available. To install it do:
1) add neurogeek overlay in your overlay list
2) emerge autodep
3) use autodep and emerge_strict commands.

This version was heavily on my few computers for a month.
- Fixed a crash when file with non-unicode character was in the name of file.
- Some packages redefined snprintf function and autodep wasn't
expected it. Now it uses own snprintf function.

I am testing 0.3 version now. I slightly rewrote an algo of allowed
packages list composing:
http://git.overlays.gentoo.org/gitweb/?p=proj/autodep.git;a=commitdiff;h=e541c423ce50003e05b9a154a0893cafe8bd7445

Best,
Alexander Bersenev

2011/9/7 Alexander Bersenev <bay@hackerdom.ru>:
> I already have depcheck and depcheckstrict FEATURES. Emerge_strict script launch modified version of emerge enabling depcheckstrict feature.
>
> I have a 0.2 version of this utility with some bugs fixed, I testing it now.
>
> I plan to move this package into main tree and I am going to talk about integration of this patch into portage but I think some things must be changed in it before.
>
> Best,
>
> Alexander Bersenev
>
> On 07.09.2011, at 15:08, Tomáš Chvátal <scarabeus@gentoo.org> wrote:
>
>> Hi,
>> really cool thing you create :)
>>
>> Would it be possible to move that package to main tree, and merge
>> or possibly add new FEATURES option to portage like "autobuildchecks" that
>> would be set by -dev profile?
>>
>> Cheers
>> Tom
>>
>

Re: [gentoo-dev] Autodep project

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 699 bytes --]

On Mon, 17 Oct 2011 06:10:26 +0000
Александр Берсенев <bay@hackerdom.ru> wrote:

> Sorry for a long delay.
> 
> New version of autodep is available. To install it do:
> 1) add neurogeek overlay in your overlay list
> 2) emerge autodep
> 3) use autodep and emerge_strict commands.

First thing I notice is:

Blocking an access to /usr/bin/debugedit

So it breaks FEATURES=splitdebug and doesn't install the debug data
then.

Second one:

[NOT IN DEPS] app-shells/dash-0.5.7.1                 : [u'configure',
u'compile'] /bin/dash
  blocked              

dash is my /bin/sh symlink. I don't think it should be bothered.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] Autodep project

[justin]

[-- Attachment #1: Type: text/plain, Size: 1806 bytes --]

On 17/10/11 08:10, Александр Берсенев wrote:
> Sorry for a long delay.
> 
> New version of autodep is available. To install it do:
> 1) add neurogeek overlay in your overlay list
> 2) emerge autodep
> 3) use autodep and emerge_strict commands.
> 
> This version was heavily on my few computers for a month.
> - Fixed a crash when file with non-unicode character was in the name of file.
> - Some packages redefined snprintf function and autodep wasn't
> expected it. Now it uses own snprintf function.
> 
> I am testing 0.3 version now. I slightly rewrote an algo of allowed
> packages list composing:
> http://git.overlays.gentoo.org/gitweb/?p=proj/autodep.git;a=commitdiff;h=e541c423ce50003e05b9a154a0893cafe8bd7445
> 
> Best,
> Alexander Bersenev
> 
> 2011/9/7 Alexander Bersenev <bay@hackerdom.ru>:
>> I already have depcheck and depcheckstrict FEATURES. Emerge_strict script launch modified version of emerge enabling depcheckstrict feature.
>>
>> I have a 0.2 version of this utility with some bugs fixed, I testing it now.
>>
>> I plan to move this package into main tree and I am going to talk about integration of this patch into portage but I think some things must be changed in it before.
>>
>> Best,
>>
>> Alexander Bersenev
>>
>> On 07.09.2011, at 15:08, Tomáš Chvátal <scarabeus@gentoo.org> wrote:
>>
>>> Hi,
>>> really cool thing you create :)
>>>
>>> Would it be possible to move that package to main tree, and merge
>>> or possibly add new FEATURES option to portage like "autobuildchecks" that
>>> would be set by -dev profile?
>>>
>>> Cheers
>>> Tom
>>>
>>
> 

Please fix this https://bugs.gentoo.org/show_bug.cgi?id=386873 and a
valid bgo account mail address to the metadata.xml. (And stop topposting)

justin



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]