[gentoo-soc] Ebuild generator week 9 update

[gentoo-soc] Ebuild generator week 9 update

[darkdefende]

Sorry for the delay of last weeks update. I got sick on the weekend so i
didn't get the update done when I wanted to.
Anyways here it is.

Last week I worked on how to interpret the information that I get from
the autoconf and automake parser. I would then combine that information
and create an ebuild.

However it didn't go as smoothly as I planned so this is still not done.
The main problem is that I need to know what the "switches" in the
configure script does so I can link them to useflags with correct deps.
If I just were to write a autoconf replacement in python there wouldn't
be much of a problem as I just have to "do" not try to guess what it
does.

Right now I have problems with figuring out how I should handle stuff in
the autoconf file that checks for headers,libs or package checks.
Granted, in most cases they are just there to see if you have the
dependecy or if you have the correct version.
So basicly you offen check if the correct dependecy is installed on the
system and then give green light to build some extra features.

But the problem is you can have it go the other way also. I.E:
If not the correct dependecy is installed then disable some features or
use old API.

Because my ebuild generator should be able generate ebuilds without all
the required dependencies this is a problem.
I can't run the checks as they will fail as everything required to build
the program might not be installed. And if I skip the checks there might
be problems as I can't think of a way that I can guess what it will try
to do.

Do you guy have any ideas how I should solve this problem? Or should I
just "bindly" skip the checks and do what it would do if that check
would have passed?

Re: [gentoo-soc] Ebuild generator week 9 update

[Donnie Berkholz]

[-- Attachment #1: Type: text/plain, Size: 2768 bytes --]

On 21:14 Mon 18 Jul     , darkdefende@gmail.com wrote:
> However it didn't go as smoothly as I planned so this is still not 
> done. The main problem is that I need to know what the "switches" in 
> the configure script does so I can link them to useflags with correct 
> deps. If I just were to write a autoconf replacement in python there 
> wouldn't be much of a problem as I just have to "do" not try to guess 
> what it does.
> 
> Right now I have problems with figuring out how I should handle stuff in
> the autoconf file that checks for headers,libs or package checks.
> Granted, in most cases they are just there to see if you have the
> dependecy or if you have the correct version.
> So basicly you offen check if the correct dependecy is installed on the
> system and then give green light to build some extra features.
> 
> Because my ebuild generator should be able generate ebuilds without all
> the required dependencies this is a problem.
> I can't run the checks as they will fail as everything required to build
> the program might not be installed. And if I skip the checks there might
> be problems as I can't think of a way that I can guess what it will try
> to do.
> 
> Do you guy have any ideas how I should solve this problem? Or should I
> just "bindly" skip the checks and do what it would do if that check
> would have passed?

Let me see if I understand this. Autoconf files are basically a mixture 
of m4 and shell syntax. You've discovered that to get all the 
conditionals right, you need to write a shell parser, or somehow run the 
code rather than parsing it to get the result, or punt on the problem.

Here are some ways you could do those things:

Option 1. (Write a parser) See libbash. Not sure how it will work with 
the m4 mixed in, you might need to run autoconf and check the generated 
configure script instead.

Option 2a. (Run the code) Save code until you get to some variable/macro 
you recognize is important because it's used during the build (e.g. 
LIBS, *_LIBS), then run everything around it. It may also be useful to 
notice when files are mentioned and save a separate list of them. You 
might need to trap file access, e.g. using Alex's script for dependency 
discovery.

Option 3. (Punt) Pretend everything is mandatory. Don't handle USE flags 
at all, just put everything as a required dependency. Or, you could 
notice that a feature is optional and add a USE flag to build it; but 
say that the design of your tool doesn't allow for optional dependency 
detection so developers will need to split those out of the full 
dependency list.

-- 
Thanks,
Donnie

Donnie Berkholz
Admin, Summer of Code
Gentoo Linux and X.Org
Blog: http://dberkholz.com


[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-soc] Ebuild generator week 9 update

[darkdefende]

Donnie Berkholz writes:

> You've discovered that to get all the
> conditionals right, you need to write a shell parser, or somehow run the
> code rather than parsing it to get the result, or punt on the problem.
>
I've written (a so far) very simple parser to handle the if statments.
This does however take for granted that the if statements uses "test" to
do for returning True/False. I'm waiting a bit to flesh it out to see if
the approach I currently thinking about implementing works.

I'm thinking about a structure something like this:

I will parse the autoconf file and have all variables stored like this:
{"variable" : {"with_a" : "what ever 'variable' was defined as" , "!with_b"
: ...}}
(Note that with_a == with_a="yes" and !with_b == with_b="no")

So when the variable turns up in the automake files it will know what
switches can affect that variable and then see what extra files it pulls
in.

If the variable doesn't pull in files directly (I.E subdirs) I will just
scan the automake file in the new subdir(s) and the the files they pull in
to that switch/useflag.
I will store the information kinda like I do with includes:
{"useflag" : [["list of files it pulls in"], {"dict with useflags"}]}

The last dict will have the same structure as the first one so I cover
nested useflags.
Then I will then scan files for includes and link them to deps and use
them with the useflags in the ebuild.

The problem I have now is those checks in the autoconf files that checks
for files/libs on the system.
For example:

if test "$with_a"; then
    AC_CHECK_HEADERS([a.h])
    AC_CHECK_LIB([a],[new_awesome_call])
    if test "$ac_cv_header_aheader_h" = "yes"
    then
        dnl this works with every version of "a"
        dnl Add the basic stuff
        EXTRA_OBJS="$EXTRA_OBJS a_basic.o"
        if test "$ac_cv_lib_a_new_awesome_call" = "yes"
        then
            dnl goodies supported in the new version
            EXTRA_OBJS="$EXTRA_OBJS a_extra.o"
        fi
    fi
fi

I might check the header files against a database to see what package they
belong to and what version and do it that way.
However I have no idea what to do with "AC_CHECK_LIB" that "probes" the
lib with a function call to see if it's there.

> Here are some ways you could do those things:
>
> Option 1. (Write a parser) See libbash. Not sure how it will work with
> the m4 mixed in, you might need to run autoconf and check the generated
> configure script instead.
>
Most if the shell scripting seem to be kept as a fairly simple level in
the autoconf files. Granted to have really good support it should be able
to correctly parse all shell scripts.
But I hope I can get away with supporting the most common stuff inside if
and case statments.

> Option 2a. (Run the code) Save code until you get to some variable/macro
> you recognize is important because it's used during the build (e.g.
> LIBS, *_LIBS), then run everything around it. It may also be useful to
> notice when files are mentioned and save a separate list of them. You
> might need to trap file access, e.g. using Alex's script for dependency
> discovery.
>
Hmm. I don't really know if running the code is a good idea as stuff that
is not installed will be left out I think.
Take for example a program with different sound backends.
I use OSS4 because my sound card doesn't work with ALSA. So if I run the
code it would only detect optional deps for OSS4 and the ALSA checks will
fail in the autoconf file and not pull in anything.

> Option 3. (Punt) Pretend everything is mandatory. Don't handle USE flags
> at all, just put everything as a required dependency. Or, you could
> notice that a feature is optional and add a USE flag to build it; but
> say that the design of your tool doesn't allow for optional dependency
> detection so developers will need to split those out of the full
> dependency list.
>
Well I can definitely have useflags but don't link them to any deps as I
can check what can be switched on/off in the autoconf file and then guess
if they use any non standard assingments. I.E --with_lib="version2".

I would really like my generator guess what deps useflags pull in.
But if it's too troublesome I'll skip it for now so it can atleast
generate an ebuild that pulls in everything for autoconf projects.