[gentoo-soc] libbash Weekly Update

[gentoo-soc] libbash Weekly Update

[Nathan Eloe]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I've had a productive week overall, with lots of good progress towards
the final goal of this project.

As the grammar becomes finalized, a change must be made.  All of the
debugging tools are used for grammars that output Java code, but I want
a C/C++ library to handle everything.  So this meant the following things:
1) Grammar file had to work regardless of what the output language was.
2) I needed to know just how the C runtime worked for ANTLR.
3) Since I'm starting to write library code (not just a grammar in a
fancy IDE) I need to document my coding standards for any C/C++ code
written over the next half of the project.

And that indeed is what I've done.  I now have coding style guidelines
(which can be found at http://web.mst.edu/~nwe5g8/coding_standard.pdf,
though as changes are made to the .tex file in the repository, the
guaranteed up to date version of the guidelines can be generated from that).

Also, I've written a quick sample program that parses a single ebuild.
This is mostly a proof of concept, not a final product.  All I was
trying to do was get familiar with the C runtime and see how it worked.

In a previous email I wrote about "infinite loops" in the parser, or at
least loops that I was too impatient to wait to finish (5 minutes is too
long to wait for a parser to finish parsing something).  I found the
problem: I was attempting to recursively aggregate complex strings in
the parser, and the backtracking became too complex for it to handle.
So, I removed the aggregation step, instead just showing the individual
portions of the strings.  Aggregation will be handled by the C++
implementation (which has to do other things with variable expansions in
strings, etc).  So that was a major problem solved, and it tuned the
performance of the parser quite nicely.

The last thing I needed to do was get a language agnostic grammar.  C++
already has functions like exp, and bitor, bitxor, and bitand are C++
keywords.  The runtime doesn't do anything to fix/catch any of this.  As
such I had to go back and do that manually.  However, the changes I made
seemed to make it a grammar that was easier to read.


This week is mostly setup and input.  I'm writing a big bash script that
demonstrates all the features that the grammar supports, and will be
sending the resulting AST to the bash upstream mailing list and the
portage_devs mailing list to see if they have suggestions.  I'm also
going to scour man bash for missing features, set up the build system
for the libbash portion of the code (most likely cmake), Implement a
very simple bash builtin (echo), and finally implement a symbol table.
A busy week, but a week that should leave me in excellent shape for the
rest of the project.

Nate
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.14 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkxEczgACgkQFpoRlVgtqKbq2QCZAU3Fy549/3WTRc3X5i6xmhlt
o9AAoIOv4spJ3BkwFIlcRvdFhzqCjpKN
=eJON
-----END PGP SIGNATURE-----

Re: [gentoo-soc] libbash Weekly Update

[Luca Barbato]

On 07/19/2010 05:46 PM, Nathan Eloe wrote:
> I've had a productive week overall, with lots of good progress towards
> the final goal of this project.
> 
> As the grammar becomes finalized, a change must be made.  All of the
> debugging tools are used for grammars that output Java code, but I want
> a C/C++ library to handle everything.  So this meant the following things:
> 1) Grammar file had to work regardless of what the output language was.
> 2) I needed to know just how the C runtime worked for ANTLR.
> 3) Since I'm starting to write library code (not just a grammar in a
> fancy IDE) I need to document my coding standards for any C/C++ code
> written over the next half of the project.
> 
> And that indeed is what I've done.  I now have coding style guidelines
> (which can be found at http://web.mst.edu/~nwe5g8/coding_standard.pdf,
> though as changes are made to the .tex file in the repository, the
> guaranteed up to date version of the guidelines can be generated from that).
> 
> Also, I've written a quick sample program that parses a single ebuild.
> This is mostly a proof of concept, not a final product.  All I was
> trying to do was get familiar with the C runtime and see how it worked.
> 
> In a previous email I wrote about "infinite loops" in the parser, or at
> least loops that I was too impatient to wait to finish (5 minutes is too
> long to wait for a parser to finish parsing something).  I found the
> problem: I was attempting to recursively aggregate complex strings in
> the parser, and the backtracking became too complex for it to handle.
> So, I removed the aggregation step, instead just showing the individual
> portions of the strings.  Aggregation will be handled by the C++
> implementation (which has to do other things with variable expansions in
> strings, etc).  So that was a major problem solved, and it tuned the
> performance of the parser quite nicely.
> 
> The last thing I needed to do was get a language agnostic grammar.  C++
> already has functions like exp, and bitor, bitxor, and bitand are C++
> keywords.  The runtime doesn't do anything to fix/catch any of this.  As
> such I had to go back and do that manually.  However, the changes I made
> seemed to make it a grammar that was easier to read.
> 
> 
> This week is mostly setup and input.  I'm writing a big bash script that
> demonstrates all the features that the grammar supports, and will be
> sending the resulting AST to the bash upstream mailing list and the
> portage_devs mailing list to see if they have suggestions.  I'm also
> going to scour man bash for missing features, set up the build system
> for the libbash portion of the code (most likely cmake), Implement a
> very simple bash builtin (echo), and finally implement a symbol table.
> A busy week, but a week that should leave me in excellent shape for the
> rest of the project.

If it has to be used in portage please:

- use autotools or ask for help about using them.
- consider staying on plain C as much as possible, breakage in C++
software because of tiny differences in the stdc++ runtime are a
constant and portage must survive them.

lu

-- 

Luca Barbato
Gentoo/linux
http://dev.gentoo.org/~lu_zero

Re: [gentoo-soc] libbash Weekly Update

[Fabian Groffen]

On 20-07-2010 12:34:18 +0200, Luca Barbato wrote:
> If it has to be used in portage please:
> 
> - use autotools or ask for help about using them.
> - consider staying on plain C as much as possible, breakage in C++
> software because of tiny differences in the stdc++ runtime are a
> constant and portage must survive them.

A big +1.

I'm willing to help you with this if needed.


-- 
Fabian Groffen
Gentoo on a different level

Re: [gentoo-soc] libbash Weekly Update

[Nathan Eloe]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 7/20/10 5:40 AM, Fabian Groffen wrote:
> On 20-07-2010 12:34:18 +0200, Luca Barbato wrote:
>> If it has to be used in portage please:
>>
>> - use autotools or ask for help about using them.
>> - consider staying on plain C as much as possible, breakage in C++
>> software because of tiny differences in the stdc++ runtime are a
>> constant and portage must survive them.
> 
> A big +1.
> 
> I'm willing to help you with this if needed.
> 
> 

I'm fine with using autotools.  I hadn't made the final decision yet (I
was going to work on the build system tomorrow), so with this
recommendation I will use that instead of cmake.  I haven't used either
extensively (I know enough about both to do something very simple with
either), so changing the build system won't throw me off very much.

Staying with straight C is not something I am as comfortable with.  I've
talked with my mentor about it, and we've decided that since the GCC
does allow C++, it's a mature enough language to use.  While C and C++
are very similar, the differences would make it difficult to accomplish
(for me) as much as I would like to get done this summer while still
producing good work.

One of the things being worked on is keeping the grammar so it doesn't
conflict with any language.  Perhaps a straight C version could be
worked on in the future, but for right now, I would like to stay with C++.

Thanks for the input.  Any and all input I get at this stage is
appreciated and helpful.

Nathan
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.14 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkxFur4ACgkQFpoRlVgtqKaviACfa+BadGX/oyBUe0tY+UAAvIqN
fS0An3aZ5mhnRz9AzW6uPzVZbazv9Eb6
=NRoB
-----END PGP SIGNATURE-----

Re: [gentoo-soc] libbash Weekly Update

[Luca Barbato]

On 07/20/2010 05:03 PM, Nathan Eloe wrote:
> I'm fine with using autotools.  I hadn't made the final decision yet (I
> was going to work on the build system tomorrow), so with this
> recommendation I will use that instead of cmake.  I haven't used either
> extensively (I know enough about both to do something very simple with
> either), so changing the build system won't throw me off very much.
> 
> Staying with straight C is not something I am as comfortable with.  I've
> talked with my mentor about it, and we've decided that since the GCC
> does allow C++, it's a mature enough language to use.

The problem is that it isn't. the gcc move to C++ might straight the
things out in the long term. in the middle one I'd be quite wary of the
outcome.

lu

-- 

Luca Barbato
Gentoo/linux
http://dev.gentoo.org/~lu_zero

Re: [gentoo-soc] libbash Weekly Update

[Petteri Räty]

On 21.7.2010 13.34, Luca Barbato wrote:
> On 07/20/2010 05:03 PM, Nathan Eloe wrote:
>> I'm fine with using autotools.  I hadn't made the final decision yet (I
>> was going to work on the build system tomorrow), so with this
>> recommendation I will use that instead of cmake.  I haven't used either
>> extensively (I know enough about both to do something very simple with
>> either), so changing the build system won't throw me off very much.
>>
>> Staying with straight C is not something I am as comfortable with.  I've
>> talked with my mentor about it, and we've decided that since the GCC
>> does allow C++, it's a mature enough language to use.
> 
> The problem is that it isn't. the gcc move to C++ might straight the
> things out in the long term. in the middle one I'd be quite wary of the
> outcome.
> 

We prefer developer productivity over issues that might not even exist.
The project will not be mature enough to be used as the default by
Portage any time soon so the current way of doing this will stick
around. The current way could also be kept as a fallback for problematic
upgrades.

Regards,
Petteri

Re: [gentoo-soc] libbash Weekly Update

[Luca Barbato]

On 07/21/2010 12:51 PM, Petteri Räty wrote:
> On 21.7.2010 13.34, Luca Barbato wrote:
>> On 07/20/2010 05:03 PM, Nathan Eloe wrote:
>>> I'm fine with using autotools.  I hadn't made the final decision yet (I
>>> was going to work on the build system tomorrow), so with this
>>> recommendation I will use that instead of cmake.  I haven't used either
>>> extensively (I know enough about both to do something very simple with
>>> either), so changing the build system won't throw me off very much.
>>>
>>> Staying with straight C is not something I am as comfortable with.  I've
>>> talked with my mentor about it, and we've decided that since the GCC
>>> does allow C++, it's a mature enough language to use.
>>
>> The problem is that it isn't. the gcc move to C++ might straight the
>> things out in the long term. in the middle one I'd be quite wary of the
>> outcome.
>>
> 
> We prefer developer productivity over issues that might not even exist.

our bugzilla and gcc one has plenty of examples regarding how well gcc
behaves regarding C++, this[1] is one that got me today ^^;

Anyway it's not so related to the project at hand. Still I wonder why
not using a quick language (e.g. ruby, python) for this project if the
aim is to have a prototype. C and C++ are useful if you plan to optimize
a lot.

[1] http://bugs.gentoo.org/show_bug.cgi?id=262912




-- 

Luca Barbato
Gentoo/linux
http://dev.gentoo.org/~lu_zero

Re: [gentoo-soc] libbash Weekly Update

[Petteri Räty]

[-- Attachment #1: Type: text/plain, Size: 440 bytes --]

On 07/21/2010 06:59 PM, Luca Barbato wrote:

> 
> Anyway it's not so related to the project at hand. Still I wonder why
> not using a quick language (e.g. ruby, python) for this project if the
> aim is to have a prototype. C and C++ are useful if you plan to optimize
> a lot.
> 

Anltr has two mature backends as far as I know: C and Java. My guess is
that most of the people wouldn't approve the latter.

Regards,
Petteri


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]

Re: [gentoo-soc] libbash Weekly Update

[Fabian Groffen]

On 21-07-2010 19:41:19 +0300, Petteri Räty wrote:
> On 07/21/2010 06:59 PM, Luca Barbato wrote:
> > Anyway it's not so related to the project at hand. Still I wonder why
> > not using a quick language (e.g. ruby, python) for this project if the
> > aim is to have a prototype. C and C++ are useful if you plan to optimize
> > a lot.
> 
> Anltr has two mature backends as far as I know: C and Java. My guess is
> that most of the people wouldn't approve the latter.

If you're prototyping anyway, I prefer Java over Python or Ruby, since
Java is much more close to C in semantics.


-- 
Fabian Groffen
Gentoo on a different level

Re: [gentoo-soc] libbash Weekly Update

[Petteri Räty]

On 21.7.2010 19.55, Fabian Groffen wrote:
> On 21-07-2010 19:41:19 +0300, Petteri Räty wrote:
>> On 07/21/2010 06:59 PM, Luca Barbato wrote:
>>> Anyway it's not so related to the project at hand. Still I wonder why
>>> not using a quick language (e.g. ruby, python) for this project if the
>>> aim is to have a prototype. C and C++ are useful if you plan to optimize
>>> a lot.
>>
>> Anltr has two mature backends as far as I know: C and Java. My guess is
>> that most of the people wouldn't approve the latter.
> 
> If you're prototyping anyway, I prefer Java over Python or Ruby, since
> Java is much more close to C in semantics.
> 
> 

One of the goals of the second half of the project is to prototype how
the integration with package managers will work. Ideally we would like
the project to be usable by all package managers and that's kind of hard
to do for paludis with Ruby or Python.

Regards,
Petteri

[gentoo-soc] libbash Weekly update

[Nathan Eloe]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Things are proceeding very nicely on this end, though I did not make as
much progress as I would have liked last week.

As of Sunday, an build system (using autotools) has been created (though
all it does is make the changes to the grammar file that are required to
generate the antlr headers and generates them). Also, a class that wraps
the parser, lexer, and tree has been written with some basic functionality.

I have a system implemented for builtins, but I want to convert it to
use classes instead of the mess of function pointers I have now.  That
should make things cleaner and easier to use.

I sent emails to both the portage developers and bash upstream and got
some input from both.  Poring through man bash yielded even more small
missing features that I added to my backlog of work.

I never got around to implementing the symbol table, but that's on the
agenda for this week.

So, for this week it boils down to lots of little fixes (missing
features, things like that), implement the plan that came out of
communications with my mentor for the builtins, and add a symbol table.
After that, I'm going to start looking at making the antlr grammar
easier to read/understand (with comments, better names, etc).

Nathan
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.14 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkxN0q0ACgkQFpoRlVgtqKZeHACfZwFOYKYKKx5lcp1j65rG79i/
PwUAn09l3XmcZ4aXKISbfWPzlRlmWYDD
=9Ryd
-----END PGP SIGNATURE-----