* [gentoo-commits] proj/libbash:master commit in: /, src/core/, test/, utils/, bashast/features_script/
@ 2011-06-25 10:30 Petteri Räty
0 siblings, 0 replies; only message in thread
From: Petteri Räty @ 2011-06-25 10:30 UTC (permalink / raw
To: gentoo-commits
commit: 6db29285fb16aea68a9d2527fe9cfd4e1918b247
Author: Mu Qiao <qiaomuf <AT> gentoo <DOT> org>
AuthorDate: Mon Jun 20 09:52:21 2011 +0000
Commit: Petteri Räty <betelgeuse <AT> gentoo <DOT> org>
CommitDate: Wed Jun 22 11:52:27 2011 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/libbash.git;a=commit;h=6db29285
Utility: support printing tokens received by walker
It's helpful if ast_printer can print tokens received by walker grammar.
Now this is supported with the -w option.
---
Makefile.am | 1 +
bashast/features_script/features.sh.walker.tokens | 2 +
src/core/bash_ast.cpp | 22 +++++++++++-
src/core/bash_ast.h | 4 ++-
test/ast_printer_test.sh | 4 ++
utils/ast_printer.cpp | 40 +++++++++++++++++----
6 files changed, 64 insertions(+), 9 deletions(-)
diff --git a/Makefile.am b/Makefile.am
index a706c43..ea13cdd 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -244,6 +244,7 @@ EXTRA_DIST = bashast/bashast.g \
bashast/features_script/features.sh \
bashast/features_script/features.sh.ast \
bashast/features_script/features.sh.tokens \
+ bashast/features_script/features.sh.walker.tokens \
test/ast_printer_test.sh \
test/verify_bashs_test.sh \
test/verify_error_output_test.sh \
diff --git a/bashast/features_script/features.sh.walker.tokens b/bashast/features_script/features.sh.walker.tokens
new file mode 100644
index 0000000..673b88b
--- /dev/null
+++ b/bashast/features_script/features.sh.walker.tokens
@@ -0,0 +1,2 @@
+LIST DOWN COMMAND DOWN FUNCTION DOWN STRING DOWN NAME UP CURRENT_SHELL DOWN LIST DOWN COMMAND DOWN STRING DOWN NAME UP STRING DOWN DOUBLE_QUOTED_STRING DOWN NAME BLANK NAME BLANK NAME BLANK NAME BLANK NAME BLANK NAME BLANK NAME COLON BLANK VAR_REF DOWN NAME UP UP UP UP COMMAND DOWN STRING DOWN NAME UP STRING DOWN SINGLE_QUOTED_STRING DOWN DOLLAR NAME UP UP UP COMMAND DOWN STRING DOWN NAME UP STRING DOWN VAR_REF DOWN NAME UP UP UP UP UP UP UP COMMAND DOWN FUNCTION DOWN STRING DOWN NAME UP CURRENT_SHELL DOWN LIST DOWN COMMAND DOWN COMPOUND_ARITH DOWN TIMES DOWN DIGIT DIGIT UP UP UP COMMAND DOWN COMPOUND_ARITH DOWN EXP DOWN DIGIT DIGIT UP UP UP COMMAND DOWN COMPOUND_ARITH DOWN PLUS DOWN VAR_REF DOWN NAME UP SLASH DOWN DIGIT DIGIT UP UP UP UP COMMAND DOWN COMPOUND_ARITH DOWN PLUS DOWN VAR_REF DOWN LETTER UP MINUS_SIGN DOWN DIGIT UP UP UP UP UP UP UP UP COMMAND DOWN FUNCTION DOWN STRING DOWN NAME UP SUBSHELL DOWN LIST DOWN COMMAND DOWN VARIABLE_DEFINITIONS DOWN EQUALS DOWN NAME A
RRAY DOWN STRING DOWN LETTER UP STRING DOWN LETTER UP STRING DOWN LETTER UP STRING DOWN LETTER UP UP UP UP UP COMMAND DOWN STRING DOWN NAME UP STRING DOWN VAR_REF DOWN NAME DOWN DIGIT UP UP UP UP COMMAND DOWN VARIABLE_DEFINITIONS DOWN EQUALS DOWN NAME ARRAY DOWN STRING DOWN COMMAND_SUB DOWN LIST DOWN COMMAND DOWN STRING DOWN NAME UP STRING DOWN DIGIT UP UP UP UP UP STRING DOWN LETTER UP STRING DOWN LETTER UP STRING DOWN LETTER UP UP UP UP UP COMMAND DOWN VARIABLE_DEFINITIONS DOWN EQUALS DOWN NAME DOWN VAR_REF DOWN NAME UP UP STRING DOWN DIGIT UP UP UP UP COMMAND DOWN VARIABLE_DEFINITIONS DOWN EQUALS DOWN NAME ARRAY DOWN STRING DOWN LETTER UP STRING DOWN LETTER UP EQUALS DOWN DIGIT STRING DOWN LETTER UP UP UP UP UP UP UP UP UP UP COMMAND DOWN STRING DOWN NAME UP STRING DOWN BRACE_EXP DOWN STRING DOWN LETTER UP STRING DOWN LETTER UP UP UP UP COMMAND DOWN STRING DOWN NAME UP STRING DOWN BRACE_EXP DOWN DOTDOT DOWN LETTER LETTER UP UP UP UP COMMAND DOWN STRING DOWN NAME UP STRING
DOWN BRACE_EXP DOWN STRING DOWN BRACE_EXP DOWN STRING DOWN LETTER UP STRING DOWN LETTER UP UP UP STRING DOWN LETTER UP STRING DOWN LETTER UP UP UP UP COMMAND DOWN STRING DOWN NAME UP STRING DOWN LETTER BRACE_EXP DOWN STRING DOWN LETTER UP STRING DOWN LETTER UP UP UP UP COMMAND DOWN STRING DOWN COMMAND_SUB DOWN LIST DOWN COMMAND DOWN STRING DOWN NAME UP STRING DOWN NAME UP UP UP UP UP UP PIPE DOWN COMMAND DOWN STRING DOWN NAME UP UP COMMAND DOWN STRING DOWN NAME UP STRING DOWN NAME UP REDIR DOWN OP STRING DOWN NAME UP UP UP UP COMMAND DOWN CASE DOWN STRING DOWN COMMAND_SUB DOWN LIST DOWN COMMAND DOWN STRING DOWN NAME UP STRING DOWN NAME UP UP UP UP UP CASE_PATTERN DOWN STRING DOWN NAME UP CASE_COMMAND LIST DOWN COMMAND DOWN STRING DOWN NAME UP STRING DOWN NAME UP UP UP UP CASE_PATTERN DOWN STRING DOWN NAME UP CASE_COMMAND LIST DOWN COMMAND DOWN STRING DOWN NAME UP STRING DOWN NAME UP UP UP UP CASE_PATTERN DOWN STRING DOWN MATCH_ALL UP CASE_COMMAND LIST DOWN COMMAND DOWN STRI
NG DOWN NAME UP STRING DOWN NAME UP UP UP UP UP UP COMMAND DOWN FOR DOWN NAME STRING DOWN COMMAND_SUB DOWN LIST DOWN PIPE DOWN COMMAND DOWN STRING DOWN NAME UP UP COMMAND DOWN STRING DOWN NAME UP STRING DOWN NAME UP UP UP UP UP UP LIST DOWN COMMAND DOWN STRING DOWN NAME UP STRING DOWN VAR_REF DOWN NAME UP UP UP COMMAND DOWN STRING DOWN NAME UP STRING DOWN NAME UP UP UP UP UP COMMAND DOWN CFOR DOWN FOR_INIT DOWN PLUS DOWN DIGIT DIGIT UP UP FOR_COND DOWN PLUS DOWN DIGIT DIGIT UP UP LIST DOWN COMMAND DOWN STRING DOWN NAME UP STRING DOWN NAME UP UP UP FOR_MOD DOWN PLUS DOWN DIGIT DIGIT UP UP UP UP COMMAND DOWN SELECT DOWN NAME STRING DOWN COMMAND_SUB DOWN LIST DOWN PIPE DOWN COMMAND DOWN STRING DOWN NAME UP UP COMMAND DOWN STRING DOWN NAME UP STRING DOWN NAME UP UP UP UP UP UP LIST DOWN COMMAND DOWN STRING DOWN NAME UP STRING DOWN NAME UP STRING DOWN DIGIT UP REDIR DOWN GREATER_THAN STRING DOWN SLASH NAME SLASH NAME UP UP UP UP UP UP COMMAND DOWN IF_STATEMENT DOWN IF DOWN LIST D
OWN COMMAND DOWN STRING DOWN NAME UP STRING DOWN NAME UP UP UP LIST DOWN COMMAND DOWN STRING DOWN NAME UP STRING DOWN NAME UP UP UP UP UP UP COMMAND DOWN UNTIL DOWN LIST DOWN COMMAND DOWN COMPOUND_COND DOWN KEYWORD_TEST DOWN LETTER DOWN STRING DOWN NAME SLASH NAME DOT NAME UP UP UP UP UP UP LIST DOWN COMMAND DOWN STRING DOWN NAME UP STRING DOWN NAME SLASH NAME DOT NAME UP UP UP UP UP COMMAND DOWN WHILE DOWN LIST DOWN COMMAND DOWN COMPOUND_COND DOWN BUILTIN_TEST DOWN LETTER DOWN STRING DOWN NAME UP UP UP UP UP UP LIST DOWN COMMAND DOWN STRING DOWN NAME UP STRING DOWN DOUBLE_QUOTED_STRING DOWN NAME BLANK NAME UP UP UP UP UP UP COMMAND DOWN IF_STATEMENT DOWN IF DOWN LIST DOWN COMMAND DOWN COMPOUND_COND DOWN BUILTIN_TEST DOWN NAME DOWN STRING DOWN DIGIT UP STRING DOWN DIGIT UP UP UP UP UP UP LIST DOWN COMMAND DOWN STRING DOWN NAME UP STRING DOWN DOUBLE_QUOTED_STRING DOWN NAME SQUOTE LETTER BLANK NAME UP UP UP UP UP UP UP COMMAND DOWN STRING DOWN NAME UP STRING DOWN NAME UP STRIN
G DOWN NAME UP STRING DOWN NAME UP STRING DOWN NAME UP STRING DOWN NAME UP UP COMMAND DOWN STRING DOWN NAME UP PROCESS_SUBSTITUTION DOWN LESS_THAN LIST DOWN COMMAND DOWN STRING DOWN NAME UP STRING DOWN SLASH NAME SLASH NAME SLASH NAME SLASH NAME DOT NAME UP UP UP UP UP LOGICOR DOWN LOGICAND DOWN LOGICAND DOWN LOGICAND DOWN COMMAND DOWN STRING DOWN NAME UP STRING DOWN NAME UP UP COMMAND DOWN STRING DOWN DOT SLASH NAME UP UP UP COMMAND DOWN STRING DOWN NAME UP UP UP COMMAND DOWN STRING DOWN NAME UP UP UP COMMAND DOWN STRING DOWN NAME UP STRING DOWN NAME UP UP UP COMMAND DOWN STRING DOWN NAME UP STRING DOWN SLASH NAME SLASH NAME UP UP PIPE DOWN COMMAND DOWN STRING DOWN NAME UP STRING DOWN MINUS NAME UP UP COMMAND DOWN STRING DOWN NAME UP STRING DOWN NAME UP UP UP COMMAND DOWN VARIABLE_DEFINITIONS DOWN EQUALS DOWN NAME STRING DOWN NAME UP UP UP UP COMMAND DOWN STRING DOWN VAR_REF DOWN USE_DEFAULT_WHEN_UNSET_OR_NULL DOWN NAME STRING DOWN NAME UP UP UP UP UP COMMAND DOWN STRING DO
WN VAR_REF DOWN OFFSET DOWN NAME DIGIT UP UP UP UP COMMAND DOWN STRING DOWN VAR_REF DOWN BANG DOWN NAME TIMES UP UP UP UP COMMAND DOWN STRING DOWN VAR_REF DOWN BANG DOWN NAME AT UP UP UP UP COMMAND DOWN STRING DOWN VAR_REF DOWN POUND DOWN NAME UP UP UP UP COMMAND DOWN STRING DOWN VAR_REF DOWN REPLACE_FIRST DOWN NAME STRING DOWN NAME UP STRING DOWN NAME UP UP UP UP UP COMMAND DOWN STRING DOWN VAR_REF DOWN LAZY_REMOVE_AT_START DOWN NAME STRING DOWN NAME UP UP UP UP UP COMMAND DOWN STRING DOWN VAR_REF DOWN REPLACE_AT_START DOWN NAME STRING DOWN NAME UP UP UP UP UP COMMAND DOWN STRING DOWN VAR_REF DOWN LAZY_REMOVE_AT_END DOWN NAME STRING DOWN NAME UP UP UP UP UP COMMAND DOWN STRING DOWN VAR_REF DOWN LAZY_REMOVE_AT_END DOWN NAME STRING DOWN NAME UP UP UP UP UP COMMAND DOWN STRING DOWN VAR_REF DOWN DIGIT UP UP STRING DOWN VAR_REF DOWN AT UP UP STRING DOWN VAR_REF DOWN TIMES UP UP UP COMMAND DOWN STRING DOWN VAR_REF DOWN QMARK UP UP UP COMMAND DOWN STRING DOWN VAR_REF DOWN REPLACE_
ALL DOWN NAME STRING DOWN DOT UP STRING DOWN UNDERSCORE UP UP UP UP UP COMMAND DOWN STRING DOWN VAR_REF DOWN REPLACE_AT_START DOWN NAME STRING DOWN NAME UP STRING DOWN NAME UP UP UP UP UP COMMAND DOWN STRING DOWN VAR_REF DOWN REPLACE_AT_END DOWN NAME STRING DOWN NAME UP STRING DOWN NAME UP UP UP UP UP COMMAND DOWN VARIABLE_DEFINITIONS DOWN EQUALS DOWN NAME STRING DOWN VAR_REF DOWN REPLACE_FIRST DOWN NAME STRING DOWN NAME MINUS UP UP UP UP UP UP UP PIPE DOWN COMMAND DOWN STRING DOWN NAME UP STRING DOWN NAME UP UP COMMAND DOWN STRING DOWN NAME UP STRING DOWN NAME UP STRING DOWN DIGIT UP REDIR DOWN OP FILE_DESCRIPTOR DOWN DIGIT UP UP REDIR DOWN GREATER_THAN STRING DOWN SLASH NAME SLASH NAME UP UP UP UP COMMAND DOWN STRING DOWN NAME UP STRING DOWN NAME UP REDIR DOWN OP STRING DOWN NAME UP UP UP COMMAND DOWN STRING DOWN NAME UP STRING DOWN NAME UP HERE_STRING_OP DOWN STRING DOWN NAME UP UP UP UP
+
diff --git a/src/core/bash_ast.cpp b/src/core/bash_ast.cpp
index 281f076..dc9501c 100644
--- a/src/core/bash_ast.cpp
+++ b/src/core/bash_ast.cpp
@@ -112,7 +112,7 @@ namespace
}
}
-std::string bash_ast::get_tokens(std::function<std::string(ANTLR3_UINT32)> token_map)
+std::string bash_ast::get_parser_tokens(std::function<std::string(ANTLR3_UINT32)> token_map)
{
std::stringstream result;
int line_counter = 1;
@@ -147,6 +147,26 @@ std::string bash_ast::get_tokens(std::function<std::string(ANTLR3_UINT32)> token
print_line_counter(result, token, line_counter, tokenName == "CONTINUE_LINE"? 1 : 0);
}
}
+ return result.str();
+}
+
+std::string bash_ast::get_walker_tokens(std::function<std::string(ANTLR3_UINT32)> token_map)
+{
+ std::stringstream result;
+ pANTLR3_INT_STREAM istream = nodes->tnstream->istream;
+ auto istream_size = istream->size(istream);
+
+ for(ANTLR3_UINT32 i = 1; i <= istream_size; ++i)
+ {
+ ANTLR3_UINT32 token = istream->_LA(istream, boost::numeric_cast<ANTLR3_INT32>(i));
+ if(token == 2)
+ result << "DOWN ";
+ else if(token == 3)
+ result << "UP ";
+ else
+ result << token_map(istream->_LA(istream, boost::numeric_cast<ANTLR3_INT32>(i))) << " ";
+ }
+ result << std::endl;
return result.str();
}
diff --git a/src/core/bash_ast.h b/src/core/bash_ast.h
index 63e35d1..ae6bb0f 100644
--- a/src/core/bash_ast.h
+++ b/src/core/bash_ast.h
@@ -105,7 +105,9 @@ public:
std::string get_string_tree();
- std::string get_tokens(std::function<std::string(ANTLR3_UINT32)>);
+ std::string get_parser_tokens(std::function<std::string(ANTLR3_UINT32)>);
+
+ std::string get_walker_tokens(std::function<std::string(ANTLR3_UINT32)>);
};
#endif
diff --git a/test/ast_printer_test.sh b/test/ast_printer_test.sh
index e0c930f..376b339 100755
--- a/test/ast_printer_test.sh
+++ b/test/ast_printer_test.sh
@@ -10,6 +10,10 @@ error+=$?
| diff -u $srcdir/bashast/features_script/features.sh.tokens -
error+=$?
+./ast_printer -w libbashWalker.tokens -f $srcdir/bashast/features_script/features.sh\
+ | diff -u $srcdir/bashast/features_script/features.sh.walker.tokens -
+error+=$?
+
./ast_printer -f $srcdir/bashast/features_script/illegal_script.sh 2 > /dev/null
if [[ $? == 0 ]]
then
diff --git a/utils/ast_printer.cpp b/utils/ast_printer.cpp
index 6fa7c38..fa309c5 100644
--- a/utils/ast_printer.cpp
+++ b/utils/ast_printer.cpp
@@ -84,9 +84,9 @@ static bool build_token_map(std::unordered_map<ANTLR3_INT32, std::string>& token
return qi::parse(first, last, line % qi::eol >> qi::eol, token_map) && first == last;
}
-static void print_token(std::istream& input,
- const std::string& token_path,
- bool silent)
+static void print_parser_token(std::istream& input,
+ const std::string& token_path,
+ bool silent)
{
if(silent)
return;
@@ -95,9 +95,28 @@ static void print_token(std::istream& input,
std::unordered_map<ANTLR3_INT32, std::string> token_map;
if(build_token_map(token_map, token_path))
- std::cout << ast.get_tokens(std::bind(&token_mapper,
- token_map,
- std::placeholders::_1))
+ std::cout << ast.get_parser_tokens(std::bind(&token_mapper,
+ token_map,
+ std::placeholders::_1))
+ << std::endl;
+ else
+ std::cerr << "Building token map failed" << std::endl;
+}
+
+static void print_walker_token(std::istream& input,
+ const std::string& token_path,
+ bool silent)
+{
+ if(silent)
+ return;
+
+ bash_ast ast(input);
+ std::unordered_map<ANTLR3_INT32, std::string> token_map;
+
+ if(build_token_map(token_map, token_path))
+ std::cout << ast.get_walker_tokens(std::bind(&token_mapper,
+ token_map,
+ std::placeholders::_1))
<< std::endl;
else
std::cerr << "Building token map failed" << std::endl;
@@ -144,6 +163,8 @@ int main(int argc, char** argv)
("dot,d", "print graphviz doc file instead of tree string if -s is not specified")
("token,t", po::value<std::string>(), "Print all tokens instead of AST. "
"The argument is the path to libbash.tokens")
+ ("walker-token,w", po::value<std::string>(), "Print all tokens received by the walker. "
+ "The argument is the path to libbashWalker.tokens")
("name,n", "When using files as input scripts, print out file names")
("silent,s", "do not print any AST")
;
@@ -160,10 +181,15 @@ int main(int argc, char** argv)
std::function<void(std::istream&)> printer;
if(vm.count("token"))
- printer = std::bind(&print_token,
+ printer = std::bind(&print_parser_token,
std::placeholders::_1,
vm["token"].as<std::string>(),
vm.count("silent"));
+ else if(vm.count("walker-token"))
+ printer = std::bind(&print_walker_token,
+ std::placeholders::_1,
+ vm["walker-token"].as<std::string>(),
+ vm.count("silent"));
else
printer = std::bind(&print_ast,
std::placeholders::_1,
^ permalink raw reply related [flat|nested] only message in thread
only message in thread, other threads:[~2011-06-25 10:30 UTC | newest]
Thread overview: (only message) (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-06-25 10:30 [gentoo-commits] proj/libbash:master commit in: /, src/core/, test/, utils/, bashast/features_script/ Petteri Räty
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox