From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from pigeon.gentoo.org ([208.92.234.80] helo=lists.gentoo.org) by finch.gentoo.org with esmtp (Exim 4.60) (envelope-from ) id 1QaQ8A-0006Xl-98 for garchives@archives.gentoo.org; Sat, 25 Jun 2011 10:30:30 +0000 Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 016541C00F; Sat, 25 Jun 2011 10:30:17 +0000 (UTC) Received: from smtp.gentoo.org (smtp.gentoo.org [140.211.166.183]) by pigeon.gentoo.org (Postfix) with ESMTP id 8C7721C00F for ; Sat, 25 Jun 2011 10:30:17 +0000 (UTC) Received: from pelican.gentoo.org (unknown [66.219.59.40]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.gentoo.org (Postfix) with ESMTPS id C72331B4015 for ; Sat, 25 Jun 2011 10:30:16 +0000 (UTC) Received: from localhost.localdomain (localhost [127.0.0.1]) by pelican.gentoo.org (Postfix) with ESMTP id 1B7D48003C for ; Sat, 25 Jun 2011 10:30:16 +0000 (UTC) From: "Petteri Räty" To: gentoo-commits@lists.gentoo.org Content-type: text/plain; charset=UTF-8 Reply-To: gentoo-dev@lists.gentoo.org, "Petteri Räty" Message-ID: <6db29285fb16aea68a9d2527fe9cfd4e1918b247.betelgeuse@gentoo> Subject: [gentoo-commits] proj/libbash:master commit in: /, src/core/, test/, utils/, bashast/features_script/ X-VCS-Repository: proj/libbash X-VCS-Files: Makefile.am bashast/features_script/features.sh.walker.tokens src/core/bash_ast.cpp src/core/bash_ast.h test/ast_printer_test.sh utils/ast_printer.cpp X-VCS-Directories: / src/core/ test/ utils/ bashast/features_script/ X-VCS-Committer: betelgeuse X-VCS-Committer-Name: Petteri Räty X-VCS-Revision: 6db29285fb16aea68a9d2527fe9cfd4e1918b247 Date: Sat, 25 Jun 2011 10:30:16 +0000 (UTC) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-commits@lists.gentoo.org Content-Transfer-Encoding: quoted-printable X-Archives-Salt: X-Archives-Hash: 2d1eb83a150c5a0602956a66fec312af commit: 6db29285fb16aea68a9d2527fe9cfd4e1918b247 Author: Mu Qiao gentoo org> AuthorDate: Mon Jun 20 09:52:21 2011 +0000 Commit: Petteri R=C3=A4ty gentoo org> CommitDate: Wed Jun 22 11:52:27 2011 +0000 URL: http://git.overlays.gentoo.org/gitweb/?p=3Dproj/libbash.git;a= =3Dcommit;h=3D6db29285 Utility: support printing tokens received by walker It's helpful if ast_printer can print tokens received by walker grammar. Now this is supported with the -w option. --- Makefile.am | 1 + bashast/features_script/features.sh.walker.tokens | 2 + src/core/bash_ast.cpp | 22 +++++++++++- src/core/bash_ast.h | 4 ++- test/ast_printer_test.sh | 4 ++ utils/ast_printer.cpp | 40 +++++++++++++++= ++---- 6 files changed, 64 insertions(+), 9 deletions(-) diff --git a/Makefile.am b/Makefile.am index a706c43..ea13cdd 100644 --- a/Makefile.am +++ b/Makefile.am @@ -244,6 +244,7 @@ EXTRA_DIST =3D bashast/bashast.g \ bashast/features_script/features.sh \ bashast/features_script/features.sh.ast \ bashast/features_script/features.sh.tokens \ + bashast/features_script/features.sh.walker.tokens \ test/ast_printer_test.sh \ test/verify_bashs_test.sh \ test/verify_error_output_test.sh \ diff --git a/bashast/features_script/features.sh.walker.tokens b/bashast/= features_script/features.sh.walker.tokens new file mode 100644 index 0000000..673b88b --- /dev/null +++ b/bashast/features_script/features.sh.walker.tokens @@ -0,0 +1,2 @@ +LIST DOWN COMMAND DOWN FUNCTION DOWN STRING DOWN NAME UP CURRENT_SHELL D= OWN LIST DOWN COMMAND DOWN STRING DOWN NAME UP STRING DOWN DOUBLE_QUOTED_= STRING DOWN NAME BLANK NAME BLANK NAME BLANK NAME BLANK NAME BLANK NAME B= LANK NAME COLON BLANK VAR_REF DOWN NAME UP UP UP UP COMMAND DOWN STRING D= OWN NAME UP STRING DOWN SINGLE_QUOTED_STRING DOWN DOLLAR NAME UP UP UP CO= MMAND DOWN STRING DOWN NAME UP STRING DOWN VAR_REF DOWN NAME UP UP UP UP = UP UP UP COMMAND DOWN FUNCTION DOWN STRING DOWN NAME UP CURRENT_SHELL DOW= N LIST DOWN COMMAND DOWN COMPOUND_ARITH DOWN TIMES DOWN DIGIT DIGIT UP UP= UP COMMAND DOWN COMPOUND_ARITH DOWN EXP DOWN DIGIT DIGIT UP UP UP COMMAN= D DOWN COMPOUND_ARITH DOWN PLUS DOWN VAR_REF DOWN NAME UP SLASH DOWN DIGI= T DIGIT UP UP UP UP COMMAND DOWN COMPOUND_ARITH DOWN PLUS DOWN VAR_REF DO= WN LETTER UP MINUS_SIGN DOWN DIGIT UP UP UP UP UP UP UP UP COMMAND DOWN F= UNCTION DOWN STRING DOWN NAME UP SUBSHELL DOWN LIST DOWN COMMAND DOWN VAR= IABLE_DEFINITIONS DOWN EQUALS DOWN NAME A RRAY DOWN STRING DOWN LETTER UP STRING DOWN LETTER UP STRING DOWN LETTER= UP STRING DOWN LETTER UP UP UP UP UP COMMAND DOWN STRING DOWN NAME UP ST= RING DOWN VAR_REF DOWN NAME DOWN DIGIT UP UP UP UP COMMAND DOWN VARIABLE_= DEFINITIONS DOWN EQUALS DOWN NAME ARRAY DOWN STRING DOWN COMMAND_SUB DOWN= LIST DOWN COMMAND DOWN STRING DOWN NAME UP STRING DOWN DIGIT UP UP UP UP= UP STRING DOWN LETTER UP STRING DOWN LETTER UP STRING DOWN LETTER UP UP = UP UP UP COMMAND DOWN VARIABLE_DEFINITIONS DOWN EQUALS DOWN NAME DOWN VAR= _REF DOWN NAME UP UP STRING DOWN DIGIT UP UP UP UP COMMAND DOWN VARIABLE_= DEFINITIONS DOWN EQUALS DOWN NAME ARRAY DOWN STRING DOWN LETTER UP STRING= DOWN LETTER UP EQUALS DOWN DIGIT STRING DOWN LETTER UP UP UP UP UP UP UP= UP UP UP COMMAND DOWN STRING DOWN NAME UP STRING DOWN BRACE_EXP DOWN STR= ING DOWN LETTER UP STRING DOWN LETTER UP UP UP UP COMMAND DOWN STRING DOW= N NAME UP STRING DOWN BRACE_EXP DOWN DOTDOT DOWN LETTER LETTER UP UP UP U= P COMMAND DOWN STRING DOWN NAME UP STRING DOWN BRACE_EXP DOWN STRING DOWN BRACE_EXP DOWN STRING DOWN LETTER UP ST= RING DOWN LETTER UP UP UP STRING DOWN LETTER UP STRING DOWN LETTER UP UP = UP UP COMMAND DOWN STRING DOWN NAME UP STRING DOWN LETTER BRACE_EXP DOWN = STRING DOWN LETTER UP STRING DOWN LETTER UP UP UP UP COMMAND DOWN STRING = DOWN COMMAND_SUB DOWN LIST DOWN COMMAND DOWN STRING DOWN NAME UP STRING D= OWN NAME UP UP UP UP UP UP PIPE DOWN COMMAND DOWN STRING DOWN NAME UP UP = COMMAND DOWN STRING DOWN NAME UP STRING DOWN NAME UP REDIR DOWN OP STRING= DOWN NAME UP UP UP UP COMMAND DOWN CASE DOWN STRING DOWN COMMAND_SUB DOW= N LIST DOWN COMMAND DOWN STRING DOWN NAME UP STRING DOWN NAME UP UP UP UP= UP CASE_PATTERN DOWN STRING DOWN NAME UP CASE_COMMAND LIST DOWN COMMAND = DOWN STRING DOWN NAME UP STRING DOWN NAME UP UP UP UP CASE_PATTERN DOWN S= TRING DOWN NAME UP CASE_COMMAND LIST DOWN COMMAND DOWN STRING DOWN NAME U= P STRING DOWN NAME UP UP UP UP CASE_PATTERN DOWN STRING DOWN MATCH_ALL UP= CASE_COMMAND LIST DOWN COMMAND DOWN STRI NG DOWN NAME UP STRING DOWN NAME UP UP UP UP UP UP COMMAND DOWN FOR DOWN= NAME STRING DOWN COMMAND_SUB DOWN LIST DOWN PIPE DOWN COMMAND DOWN STRIN= G DOWN NAME UP UP COMMAND DOWN STRING DOWN NAME UP STRING DOWN NAME UP UP= UP UP UP UP LIST DOWN COMMAND DOWN STRING DOWN NAME UP STRING DOWN VAR_R= EF DOWN NAME UP UP UP COMMAND DOWN STRING DOWN NAME UP STRING DOWN NAME U= P UP UP UP UP COMMAND DOWN CFOR DOWN FOR_INIT DOWN PLUS DOWN DIGIT DIGIT = UP UP FOR_COND DOWN PLUS DOWN DIGIT DIGIT UP UP LIST DOWN COMMAND DOWN ST= RING DOWN NAME UP STRING DOWN NAME UP UP UP FOR_MOD DOWN PLUS DOWN DIGIT = DIGIT UP UP UP UP COMMAND DOWN SELECT DOWN NAME STRING DOWN COMMAND_SUB D= OWN LIST DOWN PIPE DOWN COMMAND DOWN STRING DOWN NAME UP UP COMMAND DOWN = STRING DOWN NAME UP STRING DOWN NAME UP UP UP UP UP UP LIST DOWN COMMAND = DOWN STRING DOWN NAME UP STRING DOWN NAME UP STRING DOWN DIGIT UP REDIR D= OWN GREATER_THAN STRING DOWN SLASH NAME SLASH NAME UP UP UP UP UP UP COMM= AND DOWN IF_STATEMENT DOWN IF DOWN LIST D OWN COMMAND DOWN STRING DOWN NAME UP STRING DOWN NAME UP UP UP LIST DOWN= COMMAND DOWN STRING DOWN NAME UP STRING DOWN NAME UP UP UP UP UP UP COMM= AND DOWN UNTIL DOWN LIST DOWN COMMAND DOWN COMPOUND_COND DOWN KEYWORD_TES= T DOWN LETTER DOWN STRING DOWN NAME SLASH NAME DOT NAME UP UP UP UP UP UP= LIST DOWN COMMAND DOWN STRING DOWN NAME UP STRING DOWN NAME SLASH NAME D= OT NAME UP UP UP UP UP COMMAND DOWN WHILE DOWN LIST DOWN COMMAND DOWN COM= POUND_COND DOWN BUILTIN_TEST DOWN LETTER DOWN STRING DOWN NAME UP UP UP U= P UP UP LIST DOWN COMMAND DOWN STRING DOWN NAME UP STRING DOWN DOUBLE_QUO= TED_STRING DOWN NAME BLANK NAME UP UP UP UP UP UP COMMAND DOWN IF_STATEME= NT DOWN IF DOWN LIST DOWN COMMAND DOWN COMPOUND_COND DOWN BUILTIN_TEST DO= WN NAME DOWN STRING DOWN DIGIT UP STRING DOWN DIGIT UP UP UP UP UP UP LIS= T DOWN COMMAND DOWN STRING DOWN NAME UP STRING DOWN DOUBLE_QUOTED_STRING = DOWN NAME SQUOTE LETTER BLANK NAME UP UP UP UP UP UP UP COMMAND DOWN STRI= NG DOWN NAME UP STRING DOWN NAME UP STRIN G DOWN NAME UP STRING DOWN NAME UP STRING DOWN NAME UP STRING DOWN NAME = UP UP COMMAND DOWN STRING DOWN NAME UP PROCESS_SUBSTITUTION DOWN LESS_THA= N LIST DOWN COMMAND DOWN STRING DOWN NAME UP STRING DOWN SLASH NAME SLASH= NAME SLASH NAME SLASH NAME DOT NAME UP UP UP UP UP LOGICOR DOWN LOGICAND= DOWN LOGICAND DOWN LOGICAND DOWN COMMAND DOWN STRING DOWN NAME UP STRING= DOWN NAME UP UP COMMAND DOWN STRING DOWN DOT SLASH NAME UP UP UP COMMAND= DOWN STRING DOWN NAME UP UP UP COMMAND DOWN STRING DOWN NAME UP UP UP CO= MMAND DOWN STRING DOWN NAME UP STRING DOWN NAME UP UP UP COMMAND DOWN STR= ING DOWN NAME UP STRING DOWN SLASH NAME SLASH NAME UP UP PIPE DOWN COMMAN= D DOWN STRING DOWN NAME UP STRING DOWN MINUS NAME UP UP COMMAND DOWN STRI= NG DOWN NAME UP STRING DOWN NAME UP UP UP COMMAND DOWN VARIABLE_DEFINITIO= NS DOWN EQUALS DOWN NAME STRING DOWN NAME UP UP UP UP COMMAND DOWN STRING= DOWN VAR_REF DOWN USE_DEFAULT_WHEN_UNSET_OR_NULL DOWN NAME STRING DOWN N= AME UP UP UP UP UP COMMAND DOWN STRING DO WN VAR_REF DOWN OFFSET DOWN NAME DIGIT UP UP UP UP COMMAND DOWN STRING D= OWN VAR_REF DOWN BANG DOWN NAME TIMES UP UP UP UP COMMAND DOWN STRING DOW= N VAR_REF DOWN BANG DOWN NAME AT UP UP UP UP COMMAND DOWN STRING DOWN VAR= _REF DOWN POUND DOWN NAME UP UP UP UP COMMAND DOWN STRING DOWN VAR_REF DO= WN REPLACE_FIRST DOWN NAME STRING DOWN NAME UP STRING DOWN NAME UP UP UP = UP UP COMMAND DOWN STRING DOWN VAR_REF DOWN LAZY_REMOVE_AT_START DOWN NAM= E STRING DOWN NAME UP UP UP UP UP COMMAND DOWN STRING DOWN VAR_REF DOWN R= EPLACE_AT_START DOWN NAME STRING DOWN NAME UP UP UP UP UP COMMAND DOWN ST= RING DOWN VAR_REF DOWN LAZY_REMOVE_AT_END DOWN NAME STRING DOWN NAME UP U= P UP UP UP COMMAND DOWN STRING DOWN VAR_REF DOWN LAZY_REMOVE_AT_END DOWN = NAME STRING DOWN NAME UP UP UP UP UP COMMAND DOWN STRING DOWN VAR_REF DOW= N DIGIT UP UP STRING DOWN VAR_REF DOWN AT UP UP STRING DOWN VAR_REF DOWN = TIMES UP UP UP COMMAND DOWN STRING DOWN VAR_REF DOWN QMARK UP UP UP COMMA= ND DOWN STRING DOWN VAR_REF DOWN REPLACE_ ALL DOWN NAME STRING DOWN DOT UP STRING DOWN UNDERSCORE UP UP UP UP UP C= OMMAND DOWN STRING DOWN VAR_REF DOWN REPLACE_AT_START DOWN NAME STRING DO= WN NAME UP STRING DOWN NAME UP UP UP UP UP COMMAND DOWN STRING DOWN VAR_R= EF DOWN REPLACE_AT_END DOWN NAME STRING DOWN NAME UP STRING DOWN NAME UP = UP UP UP UP COMMAND DOWN VARIABLE_DEFINITIONS DOWN EQUALS DOWN NAME STRIN= G DOWN VAR_REF DOWN REPLACE_FIRST DOWN NAME STRING DOWN NAME MINUS UP UP = UP UP UP UP UP PIPE DOWN COMMAND DOWN STRING DOWN NAME UP STRING DOWN NAM= E UP UP COMMAND DOWN STRING DOWN NAME UP STRING DOWN NAME UP STRING DOWN = DIGIT UP REDIR DOWN OP FILE_DESCRIPTOR DOWN DIGIT UP UP REDIR DOWN GREATE= R_THAN STRING DOWN SLASH NAME SLASH NAME UP UP UP UP COMMAND DOWN STRING = DOWN NAME UP STRING DOWN NAME UP REDIR DOWN OP STRING DOWN NAME UP UP UP = COMMAND DOWN STRING DOWN NAME UP STRING DOWN NAME UP HERE_STRING_OP DOWN = STRING DOWN NAME UP UP UP UP=20 + diff --git a/src/core/bash_ast.cpp b/src/core/bash_ast.cpp index 281f076..dc9501c 100644 --- a/src/core/bash_ast.cpp +++ b/src/core/bash_ast.cpp @@ -112,7 +112,7 @@ namespace } } =20 -std::string bash_ast::get_tokens(std::function token_map) +std::string bash_ast::get_parser_tokens(std::function token_map) { std::stringstream result; int line_counter =3D 1; @@ -147,6 +147,26 @@ std::string bash_ast::get_tokens(std::function token print_line_counter(result, token, line_counter, tokenName =3D=3D "= CONTINUE_LINE"? 1 : 0); } } + return result.str(); +} + +std::string bash_ast::get_walker_tokens(std::function token_map) +{ + std::stringstream result; + pANTLR3_INT_STREAM istream =3D nodes->tnstream->istream; + auto istream_size =3D istream->size(istream); + + for(ANTLR3_UINT32 i =3D 1; i <=3D istream_size; ++i) + { + ANTLR3_UINT32 token =3D istream->_LA(istream, boost::numeric_cast(i)); + if(token =3D=3D 2) + result << "DOWN "; + else if(token =3D=3D 3) + result << "UP "; + else + result << token_map(istream->_LA(istream, boost::numeric_cast(i))) << " "; + } + result << std::endl; =20 return result.str(); } diff --git a/src/core/bash_ast.h b/src/core/bash_ast.h index 63e35d1..ae6bb0f 100644 --- a/src/core/bash_ast.h +++ b/src/core/bash_ast.h @@ -105,7 +105,9 @@ public: =20 std::string get_string_tree(); =20 - std::string get_tokens(std::function); + std::string get_parser_tokens(std::function); + + std::string get_walker_tokens(std::function); }; =20 #endif diff --git a/test/ast_printer_test.sh b/test/ast_printer_test.sh index e0c930f..376b339 100755 --- a/test/ast_printer_test.sh +++ b/test/ast_printer_test.sh @@ -10,6 +10,10 @@ error+=3D$? | diff -u $srcdir/bashast/features_script/features.sh.tokens - error+=3D$? =20 +./ast_printer -w libbashWalker.tokens -f $srcdir/bashast/features_script= /features.sh\ + | diff -u $srcdir/bashast/features_script/features.sh.walker.tokens = - +error+=3D$? + ./ast_printer -f $srcdir/bashast/features_script/illegal_script.sh 2 > /= dev/null if [[ $? =3D=3D 0 ]] then diff --git a/utils/ast_printer.cpp b/utils/ast_printer.cpp index 6fa7c38..fa309c5 100644 --- a/utils/ast_printer.cpp +++ b/utils/ast_printer.cpp @@ -84,9 +84,9 @@ static bool build_token_map(std::unordered_map& token return qi::parse(first, last, line % qi::eol >> qi::eol, token_map) &&= first =3D=3D last; } =20 -static void print_token(std::istream& input, - const std::string& token_path, - bool silent) +static void print_parser_token(std::istream& input, + const std::string& token_path, + bool silent) { if(silent) return; @@ -95,9 +95,28 @@ static void print_token(std::istream& input, std::unordered_map token_map; =20 if(build_token_map(token_map, token_path)) - std::cout << ast.get_tokens(std::bind(&token_mapper, - token_map, - std::placeholders::_1)) + std::cout << ast.get_parser_tokens(std::bind(&token_mapper, + token_map, + std::placeholders::_1)) + << std::endl; + else + std::cerr << "Building token map failed" << std::endl; +} + +static void print_walker_token(std::istream& input, + const std::string& token_path, + bool silent) +{ + if(silent) + return; + + bash_ast ast(input); + std::unordered_map token_map; + + if(build_token_map(token_map, token_path)) + std::cout << ast.get_walker_tokens(std::bind(&token_mapper, + token_map, + std::placeholders::_1)) << std::endl; else std::cerr << "Building token map failed" << std::endl; @@ -144,6 +163,8 @@ int main(int argc, char** argv) ("dot,d", "print graphviz doc file instead of tree string if -s is n= ot specified") ("token,t", po::value(), "Print all tokens instead of A= ST. " "The argument is the path to l= ibbash.tokens") + ("walker-token,w", po::value(), "Print all tokens recei= ved by the walker. " + "The argument is the path to l= ibbashWalker.tokens") ("name,n", "When using files as input scripts, print out file names"= ) ("silent,s", "do not print any AST") ; @@ -160,10 +181,15 @@ int main(int argc, char** argv) =20 std::function printer; if(vm.count("token")) - printer =3D std::bind(&print_token, + printer =3D std::bind(&print_parser_token, std::placeholders::_1, vm["token"].as(), vm.count("silent")); + else if(vm.count("walker-token")) + printer =3D std::bind(&print_walker_token, + std::placeholders::_1, + vm["walker-token"].as(), + vm.count("silent")); else printer =3D std::bind(&print_ast, std::placeholders::_1,