3.1.1 Raw code

%code { actions | copyright_license } [spec | body | context | pre | post]... %{ <output language code> }%

Raw code declarations contain arbitrary code, copied verbatim into the output. The keywords following %code determine where the section is output.

Sometimes the generator cannot tell what context clauses for other packages are required in the actions package body; then you must use a %code declaration to add them.

3.1.2 Keywords

%keyword <name> <string>

example:

%keyword SEMICOLON ";"

“Keywords” are reserved words or symbols in the target language; the lexers recognize them by the given string.

The keyword is case insensisitive if the string delimiters are single quotes or the case_insensitive declaration is present.

3.1.3 Tokens

%token < kind > name regexp repair_image

example:

%token <symbol> IDENTIFIER %[ ... ]% "A_Bogus_Identifier"
%token <punctuation> TICK "'"

The syntax of the regular expression is determined by the lexer generator. For tree_sitter, it is a Javascript expression that evaluates to a regexp object; /foo/u or new RegExp ("foo", "u"). See https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions/Cheatsheet

repair_image is used in error repair information; it should be inserted by an editor at the place of the expected but missing token.

The meaning of the following values of kind are defined by the WisiToken generator. Other token kinds may be used for documentation; they just declare a token name and regular expression.

<string-double-one-line>

%token <string-double-one-line> STRING_LITERAL %[ ... ]%

A string of characters that have string syntax, with double quote delimiters, may not contain a new-line.

The regular expression is assumed to match such a string; this is not checked.

The restriction of not containing a new-line allows improving incremental parse when inserting/deleting string quotes; the text is affected only thru the following new-line. Without this restriction, when a string quote is inserted/deleted, the entire following text must be scanned by the lexer, and parsed.

If an embedded quote is escaped by doubling it (as for Ada strings), include the declaration %escape_delimiter_doubled <token_name>.

<string-single-one-line>

%token <string-single-one-line> CHARACTER_LITERAL %[ ... ]%

A string of characters that have string syntax, with single quote delimiters, may not contain a new-line.

The regular expression is assumed to match such a string; this is not checked.

<string-double>

%token <string-double> STRING_LITERAL %[ ... ]%

A string of characters that have string syntax, with double quote delimiters, may contain a newline.

The regular expression is assumed to match such a string; this is not checked.

<string-single>

%token <string-single> CHARACTER_LITERAL %[ ... ]%

A string of characters that have string syntax, with single quote delimiters, may contain a newline.

The regular expression is assumed to match such a string; this is not checked.

<new-line>

%token <new-line> NEW_LINE

Declares the non-grammar new-line token, used to count lines. It has no regexp argument; DOS and Unix line endings are added internally. This must be declared if line information is desired.

For backward compatibility with previous WisiToken versions, any regexp present is ignored.

<comment-new-line>

%token <comment-new-line> COMMENT "--"

Declares a non-grammar comment token that is terminated by a line end or end of input. The string argument must match only the comment start (the example shows the Ada comment start); DOS and Unix line endings, and end of input, are added internally. The token includes the line end; no separate new-line token is produced after the comment token.

<comment-one-line>

%token <comment-one-line> PLACEHOLDER "{" "}"

Declares a non-grammar comment token that is terminated by the end delimiter, may not contain a new-line. The two arguments are strings, and must match only the comment start and end (the example shows a template placeholder); DOS and Unix line endings, and end of input, are added internally. The delimiters must be different; this is checked at grammar generation time.

Does not handle nested delimiters; the token is terminated by the first end delimiter.

<delimited-text>

%token <delimited-text> RAW_CODE "%{" "}%"

A token that contains arbitrary text (including new-line), delimited by the two strings. The arguments provide the start and end delimiters - the rest of the regular expression is provided by the generator. The delimiters must be different; this is checked at grammar generation time.

Does not handle nested delimiters; the token is terminated by the first end delimiter.

<non-reporting>

%token <non-reporting> WHITESPACE %[ [ \t] ]%

A token that is recognized by the lexer, but not returned to the parser.

3.1.4 Error recovery

The parser uses an error recovery algorithm when it encounters a syntax error; if a solution is found, the parse continues.

Several grammar file declarations set parameters for the error recovery. If none of these parameters are present in the grammar file, the generated parser does not do error recovery.

The error recovery algorithm generates possible solutions based on the parse state preceding the error point, by inserting, deleting, or pushing back tokens. Each possible solution is given a cost, and enqueued to be checked later. Solutions are checked in cost order (lowest first).

%no_error_recover

States that there is no error recovery for this parser. This is the default if no error recover parameters are specified.

%mckenzie_check_limit <limit>

The number of tokens past the error point that must be parsed successfully for a solution to be deemed successful. Smaller values give faster recovery; larger values give better solutions. Too large a value risks encountering another user error, making a solution impossible. 3 or 4 works well in practice; default is 4.

mckenzie_check_delta_limit <limit>

When error recovery is entered with multiple parsers active, once a solution has been found for one parser, the other parsers are allowed to check only mckenzie_check_delta_limit possible solutions before they fail. This prevents long recovery times.

%mckenzie_zombie_limit <limit>

When a parser encounters an error, it is not terminated immediately; it becomes a zombie. The other parsers must advance zombie limit tokens past the error point without error before the zombie is terminated.

Smaller values give faster parsing, because parallel parsers are terminated sooner; in particular, in a language with conflicts that occur often, a small value gives the generalized parser a chance to get down to 1 parser, which is a significant gain in speed.

Larger values give better error solutions, because they may include the original intended code. For example, consider the following Ada code:

procedure Pattern_1 is new Ada.Containers.Indefinite_Doubly_Linked_Lists (Pattern);
procedure Pattern_2 is Ada : Integer; begin null; end;

Pattern_1 is a generic instantiation; Pattern_2 is a procedure body. There is a grammar conflict on “is”, so a second parser is spawned there, with one expecting to see a generic instantiation, the other a procedure body. If the “new” is missing, the generic instantiation parser errors on “Ada”, but the procedure body parser doesn’t error until “.” (a variable declaration cannot have “.” in the name). If mckenzie_zombie_limit is 1, the generic instantiation parser is terminated before error recover is started, so the correct solution (insert “new”) is not found.

Setting mckenzie_zombie_limit the same as mckenzie_check_limit works well in practice, unless it needs to be smaller; default is 4.

%mckenzie_cost_default <insert> <delete> <push back> <ignore check fail>

McKenzie error recovery default costs for insert, delete, push back single tokens, and for ignoring a semantic check failure; four floating point numbers.

“Push back” means undo parsing; remove tokens from the parse stack and put them back into the input stream. This moves the insert/delete point, allowing better solutions. The push back default cost is also the undo reduce default cost.

If not specified, costs are zero. Costs can be negative; they all add linearly.

%mckenzie_cost_delete <token> <cost>

McKenzie error recovery delete cost for a specific token.

%mckenzie_cost_fast_forward <cost>

McKenzie error recovery cost for parsing ahead after fixing one error, moving to the next error location.

%mckenzie_cost_insert <token> <cost>

McKenzie error recovery insert cost for a specific token.

%mckenzie_cost_fast_forward <cost>

McKenzie error recovery cost for using the matching_begin strategy.

%mckenzie_cost_push_back <token> <cost>

McKenzie error recovery push back cost for a specific token.

%mckenzie_cost_undo_reduce <token> <cost>

McKenzie error recovery undo reduce cost for a specific token.

%mckenzie_enqueue_limit <integer>

McKenzie error recovery limit on possible solutions enqueued (to be checked); default max integer.

%mckenzie_minimal_complete_cost_delta <cost>

McKenzie error recovery cost added to the cost of an inserted token when the insert is done by the minimal complete strategy; this cost is normally negative.

3.1.5 Other declarations

%case_insensitive

If present, keywords are case insensitive in the lexer.

%conflict <conflict description>

Declare a known conflict.

Example conflict declaration:

%conflict REDUCE abstract_limited_opt | REDUCE abstract_limited_synchronized_opt on token NEW

The conflict description is output by wisitoken-bnf-generate when an undeclared conflict is detected. If the user decides to not fix the conflict, the description can be copied into the grammar source file, so it will be ignored next time around. Or it can be converted to a %conflict_resolution; see the next item.

If a conflict has more than two branches, it must be declared more than once, first with two branches, then with one more, etc. This is due to the way conflicts are found during the parse table generation process.

When translating a wisitoken grammar to a tree-sitter grammar, it is sometimes necessary to express the conflicts differently. In that case, the syntax is:

%conflict abstract_limited_opt abstract_limited_synchronized_opt

which translates to one item in the conflicts list:

   [$.abstract_limited_opt, $.abstract_limited_synchronized_opt]

Resolving conflicts in the grammar can be difficult, but leaving them in can increase parse time and cause ambiguous parses.

In Emacs, wisitoken-parse_table-mode provides a command wisitoken-parse_table-conflict-goto that will find a conflict in the parse table file, which has more information that might help resolve the conflict. wisitoken-grammar-mode binds that command to ^c ..

%conflict_resolution <conflict description> : <resolution>

Declare a conflict and a resolution for it. The conflict description is the same as in a %conflict declaration; the resolution says which branch of the conflict to take.

Only one kind of resolution is supported: a token name, which must match one of the token names in the conflict description; the branch that contains that token is taken.

Example conflict resolution declaration:

%conflict REDUCE abstract_limited_opt
 | REDUCE abstract_limited_synchronized_opt on token NEW
 : abstract_limited_opt

This says to always reduce to abstract_limited_opt.

%elisp_face <name>

Declare a name for an elisp face constant.

When generating Ada code for Emacs, the elisp faces applied by wisi-face-apply actions must be declared, so the elisp and Ada code aggree on what they mean.

%elisp_indent <elisp name> <Ada name> [<arg_count> [token_arg_index]...]

Declare elisp and Ada names for an indent variable or function.

When generating Ada code for Emacs, names used in wisi-indent-action actions that are not recognized are assumed to be elisp and Ada variables, with the Ada name derived from elisp name by replacing - with _, and converting to Mixed_Case.

Indent variables that don’t meet that naming convention must be declared, so the elisp and Ada code agree on what they mean.

Custom indent functions are implemented in Ada; the elisp name is the name used in grammar file actions. The declaration includes the argument count and which arguments are token indicies. Token index arguments are converted to token labels if automatic labeling is in effect.

For example, Ada declares:

%elisp_indent "ada-indent-record*" Ada_Indent_Record_1  3 1 2

The name used in grammar file actions is ada-indent-record*. The Ada function name is Ada_Indent_Record_1; it must be visible in the generated code body, either by being declared in the language runtime package spec, or in a package made use-visible by a use-clause in a %code block. It takes three arguments; the first two are token indices.

%elisp_action <elisp name> <Ada name>

Declare elisp and Ada names for a custom action subprogram written in Ada.

The elisp name is used in grammar actions.

Note that custom Ada functions can also be declared by %elisp_indent; those must appear as an argument to a wisi-indent-action grammar action; elisp_action are grammar actions.

%end_names_optional_option <name>

When generating Ada code for Emacs, the name of the Ada variable determining whether end block names are optional.

In the Ada language, block names can be repeated at the end; for example:

Get_Inputs :
loop
...
end loop Get_Inputs;

These names are optional in the Ada standard. Making them required improves error recovery; the McKenzie recovery algorithm can use matching names to isolate the error.

%escape_delimiter_doubled <token_name>

The named token escapes embedded delimiters by doubling them, as for Ada strings. This is used by incremental parse when editing such tokens.

%generate <generate_algorithm> <output_language>

<generate_algorithm> is one of LALR | LR1 | Packrat_Gen | Packrat_Proc | External | Tree_Sitter.

<output_language> is one of Ada | Ada_Emacs.

The algorithm/output_language pair declares one output source set. Multiple sets can be declared; they are all generated together.

More detail on generate_algorithm:

LALR | LR1

Generates a parse table using the LALR or LR1 algorithms (see https://en.wikipedia.org/wiki/LR_parser). At runtime, an error correcting generalized LR parser uses the parse table to parse the input text.

An additional generate parameter text_rep determines how the parse table is represented; if present, it is in a text file that is loaded at parser run time. If absent, it is in the code. For very large parse tables, such as for an LR1 parser for a large language like Ada, the text representation may be needed, because the Ada compiler can’t handle the very large number of statements that represent the parser table in the code. The text file can take a long time to read at parser startup (a few seconds for the Ada language).

Packrat_Gen

Generates Ada code that implements a packrat parser. Left recursive grammar productions are not supported. See https://en.wikipedia.org/wiki/Parsing_expression_grammar.

Packrat_Proc

Generates Ada code that interprets the grammar using a packrat parser. Left recursive grammar productions are not supported. See https://en.wikipedia.org/wiki/Parsing_expression_grammar.

This uses the same parsing algorithm as Packrat_Gen; it is slower, but easier to debug.

External

Generates code that implements the grammar actions only, for use with a parser that is generated by an external program.

Tree_Sitter

Translates the grammar file to a tree-sitter grammar file, and generates code that impements the grammar actions.

The grammar should define a nonterminal word; see https://tree-sitter.github.io/tree-sitter/creating-parsers#keywords.

In addition, if keywords are case insensitive, they must all be explicitly declared.

<output_language> determines both what code is generated, and what language is used for the grammar actions. For Ada, the grammar action language is Ada, and it is copied verbatim into the generated grammar action code. For Emacs_Ada, the grammar action language is elisp, which is translated to Ada by assuming it will be used by the GNU ELPA package wisi (see https://elpa.gnu.org/packages/wisi.html).

When the output language is emacs_ada, an additional parameter is required: Process | Module. For Process, the generated code runs in an Emacs background process. For Module, the generated code runs as an Emacs loadable module (currently not supported).

%language_runtime <string>

Specify an alternate name for the language runtime package; the default is Wisi.<language_name>, where <language_name> is the simple name of the language grammar file, without the file extension. The value must be enclosed in quotes.

%lr1_hash_table_size <integer>

Specify the size of the hash table used when generating an LR1 parse table. The default size is 113; larger size can decrease generate time on larger languages, but only by 10%. A prime larger than the requested size is used.

%max_parallel <integer>

Maximum number of parallel parsers during main parsing. Default is 15; a language with many conflicts may need more.

%meta_syntax [BNF | EBNF]

Declares the syntax used by the grammar file. BNF is a minor extension of standard Backus Naur Form; EBNF is a large extension. The default is BNF.

%no_enum

By default, the generated Ada code includes an enumeration type declaring each token. This makes the language-specific runtime easier to write (without this type, tokens are identified by integers).

%no_enum causes the generated code to not include the token enumeration type.

%no_language_runtime

When generating Ada code for Emacs, %no_language_runtime causes the generated code to not include the runtime. Some grammars may need no runtime, particularly if they are small grammars intendend to test some generator feature.

%precedence <name> ...

Specifies a list of precedence names, and the order relation between them. All precedence names must be declared in a %precedence declaration before being used in a nonterminal.

%partial_recursion

The error recovery algorithm requires computing the recursion present in the language grammar. For some grammars (such as Java), this is too hard; %partial_recursion tells WisiToken to use a simpler approximate calculation. This will affect the quality of the error recovery, but it will still be robust.

%start <nontermininal>

The start token for the grammar.

%suppress <nontermininal> <warning label>

Suppress the indicated warning for the nonterminal.

<warning label> is a copy of part of the text of the warning message. The supported warnings are:

• may never match; it shares a prefix

%lexer_regexp <name> <value>

Declare a named regular expression (or fragment) with name and current lexer syntax. The name may then occur in another lexer regular expression.