Yet Another Shell

Reorder parser functions

Now functions are arranged in sections.

Refactor successive word parser

The simple command is now parsed by the new parse_simple_command_tokens
function. Successive words are now parsed by the new parse_words
function. The parse_assignments_and_redirects, parse_words_and_redirects
and parse_words_to_paren functions are removed in favor of them.

The is_command_delimiter_tokentype function is removed as it is no
longer used.

Remove unused functions in parser.c

The parse_word_to function is renamed to parse_word, which is one of the
functions removed in this commit.

Optimize candidate name check in alias substitution

This refactoring commit moves the
is_token_delimiter_char(buf->contents[j])
and
is_redir_fd(&buf->contents[i])
calls out of the substitute_alias_range function. These calls are
omitted in the psubstitute_alias function since the token has already
been delimited and categorized.

The psubstitute_alias function no longer uses the count_name_length
function to find the end of the token. Now the token passed from
psubstitute_alias to substitute_alias_range can contain non-alias-name
characters, but such a token will not be substituted anyway since no
alias can be defined with such a name.

The is_alias_name_char function is now defined with an internal linkage.

Simplify count_name_length function

Use parse_word_to directly

This refactoring removes the use of the parse_word function in favor of
the parse_word_to function.

Use maybe_line_continuations instead of ensure_buffer

The ensure_buffer function can read more than needed. The
maybe_line_continuations is preferred.

Restructure parser token error printing

The check_closing_token function is changed to the is_closing_tokentype
function. The get_errmsg_unexpected_token function is changed to the
get_errmsg_unexpected_tokentype function. The
print_errmsg_token_unexpected function is added.

The tryparse_redirect function is modified to keep the redundant error
message suppressed.

Simplify parse_while function

Simplify parse_group function

Simplify parse_compound_command function

A test case for $LINENO is modified as the parser no longer removes the
line continuation before saving the line number of the command.

Always reject unclosed arithmetic expansion

According to POSIX, it is a syntax error if the shell reaches the end of
input before determining whether an expansion starting with "$((" is a
valid arithmetic expansion. Previously, yash treated such a case as if
it was determined not being an arithmetic expansion, then re-parsed it
as a command substitution. In most cases, the result was an invalid
command substitution, which caused a syntax error anyway. However, there
are corner cases where the expansion can be re-parsed as a valid command
substitution. Yash now correctly rejects it.

Rewrite parser functions (token-based parsing)

One test case is set to be skipped as it reveals an existing bug.

Define wordunitfree function

Define is_command_delimiter_tokentype function

Define parse_newline_list function

Define next_token function

Define maybe_line_continuation function

Add next_index to parsestate_T

Update comments for parsestate_T members

Define reserved word identifying functions

Define token type

Define is_digits_only function

Rename is_literal_function_name to is_name_word

This function could be used to test any name word, not limited to a
function name.

Create branch token_based_parser

Trim blanks from alias-substituted word

The word parsed in the parse_word_as_wcs function is subject to alias
substitution. If the value of the alias substitution begins with a
blank, it should not be included in the final word.

More test on command substitution starting with $((

Test line continuations in expansions

Count line number correctly after rewinding index (#38586)

This commit adds two test cases, the second of which is commented out as
I am not going to fix it soon. Since line continuations are entirely
removed from `ps->src' on the first pass, we cannot re-increment the
line number when incrementing the index over the removed line
continuation on the second pass.

Count line number in multi-line arithmetic expansion (#38587)