Adds documentation for TokenStream's chars and skip_whitespace, for
OffsetStr, in particular describing why do we need that structure, and
can't just use Rust's builtin &str, for Offset::from, explaining the
unsafety behind calling this function, and describing how unsafe are
implementations of Display and Debug for OffsetStr.
It was never supposed to be part of a public interface of TokenStream,
but it got added during debugging lifetimes issue. This commit finally
reverts it back to what it was supposed to be.
Now it also handles expression such as 11 * 13, 11 + 13 * 17, and even
11 + 13 * 17 + 19. We also introduced new function, is_binary_operator,
which allows for simpler checking of operators in the expression parser.
We fixed the issue that parser didn't parse its input exhaustively, i.e.
when given '+17 + 23 + +21' it would return only +17 + 23, which is far
from what we would expect.
Now when calling peek, TokenStream will remember the token it parses and
store it in a field. Then, when calling next, it looks at that field and
if it's not empty, it returns it instead. This allows us to avoid having
to parse same token in the input two times, which ultimately results in
better performance.
Now we can also parse unary +, i.e.
+17,
and also expressions that contain it, i.e.
+17 + 23.
We also now have custom Debug implementation for Expression, which
prints them in more useful, s-expression-like syntax, i.e.
+17 + 23 => (+ +17 23).
We also change implementation of `TokenStream`s `next` and `parse_next`
methods to allow to easily implement `peek` method for looking at the
(possible) next token without advancing the stream.
Adds `--unpretty` unstable flag, which currently accepts two arguments,
`dot` and `graphdotviz`, which are synonymous, and which prints the
expression we parsed in a graphdotviz-compatible format, i.e. in a form
of a directed graph.
Currently can only parse addition in a right-associative way, i.e.
1 + 2 + 3 => 1 + (2 + 3).
The reason why are we using raw pointers in a form of OffsetStr struct
is because I don't think there is a way to prove to Rust compiler that
tokens the parse_next function returns are always valid references to
the TokenStream source string. Therefore we simply have to bypass the
borrow checker and handle this ourselves.
Aodhnait Étaín