Ruby
3.4.0dev (2024-11-23 revision b4d13fac3dd5420475aa1e14fdad8137da7e3ee0)
|
When we're lexing certain types (strings, symbols, lists, etc.) we have string content associated with the tokens. More...
Data Fields | |
pm_buffer_t | buffer |
The buffer that we're using to keep track of the string content. More... | |
const uint8_t * | cursor |
The cursor into the source string that points to how far we have currently copied into the buffer. More... | |
When we're lexing certain types (strings, symbols, lists, etc.) we have string content associated with the tokens.
For example:
"foo"
In this case, the string content is foo. Since there is no escaping, there's no need to track additional information and the token can be returned as normal. However, if we have escape sequences:
"foo\n"
then the bytes in the string are "f", "o", "o", "\", "n", but we want to provide our consumers with the string content "f", "o", "o", "\n". In these cases, when we find the first escape sequence, we initialize a pm_buffer_t to keep track of the string content. Then in the parser, it will automatically attach the string content to the node that it belongs to.
pm_buffer_t pm_token_buffer_t::buffer |
const uint8_t* pm_token_buffer_t::cursor |