pm_token_buffer_t Struct Reference

When we're lexing certain types (strings, symbols, lists, etc.) we have string content associated with the tokens. More...

Data Fields
pm_buffer_t	buffer
	The buffer that we're using to keep track of the string content. More...
const uint8_t *	cursor
	The cursor into the source string that points to how far we have currently copied into the buffer. More...

Detailed Description

When we're lexing certain types (strings, symbols, lists, etc.) we have string content associated with the tokens.

For example:

"foo"

In this case, the string content is foo. Since there is no escaping, there's no need to track additional information and the token can be returned as normal. However, if we have escape sequences:

"foo\n"

then the bytes in the string are "f", "o", "o", "\", "n", but we want to provide our consumers with the string content "f", "o", "o", "\n". In these cases, when we find the first escape sequence, we initialize a pm_buffer_t to keep track of the string content. Then in the parser, it will automatically attach the string content to the node that it belongs to.

Definition at line 10345 of file prism.c.

Field Documentation

buffer

pm_buffer_t pm_token_buffer_t::buffer

The buffer that we're using to keep track of the string content.

It will only be initialized if we receive an escape sequence.

Definition at line 10350 of file prism.c.

cursor

const uint8_t* pm_token_buffer_t::cursor

The cursor into the source string that points to how far we have currently copied into the buffer.

Definition at line 10356 of file prism.c.

---

The documentation for this struct was generated from the following file:

• prism/prism.c (892c46283a5ea4179500d951c9d4866c0051f27b)