|
Ruby
3.4.0dev (2024-12-06 revision 892c46283a5ea4179500d951c9d4866c0051f27b)
|
When lexing Ruby source, the lexer has a small amount of state to tell which kind of token it is currently lexing. More...
#include <parser.h>
Public Types | |
| enum | { PM_LEX_DEFAULT , PM_LEX_EMBEXPR , PM_LEX_EMBVAR , PM_LEX_HEREDOC , PM_LEX_LIST , PM_LEX_REGEXP , PM_LEX_STRING } |
| The type of this lex mode. More... | |
Data Fields | |
| enum pm_lex_mode:: { ... } | mode |
| The type of this lex mode. More... | |
| union { | |
| struct { | |
| size_t nesting | |
| This keeps track of the nesting level of the list. More... | |
| bool interpolation | |
| Whether or not interpolation is allowed in this list. More... | |
| uint8_t incrementor | |
| When lexing a list, it takes into account balancing the terminator if the terminator is one of (), [], {}, or <>. More... | |
| uint8_t terminator | |
| This is the terminator of the list literal. More... | |
| uint8_t breakpoints [11] | |
| This is the character set that should be used to delimit the tokens within the list. More... | |
| } list | |
| struct { | |
| size_t nesting | |
| This keeps track of the nesting level of the regular expression. More... | |
| uint8_t incrementor | |
| When lexing a regular expression, it takes into account balancing the terminator if the terminator is one of (), [], {}, or <>. More... | |
| uint8_t terminator | |
| This is the terminator of the regular expression. More... | |
| uint8_t breakpoints [7] | |
| This is the character set that should be used to delimit the tokens within the regular expression. More... | |
| } regexp | |
| struct { | |
| size_t nesting | |
| This keeps track of the nesting level of the string. More... | |
| bool interpolation | |
| Whether or not interpolation is allowed in this string. More... | |
| bool label_allowed | |
| Whether or not at the end of the string we should allow a :, which would indicate this was a dynamic symbol instead of a string. More... | |
| uint8_t incrementor | |
| When lexing a string, it takes into account balancing the terminator if the terminator is one of (), [], {}, or <>. More... | |
| uint8_t terminator | |
| This is the terminator of the string. More... | |
| uint8_t breakpoints [7] | |
| This is the character set that should be used to delimit the tokens within the string. More... | |
| } string | |
| struct { | |
| pm_heredoc_lex_mode_t base | |
| All of the data necessary to lex a heredoc. More... | |
| const uint8_t * next_start | |
| This is the pointer to the character where lexing should resume once the heredoc has been completely processed. More... | |
| size_t * common_whitespace | |
| This is used to track the amount of common whitespace on each line so that we know how much to dedent each line in the case of a tilde heredoc. More... | |
| bool line_continuation | |
| True if the previous token ended with a line continuation. More... | |
| } heredoc | |
| } | as |
| The data associated with this type of lex mode. More... | |
| struct pm_lex_mode * | prev |
| The previous lex state so that it knows how to pop. More... | |
When lexing Ruby source, the lexer has a small amount of state to tell which kind of token it is currently lexing.
For example, when we find the start of a string, the first token that we return is a TOKEN_STRING_BEGIN token. After that the lexer is now in the PM_LEX_STRING mode, and will return tokens that are found as part of a string.
| anonymous enum |
The type of this lex mode.
| union { ... } pm_lex_mode::as |
The data associated with this type of lex mode.
| pm_heredoc_lex_mode_t pm_lex_mode::base |
| uint8_t pm_lex_mode::breakpoints[7] |
This is the character set that should be used to delimit the tokens within the list.
This is the character set that should be used to delimit the tokens within the string.
This is the character set that should be used to delimit the tokens within the regular expression.
| size_t* pm_lex_mode::common_whitespace |
| uint8_t pm_lex_mode::incrementor |
When lexing a list, it takes into account balancing the terminator if the terminator is one of (), [], {}, or <>.
When lexing a string, it takes into account balancing the terminator if the terminator is one of (), [], {}, or <>.
When lexing a regular expression, it takes into account balancing the terminator if the terminator is one of (), [], {}, or <>.
| bool pm_lex_mode::interpolation |
| bool pm_lex_mode::label_allowed |
| bool pm_lex_mode::line_continuation |
| enum { ... } pm_lex_mode::mode |
The type of this lex mode.
| size_t pm_lex_mode::nesting |
| const uint8_t* pm_lex_mode::next_start |
| struct pm_lex_mode* pm_lex_mode::prev |
| uint8_t pm_lex_mode::terminator |
1.9.1