module RDoc::TokenStream
A TokenStream is a list of tokens, gathered during the parse of some entity (say a method). Entities populate these streams by being registered with the lexer. Any class can collect tokens by including TokenStream. From the outside, you use such an object by calling the #start_collecting_tokens method, followed by calls to #add_token and pop_token.
Public Class Methods
Converts token_stream
to HTML wrapping various tokens with
<span>
elements. The following tokens types are wrapped
in spans with the given class names:
- TkCONSTANT
-
'ruby-constant'
- TkKW
-
'ruby-keyword'
- TkIVAR
-
'ruby-ivar'
- TkOp
-
'ruby-operator'
- TkId
-
'ruby-identifier'
- TkNode
-
'ruby-node'
- TkCOMMENT
-
'ruby-comment'
- TkREGEXP
-
'ruby-regexp'
- TkSTRING
-
'ruby-string'
- TkVal
-
'ruby-value'
Other token types are not wrapped in spans.
# File lib/rdoc/token_stream.rb, line 28 def self.to_html token_stream token_stream.map do |t| next unless t style = case t when RDoc::RubyToken::TkCONSTANT then 'ruby-constant' when RDoc::RubyToken::TkKW then 'ruby-keyword' when RDoc::RubyToken::TkIVAR then 'ruby-ivar' when RDoc::RubyToken::TkOp then 'ruby-operator' when RDoc::RubyToken::TkId then 'ruby-identifier' when RDoc::RubyToken::TkNode then 'ruby-node' when RDoc::RubyToken::TkCOMMENT then 'ruby-comment' when RDoc::RubyToken::TkREGEXP then 'ruby-regexp' when RDoc::RubyToken::TkSTRING then 'ruby-string' when RDoc::RubyToken::TkVal then 'ruby-value' end text = CGI.escapeHTML t.text if style then "<span class=\"#{style}\">#{text}</span>" else text end end.join end
Public Instance Methods
Adds tokens
to the collected tokens
# File lib/rdoc/token_stream.rb, line 58 def add_tokens(*tokens) tokens.flatten.each { |token| @token_stream << token } end
Starts collecting tokens
# File lib/rdoc/token_stream.rb, line 67 def collect_tokens @token_stream = [] end
Remove the last token from the collected tokens
# File lib/rdoc/token_stream.rb, line 76 def pop_token @token_stream.pop end
Current token stream
# File lib/rdoc/token_stream.rb, line 83 def token_stream @token_stream end
Returns a string representation of the token stream
# File lib/rdoc/token_stream.rb, line 90 def tokens_to_s token_stream.map { |token| token.text }.join '' end