Module python_ast::lexer [−] [src]

Transform bytes into tokens

Structs

Lexer

Struct that provides the operations to take a slice of bytes and convert them into Tk tokens.

Enums

LexResult

Holds the result of parsing functions

Constants

_IMPL_SERIALIZE_FOR_Lexer

Traits

Ident

The Ident trait and related function ident() are special forms if nom::alphanum to handle the special rules around identifiers instead of creating a more complex ident parser.

Functions

as_keyword	Use a hardcoded match expression to match if a string is a keyword.
endline
error_marker
ident	Recognizes a python identifier in a form defined by the regular expression `[_a-zA-Z][_a-zA-Z0-9]*`
identifier
number
operator
space
string
sublex_bin
sublex_bprefix_string
sublex_comment_string
sublex_complex
sublex_dquote_string_u8
sublex_float
sublex_fprefix_string
sublex_hex
sublex_octal
sublex_prefixed_string_u8
sublex_rprefix_string
sublex_squote_string_u8
sublex_string
sublex_string_u8
symbol
tokenize_bytes