Struct python_ast::lexer::Lexer

pub struct Lexer;

Struct that provides the operations to take a slice of bytes and convert them into Tk tokens.

Methods

impl Lexer

fn new() -> Self

fn tokenize<'a>(&self, bytes: &'a [u8]) -> Rc<LexResult<&'a [u8], Vec<Tk<'a>>>>

Deprecated

Convert a slice of bytes into a Rc<Vec<Tk>>

fn tokenize2<'a>(&self, bytes: &'a [u8]) -> LexResult<&'a [u8], Vec<Tk<'a>>>

Convert a slice of bytes into Vec<Tk<'a>>

Trait Implementations

impl Debug for Lexer

fn fmt(&self, __arg_0: &mut Formatter) -> Result

Formats the value using the given formatter.

impl Clone for Lexer

fn clone(&self) -> Lexer

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for Lexer