Struct python_ast::Lexer
[−]
[src]
pub struct Lexer;
Struct that provides the operations to take a slice of bytes and
convert them into Tk
tokens.
Methods
impl Lexer
[src]
fn new() -> Self
fn tokenize<'a>(&self, bytes: &'a [u8]) -> Rc<LexResult<&'a [u8], Vec<Tk<'a>>>>
Deprecated
Convert a slice of bytes into a Rc<Vec<Tk>>
fn tokenize2<'a>(&self, bytes: &'a [u8]) -> LexResult<&'a [u8], Vec<Tk<'a>>>
Convert a slice of bytes into Vec<Tk<'a>>
Trait Implementations
impl Debug for Lexer
[src]
impl Clone for Lexer
[src]
fn clone(&self) -> Lexer
Returns a copy of the value. Read more
fn clone_from(&mut self, source: &Self)
1.0.0
Performs copy-assignment from source
. Read more