What is the idiomatic Rust way to write this naive JavaScript token scanner code fragment?
You should become very familiar with all the methods on Iterator; in this case take_while would be highly relevant.
Other parts of the ecosystem that deal with iterators are invaluable, like String::extend.
Extract the logic about what is an identifier character, then you can write that code much simpler: identifier.extend(chars.clone().take_while(is_id_char)).
In most cases, you can use a for loop instead of while let. scan_string is a good example of this. scan is not, because you wish to pass the iterator into further methods inside the loop body, which is not possible with the for syntax.
In addition to the standard library Iterator methods, you should internalize what Itertools provides.
For example, take_while_ref avoids the need to clone the input iterator and then drive the original with nth.
You could chose to accept a generic iterator instead of Chars specifically.




extern crate itertools;

use itertools::Itertools;

fn is_id_char(c: &amp;char) -&gt; bool {
    match *c {
        'a'...'z' | 'A'...'Z' | '0'...'9' =&gt; true,
        _ =&gt; false,
    }
}

fn scan_identifier&lt;I&gt;(first: char, chars: &amp;mut I)
    where I: Iterator&lt;Item = char&gt; + Clone
{
    let mut identifier = String::new();
    identifier.push(first);
    identifier.extend(chars.take_while_ref(is_id_char));

    println!("{}", identifier);
}

fn scan_string&lt;I&gt;(chars: &amp;mut I)
    where I: Iterator&lt;Item = char&gt; + Clone
{
    let string: String = chars.take_while_ref(|&amp;c| c != '\'').collect();

    if Some('\'') != chars.next() {
        println!("unterminated string! '{}", string);
    } else {
        println!("'{}'", string);
    }
}

fn scan(src: &amp;str) {
    let mut chars = src.chars();
    while let Some(c) = chars.next() {
        match c {
            '(' =&gt; println!("("),
            ')' =&gt; println!(")"),
            ';' =&gt; println!(";"),
            '\'' =&gt; scan_string(&amp;mut chars),
            _ =&gt; scan_identifier(c, &amp;mut chars),
        }
    }
}

fn main() {
    let input = "print('Hello, world!');";
    println!("Input:\n\n{}\n\nTokens:\n", input);
    scan(input);
}




I wouldn't write any parsing / tokenizing / scanning that operates on characters, however. Especially since you are converting from a &amp;str to a String, when you could just be returning string slices and avoiding extra allocation.