add unit tests for tokenize_string() and get_next_token()

It looks like there aren't any.

Trac:
Username: cyberpunks