add unit tests for tokenize_string() and get_next_token()

It looks like there aren't any.

Trac:
Username: cyberpunks

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information