add unit tests for tokenize_string() and get_next_token()
It looks like there aren't any. **Trac**: **Username**: cyberpunks
issue