'idutils' - How to find tokens inside strings?
I'm trying to figure out how to get 'idutils' (mkid & xtokid) to report tokens from inside statically defined strings (C/C++). It only seems to work on strings with a single token.
For example in
'xtokid' finds the token "string".
const char* str = "string";
'xtokid' finds nothing. I'd like it to find each of those individual words as tokens (i.e. "string", "with", "several", & "tokens" are all found).
const char* str = "string with several tokens";
Does anyone have experience with this?