'idutils' - How to find tokens inside strings?

[pajamabama]

I'm trying to figure out how to get 'idutils' (mkid & xtokid) to report tokens from inside statically defined strings (C/C++). It only seems to work on strings with a single token.

For example in

const char* str = "string";

'xtokid' finds the token "string".

But in

const char* str = "string with several tokens";

'xtokid' finds nothing. I'd like it to find each of those individual words as tokens (i.e. "string", "with", "several", & "tokens" are all found).

Does anyone have experience with this?

Thanks.

[Rubberman]

I've had to design and develop some very sophisticated parsing/tokenizing libraries in the past, and never found anything in the standard API's that worked the way we needed it to. Just to note. In fact, I've done this so many times, such as to parse TCL streams, XML, messages used in stock/option trading data streams (VERY high volume data - a colleague once compared them to trying to drink from a fire hose!), etc. in real time, that it is almost second nature. The few times that I've used standard API's such as strtok, et al, I've found limitations that made them unsuitable for production use. Maybe it's just me...

So, know what you want to do, what data may or may not be in the data stream, and learn how to write a parser. However, make sure you know how to program a finite-state machine before you do if you want it to work reliably!

[pajamabama]

Nice idea. I think writing a custom FSM parser for generating a symbol table is probably overkill. 'idutils' does the trick for the most part. And someone suggested that parsing as text rather than C/C++ might work. It could be an interesting challenge though.

[Rubberman]

Well, the KISS principle is a good one to remember. If you don't have to "reinvent the wheel", then stick with Michelin. If the standard API's work for you, great! If not...