Find the answer to your Linux question:
Results 1 to 4 of 4
I'm trying to figure out how to get 'idutils' (mkid & xtokid) to report tokens from inside statically defined strings (C/C++). It only seems to work on strings with a ...
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1
    Linux Newbie pajamabama's Avatar
    Join Date
    Jul 2005
    Location
    In another time's forgotten space
    Posts
    150

    'idutils' - How to find tokens inside strings?


    I'm trying to figure out how to get 'idutils' (mkid & xtokid) to report tokens from inside statically defined strings (C/C++). It only seems to work on strings with a single token.

    For example in
    Code:
    const char* str = "string";
    'xtokid' finds the token "string".

    But in
    Code:
    const char* str = "string with several tokens";
    'xtokid' finds nothing. I'd like it to find each of those individual words as tokens (i.e. "string", "with", "several", & "tokens" are all found).

    Does anyone have experience with this?

    Thanks.
    Last edited by pajamabama; 03-02-2011 at 04:14 PM.
    HP Pavilion dv6000t
    Intel Centrino Duo 2.0GHz
    nVidia GeForce Go 7400
    Fedora 10

    ----------------------
    The real question is what time is it and why the hell am I still screwing around with my computer?

  2. #2
    Linux Guru Rubberman's Avatar
    Join Date
    Apr 2009
    Location
    I can be found either 40 miles west of Chicago, in Chicago, or in a galaxy far, far away.
    Posts
    11,452
    I've had to design and develop some very sophisticated parsing/tokenizing libraries in the past, and never found anything in the standard API's that worked the way we needed it to. Just to note. In fact, I've done this so many times, such as to parse TCL streams, XML, messages used in stock/option trading data streams (VERY high volume data - a colleague once compared them to trying to drink from a fire hose!), etc. in real time, that it is almost second nature. The few times that I've used standard API's such as strtok, et al, I've found limitations that made them unsuitable for production use. Maybe it's just me...

    So, know what you want to do, what data may or may not be in the data stream, and learn how to write a parser. However, make sure you know how to program a finite-state machine before you do if you want it to work reliably!
    Sometimes, real fast is almost as good as real time.
    Just remember, Semper Gumbi - always be flexible!

  3. #3
    Linux Newbie pajamabama's Avatar
    Join Date
    Jul 2005
    Location
    In another time's forgotten space
    Posts
    150
    Nice idea. I think writing a custom FSM parser for generating a symbol table is probably overkill. 'idutils' does the trick for the most part. And someone suggested that parsing as text rather than C/C++ might work. It could be an interesting challenge though.
    HP Pavilion dv6000t
    Intel Centrino Duo 2.0GHz
    nVidia GeForce Go 7400
    Fedora 10

    ----------------------
    The real question is what time is it and why the hell am I still screwing around with my computer?

  4. #4
    Linux Guru Rubberman's Avatar
    Join Date
    Apr 2009
    Location
    I can be found either 40 miles west of Chicago, in Chicago, or in a galaxy far, far away.
    Posts
    11,452
    Well, the KISS principle is a good one to remember. If you don't have to "reinvent the wheel", then stick with Michelin. If the standard API's work for you, great! If not...
    Sometimes, real fast is almost as good as real time.
    Just remember, Semper Gumbi - always be flexible!

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •