Re: Tokens in a string
On 7 June, 17:33, Victor Bazarov <v.baza...@comcast.invalid> wrote:
On 6/7/2010 12:09 PM, gervaz wrote:
Hi all, I need to search for tokens in a string, can you suggest some
function?
Basically in a string like "This is a test string. It'a test ok?"
Given some token, I would like to split the string, like "This is a" "
string. It's a " " ok?" using "test" as a delimiter.
Any help?
Have you looked at the interface of 'std::string'
(std::basic_string template specialization on char)? There is
'find', there is 'find_first_of', 'find_last_of'... Those
could be just what you need. Essentially you just let the
standard library find the token for you and then split the
original string (by using 'substr') into pieces.
I generally prefer to use algorithms in the standard library for
this: something less to learn, since they're what I also use
with std::vector, etc.; std::find, std::find_if,
std::find_first_of and std::search are probably the most useful.
Once you've got iterators to the start and end of whatever
you're looking for, then the two iterator constructor of
std::string gives you the string. (I'd also create a series of
predicate objects around std::locale, for testing character
categories.)
There are probably ready-to-wear tokenizers out there, too.
I don't have a link ('cept forwww.google.com<g>).
Boost::regex is a a pretty powerful tool for this sort of thing,
and will be part of the next release of the standard.
Otherwise, with a little bit of work, you can use flex pretty
effectively.
--
James Kanze