Re: C++ strtok
On 24.04.2012 11:46, abcd wrote:
I have a following question regarding strtok function used for string
tokenizing. As I understand, strtok internally uses static variable to
keep track of the string passed to it so that tokens can be searched
based on delimiter.
After the strtok returns NULL, it means that no tokens are available.
What if now strtok is invoked with another string to search for
tokens??
In this case the static state is discarded. Once you passed another
string you cannot continue to tokenize the first one.
What happens to the internal static buffer which was
initialized to the previous string, when is that released??
There is nothing to release. The internal state has fixed size and
refers to the string buffer you supplied at the first call. The state is
globally allocated in the data segment of the C++ runtime.
More exactly, modern thread-safe C++ runtimes allocate the storage for
the internal state of strtok as thread local storage. Otherwise strtok
would be almost useless.
In practice I avoid to use strtok at all.
Firstly, because it is not re-entrant. I.e. you must not parse another
string while you have to complete the first one. This divides the
functions that you are allowed to call from within the parser loop into
the ones that never call strtok and the functions that might call strtok.
While it is trivial to decide this for runtime library functions it
becomes error prone for your own code. E.g. an object method you call
might internally call methods that use strtok. You might not be aware of
that.
Secondly strtok modifies the original string in a C style way. C like
string manipulation should not be used in C++ programs because it is
error prone and often a backdoor for security vulnerabilities. As long
as you do not deal with char* in C++ and you only use const char* the
probability of security vulnerabilities is significantly reduced.
Use strspn and strcspn for C style parsing in C++. They will easily
achieve the same behavior than strtok without it's disadvantages. I.e.
they do not modify the input buffer and the internal state is kept at
the local stack.
strtok is mainly supported for C compatibility by the C++ runtime.
Marcel