Re: Does StreamTokenizer.pushBack correctly update TT_EOF ?
On Sep 20, 7:44 pm, "John B. Matthews" <nos...@nospam.invalid> wrote:
In article
<e458e6e8-7543-4913-bf9f-865022e65...@r15g2000prh.googlegroups.com>, en=
okaco...@gmail.com wrote:
[...]
I have to say this still seems counterintuitive. What's left to do af=
ter
seeing TT_EOF?
[...]
You see, if I push back something near the end of a loop I want it
processed at the start of the next loop. If TT_EOF is not set
correctly the last token will not be processed correctly.
This is what I tried to resolve with the logic I described earlier.
I am still having my original problem though !
Trying to debug this I tried a small test and found that - nextToken()
immediately followed by pushBack() does not give the same result as
not having both statements !!
Any thoughts on this ?
Yes, the effect of pushBack() is limited to giving the same result from
nextToken(); nval and sval are unchanged. The mechanism seems intended
to provide inexpensive, one-token look-ahead, rather than unlimited undo.
<code>
import java.io.*;
public class Tokenizer {
public static void main(String[] args) throws IOException {
StreamTokenizer tokens = new
StreamTokenizer(new StringReader("Test"));
int token;
token = tokens.nextToken();
print(tokens, token);
token = tokens.nextToken();
print(tokens, token);
tokens.pushBack();
token = tokens.nextToken();
print(tokens, token);
}
private static void print(StreamTokenizer tokens, int token) {
System.out.println((token == StreamTokenizer.TT_EOF)
+ " " + tokens.sval);
}}
</code>
<console>
false Test
true null
true null
</console>
Here's a more elaborate example:
<http://sites.google.com/site/drjohnbmatthews/enumerated-functions>
Notice how encountering TT_EOF (defined as Symbol.END) signals
successful parsing; any other result represents a (not otherwise
identified) syntax error.
--
John B. Matthews
trashgod at gmail dot com
home dot woh dot rr dot com slash jbmatthews- Hide quoted text -
- Show quoted text -
Many thanks once again for the code snippet and reference.
The snippet of course works as expected.
My issue is that I thought my code should give the identical output
either if
a) I insert a readToken() immediately followed by a pushBack() or
b) with neither of these statements
(all else being the same).
I find that this is not the case.
After further debugging I removed
while (sis.ttype != StreamTokenizer.TT_EOF) and put in a
while (!EOF)
where EOF is set if (sis.ttype == StreamTokenizer.TT_EOF)
immediately after the "chief" nextToken() - which is not subject to
pushBack().
This solved the problem !!
By the way, I visited your web site and was intrigued to find that you
are a doctor of medicine in addition to being a computer expert !
Chris