Package totalcross.io

Class TokenReader

  • public class TokenReader
extends LineReader
    Used to read an array of tokens in a line ending with \r\n (enter/linefeed) from a stream. Consecutive newlines are skipped. This class does not work well with multi-byte characters when the second byte contains the delimiter or enter/linefeed.

    The usual way to parse a CSV file is: 
     do
   line = readline
   Convert.tokenizeString(line)
    Using this class takes less memory, because the line is read in tokens. For example, suppose a line contains 200 chars, and splitting them contains 10 tokens of 20 chars each. Using the first approach (readline/tokenizeString), the readline will create a string with 200 chars, then that String will be tokenized into 10 smaller strings of 20 chars each.

    Using this class, it will read the 10 tokens of 20 chars each directly, no longer having to create the temporary string of 200 chars.

    The delimiter can be any character except for \r and \n. Note that two consecutive delimiters are considered a single token. So ;a;; is returned as {"","a","",""}.

    Here's a sample that parses the input from a file: 
     TokenReader reader = new TokenReader(new File("text.csv",File.READ_WRITE), ',');
 String[] tokens;
 while ((tokens = reader.readTokens()) != null)
 {
    ... do whatever you want with the tokens.
 }
    And here's another sample that parses from a string: 
     String lines = "a;;;a;\na;;;a; \nb;;b;b;b \nb;;b;b;b;\nb\nb;\n;b\n;\n b ;\n b \n  b \n b  \nb  \n  b";
 String ll[] = Convert.tokenizeString(lines,'\n');
 TokenReader tk = new TokenReader(new ByteArrayStream(lines.getBytes()),';');
 tk.doTrim = true;
 String []line;
 for (int j =0; ((line = tk.readTokens()) != null); j++)
 {
    Vm.debug('"'+ll[j]+'"');
    for (int i =0; i < line.length; i++)
       Vm.debug(i+": '"+line[i]+"'");
    Vm.debug("");
 }
    The output is: 
     "a;;;a;"
 0: 'a'
 1: ''
 2: ''
 3: 'a'
 4: ''
 
 "a;;;a; "
 0: 'a'
 1: ''
 2: ''
 3: 'a'
 4: ''
 
 "b;;b;b;b "
 0: 'b'
 1: ''
 2: 'b'
 3: 'b'
 4: 'b'
 
 "b;;b;b;b;"
 0: 'b'
 1: ''
 2: 'b'
 3: 'b'
 4: 'b'
 5: ''
 
 "b"
 0: 'b'
 
 "b;"
 0: 'b'
 1: ''
 
 ";b"
 0: ''
 1: 'b'
 
 ";"
 0: ''
 1: ''
 
 " b ;"
 0: 'b'
 1: ''
 
 " b "
 0: 'b'
 
 "  b "
 0: 'b'
 
 " b  "
 0: 'b'
 
 "b  "
 0: 'b'
 
 "  b"
 0: 'b'
    Note that this class already uses a buffer for faster detection of the newline and delimiters. Don't use TokenReader with a BufferedStream, it's nonsense and it will throw a warning on the desktop.
    Since:
    TotalCross 1.23

    • Constructor Summary

      Constructors 
      Constructor Description
      TokenReader​(Stream f, char delimiter)
      Constructs a new TokenReader and sets maxTries accordingly to the type of class: 10 if its a Socket or a PortConnector; 0, otherwise.
      TokenReader​(Stream f, char delimiter, byte[] buffer, int start, int len)
      Constructs a new TokenReader and sets maxTries accordingly to the type of class: 10 if its a Socket or a PortConnector, 0 otherwise.

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      java.lang.String readLine()
      Cannot be used; throws a RuntimeException.
      java.lang.String[] readTokens()
      Returns the next tokens available in this stream or null if none.

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Field Detail

      • delimiter

        protected char delimiter

    • Constructor Detail

      • TokenReader

        public TokenReader​(Stream f,
                   char delimiter)
            throws IOException
        Constructs a new TokenReader and sets maxTries accordingly to the type of class: 10 if its a Socket or a PortConnector; 0, otherwise.
        Throws:
        IOException

      • TokenReader

        public TokenReader​(Stream f,
                   char delimiter,
                   byte[] buffer,
                   int start,
                   int len)
            throws IOException
        Constructs a new TokenReader and sets maxTries accordingly to the type of class: 10 if its a Socket or a PortConnector, 0 otherwise. The given buffer contents are added to the internal buffer to start reading from them.
        Throws:
        IOException
        Since:
        TotalCross 1.25

    • Method Detail

      • readLine

        public java.lang.String readLine()
        Cannot be used; throws a RuntimeException.
        Overrides:
        readLine in class LineReader
        See Also:
        readTokens()

      • readTokens

        public java.lang.String[] readTokens()
                              throws IOException
        Returns the next tokens available in this stream or null if none. Empty lines are skipped.
        Throws:
        IOException