Examples of com.ibm.icu.util.StringTokenizer

com.ibm.icu.util.StringTokenizer
{@icuenhanced java.util.Calendar}. {@icu _usage_}
The string tokenizer class allows an application to break a string into tokens by performing code point comparison. The StringTokenizer methods do not distinguish among identifiers, numbers, and quoted strings, nor do they recognize and skip comments.

The set of delimiters (the codepoints that separate tokens) may be specified either at creation time or on a per-token basis.

An instance of StringTokenizer behaves in one of three ways, depending on whether it was created with the returnDelims and coalesceDelims flags having the value true or false:
- If returnDelims is false, delimiter code points serve to separate tokens. A token is a maximal sequence of consecutive code points that are not delimiters.
- If returnDelims is true, delimiter code points are themselves considered to be tokens. In this case, if coalesceDelims is true, such tokens will be the maximal sequence of consecutive code points that are delimiters. If coalesceDelims is false, a token will be received for each delimiter code point.
A token is thus either one delimiter code point, a maximal sequence of consecutive code points that are delimiters, or a maximal sequence of consecutive code points that are not delimiters.

A StringTokenizer object internally maintains a current position within the string to be tokenized. Some operations advance this current position past the code point processed.

A token is returned by taking a substring of the string that was used to create the StringTokenizer object.

Example of the use of the default delimiter tokenizer.
```
 StringTokenizer st = new StringTokenizer("this is a test"); while (st.hasMoreTokens()) { println(st.nextToken()); } 
```
prints the following output:
```
 this is a test 
```
Example of the use of the tokenizer with user specified delimiter.
```
 StringTokenizer st = new StringTokenizer("this is a test with supplementary characters \ud800\ud800\udc00\udc00", " \ud800\udc00"); while (st.hasMoreTokens()) { println(st.nextToken()); } 
```
prints the following output:
```
 this is a test with supplementary characters \ud800 \udc00 
```
@author syn wee @stable ICU 2.4

    {
        String str = "this\tis\na\rstring\ftesting\tStringTokenizer\nconstructors!";
        String delimiter = " \t\n\r\f";
        String expected[] = {"this", "is", "a", "string", "testing", 
                             "StringTokenizer", "constructors!"};
        StringTokenizer defaultst = new StringTokenizer(str);
        StringTokenizer stdelimiter = new StringTokenizer(str, delimiter);
        StringTokenizer stdelimiterreturn = new StringTokenizer(str, delimiter,
                                                                false);
        UnicodeSet delimiterset = new UnicodeSet("[" + delimiter + "]", false);
        StringTokenizer stdelimiterset = new StringTokenizer(str, delimiterset);
        StringTokenizer stdelimitersetreturn = new StringTokenizer(str, 
                                                                delimiterset,
                                                                false);
        for (int i = 0; i < expected.length; i ++) {
            if (!(defaultst.nextElement().equals(expected[i]) 
                  && stdelimiter.nextElement().equals(expected[i])
                  && stdelimiterreturn.nextElement().equals(expected[i])
                  && stdelimiterset.nextElement().equals(expected[i])
                  && stdelimitersetreturn.nextElement().equals(expected[i]))) {
                errln("Constructor with default delimiter gives wrong results");
            }
        }
        
        String expected1[] = {"this", "\t", "is", "\n", "a", "\r", "string", "\f",
                            "testing", "\t", "StringTokenizer", "\n",
                            "constructors!"};
        stdelimiterreturn = new StringTokenizer(str, delimiter, true);
        stdelimitersetreturn = new StringTokenizer(str, delimiterset, true);
        for (int i = 0; i < expected1.length; i ++) {
            if (!(stdelimiterreturn.nextElement().equals(expected1[i])
                  && stdelimitersetreturn.nextElement().equals(expected1[i]))) {
                errln("Constructor with default delimiter and delimiter tokens gives wrong results");
            }
        }
                            
        stdelimiter = new StringTokenizer(str, (String)null);
        stdelimiterreturn = new StringTokenizer(str, (String)null, false);
        delimiterset = null;
        stdelimiterset = new StringTokenizer(str, delimiterset);
        stdelimitersetreturn = new StringTokenizer(str, delimiterset, false);
        
        if (!(stdelimiter.nextElement().equals(str)
              && stdelimiterreturn.nextElement().equals(str)
              && stdelimiterset.nextElement().equals(str)
              && stdelimitersetreturn.nextElement().equals(str))) {
            errln("Constructor with null delimiter gives wrong results");
        }
        
        delimiter = "";
        stdelimiter = new StringTokenizer(str, delimiter);
        stdelimiterreturn = new StringTokenizer(str, delimiter, false);
        delimiterset = new UnicodeSet();
        stdelimiterset = new StringTokenizer(str, delimiterset);
        stdelimitersetreturn = new StringTokenizer(str, delimiterset, false);
        
        if (!(stdelimiter.nextElement().equals(str)
              && stdelimiterreturn.nextElement().equals(str)
              && stdelimiterset.nextElement().equals(str)
              && stdelimitersetreturn.nextElement().equals(str))) {
            errln("Constructor with empty delimiter gives wrong results");
        }
        
        try {
            defaultst = new StringTokenizer(null);
            errln("null string should throw an exception");
        } catch (Exception e) {
            logln("PASS: Constructor with null string failed as expected");
        }
        try {
            stdelimiter = new StringTokenizer(null, delimiter);
            errln("null string should throw an exception");
        } catch (Exception e) {
            logln("PASS: Constructor with null string failed as expected");
        }
        try {
            stdelimiterreturn = new StringTokenizer(null, delimiter, false);
            errln("null string should throw an exception");
        } catch (Exception e) {
            logln("PASS: Constructor with null string failed as expected");
        }
        try {
            stdelimiterset = new StringTokenizer(null, delimiterset);
            errln("null string should throw an exception");
        } catch (Exception e) {
            logln("PASS: Constructor with null string failed as expected");
        }
        try {
            stdelimitersetreturn = new StringTokenizer(null, delimiterset,
                                                       false);
            errln("null string should throw an exception");
        } catch (Exception e) {
            logln("PASS: Constructor with null string failed as expected");
        }

View Full Code Here

    {
        String str = "bmp string \ud800 with a unmatched surrogate character";
        String delimiter = "\ud800\udc00";
        String expected[] = {str};
            
        StringTokenizer tokenizer = new StringTokenizer(str, delimiter);
        if (!tokenizer.nextElement().equals(expected[0])) {
            errln("Error parsing \"" + Utility.hex(str) + "\"");
        }
        if (tokenizer.hasMoreElements()) {
            errln("Number of tokens exceeded expected");
        }
        delimiter = "\ud800";
        String expected1[] = {"bmp string ", 
                              " with a unmatched surrogate character"};
        tokenizer = new StringTokenizer(str, delimiter);
        int i = 0;
        while (tokenizer.hasMoreElements()) {
            if (!tokenizer.nextElement().equals(expected1[i ++])) {
                errln("Error parsing \"" + Utility.hex(str) + "\"");
            }
        }
        if (tokenizer.hasMoreElements()) {
            errln("Number of tokens exceeded expected");
        }
        
        str = "string \ud800\udc00 with supplementary character";
        delimiter = "\ud800";
        String expected2[] = {str};
        tokenizer = new StringTokenizer(str, delimiter);
        if (!tokenizer.nextElement().equals(expected2[0])) {
            errln("Error parsing \"" + Utility.hex(str) + "\"");
        }
        if (tokenizer.hasMoreElements()) {
            errln("Number of tokens exceeded expected");
        }
  
        delimiter = "\ud800\udc00";
        String expected3[] = {"string ", " with supplementary character"};
        tokenizer = new StringTokenizer(str, delimiter);
        i = 0;
        while (tokenizer.hasMoreElements()) {
            if (!tokenizer.nextElement().equals(expected3[i ++])) {
                errln("Error parsing \"" + Utility.hex(str) + "\"");
            }
        }
        if (tokenizer.hasMoreElements()) {
            errln("Number of tokens exceeded expected");
        }
        
        str = "\ud800 \ud800\udc00 \ud800 \ud800\udc00";
        delimiter = "\ud800";
        String expected4[] = {" \ud800\udc00 ", " \ud800\udc00"};
        i = 0;
        while (tokenizer.hasMoreElements()) {
            if (!tokenizer.nextElement().equals(expected4[i ++])) {
                errln("Error parsing \"" + Utility.hex(str) + "\"");
            }
        }
        if (tokenizer.hasMoreElements()) {
            errln("Number of tokens exceeded expected");
        }
        
        delimiter = "\ud800\udc00";
        String expected5[] = {"\ud800 ", " \ud800 "};
        i = 0;
        while (tokenizer.hasMoreElements()) {
            if (!tokenizer.nextElement().equals(expected5[i ++])) {
                errln("Error parsing \"" + Utility.hex(str) + "\"");
            }
        }
        if (tokenizer.hasMoreElements()) {
            errln("Number of tokens exceeded expected");
        }
    }

View Full Code Here

        String str = "  ,  1 2 3  AHHHHH! 5.5 6 7    ,        8\n";
        String expected[] = {",", "1", "2", "3", "AHHHHH!", "5.5", "6", "7", 
                             ",", "8\n"};
        String delimiter = " ";
                           
        StringTokenizer tokenizer = new StringTokenizer(str, delimiter);
        int currtoken = 0;
        while (tokenizer.hasMoreElements()) {
            if (!tokenizer.nextElement().equals(expected[currtoken])) {
                errln("Error token mismatch, expected " + expected[currtoken]);
            }
            currtoken ++;
        }


        if (currtoken != expected.length) {
            errln("Didn't get correct number of tokens");
        }
        
        tokenizer = new StringTokenizer("", delimiter);
        if (tokenizer.hasMoreElements()) {
            errln("Empty string should not have any tokens");
        }
        try {
            tokenizer.nextElement();
            errln("Empty string should not have any tokens");
        } catch (Exception e) {
            logln("PASS: empty string failed as expected");
        }
        
        tokenizer = new StringTokenizer(", ,", ", ");
        if (tokenizer.hasMoreElements()) {
            errln("String with only delimiters should not have any tokens");
        }
        try {
            tokenizer.nextElement();
            errln("String with only delimiters should not have any tokens");
        } catch (Exception e) {
            logln("PASS: String with only delimiters failed as expected");
        }


        tokenizer = new StringTokenizer("q, ,", ", ");
        if (!tokenizer.hasMoreElements()) {
            errln("String that does not begin with delimiters should have some tokens");
        }
        if (!tokenizer.nextElement().equals("q")) {
            errln("String that does not begin with delimiters should have some tokens");
        } 
        try {
            tokenizer.nextElement();
            errln("String has only one token");
        } catch (Exception e) {
            logln("PASS: String with only one token failed as expected");
        }


        try {
            tokenizer = new StringTokenizer(null, delimiter);
            errln("StringTokenizer constructed with null source should throw a nullpointerexception");
        } catch (Exception e) {
            logln("PASS: StringTokenizer constructed with null source failed as expected");
        }


        tokenizer = new StringTokenizer(str, "q");
        if (!tokenizer.nextElement().equals(str)) {
            errln("Should have received the same string when there are no delimiters");
        }
    }

View Full Code Here

        String expected[] = {"  ", ",", "  ", "1", " ", "2", " ", "3", "  ",
                             "AHHHHH!", " ", "5.5", " ", "6", " ", "7", "    ",
                             ",", "        ", "8\n"};
        String delimiter = " ";
                           
        StringTokenizer tokenizer = new StringTokenizer(str, delimiter, true, true);


        int currtoken = 0;
        while (tokenizer.hasMoreElements()) {
            if (!tokenizer.nextElement().equals(expected[currtoken])) {
                errln("Error token mismatch, expected " + expected[currtoken]);
            }
            currtoken ++;
        }


        if (currtoken != expected.length) {
            errln("Didn't get correct number of tokens");
        }
        
        tokenizer = new StringTokenizer("", delimiter, true);
        if (tokenizer.hasMoreElements()) {
            errln("Empty string should not have any tokens");
        }
        try {
            tokenizer.nextElement();
            errln("Empty string should not have any tokens");
        } catch (Exception e) {
            logln("PASS: Empty string failed as expected");
        }
        
        tokenizer = new StringTokenizer(", ,", ", ", true, true);
        if (!tokenizer.hasMoreElements()) {
            errln("String with only delimiters should have tokens when delimiter is treated as tokens");
        }
        if (!tokenizer.nextElement().equals(", ,")) {
            errln("String with only delimiters should return itself when delimiter is treated as tokens");
        }


        tokenizer = new StringTokenizer("q, ,", ", ", true, true);
        
        if (!tokenizer.hasMoreElements()) {
            errln("String should have some tokens");
        }
        if (!tokenizer.nextElement().equals("q") 
            || !tokenizer.nextElement().equals(", ,")) {
            errln("String tokens do not match expected results");
        } 


        try {
            tokenizer = new StringTokenizer(null, delimiter, true);
            errln("StringTokenizer constructed with null source should throw a nullpointerexception");
        } catch (Exception e) {
            logln("PASS: StringTokenizer constructed with null source failed as expected");
        }


        tokenizer = new StringTokenizer(str, "q", true);
        if (!tokenizer.nextElement().equals(str)) {
            errln("Should have recieved the same string when there are no delimiters");
        }
    }

View Full Code Here

        String expected[] = {"this", "is", "a", "string", "testing", 
                             "StringTokenizer", "constructors!"};
        String expectedreturn[] = {"this", "\t", "is", "\n", "a", "\r", 
                                   "string", "\f", "testing", "\t", 
                                   "StringTokenizer", "\n", "constructors!"};
        StringTokenizer st = new StringTokenizer(str, delimiter);
        StringTokenizer streturn = new StringTokenizer(str, delimiter, true);
        if (st.countTokens() != expected.length) {
            errln("CountTokens failed for non-delimiter tokens");
        }
        if (streturn.countTokens() != expectedreturn.length) {
            errln("CountTokens failed for delimiter tokens");
        }
        for (int i = 0; i < expected.length; i ++) {
            if (!st.nextElement().equals(expected[i])
                || st.countTokens() != expected.length - i - 1) {
                errln("CountTokens default delimiter gives wrong results");
            }
        }
        for (int i = 0; i < expectedreturn.length; i ++) {
            if (!streturn.nextElement().equals(expectedreturn[i])
                || streturn.countTokens() != expectedreturn.length - i - 1) {
                errln("CountTokens with default delimiter and delimiter tokens gives wrong results");
            }
        }    
    }

View Full Code Here

        ICUResourceBundle sub = null;
        if (requested == null) {
            requested = actualBundle;
        }
        while (actualBundle != null) {
            StringTokenizer st = new StringTokenizer(path, "/");
            ICUResourceBundle current = (ICUResourceBundle) actualBundle;
            while (st.hasMoreTokens()) {
                String subKey = st.nextToken();
                sub =  (ICUResourceBundle)current.handleGet(subKey, null, requested);
                if (sub == null) {
                    break;
                }
                current = sub;

View Full Code Here

                               {"def", "stu", "hij"},
                               {"ghi", "vwx", "klm"},
                               {"jkl", "yza", "nop"},
                               {"mno", "bcd", "qrs"}
                              };
        StringTokenizer st = new StringTokenizer(str);
        int size = expected[0].length;
        for (int i = 0; i < size; i ++) {
            for (int j = 0; j < delimiter.length; j ++) {
                if (!st.nextToken(delimiter[j]).equals(expected[j][i])) {
                    errln("nextToken() with delimiters error " + i + " " + j);
                }
                if (st.countTokens() != expected[j].length - i) {            
                    errln("countTokens() after nextToken() with delimiters error"
                          + i + " " + j);
                }
            }
        }    
        st = new StringTokenizer(str);
        String delimiter1[] = {"0", "2", "4"};
        String expected1[] = {"abc", "def1ghi", "jkl3mno", "pqr", "stu1vwx", 
                              "yza3bcd", "efg", "hij1klm", "nop3qrs", "tuv"};
        for (int i = 0; i < expected1.length; i ++) {
            if (!st.nextToken(delimiter1[i % delimiter1.length]).equals(
                                                            expected1[i])) {
                errln("nextToken() with delimiters error " + i);
            }
        }
    }

View Full Code Here

            } else {
                bundle = (ICUResourceBundle) getBundleInstance(bundleName, locale,
                         loaderToUse, false);
            }
            if (keyPath != null) {
                StringTokenizer st = new StringTokenizer(keyPath, "/");
                ICUResourceBundle current = bundle;
                while (st.hasMoreTokens()) {
                    String subKey = st.nextToken();
                    sub = (ICUResourceBundle)((ICUResourceBundle) current).get(subKey, table, requested);
                    if (sub == null) {
                        break;
                    }
                    current = sub;

View Full Code Here

    public void TestBug4423()
    {
        // bug 4423:  a bad interaction between countTokens() and hasMoreTokens().
        //
        String s1 = "This is a test";
        StringTokenizer tzr = new StringTokenizer(s1);
        int  tokenCount = 0;
        
        int t = tzr.countTokens();
        if (t!= 4) {
            errln("tzr.countTokens() returned " + t + ".  Expected 4");
        }
        while (tzr.hasMoreTokens()) {
            String  tok = tzr.nextToken();
            if (tok.length() == 0) {
                errln("token with length == 0");
            }
            tokenCount++;
        }
        if (tokenCount != 4) {
            errln("Incorrect number of tokens found = " + tokenCount);
        }
        
        // Precomputed tokens arrays can grow.  Check for edge cases around
        //  boundary where growth is forced.  Array grows in increments of 100 tokens.
        String s2 = "";
        for (int i=1; i<250; i++) {
            s2 = s2 + " " + i;
            StringTokenizer tzb = new StringTokenizer(s2);
            int t2 = tzb.countTokens();
            if (t2 != i) {
                errln("tzb.countTokens() returned " + t + ".  Expected " + i);
                break;
            }
            int j = 0;
            while (tzb.hasMoreTokens()) {
                String tok = tzb.nextToken();
                j++;
                if (tok.equals(Integer.toString(j)) == false) {
                    errln("Wrong token string.  Expected \"" + j + "\", got \""
                            + tok + "\".");
                    break;

View Full Code Here


    public void TestCountTokensNoCoalesce() {
        // jitterbug 5207
        String str = "\"\"";
        String del = "\"";
        StringTokenizer st = new StringTokenizer(str, del, true);
        int count = 0;
        while (st.hasMoreTokens()) {
            String t = st.nextToken();
            logln("[" + count + "] '" + t + "'");
            ++count;
        }
        st = new StringTokenizer(str, del, true);
        int ncount = st.countTokens();
        int xcount = 0;
        while (st.hasMoreTokens()) {
            String t = st.nextToken();
            logln("[" + xcount + "] '" + t + "'");
            ++xcount;
        }
        if (count != ncount || count != xcount) {
            errln("inconsistent counts " + count + ", " + ncount + ", " + xcount);

View Full Code Here

0 1 2 3 4 5 6 7 8 9

TOP

Related Classes of com.ibm.icu.util.StringTokenizer

com.hp.gloze.www_w3_org_2001_XMLSchema.list

com.hp.gloze.www_w3_org_2001_XMLSchema.schema

com.hp.gloze.www_w3_org_2001_XMLSchema.union

com.hp.hpl.jena.graph.query.test.TestQueryTriple

com.ibm.icu.dev.test.util.StringTokenizerTest

com.ibm.icu.impl.ICUResourceBundle

com.ibm.icu.impl.ICUResourceBundleImpl

com.ibm.icu.text.UnicodeSet

org.eclipse.jst.jsp.core.internal.contenttype.JSPResourceEncodingDetector

org.eclipse.jst.jsp.core.internal.document.PageDirectiveAdapterImpl

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.