Package org.languagetool.tokenizers

Examples of org.languagetool.tokenizers.CompoundWordTokenizer


    if (compoundTokenizer == null) {
      try {
        final AbstractWordSplitter wordSplitter = new GermanWordSplitter(false);
        wordSplitter.setStrictMode(false); // there's a spelling mistake in (at least) one part, so strict mode wouldn't split the word
        ((GermanWordSplitter)wordSplitter).setMinimumWordLength(3);
        compoundTokenizer = new CompoundWordTokenizer() {
          @Override
          public List<String> tokenize(String word) {
            return new ArrayList<>(wordSplitter.splitWord(word));
          }
        };
View Full Code Here

TOP

Related Classes of org.languagetool.tokenizers.CompoundWordTokenizer

Copyright © 2018 www.massapicom. All rights reserved.
All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.