Examples of GermanCompoundTokenizer

org.languagetool.tokenizers.de.GermanCompoundTokenizer
Split German nouns using the jWordSplitter library. @author Daniel Naber

Examples of org.languagetool.tokenizers.de.GermanCompoundTokenizer

    }
    if (manualTagger == null) {
      manualTagger = new ManualTagger(JLanguageTool.getDataBroker().getFromResourceDirAsStream(USER_DICT_FILENAME));
    }
    if (compoundTokenizer == null) {
      compoundTokenizer = new GermanCompoundTokenizer();
    }


    for (String word: sentenceTokens) {
      final List<AnalyzedGermanToken> l = new ArrayList<>();
      taggerTokens = lexiconLookup(word);

View Full Code Here

Examples of org.languagetool.tokenizers.de.GermanCompoundTokenizer


  protected void initialize() throws IOException {
    final URL url = JLanguageTool.getDataBroker().getFromResourceDirAsUrl(DICT_FILENAME);
    dictionary = Dictionary.read(url);
    manualTagger = new ManualTagger(JLanguageTool.getDataBroker().getFromResourceDirAsStream(USER_DICT_FILENAME));
    compoundTokenizer = new GermanCompoundTokenizer();
  }

View Full Code Here

Examples of org.languagetool.tokenizers.de.GermanCompoundTokenizer


  protected void initializeIfRequired() throws IOException {
    // Lazy initialize all fields when needed and only once.
    Dictionary dict = dictionary;
    ManualTagger mTagger = manualTagger;
    GermanCompoundTokenizer gTokenizer = compoundTokenizer;
    if (dict == null || mTagger == null || gTokenizer == null) {
      synchronized (this) {
        dict = dictionary;
        mTagger = manualTagger;
        gTokenizer = compoundTokenizer;

View Full Code Here

Examples of org.languagetool.tokenizers.de.GermanCompoundTokenizer

   * @since 2.7
   */
  public GermanCompoundTokenizer getStrictCompoundTokenizer() {
    if (strictCompoundTokenizer == null) {
      try {
        strictCompoundTokenizer = new GermanCompoundTokenizer();
      } catch (IOException e) {
        throw new RuntimeException("Could not set up strict German compound splitter", e);
      }
    }
    return strictCompoundTokenizer;

View Full Code Here

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.