Examples of TokenizerFactory

com.aliasi.tokenizer.TokenizerFactory
com.google.gwt.place.testplaces.TokenizerFactory
Used by tests of {@link com.google.gwt.place.rebind.PlaceHistoryMapperGenerator}.
edu.buffalo.cse.ir.wikiindexer.tokenizer.TokenizerFactory
Factory class to instantiate a Tokenizer instance The expectation is that you need to decide which rules to apply for which field Thus, given a field type, initialize the applicable rules and create the tokenizer @author nikhillo
opennlp.tools.tokenize.TokenizerFactory
The factory that provides {@link Tokenizer} default implementations andresources. Users can extend this class if their application requires overriding the {@link TokenContextGenerator}, {@link Dictionary} etc.
org.apache.lucene.analysis.util.TokenizerFactory
Abstract parent class for analysis factories that create {@link Tokenizer}instances.
org.apache.solr.analysis.TokenizerFactory
A TokenizerFactory breaks up a stream of characters into tokens.
TokenizerFactories are registered for FieldTypes with the IndexSchema through the schema.xml file.

Example schema.xml entry to register a TokenizerFactory implementation to tokenize fields of type "cool"
```
 <fieldtype name="cool" class="solr.TextField"> <analyzer> <tokenizer class="solr.StandardTokenizerFactory"/> ... 
```
A single instance of any registered TokenizerFactory is created via the default constructor and is reused for each FieldType.
@version $Id: TokenizerFactory.java 929782 2010-04-01 02:15:27Z rmuir $

Examples of org.apache.solr.analysis.TokenizerFactory

   * @param analyzerDef The <code>AnalyzerDef</code> annotation as found in the annotated domain class.
   * @return a Lucene <code>Analyzer</code>
   */
  public static Analyzer buildAnalyzer(AnalyzerDef analyzerDef) {
    TokenizerDef token = analyzerDef.tokenizer();
    TokenizerFactory tokenFactory = ( TokenizerFactory ) instantiate( token.factory() );
    tokenFactory.init( getMapOfParameters( token.params() ) );


    final int length = analyzerDef.filters().length;
    TokenFilterFactory[] filters = new TokenFilterFactory[length];
    ResourceLoader resourceLoader = new HibernateSearchResourceLoader();
    for ( int index = 0 ; index < length ; index++ ) {

View Full Code Here

0 1 2 3 4 5

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.