Examples of Partitioner

com.linkedin.camus.etl.Partitioner
com.netflix.astyanax.partitioner.Partitioner
Base interface for token partitioning utilities @author elandau
com.thinkaurelius.titan.diskstorage.cassandra.AbstractCassandraStoreManager.Partitioner
edu.cmu.sphinx.decoder.search.Partitioner
Partitions a list of tokens according to the token score, used in {@link PartitionActiveListFactory}. This method is supposed to provide O(n) performance so it's more preferable than
eu.admire.gateway.location.Partitioner
org.apache.accumulo.core.util.LocalityGroupUtil.Partitioner
org.apache.hadoop.mapred.Partitioner
Partitions the key space.
Partitioner controls the partitioning of the keys of the intermediate map-outputs. The key (or a subset of the key) is used to derive the partition, typically by a hash function. The total number of partitions is the same as the number of reduce tasks for the job. Hence this controls which of the m reduce tasks the intermediate key (and hence the record) is sent for reduction.
@see Reducer
org.apache.kafka.clients.producer.internals.Partitioner
The default partitioning strategy:
- If a partition is specified in the record, use it
- If no partition is specified but a key is present choose a partition based on a hash of the key
- If no partition or key is present choose a partition in a round-robin fashion
org.apache.sqoop.job.etl.Partitioner
This allows connector to define how input data to be partitioned. The number of data partitions also determines the degree of parallelism.
org.apache.tez.runtime.library.api.Partitioner
{@link Partitioner} is used by the TEZ framework to partition outputkey/value pairs. Partitioner Initialization
The Partitioner class is picked up using the TEZ_RUNTIME_PARTITIONER_CLASS attribute in {@link TezRuntimeConfiguration}TODO NEWTEZ Change construction to first check for a Constructor with a bytep[] payload Partitioners need to provide a single argument ( {@link Configuration}) constructor or a 0 argument constructor. If both exist, preference is given to the single argument constructor. This is primarily for MR support. If using the configuration constructor, TEZ_RUNTIME_NUM_EXPECTED_PARTITIONS will be set in the configuration, to indicate the max number of expected partitions.
org.jamesii.core.distributed.partitioner.Partitioner
Main class of the partitioning framework. Calls the factories creation functions to obtain suitable factories, then uses the factory to get suitable infrastructure analysis -, model analysis - and partitioning algorithms. Created Nov 15, 2004 @author Roland Ewald
org.openpixi.pixi.distributed.partitioning.Partitioner
Interface for partitioning the simulation area to smaller rectangles. FUTURE: implement more elaborate partitioners

Examples of org.apache.kafka.clients.producer.internals.Partitioner

        String jmxPrefix = "kafka.producer." + (clientId.length() > 0 ? clientId + "." : "");
        List<MetricsReporter> reporters = config.getConfiguredInstances(ProducerConfig.METRIC_REPORTER_CLASSES_CONFIG,
                                                                        MetricsReporter.class);
        reporters.add(new JmxReporter(jmxPrefix));
        this.metrics = new Metrics(metricConfig, reporters, time);
        this.partitioner = new Partitioner();
        long retryBackoffMs = config.getLong(ProducerConfig.RETRY_BACKOFF_MS_CONFIG);
        this.metadataFetchTimeoutMs = config.getLong(ProducerConfig.METADATA_FETCH_TIMEOUT_CONFIG);
        this.metadata = new Metadata(retryBackoffMs, config.getLong(ProducerConfig.METADATA_MAX_AGE_CONFIG));
        this.maxRequestSize = config.getInt(ProducerConfig.MAX_REQUEST_SIZE_CONFIG);
        this.totalMemorySize = config.getLong(ProducerConfig.BUFFER_MEMORY_CONFIG);

View Full Code Here

Examples of org.apache.sqoop.job.etl.Partitioner

  public List<InputSplit> getSplits(JobContext context)
      throws IOException, InterruptedException {
    Configuration conf = context.getConfiguration();


    String partitionerName = conf.get(JobConstants.JOB_ETL_PARTITIONER);
    Partitioner partitioner = (Partitioner) ClassUtils.instantiate(partitionerName);


    PrefixContext connectorContext = new PrefixContext(conf, JobConstants.PREFIX_CONNECTOR_CONTEXT);
    Object connectorConnection = ConfigurationUtils.getConfigConnectorConnection(conf);
    Object connectorJob = ConfigurationUtils.getConfigConnectorJob(conf);
    Schema schema = ConfigurationUtils.getConnectorSchema(conf);


    long maxPartitions = conf.getLong(JobConstants.JOB_ETL_EXTRACTOR_NUM, 10);
    PartitionerContext partitionerContext = new PartitionerContext(connectorContext, maxPartitions, schema);


    List<Partition> partitions = partitioner.getPartitions(partitionerContext, connectorConnection, connectorJob);
    List<InputSplit> splits = new LinkedList<InputSplit>();
    for (Partition partition : partitions) {
      LOG.debug("Partition: " + partition);
      SqoopSplit split = new SqoopSplit();
      split.setPartition(partition);

View Full Code Here

Examples of org.apache.sqoop.job.etl.Partitioner

        String.valueOf(START + NUMBER_OF_ROWS - 1));


    ConnectionConfiguration connConf = new ConnectionConfiguration();
    ImportJobConfiguration jobConf = new ImportJobConfiguration();


    Partitioner partitioner = new GenericJdbcImportPartitioner();
    PartitionerContext partitionerContext = new PartitionerContext(context, 5, null);
    List<Partition> partitions = partitioner.getPartitions(partitionerContext, connConf, jobConf);


    verifyResult(partitions, new String[] {
        "-5 <= ICOL AND ICOL < -3",
        "-3 <= ICOL AND ICOL < -1",
        "-1 <= ICOL AND ICOL < 1",

View Full Code Here

Examples of org.apache.sqoop.job.etl.Partitioner

        String.valueOf(START + NUMBER_OF_ROWS - 1));


    ConnectionConfiguration connConf = new ConnectionConfiguration();
    ImportJobConfiguration jobConf = new ImportJobConfiguration();


    Partitioner partitioner = new GenericJdbcImportPartitioner();
    PartitionerContext partitionerContext = new PartitionerContext(context, 3, null);
    List<Partition> partitions = partitioner.getPartitions(partitionerContext, connConf, jobConf);


    verifyResult(partitions, new String[] {
        "-5 <= ICOL AND ICOL < -1",
        "-1 <= ICOL AND ICOL < 2",
        "2 <= ICOL AND ICOL <= 5"

View Full Code Here

Examples of org.apache.sqoop.job.etl.Partitioner

        String.valueOf(START + NUMBER_OF_ROWS - 1));


    ConnectionConfiguration connConf = new ConnectionConfiguration();
    ImportJobConfiguration jobConf = new ImportJobConfiguration();


    Partitioner partitioner = new GenericJdbcImportPartitioner();
    PartitionerContext partitionerContext = new PartitionerContext(context, 13, null);
    List<Partition> partitions = partitioner.getPartitions(partitionerContext, connConf, jobConf);


    verifyResult(partitions, new String[] {
        "-5 <= ICOL AND ICOL < -4",
        "-4 <= ICOL AND ICOL < -3",
        "-3 <= ICOL AND ICOL < -2",

View Full Code Here

Examples of org.apache.sqoop.job.etl.Partitioner

        String.valueOf((double)(START + NUMBER_OF_ROWS - 1)));


    ConnectionConfiguration connConf = new ConnectionConfiguration();
    ImportJobConfiguration jobConf = new ImportJobConfiguration();


    Partitioner partitioner = new GenericJdbcImportPartitioner();
    PartitionerContext partitionerContext = new PartitionerContext(context, 5, null);
    List<Partition> partitions = partitioner.getPartitions(partitionerContext, connConf, jobConf);


    verifyResult(partitions, new String[] {
        "-5.0 <= DCOL AND DCOL < -3.0",
        "-3.0 <= DCOL AND DCOL < -1.0",
        "-1.0 <= DCOL AND DCOL < 1.0",

View Full Code Here

Examples of org.apache.sqoop.job.etl.Partitioner

        String.valueOf((double)(START + NUMBER_OF_ROWS - 1)));


    ConnectionConfiguration connConf = new ConnectionConfiguration();
    ImportJobConfiguration jobConf = new ImportJobConfiguration();


    Partitioner partitioner = new GenericJdbcImportPartitioner();
    PartitionerContext partitionerContext = new PartitionerContext(context, 3, null);
    List<Partition> partitions = partitioner.getPartitions(partitionerContext, connConf, jobConf);


    verifyResult(partitions, new String[] {
        "-5.0 <= DCOL AND DCOL < -1.6666666666666665",
        "-1.6666666666666665 <= DCOL AND DCOL < 1.666666666666667",
        "1.666666666666667 <= DCOL AND DCOL <= 5.0"

View Full Code Here

Examples of org.apache.sqoop.job.etl.Partitioner

    context.setString(GenericJdbcConnectorConstants.CONNECTOR_JDBC_PARTITION_MAXVALUE, String.valueOf(START + NUMBER_OF_ROWS - 1));


    ConnectionConfiguration connConf = new ConnectionConfiguration();
    ImportJobConfiguration jobConf = new ImportJobConfiguration();


    Partitioner partitioner = new GenericJdbcImportPartitioner();
    PartitionerContext partitionerContext = new PartitionerContext(context, 5, null);
    List<Partition> partitions = partitioner.getPartitions(partitionerContext, connConf, jobConf);


    verifyResult(partitions, new String[] {
        "-5 <= ICOL AND ICOL < -3",
        "-3 <= ICOL AND ICOL < -1",
        "-1 <= ICOL AND ICOL < 1",

View Full Code Here

Examples of org.apache.sqoop.job.etl.Partitioner

    context.setString(GenericJdbcConnectorConstants.CONNECTOR_JDBC_PARTITION_MAXVALUE, String.valueOf(new BigDecimal(START + NUMBER_OF_ROWS - 1)));


    ConnectionConfiguration connConf = new ConnectionConfiguration();
    ImportJobConfiguration jobConf = new ImportJobConfiguration();


    Partitioner partitioner = new GenericJdbcImportPartitioner();
    PartitionerContext partitionerContext = new PartitionerContext(context, 3, null);
    List<Partition> partitions = partitioner.getPartitions(partitionerContext, connConf, jobConf);


    verifyResult(partitions, new String[]{
      "-5 <= DCOL AND DCOL < -2",
      "-2 <= DCOL AND DCOL < 1",
      "1 <= DCOL AND DCOL <= 5"

View Full Code Here

Examples of org.apache.sqoop.job.etl.Partitioner

    context.setString(GenericJdbcConnectorConstants.CONNECTOR_JDBC_PARTITION_MAXVALUE, String.valueOf(new BigDecimal(START)));


    ConnectionConfiguration connConf = new ConnectionConfiguration();
    ImportJobConfiguration jobConf = new ImportJobConfiguration();


    Partitioner partitioner = new GenericJdbcImportPartitioner();
    PartitionerContext partitionerContext = new PartitionerContext(context, 3, null);
    List<Partition> partitions = partitioner.getPartitions(partitionerContext, connConf, jobConf);


    verifyResult(partitions, new String[]{
      "DCOL = -5",
    });
  }

View Full Code Here

0 1 2 3 4 5

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.